Joseph Doherty
2ee84af1c0
Phase 0 of the ClusterClient→gRPC migration
(docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Standalone hardening: it
closes a gap that exists today and is a precondition for moving command/control
onto gRPC in later phases.
T0.1 — delete the ManagementActor ClusterClientReceptionist registration.
It was built for an out-of-cluster CLI that was never written: the shipped CLI
speaks HTTP Basic to /management, which asks the actor in-process through
ManagementActorHolder. Nothing in the repo ever sent to /user/management. The
actor still runs there; only the cross-boundary advertisement is gone. Six
documents claimed the CLI used ClusterClient — including the CLI's own README
"Architecture Notes" — and are corrected here rather than left to rot.
T0.2 — record, do not port, the dead integration-routing path.
IntegrationCallRequest is unwired at BOTH ends: RouteIntegrationCallAsync has
zero callers anywhere, and RegisterLocalHandler(Integration, …) appears only in
a test, so production always answers "Integration handler not available". It is
excluded from the gRPC contract (28 of 29 commands migrate) rather than
enshrined on an additive-only wire format, and deleting it during a
transport migration would mix a behavioural change into a change whose whole
value is that behaviour is identical. See
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.
T0.3 — preshared-key authentication on SiteStreamService.
The service shipped with no auth at all: plaintext h2c, no interceptor, so
anything that could reach a site node's :8083 could open a live data stream or
read audit rows back via PullAuditEvents/PullSiteCalls. ControlPlaneAuthInterceptor
now gates /sitestream.SiteStreamService/ — modeled on LocalDbSyncAuthInterceptor
(constant-time compare, fail-closed, PermissionDenied) but gating a SET of
service prefixes so phases 1A/1B add services rather than interceptors. LocalDb
sync keeps its own separate key: it authenticates the pair partner, not central,
and collapsing the two would make a site's central-facing key also admit writes
into its database.
Keys are per site (SB-GRPC-PSK-<siteId>), never fleet-wide, so a compromised
site yields only its own. Central attaches them through ControlPlaneCredentials,
which binds CallCredentials to the channel — covering unary and streaming
uniformly, and letting the key resolve asynchronously, which a client
interceptor could not do without blocking. All three central→site channel
creation sites go through it (SiteStreamGrpcClient and both audit pull invokers);
the pull invokers' channel caches are re-keyed by (site, endpoint) because
credentials are per-site and bound to the channel.
Two decisions beyond the plan:
* StartupValidator now requires GrpcPsk on Site nodes. The plan specified only
the runtime gate, but fail-closed with no boot check produces a node that
joins, answers heartbeats and reports healthy while refusing every stream,
audit pull and telemetry ingest — silent and total. Same reasoning as the
existing inbound API-key pepper rule.
* Added Communication:SitePsks as a central-side key map. The plan assumed
central would read the store, seeded via a dev KEK; the docker rig
deliberately boots with no master key, so store-only resolution would leave
it unable to dial its own sites. The store stays primary — it is the only
source that can serve a site added at runtime — with the map covering
key-less hosts and one-off pins. Neither source falling back to
"unauthenticated" is the invariant.
T0.4 — dev keys on both rigs and tests.
34 tests. The seven that matter most exercise a real in-process gRPC stack over
TestServer: the unit tests on either side of the wire would both stay green if
the halves disagreed, and gRPC refuses call credentials on a plaintext channel
by default — the UnsafeUseInsecureChannelCallCredentials opt-in is only provable
by making a real call. They confirm correct key passes on unary AND streaming,
wrong key and no-credentials both get PermissionDenied, and an unresolvable key
fails the call with nothing reaching the service.
OPERATIONAL: a site node upgraded to this build without a key will not boot.
That includes the gitignored deploy/wonder-app-vd03/ overlay.
225 lines
19 KiB
Markdown
225 lines
19 KiB
Markdown
# Component: Host
|
||
|
||
## Purpose
|
||
|
||
The Host component is the single deployable executable for the entire ScadaBridge system. The same binary runs on every node — central and site alike. The node's role is determined entirely by configuration (`appsettings.json`), not by which binary is deployed. On central nodes the Host additionally bootstraps ASP.NET Core to serve the Central UI and Inbound API web endpoints.
|
||
|
||
## Location
|
||
|
||
All nodes (central and site).
|
||
|
||
## Responsibilities
|
||
|
||
- Serve as the single entry point (`Program.cs`) for the ScadaBridge process.
|
||
- Read and validate node configuration at startup before any actor system is created.
|
||
- Register the correct set of component services and actors based on the configured node role.
|
||
- Bootstrap the Akka.NET actor system with Remoting, Clustering, and split-brain resolution from a hand-assembled, injection-safe HOCON document (`AkkaHostedService.BuildHocon`; see REQ-HOST-6).
|
||
- Host ASP.NET Core web endpoints on central nodes only.
|
||
- Configure structured logging (Serilog) with environment-specific enrichment.
|
||
- Support running as a Windows Service in production and as a console application during development.
|
||
- Perform graceful shutdown via Akka.NET CoordinatedShutdown when the service is stopped.
|
||
|
||
---
|
||
|
||
## Requirements
|
||
|
||
### REQ-HOST-1: Single Binary Deployment
|
||
|
||
The same compiled binary must be deployable to both central and site nodes. The node's role (Central or Site) is determined solely by configuration values in `appsettings.json` (or environment-specific overrides). There must be no separate build targets, projects, or conditional compilation symbols for central vs. site.
|
||
|
||
### REQ-HOST-2: Role-Based Service Registration
|
||
|
||
At startup the Host must inspect the configured node role and register only the component services appropriate for that role:
|
||
|
||
- **Shared** (both Central and Site): ClusterInfrastructure, Communication, HealthMonitoring, ExternalSystemGateway.
|
||
- **Central only**: TemplateEngine, DeploymentManager, Security, AuditLogging, CentralUI, InboundAPI, ManagementService, NotificationService, NotificationOutbox, SiteCallAudit.
|
||
- **Site only**: SiteRuntime, DataConnectionLayer, StoreAndForward, SiteEventLogging.
|
||
|
||
Components not applicable to the current role must not be registered in the DI container or the Akka.NET actor system.
|
||
|
||
### REQ-HOST-3: Configuration Binding
|
||
|
||
The Host must bind configuration sections from `appsettings.json` to strongly-typed options classes using the .NET **Options pattern** (`IOptions<T>` / `IOptionsSnapshot<T>`). Each component has its own configuration section under `ScadaBridge`, mapped to a dedicated configuration class owned by that component.
|
||
|
||
#### Infrastructure Sections
|
||
|
||
| Section | Options Class | Owner | Contents |
|
||
|---------|--------------|-------|----------|
|
||
| `ScadaBridge:Node` | `NodeOptions` | Host | Role, NodeHostname, SiteId, RemotingPort, GrpcPort (site only, default 8083) |
|
||
| `ScadaBridge:Cluster` | `ClusterOptions` | ClusterInfrastructure | SeedNodes, SplitBrainResolverStrategy, StableAfter, HeartbeatInterval, FailureDetectionThreshold, MinNrOfMembers |
|
||
| `ScadaBridge:Database` | `DatabaseOptions` | Host | Central: ConfigurationDb, MachineDataDb connection strings; Site: SQLite paths |
|
||
|
||
#### Per-Component Sections
|
||
|
||
| Section | Options Class | Owner | Contents |
|
||
|---------|--------------|-------|----------|
|
||
| `ScadaBridge:DataConnection` | `DataConnectionOptions` | Data Connection Layer | ReconnectInterval, TagResolutionRetryInterval, WriteTimeout |
|
||
| `ScadaBridge:StoreAndForward` | `StoreAndForwardOptions` | Store-and-Forward | SqliteDbPath, ReplicationEnabled |
|
||
| `ScadaBridge:HealthMonitoring` | `HealthMonitoringOptions` | Health Monitoring | ReportInterval, OfflineTimeout |
|
||
| `ScadaBridge:SiteEventLog` | `SiteEventLogOptions` | Site Event Logging | RetentionDays, MaxStorageMb, PurgeScheduleCron |
|
||
| `ScadaBridge:Communication` | `CommunicationOptions` | Communication | DeploymentTimeout, LifecycleTimeout, QueryTimeout, TransportHeartbeatInterval, TransportFailureThreshold |
|
||
| `ScadaBridge:Security` | `SecurityOptions` | Security & Auth | LdapServer, LdapPort, LdapUseTls, JwtSigningKey, JwtExpiryMinutes, IdleTimeoutMinutes |
|
||
| `ScadaBridge:InboundApi` | `InboundApiOptions` | Inbound API | DefaultMethodTimeout |
|
||
| `ScadaBridge:Notification` | `NotificationOptions` | Notification Service | (SMTP config is stored in the central config DB, not in appsettings) |
|
||
| `ScadaBridge:NotificationOutbox` | `NotificationOutboxOptions` | Notification Outbox | Dispatcher poll interval, stuck-age threshold, retention window (delivery retry settings reuse the central SMTP configuration) |
|
||
| `ScadaBridge:SiteCallAudit` | `SiteCallAuditOptions` | Site Call Audit | Reconciliation pull interval, stuck-age threshold, retention window |
|
||
| `ScadaBridge:ManagementService` | `ManagementServiceOptions` | Management Service | (Reserved for future configuration) |
|
||
| `ScadaBridge:Logging` | `LoggingOptions` | Host | Serilog sink configuration, log level overrides |
|
||
|
||
#### Convention
|
||
|
||
- Each component defines its own options class (e.g., `DataConnectionOptions`) in its own project. The class is a plain POCO with properties matching the JSON section keys.
|
||
- The Host binds each section during startup via `services.Configure<T>(configuration.GetSection("ScadaBridge:<ComponentName>"))`.
|
||
- Each component's `AddXxx()` extension method accepts `IServiceCollection` and reads its options via `IOptions<T>` — the component never reads `IConfiguration` directly.
|
||
- Options classes live in the component project, not in Commons, because they are component-specific configuration — not shared contracts.
|
||
- Startup validation (REQ-HOST-4) validates all required options before the actor system starts.
|
||
|
||
### REQ-HOST-4: Startup Validation
|
||
|
||
Before the Akka.NET actor system is created, the Host must validate all required configuration values and fail fast with a clear error message if any are missing or invalid. Validation rules include:
|
||
|
||
- `NodeConfiguration.Role` must be a valid `NodeRole` value.
|
||
- `NodeConfiguration.NodeHostname` must not be null or empty.
|
||
- `NodeConfiguration.RemotingPort` must be in valid port range (1–65535).
|
||
- Site nodes must have `GrpcPort` in valid port range (1–65535) and different from `RemotingPort`.
|
||
- Site nodes must have a non-empty `SiteId`.
|
||
- Central nodes must have non-empty `ConfigurationDb` and `MachineDataDb` connection strings.
|
||
- Site nodes must have non-empty SQLite path values. Site nodes do **not** require a `ConfigurationDb` connection string — all configuration is received via artifact deployment and read from local SQLite.
|
||
- At least two seed nodes must be configured.
|
||
|
||
### REQ-HOST-4a: Readiness Gating
|
||
|
||
On central nodes, the ASP.NET Core web endpoints (Central UI, Inbound API) must **not accept traffic** until the node is fully operational:
|
||
|
||
- Akka.NET cluster membership is established.
|
||
- Database connectivity (MS SQL) is verified.
|
||
- Required cluster singletons are running (if applicable).
|
||
|
||
These are implemented as three `Ready`-tagged health checks registered in the Central-role branch of `Program.cs` (so they are naturally role-scoped — site nodes do not run them): `database` (`DatabaseHealthCheck<ScadaBridgeDbContext>`), `akka-cluster` (`AkkaClusterHealthCheck`), and `required-singletons` (`RequiredSingletonsHealthCheck`). The last verifies each *required-always* central singleton is reachable by Asking its local `ClusterSingletonProxy` an `Identify` with a short bounded timeout (~2s, probes run concurrently) and treating a non-null `ActorIdentity.Subject` as reachable; any unreachable required singleton degrades the check to **Unhealthy**, naming it. The required-always set is the five unconditional central singletons: notification-outbox, audit-log-ingest, site-call-audit, audit-log-purge, and site-audit-reconciliation. Feature-gated singletons are the "if applicable" case and are not probed when their feature is off. The check is leadership-agnostic — the proxy reaches the singleton from either central node, so a ready standby still reports ready (readiness must NOT require cluster leadership; that is the `Active` tier's job). During a brief singleton handover the probe may momentarily time out and the node may flap to not-ready, which is correct: a node mid-handover is legitimately not fully ready (no retries are used, to keep readiness polling fast).
|
||
|
||
A standard ASP.NET Core health check endpoint (`/health/ready`) reports readiness status. The load balancer uses this endpoint to determine when to route traffic to the node. During startup or failover, the node returns `503 Service Unavailable` until ready.
|
||
|
||
### REQ-HOST-5: Windows Service Hosting
|
||
|
||
The Host must support running as a Windows Service via `UseWindowsService()`. When launched outside of a Windows Service context (e.g., during development), it must run as a standard console application. No code changes or conditional compilation are required to switch between the two modes.
|
||
|
||
### REQ-HOST-6: Akka.NET Bootstrap
|
||
|
||
The Host bootstraps the Akka.NET actor system from a **hand-assembled, injection-safe HOCON document** (`AkkaHostedService.BuildHocon`) passed to `ActorSystem.Create` — **not** Akka.Hosting's `AkkaConfigurationBuilder`/`AddAkka()`. Every interpolated value is routed through `QuoteHocon`/`DurationHocon` so a hostname, seed URI or strategy containing a quote, backslash or whitespace cannot corrupt the document, and each timing is rendered in milliseconds so sub-second values survive. The bound `ClusterOptions`/`NodeOptions`/`CommunicationOptions` are startup-validated (`ClusterOptionsValidator`, `StartupValidator`) so a misconfiguration fails fast with a key-naming message. The document configures:
|
||
|
||
- **Remoting**: the node's hostname and port from `NodeOptions`.
|
||
- **Clustering**: seed nodes and the node's cluster role(s) from configuration.
|
||
- **Split-Brain Resolver**: the strategy, stable-after and `down-if-alone` from `ClusterOptions`, **with the downing provider named explicitly** (`akka.cluster.downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"`). Akka's default is `NoDowning`, under which the entire SBR section is inert and singletons never migrate on a crash or partition — omitting this one line silently disables failover. This exact omission was arch-review 01's Critical finding; it is now guarded by tests asserting the parsed HOCON key **and** a two-node kill test asserting the behavior (downing + singleton migration).
|
||
- **Coordinated shutdown**: `run-coordinated-shutdown-when-down = on` plus a cluster-leave phase timeout above the singleton drain budget (see REQ-HOST-4a / the down-if-alone recovery contract in Component-ClusterInfrastructure.md).
|
||
- **Actor registration**: each component's actors registered conditional on the node's role.
|
||
|
||
> **Bootstrap-mechanism note.** The `Akka.Hosting` / `Akka.Cluster.Hosting` / `Akka.Remote.Hosting` typed-builder packages were referenced but unused and were dropped (arch-review 01 Task 22). The hand-rolled HOCON path is a single hardened, production-shape code path with full test coverage; migrating to Akka.Hosting's typed options is deliberately **not** done here and is tracked as possible future work rather than carried as an unused dependency. Only `Akka.Cluster.Tools` (ClusterSingleton/ClusterClient, transitively Akka.Cluster) remains.
|
||
|
||
> **Persistence note.** ScadaBridge does not use Akka.Persistence. Durable state
|
||
> (store-and-forward buffers, site event logs, static attribute writes,
|
||
> deployment records, configuration) is owned by the individual components and
|
||
> persisted through component-owned stores — SQLite at sites, MS SQL centrally —
|
||
> not through an Akka journal/snapshot store. The Host therefore configures no
|
||
> `akka.persistence` section and references no persistence plugin. There are no
|
||
> `PersistentActor` subclasses in the system by design.
|
||
|
||
### REQ-HOST-6a: ClusterClientReceptionist (Central Only)
|
||
|
||
On central nodes, the Host must configure the Akka.NET **ClusterClientReceptionist** and register the **CentralCommunicationActor** with it, so that site clusters' ClusterClients can reach the central command/control endpoint without joining the central cluster.
|
||
|
||
**The ManagementActor is NOT registered with the receptionist** (removed 2026-07-22). That registration was written for an out-of-cluster CLI that was never built: the shipped CLI speaks HTTP Basic to the central `/management` endpoints, which ask the ManagementActor **in-process** via `ManagementActorHolder` (`ManagementEndpoints.cs`). No sender to `/user/management` existed anywhere in the repo, so the registration only widened the cluster-client surface for nothing. The actor itself still runs at `/user/management`; only its cross-boundary advertisement is gone.
|
||
|
||
> **Migration note.** This receptionist registration — and the `CentralCommunicationActor` one that remains — are scheduled for deletion once the site↔central transport moves to gRPC. See `docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`.
|
||
|
||
### REQ-HOST-7: ASP.NET Web Endpoints
|
||
|
||
On central nodes, the Host must use `WebApplication.CreateBuilder` to produce a full ASP.NET Core host with Kestrel, and must map web endpoints for:
|
||
|
||
- Central UI (via `MapCentralUI()` extension method).
|
||
- Inbound API (via `MapInboundAPI()` extension method).
|
||
|
||
On site nodes, the Host must also use `WebApplication.CreateBuilder` (not `Host.CreateDefaultBuilder`) to host the **SiteStreamGrpcServer** via Kestrel HTTP/2 on the configured `GrpcPort` (default 8083). Kestrel is configured with `HttpProtocols.Http2` on the gRPC port only — no HTTP/1.1 web endpoints are exposed. The gRPC service is mapped via `MapGrpcService<SiteStreamGrpcServer>()`.
|
||
|
||
**Startup ordering (site nodes)**:
|
||
1. Actor system and SiteStreamManager must be initialized before gRPC begins accepting connections.
|
||
2. The gRPC server rejects streams with `StatusCode.Unavailable` until the actor system is ready.
|
||
|
||
**Shutdown ordering (site nodes)**:
|
||
1. On `CoordinatedShutdown`, stop accepting new gRPC streams first.
|
||
2. Cancel all active gRPC streams (triggering client-side reconnect).
|
||
3. Tear down actors.
|
||
4. Use `IHostApplicationLifetime.ApplicationStopping` to signal the gRPC server.
|
||
|
||
### REQ-HOST-8: Structured Logging
|
||
|
||
The Host must configure Serilog as the logging provider with:
|
||
|
||
- Configuration-driven sink setup (console and file sinks at minimum).
|
||
- Automatic enrichment of every log entry with `SiteId`, `NodeHostname`, and `NodeRole` properties sourced from `NodeConfiguration`.
|
||
- Structured (machine-parseable) output format.
|
||
|
||
### REQ-HOST-8a: Dead Letter Monitoring
|
||
|
||
The Host must subscribe to the Akka.NET `DeadLetter` event stream and log dead letters at Warning level. Dead letters indicate messages sent to actors that no longer exist — a common symptom of failover timing issues, stale actor references, or race conditions during instance lifecycle transitions. The dead letter count is reported as a health metric (see Health Monitoring).
|
||
|
||
### REQ-HOST-9: Graceful Shutdown
|
||
|
||
When the Host process receives a stop signal (Windows Service stop, `Ctrl+C`, or SIGTERM), it must trigger Akka.NET CoordinatedShutdown to allow actors to drain in-flight work before the process exits. The Host must not call `Environment.Exit()` or forcibly terminate the actor system without coordinated shutdown.
|
||
|
||
### REQ-HOST-10: Extension Method Convention
|
||
|
||
Each component library must expose its services to the Host via a consistent set of extension methods:
|
||
|
||
- `IServiceCollection.AddXxx()` — registers the component's DI services.
|
||
- `AkkaConfigurationBuilder.AddXxxActors()` — registers the component's actors with the Akka.NET actor system (for components that have actors).
|
||
- `WebApplication.MapXxx()` — maps the component's web endpoints (only for CentralUI and InboundAPI).
|
||
|
||
The Host's `Program.cs` calls these extension methods; the component libraries own the registration logic. This keeps the Host thin and each component self-contained. The ManagementService component additionally registers the ManagementActor with ClusterClientReceptionist in its `AddManagementServiceActors()` method.
|
||
|
||
---
|
||
|
||
## Component Registration Matrix
|
||
|
||
| Component | Central | Site | DI (`AddXxx`) | Actors (`AddXxxActors`) | Endpoints (`MapXxx`) |
|
||
|---|---|---|---|---|---|
|
||
| ClusterInfrastructure | Yes | Yes | Yes | Yes | No |
|
||
| Communication | Yes | Yes | Yes | Yes | No |
|
||
| HealthMonitoring | Yes | Yes | Yes | Yes | No |
|
||
| ExternalSystemGateway | Yes | Yes | Yes | Yes | No |
|
||
| AuditLog | Yes | Yes | Yes | Yes | No |
|
||
| NotificationService | Yes | No | Yes | Yes | No |
|
||
| NotificationOutbox | Yes | No | Yes | Yes | No |
|
||
| SiteCallAudit | Yes | No | Yes | Yes | No |
|
||
| KpiHistory | Yes | No | Yes | Yes | No |
|
||
| Transport | Yes | No | Yes | No | No |
|
||
| TemplateEngine | Yes | No | Yes | Yes | No |
|
||
| DeploymentManager | Yes | No | Yes | Yes | No |
|
||
| Security | Yes | No | Yes | Yes | No |
|
||
| CentralUI | Yes | No | Yes | No | Yes |
|
||
| InboundAPI | Yes | No | Yes | No | Yes |
|
||
| ManagementService | Yes | No | Yes | Yes | No |
|
||
| SiteRuntime | No | Yes | Yes | Yes | No |
|
||
| DataConnectionLayer | No | Yes | Yes | Yes | No |
|
||
| StoreAndForward | No | Yes | Yes | Yes | No |
|
||
| SiteEventLogging | No | Yes | Yes | Yes | No |
|
||
| ConfigurationDatabase | Yes | No | Yes | No | No |
|
||
|
||
---
|
||
|
||
## Dependencies
|
||
|
||
- **All 19 component libraries**: The Host references every component project to call their extension methods (excludes CLI, which is a separate executable). Audit Log (#23) ships its central+site code in `ZB.MOM.WW.ScadaBridge.AuditLog`; the Host calls `AddAuditLog()` on both roles, M2+ will add `AddAuditLogActors()`.
|
||
- **Akka.Cluster.Tools**: ClusterSingleton (singleton manager/proxy) and ClusterClient/ClusterClientReceptionist; transitively pulls Akka.Cluster (SBR provider) and Akka.Remote. The actor system is built from hand-assembled HOCON (REQ-HOST-6), so the `Akka.Hosting`/`Akka.Remote.Hosting`/`Akka.Cluster.Hosting` typed-builder packages are **not** referenced (dropped in arch-review 01 Task 22). No Akka.Persistence plugin — see the Persistence note under REQ-HOST-6.
|
||
- **Serilog.AspNetCore**: For structured logging integration.
|
||
- **Microsoft.Extensions.Hosting.WindowsServices**: For Windows Service support.
|
||
- **ASP.NET Core** (central only): For web endpoint hosting.
|
||
|
||
## Interactions
|
||
|
||
- **All components**: The Host is the composition root — it wires every component into the DI container and actor system.
|
||
- **Configuration Database**: The Host registers the DbContext and wires repository implementations to their interfaces. In development, triggers auto-migration; in production, validates schema version.
|
||
- **ClusterInfrastructure**: The Host configures the underlying Akka.NET cluster that ClusterInfrastructure manages at runtime.
|
||
- **CentralUI / InboundAPI**: The Host maps their web endpoints into the ASP.NET Core pipeline on central nodes.
|
||
- **ManagementService**: The Host registers the ManagementActor and configures ClusterClientReceptionist on central nodes, enabling CLI access.
|
||
- **HealthMonitoring**: The Host's startup validation and logging configuration provide the foundation for health reporting.
|