diff --git a/docs/components/ClusterInfrastructure.md b/docs/components/ClusterInfrastructure.md index e442017a..f7b6529b 100644 --- a/docs/components/ClusterInfrastructure.md +++ b/docs/components/ClusterInfrastructure.md @@ -1,23 +1,33 @@ # Cluster Infrastructure -The Cluster Infrastructure component manages Akka.NET cluster formation, active/standby failover, split-brain resolution, and the singleton hosting that all other ScadaBridge components depend on. Every site and central cluster is a two-node active/standby pair governed by the same configuration contract and bootstrap logic. +The Cluster Infrastructure component manages Akka.NET cluster formation, active/standby failover, the downing strategy for unreachable members, and the singleton hosting that all other ScadaBridge components depend on. Every site and central cluster is a two-node active/standby pair governed by the same configuration contract and bootstrap logic. ## Overview Cluster Infrastructure (#13) is a **design responsibility** spanning two projects rather than a single buildable project: -- **`src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/`** owns the cluster configuration contract: `ClusterOptions` (seed nodes, failure-detection timings, split-brain settings), `ClusterOptionsValidator`, and the `AddClusterInfrastructure` DI extension that registers the validator. It does not start an actor system. +- **`src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/`** owns the cluster configuration contract: `ClusterOptions` (seed nodes, failure-detection timings, downing strategy), `ClusterOptionsValidator`, and the `AddClusterInfrastructure` DI extension that registers the validator. It does not start an actor system. - **`src/ZB.MOM.WW.ScadaBridge.Host/`** owns the cluster bootstrap and runtime wiring: `AkkaHostedService` builds the Akka HOCON from `ClusterOptions` and `NodeOptions`, starts the `ActorSystem`, wires `CoordinatedShutdown`, and creates all role-specific actors including the cluster singletons. This split is deliberate. The Host is the single deployable binary and the only project that performs Akka.NET bootstrap, so all cluster bring-up lives there. `ClusterInfrastructure` is the portable configuration contract that the Host consumes — it can be referenced by tests and other components without pulling in the Host. -Both central and site clusters run this same topology: two nodes, one active (cluster leader), one standby, with automatic failover and no manual intervention required for dual-node recovery. +Both central and site clusters run this same topology: two nodes, one active (the oldest `Up` member), one standby, with automatic failover and no manual intervention required for dual-node recovery. ## Key Concepts -### Active/standby via cluster leadership +### One `ActorSystem` name for every cluster -Akka.NET cluster leadership determines which node is "active". The cluster leader is the oldest node in the cluster, as tracked by the keep-oldest split-brain resolver. `ActiveNodeGate` (in the Host) exposes `IsActiveNode` by checking whether `cluster.SelfMember.Status == MemberStatus.Up` and `cluster.State.Leader == cluster.SelfAddress`. Cluster singletons — which run on the oldest `Up` member — automatically migrate to the surviving node on failover. +Every node in every cluster — central and all sites — joins an `ActorSystem` named **`"scadabridge"`**, hardcoded at `AkkaHostedService.cs:191` (`ActorSystem.Create("scadabridge", config)`). Central and each site are separate clusters *only* by seed-node partitioning, not by system name. This is required rather than incidental: Akka.Remote matches addresses including the system name, so a `ClusterClient` could not reach a differently-named system. + +### Active/standby is the oldest `Up` member — never the cluster leader + +A node is "active" when it is the **oldest `Up` member** of its role scope — the member `ClusterSingletonManager` places singletons on. Akka's *cluster leader* (lowest address) is a different, Akka-internal concept: it diverges from singleton placement permanently once the original first node restarts and rejoins. Every product-level active/standby decision therefore goes through one evaluator and never reads `cluster.State.Leader`: + +- `ActiveNodeEvaluator.SelfIsOldestUp(Cluster, string? role)` (`Communication/ClusterState/ActiveNodeEvaluator.cs:35`) is the single implementation — self is `Up`, carries the role when one is given, and no other `Up` member in that scope is older (`self.IsOlderThan(m)`). +- `ClusterActivityEvaluator.SelfIsOldest` (`Host/Health/ClusterActivityEvaluator.cs:23`) delegates to it, and is what `ActiveNodeGate.IsActiveNode` (`Host/Health/ActiveNodeGate.cs:48`), `OldestNodeActiveHealthCheck`, and `AkkaClusterNodeProvider.SelfIsPrimary` all call. +- `SiteCommunicationActor` stamps its heartbeat's `IsActive` from the same evaluator (`Communication/Actors/SiteCommunicationActor.cs:517-518`). + +Cluster singletons automatically migrate to the surviving node on failover, and because "active" is defined as the singleton-placement member, the health/routing view and the singleton view can never disagree. ### Configuration contract vs. bootstrap split @@ -33,22 +43,26 @@ Cluster Infrastructure provides the hosting platform; each singleton is owned an `AkkaHostedService.BuildHocon` constructs the Akka HOCON document from the bound options at startup. All interpolated values pass through `QuoteHocon` (string escaping) and `DurationHocon` (millisecond rendering) so the document is never corrupted by hostnames or timing values containing special characters or sub-second precision. -The snippet below is abbreviated to highlight the cluster stanzas. The full method also emits three additional stanzas: `akka.extensions` (registers `DistributedPubSubExtensionProvider`), `akka.remote.dot-netty.tcp` (binds `NodeOptions.NodeHostname` and `NodeOptions.RemotingPort`), and `akka.remote.transport-failure-detector` (heartbeat interval and acceptable-heartbeat-pause from `CommunicationOptions.TransportHeartbeatInterval` / `TransportFailureThreshold`). +The snippet below is abbreviated to highlight the cluster stanzas. The full method also emits `akka.extensions` (registers `DistributedPubSubExtensionProvider`), `akka.remote.dot-netty.tcp` (binds `NodeOptions.NodeHostname` and `NodeOptions.RemotingPort`), and `akka.remote.transport-failure-detector` (heartbeat interval and acceptable-heartbeat-pause from `CommunicationOptions.TransportHeartbeatInterval` / `TransportFailureThreshold`). + +The downing block is **not** a fixed stanza — `BuildHocon` branches on `ClusterOptions.SplitBrainResolverStrategy` and emits one of two shapes (`AkkaHostedService.cs:275-286`): ```csharp // Abbreviated — see AkkaHostedService.BuildHocon for the full method. -public static string BuildHocon( - NodeOptions nodeOptions, - ClusterOptions clusterOptions, - IEnumerable roles, - TimeSpan transportHeartbeat, - TimeSpan transportFailure) -{ - var seedNodesStr = string.Join(",", - clusterOptions.SeedNodes.Select(QuoteHocon)); - var rolesStr = string.Join(",", roles.Select(QuoteHocon)); +var downingBlock = string.Equals( + clusterOptions.SplitBrainResolverStrategy, "auto-down", StringComparison.OrdinalIgnoreCase) + ? $@"downing-provider-class = ""Akka.Cluster.AutoDowning, Akka.Cluster"" +auto-down-unreachable-after = {DurationHocon(clusterOptions.StableAfter)}" + : $@"downing-provider-class = ""Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"" +split-brain-resolver {{ + active-strategy = {QuoteHocon(clusterOptions.SplitBrainResolverStrategy)} + stable-after = {DurationHocon(clusterOptions.StableAfter)} + keep-oldest {{ + down-if-alone = {(clusterOptions.DownIfAlone ? "on" : "off")} + }} +}}"; - return $@" +return $@" audit-telemetry-dispatcher {{ type = ForkJoinDispatcher throughput = 100 @@ -66,13 +80,7 @@ akka {{ seed-nodes = [{seedNodesStr}] roles = [{rolesStr}] min-nr-of-members = {clusterOptions.MinNrOfMembers} - split-brain-resolver {{ - active-strategy = {QuoteHocon(clusterOptions.SplitBrainResolverStrategy)} - stable-after = {DurationHocon(clusterOptions.StableAfter)} - keep-oldest {{ - down-if-alone = {(clusterOptions.DownIfAlone ? "on" : "off")} - }} - }} + {downingBlock} failure-detector {{ heartbeat-interval = {DurationHocon(clusterOptions.HeartbeatInterval)} acceptable-heartbeat-pause = {DurationHocon(clusterOptions.FailureDetectionThreshold)} @@ -83,23 +91,35 @@ akka {{ run-by-clr-shutdown-hook = on }} }}"; -} ``` +A `downing-provider-class` is always named explicitly. Akka defaults to `NoDowning`, under which the downing configuration is inert and singletons never migrate on a hard crash or partition; naming the provider is what activates automatic downing. + The HOCON also defines the `audit-telemetry-dispatcher` (a two-thread `ForkJoinDispatcher`) so `SiteAuditTelemetryActor`'s SQLite reads and gRPC pushes never contend with the default dispatcher used by hot-path actors. -### Split-brain resolution +Nothing in the emitted document enables remoting TLS or an Akka secure cookie — there is no `enable-ssl`, no `require-cookie`, no `trusted-selection-paths`. Akka remoting between nodes and from a `ClusterClient` is plaintext and unauthenticated; the deployment is assumed to sit on a trusted network. -The keep-oldest strategy is the only strategy `ClusterOptionsValidator` permits for ScadaBridge's two-node clusters. Quorum strategies (`keep-majority`, `static-quorum`) cannot distinguish a crash from a partition with two nodes — both sides would be below quorum and both would shut down. Keep-oldest with `down-if-alone = on` ensures at most one node runs the cluster at any time: +### Downing strategy (auto-down — availability-first) -- On a network partition, the older node stays active; the younger node downs itself. -- If the oldest node finds itself alone (no reachable members), it downs itself rather than running in isolation. Without `down-if-alone`, the oldest node could run as a single-node cluster while the younger node forms its own — producing two live clusters with divergent singleton state. +**Decision 2026-07-21** (`docs/plans/2026-07-21-auto-down-availability-decision.md`): the default strategy is **`auto-down`** — Akka's `AutoDowning` provider with `auto-down-unreachable-after` = `StableAfter` (15 s). The leader among the *reachable* members downs the unreachable peer once the stability window elapses. + +- **Either-node crash is survivable.** If the standby crashes, the active node downs it and continues. If the **active/oldest** node crashes, the younger survivor downs the dead oldest, becomes the oldest itself, re-hosts every cluster singleton, and `/health/active` flips to it — no operator action and no victim restart. +- **The accepted trade is dual-active during a real network partition.** With both nodes alive but the link cut, each side downs the other and continues as a one-node cluster; both claim active until an operator restarts one side after the partition heals. This was chosen deliberately — pairs run one node per VM with no shared lease store (no Kubernetes, no site-side SQL) to arbitrate, and a stalled system is a bigger operational risk than a rare LAN partition. +- **`StableAfter` is the debounce**, not a resolver phase: 15 s of sustained unreachability before downing, which absorbs startup, rolling restarts, and transient blips. + +`keep-oldest` remains a supported value (`ClusterOptionsValidator` allows exactly `auto-down` and `keep-oldest`) for deployments that prefer partition-safety, but it **cannot survive a crash of the oldest node in a two-node cluster**: Akka's `down-if-alone` only rescues the survivor when its own side has ≥ 2 members, so a 1-vs-1 survivor takes `DownReachable` and downs *itself*. Quorum strategies are rejected outright — `static-quorum` with quorum 1 trips Akka's `IsTooManyMembers` guard and downs *all* members on any unreachability, and `keep-majority` merely moves the fatal crash from the oldest node to the lowest-address node. + +### Downed-node recovery + +`run-coordinated-shutdown-when-down = on` means a downed node runs `CoordinatedShutdown` and terminates its own `ActorSystem`. The Host watches `ActorSystem.WhenTerminated`; a termination that is not the host's own `StopAsync` calls `IHostApplicationLifetime.StopApplication()` so the process exits and the service supervisor (docker `restart: unless-stopped`, Windows service recovery) restarts it as a fresh incarnation (`AkkaHostedService.cs:203-218`). + +**Seed-node bootstrap constraint.** Only the *first* seed listed in `Cluster:SeedNodes` may self-join to form a new cluster, and all nodes list the same first seed. Under `auto-down` this no longer causes an active-crash outage — the survivor keeps running and never restarts — but it still bites when a node must **boot alone** (a cold start of only the non-first-seed VM, or the survivor crashing while its peer is still dead): that node loops on `InitJoin` until its peer returns. Recovery is operator-driven. ### Failure detection and failover timeline Detection uses two independent Akka heartbeat channels: -- **Cluster failure detector** (`akka.cluster.failure-detector`): monitors membership, triggers `Unreachable` events that the split-brain resolver acts on. +- **Cluster failure detector** (`akka.cluster.failure-detector`): monitors membership, triggers the `Unreachable` events the downing provider acts on. - **Transport failure detector** (`akka.remote.transport-failure-detector`): monitors the underlying TCP transport between nodes; configured separately from `CommunicationOptions.TransportHeartbeatInterval` / `TransportFailureThreshold`. With the defaults in `ClusterOptions`, the total failover budget is approximately 25 seconds: @@ -107,28 +127,33 @@ With the defaults in `ClusterOptions`, the total failover budget is approximatel | Phase | Duration | Source | |-------|----------|--------| | Failure detection (`acceptable-heartbeat-pause`) | 10 s | `ClusterOptions.FailureDetectionThreshold` | -| Split-brain stable-after | 15 s | `ClusterOptions.StableAfter` | +| Downing window (`auto-down-unreachable-after`) | 15 s | `ClusterOptions.StableAfter` | | Singleton restart | < 1 s | Actor `PreStart` | +The docker failover drill (`docker/failover-drill.sh`) measures both directions — `standby` mode kills the younger node, `active` mode kills the active/oldest node and asserts the survivor takes over while the victim is still down. + ### Graceful shutdown and singleton handover -When a node is stopped cleanly, `CoordinatedShutdown` runs before the CLR exits (`run-by-clr-shutdown-hook = on`). The cluster-leave phase signals Akka to migrate singletons before the actor system terminates, so handover happens in seconds rather than waiting for the full failure-detection timeout. `SiteCallAuditActor` has an explicit graceful-stop task registered on `PhaseClusterLeave` with a 10-second timeout to drain any in-flight EF Core upsert before handover opens: +When a node is stopped cleanly, `CoordinatedShutdown` runs before the CLR exits (`run-by-clr-shutdown-hook = on`). The cluster-leave phase signals Akka to migrate singletons before the actor system terminates, so handover happens in seconds rather than waiting for the full failure-detection timeout. + +Every singleton is created through the shared `SingletonRegistrar.Start` helper (`Host/Actors/SingletonRegistrar.cs`), so the drain is uniform rather than per-singleton boilerplate. The registrar applies the canonical `{name}-singleton` / `{name}-proxy` naming, a `PoisonPill` termination message, an optional `.WithRole(role)` on both the manager and proxy settings, and a `PhaseClusterLeave` task that `GracefulStop`s the manager (10-second default) so in-flight EF Core (central) or SQLite (site) work completes before handover opens: ```csharp -siteCallAuditShutdown.AddTask( +// SingletonRegistrar.Start — the drain task registered for every singleton +Akka.Actor.CoordinatedShutdown.Get(system).AddTask( Akka.Actor.CoordinatedShutdown.PhaseClusterLeave, - "drain-site-call-audit-singleton", + $"drain-{name}-singleton", async () => { try { - await siteCallAuditSingletonManager.GracefulStop(TimeSpan.FromSeconds(10)); + await manager.GracefulStop(timeout); } catch (Exception ex) { - _logger.LogWarning(ex, - "SiteCallAudit singleton did not drain within the graceful-stop " - + "timeout; falling through to PoisonPill handover"); + logger.LogWarning(ex, + "{Singleton} singleton did not drain within the graceful-stop timeout; " + + "falling through to PoisonPill handover", name); } return Akka.Done.Instance; }); @@ -136,25 +161,29 @@ siteCallAuditShutdown.AddTask( ### Cluster roles and singleton scoping -Each node carries one or more cluster roles set in the HOCON `roles` list. Site nodes carry both a base `"Site"` role and a site-specific role (`"site-{SiteId}"`, e.g. `"site-site-a"`). Singletons on site clusters are scoped to the site-specific role so each site's singleton runs on exactly one node of that site's cluster, not on any other site's nodes. Central singletons use no role scope — all central nodes share the `"Central"` role. +Each node carries one or more cluster roles set in the HOCON `roles` list, built by `AkkaHostedService.BuildRoles` (`AkkaHostedService.cs:406-417`). Site nodes carry **two** roles: the base `"Site"` role plus a site-specific `"site-{SiteId}"` (a node with `SiteId: "site-a"` gets `"site-site-a"`). Singletons on site clusters are scoped to the site-specific role so each site's singleton runs on exactly one node of that site's cluster. Central singletons pass no role to the registrar and so are unscoped — all central nodes share the `"Central"` role. ### Dual-node recovery -Because both nodes are configured as seed nodes, whichever node starts first after a simultaneous failure forms a new cluster; the second joins when it comes up. No startup ordering dependency exists, and no manual intervention is required. The keep-oldest resolver handles the "both starting fresh" case naturally — there is no pre-existing cluster to conflict with. +Because both nodes are configured as seed nodes, whichever node starts first after a simultaneous failure forms a new cluster; the second joins when it comes up. There is no pre-existing cluster to conflict with, so the "both starting fresh" case needs no downing decision at all. The one ordering dependency that does exist is the seed-node bootstrap constraint above: a node that must boot *alone* forms a cluster only if it is the first seed listed. ### Cluster singletons hosted -The Host wires the following singletons. Cluster Infrastructure provides the `ClusterSingletonManager` / `ClusterSingletonProxy` pattern; each singleton's behaviour is documented in the owning component. +The Host wires the following singletons through `SingletonRegistrar.Start`. Cluster Infrastructure provides the `ClusterSingletonManager` / `ClusterSingletonProxy` pattern and the drain hook; each singleton's behaviour is documented in the owning component. -**Central singletons (active central node, no role scope):** +**Central singletons (oldest `Up` central node, no role scope):** | Singleton name | Actor class | Owner | |----------------|-------------|-------| | `notification-outbox` | `NotificationOutboxActor` | Notification Outbox (#21) | | `audit-log-ingest` | `AuditLogIngestActor` | Audit Log (#23) | | `site-call-audit` | `SiteCallAuditActor` | Site Call Audit (#22) | +| `audit-log-purge` | `AuditLogPurgeActor` | Audit Log (#23) | +| `site-audit-reconciliation` | `SiteAuditReconciliationActor` | Audit Log (#23) | +| `kpi-history-recorder` | `KpiHistoryRecorderActor` | KPI History | +| `pending-deployment-purge` | `PendingDeploymentPurgeActor` | Deployment Manager (#2) | -**Site singletons (active site node, scoped to `"site-{SiteId}"` role):** +**Site singletons (oldest `Up` node of that site, scoped to the `"site-{SiteId}"` role):** | Singleton name | Actor class | Owner | |----------------|-------------|-------| @@ -173,29 +202,29 @@ Every host calls `AddClusterInfrastructure` to register `ClusterOptionsValidator services.AddClusterInfrastructure(); ``` -This registers `ClusterOptionsValidator` as an `IValidateOptions` singleton. Because the Host binds `ClusterOptions` with `ValidateOnStart`, a misconfigured `ScadaBridge:Cluster` section (wrong strategy, `MinNrOfMembers != 1`, `DownIfAlone = false`, fewer than two seed nodes) throws an `OptionsValidationException` at startup rather than booting into a broken cluster. +This registers `ClusterOptionsValidator` as an `IValidateOptions` singleton. Because the Host binds `ClusterOptions` with `ValidateOnStart`, a misconfigured `ScadaBridge:Cluster` section throws an `OptionsValidationException` at startup rather than booting into a broken cluster. The validator rejects: a strategy other than `auto-down` or `keep-oldest`; `MinNrOfMembers != 1`; a non-positive `StableAfter`, `HeartbeatInterval` or `FailureDetectionThreshold`; a `HeartbeatInterval` not below `FailureDetectionThreshold`; fewer than two seed nodes unless `AllowSingleNodeCluster = true`; and `DownIfAlone = false` **only when the strategy is `keep-oldest`** (the flag is inert under `auto-down`, so any value passes there). ### Checking active-node status -Components that must run only on the active node resolve `IActiveNodeGate` (registered by the Host's Central composition root): +Components that must run only on the active node resolve `IActiveNodeGate` (registered by the Host's Central composition root). The gate is a thin wrapper over the oldest-`Up` evaluator — it never inspects cluster leadership: ```csharp +// Host/Health/ActiveNodeGate.cs public bool IsActiveNode { get { var system = _akkaService.ActorSystem; - if (system == null) return false; + if (system == null) + return false; + var cluster = Cluster.Get(system); - var self = cluster.SelfMember; - if (self.Status != MemberStatus.Up) return false; - var leader = cluster.State.Leader; - return leader != null && leader == self.Address; + return ClusterActivityEvaluator.SelfIsOldest(cluster); } } ``` -This returns `false` while the actor system is warming up — the safe-by-default answer matching the standby case. The Inbound API uses this gate to return HTTP 503 on standby nodes. +This returns `false` while the actor system is warming up, and `SelfIsOldest` returns `false` unless the node has reached `MemberStatus.Up` — the safe-by-default answer matching the standby case. The Inbound API uses this gate to return HTTP 503 on standby nodes, and `OldestNodeActiveHealthCheck` backs `/health/active` off the same evaluator, so the proxy's routing decision and the API's gating decision can never disagree. ## Configuration @@ -205,13 +234,14 @@ This returns `false` while the actor system is warming up — the safe-by-defaul | Key | Type | Default | Description | |-----|------|---------|-------------| -| `SeedNodes` | `List` | (required) | Akka seed-node URIs. Must contain at least 2 entries; both nodes list both themselves and their partner. | -| `SplitBrainResolverStrategy` | `string` | `"keep-oldest"` | Must be `"keep-oldest"`. Quorum strategies are rejected by `ClusterOptionsValidator`. | -| `StableAfter` | `TimeSpan` | `00:00:15` | Cluster must be stable for this duration before the resolver acts to down unreachable nodes. | +| `SeedNodes` | `List` | (required) | Akka seed-node URIs. Must contain at least 2 entries (1 with `AllowSingleNodeCluster`); both nodes list both themselves and their partner. Only the **first** entry may self-form a new cluster. | +| `SplitBrainResolverStrategy` | `string` | `"auto-down"` | `"auto-down"` or `"keep-oldest"`. Quorum strategies are rejected by `ClusterOptionsValidator`. See downing strategy above. | +| `StableAfter` | `TimeSpan` | `00:00:15` | Sustained unreachability before downing. Emitted as `auto-down-unreachable-after` under `auto-down`, as the SBR `stable-after` under `keep-oldest`. | | `HeartbeatInterval` | `TimeSpan` | `00:00:02` | Cluster failure-detector heartbeat frequency. Must be less than `FailureDetectionThreshold`. | | `FailureDetectionThreshold` | `TimeSpan` | `00:00:10` | `acceptable-heartbeat-pause` for the cluster failure detector. | | `MinNrOfMembers` | `int` | `1` | Must be `1`. A value of `2` blocks the cluster singleton after failover. | -| `DownIfAlone` | `bool` | `true` | Must be `true`. See split-brain resolution above. | +| `DownIfAlone` | `bool` | `true` | `keep-oldest` only — inert under `auto-down`. Validated as `true` only when the strategy is `keep-oldest`. | +| `AllowSingleNodeCluster` | `bool` | `false` | Acknowledges a deliberate single-node install: permits exactly one seed node instead of the usual two. | ### `ScadaBridge:Node` @@ -241,7 +271,7 @@ This returns `false` while the actor system is warming up — the safe-by-defaul "akka.tcp://scadabridge@scadabridge-central-a:8081", "akka.tcp://scadabridge@scadabridge-central-b:8081" ], - "SplitBrainResolverStrategy": "keep-oldest", + "SplitBrainResolverStrategy": "auto-down", "StableAfter": "00:00:15", "HeartbeatInterval": "00:00:02", "FailureDetectionThreshold": "00:00:10", @@ -251,7 +281,7 @@ This returns `false` while the actor system is warming up — the safe-by-defaul } ``` -`DownIfAlone` is not present in the docker files because its default value of `true` is correct and `ClusterOptionsValidator` rejects `false`. +`DownIfAlone` is not present in the docker files because it is a `keep-oldest`-only knob and every shipped deployment runs `auto-down`, under which the flag is inert. ## Dependencies & Interactions @@ -261,22 +291,26 @@ This returns `false` while the actor system is warming up — the safe-by-defaul - [Site Runtime (#3)](./SiteRuntime.md) — the Deployment Manager singleton is the most operationally critical singleton this infrastructure hosts. It re-creates the full Instance Actor hierarchy from local SQLite on failover. Staggered Instance Actor startup after failover is Site Runtime's responsibility; this component provides the singleton placement guarantee. - [Notification Outbox (#21)](./NotificationOutbox.md), [Site Call Audit (#22)](./SiteCallAudit.md), [Audit Log (#23)](./AuditLog.md) — each hosts one or more central singletons wired by `RegisterCentralActors`. Cluster Infrastructure provides the `ClusterSingletonManager`/`ClusterSingletonProxy` boilerplate and the graceful-shutdown hooks; the business logic lives in the owning component. - [Central–Site Communication (#5)](./Communication.md) — `CentralCommunicationActor` and `SiteCommunicationActor` are created and registered with `ClusterClientReceptionist` inside the same `AkkaHostedService` startup, making them addressable by remote `ClusterClient` instances. The transport-level heartbeat (`TransportHeartbeatInterval`, `TransportFailureThreshold`) is configured separately from the cluster failure-detector and comes from `CommunicationOptions`. -- [Inbound API (#14)](./InboundAPI.md) — resolves `IActiveNodeGate` to return HTTP 503 on standby central nodes. Gate returns `false` until the actor system is `Up` and this node is the cluster leader. +- [Inbound API (#14)](./InboundAPI.md) — resolves `IActiveNodeGate` to return HTTP 503 on standby central nodes. Gate returns `false` until the actor system is `Up` and this node is the oldest `Up` member. - Design spec: [Component-ClusterInfrastructure.md](../requirements/Component-ClusterInfrastructure.md). ## Troubleshooting ### Node fails to join cluster on startup -`ClusterOptionsValidator` rejects fewer than two seed nodes, a non-`keep-oldest` strategy, `MinNrOfMembers != 1`, or `DownIfAlone = false` at startup with an `OptionsValidationException`. Check that both seed-node URIs reference the Akka remoting port, not the gRPC port (8083) or metrics port (8084) — on site nodes, `StartupValidator` explicitly rejects seed entries whose port matches `GrpcPort`. +`ClusterOptionsValidator` rejects fewer than two seed nodes (without `AllowSingleNodeCluster`), a strategy outside `auto-down` / `keep-oldest`, `MinNrOfMembers != 1`, or `DownIfAlone = false` under `keep-oldest`, at startup with an `OptionsValidationException`. Check that both seed-node URIs reference the Akka remoting port, not the gRPC port (8083) or metrics port (8084) — on site nodes, `StartupValidator` explicitly rejects seed entries whose port matches `GrpcPort`. + +A node that boots, logs no validation error, but never reaches `Up` is usually hitting the seed-node bootstrap constraint: it is not the first entry in `SeedNodes` and the first seed is down, so it loops on `InitJoin` waiting for a peer that can form the cluster. ### Singleton not starting after failover If the surviving node is `Up` but singletons do not start, `MinNrOfMembers` is the first thing to check. A value of `2` keeps the surviving node waiting for a second member indefinitely. The validator enforces `1`, but a manually patched `appsettings.json` that bypasses the validator could produce this. -### Two live clusters (split-brain) +### Two live clusters (dual-active) -If `DownIfAlone = false` were accepted (the validator rejects it), the oldest node could run alone while the younger forms its own cluster, producing two live clusters with divergent singleton state and dual MS SQL writers on central. `ClusterOptionsValidator` makes this configuration impossible to boot. +Under `auto-down` this is the **accepted trade, not a misconfiguration**: during a real network partition each side downs the other and continues as a one-node cluster, so both nodes are oldest-`Up`, both host a full set of singletons, and both answer `/health/active` with 200 — including dual MS SQL writers on central. Monitoring surfaces it directly (both nodes stamp `IsActive` on their heartbeats; the Health dashboard shows two Primaries). The two sides do **not** merge on their own — the mutual downing quarantines the association. Recovery is operator-driven: once the link is restored, restart **one** side; it rejoins its peer as a fresh incarnation and comes back as standby. + +Deployments that would rather lose availability than run dual-active should set `SplitBrainResolverStrategy: "keep-oldest"` (with `DownIfAlone = true`), accepting that a crash of the oldest node is then a total outage. ### Graceful shutdown takes longer than expected @@ -285,6 +319,7 @@ If a clean node stop takes up to 25 seconds instead of seconds, `CoordinatedShut ## Related Documentation - [Cluster Infrastructure design specification](../requirements/Component-ClusterInfrastructure.md) +- [Auto-down downing strategy — availability over partition-safety (decision, 2026-07-21)](../plans/2026-07-21-auto-down-availability-decision.md) - [Host](./Host.md) - [Site Runtime](./SiteRuntime.md) - [Health Monitoring](./HealthMonitoring.md) diff --git a/docs/components/Communication.md b/docs/components/Communication.md index 16647f24..4a9635d6 100644 --- a/docs/components/Communication.md +++ b/docs/components/Communication.md @@ -1,6 +1,6 @@ # Central–Site Communication -The Central–Site Communication component is the transport layer that connects the central cluster to every site cluster. It provides two independent transports — Akka.NET `ClusterClient` for command/control and gRPC server-streaming for real-time data — wired together through a pair of actors that each cluster registers with the `ClusterClientReceptionist`. +The Central–Site Communication component is the transport layer that connects the central cluster to every site cluster. It provides three independent transports — Akka.NET `ClusterClient` for command/control, gRPC server-streaming for real-time data, and plain token-gated HTTP for the deployment-config fetch — anchored by a pair of actors that each cluster registers with the `ClusterClientReceptionist`. ## Overview @@ -18,15 +18,33 @@ DI registration is called from the Host composition root via `AddCommunication`. ## Key Concepts -### Two transports, two concerns +### Three transports, three concerns -| Transport | Direction | Purpose | -|-----------|-----------|---------| -| Akka.NET `ClusterClient` | bidirectional (command/control) | Deployments, lifecycle, subscribe/unsubscribe handshake, snapshots, heartbeats, health reports, telemetry, notifications | -| gRPC server-streaming (`SiteStreamService`) | site → central | Real-time attribute value and alarm state changes | +| Transport | Who dials | Data direction | Purpose | +|-----------|-----------|----------------|---------| +| Akka.NET `ClusterClient` | both (central → site per site; site → central) | bidirectional | Deploy notifies, lifecycle, subscribe/unsubscribe handshake, snapshots, heartbeats, health reports, telemetry, notifications | +| gRPC (`SiteStreamService`) | **central dials the site** | mostly site → central | Real-time attribute value and alarm state changes (server-streaming), plus the audit ingest/pull unary RPCs | +| HTTP `GET` (`/api/internal/deployments/{id}/config`) | **site dials central** | central → site | The flattened deployment config itself (notify-and-fetch), gated by a per-deployment `X-Deployment-Token` | The transports are independent. A gRPC stream interruption does not affect in-flight `ClusterClient` commands, and vice versa. +**The gRPC dial direction is inverted from its data direction.** Values flow site → central, but each **site node hosts the gRPC server** and **central is the client**. `MapGrpcService()` appears exactly once in the tree, inside the Site branch of `Program.cs` (`Host/Program.cs:542`); there is **no gRPC server on a central node at all**. That is why the two `Ingest*` unary RPCs — nominally a central-side ingest surface — are dead in the shipped topology (acknowledged in `AkkaHostedService.cs:505-508`: "when the gRPC server is not registered (current central topology)"); sites push audit telemetry to central over `ClusterClient` instead, and central pulls with `PullAuditEvents` / `PullSiteCalls` by dialling the site. + +**None of the three transports carries a mutual-auth or transport-encryption story.** `BuildHocon` emits no Akka `enable-ssl`, no secure cookie and no `trusted-selection-paths`, so remoting and `ClusterClient` are plaintext and unauthenticated. The site gRPC listener is **h2c** — `ListenAnyIP(grpcPort, o => o.Protocols = HttpProtocols.Http2)` with no `UseHttps` (`Host/Program.cs:501-506`) — and `SiteStreamGrpcClient` opens a bare `GrpcChannel.ForAddress(endpoint, …)` with no credentials. `LocalDbSyncAuthInterceptor` shares that listener but gates **only** methods under `/localdb_sync.v1.LocalDbSync/`, explicitly letting every `SiteStreamService` call through untouched, so the whole surface — including the `PullAuditEvents` / `PullSiteCalls` RPCs that return audit rows — is callable by anything that can reach the site's gRPC port. The one authenticated piece is the HTTP config fetch, and its per-deployment token is the *entire* security boundary (the endpoint is `AllowAnonymous`). The design assumes a trusted network between central and sites. + +### Notify-and-fetch: the deployment-config HTTP path + +An instance deployment does not carry its flattened configuration inside the Akka message. Central stages a `PendingDeployment` row (config JSON + a freshly generated `DeploymentFetchToken` + a TTL) and sends only a small `RefreshDeploymentCommand` over `ClusterClient`, carrying the deployment id, revision hash, `CentralFetchBaseUrl` and the fetch token. The site's Deployment Manager singleton then calls back to central over plain HTTP: + +```csharp +// SiteRuntime/Deployment/HttpDeploymentConfigFetcher.cs +var url = $"{centralFetchBaseUrl.TrimEnd('/')}/api/internal/deployments/{Uri.EscapeDataString(deploymentId)}/config"; +using var req = new HttpRequestMessage(HttpMethod.Get, url); +req.Headers.Add("X-Deployment-Token", token); +``` + +`DeploymentConfigEndpoints.Resolve` (`ManagementService/DeploymentConfigEndpoints.cs:101`) checks existence and TTL *before* the token, so unknown, superseded and expired deployments are all indistinguishable `404`s; a live row with a wrong or missing token is `401`. The token comparison is constant-time. This exists because a flattened config can exceed the default 128 KB Akka frame size, which drops the single oversized message without tearing down the association — heartbeats keep flowing, the site still reports healthy, and the deploy just hangs to its Ask timeout. See `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`. `DeployArtifactsCommand` was **not** moved to this path and still carries its payload inline. + ### Hub-and-spoke topology Sites do not communicate with each other. All inter-cluster traffic flows through central. Central maintains one `ClusterClient` per site; each site maintains a single `ClusterClient` pointed at both central nodes. @@ -36,9 +54,9 @@ Sites do not communicate with each other. All inter-cluster traffic flows throug Central-side callers wrap outbound messages in a `SiteEnvelope(SiteId, Message)`. `CentralCommunicationActor` resolves the site's `ClusterClient` by `SiteId` and forwards the inner message to `/user/site-communication` on the site: ```csharp -// CommunicationService.cs — deployment pattern -public async Task DeployInstanceAsync( - string siteId, DeployInstanceCommand command, CancellationToken cancellationToken = default) +// CommunicationService.cs — deployment pattern (notify-and-fetch) +public async Task RefreshDeploymentAsync( + string siteId, RefreshDeploymentCommand command, CancellationToken cancellationToken = default) { var envelope = new SiteEnvelope(siteId, command); return await GetActor().Ask( @@ -86,7 +104,7 @@ If a site is unreachable when a command arrives, the caller's Ask times out. Cen `SiteCommunicationActor` is a `ReceiveActor` created at `/user/site-communication` and registered with `ClusterClientReceptionist`. It owns: - An `IActorRef? _centralClient` — the site's outbound `ClusterClient` to central. Injected post-construction via `RegisterCentralClient`. -- A `Timers`-based heartbeat (default 5-second interval, first tick after 1 second). Each tick sends a `HeartbeatMessage` with `IsActive` stamped from the Akka `Cluster` leader check — the node is active when its `MemberStatus` is `Up` and it holds cluster leadership. +- A `Timers`-based heartbeat on `CommunicationOptions.ApplicationHeartbeatInterval` (default 5 s; deliberately distinct from the Akka.Remote `TransportHeartbeatInterval`, so retuning the transport failure detector cannot silently retune the health heartbeat). Each tick sends a `HeartbeatMessage` whose `IsActive` is stamped from `ActiveNodeEvaluator.SelfIsOldestUp` — the node is active when it is the **oldest `Up` member**, *not* when it holds cluster leadership (`SiteCommunicationActor.cs:517-518`). A throwing active-check is caught and reported as `IsActive = false`. - Dispatch to local handlers for every inbound command pattern. Handlers for event-log, parked-message, integration, and artifact patterns are registered post-construction via `RegisterLocalHandler`; unregistered patterns receive an inline error reply so the central Ask does not stall. Site-to-central messages (health reports, audit batches, notification submissions) are sent via: @@ -108,9 +126,9 @@ A malformed address for one site does not abort the refresh loop — the actor c ### gRPC real-time data transport -Real-time attribute value and alarm state changes are delivered over `SiteStreamService`, a gRPC server-streaming service defined in `sitestream.proto`. +Real-time attribute value and alarm state changes are delivered over `SiteStreamService`, defined in `sitestream.proto`. The **server runs on every site node and the client runs on central** — central dials in to receive the stream (see the transport table above). -**Site-side** — `SiteStreamGrpcServer` (Kestrel HTTP/2, port 8083): +**Site-side** — `SiteStreamGrpcServer` (Kestrel h2c, HTTP/2 only, port 8083): - Implements `SiteStreamService.SiteStreamServiceBase`. - For each `SubscribeInstance` call, creates a `StreamRelayActor` (named `stream-relay-{correlationId}-{seq}`) and subscribes it to `ISiteStreamSubscriber` (implemented by `SiteStreamManager` in the Site Runtime project — `SiteStreamGrpcServer` holds only the interface so it does not reference `SiteRuntime` directly). @@ -144,7 +162,7 @@ private void HandleAttributeValueChanged(AttributeValueChanged msg) **Central-side** — `SiteStreamGrpcClient` / `SiteStreamGrpcClientFactory`: -- `SiteStreamGrpcClientFactory` (singleton) caches one `SiteStreamGrpcClient` per site identifier. On `GetOrCreate`, it compares the cached client's `Endpoint` to the requested endpoint and atomically replaces a stale client (different endpoint — NodeA→NodeB failover flip, or an edited address) with a fresh one. +- `SiteStreamGrpcClientFactory` (singleton) caches one `SiteStreamGrpcClient` per **`(site, endpoint)` pair** — a `ConcurrentDictionary<(string Site, string Endpoint), SiteStreamGrpcClient>`. The key was widened from site-only to fix an arch-review High: with a site-only key, one debug session's NodeA→NodeB failover flip disposed a channel another session was still using. `GetOrCreate` therefore no longer disposes on endpoint mismatch; both of a site's node channels coexist, and site *removal* (`RemoveSiteAsync`) is the only shared-disposal path. The trade-off is that an edited gRPC address leaves the old endpoint's idle channel cached until site removal or process shutdown — bounded at a handful of entries per site. - `SiteStreamGrpcClient` opens a `GrpcChannel` with HTTP/2 keepalive (`KeepAlivePingDelay` default 15 s, `KeepAlivePingTimeout` default 10 s, `KeepAlivePingPolicy.Always`). `SubscribeAsync` is a plain `async Task` that calls `SubscribeInstance` and reads the response stream with `await foreach`, invoking `onEvent` for each received event and `onError` on any non-cancellation exception. The caller (`DebugStreamBridgeActor.OpenGrpcStream`) launches it inside a `Task.Run` so the long-running stream loop runs off the actor thread. ### Debug stream session lifecycle @@ -162,18 +180,22 @@ private void HandleAttributeValueChanged(AttributeValueChanged msg) ### Proto definition summary ```proto -// Protos/sitestream.proto +// Protos/sitestream.proto — six RPCs, all served by the SITE service SiteStreamService { rpc SubscribeInstance(InstanceStreamRequest) returns (stream SiteStreamEvent); + rpc SubscribeSite(SiteStreamRequest) returns (stream SiteStreamEvent); rpc IngestAuditEvents(AuditEventBatch) returns (IngestAck); rpc IngestCachedTelemetry(CachedTelemetryBatch) returns (IngestAck); rpc PullAuditEvents(PullAuditEventsRequest) returns (PullAuditEventsResponse); + rpc PullSiteCalls(PullSiteCallsRequest) returns (PullSiteCallsResponse); } ``` -`SubscribeInstance` carries the real-time data stream. The other three RPCs (`IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents`) serve the Audit Log component's gRPC telemetry push and reconciliation pull paths — `SiteStreamGrpcServer` hosts them on the same port because sites already listen there. +Two are server-streaming: `SubscribeInstance` carries the per-instance real-time stream; `SubscribeSite` is the **site-wide, alarm-only** stream (no instance filter, attribute updates never carried) that feeds the aggregated central live alarm cache. The four unary RPCs serve the Audit Log and Site Call Audit push/pull paths — `SiteStreamGrpcServer` hosts them on the same port because sites already listen there. As noted above, the two `Ingest*` RPCs are dead in the shipped topology (no central gRPC server exists for a site to dial); the two `Pull*` RPCs are live, with central as the caller. -`SiteStreamEvent` uses a `oneof event { AttributeValueUpdate, AlarmStateUpdate }` discriminator. `AlarmStateUpdate` carries the full native alarm condition (fields 8–21) alongside the base computed-alarm fields (1–7), added additively so old clients ignoring unknown fields continue to work. +`SiteStreamEvent` uses a `oneof event { AttributeValueUpdate, AlarmStateUpdate }` discriminator. `AlarmStateUpdate` carries the full native alarm condition (fields 8–23) alongside the base computed-alarm fields (1–7), added additively so old clients ignoring unknown fields continue to work. Field numbers are never reused and evolution is additive only. + +The generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `` include commented out, so editing `sitestream.proto` does not regenerate on build — regeneration is a manual toggle-build-copy-untoggle. ## Usage @@ -181,7 +203,7 @@ Central callers interact through `CommunicationService`, which wraps each comman | Pattern | Method | Timeout | |---------|--------|---------| -| Instance deployment | `DeployInstanceAsync` | 120 s | +| Instance deployment (notify-and-fetch) | `RefreshDeploymentAsync` | 120 s | | Instance lifecycle | `DisableInstanceAsync`, `EnableInstanceAsync`, `DeleteInstanceAsync` | 30 s | | Artifact deployment | `DeployArtifactsAsync` | 60 s | | Integration routing | `RouteIntegrationCallAsync` | 30 s | @@ -197,11 +219,11 @@ For real-time streaming, callers use `DebugStreamService.StartStreamAsync`, whic ## Configuration -All options are bound from the `Communication` section via `CommunicationOptions`: +All options are bound from the `ScadaBridge:Communication` section via `CommunicationOptions`: | Key | Default | Description | |-----|---------|-------------| -| `DeploymentTimeout` | `00:02:00` | Ask timeout for instance deployment commands. | +| `DeploymentTimeout` | `00:02:00` | Ask timeout for the `RefreshDeploymentCommand` round-trip (covers the site's HTTP config fetch and apply). | | `LifecycleTimeout` | `00:00:30` | Ask timeout for lifecycle commands (disable, enable, delete). | | `ArtifactDeploymentTimeout` | `00:01:00` | Ask timeout for system-wide artifact deployment. | | `QueryTimeout` | `00:00:30` | Ask timeout for remote queries and management commands. | @@ -213,8 +235,12 @@ All options are bound from the `Communication` section via `CommunicationOptions | `GrpcKeepAlivePingTimeout` | `00:00:10` | HTTP/2 keepalive PING timeout. | | `GrpcMaxStreamLifetime` | `04:00:00` | Per-stream session timeout; forces reconnect of zombie streams. | | `GrpcMaxConcurrentStreams` | `100` | Max concurrent `SubscribeInstance` streams per site node. | -| `TransportHeartbeatInterval` | `00:00:05` | `SiteCommunicationActor` heartbeat cadence. | +| `ApplicationHeartbeatInterval` | `00:00:05` | `SiteCommunicationActor` site→central heartbeat cadence. | +| `TransportHeartbeatInterval` | `00:00:05` | Akka.Remote transport failure-detector heartbeat interval (emitted into the HOCON by the Host). Distinct from the application heartbeat above. | | `TransportFailureThreshold` | `00:00:15` | Akka remoting failure-detection threshold. | +| `CentralFetchBaseUrl` | `""` | Base URL (Traefik/LB) the site uses to fetch deploy configs from central. Carried in `RefreshDeploymentCommand` so sites need no standing config; **empty makes a deploy impossible** — `DeploymentService` fails fast. | +| `PendingDeploymentTtl` | `00:05:00` | How long a staged `PendingDeployment` row and its fetch token stay valid. Must comfortably cover both site nodes' fetches within one deploy window. | +| `PendingDeploymentPurgeInterval` | `01:00:00` | Cadence of the central `pending-deployment-purge` singleton that sweeps TTL-expired staging rows. Hygiene only — the fetch endpoint already enforces the TTL. | Three layers of dead-client detection protect the gRPC stream path: @@ -227,16 +253,16 @@ Three layers of dead-client detection protect the gRPC stream path: ## Dependencies & Interactions - [Commons (#16)](./Commons.md) — owns all message contracts used by this component: `DeployInstanceCommand`, `SiteEnvelope`, `HeartbeatMessage`, `SiteHealthReport`, `SiteHealthReportReplica`, `RegisterNotificationOutbox`, `RegisterAuditIngest`, `IngestAuditEventsCommand`, `IngestCachedTelemetryCommand`, and all other request/response records. Commons does not hold an Akka package reference, so `RegisterAuditIngest` (which carries an `IActorRef`) lives in this project. -- [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — provides `ClusterClientReceptionist` registration and the active/standby leader model that `SiteCommunicationActor`'s `IsActive` check and `CentralCommunicationActor`'s `DistributedPubSub` fanout both depend on. +- [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — provides `ClusterClientReceptionist` registration and the oldest-`Up` active/standby model that `SiteCommunicationActor`'s `IsActive` stamp depends on, plus the single `"scadabridge"` `ActorSystem` name that makes cross-cluster `ClusterClient` addressing possible at all. `CentralCommunicationActor`'s `DistributedPubSub` fanout keeps both central nodes in sync regardless of which one a site's report landed on. - [Configuration Database (#17)](./ConfigurationDatabase.md) — provides `ISiteRepository.GetAllSitesAsync` for address loading; site records carry `NodeAAddress`, `NodeBAddress`, `GrpcNodeAAddress`, `GrpcNodeBAddress`. -- [Deployment Manager (#2)](./DeploymentManager.md) — the primary consumer of command/control patterns 1–3. `CommunicationService` is injected into the Deployment Manager actor to send deployments, lifecycle commands, and artifact deployments to sites. +- [Deployment Manager (#2)](./DeploymentManager.md) — the primary consumer of command/control patterns 1–3. `CommunicationService` is injected into the Deployment Manager actor to send deploy notifies, lifecycle commands, and artifact deployments to sites. It also owns the staging half of the notify-and-fetch HTTP path (`PendingDeployment` rows + fetch tokens); the endpoint itself is served by the Management Service. - [Site Runtime (#3)](./SiteRuntime.md) — `SiteCommunicationActor` forwards inbound commands to the `DeploymentManager` singleton proxy. `SiteStreamManager` (in Site Runtime) implements `ISiteStreamSubscriber` so `SiteStreamGrpcServer` can subscribe relay actors to instance event feeds without referencing Site Runtime directly. - [Health Monitoring (#11)](./HealthMonitoring.md) — `CentralCommunicationActor` calls `ICentralHealthAggregator.MarkHeartbeat` and `ProcessReport` for every inbound heartbeat and health report. `DistributedPubSub` fanout keeps both central nodes' aggregators in sync. -- [Audit Log (#23)](./AuditLog.md) — `SiteStreamGrpcServer` hosts `IngestAuditEvents`, `IngestCachedTelemetry`, and `PullAuditEvents` RPCs. `CentralCommunicationActor` routes `IngestAuditEventsCommand` / `IngestCachedTelemetryCommand` ClusterClient messages to the `AuditLogIngestActor` proxy. +- [Audit Log (#23)](./AuditLog.md) — `SiteStreamGrpcServer` hosts the `IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents` and `PullSiteCalls` RPCs. Because there is no central gRPC server, the `Ingest*` pair is unused in the shipped topology: sites push audit telemetry over ClusterClient, and `CentralCommunicationActor` routes `IngestAuditEventsCommand` / `IngestCachedTelemetryCommand` to the `AuditLogIngestActor` proxy. The `Pull*` reconciliation RPCs run the other way, with the central `site-audit-reconciliation` singleton dialling each site. - [Notification Outbox (#21)](./NotificationOutbox.md) — `CentralCommunicationActor` routes `NotificationSubmit` / `NotificationStatusQuery` messages from sites to the `NotificationOutboxActor` proxy. `CommunicationService` Asks the proxy directly for central-UI outbox management calls. - [Site Call Audit (#22)](./SiteCallAudit.md) — `CommunicationService` Asks the `SiteCallAuditActor` proxy directly for query and relay operations. `SiteCallAuditActor` issues `RetryParkedOperation` / `DiscardParkedOperation` relay commands to sites via `SiteEnvelope`; `SiteCommunicationActor` dispatches them to `_parkedMessageHandler`. - [Store-and-Forward Engine (#6)](./StoreAndForward.md) — the site S&F Engine drives `NotificationSubmit` forwarding and cached-call telemetry emission through `SiteCommunicationActor`. Parked-message queries and retry/discard relay commands flow back the other way. -- [Management Service (#18)](./ManagementService.md) — `ManagementActor` is registered with `ClusterClientReceptionist` at `/user/management` on central; the CLI connects via its own separate `ClusterClient`. This is a distinct `ClusterClient` usage from the inter-cluster hub-and-spoke connections managed by this component. +- [Management Service (#18)](./ManagementService.md) — `ManagementActor` is registered with `ClusterClientReceptionist` at `/user/management` on central; the CLI connects via its own separate `ClusterClient`. This is a distinct `ClusterClient` usage from the inter-cluster hub-and-spoke connections managed by this component. Management Service also hosts `DeploymentConfigEndpoints` — the `GET /api/internal/deployments/{id}/config` route that terminates the third (HTTP) transport, mapped in the central-role block alongside `/api/audit/*` and `/management`. - Design spec: [Component-Communication.md](../requirements/Component-Communication.md). ## Troubleshooting @@ -257,6 +283,10 @@ A `Warning` at the `Status.Failure` handler in `CentralCommunicationActor` means After a site node failover, the `DebugStreamBridgeActor` attempts to reconnect to the other node endpoint (`_useNodeA` flips on each error). If both nodes are unreachable, the actor exhausts its 3-retry budget and calls `onTerminated`. The engineer must restart the debug session. +### Deployments fail immediately with a config-fetch error + +The site received the `RefreshDeploymentCommand` over ClusterClient but could not complete the HTTP leg. Check `CentralFetchBaseUrl` first — it must be reachable *from the site*, so a value that only resolves inside the central network fails every deploy. A `404` from the fetch means the staged row was unknown, superseded, or past `PendingDeploymentTtl`; a `401` means the row is live but the token did not match. Because the endpoint hides existence, a `404` cannot distinguish "wrong id" from "expired". + ### Heartbeats arrive but health reports do not `SiteCommunicationActor` sends heartbeats and health reports via separate paths. Health reports are sent only when the site's `HealthReportSender` publishes them (every 30 s by default). If heartbeats arrive but reports do not, the health-report sender on the site may have faulted — check site-side logs for errors in `HealthReportSender`. diff --git a/docs/components/DeploymentManager.md b/docs/components/DeploymentManager.md index f709b48f..e0be146e 100644 --- a/docs/components/DeploymentManager.md +++ b/docs/components/DeploymentManager.md @@ -26,10 +26,20 @@ Every instance deployment carries two correlated identifiers: - **`DeploymentId`** — a new `Guid` (formatted `"N"`) minted by `DeploymentService` at the start of each `DeployInstanceAsync` call. - **`RevisionHash`** — computed by the Template Engine's `RevisionHashService` over the fully resolved `FlattenedConfiguration`. The hash captures the template state at the moment of flattening, so concurrent last-write-wins template edits do not affect an in-flight deployment. -The pair travels inside `DeployInstanceCommand` to the site. The site uses the `DeploymentId` to detect an already-applied identical command (idempotent re-delivery) and uses the `RevisionHash` to reject a stale configuration that predates what is already running. +The pair travels to the site inside the `RefreshDeploymentCommand` notify and is echoed back on the fetched config. The site uses the `DeploymentId` to detect an already-applied identical command (idempotent re-delivery) and uses the `RevisionHash` to reject a stale configuration that predates what is already running. Central stores the `RevisionHash` on `DeploymentRecord` and, after a confirmed success, on `DeployedConfigSnapshot`. Comparing the snapshot hash against the current-template hash determines whether an instance is stale without a site round-trip. +### Notify-and-fetch: the config does not travel in the Akka message + +A deployment crosses the central↔site boundary over **two** transports, not one. Central stages the flattened configuration in a `PendingDeployment` row (config JSON, a generated `DeploymentFetchToken`, and an expiry of `CommunicationOptions.PendingDeploymentTtl`, default 5 minutes) and then sends only a small `RefreshDeploymentCommand` over ClusterClient carrying the deployment id, instance name, revision hash, `CentralFetchBaseUrl` and the fetch token. The site's Deployment Manager singleton fetches the config back over plain HTTP — `GET {CentralFetchBaseUrl}/api/internal/deployments/{deploymentId}/config` with an `X-Deployment-Token` header — and only then runs its normal apply path. + +This exists because a flattened configuration can exceed the default 128 KB Akka frame size, and an over-limit message is dropped silently without tearing down the association — the deploy then simply hangs to its Ask timeout. `CentralFetchBaseUrl` is therefore mandatory: `DeployInstanceAsync` fails fast with "CentralFetchBaseUrl is not configured — required for deployment (notify-and-fetch)" rather than attempting a deploy that cannot complete. Note that `DeployArtifactsCommand` was **not** moved to this path — artifact deployment still carries its payload inline and remains exposed to the frame limit. + +Staged rows are cleaned up by **TTL only** — they are deliberately not deleted on success or in the failure path. Three things keep that safe: `AddPendingDeploymentAsync` supersedes (deletes) any prior pending row for the same instance before inserting, so at most one row exists per instance; the fetch endpoint enforces the TTL itself, so an un-purged row is not a usable one; and the central `pending-deployment-purge` singleton sweeps expired rows on `PendingDeploymentPurgeInterval` (default 1 hour). + +The site's **startup reconciliation** path uses the same endpoint but stages its own rows: a site node reports its local instance→revision-hash map on boot, and central's `ReconcileService` diffs it against the expected deployed set, stages a fresh `PendingDeployment` (with a new token) for each missing or stale instance, and returns the gap plus `CentralFetchBaseUrl` for the node to fetch. Intra-site replication to the standby node does **not** use this path — `deployed_configurations` is a replicated LocalDb table, so the active node's write reaches the peer as an ordinary row change. + ### Per-instance operation lock `OperationLockManager` holds a `Dictionary` keyed by instance `UniqueName`. Each `LockEntry` wraps a `SemaphoreSlim(1,1)` with a reference count so the semaphore is created on first contention and disposed when the last waiter clears. The lock covers all four mutating operations — deploy, disable, enable, delete — so they can never interleave on a single instance. Operations on different instances proceed in parallel. @@ -67,7 +77,7 @@ The operation lock is in-memory. If the active central node fails mid-deployment 3. **Flatten and validate** — `IFlatteningPipeline.FlattenAndValidateAsync` runs the Template Engine pipeline and returns a `FlatteningPipelineResult` containing the `FlattenedConfiguration`, `RevisionHash`, and a `ValidationResult`. Semantic validation failures (call targets, argument types, trigger operand types, connection binding completeness) are returned to the caller before any record is written. 4. **Pre-deploy site reconciliation** — when the prior `DeploymentRecord` for the instance is `InProgress` or `Failed` with a timeout marker (`"Communication failure:"`), the service queries the site via `CommunicationService.QueryDeploymentStateAsync`. If the site already holds the target revision hash, the prior record is updated to `Success` and no new deployment is sent. 5. **Write `InProgress` record** — a single `DeploymentRecord` insert directly at `InProgress` status (no transient `Pending` hop). `IDeploymentStatusNotifier.NotifyStatusChanged` fires to push the status to the UI. -6. **Send `DeployInstanceCommand`** — the command carries `DeploymentId`, `InstanceUniqueName`, `RevisionHash`, `FlattenedConfigurationJson`, `DeployedBy`, and `Timestamp`. +6. **Stage and notify** — insert a `PendingDeployment` row holding the flattened config JSON and a fresh fetch token, then send `RefreshDeploymentCommand` (`DeploymentId`, `InstanceUniqueName`, `RevisionHash`, `DeployedBy`, staging timestamp, `CentralFetchBaseUrl`, `FetchToken`) via `CommunicationService.RefreshDeploymentAsync`. The site fetches the config over HTTP and replies with the same `DeploymentStatusResponse` as before. 7. **Commit terminal status** — the `DeploymentRecord` is updated to `Success` or `Failed` and saved before any post-success side effects run. This ordering ensures the recorded outcome can never be lost if a post-success write fails. 8. **Post-success side effects** — `ApplyPostSuccessSideEffectsAsync` sets `Instance.State = Enabled` (or preserves `Disabled` on the reconciliation path) and upserts the `DeployedConfigSnapshot`. These writes are best-effort: a failure here is logged at `Error` but does not flip the already-committed `Success` record back to `Failed`. 9. **Audit log** — `IAuditService.LogAsync` records `Deploy` / `DeployFailed` / `DeployReconciled` with the `DeploymentId`, status, and user. @@ -76,7 +86,7 @@ Any exception in the site round-trip (steps 6–7) writes `DeploymentStatus.Fail ```csharp // DeploymentService.DeployInstanceAsync — exception handler -var isTimeout = ex is TimeoutException or OperationCanceledException; +var isTimeout = ex is TimeoutException or OperationCanceledException or Akka.Actor.AskTimeoutException; record.Status = DeploymentStatus.Failed; record.ErrorMessage = isTimeout @@ -171,11 +181,11 @@ Options are registered via `AddDeploymentManager` and bound from `ScadaBridge:De - [Template Engine (#1)](./TemplateEngine.md) — `FlatteningPipeline` delegates to `FlatteningService`, `ValidationService`, and `RevisionHashService`. Template state is captured at flatten time; last-write-wins edits made after flatten do not affect the in-flight deployment. `DiffService.ComputeDiff` powers the deployment diff view. - [Configuration Database (#17)](./ConfigurationDatabase.md) — owns the EF Core implementation of `IDeploymentManagerRepository`, which stores `DeploymentRecord`, `DeployedConfigSnapshot`, and `SystemArtifactDeploymentRecord`. `IAuditService` (also registered by the Configuration Database component) writes all deployment audit rows. -- [Central–Site Communication (#5)](./Communication.md) — `CommunicationService` provides `DeployInstanceAsync`, `QueryDeploymentStateAsync`, `DeployArtifactsAsync`, `DisableInstanceAsync`, `EnableInstanceAsync`, and `DeleteInstanceAsync`. The communication layer routes by `SiteIdentifier` (string), not DB id; `DeploymentService.ResolveSiteIdentifierAsync` resolves the numeric `SiteId` before each cross-cluster call and treats a missing site row as a hard failure. -- [Commons (#16)](./Commons.md) — owns `DeploymentRecord`, `DeployedConfigSnapshot`, `SystemArtifactDeploymentRecord`, `DeploymentStatus`, `InstanceState`, `DeployInstanceCommand`, `DeployArtifactsCommand`, `DeploymentStateQueryRequest/Response`, `InstanceLifecycleResponse`, and the `IDeploymentManagerRepository` interface. +- [Central–Site Communication (#5)](./Communication.md) — `CommunicationService` provides `RefreshDeploymentAsync`, `QueryDeploymentStateAsync`, `DeployArtifactsAsync`, `DisableInstanceAsync`, `EnableInstanceAsync`, and `DeleteInstanceAsync`, all over the ClusterClient command/control transport. The communication layer routes by `SiteIdentifier` (string), not DB id; `DeploymentService.ResolveSiteIdentifierAsync` resolves the numeric `SiteId` before each cross-cluster call and treats a missing site row as a hard failure. `CommunicationOptions.CentralFetchBaseUrl` / `PendingDeploymentTtl` (also owned by that component) parameterise the notify-and-fetch HTTP leg. +- [Commons (#16)](./Commons.md) — owns `DeploymentRecord`, `DeployedConfigSnapshot`, `SystemArtifactDeploymentRecord`, `PendingDeployment`, `DeploymentFetchToken`, `DeploymentStatus`, `InstanceState`, `RefreshDeploymentCommand`, `DeployInstanceCommand` (retained as the site-side in-process apply DTO), `DeployArtifactsCommand`, `DeploymentStateQueryRequest/Response`, `InstanceLifecycleResponse`, and the `IDeploymentManagerRepository` interface. - [Site Runtime (#3)](./SiteRuntime.md) — receives `DeployInstanceCommand` and `DeployArtifactsCommand` via the Communication Layer. Site-side apply is all-or-nothing per instance: the Deployment Manager singleton at the site stores the config, compiles all scripts, and creates or replaces the Instance Actor as a unit. A failure at any step is reported back with the specific error message and the previous configuration remains active. - [Central UI (#9)](./CentralUI.md) — engineers trigger deployments, view diffs, manage instance lifecycle, and deploy system-wide artifacts through the UI. The deployment status page subscribes to `IDeploymentStatusNotifier.StatusChanged` for real-time push updates via Blazor Server SignalR. -- [Management Service (#18)](./ManagementService.md) — the actor-layer entry point for deployment commands received over ClusterClient. It resolves `DeploymentService` and `ArtifactDeploymentService` from a per-message DI scope and forwards `MgmtDeployArtifactsCommand`, `GetDeploymentDiffCommand`, and instance lifecycle requests. +- [Management Service (#18)](./ManagementService.md) — the actor-layer entry point for deployment commands received over ClusterClient. It resolves `DeploymentService` and `ArtifactDeploymentService` from a per-message DI scope and forwards `MgmtDeployArtifactsCommand`, `GetDeploymentDiffCommand`, and instance lifecycle requests. It also hosts `DeploymentConfigEndpoints` — the `GET /api/internal/deployments/{id}/config` route a site calls to fetch a staged config. That endpoint is `AllowAnonymous`; the per-deployment token, compared in constant time, is the entire security boundary, and existence/TTL are checked before the token so unknown, superseded and expired ids are indistinguishable `404`s. - [Security & Auth (#10)](./Security.md) — the Deployment role is required for all deploy and artifact operations; site-scoped permissions are enforced by the Central UI and Management Service before commands reach `DeploymentService`. ## Troubleshooting @@ -188,6 +198,10 @@ The operation lock is in-memory. On failover the new active node has no lock ent The site round-trip timed out or was cancelled before a response arrived. The site may or may not have applied the config. On the next deploy attempt the reconciliation query determines the ground truth. If the query also fails (site unreachable), a new `DeployInstanceCommand` is sent; the site rejects it with "already applied" if it ran the previous one. +### A deployment fails with a config-fetch error + +The notify reached the site but the HTTP leg did not complete. `CentralFetchBaseUrl` must be resolvable and reachable **from the site** — a value that only works inside the central network fails every deploy. A `404` from the fetch means the staged row was unknown, superseded, or past `PendingDeploymentTtl` (existence is hidden, so those are indistinguishable); a `401` means the row is live but the presented token did not match. A fetch failure applies nothing, and the site replies `Failed` rather than letting central's Ask hang to timeout. + ### DeleteOrphaned audit entry The site destroyed the Instance Actor but the central DB removal failed. The instance record exists in the central DB but has no corresponding site actor. It cannot be deleted through the normal UI path (the site will reject the delete command because the instance does not exist). Reconcile by removing the central record directly via the Management API or database, referencing the `CommandId` in the audit entry.