refactor(inbound-api): use the shared ZB.MOM.WW.Health IActiveNodeGate (ScadaBridge#80)

The InboundAPI carried its own IActiveNodeGate (implemented by Host/Health/ActiveNodeGate)
whose rule already matched the shared Health 0.3.0 AkkaActiveNodeGate — both route to
ClusterActiveNode (oldest Up member). The private interface and implementation are deleted;
InboundAPI references ZB.MOM.WW.Health (0.3.0, already pinned) and InboundApiEndpointFilter,
AuditWriteMiddleware, InboundMethodWarmupService and FlowDebugRunner consume
ZB.MOM.WW.Health.IActiveNodeGate. The Central branch of Program.cs registers
AkkaActiveNodeGate (unscoped, like the active-node health check).

Warm-up behaviour is kept: the shared gate answers false while no ActorSystem resolves, while
this member is not Up, and on any cluster-init fault. It resolves the ActorSystem through the
existing DI singleton bridge (GetOrCreateActorSystem), the same path the health probes use.

Tests: CompositionRoot asserts the shared gate type (fails with the registration removed);
the evaluator wiring test now drives AkkaActiveNodeGate; new warm-up test (no ActorSystem ->
not active). Nothing was missing from the Health 0.3.0 package.

(ScadaBridge#80)

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 13:39:54 -05:00
parent 1ee2e171cf
commit 23c35df234
18 changed files with 52 additions and 91 deletions
+1 -1
View File
@@ -201,7 +201,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
- Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s (drill-measured 2026-07-21 under auto-down: active-crash TAKEOVER in 28s, standby-crash removal in 27s with 0 routing blips — `docker/failover-drill.sh`).
- CoordinatedShutdown for graceful singleton handover.
- Automatic dual-node recovery from persistent storage.
- **Active/standby is decided by `ActiveNodeEvaluator.SelfIsOldestUp`, never by cluster leadership** — see the Architecture note above. **Central's** `/health/active` backs both Traefik's active-node routing and `IActiveNodeGate`, so the proxy and the Inbound API always agree on which node is active. **Site nodes DO map `/health/*` too** — the "site nodes map no `/health/*` at all" claim that stood here was stale: `SiteServiceRegistration` registers `akka-cluster` + `localdb` (Ready) and `SitePairActiveNodeHealthCheck` (Active, role-scoped to `site-{SiteId}`, deliberately NOT central's unscoped shared check), and `Program.cs`'s Site branch calls `MapZbHealth()` on the HTTP/1.1 listener so the family overview dashboard can probe every node identically. Nothing in this repo probes a site's `/health/active` (Traefik health-probes central only), and there is **no `IHealthCheckPublisher` anywhere**, so those checks run only on an inbound HTTP probe. Consequence worth knowing: the standby's correct-by-design `active-node` Unhealthy verdict is logged by `DefaultHealthCheckService` at **Error** on every probe, which `LoggerConfigurationFactory` now suppresses with one narrowly-keyed Serilog filter (register row 71 — see `Component-Host.md` → *Standby health-check log noise*; genuine failures of every other check still log at Error). Central reaches a site by dialling `GrpcNodeAAddress`/`GrpcNodeBAddress` explicitly and flipping on error; whichever node answers, the site-internal `ClusterSingletonProxy` lands the work on the active node for free, so central still never needs to track which *site* node is active.
- **Active/standby is decided by `ActiveNodeEvaluator.SelfIsOldestUp`, never by cluster leadership** — see the Architecture note above. **Central's** `/health/active` backs Traefik's active-node routing and the Inbound API uses the shared `ZB.MOM.WW.Health.IActiveNodeGate` (`AkkaActiveNodeGate`, same `ClusterActiveNode` rule; the private InboundAPI gate was removed in Gitea #80), so the proxy and the Inbound API always agree on which node is active. **Site nodes DO map `/health/*` too** — the "site nodes map no `/health/*` at all" claim that stood here was stale: `SiteServiceRegistration` registers `akka-cluster` + `localdb` (Ready) and `SitePairActiveNodeHealthCheck` (Active, role-scoped to `site-{SiteId}`, deliberately NOT central's unscoped shared check), and `Program.cs`'s Site branch calls `MapZbHealth()` on the HTTP/1.1 listener so the family overview dashboard can probe every node identically. Nothing in this repo probes a site's `/health/active` (Traefik health-probes central only), and there is **no `IHealthCheckPublisher` anywhere**, so those checks run only on an inbound HTTP probe. Consequence worth knowing: the standby's correct-by-design `active-node` Unhealthy verdict is logged by `DefaultHealthCheckService` at **Error** on every probe, which `LoggerConfigurationFactory` now suppresses with one narrowly-keyed Serilog filter (register row 71 — see `Component-Host.md` → *Standby health-check log noise*; genuine failures of every other check still log at Error). Central reaches a site by dialling `GrpcNodeAAddress`/`GrpcNodeBAddress` explicitly and flipping on error; whichever node answers, the site-internal `ClusterSingletonProxy` lands the work on the active node for free, so central still never needs to track which *site* node is active.
- **Seed-node ordering: every node lists ITSELF first (decision 2026-07-22) — the boot-alone gap is CLOSED.** Only `seed-nodes[0]` may self-join to form a new cluster (Akka runs `FirstSeedNodeProcess` for it, `JoinSeedNodeProcess` — which can never form one — for everyone else). All 14 shipped node appsettings now lead with the node's own address, so any node can cold-start alone and become operational unattended (~5s, `seed-node-timeout`); `StartupValidator` fails the boot if the ordering is broken (compares host AND port; Akka does no DNS canonicalisation). Two nodes cold-starting together while mutually reachable converge on ONE cluster via the `InitJoin` handshake — they split only under a genuine boot-time partition, the same class auto-down accepts. **An external self-form timer (`Cluster.Join(SelfAddress)` after a window) was implemented and REJECTED:** it sits outside the join handshake, so on a routine standby restart — where the peer is alive but the join is stalled behind removal of the node's own stale incarnation — it fires mid-join and permanently splits the pair (measured: still split after 90s). Regression tests: `SelfFirstSeedBootstrapTests`. The keep-oldest active-crash total outage was separately closed by the auto-down decision. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering.
- **Simultaneous-cold-start split-brain guard (opt-in, Gitea #33) — the residual self-first cost, closed.** Self-first-on-both means a *truly simultaneous* cold start (shared power/hypervisor event) races `FirstSeedNodeProcess` on BOTH nodes → two 1-node clusters that never merge (in-process loopback tests converge and hide it; real parallel VM starts don't — OtOpcUa reproduced it live). Ported from OtOpcUa (`lmxopcua` `d1dac87f`): a **dark switch `ScadaBridge:Cluster:BootstrapGuard:Enabled` (default OFF, guard-off byte-identical)**. On: `BuildHocon` emits an EMPTY seed list (Akka does not auto-join) and `ClusterBootstrapCoordinator` (`IHostedService`, registered in BOTH the Central and Site composition roots) issues ONE reachability-gated `JoinSeedNodes` — the pure `ClusterBootstrapGuard` core makes the lower canonical `host:port` the **founder** (self-first, forms immediately), the higher node TCP-probes the founder up to `PartnerProbeSeconds` (25s) and joins **peer-first** if reachable else **self-first** (cold-start-alone preserved). Decided BEFORE a single join, **never re-forms mid-handshake** (that was the rejected self-form-timer's flaw); case-insensitive founder tie-break; probe timings validated `>0` when enabled. Residual accepted trade: founder dies in the probe→join window → higher node hangs unjoined, coordinator WARNs, a restart recovers it. Tests: `ClusterBootstrapGuardTests` (pure) + `ClusterBootstrapCoordinatorTests` (real-ActorSystem, incl. both-cold-start-together-form-one-cluster). **ENABLED on the docker rig (all 8 nodes) 2026-08-02** after `deploy.sh`'s simultaneous recreate split site pairs twice on 2026-08-01 with the guard off; live gate PASS — two consecutive simultaneous-start trials (a full redeploy + a full-topology `docker compose restart`), all four pairs converged (every lower-address node founded self-first, every higher node probed-then-joined peer-first), zero splits. Default remains OFF elsewhere (wonder-app-vd03, docker-env2, Host defaults). See `docs/requirements/Component-ClusterInfrastructure.md` → Simultaneous cold start.
+1 -1
View File
@@ -388,7 +388,7 @@ A standby now **relays** the request to the active peer instead (`InboundApiEndp
- **Configuration Database (via IAuditService)**: All API key and method definition changes are audit logged.
- **Script Analysis (#25)**: `ForbiddenApiChecker` delegates to `ScriptTrustValidator.FindViolations` for the authoritative forbidden-API verdict during script compilation and validation.
- **Audit Log (#23)**: Every inbound API request emits an `ApiInbound.Completed` row via `ICentralAuditWriter` from request middleware (non-blocking for the HTTP response). Request and response bodies are captured in full up to `AuditLog:InboundMaxBytes` (default 1 MiB) per the Audit Log Payload Capture Policy; redaction (headers + per-target body redactors) still applies before persistence.
- **Cluster Infrastructure**: method execution is hosted on the active central node and fails over with it; `IActiveNodeGate` (Host) tells the endpoint filter whether to serve locally or relay.
- **Cluster Infrastructure**: method execution is hosted on the active central node and fails over with it; the shared `ZB.MOM.WW.Health.IActiveNodeGate` (registered by the Host as `ZB.MOM.WW.Health.Akka.AkkaActiveNodeGate` — oldest Up member via `ClusterActiveNode`, the same rule as `/health/active`; `false` until the ActorSystem exists and this member is Up) tells the endpoint filter whether to serve locally or relay. ScadaBridge has no private gate interface since Gitea #80.
## Interactions
@@ -1,52 +0,0 @@
using Akka.Cluster;
using ZB.MOM.WW.ScadaBridge.Host.Actors;
using ZB.MOM.WW.ScadaBridge.InboundAPI;
namespace ZB.MOM.WW.ScadaBridge.Host.Health;
/// <summary>
/// Production implementation of
/// <see cref="IActiveNodeGate"/> backed by the running Akka.NET cluster.
///
/// The inbound API is "Central cluster only (active node)" — a standby central
/// node must not execute method scripts or <c>Route.To()</c> calls. Active =
/// the oldest Up member (the singleton host), evaluated by
/// <see cref="ClusterActivityEvaluator.SelfIsOldest(Cluster, string?)"/> — NOT
/// the cluster leader, whose address-ordered definition diverges from singleton
/// placement after a restart (review 01 [High]). This is the same evaluator
/// backing the shared <c>ActiveNodeHealthCheck</c> registered on the active tier
/// — both route to <c>ClusterActiveNode</c> — so
/// <see cref="InboundApiEndpointFilter"/> can return HTTP 503 on a standby.
///
/// Registered only in the Central-role branch of <c>Program.cs</c>. The gate
/// is resolved per request from <c>HttpContext.RequestServices</c>; while the
/// <c>AkkaHostedService</c> is still warming up (<c>ActorSystem == null</c>)
/// or the node has not yet reached <see cref="MemberStatus.Up"/>, this
/// implementation reports <c>IsActiveNode == false</c> — the safe-by-default
/// answer matching the standby case.
/// </summary>
public sealed class ActiveNodeGate : IActiveNodeGate
{
private readonly AkkaHostedService _akkaService;
/// <summary>Initializes a new <see cref="ActiveNodeGate"/> bound to the given Akka hosted service.</summary>
/// <param name="akkaService">The Akka hosted service exposing the cluster's <see cref="Akka.Actor.ActorSystem"/>.</param>
public ActiveNodeGate(AkkaHostedService akkaService)
{
_akkaService = akkaService;
}
/// <inheritdoc />
public bool IsActiveNode
{
get
{
var system = _akkaService.ActorSystem;
if (system == null)
return false;
var cluster = Cluster.Get(system);
return ClusterActivityEvaluator.SelfIsOldest(cluster);
}
}
}
+11 -9
View File
@@ -603,15 +603,17 @@ try
sp.GetRequiredService<IOptions<NodeOptions>>(),
sp.GetRequiredService<ILogger<ClusterBootstrapCoordinator>>()));
// Register the production IActiveNodeGate implementation so
// standby-node gating is actually enforced (the InboundApiEndpointFilter
// consults IActiveNodeGate and defaults to "allow" when none is registered,
// which leaves the design's "central cluster only (active node)" guarantee
// unenforced in deployed binaries). The gate is backed by the same oldest-member
// evaluator (ClusterActivityEvaluator, which delegates to the shared ClusterActiveNode) as
// the active-node check above, so the inbound API and the /health/active endpoint Traefik
// routes against agree on which node is active.
builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.InboundAPI.IActiveNodeGate, ActiveNodeGate>();
// Register the production active-node gate so standby-node gating is actually enforced
// (the InboundApiEndpointFilter consults IActiveNodeGate and defaults to "allow" when none
// is registered, which leaves the design's "central cluster only (active node)" guarantee
// unenforced in deployed binaries). Since Gitea #80 this is the SHARED seam
// (ZB.MOM.WW.Health.IActiveNodeGate) with the shared AkkaActiveNodeGate, unscoped like the
// active-node check above — both route to ClusterActiveNode, so the inbound API and the
// /health/active endpoint Traefik routes against can never disagree on which node is
// active. Warm-up stays safe: the gate resolves the ActorSystem through the singleton
// bridge above and answers false until this member is Up (and on any cluster-init fault).
builder.Services.AddSingleton<ZB.MOM.WW.Health.IActiveNodeGate>(
sp => new ZB.MOM.WW.Health.Akka.AkkaActiveNodeGate(sp));
// Warm the inbound method compile cache in the BACKGROUND once the host is running and
// this node is the active one (review 2026-10-21 05/U3). Registered after
@@ -1,24 +0,0 @@
namespace ZB.MOM.WW.ScadaBridge.InboundAPI;
/// <summary>
/// Abstraction the inbound API endpoint uses to determine whether
/// this node is the active (cluster-leader) central node.
///
/// The design states the inbound API is "Central cluster only (active node)" and
/// "fails over with it". A standby central node must not execute method scripts or
/// <c>Route.To()</c> calls — that can race the active node or run against stale
/// singleton state. <see cref="InboundApiEndpointFilter"/> consults this gate and
/// returns HTTP 503 from a standby so Traefik/clients only reach the live node.
///
/// The implementation lives in the Host (it needs Akka cluster state); when no
/// implementation is registered, the endpoint defaults to "allow" so non-clustered
/// hosts and tests are unaffected.
/// </summary>
public interface IActiveNodeGate
{
/// <summary>
/// <c>true</c> when this node is the active central node and may serve the
/// inbound API; <c>false</c> on a standby node.
/// </summary>
bool IsActiveNode { get; }
}
@@ -3,6 +3,7 @@ using Microsoft.AspNetCore.Http.Features;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.Health;
namespace ZB.MOM.WW.ScadaBridge.InboundAPI;
@@ -2,6 +2,7 @@ using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.Health;
namespace ZB.MOM.WW.ScadaBridge.InboundAPI;
@@ -12,6 +12,7 @@ using ZB.MOM.WW.ScadaBridge.AuditLog.Configuration;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.Health;
namespace ZB.MOM.WW.ScadaBridge.InboundAPI.Middleware;
@@ -33,6 +33,10 @@
Abstractions contracts (IApiKeyVerifier / ApiKeyOptions) transitively. -->
<PackageReference Include="ZB.MOM.WW.Auth.ApiKeys" />
<PackageReference Include="ZB.MOM.WW.Configuration" />
<!-- Active-node gating (Gitea #80): InboundApiEndpointFilter, AuditWriteMiddleware and the
warm-up service consult the shared ZB.MOM.WW.Health.IActiveNodeGate; the Host registers
ZB.MOM.WW.Health.Akka's AkkaActiveNodeGate (oldest Up member via ClusterActiveNode). -->
<PackageReference Include="ZB.MOM.WW.Health" />
</ItemGroup>
</Project>
@@ -16,6 +16,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Scripts;
using ZB.MOM.WW.ScadaBridge.DeploymentManager;
using ZB.MOM.WW.ScadaBridge.InboundAPI;
using ZB.MOM.WW.ScadaBridge.Security;
using ZB.MOM.WW.Health;
namespace ZB.MOM.WW.ScadaBridge.ManagementService.FlowDebug;
@@ -93,7 +93,12 @@ akka {{
while (cluster.SelfMember.Status != Akka.Cluster.MemberStatus.Up && DateTime.UtcNow < deadline)
await Task.Delay(100);
var gate = new ActiveNodeGate(akkaService);
// Gitea #80: the inbound API's gate is the shared AkkaActiveNodeGate, resolving the
// ActorSystem from DI exactly as Program.cs's singleton bridge serves it.
using var sp = new ServiceCollection()
.AddSingleton(_ => akkaService.GetOrCreateActorSystem())
.BuildServiceProvider();
var gate = new ZB.MOM.WW.Health.Akka.AkkaActiveNodeGate(sp);
var provider = new AkkaClusterNodeProvider(akkaService, "Central");
Assert.True(gate.IsActiveNode);
Assert.True(provider.SelfIsPrimary);
@@ -104,6 +109,17 @@ akka {{
}
}
/// <summary>
/// Gitea #80 warm-up contract: while no ActorSystem is available the shared gate reports
/// standby, never active — the safe default the retired ScadaBridge gate also had.
/// </summary>
[Fact]
public void ActiveNodeGate_WithoutActorSystem_IsNotActive()
{
using var sp = new ServiceCollection().BuildServiceProvider();
Assert.False(new ZB.MOM.WW.Health.Akka.AkkaActiveNodeGate(sp).IsActiveNode);
}
private static int FreePort()
{
var l = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0);
@@ -332,7 +332,7 @@ public class CentralCompositionRootTests : IDisposable
/// <summary>
/// Regression test: the Central composition root MUST register a
/// concrete <see cref="IActiveNodeGate"/> implementation. Without it,
/// concrete <see cref="ZB.MOM.WW.Health.IActiveNodeGate"/> implementation. Without it,
/// <see cref="InboundApiEndpointFilter"/> falls through to "allow" and a
/// standby central node continues to serve the inbound API, racing the
/// active node and executing scripts against stale singleton state. The
@@ -344,12 +344,17 @@ public class CentralCompositionRootTests : IDisposable
/// indistinguishable from "missing" at runtime, which is the failure mode
/// the finding describes.
/// </summary>
/// <remarks>
/// Gitea #80: the gate is the shared <see cref="ZB.MOM.WW.Health.IActiveNodeGate"/> backed by
/// <see cref="ZB.MOM.WW.Health.Akka.AkkaActiveNodeGate"/> — the same ClusterActiveNode rule as
/// the /health/active probe — not a ScadaBridge-private seam.
/// </remarks>
[Fact]
public void Central_IActiveNodeGate_IsRegisteredAsActiveNodeGate()
{
var gate = _factory.Services.GetService<IActiveNodeGate>();
var gate = _factory.Services.GetService<ZB.MOM.WW.Health.IActiveNodeGate>();
Assert.NotNull(gate);
Assert.IsType<ActiveNodeGate>(gate);
Assert.IsType<ZB.MOM.WW.Health.Akka.AkkaActiveNodeGate>(gate);
}
// --- ScadaBridge#22 regression ---
@@ -3,6 +3,7 @@ using Microsoft.AspNetCore.Http.Features;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.Health;
namespace ZB.MOM.WW.ScadaBridge.InboundAPI.Tests;
@@ -4,6 +4,7 @@ using Microsoft.Extensions.Logging.Abstractions;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.InboundApi;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.Health;
namespace ZB.MOM.WW.ScadaBridge.InboundAPI.Tests;
@@ -12,6 +12,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.InboundAPI.Middleware;
using ZB.MOM.WW.Health;
namespace ZB.MOM.WW.ScadaBridge.InboundAPI.Tests.Middleware;
@@ -10,6 +10,7 @@ using ZB.MOM.WW.ScadaBridge.AuditLog.Configuration;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
using ZB.MOM.WW.ScadaBridge.InboundAPI.Middleware;
using ZB.MOM.WW.Health;
namespace ZB.MOM.WW.ScadaBridge.InboundAPI.Tests;
@@ -4,6 +4,7 @@ using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.Health;
namespace ZB.MOM.WW.ScadaBridge.InboundAPI.Tests;
@@ -18,6 +18,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening;
using ZB.MOM.WW.ScadaBridge.DeploymentManager;
using ZB.MOM.WW.ScadaBridge.InboundAPI;
using ZB.MOM.WW.ScadaBridge.ManagementService.FlowDebug;
using ZB.MOM.WW.Health;
namespace ZB.MOM.WW.ScadaBridge.ManagementService.Tests;