Joseph Doherty
1557 lines
89 KiB
Markdown
1557 lines
89 KiB
Markdown
# Cluster, Host & Failover Fixes Implementation Plan
|
|
|
|
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
|
|
|
|
**Goal:** Make automatic failover actually work (enable the SBR downing provider and prove it with a real two-node kill test), unify the "active node" definition on singleton-host (oldest) semantics, and close the CoordinatedShutdown, health-report-loss, offline-detection, and config-validation gaps from architecture review 01.
|
|
|
|
**Architecture:** All cluster bootstrap lives in `AkkaHostedService.BuildHocon` (hand-rolled, injection-safe HOCON) — the Critical fix is adding `akka.cluster.downing-provider-class` there, verified by a new reusable two-node in-process cluster fixture (`TwoNodeClusterFixture`) that other plans (02) also depend on. "Active node" is re-derived everywhere from a single new `ClusterActivityEvaluator` (self is the oldest `Up` member = singleton host), replacing the four divergent leader-based checks; the five copy-pasted central-singleton registration blocks are collapsed into a `CentralSingletonRegistrar` helper that also gives the two hot singletons (notification-outbox, audit-log-ingest) the drain tasks they were missing.
|
|
|
|
**Tech Stack:** C#/.NET, Akka.NET 1.5.62 (Akka.Cluster, Akka.Cluster.Tools, Akka.TestKit.Xunit2), xUnit + NSubstitute, Docker Compose + Traefik, Blazor Server (one small UI badge).
|
|
|
|
Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/<project>`. Docker rig: `bash docker/deploy.sh`.
|
|
|
|
## Findings Coverage
|
|
|
|
| # | Report finding | Severity | Task(s) / Disposition |
|
|
|---|---------------|----------|----------------------|
|
|
| S1 | Split-brain resolver never enabled — no automatic failover on crash/partition | Critical | Tasks 1, 3, 4, 21 |
|
|
| S2 | "Active node" = leader vs oldest diverge (purge on wrong node, wrong Primary label, wrong self-report, Traefik→non-singleton node) | High | Tasks 5, 6, 7, 8 |
|
|
| S3 | Partition: both central nodes return 200 on `/health/active`, Traefik serves both | High | Tasks 1 (SBR resolves partition), 7 (oldest-based + DB-required active check), 4 (behavioral proof) |
|
|
| S4 | Cluster-leave drain tasks budgeted 10s inside 5s CoordinatedShutdown phase | Medium | Task 2 |
|
|
| S5 | NotificationOutbox + AuditLogIngest singletons have no drain task | Medium | Tasks 9, 10 |
|
|
| S6 | Health-report loss recovery is dead code (fire-and-forget transport never throws) | Medium | Tasks 11, 12 |
|
|
| S7 | Offline detection keys off heartbeats, masking a dead metrics pipeline; spec/code disagree | Medium | Tasks 13, 14 |
|
|
| S8 | `appsettings.Site.json` second seed targets MetricsPort 8084; validator gap | Medium | Task 15 |
|
|
| S9 | Docker `stop_grace_period` shorter than full CoordinatedShutdown | Medium | Task 17 |
|
|
| S10 | Aggregator never forgets deleted sites | Low | Task 18 |
|
|
| S11 | Dead-letter monitor: unbounded warning volume | Low | Task 19 |
|
|
| S12 | Mandatory two-seed rule forces phantom seeds in single-node installs | Low | Task 16 |
|
|
| P1 | Readiness probes fan out 5 cluster Asks per poll | Low | **Accepted** — bounded (2s, concurrent), cheap at current poll rates; revisit only if an orchestrator tightens the probe interval |
|
|
| P2 | Startup compiles every inbound method sequentially | Low | **Deferred** — Inbound API domain (plan 06 scope); report itself says "acceptable now" |
|
|
| P3 | Per-tick allocations in health collection | Low | **Won't-fix** — report: "trivial at this cadence. No action" |
|
|
| C1 | REQ-HOST-6 mandates Akka.Hosting; code hand-rolls HOCON | Medium | Task 22 |
|
|
| C2 | `deploy/wonder-app-vd03` ships `DisableLogin: true` | High | **Deferred → Plan 07** (explicit cross-plan ownership) |
|
|
| C3 | LDAP plaintext in deploy artifact | Medium | **Deferred → Plan 07** (deploy-artifact security posture is plan 07's) |
|
|
| C4 | Secrets in checked-in docker dev configs | Medium | **Accepted** — labelled dev-only, `ConfigSecretsTests` tracks it, the wonder-app-vd03 `${...}` pattern already exists as the model |
|
|
| C5 | Stale `Component-TraefikProxy.md` references | Low | Task 23 |
|
|
| C6 | Five copy-pasted ~60-line singleton registration blocks | Low | Tasks 9, 10 |
|
|
| U1 | No real multi-node failover test anywhere | — | Tasks 3, 4, 8, 21 (the rig other plans reference) |
|
|
| U2 | Windows Service lifecycle / down-if-alone recovery story undefined | — | Task 20 |
|
|
| U3 | TLS absent at every layer of this domain | — | **Deferred** — documented lab posture; Task 23 adds an explicit "production TLS profile — not yet implemented" roadmap note so the gap is spec-visible |
|
|
| U4 | Dockerfile has no HEALTHCHECK; `/healthz` runs zero checks | — | **Deferred** — report calls this "acceptable minimalism"; prerequisite work for a future orchestrator migration, not this hardening pass |
|
|
| U5 | `NodeName` missing from wonder-app-vd03 overlays → NULL `SourceNode` | — | Task 23 |
|
|
| U6 | Known-but-open follow-ups (SecuredWrite SourceNode NULL, cert-trust persistence, docker-env2 mirrors findings) | — | **No action** — already tracked in CLAUDE.md/logged follow-ups; docker-env2 compose IS covered by Task 17 |
|
|
| U7 | Online/offline transitions unrecorded (post-incident "when did the site drop") | — | Tasks 13, 14 (`LastStatusChangeAt`) |
|
|
|
|
---
|
|
|
|
### Task 1: Enable the SBR downing provider in BuildHocon
|
|
|
|
**Classification:** high-risk (cluster failover semantics — the Critical finding)
|
|
**Estimated implement time:** ~3 min
|
|
**Parallelizable with:** 5, 11, 13, 15, 16, 17, 19, 22, 23
|
|
**Files:**
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (BuildHocon, cluster block lines 250-266)
|
|
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs`
|
|
|
|
1. Write the failing test in `HoconBuilderTests.cs` (reuse the existing `DefaultCluster()` helper):
|
|
|
|
```csharp
|
|
[Fact]
|
|
public void BuildHocon_EnablesSplitBrainResolverDowningProvider()
|
|
{
|
|
// Review 01 [Critical]: without downing-provider-class the entire
|
|
// split-brain-resolver section is inert (Akka default = NoDowning) and
|
|
// singletons never migrate on a hard crash.
|
|
var node = new NodeOptions { Role = "Central", NodeHostname = "node1", RemotingPort = 8081 };
|
|
|
|
var hocon = AkkaHostedService.BuildHocon(
|
|
node, DefaultCluster(), new[] { "Central" },
|
|
TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15));
|
|
|
|
var config = ConfigurationFactory.ParseString(hocon);
|
|
Assert.Equal(
|
|
"Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster",
|
|
config.GetString("akka.cluster.downing-provider-class"));
|
|
}
|
|
```
|
|
|
|
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~BuildHocon_EnablesSplitBrainResolverDowningProvider"` — expect FAIL (key missing).
|
|
3. In `BuildHocon`, add one line inside the `cluster {{` block, directly above `split-brain-resolver {{` (after `min-nr-of-members = {clusterOptions.MinNrOfMembers}`):
|
|
|
|
```csharp
|
|
downing-provider-class = ""Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster""
|
|
```
|
|
|
|
(inside the interpolated string — note doubled quotes). Also update the method's XML doc comment to state the downing provider is explicitly installed because Akka defaults to NoDowning.
|
|
4. Run the test again — expect PASS. Run the full file: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~HoconBuilderTests"` — all pass.
|
|
5. Commit:
|
|
```
|
|
git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs
|
|
git commit -m "fix(cluster): enable SBR downing provider — automatic failover was inert for crashes/partitions"
|
|
```
|
|
|
|
### Task 2: Raise the cluster-leave CoordinatedShutdown phase timeout above the drain budget
|
|
|
|
**Classification:** small
|
|
**Estimated implement time:** ~3 min
|
|
**Parallelizable with:** 5, 11, 13, 15, 16, 17, 19, 22, 23
|
|
**Files:**
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (BuildHocon, coordinated-shutdown block lines 267-269)
|
|
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs`
|
|
|
|
1. Write the failing test:
|
|
|
|
```csharp
|
|
[Fact]
|
|
public void BuildHocon_ClusterLeavePhaseTimeout_ExceedsDrainBudget()
|
|
{
|
|
// Review 01 [Medium]: the five singleton drain tasks GracefulStop(10s)
|
|
// inside cluster-leave, whose default phase timeout is 5s — Akka abandons
|
|
// the task at 5s and the drain guarantee is silently halved.
|
|
var node = new NodeOptions { Role = "Central", NodeHostname = "node1", RemotingPort = 8081 };
|
|
var hocon = AkkaHostedService.BuildHocon(
|
|
node, DefaultCluster(), new[] { "Central" },
|
|
TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15));
|
|
|
|
var config = ConfigurationFactory.ParseString(hocon);
|
|
var phaseTimeout = config.GetTimeSpan("akka.coordinated-shutdown.phases.cluster-leave.timeout");
|
|
Assert.True(phaseTimeout >= TimeSpan.FromSeconds(15),
|
|
$"cluster-leave phase timeout ({phaseTimeout}) must exceed the 10s singleton GracefulStop budget");
|
|
}
|
|
```
|
|
|
|
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~ClusterLeavePhaseTimeout"` — expect FAIL.
|
|
3. In `BuildHocon`, extend the `coordinated-shutdown` block:
|
|
|
|
```
|
|
coordinated-shutdown {{
|
|
run-by-clr-shutdown-hook = on
|
|
phases.cluster-leave.timeout = 15s
|
|
}}
|
|
```
|
|
|
|
4. Run again — expect PASS. Also run `--filter "FullyQualifiedName~CoordinatedShutdownTests"` (source-grep tests must stay green).
|
|
5. Commit:
|
|
```
|
|
git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs
|
|
git commit -m "fix(cluster): give cluster-leave phase a 15s timeout so 10s singleton drains can complete"
|
|
```
|
|
|
|
### Task 3: Two-node in-process cluster fixture (the reusable failover rig)
|
|
|
|
**Classification:** high-risk (real cluster formation in tests)
|
|
**Estimated implement time:** ~5 min
|
|
**Parallelizable with:** 5, 11, 13, 15, 16, 17, 19, 22, 23
|
|
**Files:**
|
|
- Create: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixture.cs`
|
|
- Test: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixtureTests.cs`
|
|
|
|
This fixture is the rig other plans (02: S&F standby gating) reference — keep its API small and public. It builds each node's config from the **production** `AkkaHostedService.BuildHocon` (so the config under test is the config that ships), overlaid with test-only crash-simulation settings.
|
|
|
|
1. Create the fixture:
|
|
|
|
```csharp
|
|
using Akka.Actor;
|
|
using Akka.Cluster;
|
|
using Akka.Configuration;
|
|
using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
|
|
using ZB.MOM.WW.ScadaBridge.Host;
|
|
using ZB.MOM.WW.ScadaBridge.Host.Actors;
|
|
|
|
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
|
|
|
|
/// <summary>
|
|
/// Forms a REAL two-node Akka cluster in-process from the production
|
|
/// BuildHocon output. NodeA is always started first (=> oldest member /
|
|
/// singleton host). CrashNode() terminates a node WITHOUT cluster-leave
|
|
/// (coordinated-shutdown by-terminate disabled on both nodes) to simulate
|
|
/// a hard crash — the scenario review 01 found untested.
|
|
/// </summary>
|
|
public sealed class TwoNodeClusterFixture : IAsyncDisposable
|
|
{
|
|
public ActorSystem NodeA { get; private set; } = null!;
|
|
public ActorSystem NodeB { get; private set; } = null!;
|
|
public int PortA { get; private set; }
|
|
public int PortB { get; private set; }
|
|
|
|
public static async Task<TwoNodeClusterFixture> StartAsync(
|
|
string role = "Central", TimeSpan? stableAfter = null)
|
|
{
|
|
var f = new TwoNodeClusterFixture();
|
|
f.PortA = GetFreeTcpPort();
|
|
f.PortB = GetFreeTcpPort();
|
|
f.NodeA = f.StartNode(f.PortA, role, stableAfter);
|
|
await WaitForMembersUp(f.NodeA, 1, TimeSpan.FromSeconds(20));
|
|
f.NodeB = f.StartNode(f.PortB, role, stableAfter);
|
|
await WaitForMembersUp(f.NodeA, 2, TimeSpan.FromSeconds(20));
|
|
await WaitForMembersUp(f.NodeB, 2, TimeSpan.FromSeconds(20));
|
|
return f;
|
|
}
|
|
|
|
/// <summary>Starts a node from production HOCON; used by StartAsync and by restart-scenarios.</summary>
|
|
public ActorSystem StartNode(int port, string role, TimeSpan? stableAfter = null)
|
|
{
|
|
var nodeOptions = new NodeOptions { Role = role, NodeHostname = "127.0.0.1", RemotingPort = port };
|
|
var clusterOptions = new ClusterOptions
|
|
{
|
|
SeedNodes = new List<string>
|
|
{
|
|
$"akka.tcp://scadabridge@127.0.0.1:{PortA}",
|
|
$"akka.tcp://scadabridge@127.0.0.1:{PortB}",
|
|
},
|
|
SplitBrainResolverStrategy = "keep-oldest",
|
|
StableAfter = stableAfter ?? TimeSpan.FromSeconds(3),
|
|
HeartbeatInterval = TimeSpan.FromMilliseconds(500),
|
|
FailureDetectionThreshold = TimeSpan.FromSeconds(2),
|
|
MinNrOfMembers = 1,
|
|
DownIfAlone = true,
|
|
};
|
|
var hocon = AkkaHostedService.BuildHocon(
|
|
nodeOptions, clusterOptions, new[] { role },
|
|
TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(3));
|
|
// Crash simulation: Terminate() must NOT run CoordinatedShutdown
|
|
// (no Leave gossip) or the test degenerates to the graceful path.
|
|
var config = ConfigurationFactory
|
|
.ParseString("akka.coordinated-shutdown.run-by-actor-system-terminate = off")
|
|
.WithFallback(ConfigurationFactory.ParseString(hocon));
|
|
return ActorSystem.Create("scadabridge", config);
|
|
}
|
|
|
|
/// <summary>Hard-crash a node: abrupt terminate, no cluster leave.</summary>
|
|
public static Task CrashNode(ActorSystem node) => node.Terminate();
|
|
|
|
public static async Task WaitForMembersUp(ActorSystem system, int count, TimeSpan timeout)
|
|
{
|
|
var cluster = Akka.Cluster.Cluster.Get(system);
|
|
var deadline = DateTime.UtcNow + timeout;
|
|
while (DateTime.UtcNow < deadline)
|
|
{
|
|
if (cluster.State.Members.Count(m => m.Status == MemberStatus.Up) == count
|
|
&& !cluster.State.Unreachable.Any())
|
|
return;
|
|
await Task.Delay(200);
|
|
}
|
|
throw new TimeoutException(
|
|
$"Cluster on {cluster.SelfAddress} did not reach {count} Up members within {timeout}. " +
|
|
$"Members: [{string.Join(", ", cluster.State.Members)}], " +
|
|
$"Unreachable: [{string.Join(", ", cluster.State.Unreachable)}]");
|
|
}
|
|
|
|
public static async Task WaitForMemberRemoved(ActorSystem survivor, Address removed, TimeSpan timeout)
|
|
{
|
|
var cluster = Akka.Cluster.Cluster.Get(survivor);
|
|
var deadline = DateTime.UtcNow + timeout;
|
|
while (DateTime.UtcNow < deadline)
|
|
{
|
|
if (cluster.State.Members.All(m => m.Address != removed)
|
|
&& cluster.State.Unreachable.All(m => m.Address != removed))
|
|
return;
|
|
await Task.Delay(200);
|
|
}
|
|
throw new TimeoutException($"Member {removed} was never removed from {cluster.SelfAddress}'s view — SBR did not down it.");
|
|
}
|
|
|
|
private static int GetFreeTcpPort()
|
|
{
|
|
var listener = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0);
|
|
listener.Start();
|
|
var port = ((System.Net.IPEndPoint)listener.LocalEndpoint).Port;
|
|
listener.Stop();
|
|
return port;
|
|
}
|
|
|
|
public async ValueTask DisposeAsync()
|
|
{
|
|
foreach (var sys in new[] { NodeA, NodeB })
|
|
{
|
|
if (sys is { WhenTerminated.IsCompleted: false })
|
|
{
|
|
try { await sys.Terminate().WaitAsync(TimeSpan.FromSeconds(10)); } catch { /* teardown */ }
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
2. Write the smoke test proving the fixture forms a real cluster:
|
|
|
|
```csharp
|
|
public class TwoNodeClusterFixtureTests
|
|
{
|
|
[Fact]
|
|
public async Task Fixture_FormsTwoNodeCluster_NodeAIsOldest()
|
|
{
|
|
await using var cluster = await TwoNodeClusterFixture.StartAsync();
|
|
var a = Akka.Cluster.Cluster.Get(cluster.NodeA);
|
|
var b = Akka.Cluster.Cluster.Get(cluster.NodeB);
|
|
Assert.Equal(2, a.State.Members.Count);
|
|
Assert.Equal(2, b.State.Members.Count);
|
|
// NodeA started first => oldest (the singleton host).
|
|
Assert.True(a.SelfMember.IsOlderThan(b.State.Members.First(m => m.Address == b.SelfAddress)));
|
|
}
|
|
}
|
|
```
|
|
|
|
3. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~TwoNodeClusterFixtureTests"` — expect FAIL first (files don't exist / compile), then PASS once implemented. (This task is fixture-first; the "failing test" step is the compile failure.)
|
|
4. If `NodeOptions`/`BuildHocon` visibility blocks compilation, note that `IntegrationTests` already references the Host project (`.csproj` line 37) and both are public — no changes to `src/` are expected in this task.
|
|
5. Commit:
|
|
```
|
|
git add tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/
|
|
git commit -m "test(cluster): add TwoNodeClusterFixture — real two-node in-process cluster rig from production HOCON"
|
|
```
|
|
|
|
### Task 4: SBR behavioral proof — kill the oldest node, assert downing + singleton migration
|
|
|
|
**Classification:** high-risk
|
|
**Estimated implement time:** ~5 min
|
|
**Parallelizable with:** 5, 11, 13, 15, 16, 17, 19, 22, 23
|
|
**Files:**
|
|
- Create: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SbrFailoverTests.cs`
|
|
|
|
1. Write the test (this is the test that would have caught the Critical finding — it must FAIL if Task 1's HOCON line is reverted; verify that once by stashing Task 1 if cheap, otherwise trust the assertion on member removal, which is impossible under NoDowning):
|
|
|
|
```csharp
|
|
using Akka.Actor;
|
|
using Akka.Cluster.Tools.Singleton;
|
|
|
|
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
|
|
|
|
public class SbrFailoverTests
|
|
{
|
|
private sealed class EchoActor : ReceiveActor
|
|
{
|
|
public EchoActor() => ReceiveAny(msg => Sender.Tell(msg));
|
|
}
|
|
|
|
private static (IActorRef manager, IActorRef proxy) StartSingleton(ActorSystem sys)
|
|
{
|
|
var manager = sys.ActorOf(ClusterSingletonManager.Props(
|
|
Props.Create(() => new EchoActor()),
|
|
PoisonPill.Instance,
|
|
ClusterSingletonManagerSettings.Create(sys).WithSingletonName("failover-probe")),
|
|
"failover-probe-singleton");
|
|
var proxy = sys.ActorOf(ClusterSingletonProxy.Props(
|
|
"/user/failover-probe-singleton",
|
|
ClusterSingletonProxySettings.Create(sys).WithSingletonName("failover-probe")),
|
|
"failover-probe-proxy");
|
|
return (manager, proxy);
|
|
}
|
|
|
|
[Fact]
|
|
public async Task HardCrashOfOldestNode_SbrDownsIt_AndSingletonMigratesToSurvivor()
|
|
{
|
|
await using var cluster = await TwoNodeClusterFixture.StartAsync();
|
|
StartSingleton(cluster.NodeA);
|
|
var (_, proxyB) = StartSingleton(cluster.NodeB);
|
|
|
|
// Singleton is reachable from B while A (oldest) hosts it.
|
|
var echo = await proxyB.Ask<string>("ping", TimeSpan.FromSeconds(20));
|
|
Assert.Equal("ping", echo);
|
|
|
|
var victimAddress = Akka.Cluster.Cluster.Get(cluster.NodeA).SelfAddress;
|
|
await TwoNodeClusterFixture.CrashNode(cluster.NodeA);
|
|
|
|
// 1) SBR must DOWN and REMOVE the crashed member (NoDowning => this
|
|
// times out; the member stays unreachable forever).
|
|
// Budget: failure detection (~2s) + stable-after (3s) + gossip margin.
|
|
await TwoNodeClusterFixture.WaitForMemberRemoved(
|
|
cluster.NodeB, victimAddress, TimeSpan.FromSeconds(30));
|
|
|
|
// 2) The singleton must re-host on the survivor and answer again.
|
|
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30);
|
|
Exception? last = null;
|
|
while (DateTime.UtcNow < deadline)
|
|
{
|
|
try
|
|
{
|
|
var echo2 = await proxyB.Ask<string>("ping-after-failover", TimeSpan.FromSeconds(3));
|
|
Assert.Equal("ping-after-failover", echo2);
|
|
return;
|
|
}
|
|
catch (Exception ex) { last = ex; }
|
|
}
|
|
throw new Xunit.Sdk.XunitException($"Singleton never re-hosted on the survivor after SBR downing: {last}");
|
|
}
|
|
}
|
|
```
|
|
|
|
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~SbrFailoverTests"` — expect PASS (Task 1 is in). To confirm the test has teeth, temporarily comment out the `downing-provider-class` line in `BuildHocon`, re-run, expect FAIL at `WaitForMemberRemoved` (timeout), then restore the line. Do NOT commit the reverted state.
|
|
3. Commit:
|
|
```
|
|
git add tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SbrFailoverTests.cs
|
|
git commit -m "test(cluster): prove SBR downs a hard-crashed oldest node and the singleton migrates"
|
|
```
|
|
|
|
### Task 5: ClusterActivityEvaluator — single oldest-member "active node" definition
|
|
|
|
**Classification:** standard
|
|
**Estimated implement time:** ~4 min
|
|
**Parallelizable with:** 1, 2, 3, 4, 11, 13, 15, 16, 17, 19, 22, 23
|
|
**Files:**
|
|
- Create: `src/ZB.MOM.WW.ScadaBridge.Host/Health/ClusterActivityEvaluator.cs`
|
|
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ClusterActivityEvaluatorTests.cs`
|
|
|
|
1. Write the failing test (real single-node cluster via TestKit-style bootstrap, mirroring `AkkaBootstrapTests`):
|
|
|
|
```csharp
|
|
using Akka.Actor;
|
|
using Akka.Configuration;
|
|
using ZB.MOM.WW.ScadaBridge.Host.Health;
|
|
|
|
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
|
|
|
|
public class ClusterActivityEvaluatorTests : IAsyncLifetime
|
|
{
|
|
private ActorSystem? _system;
|
|
|
|
public async Task InitializeAsync()
|
|
{
|
|
var port = FreePort();
|
|
var config = ConfigurationFactory.ParseString($@"
|
|
akka {{
|
|
actor.provider = cluster
|
|
remote.dot-netty.tcp {{ hostname = ""127.0.0.1"", port = {port} }}
|
|
cluster {{
|
|
seed-nodes = [""akka.tcp://eval-test@127.0.0.1:{port}""]
|
|
roles = [""Central""]
|
|
min-nr-of-members = 1
|
|
}}
|
|
}}");
|
|
_system = ActorSystem.Create("eval-test", config);
|
|
var cluster = Akka.Cluster.Cluster.Get(_system);
|
|
var deadline = DateTime.UtcNow.AddSeconds(20);
|
|
while (cluster.SelfMember.Status != Akka.Cluster.MemberStatus.Up && DateTime.UtcNow < deadline)
|
|
await Task.Delay(100);
|
|
}
|
|
|
|
public async Task DisposeAsync() { if (_system != null) await _system.Terminate(); }
|
|
|
|
[Fact]
|
|
public void SelfIsOldest_SoleUpMember_ReturnsTrue()
|
|
{
|
|
var cluster = Akka.Cluster.Cluster.Get(_system!);
|
|
Assert.True(ClusterActivityEvaluator.SelfIsOldest(cluster));
|
|
Assert.True(ClusterActivityEvaluator.SelfIsOldest(cluster, "Central"));
|
|
}
|
|
|
|
[Fact]
|
|
public void SelfIsOldest_RoleNotHeld_ReturnsFalse()
|
|
{
|
|
var cluster = Akka.Cluster.Cluster.Get(_system!);
|
|
// Self doesn't carry the role => it can never host that role's singletons.
|
|
Assert.False(ClusterActivityEvaluator.SelfIsOldest(cluster, "site-nonexistent"));
|
|
}
|
|
|
|
private static int FreePort()
|
|
{
|
|
var l = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0);
|
|
l.Start(); var p = ((System.Net.IPEndPoint)l.LocalEndpoint).Port; l.Stop(); return p;
|
|
}
|
|
}
|
|
```
|
|
|
|
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~ClusterActivityEvaluatorTests"` — expect FAIL (type missing).
|
|
3. Implement:
|
|
|
|
```csharp
|
|
using Akka.Cluster;
|
|
|
|
namespace ZB.MOM.WW.ScadaBridge.Host.Health;
|
|
|
|
/// <summary>
|
|
/// THE single definition of "active node" (review 01 [High]): a node is active
|
|
/// when it is the OLDEST Up member (optionally within a role scope) — i.e. the
|
|
/// member the ClusterSingletonManager places singletons on. Cluster LEADERSHIP
|
|
/// (lowest address) is an Akka-internal concept that diverges from singleton
|
|
/// placement permanently once the original first node restarts and rejoins;
|
|
/// every product-level active/standby decision must use this evaluator, never
|
|
/// cluster.State.Leader.
|
|
/// </summary>
|
|
public static class ClusterActivityEvaluator
|
|
{
|
|
/// <summary>True when self is Up and no other Up member (in the role scope) is older.</summary>
|
|
public static bool SelfIsOldest(Cluster cluster, string? role = null)
|
|
{
|
|
var self = cluster.SelfMember;
|
|
if (self.Status != MemberStatus.Up)
|
|
return false;
|
|
if (role != null && !self.HasRole(role))
|
|
return false;
|
|
|
|
return cluster.State.Members
|
|
.Where(m => m.Status == MemberStatus.Up)
|
|
.Where(m => role == null || m.HasRole(role))
|
|
.All(m => m.UniqueAddress.Equals(self.UniqueAddress) || self.IsOlderThan(m));
|
|
}
|
|
|
|
/// <summary>The oldest Up member in the role scope, or null while none is Up. Used for Primary/Standby labelling.</summary>
|
|
public static Member? OldestUpMember(Cluster cluster, string? role = null)
|
|
{
|
|
Member? oldest = null;
|
|
foreach (var m in cluster.State.Members.Where(m => m.Status == MemberStatus.Up))
|
|
{
|
|
if (role != null && !m.HasRole(role)) continue;
|
|
if (oldest == null || m.IsOlderThan(oldest)) oldest = m;
|
|
}
|
|
return oldest;
|
|
}
|
|
}
|
|
```
|
|
|
|
4. Run — expect PASS.
|
|
5. Commit:
|
|
```
|
|
git add src/ZB.MOM.WW.ScadaBridge.Host/Health/ClusterActivityEvaluator.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ClusterActivityEvaluatorTests.cs
|
|
git commit -m "feat(host): ClusterActivityEvaluator — oldest-member (singleton-host) active-node definition"
|
|
```
|
|
|
|
### Task 6: Rewire ActiveNodeGate, AkkaClusterNodeProvider and Primary labels onto the evaluator
|
|
|
|
**Classification:** high-risk (changes which node the purge, self-report, inbound gate and dashboard consider active)
|
|
**Estimated implement time:** ~5 min
|
|
**Parallelizable with:** 11, 13, 15, 16, 17, 19, 22, 23
|
|
**Files:**
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeGate.cs` (lines 36-52)
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaClusterNodeProvider.cs` (lines 29-40, 43-81)
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/IClusterNodeProvider.cs` (SelfIsPrimary xmldoc, lines 15-21)
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs` (comment at lines 109-115 — text only)
|
|
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ClusterActivityEvaluatorTests.cs` (extend)
|
|
|
|
1. Write the failing test (extend `ClusterActivityEvaluatorTests` — the single-node cluster from Task 5 is enough because with one member, leader == oldest; the *divergence* behavior is proven in Task 8's two-node test. Here assert the wiring):
|
|
|
|
```csharp
|
|
[Fact]
|
|
public void ActiveNodeGate_And_NodeProvider_AgreeWithEvaluator()
|
|
{
|
|
// Both product gates must be backed by the oldest-member evaluator.
|
|
// A single-node Up cluster: evaluator says true, so both must say true.
|
|
var akkaService = /* construct AkkaHostedService test double or use the
|
|
pattern in ActorSystemBridgeTests to wrap _system */;
|
|
var gate = new ActiveNodeGate(akkaService);
|
|
var provider = new AkkaClusterNodeProvider(akkaService, "Central");
|
|
Assert.True(gate.IsActiveNode);
|
|
Assert.True(provider.SelfIsPrimary);
|
|
}
|
|
```
|
|
|
|
Follow the construction pattern used by `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ActorSystemBridgeTests.cs` / `HealthCheckTests.cs` for obtaining an `AkkaHostedService` whose `ActorSystem` is the test system (if `AkkaHostedService` cannot be instantiated cheaply, add an internal test seam — but check those tests first; a pattern exists because `ActiveNodeGate` is already constructor-tested).
|
|
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~AgreeWithEvaluator"` — expect FAIL (or compile error) before rewiring if you assert on a divergence scenario; if single-node passes trivially pre-change, treat this as a pinning test and rely on Task 8 for the behavioral delta.
|
|
3. Implement:
|
|
- `ActiveNodeGate.IsActiveNode`: replace the leader comparison (lines 44-51) with `return ClusterActivityEvaluator.SelfIsOldest(cluster);` (keep the null-system and `Up` guards; update the class xmldoc: "oldest Up member — the singleton host — not the cluster leader").
|
|
- `AkkaClusterNodeProvider.SelfIsPrimary`: replace leader comparison with `return ClusterActivityEvaluator.SelfIsOldest(cluster, _siteRole);`.
|
|
- `AkkaClusterNodeProvider.GetClusterNodes`: replace `var leader = cluster.State.Leader;` + `isLeader` with `var oldest = ClusterActivityEvaluator.OldestUpMember(cluster, _siteRole);` and `var isPrimary = oldest != null && member.UniqueAddress.Equals(oldest.UniqueAddress);` — label `"Primary"`/`"Standby"` from that.
|
|
- `IClusterNodeProvider.SelfIsPrimary` xmldoc: change "currently the cluster leader (Primary)" to "currently the oldest Up member for the provider's role scope — the node the ClusterSingletonManager hosts singletons on. This is the canonical product 'active node' check (review 01): the site event-log purge gate, the central self-report loop, and (via plan 02) the S&F delivery gate all consume it."
|
|
- `SiteServiceRegistration.cs` lines 109-115 comment: replace "this node is Up AND cluster leader" wording with "this node is the oldest Up member (singleton host)".
|
|
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` and `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests` — all pass (CentralHealthReportLoop/EventLogPurge consumers are interface-driven; their fakes are unaffected).
|
|
5. Commit:
|
|
```
|
|
git add src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeGate.cs src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaClusterNodeProvider.cs src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/IClusterNodeProvider.cs src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ClusterActivityEvaluatorTests.cs
|
|
git commit -m "fix(host): unify active-node on oldest-member semantics — purge/self-report/labels now track the singleton host"
|
|
```
|
|
|
|
**Seam note for plan 02:** after this task, `IClusterNodeProvider.SelfIsPrimary` (registered in site DI at `SiteServiceRegistration.cs:101-107`) IS the canonical "this node hosts the site singletons / is active" check. Plan 02's S&F delivery gating should inject `IClusterNodeProvider` (or register a delegate mirroring `SiteEventLogActiveNodeCheck`, `SiteServiceRegistration.cs:116-120`) rather than invent a new check.
|
|
|
|
### Task 7: Oldest-based /health/active check + require DB reachability for Active
|
|
|
|
**Classification:** high-risk (changes Traefik routing behavior)
|
|
**Estimated implement time:** ~5 min
|
|
**Parallelizable with:** 11, 13, 15, 16, 17, 19, 22, 23
|
|
**Files:**
|
|
- Create: `src/ZB.MOM.WW.ScadaBridge.Host/Health/OldestNodeActiveHealthCheck.cs`
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (health check registrations, lines 209-234; comment block lines 346-356)
|
|
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HealthCheckTests.cs`
|
|
|
|
1. Write failing tests (follow the existing registration-introspection style in `HealthCheckTests.cs` — it already builds the central service collection and inspects `HealthCheckServiceOptions`):
|
|
|
|
```csharp
|
|
[Fact]
|
|
public void ActiveNodeCheck_IsHostLocalOldestNodeCheck()
|
|
{
|
|
var registrations = GetCentralHealthCheckRegistrations(); // existing helper or mirror existing tests
|
|
var active = registrations.Single(r => r.Name == "active-node");
|
|
Assert.Contains(ZbHealthTags.Active, active.Tags);
|
|
// The shared package's ActiveNodeHealthCheck is LEADER-based (review 01 [High]):
|
|
// during a partition both nodes think they lead and Traefik serves both.
|
|
var check = active.Factory(BuildCentralServiceProvider());
|
|
Assert.IsType<OldestNodeActiveHealthCheck>(check);
|
|
}
|
|
|
|
[Fact]
|
|
public void DatabaseCheck_AlsoGatesActive()
|
|
{
|
|
// Review 01 [High] recommendation: /health/active should require Ready
|
|
// (DB reachable) in addition to singleton-host status, so a DB-dead
|
|
// active node is pulled from Traefik rotation.
|
|
var registrations = GetCentralHealthCheckRegistrations();
|
|
var db = registrations.Single(r => r.Name == "database");
|
|
Assert.Contains(ZbHealthTags.Ready, db.Tags);
|
|
Assert.Contains(ZbHealthTags.Active, db.Tags);
|
|
}
|
|
```
|
|
|
|
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~HealthCheckTests"` — expect FAIL.
|
|
3. Implement:
|
|
- New `OldestNodeActiveHealthCheck`:
|
|
|
|
```csharp
|
|
using Akka.Actor;
|
|
using Microsoft.Extensions.Diagnostics.HealthChecks;
|
|
|
|
namespace ZB.MOM.WW.ScadaBridge.Host.Health;
|
|
|
|
/// <summary>
|
|
/// /health/active check: Healthy only on the oldest Up member — the node that
|
|
/// hosts the cluster singletons. Replaces the shared package's leader-based
|
|
/// ActiveNodeHealthCheck (review 01 [High]): leadership is address-ordered and
|
|
/// diverges from singleton placement after a restart, and during a partition
|
|
/// BOTH nodes compute themselves leader, so Traefik served both. Oldest-member
|
|
/// semantics keep exactly one active node through a partition (the non-oldest
|
|
/// side still sees the old oldest member until SBR resolves membership).
|
|
/// </summary>
|
|
public sealed class OldestNodeActiveHealthCheck : IHealthCheck
|
|
{
|
|
private readonly ActorSystem _system;
|
|
public OldestNodeActiveHealthCheck(ActorSystem system) => _system = system;
|
|
|
|
public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken ct = default)
|
|
{
|
|
var cluster = Akka.Cluster.Cluster.Get(_system);
|
|
return Task.FromResult(ClusterActivityEvaluator.SelfIsOldest(cluster)
|
|
? HealthCheckResult.Healthy("oldest Up member (singleton host)")
|
|
: HealthCheckResult.Unhealthy("not the oldest Up member (standby)"));
|
|
}
|
|
}
|
|
```
|
|
|
|
- In `Program.cs`: replace `.AddTypeActivatedCheck<ActiveNodeHealthCheck>("active-node", ...)` with `.AddTypeActivatedCheck<OldestNodeActiveHealthCheck>("active-node", failureStatus: null, tags: new[] { ZbHealthTags.Active })`; change the database check's tags to `new[] { ZbHealthTags.Ready, ZbHealthTags.Active }`; update the tier-separation comment block (lines 204-208 and 346-356) to say "active-node = oldest Up member + database reachable" and remove the now-stale `ActiveNodeHealthCheck` mention. Keep the `ActiveNodeGate` registration comment (lines 252-260) in sync ("same oldest-member evaluator as OldestNodeActiveHealthCheck").
|
|
4. Run — expect PASS. Also run `--filter "FullyQualifiedName~RequiredSingletonsHealthCheckTests"` (readiness untouched) and build the solution.
|
|
5. Commit:
|
|
```
|
|
git add src/ZB.MOM.WW.ScadaBridge.Host/Health/OldestNodeActiveHealthCheck.cs src/ZB.MOM.WW.ScadaBridge.Host/Program.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HealthCheckTests.cs
|
|
git commit -m "fix(host): /health/active = oldest member + DB reachable — partition-safe Traefik routing"
|
|
```
|
|
|
|
### Task 8: Divergence proof — restarted first node must NOT become active again
|
|
|
|
**Classification:** high-risk
|
|
**Estimated implement time:** ~5 min
|
|
**Parallelizable with:** 11, 13, 15, 16, 17, 19, 22, 23
|
|
**Files:**
|
|
- Create: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/ActiveNodeSemanticsTests.cs`
|
|
|
|
1. Write the test using the Task 3 fixture (this is the exact scenario from the report: after the original first node restarts and rejoins, leadership returns to it by address order while the peer remains oldest — the two definitions permanently diverge):
|
|
|
|
```csharp
|
|
using ZB.MOM.WW.ScadaBridge.Host.Health;
|
|
|
|
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
|
|
|
|
public class ActiveNodeSemanticsTests
|
|
{
|
|
[Fact]
|
|
public async Task AfterOldestNodeRestarts_SurvivorStaysActive_RejoinerIsStandby()
|
|
{
|
|
await using var fixture = await TwoNodeClusterFixture.StartAsync();
|
|
|
|
// Crash A (oldest), let SBR remove it, B becomes oldest.
|
|
var victim = Akka.Cluster.Cluster.Get(fixture.NodeA).SelfAddress;
|
|
await TwoNodeClusterFixture.CrashNode(fixture.NodeA);
|
|
await TwoNodeClusterFixture.WaitForMemberRemoved(fixture.NodeB, victim, TimeSpan.FromSeconds(30));
|
|
|
|
// Restart A on the same port — it rejoins as the YOUNGEST member but
|
|
// (lowest address) may regain Akka leadership.
|
|
var rejoined = fixture.StartNode(fixture.PortA, "Central");
|
|
try
|
|
{
|
|
await TwoNodeClusterFixture.WaitForMembersUp(rejoined, 2, TimeSpan.FromSeconds(30));
|
|
await TwoNodeClusterFixture.WaitForMembersUp(fixture.NodeB, 2, TimeSpan.FromSeconds(30));
|
|
|
|
var clusterB = Akka.Cluster.Cluster.Get(fixture.NodeB);
|
|
var clusterA2 = Akka.Cluster.Cluster.Get(rejoined);
|
|
|
|
// Product active-node semantics: survivor (oldest, singleton host)
|
|
// is active; the rejoined original node is standby — even when it
|
|
// holds Akka leadership.
|
|
Assert.True(ClusterActivityEvaluator.SelfIsOldest(clusterB));
|
|
Assert.False(ClusterActivityEvaluator.SelfIsOldest(clusterA2));
|
|
}
|
|
finally
|
|
{
|
|
await rejoined.Terminate();
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.IntegrationTests --filter "FullyQualifiedName~ActiveNodeSemanticsTests"` — expect PASS. Sanity: assert the OLD leader-based expression (`clusterA2.State.Leader == clusterA2.SelfAddress`) would have returned true for the rejoiner in at least one run — if it reliably does, add it as an inline comment documenting why leader-based was wrong (don't make the test depend on Akka leadership internals).
|
|
3. Commit:
|
|
```
|
|
git add tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/ActiveNodeSemanticsTests.cs
|
|
git commit -m "test(cluster): restarted original node stays standby — oldest-member active semantics proven"
|
|
```
|
|
|
|
### Task 9: CentralSingletonRegistrar helper (manager + drain task + proxy)
|
|
|
|
**Classification:** high-risk (actor lifecycle)
|
|
**Estimated implement time:** ~5 min
|
|
**Parallelizable with:** 11, 13, 15, 16, 17, 19, 22, 23
|
|
**Files:**
|
|
- Create: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/CentralSingletonRegistrar.cs`
|
|
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CentralSingletonRegistrarTests.cs`
|
|
|
|
1. Write the failing test (single-node cluster, real CoordinatedShutdown run):
|
|
|
|
```csharp
|
|
using Akka.Actor;
|
|
using Akka.Configuration;
|
|
using Microsoft.Extensions.Logging.Abstractions;
|
|
using ZB.MOM.WW.ScadaBridge.Host.Actors;
|
|
|
|
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
|
|
|
|
public class CentralSingletonRegistrarTests
|
|
{
|
|
private sealed class EchoActor : ReceiveActor
|
|
{
|
|
public EchoActor() => ReceiveAny(m => Sender.Tell(m));
|
|
}
|
|
|
|
[Fact]
|
|
public async Task Start_CreatesManagerAndProxy_WithCanonicalNames_AndDrainTask()
|
|
{
|
|
using var system = CreateSingleNodeClusterSystem(); // same bootstrap as ClusterActivityEvaluatorTests
|
|
var handle = CentralSingletonRegistrar.Start(
|
|
system, "test-widget", Props.Create(() => new EchoActor()),
|
|
NullLogger.Instance, drainTimeout: TimeSpan.FromSeconds(2));
|
|
|
|
// Canonical naming preserved: <name>-singleton / <name>-proxy.
|
|
Assert.EndsWith("/user/test-widget-singleton", handle.Manager.Path.ToString());
|
|
Assert.EndsWith("/user/test-widget-proxy", handle.Proxy.Path.ToString());
|
|
|
|
// Singleton answers through the proxy once the member is Up.
|
|
var echo = await handle.Proxy.Ask<string>("hi", TimeSpan.FromSeconds(20));
|
|
Assert.Equal("hi", echo);
|
|
|
|
// The drain task must stop the manager during cluster-leave.
|
|
var probeSystem = system; // watch from a test actor
|
|
var watcher = probeSystem.ActorOf(Props.Create(() => new WatchActor(handle.Manager)));
|
|
await Akka.Actor.CoordinatedShutdown.Get(system).Run(Akka.Actor.CoordinatedShutdown.ClrExitReason.Instance);
|
|
// Manager terminated => drain ran (GracefulStop completed or PoisonPill fallback).
|
|
Assert.True(handle.Manager.IsNobody() || system.WhenTerminated.IsCompleted);
|
|
}
|
|
}
|
|
```
|
|
|
|
(Use a `WatchActor`/`TestProbe`-style Terminated assertion if `IsNobody()` proves flaky — the load-bearing assertions are the two path names and the pre-shutdown echo; the drain execution is additionally covered by the existing five singletons' behavior in Task 10's regression run.)
|
|
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~CentralSingletonRegistrarTests"` — expect FAIL (type missing).
|
|
3. Implement `CentralSingletonRegistrar`:
|
|
|
|
```csharp
|
|
using Akka.Actor;
|
|
using Akka.Cluster.Tools.Singleton;
|
|
using Microsoft.Extensions.Logging;
|
|
|
|
namespace ZB.MOM.WW.ScadaBridge.Host.Actors;
|
|
|
|
/// <summary>
|
|
/// Registers a central cluster singleton with the canonical naming scheme
|
|
/// (<c>{name}-singleton</c> / <c>{name}-proxy</c>), a PoisonPill termination
|
|
/// message, and a PhaseClusterLeave drain task that GracefulStops the manager
|
|
/// so in-flight EF work completes before handover. Extracted from five
|
|
/// copy-pasted ~60-line blocks (review 01 [Low]) whose drift left the two
|
|
/// busiest singletons (notification-outbox, audit-log-ingest) without drain
|
|
/// tasks (review 01 [Medium]).
|
|
/// </summary>
|
|
internal static class CentralSingletonRegistrar
|
|
{
|
|
internal sealed record Handle(IActorRef Manager, IActorRef Proxy);
|
|
|
|
internal static Handle Start(
|
|
ActorSystem system,
|
|
string name,
|
|
Props singletonProps,
|
|
ILogger logger,
|
|
TimeSpan? drainTimeout = null)
|
|
{
|
|
var manager = system.ActorOf(
|
|
ClusterSingletonManager.Props(
|
|
singletonProps,
|
|
PoisonPill.Instance,
|
|
ClusterSingletonManagerSettings.Create(system).WithSingletonName(name)),
|
|
$"{name}-singleton");
|
|
|
|
var timeout = drainTimeout ?? TimeSpan.FromSeconds(10);
|
|
Akka.Actor.CoordinatedShutdown.Get(system).AddTask(
|
|
Akka.Actor.CoordinatedShutdown.PhaseClusterLeave,
|
|
$"drain-{name}-singleton",
|
|
async () =>
|
|
{
|
|
try
|
|
{
|
|
await manager.GracefulStop(timeout);
|
|
}
|
|
catch (Exception ex)
|
|
{
|
|
logger.LogWarning(ex,
|
|
"{Singleton} singleton did not drain within the graceful-stop timeout; "
|
|
+ "falling through to PoisonPill handover", name);
|
|
}
|
|
return Akka.Done.Instance;
|
|
});
|
|
|
|
var proxy = system.ActorOf(
|
|
ClusterSingletonProxy.Props(
|
|
$"/user/{name}-singleton",
|
|
ClusterSingletonProxySettings.Create(system).WithSingletonName(name)),
|
|
$"{name}-proxy");
|
|
|
|
return new Handle(manager, proxy);
|
|
}
|
|
}
|
|
```
|
|
|
|
4. Run — expect PASS.
|
|
5. Commit:
|
|
```
|
|
git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/CentralSingletonRegistrar.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CentralSingletonRegistrarTests.cs
|
|
git commit -m "feat(host): CentralSingletonRegistrar — one registration path with drain task for central singletons"
|
|
```
|
|
|
|
### Task 10: Route all seven central singletons through the registrar (adds the missing outbox/ingest drains)
|
|
|
|
**Classification:** high-risk (actor wiring refactor)
|
|
**Estimated implement time:** ~5 min
|
|
**Parallelizable with:** 11, 13, 15, 16, 17, 19, 22, 23
|
|
**Files:**
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (lines ~395-811: notification-outbox 410-434, audit-log-ingest 451-480, site-call-audit 524-589, audit-log-purge 612-648, site-audit-reconciliation 663-700, kpi-history-recorder 722-757, pending-deployment-purge 776-811)
|
|
- Test: existing suites (regression) + `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs` (one new source assertion)
|
|
|
|
1. Write the failing test — extend `CoordinatedShutdownTests` in its existing source-grep style (the behavioral drain is covered by Task 9; this pins that the two hot singletons now go through the drain path):
|
|
|
|
```csharp
|
|
[Fact]
|
|
public void AllCentralSingletons_RegisterThroughRegistrarWithDrain()
|
|
{
|
|
var hostProjectDir = FindHostProjectDirectory();
|
|
var content = File.ReadAllText(Path.Combine(hostProjectDir!, "Actors", "AkkaHostedService.cs"));
|
|
// Review 01 [Medium]: notification-outbox and audit-log-ingest were the
|
|
// only central singletons WITHOUT a cluster-leave drain task.
|
|
foreach (var name in new[] { "notification-outbox", "audit-log-ingest", "site-call-audit",
|
|
"audit-log-purge", "site-audit-reconciliation",
|
|
"kpi-history-recorder", "pending-deployment-purge" })
|
|
{
|
|
Assert.Contains($"CentralSingletonRegistrar.Start(", content);
|
|
Assert.Contains($"\"{name}\"", content);
|
|
}
|
|
// No hand-rolled manager registrations remain in RegisterCentralActors.
|
|
Assert.DoesNotContain("ClusterSingletonManager.Props(", ExtractRegisterCentralActors(content));
|
|
}
|
|
```
|
|
|
|
(Implement `ExtractRegisterCentralActors` as a substring between `RegisterCentralActors` and the next method, or simply assert `Assert.Equal(0, CountOccurrences(content, "ClusterSingletonManager.Props("))` after checking the site branch separately — the site `deployment-manager`/event-log singletons at lines 906+ use `.WithRole(...)` and stay hand-rolled; scope the assertion to avoid them, e.g. count == 2.)
|
|
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~AllCentralSingletons"` — expect FAIL.
|
|
3. Refactor `RegisterCentralActors`: replace each of the seven manager/drain/proxy blocks with
|
|
|
|
```csharp
|
|
var outbox = CentralSingletonRegistrar.Start(
|
|
_actorSystem!, "notification-outbox",
|
|
Props.Create(() => new ZB.MOM.WW.ScadaBridge.NotificationOutbox.NotificationOutboxActor(
|
|
_serviceProvider, outboxOptions, outboxAuditWriter, outboxLogger)),
|
|
_logger);
|
|
centralCommActor.Tell(new RegisterNotificationOutbox(outbox.Proxy));
|
|
commService?.SetNotificationOutbox(outbox.Proxy);
|
|
```
|
|
|
|
and equivalently for the other six (preserve every existing proxy hand-off: `RegisterAuditIngest`, `grpcServer?.SetAuditIngestActor`, `commService?.SetSiteCallAudit`, `siteCallAuditProxy.Tell(RegisterCentralCommunication…)` → `siteCallAudit.Proxy.Tell(…)`, and the log lines). Actor names and singleton names are unchanged (the registrar reproduces them exactly), so `ActorPathTests` and `RequiredSingletonsHealthCheckTests` must stay green. Preserve the comment describing the drain rationale once at the top of `RegisterCentralActors` (delete the five per-block copies and the "not added here to keep this change minimal" note at 544-549, which this task retires).
|
|
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` (full project — ActorPath/RequiredSingletons/AuditWiring are the regression net) and `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — expect PASS.
|
|
5. Commit:
|
|
```
|
|
git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CoordinatedShutdownTests.cs
|
|
git commit -m "refactor(host): central singletons via CentralSingletonRegistrar — outbox/audit-ingest gain drain tasks"
|
|
```
|
|
|
|
### Task 11: SiteHealthReportAck message + ack semantics in the communication actors
|
|
|
|
**Classification:** high-risk (actor messaging, cross-cluster contract)
|
|
**Estimated implement time:** ~5 min
|
|
**Parallelizable with:** 1, 2, 3, 4, 5, 13, 15, 16, 17, 19, 22, 23
|
|
**Files:**
|
|
- Create: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReportAck.cs`
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteCommunicationActor.cs` (SiteHealthReport handler, lines ~393-399)
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs` (`HandleSiteHealthReport`, lines ~360-374)
|
|
- Test: `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/` (new file `HealthReportAckTests.cs`)
|
|
|
|
1. Write failing TestKit tests (mirror the existing `NotificationSubmit` ack tests in Communication.Tests — grep `NotificationSubmitAck` there for the harness pattern):
|
|
|
|
```csharp
|
|
[Fact]
|
|
public void SiteCommunicationActor_NoCentralClient_RepliesNotAccepted()
|
|
{
|
|
var siteComm = Sys.ActorOf(Props.Create(() => new SiteCommunicationActor(
|
|
"site-a", TestCommunicationOptions(), TestProbe().Ref)));
|
|
// No RegisterCentralClient sent => _centralClient is null.
|
|
siteComm.Tell(SampleReport(seq: 7));
|
|
var ack = ExpectMsg<SiteHealthReportAck>();
|
|
Assert.False(ack.Accepted);
|
|
Assert.Equal(7, ack.SequenceNumber);
|
|
}
|
|
|
|
[Fact]
|
|
public void CentralCommunicationActor_OnReport_ProcessesAndAcks()
|
|
{
|
|
var actor = CreateCentralCommunicationActorWithAggregator(out var aggregator);
|
|
actor.Tell(SampleReport(seq: 3));
|
|
var ack = ExpectMsg<SiteHealthReportAck>();
|
|
Assert.True(ack.Accepted);
|
|
Assert.Equal(3, ack.SequenceNumber);
|
|
// aggregator received the report (NSubstitute: aggregator.Received().ProcessReport(...))
|
|
}
|
|
```
|
|
|
|
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter "FullyQualifiedName~HealthReportAckTests"` — expect FAIL (message type missing).
|
|
3. Implement:
|
|
- Commons (additive message evolution — new type only):
|
|
|
|
```csharp
|
|
namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
|
|
|
|
/// <summary>
|
|
/// Acknowledgement for a <see cref="SiteHealthReport"/> forwarded site→central.
|
|
/// Review 01 [Medium]: the transport was fire-and-forget, so the sender's
|
|
/// interval-counter restore logic could never observe a loss. The ack makes
|
|
/// delivery observable end-to-end (central processed the report).
|
|
/// </summary>
|
|
public sealed record SiteHealthReportAck(
|
|
string SiteId, long SequenceNumber, bool Accepted, string? Error = null);
|
|
```
|
|
|
|
- `SiteCommunicationActor` `Receive<SiteHealthReport>`: mirror the `NotificationSubmit` pattern exactly — when `_centralClient == null`, `Sender.Tell(new SiteHealthReportAck(msg.SiteId, msg.SequenceNumber, false, "Central ClusterClient not registered"))`; otherwise `_centralClient.Tell(new ClusterClient.Send("/user/central-communication", msg), Sender)` (forward the original Sender so the central ack routes straight back to the waiting Ask — replaces the current `Self` sender).
|
|
- `CentralCommunicationActor.HandleSiteHealthReport`: after `ProcessLocally(report)` and the pub-sub replica publish, add `Sender.Tell(new SiteHealthReportAck(report.SiteId, report.SequenceNumber, true));` (guard with `!Sender.IsNobody()` so the peer-replica path, which arrives without an Ask, doesn't dead-letter).
|
|
4. Run — expect PASS. Run the whole Communication.Tests project.
|
|
5. Commit:
|
|
```
|
|
git add src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReportAck.cs src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteCommunicationActor.cs src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/HealthReportAckTests.cs
|
|
git commit -m "feat(comm): ack site health reports end-to-end (SiteHealthReportAck, additive contract)"
|
|
```
|
|
|
|
### Task 12: Acked async health-report transport so counter-restore actually fires
|
|
|
|
**Classification:** high-risk (changes the sender loop's failure semantics)
|
|
**Estimated implement time:** ~5 min
|
|
**Parallelizable with:** 13, 15, 16, 17, 19, 22, 23
|
|
**Files:**
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/IHealthReportTransport.cs`
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/AkkaHealthReportTransport.cs`
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/HealthReportSender.cs` (line 160)
|
|
- Test: `tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/HealthReportSenderTests.cs` (update fakes), `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/` (new `AkkaHealthReportTransportTests.cs`)
|
|
|
|
1. Write failing tests:
|
|
- HealthMonitoring: update the fake transport in `HealthReportSenderTests` to the new `Task SendAsync(SiteHealthReport, CancellationToken)` signature and add/adjust:
|
|
|
|
```csharp
|
|
[Fact]
|
|
public async Task SendAsyncFailure_RestoresIntervalCounters()
|
|
{
|
|
// The restore path (HealthReportSender.cs:158-175) was DEAD CODE against
|
|
// the fire-and-forget production transport. With an acked async transport
|
|
// a timeout/nack surfaces as an exception and the counters roll forward.
|
|
var collector = NewCollectorWithErrors(scriptErrors: 4);
|
|
var transport = new ThrowingTransport(); // SendAsync throws
|
|
var sender = NewSender(collector, transport);
|
|
await RunOneTick(sender);
|
|
Assert.Equal(4, CollectorScriptErrorCount(collector)); // restored, not lost
|
|
}
|
|
```
|
|
|
|
- Host: `AkkaHealthReportTransportTests` — TestKit: place a probe-backed actor at `/user/site-communication` that replies `SiteHealthReportAck(Accepted: true)`; assert `SendAsync` completes. Second test: reply `Accepted: false` → assert `SendAsync` throws `InvalidOperationException`. Third: no actor at the path → assert it throws (Ask timeout; use a short timeout via options or constant).
|
|
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter "FullyQualifiedName~HealthReportSenderTests"` — expect FAIL (compile).
|
|
3. Implement:
|
|
- `IHealthReportTransport`: replace `void Send(SiteHealthReport report);` with `Task SendAsync(SiteHealthReport report, CancellationToken cancellationToken);` (update xmldoc: "must THROW on non-delivery — the sender's counter-restore depends on it").
|
|
- `AkkaHealthReportTransport`:
|
|
|
|
```csharp
|
|
public async Task SendAsync(SiteHealthReport report, CancellationToken cancellationToken)
|
|
{
|
|
var actorSystem = _akkaService.ActorSystem
|
|
?? throw new InvalidOperationException("Actor system not started — health report not sent");
|
|
var siteComm = actorSystem.ActorSelection("/user/site-communication");
|
|
var ack = await siteComm.Ask<SiteHealthReportAck>(report, AckTimeout, cancellationToken);
|
|
if (!ack.Accepted)
|
|
throw new InvalidOperationException($"Health report #{report.SequenceNumber} rejected: {ack.Error}");
|
|
}
|
|
private static readonly TimeSpan AckTimeout = TimeSpan.FromSeconds(10);
|
|
```
|
|
|
|
- `HealthReportSender` line 160: `_transport.Send(reportWithSeq);` → `await _transport.SendAsync(reportWithSeq, stoppingToken).ConfigureAwait(false);` (the surrounding try/catch + restore stays exactly as-is — it now actually fires). Trim the comment at 142-152 to note the transport is now acked.
|
|
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests` and `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~AkkaHealthReportTransportTests"` — PASS. Build the solution (interface change ripples to any other fake).
|
|
5. Commit:
|
|
```
|
|
git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/IHealthReportTransport.cs src/ZB.MOM.WW.ScadaBridge.Host/AkkaHealthReportTransport.cs src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/HealthReportSender.cs tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests tests/ZB.MOM.WW.ScadaBridge.Host.Tests/AkkaHealthReportTransportTests.cs
|
|
git commit -m "fix(health): acked SendAsync transport — report-loss counter restore is live, not dead code"
|
|
```
|
|
|
|
### Task 13: Metrics-staleness signal + status-transition timestamps in the aggregator
|
|
|
|
**Classification:** high-risk (lock-free CAS state machine)
|
|
**Estimated implement time:** ~5 min
|
|
**Parallelizable with:** 1, 2, 3, 4, 5, 11, 15, 16, 17, 19, 22, 23
|
|
**Files:**
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthState.cs`
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthAggregator.cs` (ProcessReport, CheckForOfflineSites)
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/HealthMonitoringOptions.cs` + `HealthMonitoringOptionsValidator.cs`
|
|
- Modify: `docs/requirements/Component-HealthMonitoring.md` (offline definition, lines ~43-46 + option table)
|
|
- Test: `tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/CentralHealthAggregatorTests.cs`, `HealthMonitoringOptionsValidatorTests.cs`
|
|
|
|
1. Write failing tests in `CentralHealthAggregatorTests` (use the existing `FakeTimeProvider`/manual-time pattern already in that file):
|
|
|
|
```csharp
|
|
[Fact]
|
|
public void HeartbeatingSiteWithStaleReports_IsFlaggedMetricsStale_ButStaysOnline()
|
|
{
|
|
// Review 01 [Medium]: a site whose HealthReportSender died keeps
|
|
// heartbeating and previously showed "online with frozen metrics forever".
|
|
var (aggregator, time) = NewAggregator(metricsStaleTimeout: TimeSpan.FromMinutes(2));
|
|
aggregator.ProcessReport(Report("site-a", seq: 1));
|
|
time.Advance(TimeSpan.FromMinutes(3));
|
|
aggregator.MarkHeartbeat("site-a", time.GetUtcNow()); // heartbeats keep flowing
|
|
aggregator.CheckForOfflineSites();
|
|
var state = aggregator.GetSiteState("site-a")!;
|
|
Assert.True(state.IsOnline); // heartbeat-based liveness unchanged
|
|
Assert.True(state.IsMetricsStale); // NEW distinct signal
|
|
}
|
|
|
|
[Fact]
|
|
public void FreshReport_ClearsMetricsStale()
|
|
{
|
|
var (aggregator, time) = NewAggregator(metricsStaleTimeout: TimeSpan.FromMinutes(2));
|
|
aggregator.ProcessReport(Report("site-a", seq: 1));
|
|
time.Advance(TimeSpan.FromMinutes(3));
|
|
aggregator.CheckForOfflineSites();
|
|
aggregator.ProcessReport(Report("site-a", seq: 2));
|
|
Assert.False(aggregator.GetSiteState("site-a")!.IsMetricsStale);
|
|
}
|
|
|
|
[Fact]
|
|
public void OnlineOfflineTransitions_RecordLastStatusChangeAt()
|
|
{
|
|
// Review 01 underdeveloped #7: "when did the site drop" was unanswerable.
|
|
var (aggregator, time) = NewAggregator();
|
|
aggregator.ProcessReport(Report("site-a", seq: 1));
|
|
time.Advance(TimeSpan.FromMinutes(5)); // > OfflineTimeout
|
|
aggregator.CheckForOfflineSites();
|
|
var offlineAt = aggregator.GetSiteState("site-a")!.LastStatusChangeAt;
|
|
Assert.Equal(time.GetUtcNow(), offlineAt);
|
|
aggregator.MarkHeartbeat("site-a", time.GetUtcNow());
|
|
Assert.NotEqual(offlineAt, aggregator.GetSiteState("site-a")!.LastStatusChangeAt);
|
|
}
|
|
```
|
|
|
|
Validator test: `MetricsStaleTimeout` must be positive and ≥ `ReportInterval` (a stale window shorter than one report interval flags every site).
|
|
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter "FullyQualifiedName~CentralHealthAggregatorTests"` — expect FAIL.
|
|
3. Implement:
|
|
- `SiteHealthState`: add `public bool IsMetricsStale { get; init; }` and `public DateTimeOffset? LastStatusChangeAt { get; init; }` (xmldoc each: stale = online but no *report* within MetricsStaleTimeout; LastStatusChangeAt = last online↔offline flip).
|
|
- `HealthMonitoringOptions`: add `public TimeSpan MetricsStaleTimeout { get; set; } = TimeSpan.FromMinutes(2);` — validator rules as above.
|
|
- `CentralHealthAggregator.ProcessReport`: in both the register and update branches set `IsMetricsStale = false`; in the update branch set `LastStatusChangeAt = existing.IsOnline ? existing.LastStatusChangeAt : now`. `MarkHeartbeat`: on the offline→online promotion set `LastStatusChangeAt = now` (leave `IsMetricsStale` untouched — heartbeats say nothing about metrics).
|
|
- `CheckForOfflineSites`: when marking offline, `offline = state with { IsOnline = false, LastStatusChangeAt = now }`. Add a second pass (same loop) for online sites: if `state.LastReportReceivedAt is { } lr && now - lr > _options.MetricsStaleTimeout && !state.IsMetricsStale`, CAS to `state with { IsMetricsStale = true }` and `LogWarning("Site {SiteId} metrics are stale — online (heartbeats) but no report for {Elapsed}s", ...)`. CAS-loss handling identical to the offline swap (lose ⇒ fresh data arrived ⇒ correct to skip).
|
|
- `Component-HealthMonitoring.md` lines ~43-46: replace "offline = no report within 60s" with the two-signal model: **liveness** = no heartbeat/signal within `OfflineTimeout` (60s) → offline; **metrics staleness** = online but no report within `MetricsStaleTimeout` (2m) → "metrics stale" flag. Document `LastStatusChangeAt`. This closes the spec/code disagreement the report flagged.
|
|
4. Run the HealthMonitoring.Tests project — PASS.
|
|
5. Commit:
|
|
```
|
|
git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests docs/requirements/Component-HealthMonitoring.md
|
|
git commit -m "feat(health): metrics-stale signal + status-transition timestamps; spec now matches heartbeat-liveness code"
|
|
```
|
|
|
|
### Task 14: Surface metrics-stale + offline-since on the Health dashboard
|
|
|
|
**Classification:** small
|
|
**Estimated implement time:** ~4 min
|
|
**Parallelizable with:** 15, 16, 17, 19, 22, 23
|
|
**Files:**
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor`
|
|
- Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/` (bUnit — grep for an existing Health page test class first: `grep -rn "Health" tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --include="*.cs" -l`; extend it if present, else create `HealthPageStalenessTests.cs` following the nearest page-test pattern)
|
|
|
|
1. Write the failing bUnit test: render the page with a site state where `IsMetricsStale = true` and assert a `.badge` (existing Bootstrap badge classes used on the page) containing text `Metrics stale` is rendered; second case: `IsOnline = false, LastStatusChangeAt = X` renders `offline since` text containing the formatted timestamp.
|
|
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests --filter "FullyQualifiedName~HealthPageStaleness"` — expect FAIL.
|
|
3. Implement in `Health.razor`, on the site card/row where the online/offline badge already renders:
|
|
|
|
```razor
|
|
@if (state.IsOnline && state.IsMetricsStale)
|
|
{
|
|
<span class="badge bg-warning text-dark ms-1"
|
|
title="Site is heartbeating but has sent no health report for over @StaleTimeoutDisplay — the metrics pipeline may be dead.">
|
|
Metrics stale
|
|
</span>
|
|
}
|
|
@if (!state.IsOnline && state.LastStatusChangeAt is { } changedAt)
|
|
{
|
|
<small class="text-muted ms-1">offline since @changedAt.ToString("u")</small>
|
|
}
|
|
```
|
|
|
|
(Match the page's existing badge markup/classes — clean corporate Bootstrap, no new components.)
|
|
4. Run — PASS.
|
|
5. Commit:
|
|
```
|
|
git add src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests
|
|
git commit -m "feat(ui): health dashboard shows metrics-stale badge and offline-since timestamp"
|
|
```
|
|
|
|
### Task 15: Fix the metrics-port seed in appsettings.Site.json + close the validator gap
|
|
|
|
**Classification:** small
|
|
**Estimated implement time:** ~4 min
|
|
**Parallelizable with:** 1, 2, 3, 4, 5, 11, 13, 16, 17, 19, 22, 23
|
|
**Files:**
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json` (lines 13-17)
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs` (seed loop, lines 117-127)
|
|
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs`
|
|
|
|
1. Write the failing test (mirror the existing seed-vs-GrpcPort test in `StartupValidatorTests`):
|
|
|
|
```csharp
|
|
[Fact]
|
|
public void Validate_SeedNodeTargetingMetricsPort_Fails()
|
|
{
|
|
// Review 01 [Medium]: appsettings.Site.json shipped a seed pointing at the
|
|
// Kestrel HTTP/1.1 metrics listener (8084) — a doomed Akka.Remote
|
|
// association. The validator guarded GrpcPort but not MetricsPort.
|
|
var config = SiteConfigWith(
|
|
metricsPort: 8084,
|
|
seedNodes: new[] { "akka.tcp://scadabridge@localhost:8082", "akka.tcp://scadabridge@localhost:8084" });
|
|
var ex = Assert.Throws<StartupValidationException>(() => StartupValidator.Validate(config));
|
|
Assert.Contains("must not target the metrics port", ex.Message);
|
|
}
|
|
```
|
|
|
|
(Use the existing config-building helper in that test file; match the actual exception type/validation surface used by the current seed-vs-grpc test.)
|
|
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~SeedNodeTargetingMetricsPort"` — expect FAIL.
|
|
3. Implement:
|
|
- `StartupValidator.cs`: inside the existing `foreach (var seed in seedNodes ...)` loop add:
|
|
|
|
```csharp
|
|
p.Require("ScadaBridge:Cluster:SeedNodes",
|
|
_ => SeedNodePort(seed) != metricsPort,
|
|
$"entry '{seed}' must not target the metrics port " +
|
|
$"({metricsPort}); seed nodes must reference Akka remoting ports");
|
|
```
|
|
|
|
- `appsettings.Site.json`: change the second seed `"akka.tcp://scadabridge@localhost:8084"` → `"akka.tcp://scadabridge@localhost:8085"` and add a `"_seedNodes"` comment key: `"Host-0xx: second entry is the FUTURE node-b remoting port (8085) for a two-node localhost site. It must be an Akka remoting endpoint — never this node's GrpcPort (8083) or MetricsPort (8084); StartupValidator rejects both."`
|
|
- Sanity-check the docker per-node site configs don't carry the same defect: `grep -n "SeedNodes" -A3 docker/site-*/appsettings.Site.json` (they seed container-host:8082 pairs — expected clean; fix identically if not).
|
|
4. Run — PASS. Run the full `StartupValidatorTests` class.
|
|
5. Commit:
|
|
```
|
|
git add src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs
|
|
git commit -m "fix(host): dev site seed no longer targets the metrics port; validator rejects seed-vs-MetricsPort"
|
|
```
|
|
|
|
### Task 16: Single-node installs — explicit AllowSingleNodeCluster instead of phantom seeds
|
|
|
|
**Classification:** small
|
|
**Estimated implement time:** ~4 min
|
|
**Parallelizable with:** 1, 2, 3, 4, 5, 11, 13, 15, 17, 19, 22, 23
|
|
**Files:**
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs`
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs` (lines 24-34)
|
|
- Modify: `deploy/wonder-app-vd03/appsettings.Central.json` (Cluster section, lines 9-19)
|
|
- Test: `tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsValidatorTests.cs`
|
|
|
|
1. Write failing tests:
|
|
|
|
```csharp
|
|
[Fact]
|
|
public void SingleSeed_WithoutAcknowledgement_Fails()
|
|
{
|
|
var options = ValidOptions() with-like-mutation: SeedNodes = ["akka.tcp://scadabridge@h:8081"];
|
|
AssertInvalid(options, "AllowSingleNodeCluster");
|
|
}
|
|
|
|
[Fact]
|
|
public void SingleSeed_WithAllowSingleNodeCluster_Passes()
|
|
{
|
|
// Review 01 [Low]: the shipped single-node artifact satisfied the 2-seed
|
|
// rule with a phantom seed the node dials forever. An explicit
|
|
// acknowledgement flag replaces the fiction.
|
|
var options = ValidOptions(); options.SeedNodes = new List<string> { "akka.tcp://scadabridge@h:8081" };
|
|
options.AllowSingleNodeCluster = true;
|
|
AssertValid(options);
|
|
}
|
|
|
|
[Fact]
|
|
public void EmptySeeds_FailsEvenWithFlag() { /* zero seeds always invalid */ }
|
|
```
|
|
|
|
(Adapt to the existing `ClusterOptionsValidatorTests` helper style.)
|
|
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests --filter "FullyQualifiedName~SingleSeed"` — expect FAIL.
|
|
3. Implement:
|
|
- `ClusterOptions`: add `public bool AllowSingleNodeCluster { get; set; }` with xmldoc: "Acknowledges a deliberate single-node deployment: permits exactly one seed node (normally ≥2 are required so either node can start first). Without this flag a single-node install must list a phantom second seed, which the node dials forever — log noise and a defeated validation intent (review 01)."
|
|
- `ClusterOptionsValidator`: replace the `>= 2` rule with:
|
|
|
|
```csharp
|
|
var minSeeds = options.AllowSingleNodeCluster ? 1 : 2;
|
|
builder.RequireThat(options.SeedNodes is not null && options.SeedNodes.Count >= minSeeds,
|
|
options.AllowSingleNodeCluster
|
|
? "ClusterOptions.SeedNodes must contain at least 1 seed node."
|
|
: "ClusterOptions.SeedNodes must contain at least 2 seed nodes "
|
|
+ "(Component-ClusterInfrastructure.md → Node Configuration: both nodes are seed nodes); "
|
|
+ "for a deliberate single-node install set ClusterOptions.AllowSingleNodeCluster = true instead of listing a phantom seed.");
|
|
```
|
|
|
|
- `deploy/wonder-app-vd03/appsettings.Central.json`: remove the `localhost:8091` phantom seed, add `"AllowSingleNodeCluster": true`, and rewrite `_comment_Cluster` to explain the flag ("single-node install acknowledged; add node B's real seed and remove the flag when it exists"). Validate JSON: `python3 -m json.tool deploy/wonder-app-vd03/appsettings.Central.json > /dev/null`.
|
|
4. Run the ClusterInfrastructure.Tests project — PASS.
|
|
5. Commit:
|
|
```
|
|
git add src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests deploy/wonder-app-vd03/appsettings.Central.json
|
|
git commit -m "feat(cluster): AllowSingleNodeCluster flag — wonder-app-vd03 drops its phantom seed"
|
|
```
|
|
|
|
### Task 17: stop_grace_period for all app containers (both topologies)
|
|
|
|
**Classification:** trivial
|
|
**Estimated implement time:** ~3 min
|
|
**Parallelizable with:** everything except 21
|
|
**Files:**
|
|
- Modify: `docker/docker-compose.yml` (every `scadabridge:latest` service)
|
|
- Modify: `docker-env2/docker-compose.yml` (every `scadabridge:latest` service)
|
|
|
|
1. No unit test (compose config). Verification command is the "test": `docker compose -f docker/docker-compose.yml config | grep -c "stop_grace_period"` — expect 0 before, N (= number of app services, 8 in docker/, 4 in docker-env2/) after.
|
|
2. Add to each app service (central-a/b, all site nodes — NOT the traefik service, which has no CoordinatedShutdown):
|
|
|
|
```yaml
|
|
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
|
|
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
|
|
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
|
|
stop_grace_period: 30s
|
|
```
|
|
|
|
3. Validate: `docker compose -f docker/docker-compose.yml config > /dev/null && docker compose -f docker-env2/docker-compose.yml config > /dev/null` (syntax), then the grep from step 1 returns 8 and 4.
|
|
4. Commit:
|
|
```
|
|
git add docker/docker-compose.yml docker-env2/docker-compose.yml
|
|
git commit -m "fix(docker): 30s stop_grace_period so graceful redeploys aren't SIGKILLed mid-CoordinatedShutdown"
|
|
```
|
|
|
|
### Task 18: Evict deleted sites from the health aggregator
|
|
|
|
**Classification:** standard
|
|
**Estimated implement time:** ~5 min
|
|
**Parallelizable with:** 15, 16, 17, 19, 22, 23
|
|
**Files:**
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ICentralHealthAggregator.cs`
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthAggregator.cs`
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs` (the periodic site-address refresh handler — locate with `grep -n "Refresh" src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs`; it reloads sites from the repository every 60s and on admin changes)
|
|
- Test: `tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/CentralHealthAggregatorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/`
|
|
|
|
1. Write the failing aggregator test:
|
|
|
|
```csharp
|
|
[Fact]
|
|
public void PruneUnknownSites_RemovesDeletedSites_KeepsKnownAndCentral()
|
|
{
|
|
// Review 01 [Low]: _siteStates only grew — a deleted site remained a
|
|
// permanently-offline dashboard tile (and a KPI sample source) forever.
|
|
var (aggregator, time) = NewAggregator();
|
|
aggregator.ProcessReport(Report("site-a", 1));
|
|
aggregator.ProcessReport(Report("site-b", 1));
|
|
aggregator.ProcessReport(Report(CentralHealthReportLoop.CentralSiteId, 1));
|
|
|
|
aggregator.PruneUnknownSites(new[] { "site-a" });
|
|
|
|
Assert.NotNull(aggregator.GetSiteState("site-a"));
|
|
Assert.Null(aggregator.GetSiteState("site-b"));
|
|
Assert.NotNull(aggregator.GetSiteState(CentralHealthReportLoop.CentralSiteId)); // synthetic id always kept
|
|
}
|
|
```
|
|
|
|
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter "FullyQualifiedName~PruneUnknownSites"` — expect FAIL.
|
|
3. Implement:
|
|
- Interface + aggregator:
|
|
|
|
```csharp
|
|
/// <summary>Removes tracked sites not in <paramref name="knownSiteIds"/> (deleted from configuration). The synthetic central id is always retained. Self-healing: called from the CentralCommunicationActor's periodic site refresh, so a deleted site disappears within one refresh interval without needing a deletion event.</summary>
|
|
public void PruneUnknownSites(IReadOnlyCollection<string> knownSiteIds)
|
|
{
|
|
var known = new HashSet<string>(knownSiteIds, StringComparer.Ordinal);
|
|
foreach (var siteId in _siteStates.Keys)
|
|
{
|
|
if (siteId == CentralHealthReportLoop.CentralSiteId || known.Contains(siteId))
|
|
continue;
|
|
if (_siteStates.TryRemove(siteId, out _))
|
|
_logger.LogInformation("Site {SiteId} evicted from health aggregator (no longer configured)", siteId);
|
|
}
|
|
}
|
|
```
|
|
|
|
- In `CentralCommunicationActor`'s site-refresh handler, after the fresh site list is loaded: `_serviceProvider.GetService<ICentralHealthAggregator>()?.PruneUnknownSites(sites.Select(s => s.SiteIdentifier).ToList());` (match the identifier the aggregator is keyed on — it's the `SiteId` string sites report with; confirm by checking what `HandleSiteHealthReport` receives, and use the same property the refresh handler already uses for client-keying).
|
|
- Add a Communication.Tests TestKit test: trigger the refresh message with a repo/enumerator fake returning `["site-a"]` and an NSubstitute aggregator; assert `aggregator.Received().PruneUnknownSites(...)`.
|
|
4. Run both test projects — PASS.
|
|
5. Commit:
|
|
```
|
|
git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring src/ZB.MOM.WW.ScadaBridge.Communication/Actors/CentralCommunicationActor.cs tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests tests/ZB.MOM.WW.ScadaBridge.Communication.Tests
|
|
git commit -m "fix(health): evict deleted sites from the aggregator on the periodic site refresh"
|
|
```
|
|
|
|
### Task 19: Rate-limit dead-letter warnings
|
|
|
|
**Classification:** high-risk (actor change — timers)
|
|
**Estimated implement time:** ~5 min
|
|
**Parallelizable with:** 1, 2, 3, 4, 5, 11, 13, 15, 16, 17, 22, 23
|
|
**Files:**
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/DeadLetterMonitorActor.cs`
|
|
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/DeadLetterMonitorTests.cs`
|
|
|
|
1. Write the failing test (extend `DeadLetterMonitorTests`; use an NSubstitute `ILogger<DeadLetterMonitorActor>` and count `Log` invocations at `LogLevel.Warning`):
|
|
|
|
```csharp
|
|
[Fact]
|
|
public void DeadLetterStorm_LogsAtMostWindowLimit_ButCountsAll()
|
|
{
|
|
// Review 01 [Low]: a failover/undeploy storm produced one Warning per
|
|
// dead letter — unbounded log flood. Health metric counting is untouched.
|
|
var logger = Substitute.For<ILogger<DeadLetterMonitorActor>>();
|
|
var monitor = Sys.ActorOf(Props.Create(() => new DeadLetterMonitorActor(logger, null)));
|
|
for (var i = 0; i < 50; i++)
|
|
monitor.Tell(new DeadLetter($"m{i}", TestActor, TestActor));
|
|
|
|
var count = monitor.Ask<DeadLetterCountResponse>(GetDeadLetterCount.Instance).Result;
|
|
Assert.Equal(50, count.Count); // every dead letter still counted
|
|
var warnings = logger.ReceivedCalls().Count(c =>
|
|
c.GetMethodInfo().Name == "Log" && (LogLevel)c.GetArguments()[0]! == LogLevel.Warning);
|
|
Assert.True(warnings <= DeadLetterMonitorActor.MaxWarningsPerWindow,
|
|
$"expected <= {DeadLetterMonitorActor.MaxWarningsPerWindow} warnings, got {warnings}");
|
|
}
|
|
```
|
|
|
|
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~DeadLetterStorm"` — expect FAIL (50 warnings).
|
|
3. Implement: make the actor `IWithTimers`; add `public const int MaxWarningsPerWindow = 10;` and `internal static readonly TimeSpan WarningWindow = TimeSpan.FromMinutes(1);` In the `DeadLetter` handler:
|
|
|
|
```csharp
|
|
_deadLetterCount++;
|
|
_healthCollector?.IncrementDeadLetter();
|
|
if (_warningsThisWindow < MaxWarningsPerWindow)
|
|
{
|
|
_warningsThisWindow++;
|
|
logger.LogWarning("Dead letter: {MessageType} from {Sender} to {Recipient}",
|
|
dl.Message.GetType().Name, dl.Sender, dl.Recipient);
|
|
if (_warningsThisWindow == MaxWarningsPerWindow)
|
|
logger.LogWarning("Dead-letter warning limit ({Limit}) reached — suppressing per-letter warnings for {Window}",
|
|
MaxWarningsPerWindow, WarningWindow);
|
|
}
|
|
else
|
|
{
|
|
_suppressedThisWindow++;
|
|
}
|
|
```
|
|
|
|
Plus a private `WindowTick` timer message started in `PreStart` (`Timers.StartPeriodicTimer("dl-window", WindowTick.Instance, WarningWindow)`) whose handler logs one summary if `_suppressedThisWindow > 0` (`LogWarning("Suppressed {Count} dead letters in the last {Window}", ...)`) and resets both window counters. Update the class xmldoc.
|
|
4. Run — PASS (the summary tick fires only after 1 min, so the test sees ≤ 11 warnings; assert against `MaxWarningsPerWindow + 1` if the limit-reached notice counts).
|
|
5. Commit:
|
|
```
|
|
git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/DeadLetterMonitorActor.cs tests/ZB.MOM.WW.ScadaBridge.Host.Tests/DeadLetterMonitorTests.cs
|
|
git commit -m "fix(host): rate-limit dead-letter warnings (10/min + suppression summary); metric counting unchanged"
|
|
```
|
|
|
|
### Task 20: Down-if-alone recovery story — exit on unexpected ActorSystem termination + service restart actions
|
|
|
|
**Classification:** high-risk (process lifecycle)
|
|
**Estimated implement time:** ~5 min
|
|
**Parallelizable with:** 15, 16, 22, 23
|
|
**Files:**
|
|
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs` (ctor DI + StartAsync/GetOrCreateActorSystem + StopAsync flag)
|
|
- Modify: `deploy/wonder-app-vd03/install.ps1` (service recovery actions)
|
|
- Modify: `docs/requirements/Component-ClusterInfrastructure.md` (new "Down-if-alone recovery" subsection near the split-brain section, ~lines 110-119)
|
|
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/` (new `UnexpectedTerminationTests.cs`)
|
|
|
|
Context: with SBR enabled (Task 1), `keep-oldest down-if-alone = on` means a node that finds itself alone-and-partitioned downs ITSELF, and `run-coordinated-shutdown-when-down = on` (now live) terminates its ActorSystem. Today the Host process would keep running with a dead actor system — serving nothing, restarted by nobody. The recovery contract: unexpected ActorSystem termination ⇒ stop the host process ⇒ the supervisor (docker `restart: unless-stopped` / Windows service recovery) restarts it ⇒ it rejoins as a fresh incarnation.
|
|
|
|
1. Write the failing test:
|
|
|
|
```csharp
|
|
[Fact]
|
|
public async Task ActorSystemTerminatedOutsideStopAsync_StopsHostApplication()
|
|
{
|
|
// Down-if-alone self-down (SBR) terminates the ActorSystem via
|
|
// run-coordinated-shutdown-when-down. The process must exit so the
|
|
// service supervisor restarts it — otherwise the node idles forever
|
|
// (review 01 underdeveloped #2).
|
|
var lifetime = Substitute.For<IHostApplicationLifetime>();
|
|
var service = CreateAkkaHostedService(lifetime); // existing Host.Tests construction pattern
|
|
await service.StartAsync(CancellationToken.None);
|
|
var system = service.ActorSystem!;
|
|
|
|
await system.Terminate(); // simulated self-down, NOT StopAsync
|
|
|
|
await Task.Delay(500);
|
|
lifetime.Received(1).StopApplication();
|
|
}
|
|
|
|
[Fact]
|
|
public async Task StopAsync_DoesNotTriggerStopApplication()
|
|
{
|
|
var lifetime = Substitute.For<IHostApplicationLifetime>();
|
|
var service = CreateAkkaHostedService(lifetime);
|
|
await service.StartAsync(CancellationToken.None);
|
|
await service.StopAsync(CancellationToken.None);
|
|
lifetime.DidNotReceive().StopApplication();
|
|
}
|
|
```
|
|
|
|
(Reuse whatever construction pattern `AkkaHostedServiceAuditWiringTests`/`HostStartupTests` use for building the service with a test DI container; add `IHostApplicationLifetime` as an OPTIONAL ctor parameter — nullable, resolved via `GetService` — so every existing construction site keeps compiling.)
|
|
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~UnexpectedTermination"` — expect FAIL.
|
|
3. Implement:
|
|
- `AkkaHostedService`: add `private volatile bool _stopRequested;` set to `true` first thing in `StopAsync`. After the system is created and published (`_actorSystem = system;` in `GetOrCreateActorSystem`), register once:
|
|
|
|
```csharp
|
|
system.WhenTerminated.ContinueWith(_ =>
|
|
{
|
|
if (_stopRequested) return;
|
|
_logger.LogCritical(
|
|
"ActorSystem terminated outside host shutdown (SBR down / run-coordinated-shutdown-when-down). "
|
|
+ "Stopping the host process so the service supervisor restarts this node as a fresh incarnation.");
|
|
_appLifetime?.StopApplication();
|
|
}, TaskContinuationOptions.ExecuteSynchronously);
|
|
```
|
|
|
|
- `install.ps1`: after the `sc.exe create`/`New-Service` call, add recovery actions (find the actual service name in the script first):
|
|
|
|
```powershell
|
|
# Restart-on-failure: required for the SBR down-if-alone recovery contract —
|
|
# a self-downed node exits the process and MUST be restarted externally.
|
|
& sc.exe failure $ServiceName reset= 86400 actions= restart/5000/restart/30000/restart/60000
|
|
& sc.exe failureflag $ServiceName 1
|
|
```
|
|
|
|
- `Component-ClusterInfrastructure.md`: add a "Down-if-alone recovery" subsection: self-down ⇒ CoordinatedShutdown ⇒ process exit ⇒ supervisor restart (docker `restart: unless-stopped`; Windows service recovery actions) ⇒ clean rejoin as a new incarnation; note the drill in Task 21's script.
|
|
4. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` — PASS (full project; the new ctor param must not break existing tests).
|
|
5. Commit:
|
|
```
|
|
git add src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs deploy/wonder-app-vd03/install.ps1 docs/requirements/Component-ClusterInfrastructure.md tests/ZB.MOM.WW.ScadaBridge.Host.Tests/UnexpectedTerminationTests.cs
|
|
git commit -m "feat(host): exit process on unexpected ActorSystem termination — completes the down-if-alone recovery loop"
|
|
```
|
|
|
|
### Task 21: Docker failover drill script (the manual/CI kill test against the real topology)
|
|
|
|
**Classification:** small
|
|
**Estimated implement time:** ~5 min
|
|
**Parallelizable with:** 22, 23
|
|
**Files:**
|
|
- Create: `docker/failover-drill.sh`
|
|
- Modify: `docker/README.md` (new "Failover drill" section), `deployments/docker-cluster.md` (cross-reference)
|
|
|
|
1. No unit test — the script IS the test. Create:
|
|
|
|
```bash
|
|
#!/usr/bin/env bash
|
|
# Failover drill against the running docker cluster (bash docker/deploy.sh first).
|
|
# Kills the ACTIVE central node (docker kill = SIGKILL, the hard-crash path that
|
|
# review 01 found had NO automatic recovery before the SBR downing provider was
|
|
# enabled) and measures how long Traefik takes to route to the survivor.
|
|
set -euo pipefail
|
|
|
|
TRAEFIK_URL="${TRAEFIK_URL:-http://localhost:9000}"
|
|
TIMEOUT_S="${TIMEOUT_S:-90}"
|
|
|
|
active_container() {
|
|
if curl -sf -o /dev/null "http://localhost:9001/health/active"; then echo scadabridge-central-a
|
|
elif curl -sf -o /dev/null "http://localhost:9002/health/active"; then echo scadabridge-central-b
|
|
else echo "ERROR: no active central node found" >&2; exit 1; fi
|
|
}
|
|
|
|
VICTIM=$(active_container)
|
|
echo "Active central node: ${VICTIM} — killing it (SIGKILL, crash path)"
|
|
docker kill "${VICTIM}" > /dev/null
|
|
START=$(date +%s)
|
|
|
|
echo "Waiting for the survivor to become active through Traefik (${TRAEFIK_URL}/health/active)..."
|
|
while true; do
|
|
ELAPSED=$(( $(date +%s) - START ))
|
|
if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then
|
|
echo "PASS: failover complete in ${ELAPSED}s (design budget ~25s + Traefik health interval)"
|
|
break
|
|
fi
|
|
if (( ELAPSED > TIMEOUT_S )); then
|
|
echo "FAIL: no active central node after ${ELAPSED}s — SBR/singleton handover did not recover" >&2
|
|
docker start "${VICTIM}" > /dev/null
|
|
exit 1
|
|
fi
|
|
sleep 1
|
|
done
|
|
|
|
SURVIVOR=$([ "${VICTIM}" = scadabridge-central-a ] && echo scadabridge-central-b || echo scadabridge-central-a)
|
|
echo "Survivor singleton evidence (last 20 matching log lines from ${SURVIVOR}):"
|
|
docker logs "${SURVIVOR}" 2>&1 | grep -Ei "singleton|oldest|Downing|Removed" | tail -20 || true
|
|
|
|
echo "Restarting ${VICTIM} and waiting for it to rejoin..."
|
|
docker start "${VICTIM}" > /dev/null
|
|
sleep 5
|
|
echo "Drill complete. Verify on the Health dashboard that both nodes show Up and the survivor is Primary."
|
|
```
|
|
|
|
2. `chmod +x docker/failover-drill.sh`. Verify statically: `bash -n docker/failover-drill.sh` (syntax). If a cluster is running (`bash docker/deploy.sh`), run the drill end-to-end and record the observed failover time in `docker/README.md`; if not available in this session, mark the README section "run after next deploy" — the script is still committed.
|
|
3. Document in `docker/README.md`: what it does, the ~25s + Traefik 5s health-interval expectation, and that it exercises S1 (SBR downing), S3 (single-active through Traefik) and Task 20 (restart/rejoin). Cross-reference from `deployments/docker-cluster.md`. Note for other plans: plan 02's duplicate-delivery checks can extend this script.
|
|
4. Commit:
|
|
```
|
|
git add docker/failover-drill.sh docker/README.md deployments/docker-cluster.md
|
|
git commit -m "test(docker): failover drill script — SIGKILL the active central node, assert Traefik recovery"
|
|
```
|
|
|
|
### Task 22: Reconcile REQ-HOST-6 (Akka.Hosting) with the hand-rolled bootstrap
|
|
|
|
**Classification:** small
|
|
**Estimated implement time:** ~4 min
|
|
**Parallelizable with:** 15, 16, 17, 19, 21, 23
|
|
**Files:**
|
|
- Modify: `docs/requirements/Component-Host.md` (REQ-HOST-6, ~lines 106-114)
|
|
- Modify (conditional): `src/ZB.MOM.WW.ScadaBridge.Host/ZB.MOM.WW.ScadaBridge.Host.csproj` (lines 15-18) + `Directory.Packages.props`
|
|
|
|
Decision: keep the hand-rolled HOCON (it is hardened — `QuoteHocon`/`DurationHocon`, validated options — and now carries the downing provider explicitly per Task 1); amend the spec instead of a risky bootstrap migration. The report accepts either resolution.
|
|
|
|
1. Verify actual package usage: `grep -rn "using Akka.Hosting\|using Akka.Cluster.Hosting\|using Akka.Remote.Hosting\|AkkaConfigurationBuilder\|\.WithClustering\|\.WithRemoting" src/ --include="*.cs"` — expect zero hits (review found them referenced-but-unused).
|
|
2. If zero hits: remove `<PackageReference Include="Akka.Cluster.Hosting" />`, `<PackageReference Include="Akka.Hosting" />`, `<PackageReference Include="Akka.Remote.Hosting" />` from the Host csproj (keep `Akka.Cluster.Tools`), remove the matching `<PackageVersion>` lines from `Directory.Packages.props` **only if no other project references them** (`grep -rn "Akka.Hosting\|Akka.Cluster.Hosting\|Akka.Remote.Hosting" src/*/[A-Z]*.csproj tests/*/[A-Z]*.csproj`), and check whether the csproj's OpenTelemetry.Api transitive-override comment (lines ~28-32) still applies — if it referenced Akka.Hosting's pin, delete the override too.
|
|
3. Amend `Component-Host.md` REQ-HOST-6: replace "bootstrap via Akka.Hosting" with: bootstrap is a hand-assembled, injection-safe HOCON document (`AkkaHostedService.BuildHocon`) + `ActorSystem.Create`; the downing provider (`Akka.Cluster.SBR.SplitBrainResolverProvider`) is set explicitly — Akka's default is NoDowning and forgetting this line silently disables failover (this exact omission was review 01's Critical finding, caught because tests now assert the parsed key and a two-node kill test asserts the behavior). Note the deliberate rejection of Akka.Cluster.Hosting typed options with a one-line rationale (single hardened code path already in production shape; migration tracked as possible future work).
|
|
4. Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — expect success. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~HoconBuilderTests"` — green.
|
|
5. Commit:
|
|
```
|
|
git add docs/requirements/Component-Host.md src/ZB.MOM.WW.ScadaBridge.Host/ZB.MOM.WW.ScadaBridge.Host.csproj Directory.Packages.props
|
|
git commit -m "docs(host): REQ-HOST-6 documents the hand-rolled HOCON bootstrap; drop unused Akka.Hosting packages"
|
|
```
|
|
|
|
### Task 23: Docs & deploy-config cleanup batch (Traefik doc, NodeName overlays, TLS roadmap note)
|
|
|
|
**Classification:** trivial
|
|
**Estimated implement time:** ~4 min
|
|
**Parallelizable with:** 15, 16, 17, 19, 21, 22
|
|
**Files:**
|
|
- Modify: `docs/requirements/Component-TraefikProxy.md` (lines ~115, ~121, TLS section ~131-138)
|
|
- Modify: `deploy/wonder-app-vd03/appsettings.Central.json` (Node section), `deploy/wonder-app-vd03/appsettings.Site.json` (Node section, if the file exists — check `ls deploy/wonder-app-vd03/`)
|
|
|
|
1. `Component-TraefikProxy.md`:
|
|
- Line ~121: replace the stale `src/ZB.MOM.WW.ScadaBridge.Host/Health/ActiveNodeHealthCheck.cs` location with `src/ZB.MOM.WW.ScadaBridge.Host/Health/OldestNodeActiveHealthCheck.cs` (the Host-local oldest-member check from Task 7) and note `/health/active` also requires the database check.
|
|
- Line ~115: update the `/health/ready` description from "database + Akka cluster" to "database + akka-cluster + required-singletons" (matches Program.cs and Component-Host REQ-HOST-4a).
|
|
- TLS section (~131-138): append an explicit roadmap paragraph — "**Production TLS profile: not yet implemented.** Traefik terminates plain HTTP, Akka remoting is unencrypted TCP, and site gRPC is `http://`. Acceptable for the lab topology only; a production deployment requires TLS at all three layers and a secured Traefik dashboard. Tracked as an open roadmap item (arch review 01, underdeveloped area 3)." — this makes the accepted gap spec-visible without pretending to fix it.
|
|
2. `deploy/wonder-app-vd03/appsettings.Central.json`: add `"NodeName": "central-a"` to the `Node` section (per the base config's documented convention — without it every audit row from the one real deployment gets a NULL `SourceNode`, undermining `IX_AuditLog_Node_Occurred` and the per-node stuck KPIs). Same for the Site overlay if present (`"NodeName": "node-a"`). Validate: `python3 -m json.tool deploy/wonder-app-vd03/appsettings.Central.json > /dev/null`.
|
|
3. Cross-check no other stale `ActiveNodeHealthCheck` doc references remain: `grep -rn "ActiveNodeHealthCheck" docs/ README.md` — fix any found the same way.
|
|
4. Commit:
|
|
```
|
|
git add docs/requirements/Component-TraefikProxy.md deploy/wonder-app-vd03/
|
|
git commit -m "docs(cleanup): sync Traefik doc with shared health checks, add TLS roadmap note, NodeName in deploy overlays"
|
|
```
|
|
|
|
---
|
|
|
|
## Dependencies on other plans
|
|
|
|
- **Plan 02 (Communication & S&F) depends on THIS plan** for two facilities — design them exactly as specified here so the seam holds:
|
|
- The canonical site-side "active node / singleton host" check: after Task 6, `IClusterNodeProvider.SelfIsPrimary` (site DI registration at `SiteServiceRegistration.cs:101-107`) means "oldest Up member in the site role = the node hosting the DeploymentManager singleton". Plan 02's standby S&F delivery gating must consume this (directly or via a registered delegate mirroring `SiteEventLogActiveNodeCheck`), not reinvent a leader check.
|
|
- The two-node failover rig: `TwoNodeClusterFixture` (Task 3) is public and production-HOCON-based; plan 02's ClusterClient-recreate and S&F non-duplication tests should build on it, and the docker drill script (Task 21) is the shared live-cluster harness.
|
|
- **Plan 07 (UI/Management/Security) owns** the `DisableLogin: true` deploy artifact (C2) and the deploy-artifact LDAP plaintext posture (C3) — this plan touches `deploy/wonder-app-vd03/appsettings.Central.json` in Tasks 16 and 23 (Cluster + Node sections only); coordinate merges but do not touch the `Security` section here.
|
|
- No other plan blocks this one.
|
|
|
|
## Execution order
|
|
|
|
**P0 (do first, in order):** Task 1 (the Critical one-liner) → Task 3 (fixture) → Task 4 (behavioral proof). Ship these three even if nothing else lands — they convert "failover does not work" into "failover works and is regression-tested".
|
|
|
|
**Critical path:** 1 → 3 → 4 → 8 (needs 6), with 5 → 6 → 7 as the second spine (active-node unification). Task 2 must precede 9/10 (drain budget before drain refactor); 11 → 12 (message before transport); 13 → 14 and 13 → 18 (state shape before consumers). Task 20 touches `AkkaHostedService.cs` and should follow 10 to avoid merge churn; Task 21 follows 1 (drill is meaningless before SBR) and ideally 17.
|
|
|
|
**Freely parallel at any point:** 15, 16, 17, 19, 22, 23.
|