Compare commits
30 Commits
6dd7858619
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
| eedbf13712 | |||
| 035c5feeb8 | |||
| b02d7a92bb | |||
| 75f267c4c9 | |||
| 7dcd93c326 | |||
| 591eaac881 | |||
| 6ab50fed1f | |||
| a58d79b793 | |||
| bf75e265ab | |||
| 8698d1804b | |||
| 344bdc4da5 | |||
| 6a4fa220de | |||
| a61e041e58 | |||
| c81a6615ba | |||
| 280e57896b | |||
| af1e0a86b2 | |||
| c22b06deba | |||
| d95292c95d | |||
| 41013dcab3 | |||
| 92997b40dc | |||
| 5fe7135e2c | |||
| 61fc88c8a8 | |||
| d0110cf406 | |||
| 80668a07bd | |||
| 6721c8a224 | |||
| 3dbd6cd57c | |||
| 92015da4e0 | |||
| 1911de2c3c | |||
| b5e7bc8b3c | |||
| 4d93d2580b |
@@ -4,7 +4,7 @@ Health-check libraries for the **ZB.MOM.WW SCADA family** (OtOpcUa, MxAccessGate
|
||||
|
||||
The library normalizes the three-tier health endpoint convention (`/health/ready`, `/health/active`, `/healthz`) and provides reusable probe implementations so the three sister projects share a common surface without duplicating probe logic.
|
||||
|
||||
**Built at 0.1.0. NOT yet adopted by the three apps.** Adoption is tracked in `~/Desktop/scadaproj/components/health/GAPS.md`.
|
||||
**Built at 0.3.0 and published to the Gitea NuGet feed. Adopted by all four apps** — adoption is tracked in `~/Desktop/scadaproj/components/health/GAPS.md`.
|
||||
|
||||
---
|
||||
|
||||
@@ -13,7 +13,7 @@ The library normalizes the three-tier health endpoint convention (`/health/ready
|
||||
| Package | Responsibilities | Key Dependencies |
|
||||
|---|---|---|
|
||||
| `ZB.MOM.WW.Health` | Core tier convention, `MapZbHealth` extension, canonical JSON writer (`ZbHealthWriter`), `IActiveNodeGate` seam, `GrpcDependencyHealthCheck` reachability probe, tier-tag constants (`ZbHealthTags`). No Akka or EF dependency. | `Microsoft.AspNetCore.App` (framework ref), `Grpc.Net.Client` |
|
||||
| `ZB.MOM.WW.Health.Akka` | `AkkaClusterHealthCheck` with a configurable `AkkaClusterStatusPolicy` (presets: `Default` / `OtOpcUaCompat`), `ActiveNodeHealthCheck` with an optional role filter, and `AkkaActiveNodeGate` that backs `IActiveNodeGate` from cluster member state. | `ZB.MOM.WW.Health`, `Akka.Cluster` |
|
||||
| `ZB.MOM.WW.Health.Akka` | `AkkaClusterHealthCheck` with a configurable `AkkaClusterStatusPolicy` (presets: `Default` / `OtOpcUaCompat`); `ClusterActiveNode` — the family-wide **oldest-`Up`-member** rule, optionally role-scoped; `ActiveNodeHealthCheck` over it (role-preference list, `activeRole`/`activeNode` in the result `data`); and `AkkaActiveNodeGate`, which backs `IActiveNodeGate` from the same rule. | `ZB.MOM.WW.Health`, `Akka.Cluster` |
|
||||
| `ZB.MOM.WW.Health.EntityFrameworkCore` | `DatabaseHealthCheck<TContext>` — resolves `IDbContextFactory<TContext>` when registered, else a scoped `TContext`; pool-safe. Default probe: `CanConnectAsync`. Optional `ProbeQuery` delegate for query-based validation. | `ZB.MOM.WW.Health`, `Microsoft.EntityFrameworkCore` |
|
||||
|
||||
---
|
||||
@@ -46,14 +46,19 @@ dotnet test ZB.MOM.WW.Health.slnx
|
||||
dotnet pack ZB.MOM.WW.Health.slnx -c Release -o ./artifacts
|
||||
```
|
||||
|
||||
All three test assemblies run offline:
|
||||
All three test assemblies run offline. The Akka suite forms real in-process clusters on loopback: a
|
||||
single-node one for the cluster-data payload, and a **two-node one built so the oldest member is not
|
||||
the lowest-addressed one** — the only condition under which age- and leader-ordering disagree, and so
|
||||
the only fixture that can catch a leader-based active-node regression. It binds fixed ports
|
||||
19540/19541 (distinct from OtOpcUa's equivalent fixture) and carries a sanity check that the
|
||||
divergence actually occurred, so the assertions cannot pass for the wrong reason:
|
||||
|
||||
| Assembly | Tests |
|
||||
|---|---|
|
||||
| `ZB.MOM.WW.Health.Tests` | 20 |
|
||||
| `ZB.MOM.WW.Health.Akka.Tests` | 32 |
|
||||
| `ZB.MOM.WW.Health.Tests` | 25 |
|
||||
| `ZB.MOM.WW.Health.Akka.Tests` | 45 |
|
||||
| `ZB.MOM.WW.Health.EntityFrameworkCore.Tests` | 6 |
|
||||
| **Total** | **58** |
|
||||
| **Total** | **76** |
|
||||
|
||||
`GeneratePackageOnBuild` is off — pack explicitly with the command above.
|
||||
|
||||
@@ -61,7 +66,7 @@ All three test assemblies run offline:
|
||||
|
||||
## Status
|
||||
|
||||
Built at **0.1.0** and published to the Gitea NuGet feed. **Not yet adopted** by the three apps — adoption is tracked in the component backlog:
|
||||
Built at **0.3.0** and published to the Gitea NuGet feed. **Adopted** by all four apps — adoption is tracked in the component backlog:
|
||||
|
||||
- `~/Desktop/scadaproj/components/health/GAPS.md` — adoption order, effort, and risk
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<LangVersion>latest</LangVersion>
|
||||
<Version>0.1.0</Version>
|
||||
<Version>0.3.0</Version>
|
||||
<ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
|
||||
<!-- Emit XML docs so the public API summaries ship inside the packed nupkgs (IntelliSense for
|
||||
consumers). CS1591 (missing doc on a public member) is suppressed so undocumented test /
|
||||
|
||||
@@ -30,7 +30,19 @@ MxAccessGateway consumes the core package only — it has no Akka cluster and no
|
||||
|
||||
## Versioning
|
||||
|
||||
All three packages are versioned **lockstep** from `Directory.Build.props`. The current release is **0.1.0**. A single version bump in `Directory.Build.props` bumps all three packages simultaneously — consumers should reference the same version for all ZB.MOM.WW.Health packages.
|
||||
All three packages are versioned **lockstep** from `Directory.Build.props`. The current release is **0.2.0**. A single version bump in `Directory.Build.props` bumps all three packages simultaneously — consumers should reference the same version for all ZB.MOM.WW.Health packages.
|
||||
|
||||
### 0.2.0 — per-entry `data` (additive, non-breaking)
|
||||
|
||||
- `ZbHealthWriter` emits an optional `"data": { … }` object per entry, sourced from the check's
|
||||
`HealthCheckResult.Data`, **only when non-empty** — a check that publishes no data produces a
|
||||
byte-identical body to 0.1.0, so every existing consumer is unaffected. Data keys are written
|
||||
verbatim (the camelCase policy applies to the envelope, not to dictionary keys).
|
||||
- `AkkaClusterHealthCheck` populates that dictionary with this node's cluster view: `leader`
|
||||
(omitted while unknown), `selfAddress`, `selfRoles`, `memberCount`, `unreachableCount`. The
|
||||
startup-safety paths (no `ActorSystem`, cluster not yet accessible) stay description-only.
|
||||
- Consumers gain the leader field on a package bump alone — no application code change, provided the
|
||||
app registers the shared `AkkaClusterHealthCheck`.
|
||||
|
||||
---
|
||||
|
||||
@@ -42,14 +54,16 @@ dotnet build ZB.MOM.WW.Health.slnx
|
||||
dotnet test ZB.MOM.WW.Health.slnx
|
||||
```
|
||||
|
||||
All three test assemblies run with `dotnet test` and require no external dependencies (no running Akka cluster, no database):
|
||||
All three test assemblies run with `dotnet test` and require no external dependencies — no database and no
|
||||
external cluster (the cluster-data tests form a single-node in-process Akka cluster on a loopback port
|
||||
via `Akka.TestKit.Xunit2`):
|
||||
|
||||
| Assembly | Tests |
|
||||
|---|---|
|
||||
| `ZB.MOM.WW.Health.Tests` | 20 |
|
||||
| `ZB.MOM.WW.Health.Akka.Tests` | 32 |
|
||||
| `ZB.MOM.WW.Health.Tests` | 25 |
|
||||
| `ZB.MOM.WW.Health.Akka.Tests` | 39 |
|
||||
| `ZB.MOM.WW.Health.EntityFrameworkCore.Tests` | 6 |
|
||||
| **Total** | **58** |
|
||||
| **Total** | **70** |
|
||||
|
||||
---
|
||||
|
||||
@@ -62,9 +76,9 @@ dotnet pack ZB.MOM.WW.Health.slnx -c Release -o ./artifacts
|
||||
Produces three `.nupkg` files in `artifacts/`:
|
||||
|
||||
```
|
||||
ZB.MOM.WW.Health.0.1.0.nupkg
|
||||
ZB.MOM.WW.Health.Akka.0.1.0.nupkg
|
||||
ZB.MOM.WW.Health.EntityFrameworkCore.0.1.0.nupkg
|
||||
ZB.MOM.WW.Health.0.2.0.nupkg
|
||||
ZB.MOM.WW.Health.Akka.0.2.0.nupkg
|
||||
ZB.MOM.WW.Health.EntityFrameworkCore.0.2.0.nupkg
|
||||
```
|
||||
|
||||
`GeneratePackageOnBuild` is off — pack explicitly as above.
|
||||
@@ -73,7 +87,7 @@ ZB.MOM.WW.Health.EntityFrameworkCore.0.1.0.nupkg
|
||||
|
||||
## Status
|
||||
|
||||
**Built at 0.1.0. NOT yet adopted by the three apps.** Adoption is tracked in the component backlog:
|
||||
**Built at 0.2.0 and published to the Gitea NuGet feed. Adopted by all four apps** (OtOpcUa, MxAccessGateway, ScadaBridge, HistorianGateway). Adoption is tracked in the component backlog:
|
||||
|
||||
- `~/Desktop/scadaproj/components/health/GAPS.md`
|
||||
|
||||
|
||||
@@ -9,89 +9,80 @@ using Microsoft.Extensions.Diagnostics.HealthChecks;
|
||||
namespace ZB.MOM.WW.Health.Akka;
|
||||
|
||||
/// <summary>
|
||||
/// Pure decision function for the active / leader probe, factored out of
|
||||
/// <see cref="ActiveNodeHealthCheck"/> so the role-less and role-filtered matrices are exhaustively
|
||||
/// table-testable without forming a real cluster.
|
||||
/// </summary>
|
||||
internal static class ActiveNodeDecision
|
||||
{
|
||||
/// <summary>
|
||||
/// Maps the resolved cluster facts to a <see cref="HealthStatus"/>.
|
||||
/// </summary>
|
||||
/// <param name="selfUp">Whether the local node's member status is <c>Up</c>.</param>
|
||||
/// <param name="isLeader">
|
||||
/// Whether the local node is the leader: the cluster leader in role-less mode, or the
|
||||
/// role-singleton leader in role-filtered mode.
|
||||
/// </param>
|
||||
/// <param name="hasRole">
|
||||
/// Whether the local node carries <paramref name="requiredRole"/>. Ignored when
|
||||
/// <paramref name="requiredRole"/> is <c>null</c>.
|
||||
/// </param>
|
||||
/// <param name="requiredRole">
|
||||
/// The role to scope the check to, or <c>null</c> for the role-less (whole-cluster-leader) mode.
|
||||
/// </param>
|
||||
/// <returns>
|
||||
/// Role-less: Healthy iff the node is Up and the cluster leader, otherwise Unhealthy.
|
||||
/// Role-filtered: Healthy when the node lacks the role (probe irrelevant) or carries the role and
|
||||
/// is the role-singleton leader; Degraded when it carries the role but is not the leader.
|
||||
/// </returns>
|
||||
public static HealthStatus Evaluate(bool selfUp, bool isLeader, bool hasRole, string? requiredRole)
|
||||
{
|
||||
if (requiredRole is null)
|
||||
return selfUp && isLeader ? HealthStatus.Healthy : HealthStatus.Unhealthy;
|
||||
|
||||
if (!hasRole)
|
||||
return HealthStatus.Healthy;
|
||||
|
||||
return isLeader ? HealthStatus.Healthy : HealthStatus.Degraded;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Health check that reports whether this node is the designated active / leader node.
|
||||
/// An optional role scopes the check to nodes carrying that role. Register to the
|
||||
/// <see cref="ZbHealthTags.Active"/> tag.
|
||||
/// Active-tier probe: Healthy (200) on the one node in charge, Unhealthy (503) on its standby.
|
||||
/// Register to the <see cref="ZbHealthTags.Active"/> tag.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The <see cref="ActorSystem"/> is resolved lazily from the service provider. If it is not yet
|
||||
/// available — e.g. during startup before Akka is initialised — the check returns
|
||||
/// <see cref="HealthStatus.Degraded"/> rather than throwing, so it is startup-safe.
|
||||
/// <para>
|
||||
/// "In charge" means the <b>oldest Up member</b> — see <see cref="ClusterActiveNode"/> for
|
||||
/// why leadership is the wrong answer and what breaks when it is used. Scope the competition
|
||||
/// to a role with <see cref="ActiveNodeHealthCheckOptions.RolePreference"/>; leave it empty
|
||||
/// for a cluster whose nodes all compete for the same single active slot.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Behaviour change in 0.3.0.</b> Through 0.2.1 this check selected by
|
||||
/// <c>ClusterState.Leader</c> / <c>RoleLeader</c>, and its role-filtered mode reported
|
||||
/// Healthy for any node <i>lacking</i> the role. Both were wrong, both were found on live
|
||||
/// rigs, and both Akka consumers had already replaced it with a private copy rather than use
|
||||
/// it. It now selects by age and reports <see cref="HealthStatus.Unhealthy"/> for a node that
|
||||
/// owns no active work. A node whose active tier previously answered 200 unconditionally will
|
||||
/// now answer 503 unless it really is the active node — which is the point.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Startup-safe: before the <see cref="ActorSystem"/> or the cluster is available the result
|
||||
/// is <see cref="HealthStatus.Degraded"/> (a 200 under the tier mapping), so a node that is
|
||||
/// merely still booting is not reported as a failed standby and pulled from rotation on the
|
||||
/// way up.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class ActiveNodeHealthCheck : IHealthCheck
|
||||
{
|
||||
private readonly IServiceProvider _serviceProvider;
|
||||
private readonly string? _role;
|
||||
private readonly ActiveNodeHealthCheckOptions _options;
|
||||
|
||||
/// <summary>
|
||||
/// Role-less constructor: Healthy when the node is <c>Up</c> and the cluster leader
|
||||
/// (ScadaBridge ActiveNode pattern); Unhealthy otherwise. Degraded when the ActorSystem /
|
||||
/// cluster is not yet ready.
|
||||
/// Unscoped: the active node is the oldest Up member of the whole cluster.
|
||||
/// </summary>
|
||||
/// <param name="serviceProvider">
|
||||
/// The application service provider. The <see cref="ActorSystem"/> is resolved lazily so the
|
||||
/// check is startup-safe: if no <see cref="ActorSystem"/> is registered yet the result is Degraded.
|
||||
/// check is startup-safe.
|
||||
/// </param>
|
||||
public ActiveNodeHealthCheck(IServiceProvider serviceProvider)
|
||||
: this(serviceProvider, new ActiveNodeHealthCheckOptions())
|
||||
{
|
||||
_serviceProvider = serviceProvider ?? throw new ArgumentNullException(nameof(serviceProvider));
|
||||
_role = null;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Role-filtered constructor: Healthy when the node lacks <paramref name="role"/> or carries it
|
||||
/// and is the role-singleton leader; Degraded when it carries the role but is not the leader
|
||||
/// (OtOpcUa AdminRoleLeader pattern). Degraded when the ActorSystem / cluster is not yet ready.
|
||||
/// Role-scoped: the active node is the oldest Up member carrying <paramref name="role"/>.
|
||||
/// </summary>
|
||||
/// <param name="serviceProvider">
|
||||
/// The application service provider. The <see cref="ActorSystem"/> is resolved lazily so the
|
||||
/// check is startup-safe: if no <see cref="ActorSystem"/> is registered yet the result is Degraded.
|
||||
/// check is startup-safe.
|
||||
/// </param>
|
||||
/// <param name="role">The Akka cluster role to scope the check to.</param>
|
||||
/// <param name="role">The cluster role to scope the competition to.</param>
|
||||
public ActiveNodeHealthCheck(IServiceProvider serviceProvider, string role)
|
||||
: this(serviceProvider, NewOptions(role))
|
||||
{
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Configured with an explicit <see cref="ActiveNodeHealthCheckOptions"/>.
|
||||
/// </summary>
|
||||
/// <param name="serviceProvider">
|
||||
/// The application service provider. The <see cref="ActorSystem"/> is resolved lazily so the
|
||||
/// check is startup-safe.
|
||||
/// </param>
|
||||
/// <param name="options">The role-scoping options.</param>
|
||||
public ActiveNodeHealthCheck(IServiceProvider serviceProvider, ActiveNodeHealthCheckOptions options)
|
||||
{
|
||||
_serviceProvider = serviceProvider ?? throw new ArgumentNullException(nameof(serviceProvider));
|
||||
_options = options ?? throw new ArgumentNullException(nameof(options));
|
||||
}
|
||||
|
||||
private static ActiveNodeHealthCheckOptions NewOptions(string role)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(role);
|
||||
_role = role;
|
||||
return new ActiveNodeHealthCheckOptions { RolePreference = new[] { role } };
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
@@ -103,10 +94,8 @@ public sealed class ActiveNodeHealthCheck : IHealthCheck
|
||||
if (system is null)
|
||||
return Task.FromResult(HealthCheckResult.Degraded("ActorSystem not yet available."));
|
||||
|
||||
MemberStatus selfStatus;
|
||||
bool selfUp;
|
||||
bool hasRole;
|
||||
bool isLeader;
|
||||
Member self;
|
||||
IEnumerable<Member> members;
|
||||
try
|
||||
{
|
||||
// Reading cluster membership can throw while the ActorSystem exists but the cluster has
|
||||
@@ -114,54 +103,49 @@ public sealed class ActiveNodeHealthCheck : IHealthCheck
|
||||
// ConfigurationException). The spec's startup-safety rule maps this to Degraded rather
|
||||
// than letting the exception escape (which the host would record as Unhealthy).
|
||||
var cluster = Cluster.Get(system);
|
||||
var self = cluster.SelfMember;
|
||||
selfStatus = self.Status;
|
||||
selfUp = selfStatus == MemberStatus.Up;
|
||||
|
||||
if (_role is null)
|
||||
{
|
||||
hasRole = false;
|
||||
var leader = cluster.State.Leader;
|
||||
isLeader = leader is not null && leader == self.Address;
|
||||
}
|
||||
else
|
||||
{
|
||||
hasRole = self.HasRole(_role);
|
||||
var roleLeader = cluster.State.RoleLeader(_role);
|
||||
isLeader = roleLeader is not null && roleLeader == self.Address;
|
||||
}
|
||||
self = cluster.SelfMember;
|
||||
members = cluster.State.Members;
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
return Task.FromResult(HealthCheckResult.Degraded("Akka cluster state not yet accessible.", ex));
|
||||
}
|
||||
|
||||
var health = ActiveNodeDecision.Evaluate(selfUp, isLeader, hasRole, _role);
|
||||
var description = DescribeResult(health, selfStatus, selfUp, isLeader);
|
||||
var result = health switch
|
||||
{
|
||||
HealthStatus.Healthy => HealthCheckResult.Healthy(description),
|
||||
HealthStatus.Degraded => HealthCheckResult.Degraded(description),
|
||||
_ => HealthCheckResult.Unhealthy(description),
|
||||
};
|
||||
return Task.FromResult(result);
|
||||
}
|
||||
var scoped = _options.RolePreference.Count > 0;
|
||||
var role = ClusterActiveNode.ResolveActiveRole(self.Roles, _options.RolePreference);
|
||||
|
||||
private string DescribeResult(HealthStatus health, MemberStatus status, bool selfUp, bool isLeader)
|
||||
{
|
||||
if (_role is null)
|
||||
var data = new Dictionary<string, object>(StringComparer.Ordinal)
|
||||
{
|
||||
if (health == HealthStatus.Healthy)
|
||||
return "Active node (cluster leader).";
|
||||
return selfUp && !isLeader
|
||||
? "Standby: node is Up but not the cluster leader."
|
||||
: $"Standby: node is not Up (status: {status}).";
|
||||
["selfAddress"] = self.Address.ToString(),
|
||||
};
|
||||
if (role is not null) data["activeRole"] = role;
|
||||
|
||||
if (scoped && role is null)
|
||||
{
|
||||
// The node carries none of the preferred roles, so it owns no active work and cannot be
|
||||
// the active node for any of them. Reported per NoActiveRoleStatus — Unhealthy by
|
||||
// default, because the active tier's whole job is to keep such a node out of the pool.
|
||||
var message = $"Node carries none of the active roles: {string.Join(", ", _options.RolePreference)}.";
|
||||
return Task.FromResult(new HealthCheckResult(_options.NoActiveRoleStatus, message, exception: null, data));
|
||||
}
|
||||
|
||||
return health switch
|
||||
var active = ClusterActiveNode.OldestUpMember(members, role);
|
||||
if (active is not null) data["activeNode"] = active.Address.ToString();
|
||||
|
||||
var scopeLabel = role is null ? "the cluster" : $"role '{role}'";
|
||||
|
||||
if (active is null)
|
||||
{
|
||||
HealthStatus.Healthy => $"Active for role '{_role}' (or not a role member).",
|
||||
_ => $"Role '{_role}' member but not leader.",
|
||||
};
|
||||
// Nobody is Up in the scope — a cold start, or every holder is still Joining or already
|
||||
// Leaving. Nobody is active, so nobody may claim to be.
|
||||
return Task.FromResult(HealthCheckResult.Unhealthy(
|
||||
$"No Up member currently holds the active slot for {scopeLabel}.", exception: null, data));
|
||||
}
|
||||
|
||||
// Compared by UniqueAddress: a node that restarted on the same host and port is a different
|
||||
// member, and during the overlap the cluster briefly holds both incarnations.
|
||||
return Task.FromResult(active.UniqueAddress.Equals(self.UniqueAddress)
|
||||
? HealthCheckResult.Healthy($"Active node for {scopeLabel} (oldest Up member).", data)
|
||||
: HealthCheckResult.Unhealthy($"Standby: active node for {scopeLabel} is {active.Address}.", exception: null, data));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
using Microsoft.Extensions.Diagnostics.HealthChecks;
|
||||
|
||||
namespace ZB.MOM.WW.Health.Akka;
|
||||
|
||||
/// <summary>
|
||||
/// Configures how <see cref="ActiveNodeHealthCheck"/> decides which role's active node this node is
|
||||
/// competing to be.
|
||||
/// </summary>
|
||||
public sealed class ActiveNodeHealthCheckOptions
|
||||
{
|
||||
/// <summary>
|
||||
/// Candidate roles in descending precedence. The check answers for the first one the local node
|
||||
/// carries. Empty (the default) means unscoped: the active node is the oldest Up member of the
|
||||
/// whole cluster.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A fused node should list the role its singletons are pinned to first — e.g.
|
||||
/// <c>["admin", "driver"]</c> for a node that hosts the admin singletons when fused and competes
|
||||
/// as a plain driver when it is a driver-only node.
|
||||
/// </remarks>
|
||||
public IReadOnlyList<string> RolePreference { get; init; } = Array.Empty<string>();
|
||||
|
||||
/// <summary>
|
||||
/// The status to report when <see cref="RolePreference"/> is non-empty and the node carries none
|
||||
/// of those roles — it owns no active work at all. Defaults to
|
||||
/// <see cref="HealthStatus.Unhealthy"/>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Unhealthy is the safe default and is deliberately <b>not</b> the "not applicable ⇒ Healthy"
|
||||
/// convention used by readiness probes. The active tier exists so a load balancer can pick
|
||||
/// exactly one node; a node that owns no active work must not be in that pool. Answering Healthy
|
||||
/// there is what made the previous role-filtered check report 200 on every node and silently
|
||||
/// broke leader-pinning (lmxopcua#494). Override only when a node genuinely should stay in the
|
||||
/// active pool without holding any of the preferred roles.
|
||||
/// </remarks>
|
||||
public HealthStatus NoActiveRoleStatus { get; init; } = HealthStatus.Unhealthy;
|
||||
}
|
||||
@@ -6,27 +6,47 @@ namespace ZB.MOM.WW.Health.Akka;
|
||||
|
||||
/// <summary>
|
||||
/// <see cref="IActiveNodeGate"/> implementation that computes <see cref="IsActiveNode"/> directly
|
||||
/// from the Akka cluster state (self member <c>Up</c> and the local node is the cluster leader).
|
||||
/// Register as a singleton.
|
||||
/// from the Akka cluster state — the local node is the <b>oldest Up member</b>, optionally within a
|
||||
/// role scope. Register as a singleton.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The <see cref="ActorSystem"/> is resolved lazily from the service provider; if it is not yet
|
||||
/// available — e.g. during startup before Akka is initialised — <see cref="IsActiveNode"/> returns
|
||||
/// <c>false</c> (the safe default during startup). This gate reads the cluster state directly and
|
||||
/// does not resolve <see cref="ActiveNodeHealthCheck"/> from DI.
|
||||
/// <para>
|
||||
/// Shares its rule with <see cref="ActiveNodeHealthCheck"/> via
|
||||
/// <see cref="ClusterActiveNode"/>, so an endpoint gated by this type and the
|
||||
/// <c>/health/active</c> probe an orchestrator routes by can never disagree about which node
|
||||
/// is in charge.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Behaviour change in 0.3.0:</b> through 0.2.1 this gate selected by
|
||||
/// <c>ClusterState.Leader</c>, which is address-ordered and diverges from singleton placement
|
||||
/// after any restart. See <see cref="ClusterActiveNode"/>.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The <see cref="ActorSystem"/> is resolved lazily from the service provider; if it is not
|
||||
/// yet available — e.g. during startup before Akka is initialised —
|
||||
/// <see cref="IsActiveNode"/> returns <c>false</c> (the safe default during startup, matching
|
||||
/// the standby case). This gate reads the cluster state directly and does not resolve
|
||||
/// <see cref="ActiveNodeHealthCheck"/> from DI.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class AkkaActiveNodeGate : IActiveNodeGate
|
||||
{
|
||||
private readonly IServiceProvider _serviceProvider;
|
||||
private readonly string? _role;
|
||||
|
||||
/// <summary>Initializes a new <see cref="AkkaActiveNodeGate"/>.</summary>
|
||||
/// <param name="serviceProvider">
|
||||
/// The application service provider. The <see cref="ActorSystem"/> is resolved lazily; if it is
|
||||
/// not yet available <see cref="IsActiveNode"/> returns <c>false</c>.
|
||||
/// </param>
|
||||
public AkkaActiveNodeGate(IServiceProvider serviceProvider)
|
||||
/// <param name="role">
|
||||
/// The role to scope the competition to, or <c>null</c> (the default) to consider every Up
|
||||
/// member.
|
||||
/// </param>
|
||||
public AkkaActiveNodeGate(IServiceProvider serviceProvider, string? role = null)
|
||||
{
|
||||
_serviceProvider = serviceProvider ?? throw new ArgumentNullException(nameof(serviceProvider));
|
||||
_role = role;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
@@ -38,13 +58,16 @@ public sealed class AkkaActiveNodeGate : IActiveNodeGate
|
||||
if (system is null)
|
||||
return false;
|
||||
|
||||
var cluster = Cluster.Get(system);
|
||||
var self = cluster.SelfMember;
|
||||
if (self.Status != MemberStatus.Up)
|
||||
try
|
||||
{
|
||||
return ClusterActiveNode.SelfIsActive(Cluster.Get(system), _role);
|
||||
}
|
||||
catch (Exception)
|
||||
{
|
||||
// ActorSystem exists but the cluster is not initialised yet. Same startup-safety
|
||||
// rule as the health check, expressed as the gate's safe default: not active.
|
||||
return false;
|
||||
|
||||
var leader = cluster.State.Leader;
|
||||
return leader is not null && leader == self.Address;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -44,6 +44,7 @@ public sealed class AkkaClusterHealthCheck : IHealthCheck
|
||||
return Task.FromResult(HealthCheckResult.Degraded("ActorSystem not yet available."));
|
||||
|
||||
MemberStatus status;
|
||||
IReadOnlyDictionary<string, object> data;
|
||||
try
|
||||
{
|
||||
// Cluster.Get(system).SelfMember can throw while the ActorSystem exists but the cluster
|
||||
@@ -51,7 +52,9 @@ public sealed class AkkaClusterHealthCheck : IHealthCheck
|
||||
// ConfigurationException). The spec's startup-safety rule maps this to Degraded, not an
|
||||
// escaping exception (which the host would record as Unhealthy and pull the node from
|
||||
// rotation).
|
||||
status = Cluster.Get(system).SelfMember.Status;
|
||||
var cluster = Cluster.Get(system);
|
||||
status = cluster.SelfMember.Status;
|
||||
data = BuildClusterData(cluster);
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
@@ -62,10 +65,47 @@ public sealed class AkkaClusterHealthCheck : IHealthCheck
|
||||
var description = $"Akka cluster member status: {status}";
|
||||
var result = health switch
|
||||
{
|
||||
HealthStatus.Healthy => HealthCheckResult.Healthy(description),
|
||||
HealthStatus.Degraded => HealthCheckResult.Degraded(description),
|
||||
_ => HealthCheckResult.Unhealthy(description),
|
||||
HealthStatus.Healthy => HealthCheckResult.Healthy(description, data),
|
||||
HealthStatus.Degraded => HealthCheckResult.Degraded(description, data: data),
|
||||
_ => HealthCheckResult.Unhealthy(description, data: data),
|
||||
};
|
||||
return Task.FromResult(result);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Projects this node's view of the cluster into the check's <c>data</c> dictionary.
|
||||
/// </summary>
|
||||
/// <param name="cluster">The cluster extension for the local <see cref="ActorSystem"/>.</param>
|
||||
/// <returns>
|
||||
/// JSON-friendly values only (string / int / string[]), so the canonical health writer can emit
|
||||
/// them verbatim: <c>leader</c> (omitted while the leader is unknown, e.g. pre-join or during
|
||||
/// convergence), <c>selfAddress</c>, <c>selfRoles</c> (sorted for a stable payload),
|
||||
/// <c>memberCount</c>, <c>unreachableCount</c>.
|
||||
/// </returns>
|
||||
/// <remarks>
|
||||
/// This is deliberately each node's <em>own</em> view rather than an authoritative fleet answer —
|
||||
/// a consumer that reads it from several members of the same cluster can compare them, and a
|
||||
/// disagreement is a split-brain tell.
|
||||
/// </remarks>
|
||||
internal static IReadOnlyDictionary<string, object> BuildClusterData(Cluster cluster)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(cluster);
|
||||
|
||||
var state = cluster.State;
|
||||
var data = new Dictionary<string, object>(StringComparer.Ordinal)
|
||||
{
|
||||
["selfAddress"] = cluster.SelfAddress.ToString(),
|
||||
["selfRoles"] = cluster.SelfMember.Roles.OrderBy(static r => r, StringComparer.Ordinal).ToArray(),
|
||||
["memberCount"] = state.Members.Count,
|
||||
["unreachableCount"] = state.Unreachable.Count,
|
||||
};
|
||||
|
||||
// Omitted rather than emitted as null: a missing key reads as "this node does not know yet",
|
||||
// which is exactly the pre-join / mid-convergence state.
|
||||
var leader = state.Leader;
|
||||
if (leader is not null)
|
||||
data["leader"] = leader.ToString();
|
||||
|
||||
return data;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
using Akka.Cluster;
|
||||
|
||||
namespace ZB.MOM.WW.Health.Akka;
|
||||
|
||||
/// <summary>
|
||||
/// THE family-wide definition of "the active node": the <b>oldest Up member</b> of the cluster,
|
||||
/// optionally scoped to a role — the member <c>ClusterSingletonManager</c> places singletons on.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Cluster <b>leadership</b> (<c>ClusterState.Leader</c> / <c>RoleLeader</c>) is the wrong
|
||||
/// answer and must never be used for a product-level active/standby decision. Leadership is
|
||||
/// <i>address</i>-ordered (host, then port) and has no relationship to time; singleton
|
||||
/// placement is <i>age</i>-ordered. On a freshly-formed cluster the two agree, which is why
|
||||
/// the discrepancy hides in unit tests and in a rig that has never been restarted. They
|
||||
/// diverge permanently after any restart: the restarted node rejoins as the youngest but
|
||||
/// keeps its address, so if it holds the lower address it becomes leader while the
|
||||
/// singletons — and all the work they own — stay on the other node.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Both Akka consumers in the family independently hit this and each wrote its own
|
||||
/// replacement rather than use the shared leader-based check
|
||||
/// (ScadaBridge <c>OldestNodeActiveHealthCheck</c>, OtOpcUa <c>ClusterPrimaryHealthCheck</c>).
|
||||
/// This type is those two implementations promoted to one, so the health tier, the inbound
|
||||
/// API gate, the store-and-forward delivery gate and OtOpcUa's OPC UA <c>ServiceLevel</c>
|
||||
/// cannot drift apart about which node is in charge.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Only <see cref="MemberStatus.Up"/> members are eligible, matching singleton placement: a
|
||||
/// <c>Leaving</c> node is handing its singletons over and must not be named active.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public static class ClusterActiveNode
|
||||
{
|
||||
/// <summary>
|
||||
/// The oldest <see cref="MemberStatus.Up"/> member, optionally scoped to <paramref name="role"/>.
|
||||
/// </summary>
|
||||
/// <param name="members">The cluster members, in any order.</param>
|
||||
/// <param name="role">
|
||||
/// The role to scope the selection to, or <c>null</c> to consider every Up member.
|
||||
/// </param>
|
||||
/// <returns>The oldest eligible member, or <c>null</c> when none is Up.</returns>
|
||||
public static Member? OldestUpMember(IEnumerable<Member> members, string? role = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(members);
|
||||
|
||||
Member? oldest = null;
|
||||
foreach (var member in members)
|
||||
{
|
||||
if (member.Status != MemberStatus.Up) continue;
|
||||
if (role is not null && !member.HasRole(role)) continue;
|
||||
if (oldest is null || member.IsOlderThan(oldest)) oldest = member;
|
||||
}
|
||||
|
||||
return oldest;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The oldest <see cref="MemberStatus.Up"/> member of <paramref name="cluster"/>, optionally
|
||||
/// scoped to <paramref name="role"/>.
|
||||
/// </summary>
|
||||
/// <param name="cluster">The Akka cluster to evaluate.</param>
|
||||
/// <param name="role">
|
||||
/// The role to scope the selection to, or <c>null</c> to consider every Up member.
|
||||
/// </param>
|
||||
/// <returns>The oldest eligible member, or <c>null</c> when none is Up.</returns>
|
||||
public static Member? OldestUpMember(Cluster cluster, string? role = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(cluster);
|
||||
return OldestUpMember(cluster.State.Members, role);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Whether the local node is the active node — Up, and the oldest Up member in the role scope.
|
||||
/// </summary>
|
||||
/// <param name="cluster">The Akka cluster to evaluate.</param>
|
||||
/// <param name="role">
|
||||
/// The role to scope the selection to, or <c>null</c> to consider every Up member.
|
||||
/// </param>
|
||||
/// <returns>
|
||||
/// <c>true</c> when the local member is Up, carries <paramref name="role"/> when one is given,
|
||||
/// and no other eligible member is older.
|
||||
/// </returns>
|
||||
/// <remarks>
|
||||
/// Identity is compared by <see cref="Member.UniqueAddress"/>, not <c>Address</c>: a node that
|
||||
/// restarted on the same host and port is a different member, and during the overlap the cluster
|
||||
/// briefly holds both incarnations.
|
||||
/// </remarks>
|
||||
public static bool SelfIsActive(Cluster cluster, string? role = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(cluster);
|
||||
|
||||
var self = cluster.SelfMember;
|
||||
if (self.Status != MemberStatus.Up) return false;
|
||||
if (role is not null && !self.HasRole(role)) return false;
|
||||
|
||||
var oldest = OldestUpMember(cluster.State.Members, role);
|
||||
return oldest is not null && oldest.UniqueAddress.Equals(self.UniqueAddress);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The first role in <paramref name="rolePreference"/> that <paramref name="selfRoles"/> carries
|
||||
/// — the role whose oldest Up member owns this node's active work.
|
||||
/// </summary>
|
||||
/// <param name="selfRoles">The local member's cluster roles.</param>
|
||||
/// <param name="rolePreference">
|
||||
/// Candidate roles in descending precedence, e.g. <c>["admin", "driver"]</c> for a node that
|
||||
/// answers for <c>admin</c> when it is fused and for <c>driver</c> when it is a driver-only node.
|
||||
/// </param>
|
||||
/// <returns>
|
||||
/// The first carried role, or <c>null</c> when the node carries none of them.
|
||||
/// </returns>
|
||||
/// <remarks>
|
||||
/// Precedence matters on a fused node: it must answer for the role its singletons — and the
|
||||
/// traffic being routed to it — are pinned to. Answering for a lower-precedence role could name a
|
||||
/// different node the moment a single-role member joins the mesh.
|
||||
/// </remarks>
|
||||
public static string? ResolveActiveRole(IEnumerable<string> selfRoles, IReadOnlyList<string> rolePreference)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(selfRoles);
|
||||
ArgumentNullException.ThrowIfNull(rolePreference);
|
||||
|
||||
if (rolePreference.Count == 0) return null;
|
||||
|
||||
var roles = selfRoles as IReadOnlyCollection<string> ?? selfRoles.ToList();
|
||||
foreach (var candidate in rolePreference)
|
||||
{
|
||||
if (roles.Contains(candidate)) return candidate;
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
}
|
||||
@@ -1,4 +1,5 @@
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Serialization;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
using Microsoft.Extensions.Diagnostics.HealthChecks;
|
||||
|
||||
@@ -16,15 +17,18 @@ namespace ZB.MOM.WW.Health;
|
||||
/// "status": "Healthy|Degraded|Unhealthy",
|
||||
/// "totalDurationMs": 12.34,
|
||||
/// "entries": {
|
||||
/// "<name>": { "status": "...", "description": "...", "durationMs": 1.23 }
|
||||
/// "<name>": { "status": "...", "description": "...", "durationMs": 1.23, "data": { ... } }
|
||||
/// }
|
||||
/// }
|
||||
/// </code>
|
||||
/// The <c>description</c> key is always present; when a check supplies no description it is emitted
|
||||
/// as JSON <c>null</c> (not omitted), matching the spec example and the <c>HealthChecks.UI.Client</c>
|
||||
/// shape. The HTTP status code is left to the ASP.NET Core health-checks middleware (Healthy/Degraded
|
||||
/// → 200, Unhealthy → 503); this writer only renders the body and sets
|
||||
/// <c>Content-Type: application/json</c>.
|
||||
/// shape. The <c>data</c> key is the opposite: it is emitted only when the check supplied a non-empty
|
||||
/// <see cref="HealthCheckResult.Data"/> dictionary, so payloads from checks that publish no data stay
|
||||
/// byte-identical to the pre-0.2.0 shape. Its keys are written verbatim (the camelCase naming policy
|
||||
/// applies to the envelope properties, not to dictionary keys). The HTTP status code is left to the
|
||||
/// ASP.NET Core health-checks middleware (Healthy/Degraded → 200, Unhealthy → 503); this writer only
|
||||
/// renders the body and sets <c>Content-Type: application/json</c>.
|
||||
/// </remarks>
|
||||
public static class ZbHealthWriter
|
||||
{
|
||||
@@ -60,6 +64,10 @@ public static class ZbHealthWriter
|
||||
Status = e.Value.Status.ToString(),
|
||||
Description = e.Value.Description,
|
||||
DurationMs = e.Value.Duration.TotalMilliseconds,
|
||||
// HealthReportEntry.Data is never null (empty when the check set none); mapping
|
||||
// empty → null is what drops the key entirely, keeping data-less payloads
|
||||
// byte-identical to the pre-0.2.0 shape.
|
||||
Data = e.Value.Data.Count > 0 ? e.Value.Data : null,
|
||||
}),
|
||||
};
|
||||
|
||||
@@ -78,5 +86,11 @@ public static class ZbHealthWriter
|
||||
public string Status { get; init; } = string.Empty;
|
||||
public string? Description { get; init; }
|
||||
public double DurationMs { get; init; }
|
||||
|
||||
// Per-property ignore, deliberately NOT a global DefaultIgnoreCondition: the writer must keep
|
||||
// emitting "description": null (see SerializerOptions above) while omitting "data" entirely
|
||||
// when a check publishes none.
|
||||
[JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
|
||||
public IReadOnlyDictionary<string, object>? Data { get; init; }
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,137 +0,0 @@
|
||||
using Akka.Actor;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Diagnostics.HealthChecks;
|
||||
using ZB.MOM.WW.Health.Akka;
|
||||
|
||||
namespace ZB.MOM.WW.Health.Akka.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Table-driven tests for the pure <see cref="ActiveNodeDecision.Evaluate"/> helper covering both
|
||||
/// the role-less (ScadaBridge ActiveNode) and role-filtered (OtOpcUa AdminRoleLeader) matrices,
|
||||
/// plus the startup-safety null-guards on <see cref="ActiveNodeHealthCheck"/> and
|
||||
/// <see cref="AkkaActiveNodeGate"/> when no <see cref="ActorSystem"/> is registered.
|
||||
/// </summary>
|
||||
public sealed class ActiveNodeDecisionTests
|
||||
{
|
||||
// Role-less: requiredRole == null. hasRole is irrelevant. Healthy iff (selfUp && isLeader), else Unhealthy.
|
||||
public static IEnumerable<object?[]> RoleLessCases() => new[]
|
||||
{
|
||||
new object?[] { true, true, false, (string?)null, HealthStatus.Healthy },
|
||||
new object?[] { true, false, false, (string?)null, HealthStatus.Unhealthy },
|
||||
new object?[] { false, true, false, (string?)null, HealthStatus.Unhealthy },
|
||||
new object?[] { false, false, false, (string?)null, HealthStatus.Unhealthy },
|
||||
};
|
||||
|
||||
[Theory]
|
||||
[MemberData(nameof(RoleLessCases))]
|
||||
public void Evaluate_RoleLess(bool selfUp, bool isLeader, bool hasRole, string? requiredRole, HealthStatus expected)
|
||||
{
|
||||
Assert.Equal(expected, ActiveNodeDecision.Evaluate(selfUp, isLeader, hasRole, requiredRole));
|
||||
}
|
||||
|
||||
// Role-filtered: requiredRole != null.
|
||||
// lacks role -> Healthy (probe irrelevant for this node)
|
||||
// has role & is leader -> Healthy (selfUp is ignored — role-filtered mode only cares about leadership)
|
||||
// has role & not leader -> Degraded
|
||||
public static IEnumerable<object[]> RoleFilteredCases() => new[]
|
||||
{
|
||||
// node lacks the role -> Healthy regardless of selfUp / isLeader
|
||||
new object[] { true, true, false, "admin", HealthStatus.Healthy },
|
||||
new object[] { true, false, false, "admin", HealthStatus.Healthy },
|
||||
new object[] { false, false, false, "admin", HealthStatus.Healthy },
|
||||
// node carries the role and is leader -> Healthy (selfUp=true)
|
||||
new object[] { true, true, true, "admin", HealthStatus.Healthy },
|
||||
// node carries the role and is leader -> Healthy (selfUp=false: role-filtered mode ignores selfUp)
|
||||
new object[] { false, true, true, "admin", HealthStatus.Healthy },
|
||||
// node carries the role but is not leader -> Degraded
|
||||
new object[] { true, false, true, "admin", HealthStatus.Degraded },
|
||||
new object[] { false, false, true, "admin", HealthStatus.Degraded },
|
||||
};
|
||||
|
||||
[Theory]
|
||||
[MemberData(nameof(RoleFilteredCases))]
|
||||
public void Evaluate_RoleFiltered(bool selfUp, bool isLeader, bool hasRole, string? requiredRole, HealthStatus expected)
|
||||
{
|
||||
Assert.Equal(expected, ActiveNodeDecision.Evaluate(selfUp, isLeader, hasRole, requiredRole));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task HealthCheck_RoleLess_NoActorSystem_ReturnsDegraded()
|
||||
{
|
||||
var provider = new ServiceCollection().BuildServiceProvider();
|
||||
var check = new ActiveNodeHealthCheck(provider);
|
||||
|
||||
var result = await check.CheckHealthAsync(NewContext(check));
|
||||
|
||||
Assert.Equal(HealthStatus.Degraded, result.Status);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task HealthCheck_RoleFiltered_NoActorSystem_ReturnsDegraded()
|
||||
{
|
||||
var provider = new ServiceCollection().BuildServiceProvider();
|
||||
var check = new ActiveNodeHealthCheck(provider, "admin");
|
||||
|
||||
var result = await check.CheckHealthAsync(NewContext(check));
|
||||
|
||||
Assert.Equal(HealthStatus.Degraded, result.Status);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Gate_NoActorSystem_IsActiveNodeFalse()
|
||||
{
|
||||
var provider = new ServiceCollection().BuildServiceProvider();
|
||||
var gate = new AkkaActiveNodeGate(provider);
|
||||
|
||||
Assert.False(gate.IsActiveNode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task HealthCheck_RoleLess_ClusterInaccessible_ReturnsDegraded()
|
||||
{
|
||||
// ActorSystem present but Akka.Cluster not configured → Cluster.Get throws. The check must
|
||||
// return Degraded (startup-safety rule), not let the exception escape (→ Unhealthy).
|
||||
using var system = ActorSystem.Create("plain-no-cluster-roleless");
|
||||
try
|
||||
{
|
||||
var provider = new ServiceCollection()
|
||||
.AddSingleton(system)
|
||||
.BuildServiceProvider();
|
||||
var check = new ActiveNodeHealthCheck(provider);
|
||||
|
||||
var result = await check.CheckHealthAsync(NewContext(check));
|
||||
|
||||
Assert.Equal(HealthStatus.Degraded, result.Status);
|
||||
}
|
||||
finally
|
||||
{
|
||||
await system.Terminate();
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task HealthCheck_RoleFiltered_ClusterInaccessible_ReturnsDegraded()
|
||||
{
|
||||
using var system = ActorSystem.Create("plain-no-cluster-rolefiltered");
|
||||
try
|
||||
{
|
||||
var provider = new ServiceCollection()
|
||||
.AddSingleton(system)
|
||||
.BuildServiceProvider();
|
||||
var check = new ActiveNodeHealthCheck(provider, "admin");
|
||||
|
||||
var result = await check.CheckHealthAsync(NewContext(check));
|
||||
|
||||
Assert.Equal(HealthStatus.Degraded, result.Status);
|
||||
}
|
||||
finally
|
||||
{
|
||||
await system.Terminate();
|
||||
}
|
||||
}
|
||||
|
||||
private static HealthCheckContext NewContext(IHealthCheck check) => new()
|
||||
{
|
||||
Registration = new HealthCheckRegistration("active-node", check, HealthStatus.Unhealthy, tags: null),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Diagnostics.HealthChecks;
|
||||
using ZB.MOM.WW.Health.Akka;
|
||||
|
||||
namespace ZB.MOM.WW.Health.Akka.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// The states a node passes through on the way up, where the active tier must not report a booting
|
||||
/// node as a failed standby — now that the tier is a real 503, an Unhealthy here would pull the node
|
||||
/// out of an orchestrator's pool while it starts.
|
||||
/// </summary>
|
||||
public sealed class ActiveNodeStartupSafetyTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task NoActorSystem_IsDegraded()
|
||||
{
|
||||
var check = new ActiveNodeHealthCheck(new ServiceCollection().BuildServiceProvider());
|
||||
|
||||
var result = await check.CheckHealthAsync(NewContext(check));
|
||||
|
||||
Assert.Equal(HealthStatus.Degraded, result.Status);
|
||||
// Description-only: an empty data dictionary is what makes the writer omit "data" entirely.
|
||||
Assert.Empty(result.Data);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task NoActorSystem_RoleScoped_IsDegraded()
|
||||
{
|
||||
var check = new ActiveNodeHealthCheck(new ServiceCollection().BuildServiceProvider(), "admin");
|
||||
|
||||
Assert.Equal(HealthStatus.Degraded, (await check.CheckHealthAsync(NewContext(check))).Status);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Gate_NoActorSystem_IsNotActive()
|
||||
{
|
||||
Assert.False(new AkkaActiveNodeGate(new ServiceCollection().BuildServiceProvider()).IsActiveNode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ClusterNotConfigured_IsDegraded()
|
||||
{
|
||||
// ActorSystem present but Akka.Cluster not configured → Cluster.Get throws. The check must
|
||||
// return Degraded (the spec's startup-safety rule), not let the exception escape (→ Unhealthy).
|
||||
using var system = ActorSystem.Create("active-node-no-cluster");
|
||||
try
|
||||
{
|
||||
var provider = new ServiceCollection().AddSingleton(system).BuildServiceProvider();
|
||||
var check = new ActiveNodeHealthCheck(provider);
|
||||
|
||||
Assert.Equal(HealthStatus.Degraded, (await check.CheckHealthAsync(NewContext(check))).Status);
|
||||
}
|
||||
finally
|
||||
{
|
||||
await system.Terminate();
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Gate_ClusterNotConfigured_IsNotActive()
|
||||
{
|
||||
using var system = ActorSystem.Create("active-node-gate-no-cluster");
|
||||
try
|
||||
{
|
||||
var provider = new ServiceCollection().AddSingleton(system).BuildServiceProvider();
|
||||
|
||||
Assert.False(new AkkaActiveNodeGate(provider).IsActiveNode);
|
||||
}
|
||||
finally
|
||||
{
|
||||
await system.Terminate();
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ClusteredButNeverJoined_IsUnhealthy_NobodyIsActive()
|
||||
{
|
||||
// A cold start: the cluster provider is up but no member has reached Up. Nobody is active, so
|
||||
// nobody may claim to be — this is the one case that must NOT be Degraded, because a cluster
|
||||
// that never forms would otherwise sit at 200 forever.
|
||||
var config = ConfigurationFactory.ParseString("""
|
||||
akka {
|
||||
loglevel = "WARNING"
|
||||
actor.provider = cluster
|
||||
remote.dot-netty.tcp { hostname = "127.0.0.1", port = 0 }
|
||||
cluster.seed-nodes = []
|
||||
}
|
||||
""");
|
||||
var system = ActorSystem.Create("active-node-never-joined", config);
|
||||
try
|
||||
{
|
||||
var provider = new ServiceCollection().AddSingleton(system).BuildServiceProvider();
|
||||
var check = new ActiveNodeHealthCheck(provider);
|
||||
|
||||
var result = await check.CheckHealthAsync(NewContext(check));
|
||||
|
||||
Assert.Equal(HealthStatus.Unhealthy, result.Status);
|
||||
Assert.Contains("No Up member", result.Description);
|
||||
Assert.False(new AkkaActiveNodeGate(provider).IsActiveNode);
|
||||
}
|
||||
finally
|
||||
{
|
||||
await system.Terminate();
|
||||
}
|
||||
}
|
||||
|
||||
private static HealthCheckContext NewContext(IHealthCheck check) => new()
|
||||
{
|
||||
Registration = new HealthCheckRegistration("active-node", check, HealthStatus.Unhealthy, tags: null),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
using ZB.MOM.WW.Health.Akka;
|
||||
|
||||
namespace ZB.MOM.WW.Health.Akka.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Covers <see cref="ClusterActiveNode.ResolveActiveRole"/> — which role a node competes for. Pure,
|
||||
/// so no cluster is needed; the age-ordering half of the rule is pinned in
|
||||
/// <see cref="ClusterActiveNodeTests"/> against a real two-node cluster.
|
||||
/// </summary>
|
||||
public sealed class ActiveRoleResolutionTests
|
||||
{
|
||||
[Fact]
|
||||
public void EmptyPreference_IsUnscoped()
|
||||
{
|
||||
// Unscoped: every Up member competes for one active slot (the ScadaBridge central pattern).
|
||||
Assert.Null(ClusterActiveNode.ResolveActiveRole(["admin", "driver"], []));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FusedNode_AnswersForTheHighestPrecedenceRoleItCarries()
|
||||
{
|
||||
Assert.Equal(
|
||||
"admin",
|
||||
ClusterActiveNode.ResolveActiveRole(["admin", "driver", "cluster-MAIN"], ["admin", "driver"]));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SingleRoleNode_AnswersForThatRole()
|
||||
{
|
||||
// The regression the OtOpcUa fix existed for: under the old role-filtered check every
|
||||
// driver-only node reported Healthy — "or not a role member" — so all of them called
|
||||
// themselves active and no consumer could find the active node of a site cluster. lmxopcua#494.
|
||||
Assert.Equal(
|
||||
"driver",
|
||||
ClusterActiveNode.ResolveActiveRole(["driver", "cluster-SITE-A"], ["admin", "driver"]));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void PreferenceOrderWins_NotTheOrderTheNodeListsItsRoles()
|
||||
{
|
||||
// Self-role order is incidental (it comes from config parsing); precedence must come from the
|
||||
// caller's list alone, or a fused node's answer would depend on how its roles were spelled.
|
||||
Assert.Equal(
|
||||
"admin",
|
||||
ClusterActiveNode.ResolveActiveRole(["driver", "admin"], ["admin", "driver"]));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void NodeCarryingNoneOfThePreferredRoles_ResolvesToNull()
|
||||
{
|
||||
// Reachable, not merely defensive: OtOpcUa's RoleParser admits a dev-only or cluster-role-only
|
||||
// node. Such a node owns no active work — the health check maps this to Unhealthy so it stays
|
||||
// out of the active pool.
|
||||
Assert.Null(ClusterActiveNode.ResolveActiveRole(["dev", "cluster-MAIN"], ["admin", "driver"]));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster;
|
||||
using Akka.TestKit.Xunit2;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Diagnostics.HealthChecks;
|
||||
using ZB.MOM.WW.Health.Akka;
|
||||
|
||||
namespace ZB.MOM.WW.Health.Akka.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Covers the cluster-view <c>data</c> dictionary that <see cref="AkkaClusterHealthCheck"/> publishes
|
||||
/// from 0.2.0 on (consumed by the family overview dashboard to show each cluster's current leader).
|
||||
/// Uses a real single-node self-joined cluster via <see cref="TestKit"/> — the values come off
|
||||
/// <see cref="Cluster"/>, which cannot be faked, so a formed cluster is the only honest fixture.
|
||||
/// </summary>
|
||||
public sealed class AkkaClusterDataTests : TestKit
|
||||
{
|
||||
private const string ClusterConfig = """
|
||||
akka {
|
||||
actor.provider = cluster
|
||||
remote.dot-netty.tcp {
|
||||
hostname = "127.0.0.1"
|
||||
port = 0
|
||||
}
|
||||
cluster {
|
||||
roles = ["driver", "admin"]
|
||||
# Keep the single-node self-join deterministic and fast in-test.
|
||||
min-nr-of-members = 1
|
||||
}
|
||||
}
|
||||
""";
|
||||
|
||||
private static readonly TimeSpan FormationTimeout = TimeSpan.FromSeconds(30);
|
||||
|
||||
public AkkaClusterDataTests()
|
||||
: base(ClusterConfig, nameof(AkkaClusterDataTests))
|
||||
{
|
||||
}
|
||||
|
||||
/// <summary>Self-joins the test node and waits until it is Up with a converged leader.</summary>
|
||||
private Cluster FormSingleNodeCluster()
|
||||
{
|
||||
var cluster = Cluster.Get(Sys);
|
||||
cluster.Join(cluster.SelfAddress);
|
||||
AwaitCondition(
|
||||
() => cluster.SelfMember.Status == MemberStatus.Up && cluster.State.Leader is not null,
|
||||
FormationTimeout);
|
||||
return cluster;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BuildClusterData_SingleNodeCluster_ReportsSelfAsLeader()
|
||||
{
|
||||
var cluster = FormSingleNodeCluster();
|
||||
|
||||
var data = AkkaClusterHealthCheck.BuildClusterData(cluster);
|
||||
|
||||
Assert.Equal(cluster.SelfAddress.ToString(), Assert.IsType<string>(data["leader"]));
|
||||
Assert.Equal(cluster.SelfAddress.ToString(), Assert.IsType<string>(data["selfAddress"]));
|
||||
Assert.Equal(1, Assert.IsType<int>(data["memberCount"]));
|
||||
Assert.Equal(0, Assert.IsType<int>(data["unreachableCount"]));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BuildClusterData_RolesRoundTripSorted()
|
||||
{
|
||||
var cluster = FormSingleNodeCluster();
|
||||
|
||||
var roles = Assert.IsType<string[]>(AkkaClusterHealthCheck.BuildClusterData(cluster)["selfRoles"]);
|
||||
|
||||
// Sorted, so the payload is stable across restarts; the configured roles survive verbatim.
|
||||
Assert.Equal(new[] { "admin", "driver" }, roles);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task CheckHealth_FormedCluster_HealthyResultCarriesClusterData()
|
||||
{
|
||||
var cluster = FormSingleNodeCluster();
|
||||
var provider = new ServiceCollection().AddSingleton(Sys).BuildServiceProvider();
|
||||
var check = new AkkaClusterHealthCheck(provider, AkkaClusterStatusPolicy.Default);
|
||||
|
||||
var result = await check.CheckHealthAsync(NewContext(check));
|
||||
|
||||
Assert.Equal(HealthStatus.Healthy, result.Status);
|
||||
Assert.Equal(cluster.SelfAddress.ToString(), result.Data["leader"]);
|
||||
Assert.Equal(1, result.Data["memberCount"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task CheckHealth_NoActorSystem_CarriesNoData()
|
||||
{
|
||||
// The startup-safety paths stay description-only: an empty data dictionary is what makes the
|
||||
// writer omit the "data" key entirely, so a not-yet-started node's payload is unchanged.
|
||||
var provider = new ServiceCollection().BuildServiceProvider();
|
||||
var check = new AkkaClusterHealthCheck(provider, AkkaClusterStatusPolicy.Default);
|
||||
|
||||
var result = await check.CheckHealthAsync(NewContext(check));
|
||||
|
||||
Assert.Equal(HealthStatus.Degraded, result.Status);
|
||||
Assert.Empty(result.Data);
|
||||
}
|
||||
|
||||
private static HealthCheckContext NewContext(IHealthCheck check) => new()
|
||||
{
|
||||
Registration = new HealthCheckRegistration("akka-cluster", check, HealthStatus.Unhealthy, tags: null),
|
||||
};
|
||||
}
|
||||
@@ -88,6 +88,8 @@ public sealed class AkkaClusterStatusPolicyTests
|
||||
var result = await check.CheckHealthAsync(NewContext(check));
|
||||
|
||||
Assert.Equal(HealthStatus.Degraded, result.Status);
|
||||
// No cluster to describe → no data, so the emitted payload stays the pre-0.2.0 shape.
|
||||
Assert.Empty(result.Data);
|
||||
}
|
||||
finally
|
||||
{
|
||||
|
||||
@@ -0,0 +1,237 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster;
|
||||
using Akka.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Diagnostics.HealthChecks;
|
||||
using ZB.MOM.WW.Health.Akka;
|
||||
|
||||
namespace ZB.MOM.WW.Health.Akka.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Pins <b>which</b> node the active tier names, against a real two-node cluster deliberately built
|
||||
/// so the oldest member is <i>not</i> the lowest-addressed one.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The divergence is the whole point. Akka offers two different "first" members:
|
||||
/// <c>ClusterState.Leader</c> / <c>RoleLeader</c> is the <b>lowest-addressed</b> Up member
|
||||
/// (host, then port — nothing to do with time), while <b>oldest</b> is the member with the
|
||||
/// lowest up-number, which is what <c>ClusterSingletonManager</c> uses to place singletons.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// They coincide on a freshly-formed cluster, which is why every single-node and happy-path
|
||||
/// test missed it and why the leader-based check survived to 0.2.1. They diverge the moment a
|
||||
/// node restarts: the restarted node rejoins as the youngest while keeping its address, so if
|
||||
/// it holds the lower address it becomes leader while the singletons — and every piece of work
|
||||
/// they own — stay on the other node. This fixture reproduces that state directly instead of
|
||||
/// waiting for a restart.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class ClusterActiveNodeTests : IAsyncLifetime
|
||||
{
|
||||
// Fixed, unusual ports: the test needs a deterministic address ordering, which port 0 cannot give.
|
||||
// Distinct from OtOpcUa's equivalent fixture (19_530/19_531) so the two suites can run at once.
|
||||
private const int OldestPort = 19_541; // joins FIRST -> oldest, but the HIGHER address
|
||||
private const int YoungestPort = 19_540; // joins SECOND -> cluster leader, the LOWER address
|
||||
|
||||
private const string SystemName = "health-active-node";
|
||||
|
||||
private ActorSystem? _oldest;
|
||||
private ActorSystem? _youngest;
|
||||
|
||||
private static Config NodeConfig(int port, string roles) => ConfigurationFactory.ParseString($$"""
|
||||
akka {
|
||||
loglevel = "WARNING"
|
||||
actor.provider = "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
|
||||
remote.dot-netty.tcp {
|
||||
hostname = "127.0.0.1"
|
||||
port = {{port}}
|
||||
}
|
||||
cluster {
|
||||
seed-nodes = ["akka.tcp://{{SystemName}}@127.0.0.1:{{OldestPort}}"]
|
||||
roles = [{{roles}}]
|
||||
min-nr-of-members = 1
|
||||
run-coordinated-shutdown-when-down = off
|
||||
downing-provider-class = ""
|
||||
}
|
||||
}
|
||||
""");
|
||||
|
||||
public async Task InitializeAsync()
|
||||
{
|
||||
// Order matters: the seed forms the cluster and is therefore the oldest member. The oldest
|
||||
// node is fused (admin + driver); the younger one is driver-only, so the role-scoped
|
||||
// assertions below can distinguish "oldest of this role" from "oldest overall".
|
||||
_oldest = ActorSystem.Create(SystemName, NodeConfig(OldestPort, "\"admin\", \"driver\""));
|
||||
await WaitForUpAsync(_oldest, expectedMembers: 1);
|
||||
|
||||
_youngest = ActorSystem.Create(SystemName, NodeConfig(YoungestPort, "\"driver\""));
|
||||
await WaitForUpAsync(_oldest, expectedMembers: 2);
|
||||
await WaitForUpAsync(_youngest, expectedMembers: 2);
|
||||
}
|
||||
|
||||
public async Task DisposeAsync()
|
||||
{
|
||||
if (_youngest is not null) await _youngest.Terminate();
|
||||
if (_oldest is not null) await _oldest.Terminate();
|
||||
}
|
||||
|
||||
private static async Task WaitForUpAsync(ActorSystem system, int expectedMembers)
|
||||
{
|
||||
var cluster = Cluster.Get(system);
|
||||
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30);
|
||||
while (DateTime.UtcNow < deadline)
|
||||
{
|
||||
if (cluster.State.Members.Count(m => m.Status == MemberStatus.Up) >= expectedMembers) return;
|
||||
await Task.Delay(100);
|
||||
}
|
||||
|
||||
throw new TimeoutException(
|
||||
$"cluster on port {cluster.SelfAddress.Port} never saw {expectedMembers} Up members "
|
||||
+ $"(saw {cluster.State.Members.Count(m => m.Status == MemberStatus.Up)})");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Sanity check on the fixture itself: if the two orderings did not actually diverge, every
|
||||
/// assertion below would pass for the wrong reason.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Fixture_actually_produces_divergent_age_and_address_ordering()
|
||||
{
|
||||
var state = Cluster.Get(_oldest!).State;
|
||||
|
||||
Assert.Equal(YoungestPort, state.Leader!.Port);
|
||||
Assert.Equal(YoungestPort, state.RoleLeader("driver")!.Port);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void OldestUpMember_IsTheOldest_NotTheLeader()
|
||||
{
|
||||
var members = Cluster.Get(_oldest!).State.Members;
|
||||
|
||||
Assert.Equal(OldestPort, ClusterActiveNode.OldestUpMember(members)!.Address.Port);
|
||||
Assert.Equal(OldestPort, ClusterActiveNode.OldestUpMember(members, "driver")!.Address.Port);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void OldestUpMember_IsRoleScoped()
|
||||
{
|
||||
var members = Cluster.Get(_oldest!).State.Members;
|
||||
|
||||
// Only the fused node carries admin, so it owns the admin slot even though it is not the
|
||||
// lowest-addressed member.
|
||||
Assert.Equal(OldestPort, ClusterActiveNode.OldestUpMember(members, "admin")!.Address.Port);
|
||||
|
||||
// A role nobody carries has no owner.
|
||||
Assert.Null(ClusterActiveNode.OldestUpMember(members, "no-such-role"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SelfIsActive_IsTrueOnlyOnTheOldestNode()
|
||||
{
|
||||
Assert.True(ClusterActiveNode.SelfIsActive(Cluster.Get(_oldest!)));
|
||||
Assert.False(ClusterActiveNode.SelfIsActive(Cluster.Get(_youngest!)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SelfIsActive_RoleScoped_YoungestOwnsNoSlot()
|
||||
{
|
||||
// Driver-scoped: the fused node is still the oldest driver, so the driver-only node is standby.
|
||||
Assert.True(ClusterActiveNode.SelfIsActive(Cluster.Get(_oldest!), "driver"));
|
||||
Assert.False(ClusterActiveNode.SelfIsActive(Cluster.Get(_youngest!), "driver"));
|
||||
|
||||
// Admin-scoped: the driver-only node does not carry admin, so it can never be the admin active
|
||||
// node — even though it is the cluster leader.
|
||||
Assert.False(ClusterActiveNode.SelfIsActive(Cluster.Get(_youngest!), "admin"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task HealthCheck_IsHealthyOnTheOldestAndUnhealthyOnTheLeader()
|
||||
{
|
||||
var onOldest = await CheckAsync(_oldest!, new ActiveNodeHealthCheckOptions());
|
||||
var onYoungest = await CheckAsync(_youngest!, new ActiveNodeHealthCheckOptions());
|
||||
|
||||
Assert.Equal(HealthStatus.Healthy, onOldest.Status);
|
||||
|
||||
// The node the old leader-based check would have called active. Exactly one 200 per cluster is
|
||||
// the property an orchestrator's active-tier routing depends on.
|
||||
Assert.Equal(HealthStatus.Unhealthy, onYoungest.Status);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task HealthCheck_DataNamesTheActiveNode_EvenFromAStandby()
|
||||
{
|
||||
// A standby reporting *who* is active is what lets a dashboard render the pair from either
|
||||
// node's payload alone (the 0.2.0 per-entry data object).
|
||||
var result = await CheckAsync(_youngest!, new ActiveNodeHealthCheckOptions());
|
||||
|
||||
Assert.Equal($"akka.tcp://{SystemName}@127.0.0.1:{OldestPort}", result.Data["activeNode"]);
|
||||
Assert.Equal($"akka.tcp://{SystemName}@127.0.0.1:{YoungestPort}", result.Data["selfAddress"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task HealthCheck_RolePreference_FusedNodeAnswersForAdmin()
|
||||
{
|
||||
var options = new ActiveNodeHealthCheckOptions { RolePreference = ["admin", "driver"] };
|
||||
|
||||
var onOldest = await CheckAsync(_oldest!, options);
|
||||
var onYoungest = await CheckAsync(_youngest!, options);
|
||||
|
||||
Assert.Equal(HealthStatus.Healthy, onOldest.Status);
|
||||
Assert.Equal("admin", onOldest.Data["activeRole"]);
|
||||
|
||||
// Resolves to driver (it has no admin role) and loses to the older fused node.
|
||||
Assert.Equal(HealthStatus.Unhealthy, onYoungest.Status);
|
||||
Assert.Equal("driver", onYoungest.Data["activeRole"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task HealthCheck_NodeOwningNoneOfThePreferredRoles_IsUnhealthyByDefault()
|
||||
{
|
||||
var options = new ActiveNodeHealthCheckOptions { RolePreference = ["no-such-role"] };
|
||||
|
||||
var result = await CheckAsync(_oldest!, options);
|
||||
|
||||
// Not "not applicable ⇒ Healthy": a node that owns no active work must stay out of the pool.
|
||||
Assert.Equal(HealthStatus.Unhealthy, result.Status);
|
||||
Assert.Contains("carries none of the active roles", result.Description);
|
||||
Assert.DoesNotContain("activeRole", result.Data.Keys);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task HealthCheck_NoActiveRoleStatus_IsOverridable()
|
||||
{
|
||||
var options = new ActiveNodeHealthCheckOptions
|
||||
{
|
||||
RolePreference = ["no-such-role"],
|
||||
NoActiveRoleStatus = HealthStatus.Healthy,
|
||||
};
|
||||
|
||||
Assert.Equal(HealthStatus.Healthy, (await CheckAsync(_oldest!, options)).Status);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Gate_IsActiveOnlyOnTheOldestNode()
|
||||
{
|
||||
Assert.True(new AkkaActiveNodeGate(ProviderFor(_oldest!)).IsActiveNode);
|
||||
Assert.False(new AkkaActiveNodeGate(ProviderFor(_youngest!)).IsActiveNode);
|
||||
|
||||
// Role-scoped: the driver-only node cannot gate admin work.
|
||||
Assert.False(new AkkaActiveNodeGate(ProviderFor(_youngest!), "admin").IsActiveNode);
|
||||
}
|
||||
|
||||
private static IServiceProvider ProviderFor(ActorSystem system) =>
|
||||
new ServiceCollection().AddSingleton(system).BuildServiceProvider();
|
||||
|
||||
private static async Task<HealthCheckResult> CheckAsync(
|
||||
ActorSystem system,
|
||||
ActiveNodeHealthCheckOptions options)
|
||||
{
|
||||
var check = new ActiveNodeHealthCheck(ProviderFor(system), options);
|
||||
var context = new HealthCheckContext
|
||||
{
|
||||
Registration = new HealthCheckRegistration("active-node", check, HealthStatus.Unhealthy, tags: null),
|
||||
};
|
||||
return await check.CheckHealthAsync(context);
|
||||
}
|
||||
}
|
||||
@@ -5,6 +5,7 @@
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Akka.TestKit.Xunit2" />
|
||||
<PackageReference Include="coverlet.collector" />
|
||||
<PackageReference Include="Microsoft.NET.Test.Sdk" />
|
||||
<PackageReference Include="xunit" />
|
||||
|
||||
@@ -22,8 +22,11 @@ public sealed class ResponseWriterTests
|
||||
{
|
||||
private readonly HealthCheckResult _result;
|
||||
|
||||
public StubHealthCheck(HealthStatus status, string? description = null) =>
|
||||
_result = new HealthCheckResult(status, description);
|
||||
public StubHealthCheck(
|
||||
HealthStatus status,
|
||||
string? description = null,
|
||||
IReadOnlyDictionary<string, object>? data = null) =>
|
||||
_result = new HealthCheckResult(status, description, exception: null, data: data);
|
||||
|
||||
public Task<HealthCheckResult> CheckHealthAsync(
|
||||
HealthCheckContext context,
|
||||
@@ -31,12 +34,14 @@ public sealed class ResponseWriterTests
|
||||
}
|
||||
|
||||
private static async Task<HttpResponseMessage> GetReadyAsync(
|
||||
HealthStatus status, string? description = "db reachable")
|
||||
HealthStatus status,
|
||||
string? description = "db reachable",
|
||||
IReadOnlyDictionary<string, object>? data = null)
|
||||
{
|
||||
var builder = WebApplication.CreateBuilder();
|
||||
builder.WebHost.UseTestServer();
|
||||
builder.Services.AddHealthChecks()
|
||||
.AddCheck("db", new StubHealthCheck(status, description), tags: new[] { ZbHealthTags.Ready });
|
||||
.AddCheck("db", new StubHealthCheck(status, description, data), tags: new[] { ZbHealthTags.Ready });
|
||||
|
||||
await using var app = builder.Build();
|
||||
app.MapZbHealth();
|
||||
@@ -84,6 +89,51 @@ public sealed class ResponseWriterTests
|
||||
Assert.Equal(JsonValueKind.Null, description.ValueKind);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ReadyEndpoint_EntryWithData_EmitsDataObject_WithVerbatimKeys()
|
||||
{
|
||||
// The envelope properties are camelCased by the naming policy, but a check's own data keys are
|
||||
// dictionary keys and must survive verbatim — the dashboard reads them by exact name
|
||||
// (entries["akka-cluster"].data.leader), so a naming policy applied here would silently break
|
||||
// the contract. Values may be any JSON-friendly type (string / int / string[]).
|
||||
var data = new Dictionary<string, object>
|
||||
{
|
||||
["leader"] = "akka.tcp://sys@node-1:4053",
|
||||
["memberCount"] = 2,
|
||||
["selfRoles"] = new[] { "driver", "site-a" },
|
||||
};
|
||||
|
||||
var response = await GetReadyAsync(HealthStatus.Healthy, data: data);
|
||||
|
||||
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
|
||||
|
||||
using var doc = JsonDocument.Parse(await response.Content.ReadAsStringAsync());
|
||||
var entryData = doc.RootElement.GetProperty("entries").GetProperty("db").GetProperty("data");
|
||||
|
||||
Assert.Equal("akka.tcp://sys@node-1:4053", entryData.GetProperty("leader").GetString());
|
||||
Assert.Equal(2, entryData.GetProperty("memberCount").GetInt32());
|
||||
Assert.Equal(
|
||||
new[] { "driver", "site-a" },
|
||||
entryData.GetProperty("selfRoles").EnumerateArray().Select(static e => e.GetString()).ToArray());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ReadyEndpoint_EntryWithoutData_OmitsDataKeyEntirely()
|
||||
{
|
||||
// The 0.2.0 non-breaking invariant: a check that publishes no data must produce exactly the
|
||||
// pre-0.2.0 body. Asserted on the raw JSON string so a `"data": null` regression (which
|
||||
// TryGetProperty would still accept as "present") fails here.
|
||||
var response = await GetReadyAsync(HealthStatus.Healthy);
|
||||
|
||||
var body = await response.Content.ReadAsStringAsync();
|
||||
|
||||
Assert.DoesNotContain("\"data\"", body, StringComparison.Ordinal);
|
||||
|
||||
using var doc = JsonDocument.Parse(body);
|
||||
var db = doc.RootElement.GetProperty("entries").GetProperty("db");
|
||||
Assert.False(db.TryGetProperty("data", out _), "data key must be absent when the check set none");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ReadyEndpoint_Degraded_Returns200_WithDegradedStatus()
|
||||
{
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
# Host-side publish output copied into the runtime-only Docker image.
|
||||
docker/publish/
|
||||
|
||||
# Serilog rolling file sink, written relative to the content root.
|
||||
logs/
|
||||
@@ -0,0 +1,32 @@
|
||||
<Project>
|
||||
|
||||
<!--
|
||||
Workspace-wide build settings for ZB.MOM.WW.Overview (the family overview dashboard).
|
||||
Mirrors the HistorianGateway workspace props — the reference app layout for this family.
|
||||
-->
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<LangVersion>latest</LangVersion>
|
||||
</PropertyGroup>
|
||||
|
||||
<!--
|
||||
Build-quality floor: the family's zero-warnings invariant is a build GATE, not a convention —
|
||||
new warnings break the build and are fixed rather than suppressed.
|
||||
-->
|
||||
<PropertyGroup>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
<AnalysisLevel>latest</AnalysisLevel>
|
||||
<EnforceCodeStyleInBuild>true</EnforceCodeStyleInBuild>
|
||||
<Deterministic>true</Deterministic>
|
||||
</PropertyGroup>
|
||||
|
||||
<!--
|
||||
This is an APP, not a library: nothing here is ever packed to the Gitea feed.
|
||||
-->
|
||||
<PropertyGroup>
|
||||
<IsPackable>false</IsPackable>
|
||||
</PropertyGroup>
|
||||
|
||||
</Project>
|
||||
@@ -0,0 +1,8 @@
|
||||
<Solution>
|
||||
<Folder Name="/src/">
|
||||
<Project Path="src/ZB.MOM.WW.Overview/ZB.MOM.WW.Overview.csproj" />
|
||||
</Folder>
|
||||
<Folder Name="/tests/">
|
||||
<Project Path="tests/ZB.MOM.WW.Overview.Tests/ZB.MOM.WW.Overview.Tests.csproj" />
|
||||
</Folder>
|
||||
</Solution>
|
||||
@@ -0,0 +1,2 @@
|
||||
**/*.md
|
||||
**/.DS_Store
|
||||
@@ -0,0 +1,29 @@
|
||||
# Runtime-only image for the ZB.MOM.WW.Overview dashboard.
|
||||
#
|
||||
# The app is published FRAMEWORK-DEPENDENT on the host (which holds the Gitea NuGet feed
|
||||
# credentials in ~/.nuget) into ./publish, then copied in here. Docker performs NO restore and
|
||||
# NO build, so the authenticated Gitea feed is never needed at image-build time. Regenerate
|
||||
# ./publish with:
|
||||
# dotnet publish src/ZB.MOM.WW.Overview/ZB.MOM.WW.Overview.csproj \
|
||||
# -c Release -o docker/publish -p:UseAppHost=false
|
||||
#
|
||||
# The published IL is architecture-neutral, so the multi-arch aspnet:10.0 base runs as-is on
|
||||
# both Apple Silicon and amd64 hosts.
|
||||
FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS runtime
|
||||
WORKDIR /app
|
||||
|
||||
# The base image declares its own binding — as ASPNETCORE_HTTP_PORTS=8080 on .NET 8+, NOT the
|
||||
# ASPNETCORE_URLS everyone reaches for first (clearing only URLS leaves the warning in place, which
|
||||
# is a good way to spend an afternoon). Left set, it collides with the Kestrel endpoint section in
|
||||
# appsettings.Docker.json: Kestrel wins, but logs "Overriding address(es) 'http://*:8080'" on every
|
||||
# boot — the family's URLs-override trap arriving by default rather than by mistake. Clearing both
|
||||
# at the layer that introduced them leaves the Kestrel section as the single authority.
|
||||
ENV ASPNETCORE_URLS= \
|
||||
ASPNETCORE_HTTP_PORTS=
|
||||
|
||||
COPY publish/ ./
|
||||
|
||||
# Dashboard + /health/* + /metrics, all HTTP/1.1 and all anonymous.
|
||||
EXPOSE 5320
|
||||
|
||||
ENTRYPOINT ["dotnet", "ZB.MOM.WW.Overview.dll"]
|
||||
@@ -0,0 +1,81 @@
|
||||
# `docker/` — containerised overview dashboard
|
||||
|
||||
Runtime-only image, the HistorianGateway pattern: publish on the host (which has the Gitea feed
|
||||
credentials), copy the output into `mcr.microsoft.com/dotnet/aspnet:10.0`. Docker never restores.
|
||||
|
||||
```bash
|
||||
# from ZB.MOM.WW.Overview/
|
||||
dotnet publish src/ZB.MOM.WW.Overview/ZB.MOM.WW.Overview.csproj \
|
||||
-c Release -o docker/publish -p:UseAppHost=false
|
||||
|
||||
cd docker && docker compose up -d --build
|
||||
open http://localhost:5320
|
||||
```
|
||||
|
||||
`docker/publish/` is build output and is gitignored.
|
||||
|
||||
## Why the container, and not just `dotnet run`
|
||||
|
||||
The dev rigs publish only some node ports to the host:
|
||||
|
||||
| Fleet | Health endpoint | Reachable from the Mac host? |
|
||||
|---|---|---|
|
||||
| ScadaBridge central-a/b | container `:5000` | yes — `localhost:9001` / `:9002` |
|
||||
| ScadaBridge site-\* (6) | container `:8084` | **no** — not published |
|
||||
| OtOpcUa central-1/2 | container `:9000` | **no** — only Traefik `:9200`, load-balanced across the pair |
|
||||
| OtOpcUa site-\* (4) | container `:8080` | **no** — not published |
|
||||
| HistorianGateway | container `:5220` | yes — `localhost:5220` |
|
||||
| MxAccessGateway | windev `:5130` | yes, with the VPN up |
|
||||
|
||||
So a host-run dashboard can only see part of the fleet. This stack joins the three rig networks
|
||||
(`otopcua-dev_default`, `scadabridge-net`, `zb-historiangw_default`, all declared `external`) and
|
||||
addresses every node by container DNS name, which is how it sees all of them.
|
||||
|
||||
Ports above were established by probing the running rigs on 2026-07-24, **not** read off the
|
||||
compose files — the two disagree. In particular OtOpcUa's site nodes serve health on `:8080`, not
|
||||
the `:9000` their compose anchor implies, because their explicit Kestrel listeners for LocalDb sync
|
||||
re-bind the primary HTTP port.
|
||||
|
||||
Start the rigs before this stack. The networks are `external`, so bringing the dashboard up or
|
||||
down never disturbs a running rig.
|
||||
|
||||
## Registry shape
|
||||
|
||||
Bound from the `Overview` section; `appsettings.Docker.json` is the worked example.
|
||||
|
||||
```jsonc
|
||||
"Overview": {
|
||||
"PollIntervalSeconds": 10, // sweep cadence; per-app and per-instance overrides allowed
|
||||
"TimeoutSeconds": 3, // per-request timeout
|
||||
"StaleAfterSeconds": 45, // must exceed the poll interval, or every card reads Stale
|
||||
"Applications": [
|
||||
{
|
||||
"Name": "ScadaBridge",
|
||||
"ManagementLabel": "CentralUI", // label on each instance's management link
|
||||
"Instances": [
|
||||
{
|
||||
"Name": "central-a",
|
||||
"Group": "central", // one Akka cluster = one group; leader + split-brain
|
||||
// detection are scoped to it. Omit for standalone apps.
|
||||
"BaseUrl": "http://host:5000", // /health/ready and /health/active are probed under this
|
||||
"ManagementUrl": "http://localhost:9001/", // followed by the OPERATOR'S browser, so it
|
||||
// must be host-reachable, not container DNS
|
||||
"HasActiveRole": true // false for single-instance gateways: no Active badge,
|
||||
// and /health/active is never probed
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`appsettings.json` ships the tuning defaults with an **empty** `Applications` list, so a deployment
|
||||
that supplies no registry fails at startup naming the missing key rather than booting into an empty
|
||||
dashboard.
|
||||
|
||||
## Configuration traps
|
||||
|
||||
- **Never set `ASPNETCORE_URLS` alongside `Kestrel__Endpoints__*`.** Declaring both puts Kestrel
|
||||
into explicit-endpoints mode and one of them is silently discarded.
|
||||
- No `env_file`, no secrets, no login. The dashboard reads anonymous health endpoints and holds no
|
||||
credentials. That is a requirement of the design, not an omission.
|
||||
@@ -0,0 +1,54 @@
|
||||
# Local Docker deployment of the SCADA family overview dashboard.
|
||||
#
|
||||
# Why this exists rather than just `dotnet run`: the dev rigs publish only some node ports to the
|
||||
# host. OtOpcUa publishes no per-node HTTP port at all (only Traefik's load-balanced :9200, which
|
||||
# round-robins the central pair and so cannot identify a node), and ScadaBridge's site nodes keep
|
||||
# their health port container-internal. A dashboard on the host can therefore see a fraction of the
|
||||
# fleet; one joined to the rig networks sees all of it, addressing each node by container DNS name.
|
||||
#
|
||||
# Registry: appsettings.Docker.json, baked into the image (ASPNETCORE_ENVIRONMENT=Docker below).
|
||||
#
|
||||
# No env_file and no secrets: the dashboard reads anonymous /health endpoints, holds no
|
||||
# credentials, and serves no login. That is a requirement, not an oversight.
|
||||
#
|
||||
# Explicit project name so this stack is isolated from the other compose projects on the host.
|
||||
name: zb-overview
|
||||
|
||||
services:
|
||||
overview:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
image: zb-overview:local
|
||||
container_name: zb-overview
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "5320:5320" # dashboard + /health/* + /metrics
|
||||
environment:
|
||||
ASPNETCORE_ENVIRONMENT: Docker
|
||||
|
||||
# The endpoint lives in appsettings.Docker.json — ONE authority for the binding. Do NOT add
|
||||
# ASPNETCORE_URLS here; the Dockerfile clears the base image's default for that reason.
|
||||
|
||||
# No container healthcheck on purpose. aspnet:10.0 ships neither curl nor wget, and adding a
|
||||
# `--healthcheck` mode to the app to work around that would put a second entry point into
|
||||
# production code purely to satisfy Compose. The app's real health surface is HTTP and already
|
||||
# honest — `curl localhost:5320/health/ready` from the host, or scrape /metrics.
|
||||
|
||||
networks:
|
||||
- otopcua
|
||||
- scadabridge
|
||||
- historiangw
|
||||
|
||||
# All three are created by the rigs themselves; this stack joins them and never defines them, so
|
||||
# bringing the dashboard up or down cannot disturb a running rig. Start the rigs first.
|
||||
networks:
|
||||
otopcua:
|
||||
name: otopcua-dev_default
|
||||
external: true
|
||||
scadabridge:
|
||||
name: scadabridge-net
|
||||
external: true
|
||||
historiangw:
|
||||
name: zb-historiangw_default
|
||||
external: true
|
||||
@@ -0,0 +1,24 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<configuration>
|
||||
<packageSources>
|
||||
<clear />
|
||||
<add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
|
||||
<add key="dohertj2-gitea" value="https://gitea.dohertylan.com/api/packages/dohertj2/nuget/index.json" />
|
||||
</packageSources>
|
||||
<!-- nuget.org serves everything; the Gitea feed serves only the ZB.MOM.WW.* shared libs.
|
||||
Credentials are NOT committed: they are provided per-developer at the user level
|
||||
(~/.nuget/NuGet/NuGet.Config). -->
|
||||
<packageSourceMapping>
|
||||
<packageSource key="nuget.org">
|
||||
<package pattern="*" />
|
||||
</packageSource>
|
||||
<packageSource key="dohertj2-gitea">
|
||||
<package pattern="ZB.MOM.WW.Health" />
|
||||
<package pattern="ZB.MOM.WW.Health.*" />
|
||||
<package pattern="ZB.MOM.WW.Telemetry" />
|
||||
<package pattern="ZB.MOM.WW.Telemetry.*" />
|
||||
<package pattern="ZB.MOM.WW.Configuration" />
|
||||
<package pattern="ZB.MOM.WW.Theme" />
|
||||
</packageSource>
|
||||
</packageSourceMapping>
|
||||
</configuration>
|
||||
@@ -0,0 +1,22 @@
|
||||
@* Host page. Head order matters: Bootstrap first (the Theme kit's shell and
|
||||
TechButton are built on Bootstrap utility classes), then <ThemeHead/> which
|
||||
brings the design tokens and IBM Plex, then site.css so this app's own
|
||||
dashboard anatomy can override either. *@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<base href="/" />
|
||||
<link rel="stylesheet" href="/lib/bootstrap/css/bootstrap.min.css" />
|
||||
<ThemeHead />
|
||||
<link rel="stylesheet" href="/css/site.css" />
|
||||
<HeadOutlet @rendermode="InteractiveServer" />
|
||||
</head>
|
||||
<body>
|
||||
<Routes @rendermode="InteractiveServer" />
|
||||
<script src="/lib/bootstrap/js/bootstrap.bundle.min.js"></script>
|
||||
<ThemeScripts />
|
||||
<script src="/_framework/blazor.web.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,43 @@
|
||||
@inherits LayoutComponentBase
|
||||
@inject OverviewSnapshotStore Store
|
||||
|
||||
@* Thin layout: the side-rail chassis belongs to the shared kit's <ThemeShell>.
|
||||
No AuthorizeView anywhere — this dashboard is anonymous by requirement, so
|
||||
there is no user to show and nothing to sign out of. The rail's application
|
||||
links are fragment anchors into the single page rather than routes: the whole
|
||||
fleet is one view, and splitting it per application would defeat the point. *@
|
||||
<ThemeShell Product="SCADA Overview" Accent="#6d28d9">
|
||||
<Nav>
|
||||
<NavRailItem Href="/" Text="Overview" Match="NavLinkMatch.All" />
|
||||
@if (Applications.Count > 0)
|
||||
{
|
||||
<NavRailSection Title="Applications" Key="overview-apps">
|
||||
@foreach (var application in Applications)
|
||||
{
|
||||
<a class="rail-link" href="@($"#{Slug(application)}")">@application</a>
|
||||
}
|
||||
</NavRailSection>
|
||||
}
|
||||
</Nav>
|
||||
<RailFooter>
|
||||
<div class="rail-note">read-only · anonymous<br />ZB.MOM.WW.Overview @Version</div>
|
||||
</RailFooter>
|
||||
<ChildContent>@Body</ChildContent>
|
||||
</ThemeShell>
|
||||
|
||||
@code {
|
||||
private static readonly string Version =
|
||||
typeof(MainLayout).Assembly.GetName().Version?.ToString(3) ?? "0.0.0";
|
||||
|
||||
private IReadOnlyList<string> Applications =>
|
||||
Store.Current.Applications.Select(a => a.Name).ToList();
|
||||
|
||||
/// <summary>
|
||||
/// Builds the fragment id for an application section. Must stay identical to the id the
|
||||
/// Overview page emits, or every rail link becomes a dead anchor.
|
||||
/// </summary>
|
||||
/// <param name="name">The application's display name.</param>
|
||||
/// <returns>A URL-fragment-safe slug.</returns>
|
||||
internal static string Slug(string name) =>
|
||||
new(name.Where(c => char.IsLetterOrDigit(c) || c is '-' or '_').Select(char.ToLowerInvariant).ToArray());
|
||||
}
|
||||
@@ -0,0 +1,245 @@
|
||||
@page "/"
|
||||
@implements IDisposable
|
||||
@inject OverviewSnapshotStore Store
|
||||
@inject TimeProvider Time
|
||||
|
||||
@* The whole dashboard, rendered from the snapshot store. This page never polls
|
||||
an instance: it subscribes to the store and re-renders when the poller swaps
|
||||
a snapshot in, plus a slow local tick so ages and staleness advance on their
|
||||
own between sweeps. *@
|
||||
|
||||
@* Read once per render and threaded through everything below, so every card, KPI
|
||||
and group header on one paint agrees on what "now" is. Reading the clock at
|
||||
render — rather than caching it when a snapshot arrives — is what lets the view
|
||||
age on its own: a paused or stuck poller publishes nothing, and the cards still
|
||||
cross into Stale. *@
|
||||
@{
|
||||
var now = Time.GetUtcNow();
|
||||
}
|
||||
|
||||
<PageTitle>SCADA Overview</PageTitle>
|
||||
|
||||
<div class="page-head rise">
|
||||
<h1>Fleet Overview</h1>
|
||||
<span class="page-meta">@Meta</span>
|
||||
<span class="spacer"></span>
|
||||
<span class="conn-pill" data-state="@(_paused ? "disconnected" : "connected")">
|
||||
<span class="dot"></span>@(_paused ? "Paused" : "Polling")
|
||||
</span>
|
||||
<button class="rail-btn" type="button" data-testid="pause-toggle" @onclick="TogglePause">
|
||||
@(_paused ? "Resume auto-refresh" : "Pause auto-refresh")
|
||||
</button>
|
||||
</div>
|
||||
|
||||
@if (_snapshot.Applications.Count == 0)
|
||||
{
|
||||
<p class="page-meta" data-testid="empty-registry">
|
||||
No instances have been polled yet.
|
||||
</p>
|
||||
}
|
||||
else
|
||||
{
|
||||
<div class="agg-grid rise">
|
||||
<div class="agg-card">
|
||||
<div class="agg-label">Instances</div>
|
||||
<div class="agg-value">@_instances.Count<span class="agg-sub">registered</span></div>
|
||||
</div>
|
||||
<div class="agg-card good">
|
||||
<div class="agg-label">Up</div>
|
||||
<div class="agg-value" data-testid="kpi-up">@Count(now, InstanceStatus.Up)</div>
|
||||
</div>
|
||||
<div class="agg-card @(Count(now, InstanceStatus.Degraded) > 0 ? "caution" : null)">
|
||||
<div class="agg-label">Degraded</div>
|
||||
<div class="agg-value" data-testid="kpi-degraded">@Count(now, InstanceStatus.Degraded)</div>
|
||||
</div>
|
||||
<div class="agg-card @(Count(now, InstanceStatus.Down) > 0 ? "alert" : null)">
|
||||
<div class="agg-label">Down</div>
|
||||
<div class="agg-value" data-testid="kpi-down">@Count(now, InstanceStatus.Down)</div>
|
||||
</div>
|
||||
<div class="agg-card @(Count(now, InstanceStatus.Unreachable) > 0 ? "alert" : null)">
|
||||
<div class="agg-label">Unreachable</div>
|
||||
<div class="agg-value" data-testid="kpi-unreachable">@Count(now, InstanceStatus.Unreachable)</div>
|
||||
</div>
|
||||
<div class="agg-card quiet">
|
||||
<div class="agg-label">Stale</div>
|
||||
<div class="agg-value" data-testid="kpi-stale">@StaleCount(now)</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@foreach (var application in _snapshot.Applications)
|
||||
{
|
||||
<section class="app-section rise" id="@MainLayout.Slug(application.Name)" data-testid="application">
|
||||
<div class="app-head">
|
||||
<h2>@application.Name</h2>
|
||||
<StatusPill State="RollupState(application)">@RollupLabel(application)</StatusPill>
|
||||
<span class="spacer"></span>
|
||||
<span class="app-meta">@AppMeta(application)</span>
|
||||
</div>
|
||||
|
||||
@foreach (var group in Layout(application))
|
||||
{
|
||||
<div class="group">
|
||||
@if (group.Key is { } groupName)
|
||||
{
|
||||
var state = application.GroupFor(groupName);
|
||||
<div class="group-head">
|
||||
<span class="group-name">@groupName</span>
|
||||
@if (state?.LeaderInstance is { } leaderInstance)
|
||||
{
|
||||
<StatusPill State="StatusState.Info">Leader · @leaderInstance</StatusPill>
|
||||
}
|
||||
else if (state?.LeaderAddress is { } leaderAddress)
|
||||
{
|
||||
@* Unresolved address: shown raw rather than hidden — a leader on a
|
||||
host that is not in the registry is itself the finding. *@
|
||||
<StatusPill State="StatusState.Info">Leader · @leaderAddress</StatusPill>
|
||||
}
|
||||
@if (state?.Disagreement == true)
|
||||
{
|
||||
<StatusPill State="StatusState.Bad">
|
||||
Split brain · @string.Join(" vs ", state.ReportedLeaders)
|
||||
</StatusPill>
|
||||
}
|
||||
<span class="group-meta">@GroupMeta(group, now)</span>
|
||||
</div>
|
||||
}
|
||||
<div class="card-grid">
|
||||
@foreach (var instance in group)
|
||||
{
|
||||
<InstanceCard Instance="instance"
|
||||
Now="now"
|
||||
IsLeader="IsLeader(application, instance)" />
|
||||
}
|
||||
</div>
|
||||
</div>
|
||||
}
|
||||
</section>
|
||||
}
|
||||
}
|
||||
|
||||
@code {
|
||||
/// <summary>
|
||||
/// How often the page re-renders on its own. Independent of the poll interval: its job is to
|
||||
/// age the "last poll" figures and let a card cross into Stale even when no sweep lands —
|
||||
/// which is exactly the case a stuck poller produces.
|
||||
/// </summary>
|
||||
private static readonly TimeSpan UiTick = TimeSpan.FromSeconds(5);
|
||||
|
||||
private OverviewSnapshot _snapshot = OverviewSnapshot.Empty;
|
||||
private IReadOnlyList<InstanceSnapshot> _instances = [];
|
||||
private bool _paused;
|
||||
private ITimer? _timer;
|
||||
|
||||
protected override void OnInitialized()
|
||||
{
|
||||
Adopt(Store.Current);
|
||||
Store.Changed += OnSnapshotChanged;
|
||||
_timer = Time.CreateTimer(_ => InvokeAsync(StateHasChanged), null, UiTick, UiTick);
|
||||
}
|
||||
|
||||
private void OnSnapshotChanged(OverviewSnapshot snapshot)
|
||||
{
|
||||
// Pause stops ADOPTING new snapshots; the poller keeps sweeping. That is what makes the
|
||||
// pause useful — the frozen view then visibly ages into Stale, so a paused dashboard can
|
||||
// never be mistaken for a live one.
|
||||
if (_paused)
|
||||
return;
|
||||
|
||||
InvokeAsync(() =>
|
||||
{
|
||||
Adopt(snapshot);
|
||||
StateHasChanged();
|
||||
});
|
||||
}
|
||||
|
||||
private void Adopt(OverviewSnapshot snapshot)
|
||||
{
|
||||
_snapshot = snapshot;
|
||||
_instances = snapshot.AllInstances.ToList();
|
||||
}
|
||||
|
||||
private void TogglePause()
|
||||
{
|
||||
_paused = !_paused;
|
||||
|
||||
if (!_paused)
|
||||
Adopt(Store.Current);
|
||||
}
|
||||
|
||||
private int Count(DateTimeOffset now, InstanceStatus status) =>
|
||||
_instances.Count(i => i.Status == status && !i.IsStale(now));
|
||||
|
||||
private int StaleCount(DateTimeOffset now) => _instances.Count(i => i.IsStale(now));
|
||||
|
||||
private string Meta
|
||||
{
|
||||
get
|
||||
{
|
||||
var generated = _snapshot.GeneratedUtc == default
|
||||
? "never"
|
||||
: _snapshot.GeneratedUtc.ToLocalTime().ToString("HH:mm:ss", System.Globalization.CultureInfo.InvariantCulture);
|
||||
|
||||
return $"last sweep {generated} · {_instances.Count} instances";
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Groups an application's instances for display: named cluster groups first in registry
|
||||
/// order, then the ungrouped ones.
|
||||
/// </summary>
|
||||
/// <param name="application">The application to lay out.</param>
|
||||
/// <returns>The instance groups.</returns>
|
||||
private static IEnumerable<IGrouping<string?, InstanceSnapshot>> Layout(ApplicationSnapshot application) =>
|
||||
application.Instances
|
||||
.GroupBy(i => i.Group, StringComparer.Ordinal)
|
||||
.OrderBy(g => g.Key is null)
|
||||
.ThenBy(g => g.Key, StringComparer.Ordinal);
|
||||
|
||||
private static string AppMeta(ApplicationSnapshot application)
|
||||
{
|
||||
var clusters = application.Groups.Count;
|
||||
var instances = application.Instances.Count;
|
||||
var clusterText = clusters == 0 ? "" : $" · {clusters} cluster{(clusters == 1 ? "" : "s")}";
|
||||
|
||||
return $"{instances} instance{(instances == 1 ? "" : "s")}{clusterText}";
|
||||
}
|
||||
|
||||
private static string GroupMeta(IGrouping<string?, InstanceSnapshot> group, DateTimeOffset now)
|
||||
{
|
||||
var members = group.Count();
|
||||
var missing = group.Count(i => i.Status is InstanceStatus.Unreachable || i.IsStale(now));
|
||||
|
||||
return $"{members} member{(members == 1 ? "" : "s")} · {missing} unreachable";
|
||||
}
|
||||
|
||||
private static bool IsLeader(ApplicationSnapshot application, InstanceSnapshot instance) =>
|
||||
instance.Group is { } group &&
|
||||
application.GroupFor(group)?.LeaderInstance is { } leader &&
|
||||
string.Equals(leader, instance.Instance, StringComparison.Ordinal);
|
||||
|
||||
private static StatusState RollupState(ApplicationSnapshot application) => application.Worst switch
|
||||
{
|
||||
InstanceStatus.Up => StatusState.Ok,
|
||||
InstanceStatus.Degraded => StatusState.Warn,
|
||||
InstanceStatus.Down or InstanceStatus.Unreachable => StatusState.Bad,
|
||||
_ => StatusState.Idle,
|
||||
};
|
||||
|
||||
private static string RollupLabel(ApplicationSnapshot application) => application.Worst switch
|
||||
{
|
||||
InstanceStatus.Up => "Up",
|
||||
InstanceStatus.Degraded => "Degraded",
|
||||
InstanceStatus.Down => "Down",
|
||||
InstanceStatus.Unreachable => "Unreachable",
|
||||
_ => "Pending",
|
||||
};
|
||||
|
||||
/// <inheritdoc/>
|
||||
public void Dispose()
|
||||
{
|
||||
// Both are required: an un-detached handler would call StateHasChanged on a disposed
|
||||
// renderer once per sweep for the life of the process.
|
||||
Store.Changed -= OnSnapshotChanged;
|
||||
_timer?.Dispose();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
@* Plain RouteView, not AuthorizeRouteView. The dashboard is anonymous and
|
||||
read-only by requirement (design §2): there is no authentication pipeline to
|
||||
authorize against, and wiring one in would be the opposite of the decision. *@
|
||||
<Router AppAssembly="@typeof(Routes).Assembly">
|
||||
<Found Context="routeData">
|
||||
<RouteView RouteData="@routeData" DefaultLayout="@typeof(MainLayout)" />
|
||||
<FocusOnNavigate RouteData="@routeData" Selector="h1" />
|
||||
</Found>
|
||||
<NotFound>
|
||||
<LayoutView Layout="@typeof(MainLayout)">
|
||||
<PageTitle>Overview — Not Found</PageTitle>
|
||||
<section class="app-section">
|
||||
<h1 class="page-head">Not Found</h1>
|
||||
<p class="mb-0">The requested page does not exist.</p>
|
||||
</section>
|
||||
</LayoutView>
|
||||
</NotFound>
|
||||
</Router>
|
||||
@@ -0,0 +1,300 @@
|
||||
@* One instance = one card. Every value shown here comes from the snapshot the
|
||||
poller published; the component never probes anything itself. *@
|
||||
|
||||
<article class="inst @StateClass" data-testid="instance-card" data-instance="@Instance.Instance">
|
||||
<div class="inst-head">
|
||||
<div>
|
||||
<span class="inst-name">@Instance.Instance</span>
|
||||
<span class="inst-host">@HostAndPort</span>
|
||||
</div>
|
||||
<div class="inst-chips">
|
||||
@if (IsLeader)
|
||||
{
|
||||
<StatusPill State="StatusState.Info">Leader</StatusPill>
|
||||
}
|
||||
@if (Instance.Active is ActiveState.Active)
|
||||
{
|
||||
<StatusPill State="StatusState.Info">Active</StatusPill>
|
||||
}
|
||||
else if (Instance.Active is ActiveState.Standby)
|
||||
{
|
||||
<StatusPill State="StatusState.Idle">Standby</StatusPill>
|
||||
}
|
||||
else if (Instance.Active is ActiveState.Unknown)
|
||||
{
|
||||
<StatusPill State="StatusState.Idle">Role ?</StatusPill>
|
||||
}
|
||||
<StatusPill State="StatusPillState">@StatusLabel</StatusPill>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="inst-body">
|
||||
<div class="kv">
|
||||
<span class="k">Latency</span>
|
||||
<span class="v">@LatencyText</span>
|
||||
</div>
|
||||
<div class="kv">
|
||||
<span class="k">@(IsStale ? "Stale since" : "Last poll")</span>
|
||||
<span class="v @(IsStale ? "idle" : null)">@LastPollText</span>
|
||||
</div>
|
||||
<div class="kv">
|
||||
<span class="k">Checks</span>
|
||||
<span class="v @ChecksClass">@ChecksText</span>
|
||||
</div>
|
||||
|
||||
@if (Instance.Error is { Length: > 0 } error)
|
||||
{
|
||||
<div class="kv">
|
||||
<span class="k">Error</span>
|
||||
<span class="v bad" title="@error">@Truncate(error, 48)</span>
|
||||
</div>
|
||||
}
|
||||
|
||||
@if (Entries.Count > 0)
|
||||
{
|
||||
<details class="checks">
|
||||
<summary>@Entries.Count check@(Entries.Count == 1 ? "" : "s")</summary>
|
||||
@foreach (var entry in Entries)
|
||||
{
|
||||
<div class="check-row">
|
||||
<span class="check-dot @DotClass(entry.Value.Status)"></span>
|
||||
<span class="check-name">@entry.Key</span>
|
||||
<span class="check-desc">@entry.Value.Description</span>
|
||||
</div>
|
||||
}
|
||||
@if (ClusterDataLine is { Length: > 0 } clusterData)
|
||||
{
|
||||
@* The cluster view, rendered verbatim rather than interpreted: when a pair
|
||||
disagrees about the leader this line is the raw evidence an operator needs
|
||||
to see, not a summary of it. *@
|
||||
<div class="check-data">@((MarkupString)clusterData)</div>
|
||||
}
|
||||
</details>
|
||||
}
|
||||
</div>
|
||||
|
||||
<div class="panel-foot">
|
||||
@if (Instance.ManagementUrl is { Length: > 0 } managementUrl)
|
||||
{
|
||||
<a href="@managementUrl" target="_blank" rel="noopener">Open @Instance.ManagementLabel ↗</a>
|
||||
}
|
||||
else
|
||||
{
|
||||
<span class="foot-note">no management link</span>
|
||||
}
|
||||
<span class="foot-note" data-testid="raw-signals">@RawSignals</span>
|
||||
</div>
|
||||
</article>
|
||||
|
||||
@code {
|
||||
/// <summary>The instance to render.</summary>
|
||||
[Parameter, EditorRequired] public InstanceSnapshot Instance { get; set; } = default!;
|
||||
|
||||
/// <summary>The render-time clock, passed down so every card on a page agrees on "now".</summary>
|
||||
[Parameter, EditorRequired] public DateTimeOffset Now { get; set; }
|
||||
|
||||
/// <summary>Whether this instance is its group's leader.</summary>
|
||||
[Parameter] public bool IsLeader { get; set; }
|
||||
|
||||
private bool IsStale => Instance.IsStale(Now);
|
||||
|
||||
private IReadOnlyList<KeyValuePair<string, ZbHealthEntry>> Entries =>
|
||||
Instance.Report?.Entries.OrderBy(e => e.Key, StringComparer.Ordinal).ToList() ?? [];
|
||||
|
||||
/// <summary>
|
||||
/// Stale wins over the stored status in the class, so the card visibly greys out. What it
|
||||
/// shows is still the last real reading — the point is to mark it as no longer current, not
|
||||
/// to replace it with a guess.
|
||||
/// </summary>
|
||||
private string StateClass => Instance.Status switch
|
||||
{
|
||||
null => "pending",
|
||||
_ when IsStale => $"{Instance.Status.ToString()!.ToLowerInvariant()} stale",
|
||||
var status => status.ToString()!.ToLowerInvariant(),
|
||||
};
|
||||
|
||||
private string StatusLabel => Instance.Status switch
|
||||
{
|
||||
null => "Pending",
|
||||
_ when IsStale => "Stale",
|
||||
InstanceStatus.Up => "Up",
|
||||
InstanceStatus.Degraded => "Degraded",
|
||||
InstanceStatus.Down => "Down",
|
||||
_ => "Unreachable",
|
||||
};
|
||||
|
||||
private StatusState StatusPillState => Instance.Status switch
|
||||
{
|
||||
null => StatusState.Idle,
|
||||
_ when IsStale => StatusState.Idle,
|
||||
InstanceStatus.Up => StatusState.Ok,
|
||||
InstanceStatus.Degraded => StatusState.Warn,
|
||||
_ => StatusState.Bad,
|
||||
};
|
||||
|
||||
private string HostAndPort
|
||||
{
|
||||
get
|
||||
{
|
||||
if (!Uri.TryCreate(Instance.BaseUrl, UriKind.Absolute, out var uri))
|
||||
return Instance.BaseUrl;
|
||||
|
||||
return uri.IsDefaultPort ? uri.Host : $"{uri.Host}:{uri.Port}";
|
||||
}
|
||||
}
|
||||
|
||||
private string LatencyText =>
|
||||
Instance.Status is null ? "—" : $"{Instance.Latency.TotalMilliseconds:0} ms";
|
||||
|
||||
private string LastPollText
|
||||
{
|
||||
get
|
||||
{
|
||||
// Stale cards count from the last successful contact, not the last poll attempt:
|
||||
// "stale since 4 s ago" while an instance has actually been gone for ten minutes
|
||||
// would be actively misleading.
|
||||
var reference = IsStale ? Instance.LastReachableUtc ?? Instance.LastPolledUtc : Instance.LastPolledUtc;
|
||||
return reference is { } at ? Relative(Now - at) : "never";
|
||||
}
|
||||
}
|
||||
|
||||
private string ChecksText
|
||||
{
|
||||
get
|
||||
{
|
||||
if (Entries.Count == 0)
|
||||
return Instance.Status is null ? "—" : "no detail";
|
||||
|
||||
var healthy = Entries.Count(e => string.Equals(e.Value.Status, "Healthy", StringComparison.Ordinal));
|
||||
return $"{healthy} / {Entries.Count} healthy";
|
||||
}
|
||||
}
|
||||
|
||||
private string? ChecksClass
|
||||
{
|
||||
get
|
||||
{
|
||||
if (Entries.Count == 0)
|
||||
return null;
|
||||
|
||||
if (Entries.Any(e => string.Equals(e.Value.Status, "Unhealthy", StringComparison.Ordinal)))
|
||||
return "bad";
|
||||
|
||||
return Entries.Any(e => string.Equals(e.Value.Status, "Degraded", StringComparison.Ordinal))
|
||||
? "warn"
|
||||
: "ok";
|
||||
}
|
||||
}
|
||||
|
||||
private string RawSignals
|
||||
{
|
||||
get
|
||||
{
|
||||
var ready = Instance.Status switch
|
||||
{
|
||||
null => "—",
|
||||
InstanceStatus.Unreachable => "fail",
|
||||
InstanceStatus.Down => "503",
|
||||
_ => "200",
|
||||
};
|
||||
|
||||
var active = Instance.Active switch
|
||||
{
|
||||
ActiveState.Active => "200",
|
||||
ActiveState.Standby => "503",
|
||||
ActiveState.Unknown => "—",
|
||||
_ => null,
|
||||
};
|
||||
|
||||
return active is null ? $"ready {ready}" : $"ready {ready} · active {active}";
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Renders the cluster check's data as the mockup's monospace evidence line.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Located by the data it carries rather than by check name: the two apps register the
|
||||
/// same shared checks under different names (<c>akka-cluster</c> vs <c>akka</c>,
|
||||
/// <c>active-node</c> vs <c>cluster-primary</c>), so keying on either one silently drops
|
||||
/// the evidence line for the other.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <c>active</c> and <c>leader</c> are shown side by side <b>on purpose</b>. They come
|
||||
/// from different checks and are different nodes whenever any node has restarted — the
|
||||
/// leader is the lowest-addressed Up member, the active node is the oldest. Showing both
|
||||
/// turns the confusion that produced lmxopcua#494 into something an operator can read
|
||||
/// straight off the card, instead of a silent disagreement between the chip and the tier.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private string? ClusterDataLine
|
||||
{
|
||||
get
|
||||
{
|
||||
var entries = Instance.Report?.Entries.Values;
|
||||
if (entries is null)
|
||||
return null;
|
||||
|
||||
var cluster = entries.FirstOrDefault(e => e.DataString(LeaderResolver.LeaderDataKey) is { Length: > 0 });
|
||||
var active = entries.FirstOrDefault(e => e.DataString(LeaderResolver.ActiveNodeDataKey) is { Length: > 0 });
|
||||
|
||||
if (cluster is null && active is null)
|
||||
return null;
|
||||
|
||||
var parts = new List<string>();
|
||||
|
||||
void Add(string label, string? value)
|
||||
{
|
||||
if (!string.IsNullOrEmpty(value))
|
||||
parts.Add($"<b>{Escape(label)}</b> {Escape(value)}");
|
||||
}
|
||||
|
||||
Add("active", active?.DataString(LeaderResolver.ActiveNodeDataKey));
|
||||
Add("leader", cluster?.DataString(LeaderResolver.LeaderDataKey));
|
||||
Add("members", cluster?.DataInt("memberCount")?.ToString());
|
||||
Add("unreachable", cluster?.DataInt("unreachableCount")?.ToString());
|
||||
Add("self", (cluster ?? active)?.DataString("selfAddress"));
|
||||
|
||||
return parts.Count == 0 ? null : string.Join(" · ", parts);
|
||||
}
|
||||
}
|
||||
|
||||
private static string DotClass(string? status) => status switch
|
||||
{
|
||||
"Healthy" => "ok",
|
||||
"Degraded" => "warn",
|
||||
"Unhealthy" => "bad",
|
||||
_ => "idle",
|
||||
};
|
||||
|
||||
/// <summary>Formats an age the way an operator reads a dashboard: coarse and instant.</summary>
|
||||
/// <param name="age">The elapsed time.</param>
|
||||
/// <returns>A short relative string.</returns>
|
||||
internal static string Relative(TimeSpan age)
|
||||
{
|
||||
if (age < TimeSpan.Zero)
|
||||
age = TimeSpan.Zero;
|
||||
|
||||
if (age.TotalSeconds < 60)
|
||||
return $"{age.TotalSeconds:0} s ago";
|
||||
|
||||
if (age.TotalMinutes < 60)
|
||||
return $"{age.TotalMinutes:0} min ago";
|
||||
|
||||
return age.TotalHours < 24 ? $"{age.TotalHours:0} h ago" : $"{age.TotalDays:0} d ago";
|
||||
}
|
||||
|
||||
private static string Truncate(string value, int max) =>
|
||||
value.Length <= max ? value : string.Concat(value.AsSpan(0, max - 1), "…");
|
||||
|
||||
/// <summary>
|
||||
/// HTML-escapes a value before it goes into the one MarkupString on this card. Health data is
|
||||
/// remote input from another application — trusted operationally, but never trusted to be
|
||||
/// markup-safe.
|
||||
/// </summary>
|
||||
/// <param name="value">The raw value.</param>
|
||||
/// <returns>The escaped value.</returns>
|
||||
private static string Escape(string value) =>
|
||||
System.Net.WebUtility.HtmlEncode(value);
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
@using Microsoft.AspNetCore.Components.Routing
|
||||
@using Microsoft.AspNetCore.Components.Web
|
||||
@using ZB.MOM.WW.Overview.Components
|
||||
@using ZB.MOM.WW.Overview.Components.Layout
|
||||
@using ZB.MOM.WW.Overview.Components.Pages
|
||||
@using ZB.MOM.WW.Overview.Components.Widgets
|
||||
@using ZB.MOM.WW.Overview.Polling
|
||||
@using ZB.MOM.WW.Overview.Registry
|
||||
@using ZB.MOM.WW.Theme
|
||||
@using static Microsoft.AspNetCore.Components.Web.RenderMode
|
||||
+37
@@ -0,0 +1,37 @@
|
||||
using ZB.MOM.WW.Health;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Observability;
|
||||
|
||||
/// <summary>Registers the dashboard's own health checks.</summary>
|
||||
public static class ObservabilityServiceCollectionExtensions
|
||||
{
|
||||
/// <summary>Check name for the registry-loaded probe.</summary>
|
||||
public const string RegistryLoadedCheckName = "registry-loaded";
|
||||
|
||||
/// <summary>Check name for the poll-cycle liveness probe.</summary>
|
||||
public const string PollCycleCheckName = "last-poll-cycle-completed";
|
||||
|
||||
/// <summary>
|
||||
/// Adds the two readiness checks that make the dashboard's own <c>/health/ready</c> mean
|
||||
/// something: that it bound a registry, and that it is still polling it.
|
||||
/// </summary>
|
||||
/// <param name="services">The service collection.</param>
|
||||
/// <returns>The same collection, for chaining.</returns>
|
||||
/// <remarks>
|
||||
/// Both are tagged <see cref="ZbHealthTags.Ready"/> only. The dashboard is a single stateless
|
||||
/// process with no active/standby concept, so it registers nothing on the active tier — its own
|
||||
/// <c>/health/active</c> is deliberately an empty, always-200 report.
|
||||
/// </remarks>
|
||||
public static IServiceCollection AddOverviewHealthChecks(this IServiceCollection services)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
|
||||
services.AddHealthChecks()
|
||||
.AddTypeActivatedCheck<RegistryLoadedHealthCheck>(
|
||||
RegistryLoadedCheckName, failureStatus: null, tags: [ZbHealthTags.Ready])
|
||||
.AddTypeActivatedCheck<PollCycleHealthCheck>(
|
||||
PollCycleCheckName, failureStatus: null, tags: [ZbHealthTags.Ready]);
|
||||
|
||||
return services;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
using System.Diagnostics.Metrics;
|
||||
using ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Observability;
|
||||
|
||||
/// <summary>
|
||||
/// The dashboard's own Meter: what it currently believes about every registered instance, and how
|
||||
/// long its sweeps take.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The instance gauge is <em>observable</em> — it is read from the published snapshot at scrape
|
||||
/// time rather than written on every poll. The snapshot is already the single authority for what
|
||||
/// the dashboard believes, so deriving the gauge from it makes a disagreement between the page and
|
||||
/// the metric impossible by construction.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The instrument names follow the family convention in
|
||||
/// <c>components/observability/spec/METRIC-CONVENTIONS.md</c> §2 — lower-case, dot-separated
|
||||
/// <c><app>.<subsystem>.<event></c>, durations in seconds — so the Prometheus
|
||||
/// exporter renders them as <c>overview_instance_status</c> and
|
||||
/// <c>overview_poll_duration_seconds</c>.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class OverviewMetrics : IDisposable
|
||||
{
|
||||
/// <summary>The app's single Meter name, per the family one-meter-per-app convention.</summary>
|
||||
public const string MeterName = "ZB.MOM.WW.Overview";
|
||||
|
||||
/// <summary>Per-instance status gauge; the value is the <see cref="InstanceStatus"/> ordinal.</summary>
|
||||
public const string InstanceStatusInstrument = "overview.instance.status";
|
||||
|
||||
/// <summary>Sweep-duration histogram, in seconds.</summary>
|
||||
public const string PollDurationInstrument = "overview.poll.duration";
|
||||
|
||||
private readonly Meter _meter;
|
||||
private readonly Histogram<double> _pollDuration;
|
||||
private readonly OverviewSnapshotStore _store;
|
||||
|
||||
/// <summary>Initializes the meter and its instruments.</summary>
|
||||
/// <param name="meterFactory">Factory that scopes the meter's lifetime to the container.</param>
|
||||
/// <param name="store">The snapshot the observable gauge reads at scrape time.</param>
|
||||
public OverviewMetrics(IMeterFactory meterFactory, OverviewSnapshotStore store)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(meterFactory);
|
||||
|
||||
_store = store ?? throw new ArgumentNullException(nameof(store));
|
||||
_meter = meterFactory.Create(MeterName);
|
||||
|
||||
_meter.CreateObservableGauge(
|
||||
InstanceStatusInstrument,
|
||||
ObserveInstanceStatus,
|
||||
unit: "1",
|
||||
description: "Derived status of each registered instance (0=Up, 1=Degraded, 2=Down, 3=Unreachable).");
|
||||
|
||||
_pollDuration = _meter.CreateHistogram<double>(
|
||||
PollDurationInstrument,
|
||||
unit: "s",
|
||||
description: "Wall-clock duration of one poll sweep across every due instance.");
|
||||
}
|
||||
|
||||
/// <summary>Records the duration of one completed sweep.</summary>
|
||||
/// <param name="duration">Wall-clock time the sweep took.</param>
|
||||
public void RecordSweep(TimeSpan duration) => _pollDuration.Record(duration.TotalSeconds);
|
||||
|
||||
/// <inheritdoc/>
|
||||
public void Dispose() => _meter.Dispose();
|
||||
|
||||
private IEnumerable<Measurement<int>> ObserveInstanceStatus()
|
||||
{
|
||||
foreach (var instance in _store.Current.AllInstances)
|
||||
{
|
||||
// Instances that have never been probed are skipped rather than reported as anything.
|
||||
// Emitting a value for them would publish a status the dashboard does not yet hold —
|
||||
// and the enum's zero ordinal is Up, so the fabrication would be the reassuring one.
|
||||
if (instance.Status is not { } status)
|
||||
continue;
|
||||
|
||||
yield return new Measurement<int>(
|
||||
(int)status,
|
||||
new KeyValuePair<string, object?>("application", instance.Application),
|
||||
|
||||
// Deliberately NOT named "instance": Prometheus attaches its own `instance` label to
|
||||
// every scraped series, and a colliding metric label is silently renamed to
|
||||
// `exported_instance`, which no dashboard query would be written against.
|
||||
new KeyValuePair<string, object?>("node", instance.Instance),
|
||||
new KeyValuePair<string, object?>("group", instance.Group ?? string.Empty));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,110 @@
|
||||
using Microsoft.Extensions.Diagnostics.HealthChecks;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.Overview.Registry;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Observability;
|
||||
|
||||
/// <summary>
|
||||
/// Fails when the poller has stopped sweeping — the one failure mode that would otherwise leave the
|
||||
/// dashboard looking perfectly healthy while every card it serves quietly went stale.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// This is what makes the dashboard's own <c>/health/ready</c> honest. Without it the process
|
||||
/// answers 200 for as long as Kestrel is up, which says nothing about whether the data behind the
|
||||
/// page is still being refreshed.
|
||||
/// </remarks>
|
||||
public sealed class PollCycleHealthCheck : IHealthCheck
|
||||
{
|
||||
private readonly PollCycleHeartbeat _heartbeat;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private readonly TimeSpan _grace;
|
||||
|
||||
/// <summary>Initializes a new <see cref="PollCycleHealthCheck"/>.</summary>
|
||||
/// <param name="heartbeat">The poller's completed-sweep heartbeat.</param>
|
||||
/// <param name="options">The registry, used to derive the tolerated gap.</param>
|
||||
/// <param name="timeProvider">Clock.</param>
|
||||
public PollCycleHealthCheck(
|
||||
PollCycleHeartbeat heartbeat,
|
||||
IOptions<OverviewOptions> options,
|
||||
TimeProvider timeProvider)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
_heartbeat = heartbeat ?? throw new ArgumentNullException(nameof(heartbeat));
|
||||
_timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider));
|
||||
_grace = GraceFor(options.Value);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The longest gap between completed sweeps that is still considered healthy.
|
||||
/// </summary>
|
||||
/// <param name="options">The registry.</param>
|
||||
/// <returns>The tolerated gap.</returns>
|
||||
/// <remarks>
|
||||
/// A sweep completes on every timer tick, and the timer ticks at the fastest configured
|
||||
/// interval — so the expected gap is one interval plus however long the probes took, which is
|
||||
/// bounded by the slowest configured timeout. Twice that is comfortably past "slow" and
|
||||
/// unambiguously "stopped".
|
||||
/// </remarks>
|
||||
internal static TimeSpan GraceFor(OverviewOptions options)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
var timings = options.Applications
|
||||
.SelectMany(app => app.Instances.Select(instance =>
|
||||
EffectiveTimings.Resolve(options, app, instance)))
|
||||
.ToList();
|
||||
|
||||
// An empty registry cannot poll at all; RegistryLoadedHealthCheck is the check that reports
|
||||
// that, so fall back to the global defaults rather than dividing by an empty set here.
|
||||
var interval = timings.Count > 0
|
||||
? timings.Min(t => t.PollInterval)
|
||||
: TimeSpan.FromSeconds(options.PollIntervalSeconds);
|
||||
var timeout = timings.Count > 0
|
||||
? timings.Max(t => t.Timeout)
|
||||
: TimeSpan.FromSeconds(options.TimeoutSeconds);
|
||||
|
||||
return 2 * (interval + timeout);
|
||||
}
|
||||
|
||||
/// <inheritdoc/>
|
||||
public Task<HealthCheckResult> CheckHealthAsync(
|
||||
HealthCheckContext context,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
var now = _timeProvider.GetUtcNow();
|
||||
var last = _heartbeat.LastCompletedUtc;
|
||||
|
||||
var data = new Dictionary<string, object>(StringComparer.Ordinal)
|
||||
{
|
||||
["graceSeconds"] = Math.Round(_grace.TotalSeconds, 3),
|
||||
};
|
||||
|
||||
if (last is null)
|
||||
{
|
||||
// Nothing has swept yet. Within the grace window that is ordinary startup, so report
|
||||
// Degraded (still a 200) rather than failing readiness for the first few seconds of
|
||||
// every deployment. Past it, the poller never started and that IS a failure.
|
||||
var waiting = now - _heartbeat.StartedUtc;
|
||||
data["waitingSeconds"] = Math.Round(waiting.TotalSeconds, 3);
|
||||
|
||||
return Task.FromResult(waiting <= _grace
|
||||
? HealthCheckResult.Degraded("No poll cycle has completed yet; the poller is still starting.", data: data)
|
||||
: HealthCheckResult.Unhealthy(
|
||||
$"No poll cycle has completed in {waiting.TotalSeconds:0.#}s since startup.",
|
||||
exception: null,
|
||||
data));
|
||||
}
|
||||
|
||||
var age = now - last.Value;
|
||||
data["lastCycleUtc"] = last.Value.ToString("O", System.Globalization.CultureInfo.InvariantCulture);
|
||||
data["ageSeconds"] = Math.Round(age.TotalSeconds, 3);
|
||||
|
||||
return Task.FromResult(age <= _grace
|
||||
? HealthCheckResult.Healthy($"Last poll cycle completed {age.TotalSeconds:0.#}s ago.", data)
|
||||
: HealthCheckResult.Unhealthy(
|
||||
$"No poll cycle has completed in {age.TotalSeconds:0.#}s; the displayed data is stale.",
|
||||
exception: null,
|
||||
data));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
namespace ZB.MOM.WW.Overview.Observability;
|
||||
|
||||
/// <summary>
|
||||
/// When the poller last finished a full sweep. Written by the poller, read by
|
||||
/// <see cref="PollCycleHealthCheck"/>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Deliberately not derived from the published snapshot's <c>GeneratedUtc</c>: the poller publishes
|
||||
/// the registry once before it probes anything, so a snapshot exists from the moment the host
|
||||
/// starts. A stuck poller would therefore look like a working one. This records only sweeps that
|
||||
/// actually completed.
|
||||
/// </remarks>
|
||||
public sealed class PollCycleHeartbeat
|
||||
{
|
||||
private long _lastCompletedTicks;
|
||||
|
||||
/// <summary>Initializes the heartbeat and stamps the host start time.</summary>
|
||||
/// <param name="timeProvider">Clock.</param>
|
||||
public PollCycleHeartbeat(TimeProvider timeProvider)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(timeProvider);
|
||||
StartedUtc = timeProvider.GetUtcNow();
|
||||
}
|
||||
|
||||
/// <summary>When this heartbeat was constructed — effectively when the host started.</summary>
|
||||
/// <remarks>
|
||||
/// Lets the health check distinguish "started a moment ago and has not swept yet" from "has
|
||||
/// not swept in minutes", which are the same null reading but opposite conclusions.
|
||||
/// </remarks>
|
||||
public DateTimeOffset StartedUtc { get; }
|
||||
|
||||
/// <summary>When the last full sweep completed, or null if none has.</summary>
|
||||
public DateTimeOffset? LastCompletedUtc
|
||||
{
|
||||
get
|
||||
{
|
||||
var ticks = Interlocked.Read(ref _lastCompletedTicks);
|
||||
return ticks == 0 ? null : new DateTimeOffset(ticks, TimeSpan.Zero);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Records that a sweep completed.</summary>
|
||||
/// <param name="at">When it completed.</param>
|
||||
public void RecordCycleCompleted(DateTimeOffset at) =>
|
||||
Interlocked.Exchange(ref _lastCompletedTicks, at.UtcTicks);
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
using Microsoft.Extensions.Diagnostics.HealthChecks;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.Overview.Registry;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Observability;
|
||||
|
||||
/// <summary>
|
||||
/// Reports how much registry the dashboard actually bound, and fails if it bound none.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The pre-host <c>ConfigPreflight</c> and the options validator already refuse to start on an
|
||||
/// empty registry, so the unhealthy branch is a backstop rather than an expected state. The check
|
||||
/// earns its place on the reported counts: an operator looking at <c>/health/ready</c> after a
|
||||
/// config change can see how many instances this process is watching without reading its
|
||||
/// configuration files.
|
||||
/// </remarks>
|
||||
public sealed class RegistryLoadedHealthCheck : IHealthCheck
|
||||
{
|
||||
private readonly OverviewOptions _options;
|
||||
|
||||
/// <summary>Initializes a new <see cref="RegistryLoadedHealthCheck"/>.</summary>
|
||||
/// <param name="options">The bound registry.</param>
|
||||
public RegistryLoadedHealthCheck(IOptions<OverviewOptions> options)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
_options = options.Value;
|
||||
}
|
||||
|
||||
/// <inheritdoc/>
|
||||
public Task<HealthCheckResult> CheckHealthAsync(
|
||||
HealthCheckContext context,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
var applications = _options.Applications.Count;
|
||||
var instances = _options.Applications.Sum(a => a.Instances.Count);
|
||||
|
||||
var data = new Dictionary<string, object>(StringComparer.Ordinal)
|
||||
{
|
||||
["applications"] = applications,
|
||||
["instances"] = instances,
|
||||
};
|
||||
|
||||
return Task.FromResult(instances > 0
|
||||
? HealthCheckResult.Healthy(
|
||||
$"Watching {instances} instance(s) across {applications} application(s).",
|
||||
data)
|
||||
: HealthCheckResult.Unhealthy(
|
||||
"The registry contains no instances; the dashboard has nothing to watch.",
|
||||
exception: null,
|
||||
data));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
namespace ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
/// <summary>The derived state of one probed instance.</summary>
|
||||
public enum InstanceStatus
|
||||
{
|
||||
/// <summary>Ready tier answered 200 / <c>Healthy</c>.</summary>
|
||||
Up,
|
||||
|
||||
/// <summary>Ready tier answered 200 / <c>Degraded</c> — serving, but a check is unhappy.</summary>
|
||||
Degraded,
|
||||
|
||||
/// <summary>
|
||||
/// Ready tier answered 503 / <c>Unhealthy</c>. The app is up and talking; a dependency is not.
|
||||
/// </summary>
|
||||
Down,
|
||||
|
||||
/// <summary>
|
||||
/// Nothing answered — timeout, connection refused, DNS failure, or a body that is not the
|
||||
/// canonical health JSON. Kept distinct from <see cref="Down"/> because it usually means
|
||||
/// network / VPN / registry typo rather than a sick application, and the two lead an operator
|
||||
/// to look in completely different places.
|
||||
/// </summary>
|
||||
Unreachable,
|
||||
}
|
||||
|
||||
/// <summary>Whether a probed instance is the active half of its pair.</summary>
|
||||
public enum ActiveState
|
||||
{
|
||||
/// <summary>The instance has no active/standby concept (single-instance gateways).</summary>
|
||||
NotApplicable,
|
||||
|
||||
/// <summary>The active probe did not answer, so activity is unknown.</summary>
|
||||
Unknown,
|
||||
|
||||
/// <summary>Active tier answered 200 — this is the active node.</summary>
|
||||
Active,
|
||||
|
||||
/// <summary>Active tier answered 503 — this is the standby.</summary>
|
||||
Standby,
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Pure status derivation: probe result → <see cref="InstanceStatus"/>, with two-strike flap
|
||||
/// damping applied across polls. Static and dependency-free so every rule is table-testable.
|
||||
/// </summary>
|
||||
public static class StatusDerivation
|
||||
{
|
||||
/// <summary>
|
||||
/// Consecutive failing observations required before a previously-good instance is shown as
|
||||
/// failing. A single blip — one dropped packet, one GC pause past the timeout — is not news;
|
||||
/// two in a row is.
|
||||
/// </summary>
|
||||
public const int FailureStrikes = 2;
|
||||
|
||||
/// <summary>Classifies one ready-tier probe, before any damping.</summary>
|
||||
/// <param name="probe">The probe result.</param>
|
||||
/// <returns>The observed status.</returns>
|
||||
public static InstanceStatus Observe(HealthProbeResult probe)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(probe);
|
||||
|
||||
if (!probe.Reachable || probe.Report is null)
|
||||
return InstanceStatus.Unreachable;
|
||||
|
||||
// The reported status wins over the HTTP code: they agree by construction (Healthy/Degraded
|
||||
// → 200, Unhealthy → 503), and the body is the more specific of the two — 200 alone cannot
|
||||
// tell Up from Degraded.
|
||||
return probe.Report.Status switch
|
||||
{
|
||||
"Healthy" => InstanceStatus.Up,
|
||||
"Degraded" => InstanceStatus.Degraded,
|
||||
"Unhealthy" => InstanceStatus.Down,
|
||||
|
||||
// Parsed as JSON but carrying no status we recognise: the endpoint is answering with
|
||||
// something that is not this contract, which is the Unreachable case, not a sick app.
|
||||
_ => InstanceStatus.Unreachable,
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>Classifies one active-tier probe.</summary>
|
||||
/// <param name="hasActiveRole">Whether this instance participates in an active/standby pair.</param>
|
||||
/// <param name="probe">The active-tier probe result, or null when not probed.</param>
|
||||
/// <returns>The derived active state.</returns>
|
||||
public static ActiveState ObserveActive(bool hasActiveRole, HealthProbeResult? probe)
|
||||
{
|
||||
if (!hasActiveRole)
|
||||
return ActiveState.NotApplicable;
|
||||
|
||||
if (probe is null || !probe.Reachable || probe.StatusCode is null)
|
||||
return ActiveState.Unknown;
|
||||
|
||||
return probe.StatusCode switch
|
||||
{
|
||||
200 => ActiveState.Active,
|
||||
503 => ActiveState.Standby,
|
||||
_ => ActiveState.Unknown,
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Applies two-strike damping: a failing observation only replaces a previously-good state
|
||||
/// after <see cref="FailureStrikes"/> consecutive failures, while any good observation is
|
||||
/// adopted immediately.
|
||||
/// </summary>
|
||||
/// <param name="previous">The last displayed status, or null on the first-ever poll.</param>
|
||||
/// <param name="previousConsecutiveFailures">Consecutive failures recorded before this poll.</param>
|
||||
/// <param name="observed">This poll's observed status.</param>
|
||||
/// <returns>The status to display and the updated consecutive-failure count.</returns>
|
||||
/// <remarks>
|
||||
/// Recovery is deliberately asymmetric — instant. Damping exists to suppress false alarms, and
|
||||
/// making recovery equally slow would just convert them into false alarms that linger.
|
||||
/// </remarks>
|
||||
public static (InstanceStatus Status, int ConsecutiveFailures) Damp(
|
||||
InstanceStatus? previous,
|
||||
int previousConsecutiveFailures,
|
||||
InstanceStatus observed)
|
||||
{
|
||||
var failing = observed is InstanceStatus.Down or InstanceStatus.Unreachable;
|
||||
|
||||
if (!failing)
|
||||
return (observed, 0);
|
||||
|
||||
var failures = previousConsecutiveFailures + 1;
|
||||
|
||||
// With no previous state there is nothing to protect and nothing better to show: adopt the
|
||||
// observation. Claiming Up on a first poll that failed would be a fabrication.
|
||||
if (previous is null || failures >= FailureStrikes)
|
||||
return (observed, failures);
|
||||
|
||||
return (previous.Value, failures);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,186 @@
|
||||
namespace ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
/// <summary>
|
||||
/// Derives per-group leader state from whatever health check a member publishes its cluster view
|
||||
/// under.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Pure and static: given the same instance snapshots it always produces the same answer, which is
|
||||
/// what makes the split-brain rule — the one output an operator would act on at 3am — exhaustively
|
||||
/// testable without a cluster.
|
||||
/// </remarks>
|
||||
public static class LeaderResolver
|
||||
{
|
||||
/// <summary>
|
||||
/// The data key holding the address of the node actually in charge — the oldest Up member,
|
||||
/// published by the active tier's <c>ActiveNodeHealthCheck</c> from ZB.MOM.WW.Health 0.3.0.
|
||||
/// </summary>
|
||||
public const string ActiveNodeDataKey = "activeNode";
|
||||
|
||||
/// <summary>
|
||||
/// The data key holding the Akka cluster leader address, published by
|
||||
/// <c>AkkaClusterHealthCheck</c>. Fallback only — see <see cref="LeaderReportedBy"/>.
|
||||
/// </summary>
|
||||
public const string LeaderDataKey = "leader";
|
||||
|
||||
/// <summary>Resolves leader state for every cluster group present in <paramref name="instances"/>.</summary>
|
||||
/// <param name="instances">One application's instances.</param>
|
||||
/// <returns>Group state in first-appearance order; empty when the application has no groups.</returns>
|
||||
public static IReadOnlyList<GroupSnapshot> Resolve(IEnumerable<InstanceSnapshot> instances)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(instances);
|
||||
|
||||
var all = instances as IReadOnlyList<InstanceSnapshot> ?? instances.ToList();
|
||||
var groups = new List<GroupSnapshot>();
|
||||
|
||||
foreach (var name in all
|
||||
.Select(i => i.Group)
|
||||
.Where(g => !string.IsNullOrWhiteSpace(g))
|
||||
.Distinct(StringComparer.Ordinal))
|
||||
{
|
||||
var members = all.Where(i => string.Equals(i.Group, name, StringComparison.Ordinal)).ToList();
|
||||
|
||||
// A group of instances that have no active/standby concept has no leader to report;
|
||||
// rendering a leader chip for one would invent a distinction that does not exist.
|
||||
if (!members.Exists(m => m.HasActiveRole))
|
||||
continue;
|
||||
|
||||
groups.Add(ResolveGroup(name!, members, all));
|
||||
}
|
||||
|
||||
return groups;
|
||||
}
|
||||
|
||||
private static GroupSnapshot ResolveGroup(
|
||||
string name,
|
||||
IReadOnlyList<InstanceSnapshot> members,
|
||||
IReadOnlyList<InstanceSnapshot> allInstances)
|
||||
{
|
||||
// Only members that are actually talking get a vote. A Down or Unreachable node's last
|
||||
// known leader is stale by definition, and counting it would manufacture a disagreement
|
||||
// every time a node goes away mid-failover — the exact moment the flag must stay honest.
|
||||
var votes = members
|
||||
.Where(m => m.Status is InstanceStatus.Up or InstanceStatus.Degraded)
|
||||
.Select(LeaderReportedBy)
|
||||
.Where(address => !string.IsNullOrWhiteSpace(address))
|
||||
.Select(address => address!)
|
||||
.ToList();
|
||||
|
||||
if (votes.Count == 0)
|
||||
return new GroupSnapshot(name, null, null, Disagreement: false, []);
|
||||
|
||||
var distinct = votes.Distinct(StringComparer.Ordinal).OrderBy(a => a, StringComparer.Ordinal).ToList();
|
||||
|
||||
// The most-reported address wins the chip, ties broken ordinally so the display is stable
|
||||
// rather than flickering between two equally-supported claims on consecutive sweeps.
|
||||
var winner = votes
|
||||
.GroupBy(a => a, StringComparer.Ordinal)
|
||||
.OrderByDescending(g => g.Count())
|
||||
.ThenBy(g => g.Key, StringComparer.Ordinal)
|
||||
.First()
|
||||
.Key;
|
||||
|
||||
return new GroupSnapshot(
|
||||
name,
|
||||
ResolveToInstanceName(winner, allInstances),
|
||||
winner,
|
||||
Disagreement: distinct.Count > 1,
|
||||
distinct);
|
||||
}
|
||||
|
||||
/// <summary>Finds the address of the in-charge node one instance reports.</summary>
|
||||
/// <param name="instance">The instance whose report to search.</param>
|
||||
/// <returns>The reported address, or null when this instance publishes none.</returns>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b><c>activeNode</c> first, <c>leader</c> only as a fallback.</b> They are different
|
||||
/// nodes and the chip must show the first. The Akka cluster <i>leader</i> is the
|
||||
/// lowest-addressed Up member — address order, no relationship to time — while the node
|
||||
/// that actually holds the singletons and answers 200 on <c>/health/active</c> is the
|
||||
/// <i>oldest</i> Up member. They agree only until some node restarts, after which the
|
||||
/// restarted node rejoins youngest but keeps its address; observed live on the rebuilt
|
||||
/// OtOpcUa rig, where the leader was <c>central-1</c> while <c>central-2</c> was the
|
||||
/// active node. Ranking <c>leader</c> first would put the Active chip on the standby.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The fallback still matters: it is what an instance running Health < 0.3.0, or one
|
||||
/// whose active tier was not probed, publishes. Showing the leader there is better than
|
||||
/// showing nothing, and it is the pre-0.3.0 behaviour unchanged.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Matched on the DATA KEY, not on the check name. The two apps register the same shared
|
||||
/// checks under different names — ScadaBridge calls its cluster check
|
||||
/// <c>akka-cluster</c>, OtOpcUa calls it <c>akka</c>; the active check is
|
||||
/// <c>active-node</c> in one and <c>cluster-primary</c> in the other — and a resolver
|
||||
/// keyed on any of those names silently renders no chip for the other app, which is
|
||||
/// exactly how this was missed until a live rig showed OtOpcUa's groups bare while
|
||||
/// ScadaBridge's were fully populated. The data key is the part of the contract the
|
||||
/// shared checks actually own.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
internal static string? LeaderReportedBy(InstanceSnapshot instance)
|
||||
{
|
||||
if (instance.Report?.Entries is not { } entries)
|
||||
return null;
|
||||
|
||||
string? leaderFallback = null;
|
||||
|
||||
foreach (var entry in entries.Values)
|
||||
{
|
||||
if (entry.DataString(ActiveNodeDataKey) is { Length: > 0 } activeNode)
|
||||
return activeNode;
|
||||
|
||||
leaderFallback ??= entry.DataString(LeaderDataKey) is { Length: > 0 } leader ? leader : null;
|
||||
}
|
||||
|
||||
return leaderFallback;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Maps an Akka address to a registry instance name by host.
|
||||
/// </summary>
|
||||
/// <param name="akkaAddress">An address of the form <c>akka.tcp://system@host:port</c>.</param>
|
||||
/// <param name="instances">Candidate instances.</param>
|
||||
/// <returns>The matching instance's name, or null when nothing matches.</returns>
|
||||
/// <remarks>
|
||||
/// Host only, deliberately: the Akka remoting port and the HTTP port are different by design, so
|
||||
/// comparing ports would never match. An unmatched address is not an error — the card falls back
|
||||
/// to showing the raw address, which is still the useful thing to see.
|
||||
/// </remarks>
|
||||
internal static string? ResolveToInstanceName(string akkaAddress, IReadOnlyList<InstanceSnapshot> instances)
|
||||
{
|
||||
var host = HostOf(akkaAddress);
|
||||
if (host is null)
|
||||
return null;
|
||||
|
||||
foreach (var instance in instances)
|
||||
{
|
||||
if (Uri.TryCreate(instance.BaseUrl, UriKind.Absolute, out var uri) &&
|
||||
string.Equals(uri.Host, host, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return instance.Instance;
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
/// <summary>Extracts the host from an Akka address.</summary>
|
||||
/// <param name="akkaAddress">The address to parse.</param>
|
||||
/// <returns>The host, or null when the address is not in the expected form.</returns>
|
||||
internal static string? HostOf(string? akkaAddress)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(akkaAddress))
|
||||
return null;
|
||||
|
||||
var at = akkaAddress.LastIndexOf('@');
|
||||
if (at < 0 || at == akkaAddress.Length - 1)
|
||||
return null;
|
||||
|
||||
var hostAndPort = akkaAddress[(at + 1)..];
|
||||
var colon = hostAndPort.LastIndexOf(':');
|
||||
var host = colon > 0 ? hostAndPort[..colon] : hostAndPort;
|
||||
|
||||
return host.Length == 0 ? null : host;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,255 @@
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.Overview.Observability;
|
||||
using ZB.MOM.WW.Overview.Registry;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
/// <summary>
|
||||
/// The single writer: sweeps every registered instance on a timer and publishes each result set to
|
||||
/// <see cref="OverviewSnapshotStore"/>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// No retries. A failed probe is simply reported as failed and the next tick is the retry, which
|
||||
/// keeps the sweep bounded in time and keeps the two-strike damping in
|
||||
/// <see cref="StatusDerivation.Damp"/> the only place that decides whether a failure is worth
|
||||
/// showing. Retrying inside a sweep would silently double the effective strike count.
|
||||
/// </remarks>
|
||||
public sealed class OverviewPollerService : BackgroundService
|
||||
{
|
||||
/// <summary>Name of the pooled <see cref="HttpClient"/> used for probes.</summary>
|
||||
public const string HttpClientName = "overview-health";
|
||||
|
||||
/// <summary>
|
||||
/// Slack allowed when deciding whether an instance is due. A timer tick can arrive a hair early
|
||||
/// relative to the recorded sweep time, and without the tolerance such an instance would be
|
||||
/// skipped and end up polling at half its configured rate.
|
||||
/// </summary>
|
||||
private static readonly TimeSpan DueTolerance = TimeSpan.FromMilliseconds(250);
|
||||
|
||||
private readonly IHttpClientFactory _httpClientFactory;
|
||||
private readonly OverviewSnapshotStore _store;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private readonly ILogger<OverviewPollerService> _logger;
|
||||
private readonly OverviewMetrics _metrics;
|
||||
private readonly PollCycleHeartbeat _heartbeat;
|
||||
private readonly List<PollTarget> _targets;
|
||||
|
||||
/// <summary>Initializes a new <see cref="OverviewPollerService"/>.</summary>
|
||||
/// <param name="options">The validated registry.</param>
|
||||
/// <param name="httpClientFactory">Factory for the pooled probe client.</param>
|
||||
/// <param name="store">The snapshot store this service writes to.</param>
|
||||
/// <param name="timeProvider">Clock and timer source.</param>
|
||||
/// <param name="metrics">The app's own instruments.</param>
|
||||
/// <param name="heartbeat">Records that a sweep completed, for the self health check.</param>
|
||||
/// <param name="logger">Logger.</param>
|
||||
public OverviewPollerService(
|
||||
IOptions<OverviewOptions> options,
|
||||
IHttpClientFactory httpClientFactory,
|
||||
OverviewSnapshotStore store,
|
||||
TimeProvider timeProvider,
|
||||
OverviewMetrics metrics,
|
||||
PollCycleHeartbeat heartbeat,
|
||||
ILogger<OverviewPollerService> logger)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
_httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
|
||||
_store = store ?? throw new ArgumentNullException(nameof(store));
|
||||
_timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider));
|
||||
_metrics = metrics ?? throw new ArgumentNullException(nameof(metrics));
|
||||
_heartbeat = heartbeat ?? throw new ArgumentNullException(nameof(heartbeat));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_targets = BuildTargets(options.Value);
|
||||
}
|
||||
|
||||
/// <inheritdoc/>
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
if (_targets.Count == 0)
|
||||
{
|
||||
_logger.LogWarning("Overview registry contains no instances; the poller has nothing to do.");
|
||||
return;
|
||||
}
|
||||
|
||||
// Tick at the fastest configured cadence and let each target decide whether it is due.
|
||||
// A single timer honours per-instance overrides without spawning a timer per instance.
|
||||
var tick = _targets.Min(t => t.Timings.PollInterval);
|
||||
|
||||
// Publish the registry before probing anything, so the page paints its full layout at once
|
||||
// instead of appearing empty for the duration of the first sweep.
|
||||
_store.Publish(BuildSnapshot(_timeProvider.GetUtcNow()));
|
||||
|
||||
using var timer = new PeriodicTimer(tick, _timeProvider);
|
||||
|
||||
try
|
||||
{
|
||||
do
|
||||
{
|
||||
await SweepAsync(stoppingToken).ConfigureAwait(false);
|
||||
}
|
||||
while (await timer.WaitForNextTickAsync(stoppingToken).ConfigureAwait(false));
|
||||
}
|
||||
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
// Ordinary shutdown.
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Runs one sweep: probes every due instance and publishes the resulting snapshot.</summary>
|
||||
/// <param name="cancellationToken">Cancelled on host shutdown.</param>
|
||||
/// <returns>The snapshot that was published.</returns>
|
||||
internal async Task<OverviewSnapshot> SweepAsync(CancellationToken cancellationToken = default)
|
||||
{
|
||||
// Checked here rather than left to the transport: whether an in-flight request observes the
|
||||
// token is a property of the handler stack, but "a sweep that started after shutdown must
|
||||
// not publish" has to hold unconditionally, or the store keeps a half-probed registry as
|
||||
// its final state.
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
|
||||
var startedAt = _timeProvider.GetTimestamp();
|
||||
var now = _timeProvider.GetUtcNow();
|
||||
var due = _targets.Where(t => IsDue(t, now)).ToList();
|
||||
|
||||
if (due.Count > 0)
|
||||
{
|
||||
// One client per sweep, not per probe: IHttpClientFactory pools the handler either way,
|
||||
// and a single instance keeps connection reuse across the fan-out.
|
||||
var client = _httpClientFactory.CreateClient(HttpClientName);
|
||||
var probe = new ZbHealthReportClient(client);
|
||||
|
||||
var results = await Task
|
||||
.WhenAll(due.Select(target => ProbeAsync(probe, target, now, cancellationToken)))
|
||||
.ConfigureAwait(false);
|
||||
|
||||
for (var i = 0; i < due.Count; i++)
|
||||
due[i].Last = results[i];
|
||||
}
|
||||
|
||||
// Re-checked after the fan-out: a shutdown that lands mid-sweep leaves the probed targets
|
||||
// in an arbitrary mix of fresh and stale results, and publishing that mix would freeze a
|
||||
// misleading picture into the store for as long as the process takes to exit.
|
||||
cancellationToken.ThrowIfCancellationRequested();
|
||||
|
||||
var snapshot = BuildSnapshot(now);
|
||||
_store.Publish(snapshot);
|
||||
|
||||
_metrics.RecordSweep(_timeProvider.GetElapsedTime(startedAt));
|
||||
|
||||
// Stamped after the publish, so the heartbeat can never claim a cycle the store did not
|
||||
// receive — the health check's whole job is to notice exactly that gap.
|
||||
_heartbeat.RecordCycleCompleted(_timeProvider.GetUtcNow());
|
||||
|
||||
return snapshot;
|
||||
}
|
||||
|
||||
private bool IsDue(PollTarget target, DateTimeOffset now) =>
|
||||
target.Last?.LastPolledUtc is not { } polled || now - polled + DueTolerance >= target.Timings.PollInterval;
|
||||
|
||||
private static async Task<InstanceSnapshot> ProbeAsync(
|
||||
ZbHealthReportClient probe,
|
||||
PollTarget target,
|
||||
DateTimeOffset now,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var timeout = target.Timings.Timeout;
|
||||
var ready = await probe.ProbeAsync(target.ReadyUrl, timeout, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
// Skip the active probe when nothing answered the ready probe: it would spend a second full
|
||||
// timeout to learn what we already know, and ObserveActive maps a missing probe to Unknown
|
||||
// anyway. A 503 from ready is NOT a skip — that instance is talking, and whether it is the
|
||||
// active half is exactly what an operator wants to know about a sick node.
|
||||
HealthProbeResult? active = null;
|
||||
if (target.Instance.HasActiveRole && ready.Reachable)
|
||||
active = await probe.ProbeStatusAsync(target.ActiveUrl, timeout, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
var observed = StatusDerivation.Observe(ready);
|
||||
var (status, failures) = StatusDerivation.Damp(
|
||||
target.Last?.Status,
|
||||
target.Last?.ConsecutiveFailures ?? 0,
|
||||
observed);
|
||||
|
||||
return target.Template with
|
||||
{
|
||||
Status = status,
|
||||
Active = StatusDerivation.ObserveActive(target.Instance.HasActiveRole, active),
|
||||
ConsecutiveFailures = failures,
|
||||
LastPolledUtc = now,
|
||||
LastReachableUtc = ready.Reachable ? now : target.Last?.LastReachableUtc,
|
||||
Latency = ready.Latency,
|
||||
Error = ready.Error,
|
||||
Report = ready.Report,
|
||||
};
|
||||
}
|
||||
|
||||
private OverviewSnapshot BuildSnapshot(DateTimeOffset now)
|
||||
{
|
||||
var applications = _targets
|
||||
.GroupBy(t => t.Application, StringComparer.Ordinal)
|
||||
.Select(group =>
|
||||
{
|
||||
var instances = group.Select(t => t.Last ?? t.Template).ToList();
|
||||
return new ApplicationSnapshot(group.Key, instances, LeaderResolver.Resolve(instances));
|
||||
})
|
||||
.ToList();
|
||||
|
||||
return new OverviewSnapshot(now, applications);
|
||||
}
|
||||
|
||||
private static List<PollTarget> BuildTargets(OverviewOptions options)
|
||||
{
|
||||
var targets = new List<PollTarget>();
|
||||
|
||||
foreach (var application in options.Applications)
|
||||
{
|
||||
foreach (var instance in application.Instances)
|
||||
{
|
||||
var baseUrl = instance.BaseUrl.TrimEnd('/');
|
||||
|
||||
targets.Add(new PollTarget
|
||||
{
|
||||
Application = application.Name,
|
||||
Instance = instance,
|
||||
Timings = EffectiveTimings.Resolve(options, application, instance),
|
||||
ReadyUrl = $"{baseUrl}/health/ready",
|
||||
ActiveUrl = $"{baseUrl}/health/active",
|
||||
Template = new InstanceSnapshot
|
||||
{
|
||||
Application = application.Name,
|
||||
Instance = instance.Name,
|
||||
Group = instance.Group,
|
||||
BaseUrl = baseUrl,
|
||||
ManagementUrl = instance.ManagementUrl,
|
||||
ManagementLabel = application.ManagementLabel,
|
||||
HasActiveRole = instance.HasActiveRole,
|
||||
StaleAfter = EffectiveTimings.Resolve(options, application, instance).StaleAfter,
|
||||
},
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return targets;
|
||||
}
|
||||
|
||||
/// <summary>One instance's fixed probe configuration plus its most recent result.</summary>
|
||||
private sealed class PollTarget
|
||||
{
|
||||
public required string Application { get; init; }
|
||||
|
||||
public required InstanceEntry Instance { get; init; }
|
||||
|
||||
public required EffectiveTimings Timings { get; init; }
|
||||
|
||||
public required string ReadyUrl { get; init; }
|
||||
|
||||
public required string ActiveUrl { get; init; }
|
||||
|
||||
/// <summary>The registry-derived fields, reused as the base of every result.</summary>
|
||||
public required InstanceSnapshot Template { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// The last published result, or null before the first probe. Written only by the sweep
|
||||
/// loop after its <c>Task.WhenAll</c> has completed, so it is never touched concurrently.
|
||||
/// </summary>
|
||||
public InstanceSnapshot? Last { get; set; }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,172 @@
|
||||
namespace ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
/// <summary>
|
||||
/// Everything the dashboard knows about one instance, as of the last sweep that included it.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Immutable by construction: the poller builds a whole new object graph each sweep and swaps it
|
||||
/// into <see cref="OverviewSnapshotStore"/> atomically, so a render can never observe a half-updated
|
||||
/// registry.
|
||||
/// </remarks>
|
||||
public sealed record InstanceSnapshot
|
||||
{
|
||||
/// <summary>Owning application's display name.</summary>
|
||||
public required string Application { get; init; }
|
||||
|
||||
/// <summary>This instance's display name, unique within its application.</summary>
|
||||
public required string Instance { get; init; }
|
||||
|
||||
/// <summary>Cluster group, or null for a standalone instance.</summary>
|
||||
public string? Group { get; init; }
|
||||
|
||||
/// <summary>The probed base URL, shown on the card for diagnosis.</summary>
|
||||
public required string BaseUrl { get; init; }
|
||||
|
||||
/// <summary>Management UI link, when the registry supplies one.</summary>
|
||||
public string? ManagementUrl { get; init; }
|
||||
|
||||
/// <summary>Label for <see cref="ManagementUrl"/>, from the application entry.</summary>
|
||||
public string ManagementLabel { get; init; } = "UI";
|
||||
|
||||
/// <summary>Whether this instance is half of an active/standby pair.</summary>
|
||||
public bool HasActiveRole { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// The damped, displayable status — null until the first sweep has actually probed this
|
||||
/// instance.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Null is a real, distinct state and not a stand-in for "Up": the registry is published to the
|
||||
/// store before the first probe returns so the page renders its full layout immediately, and
|
||||
/// showing an unmeasured instance as healthy for that half-second would be a fabricated reading.
|
||||
/// </remarks>
|
||||
public InstanceStatus? Status { get; init; }
|
||||
|
||||
/// <summary>Active/standby state from the active tier.</summary>
|
||||
public ActiveState Active { get; init; } = ActiveState.NotApplicable;
|
||||
|
||||
/// <summary>Consecutive failing observations, carried across sweeps to drive flap damping.</summary>
|
||||
public int ConsecutiveFailures { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// When the sweep that produced this snapshot started; null before the first probe. Staleness
|
||||
/// is measured from here, so a stuck or paused poller ages every card even though no probe
|
||||
/// failed.
|
||||
/// </summary>
|
||||
public DateTimeOffset? LastPolledUtc { get; init; }
|
||||
|
||||
/// <summary>When this instance last answered a probe at all, regardless of what it said.</summary>
|
||||
public DateTimeOffset? LastReachableUtc { get; init; }
|
||||
|
||||
/// <summary>Wall-clock duration of the last ready probe.</summary>
|
||||
public TimeSpan Latency { get; init; }
|
||||
|
||||
/// <summary>Why the last probe failed, when it did.</summary>
|
||||
public string? Error { get; init; }
|
||||
|
||||
/// <summary>The last parsed ready-tier body, which drives the per-check detail list.</summary>
|
||||
public ZbHealthReport? Report { get; init; }
|
||||
|
||||
/// <summary>This instance's effective staleness window.</summary>
|
||||
public TimeSpan StaleAfter { get; init; }
|
||||
|
||||
/// <summary>Whether this snapshot is too old to be trusted as current.</summary>
|
||||
/// <param name="now">The current time.</param>
|
||||
/// <returns>True when the snapshot has aged past its staleness window.</returns>
|
||||
/// <remarks>
|
||||
/// Evaluated at render rather than stored, because staleness is a function of the clock and not
|
||||
/// of anything the poller observed — a snapshot that was fresh when written goes stale on its
|
||||
/// own while nothing at all happens.
|
||||
/// </remarks>
|
||||
public bool IsStale(DateTimeOffset now) =>
|
||||
Status is not null && LastPolledUtc is { } polled && now - polled > StaleAfter;
|
||||
}
|
||||
|
||||
/// <summary>Leader state for one cluster group, derived from what its members report.</summary>
|
||||
/// <param name="Name">The group name from the registry.</param>
|
||||
/// <param name="LeaderInstance">
|
||||
/// Registry name of the instance the reported leader address resolves to, or null when no member
|
||||
/// reported a leader or the address matches no registered instance.
|
||||
/// </param>
|
||||
/// <param name="LeaderAddress">The reported Akka leader address, shown raw when unresolved.</param>
|
||||
/// <param name="Disagreement">
|
||||
/// True when reachable members reported different leaders — the split-brain tell.
|
||||
/// </param>
|
||||
/// <param name="ReportedLeaders">Every distinct address reported, for the disagreement detail.</param>
|
||||
public sealed record GroupSnapshot(
|
||||
string Name,
|
||||
string? LeaderInstance,
|
||||
string? LeaderAddress,
|
||||
bool Disagreement,
|
||||
IReadOnlyList<string> ReportedLeaders);
|
||||
|
||||
/// <summary>One application section of the dashboard.</summary>
|
||||
/// <param name="Name">The application's display name.</param>
|
||||
/// <param name="Instances">Its instances, in registry order.</param>
|
||||
/// <param name="Groups">Leader state per cluster group, in registry order.</param>
|
||||
public sealed record ApplicationSnapshot(
|
||||
string Name,
|
||||
IReadOnlyList<InstanceSnapshot> Instances,
|
||||
IReadOnlyList<GroupSnapshot> Groups)
|
||||
{
|
||||
/// <summary>Finds the group state for <paramref name="group"/>.</summary>
|
||||
/// <param name="group">The group name, or null for standalone instances.</param>
|
||||
/// <returns>The group snapshot, or null when there is none.</returns>
|
||||
public GroupSnapshot? GroupFor(string? group) =>
|
||||
group is null ? null : Groups.FirstOrDefault(g => string.Equals(g.Name, group, StringComparison.Ordinal));
|
||||
|
||||
/// <summary>
|
||||
/// The most severe status across this application's probed instances, for the rollup pill;
|
||||
/// null while nothing has been probed yet.
|
||||
/// </summary>
|
||||
public InstanceStatus? Worst
|
||||
{
|
||||
get
|
||||
{
|
||||
InstanceStatus? worst = null;
|
||||
foreach (var instance in Instances)
|
||||
{
|
||||
if (instance.Status is not { } status)
|
||||
continue;
|
||||
|
||||
if (worst is null || StatusSeverity.Rank(status) > StatusSeverity.Rank(worst.Value))
|
||||
worst = status;
|
||||
}
|
||||
|
||||
return worst;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>A complete, atomically-published view of the whole registry.</summary>
|
||||
/// <param name="GeneratedUtc">When the sweep that produced this snapshot started.</param>
|
||||
/// <param name="Applications">The applications, in registry order.</param>
|
||||
public sealed record OverviewSnapshot(DateTimeOffset GeneratedUtc, IReadOnlyList<ApplicationSnapshot> Applications)
|
||||
{
|
||||
/// <summary>The pre-registry snapshot served before the poller has published anything.</summary>
|
||||
public static OverviewSnapshot Empty { get; } = new(default, []);
|
||||
|
||||
/// <summary>Every instance across every application, flattened.</summary>
|
||||
public IEnumerable<InstanceSnapshot> AllInstances => Applications.SelectMany(a => a.Instances);
|
||||
}
|
||||
|
||||
/// <summary>Severity ordering for rollups.</summary>
|
||||
public static class StatusSeverity
|
||||
{
|
||||
/// <summary>Ranks a status; higher is worse.</summary>
|
||||
/// <param name="status">The status to rank.</param>
|
||||
/// <returns>The severity rank.</returns>
|
||||
/// <remarks>
|
||||
/// <see cref="InstanceStatus.Unreachable"/> outranks <see cref="InstanceStatus.Down"/> because
|
||||
/// it carries strictly less information: a Down instance told us what is wrong, an Unreachable
|
||||
/// one did not answer at all and could be anything up to and including gone.
|
||||
/// </remarks>
|
||||
public static int Rank(InstanceStatus status) => status switch
|
||||
{
|
||||
InstanceStatus.Up => 0,
|
||||
InstanceStatus.Degraded => 1,
|
||||
InstanceStatus.Down => 2,
|
||||
InstanceStatus.Unreachable => 3,
|
||||
_ => 3,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
namespace ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
/// <summary>
|
||||
/// Holds the current <see cref="OverviewSnapshot"/> and notifies subscribers when it is replaced.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// This is the cache-first requirement made concrete: the poller is the only writer, and every
|
||||
/// render reads whatever is already here. A page load therefore paints the last known state
|
||||
/// immediately and never waits on — or triggers — a probe of its own.
|
||||
/// </remarks>
|
||||
public sealed class OverviewSnapshotStore
|
||||
{
|
||||
private OverviewSnapshot _current = OverviewSnapshot.Empty;
|
||||
|
||||
/// <summary>Raised after a new snapshot has been published.</summary>
|
||||
/// <remarks>
|
||||
/// Handlers run on the poller's thread, so a Blazor subscriber must marshal onto its own
|
||||
/// synchronization context (<c>InvokeAsync(StateHasChanged)</c>) and must unsubscribe when
|
||||
/// disposed.
|
||||
/// </remarks>
|
||||
public event Action<OverviewSnapshot>? Changed;
|
||||
|
||||
/// <summary>The current snapshot; never null.</summary>
|
||||
public OverviewSnapshot Current => Volatile.Read(ref _current);
|
||||
|
||||
/// <summary>Replaces the current snapshot and notifies subscribers.</summary>
|
||||
/// <param name="snapshot">The new snapshot.</param>
|
||||
public void Publish(OverviewSnapshot snapshot)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(snapshot);
|
||||
|
||||
// A single reference swap of an already-complete graph. Readers hold their own reference
|
||||
// for the duration of a render, so a sweep landing mid-render cannot tear the view.
|
||||
Volatile.Write(ref _current, snapshot);
|
||||
|
||||
Changed?.Invoke(snapshot);
|
||||
}
|
||||
}
|
||||
+40
@@ -0,0 +1,40 @@
|
||||
using Microsoft.Extensions.DependencyInjection.Extensions;
|
||||
using ZB.MOM.WW.Overview.Observability;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
/// <summary>Registers the polling pipeline.</summary>
|
||||
public static class PollingServiceCollectionExtensions
|
||||
{
|
||||
/// <summary>Adds the snapshot store, the probe client and the poller.</summary>
|
||||
/// <param name="services">The service collection.</param>
|
||||
/// <returns>The same collection, for chaining.</returns>
|
||||
public static IServiceCollection AddOverviewPolling(this IServiceCollection services)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
|
||||
services.TryAddSingleton(TimeProvider.System);
|
||||
services.TryAddSingleton<OverviewSnapshotStore>();
|
||||
|
||||
// IMeterFactory comes from AddMetrics, which the default web host already registers; the
|
||||
// call is idempotent and makes the dependency explicit for any non-web host.
|
||||
services.AddMetrics();
|
||||
services.TryAddSingleton<OverviewMetrics>();
|
||||
services.TryAddSingleton<PollCycleHeartbeat>();
|
||||
|
||||
services.AddHttpClient(OverviewPollerService.HttpClientName, client =>
|
||||
{
|
||||
// The per-request CancellationTokenSource in ZbHealthReportClient is the only timeout
|
||||
// that should ever fire. Leaving HttpClient's own 100s default in place would let it
|
||||
// pre-empt a per-instance override and report "timed out after 3s" for a 100s wait.
|
||||
client.Timeout = Timeout.InfiniteTimeSpan;
|
||||
|
||||
// Health bodies are small and must not be served from anywhere but the instance itself.
|
||||
client.DefaultRequestHeaders.CacheControl = new() { NoCache = true, NoStore = true };
|
||||
});
|
||||
|
||||
services.AddHostedService<OverviewPollerService>();
|
||||
|
||||
return services;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,84 @@
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
/// <summary>
|
||||
/// The canonical <c>ZbHealthWriter</c> body every family app serves from <c>MapZbHealth</c>:
|
||||
/// <c>{ status, totalDurationMs, entries: { name: { status, description, durationMs, data? } } }</c>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// This mirrors a contract owned by <c>ZB.MOM.WW.Health</c>. It lives in the app for v1 by
|
||||
/// deliberate decision (design §8.2): promoting it to a <c>ZB.MOM.WW.Health.Client</c> package is
|
||||
/// worth a publish only once a second consumer exists.
|
||||
/// </remarks>
|
||||
public sealed class ZbHealthReport
|
||||
{
|
||||
/// <summary>Aggregate tier status: <c>Healthy</c>, <c>Degraded</c> or <c>Unhealthy</c>.</summary>
|
||||
[JsonPropertyName("status")]
|
||||
public string? Status { get; init; }
|
||||
|
||||
/// <summary>Total time the tier's checks took, in milliseconds.</summary>
|
||||
[JsonPropertyName("totalDurationMs")]
|
||||
public double TotalDurationMs { get; init; }
|
||||
|
||||
/// <summary>Per-check results, keyed by check name (verbatim, e.g. <c>akka-cluster</c>).</summary>
|
||||
[JsonPropertyName("entries")]
|
||||
public Dictionary<string, ZbHealthEntry> Entries { get; init; } = new(StringComparer.Ordinal);
|
||||
}
|
||||
|
||||
/// <summary>One check's result inside a <see cref="ZbHealthReport"/>.</summary>
|
||||
public sealed class ZbHealthEntry
|
||||
{
|
||||
/// <summary>This check's status: <c>Healthy</c>, <c>Degraded</c> or <c>Unhealthy</c>.</summary>
|
||||
[JsonPropertyName("status")]
|
||||
public string? Status { get; init; }
|
||||
|
||||
/// <summary>Human-readable detail. Always present in the canonical shape; may be JSON null.</summary>
|
||||
[JsonPropertyName("description")]
|
||||
public string? Description { get; init; }
|
||||
|
||||
/// <summary>How long this check took, in milliseconds.</summary>
|
||||
[JsonPropertyName("durationMs")]
|
||||
public double DurationMs { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Optional structured data the check published (ZB.MOM.WW.Health 0.2.0+). Absent entirely on
|
||||
/// checks that publish none and on every pre-0.2.0 node — hence nullable, and hence the
|
||||
/// dashboard must render fully (minus leader) without it.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Values stay as <see cref="JsonElement"/>: this is an open contract carrying arbitrary JSON,
|
||||
/// and binding it to concrete types here would break the moment a check adds a field.
|
||||
/// </remarks>
|
||||
[JsonPropertyName("data")]
|
||||
public Dictionary<string, JsonElement>? Data { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Reads <paramref name="key"/> from <see cref="Data"/> as a string, or null when the key is
|
||||
/// absent or is not a JSON string.
|
||||
/// </summary>
|
||||
/// <param name="key">The data key, matched verbatim.</param>
|
||||
/// <returns>The string value, or null.</returns>
|
||||
public string? DataString(string key)
|
||||
{
|
||||
if (Data is null || !Data.TryGetValue(key, out var value))
|
||||
return null;
|
||||
|
||||
return value.ValueKind == JsonValueKind.String ? value.GetString() : null;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reads <paramref name="key"/> from <see cref="Data"/> as an integer, or null when the key is
|
||||
/// absent or is not a JSON number.
|
||||
/// </summary>
|
||||
/// <param name="key">The data key, matched verbatim.</param>
|
||||
/// <returns>The integer value, or null.</returns>
|
||||
public int? DataInt(string key)
|
||||
{
|
||||
if (Data is null || !Data.TryGetValue(key, out var value))
|
||||
return null;
|
||||
|
||||
return value.ValueKind == JsonValueKind.Number && value.TryGetInt32(out var parsed) ? parsed : null;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,139 @@
|
||||
using System.Diagnostics;
|
||||
using System.Text.Json;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
/// <summary>
|
||||
/// One probe of one health endpoint: whether anything answered, what it answered, and how long it
|
||||
/// took.
|
||||
/// </summary>
|
||||
/// <param name="Reachable">
|
||||
/// True when the endpoint answered with a parseable canonical body. False for timeouts, refused
|
||||
/// connections, DNS failures, and bodies that are not the canonical health JSON.
|
||||
/// </param>
|
||||
/// <param name="StatusCode">The HTTP status code, when one was received.</param>
|
||||
/// <param name="Report">The parsed body, when it parsed.</param>
|
||||
/// <param name="Latency">Wall-clock time for the request.</param>
|
||||
/// <param name="Error">A short reason when <paramref name="Reachable"/> is false.</param>
|
||||
public sealed record HealthProbeResult(
|
||||
bool Reachable,
|
||||
int? StatusCode,
|
||||
ZbHealthReport? Report,
|
||||
TimeSpan Latency,
|
||||
string? Error);
|
||||
|
||||
/// <summary>
|
||||
/// GETs a family health endpoint and parses the canonical <c>ZbHealthWriter</c> body.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Both 200 and 503 are treated as SUCCESSFUL probes: an Unhealthy tier still returns a full body,
|
||||
/// and that body is exactly what the dashboard needs to show which check failed. Only a genuine
|
||||
/// non-answer (transport failure) or an unparseable body is a failed probe.
|
||||
/// </remarks>
|
||||
public sealed class ZbHealthReportClient
|
||||
{
|
||||
private static readonly JsonSerializerOptions SerializerOptions = new()
|
||||
{
|
||||
PropertyNameCaseInsensitive = true,
|
||||
};
|
||||
|
||||
private readonly HttpClient _httpClient;
|
||||
|
||||
/// <summary>Initializes a new <see cref="ZbHealthReportClient"/>.</summary>
|
||||
/// <param name="httpClient">The HTTP client used for probes.</param>
|
||||
public ZbHealthReportClient(HttpClient httpClient) =>
|
||||
_httpClient = httpClient ?? throw new ArgumentNullException(nameof(httpClient));
|
||||
|
||||
/// <summary>Probes <paramref name="url"/> and parses the response body.</summary>
|
||||
/// <param name="url">Absolute URL of a health endpoint.</param>
|
||||
/// <param name="timeout">Per-request timeout.</param>
|
||||
/// <param name="cancellationToken">Token cancelled when the host is shutting down.</param>
|
||||
/// <returns>The probe result; this never throws for an unreachable endpoint.</returns>
|
||||
public Task<HealthProbeResult> ProbeAsync(
|
||||
string url,
|
||||
TimeSpan timeout,
|
||||
CancellationToken cancellationToken = default) =>
|
||||
SendAsync(url, timeout, bodyRequired: true, cancellationToken);
|
||||
|
||||
/// <summary>
|
||||
/// Probes <paramref name="url"/> for its status code alone, parsing the body only if one
|
||||
/// happens to be there.
|
||||
/// </summary>
|
||||
/// <param name="url">Absolute URL of a health endpoint.</param>
|
||||
/// <param name="timeout">Per-request timeout.</param>
|
||||
/// <param name="cancellationToken">Token cancelled when the host is shutting down.</param>
|
||||
/// <returns>The probe result; this never throws for an unreachable endpoint.</returns>
|
||||
/// <remarks>
|
||||
/// For the active tier the status code IS the answer — 200 means active, 503 means standby —
|
||||
/// and the body carries nothing the dashboard uses. Demanding a parseable body here would
|
||||
/// report every node behind anything that answers the tier with an empty response (a bare
|
||||
/// <c>IActiveNodeGate</c> endpoint, a load balancer, a proxy) as "role unknown" while it is
|
||||
/// plainly telling us the role.
|
||||
/// </remarks>
|
||||
public Task<HealthProbeResult> ProbeStatusAsync(
|
||||
string url,
|
||||
TimeSpan timeout,
|
||||
CancellationToken cancellationToken = default) =>
|
||||
SendAsync(url, timeout, bodyRequired: false, cancellationToken);
|
||||
|
||||
private async Task<HealthProbeResult> SendAsync(
|
||||
string url,
|
||||
TimeSpan timeout,
|
||||
bool bodyRequired,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var startedAt = Stopwatch.GetTimestamp();
|
||||
|
||||
// Linked, not a bare timeout token: shutdown must cancel an in-flight probe immediately,
|
||||
// and the two causes have to stay distinguishable so a shutdown is not misreported as a
|
||||
// timed-out instance.
|
||||
using var timeoutSource = new CancellationTokenSource(timeout);
|
||||
using var linked = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, timeoutSource.Token);
|
||||
|
||||
try
|
||||
{
|
||||
using var response = await _httpClient
|
||||
.GetAsync(url, HttpCompletionOption.ResponseContentRead, linked.Token)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
var body = await response.Content.ReadAsStringAsync(linked.Token).ConfigureAwait(false);
|
||||
var statusCode = (int)response.StatusCode;
|
||||
|
||||
ZbHealthReport? report;
|
||||
try
|
||||
{
|
||||
report = JsonSerializer.Deserialize<ZbHealthReport>(body, SerializerOptions);
|
||||
}
|
||||
catch (JsonException ex)
|
||||
{
|
||||
// Answered, but not with this contract — a reverse proxy error page, an HTML login
|
||||
// redirect, a wrong port. That is a registry/plumbing problem, not a sick app, so
|
||||
// it lands as Unreachable rather than Down.
|
||||
return bodyRequired
|
||||
? Failed(startedAt, statusCode, $"response body is not canonical health JSON: {ex.Message}")
|
||||
: new HealthProbeResult(true, statusCode, null, Stopwatch.GetElapsedTime(startedAt), null);
|
||||
}
|
||||
|
||||
if (report is null && bodyRequired)
|
||||
return Failed(startedAt, statusCode, "response body deserialized to null");
|
||||
|
||||
return new HealthProbeResult(true, statusCode, report, Stopwatch.GetElapsedTime(startedAt), null);
|
||||
}
|
||||
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
// Host shutdown, not an instance fault — let the caller abandon the sweep.
|
||||
throw;
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return Failed(startedAt, null, $"timed out after {timeout.TotalSeconds:0.##}s");
|
||||
}
|
||||
catch (HttpRequestException ex)
|
||||
{
|
||||
return Failed(startedAt, null, ex.Message);
|
||||
}
|
||||
}
|
||||
|
||||
private static HealthProbeResult Failed(long startedAt, int? statusCode, string error) =>
|
||||
new(false, statusCode, null, Stopwatch.GetElapsedTime(startedAt), error);
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
using ZB.MOM.WW.Configuration;
|
||||
using ZB.MOM.WW.Health;
|
||||
using ZB.MOM.WW.Overview.Components;
|
||||
using ZB.MOM.WW.Overview.Observability;
|
||||
using ZB.MOM.WW.Overview.Polling;
|
||||
using ZB.MOM.WW.Overview.Registry;
|
||||
using ZB.MOM.WW.Telemetry;
|
||||
using ZB.MOM.WW.Telemetry.Serilog;
|
||||
|
||||
var builder = WebApplication.CreateBuilder(args);
|
||||
|
||||
// The host calls this itself, but ONLY when the environment is literally "Development". Running the
|
||||
// build output under any other name — checking a staging registry locally, say — leaves the Theme
|
||||
// RCL's assets under _content/ unresolvable, and the page renders as a blank white screen with its
|
||||
// markup fully present and every stylesheet 404. Calling it unconditionally is a no-op once
|
||||
// published (the manifest is gone and the assets are real files on disk), so the only thing it
|
||||
// changes is that failure.
|
||||
builder.WebHost.UseStaticWebAssets();
|
||||
|
||||
// Pre-host presence check. The validator below covers shape and consistency, but it only runs once
|
||||
// the options are resolved — a registry section that is missing entirely binds to an empty
|
||||
// OverviewOptions, and the failure a reader can act on is "you configured no registry", not
|
||||
// "Applications must have at least 1 entry". Fail here, before the host is built, with the key name.
|
||||
ConfigPreflight.For(builder.Configuration)
|
||||
.RequireValue($"{OverviewOptions.SectionName}:Applications:0:Name")
|
||||
.RequireValue($"{OverviewOptions.SectionName}:Applications:0:Instances:0:BaseUrl")
|
||||
.ThrowIfInvalid();
|
||||
|
||||
builder.Services.AddValidatedOptions<OverviewOptions, OverviewOptionsValidator>(
|
||||
builder.Configuration, OverviewOptions.SectionName);
|
||||
|
||||
builder.AddZbSerilog(o => o.ServiceName = "overview");
|
||||
|
||||
builder.AddZbTelemetry(o =>
|
||||
{
|
||||
o.ServiceName = "overview";
|
||||
|
||||
// Meters is an ALLOWLIST, not a filter: a Meter absent from this array is never registered with
|
||||
// the MeterProvider, and its instruments vanish from /metrics with no warning anywhere. The
|
||||
// dashboard's own gauge is the only thing here that would go missing silently.
|
||||
o.Meters = [OverviewMetrics.MeterName];
|
||||
});
|
||||
|
||||
builder.Services.AddOverviewPolling();
|
||||
builder.Services.AddOverviewHealthChecks();
|
||||
|
||||
builder.Services.AddRazorComponents()
|
||||
.AddInteractiveServerComponents();
|
||||
|
||||
var app = builder.Build();
|
||||
|
||||
app.UseStaticFiles();
|
||||
|
||||
// No UseAuthentication/UseAuthorization and no cascading auth state: the dashboard is anonymous and
|
||||
// read-only by requirement (design §2). It renders health data that every one of these apps already
|
||||
// serves unauthenticated, and it has no endpoint that mutates anything.
|
||||
//
|
||||
// UseAntiforgery IS required despite that. AddRazorComponents stamps antiforgery metadata onto every
|
||||
// component endpoint unconditionally, and the endpoint middleware hard-fails a request whose endpoint
|
||||
// carries that metadata with no middleware to honour it — so without this line every page returns
|
||||
// 500. The alternative, MapRazorComponents(...).DisableAntiforgery(), would also boot, but it turns
|
||||
// the first form anyone ever adds into a silent CSRF hole. This app has no forms, so the middleware
|
||||
// is inert; it is here so that stays true by default rather than by vigilance.
|
||||
app.UseAntiforgery();
|
||||
|
||||
app.MapRazorComponents<App>()
|
||||
.AddInteractiveServerRenderMode();
|
||||
|
||||
// The dashboard's own three-tier health surface and Prometheus endpoint. Both are anonymous, like
|
||||
// every /health/* endpoint this app polls — which also means one Overview instance can be pointed
|
||||
// at another and would render it correctly.
|
||||
app.MapZbHealth();
|
||||
app.MapZbMetrics();
|
||||
|
||||
await app.RunAsync();
|
||||
|
||||
/// <summary>
|
||||
/// Exposed so <c>WebApplicationFactory<Program></c> can boot the real host in tests
|
||||
/// (top-level statements otherwise compile to an internal entry-point class).
|
||||
/// </summary>
|
||||
public partial class Program;
|
||||
@@ -0,0 +1,124 @@
|
||||
namespace ZB.MOM.WW.Overview.Registry;
|
||||
|
||||
/// <summary>
|
||||
/// The dashboard's registry: which application instances to probe, and how often. Bound from the
|
||||
/// <c>Overview</c> configuration section and validated by <see cref="OverviewOptionsValidator"/>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Every instance is probed the same way — GET <c>{BaseUrl}/health/ready</c>, plus
|
||||
/// <c>{BaseUrl}/health/active</c> when <see cref="InstanceEntry.HasActiveRole"/>. There is
|
||||
/// deliberately no per-instance probe-mode switch: one probe model, one status derivation.
|
||||
/// </remarks>
|
||||
public sealed class OverviewOptions
|
||||
{
|
||||
/// <summary>The configuration section this binds to.</summary>
|
||||
public const string SectionName = "Overview";
|
||||
|
||||
/// <summary>How often the poller sweeps every instance, in seconds.</summary>
|
||||
public int PollIntervalSeconds { get; set; } = 10;
|
||||
|
||||
/// <summary>Per-request timeout, in seconds.</summary>
|
||||
public int TimeoutSeconds { get; set; } = 3;
|
||||
|
||||
/// <summary>
|
||||
/// How old a snapshot may get before its instance renders as Stale (out of date / offline)
|
||||
/// regardless of its last known status, in seconds. This is what catches a stuck poller, a
|
||||
/// paused refresh or a slept machine — not just failed probes — so it must exceed the poll
|
||||
/// interval or every instance would read Stale between two healthy polls.
|
||||
/// </summary>
|
||||
public int StaleAfterSeconds { get; set; } = 45;
|
||||
|
||||
/// <summary>The registered applications, each with one or more instances.</summary>
|
||||
public IList<ApplicationEntry> Applications { get; set; } = new List<ApplicationEntry>();
|
||||
}
|
||||
|
||||
/// <summary>One registered application (a section on the dashboard).</summary>
|
||||
public sealed class ApplicationEntry
|
||||
{
|
||||
/// <summary>Display name, unique across the registry.</summary>
|
||||
public string Name { get; set; } = string.Empty;
|
||||
|
||||
/// <summary>Label for the per-instance management link, e.g. "AdminUI" or "Dashboard".</summary>
|
||||
public string ManagementLabel { get; set; } = "UI";
|
||||
|
||||
/// <summary>Overrides <see cref="OverviewOptions.PollIntervalSeconds"/> for this application.</summary>
|
||||
public int? PollIntervalSeconds { get; set; }
|
||||
|
||||
/// <summary>Overrides <see cref="OverviewOptions.TimeoutSeconds"/> for this application.</summary>
|
||||
public int? TimeoutSeconds { get; set; }
|
||||
|
||||
/// <summary>Overrides <see cref="OverviewOptions.StaleAfterSeconds"/> for this application.</summary>
|
||||
public int? StaleAfterSeconds { get; set; }
|
||||
|
||||
/// <summary>This application's instances; at least one is required.</summary>
|
||||
public IList<InstanceEntry> Instances { get; set; } = new List<InstanceEntry>();
|
||||
}
|
||||
|
||||
/// <summary>One probed instance (a card on the dashboard).</summary>
|
||||
public sealed class InstanceEntry
|
||||
{
|
||||
/// <summary>Display name, unique within its application.</summary>
|
||||
public string Name { get; set; } = string.Empty;
|
||||
|
||||
/// <summary>
|
||||
/// The cluster group this instance belongs to (e.g. "central", "site-a"). One Akka cluster is
|
||||
/// one group: leader aggregation and the split-brain check are scoped to it. Null for
|
||||
/// standalone instances.
|
||||
/// </summary>
|
||||
public string? Group { get; set; }
|
||||
|
||||
/// <summary>Absolute base URL whose <c>/health/*</c> endpoints are probed.</summary>
|
||||
public string BaseUrl { get; set; } = string.Empty;
|
||||
|
||||
/// <summary>Absolute URL of this instance's own management UI, linked from its card.</summary>
|
||||
public string? ManagementUrl { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Whether this instance participates in an active/standby pair. False for the single-instance
|
||||
/// gateways, which have no such concept — their cards show status without an Active badge and
|
||||
/// are never probed for <c>/health/active</c>.
|
||||
/// </summary>
|
||||
public bool HasActiveRole { get; set; }
|
||||
|
||||
/// <summary>Overrides the application and global poll interval for this instance.</summary>
|
||||
public int? PollIntervalSeconds { get; set; }
|
||||
|
||||
/// <summary>Overrides the application and global request timeout for this instance.</summary>
|
||||
public int? TimeoutSeconds { get; set; }
|
||||
|
||||
/// <summary>Overrides the application and global staleness window for this instance.</summary>
|
||||
public int? StaleAfterSeconds { get; set; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The effective (override-resolved) timings for one instance. Instance wins over application,
|
||||
/// application wins over global — resolved once here so no caller re-implements the precedence.
|
||||
/// </summary>
|
||||
/// <param name="PollInterval">Effective poll cadence.</param>
|
||||
/// <param name="Timeout">Effective per-request timeout.</param>
|
||||
/// <param name="StaleAfter">Effective staleness window.</param>
|
||||
public readonly record struct EffectiveTimings(TimeSpan PollInterval, TimeSpan Timeout, TimeSpan StaleAfter)
|
||||
{
|
||||
/// <summary>Resolves the effective timings for <paramref name="instance"/>.</summary>
|
||||
/// <param name="options">The global registry defaults.</param>
|
||||
/// <param name="application">The instance's application.</param>
|
||||
/// <param name="instance">The instance itself.</param>
|
||||
/// <returns>The resolved timings.</returns>
|
||||
public static EffectiveTimings Resolve(
|
||||
OverviewOptions options,
|
||||
ApplicationEntry application,
|
||||
InstanceEntry instance)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
ArgumentNullException.ThrowIfNull(application);
|
||||
ArgumentNullException.ThrowIfNull(instance);
|
||||
|
||||
return new EffectiveTimings(
|
||||
TimeSpan.FromSeconds(
|
||||
instance.PollIntervalSeconds ?? application.PollIntervalSeconds ?? options.PollIntervalSeconds),
|
||||
TimeSpan.FromSeconds(
|
||||
instance.TimeoutSeconds ?? application.TimeoutSeconds ?? options.TimeoutSeconds),
|
||||
TimeSpan.FromSeconds(
|
||||
instance.StaleAfterSeconds ?? application.StaleAfterSeconds ?? options.StaleAfterSeconds));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
using ZB.MOM.WW.Configuration;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Registry;
|
||||
|
||||
/// <summary>
|
||||
/// Validates the <c>Overview</c> registry. Registered through
|
||||
/// <c>AddValidatedOptions<OverviewOptions, OverviewOptionsValidator></c>, so a malformed
|
||||
/// registry fails the host at startup rather than surfacing as a silently-empty dashboard or a
|
||||
/// first-poll exception.
|
||||
/// </summary>
|
||||
public sealed class OverviewOptionsValidator : OptionsValidatorBase<OverviewOptions>
|
||||
{
|
||||
/// <inheritdoc />
|
||||
protected override void Validate(ValidationBuilder builder, OverviewOptions options)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(builder);
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
RequirePositive(builder, options.PollIntervalSeconds, "Overview:PollIntervalSeconds");
|
||||
RequirePositive(builder, options.TimeoutSeconds, "Overview:TimeoutSeconds");
|
||||
RequirePositive(builder, options.StaleAfterSeconds, "Overview:StaleAfterSeconds");
|
||||
|
||||
builder.MinCount(options.Applications as IReadOnlyCollection<ApplicationEntry>, 1, "Overview:Applications");
|
||||
|
||||
var seenApplications = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
for (var a = 0; a < options.Applications.Count; a++)
|
||||
{
|
||||
var application = options.Applications[a];
|
||||
var appPath = $"Overview:Applications:{a}";
|
||||
|
||||
builder.Required(application.Name, $"{appPath}:Name");
|
||||
if (!string.IsNullOrWhiteSpace(application.Name) && !seenApplications.Add(application.Name))
|
||||
builder.Add($"{appPath}:Name '{application.Name}' is duplicated; application names must be unique");
|
||||
|
||||
RequirePositiveIfSet(builder, application.PollIntervalSeconds, $"{appPath}:PollIntervalSeconds");
|
||||
RequirePositiveIfSet(builder, application.TimeoutSeconds, $"{appPath}:TimeoutSeconds");
|
||||
RequirePositiveIfSet(builder, application.StaleAfterSeconds, $"{appPath}:StaleAfterSeconds");
|
||||
|
||||
builder.MinCount(
|
||||
application.Instances as IReadOnlyCollection<InstanceEntry>, 1, $"{appPath}:Instances");
|
||||
|
||||
var seenInstances = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
for (var i = 0; i < application.Instances.Count; i++)
|
||||
{
|
||||
var instance = application.Instances[i];
|
||||
var path = $"{appPath}:Instances:{i}";
|
||||
|
||||
builder.Required(instance.Name, $"{path}:Name");
|
||||
if (!string.IsNullOrWhiteSpace(instance.Name) && !seenInstances.Add(instance.Name))
|
||||
{
|
||||
builder.Add(
|
||||
$"{path}:Name '{instance.Name}' is duplicated within application "
|
||||
+ $"'{application.Name}'; instance names must be unique per application");
|
||||
}
|
||||
|
||||
RequireAbsoluteHttpUrl(builder, instance.BaseUrl, $"{path}:BaseUrl", required: true);
|
||||
RequireAbsoluteHttpUrl(builder, instance.ManagementUrl, $"{path}:ManagementUrl", required: false);
|
||||
|
||||
RequirePositiveIfSet(builder, instance.PollIntervalSeconds, $"{path}:PollIntervalSeconds");
|
||||
RequirePositiveIfSet(builder, instance.TimeoutSeconds, $"{path}:TimeoutSeconds");
|
||||
RequirePositiveIfSet(builder, instance.StaleAfterSeconds, $"{path}:StaleAfterSeconds");
|
||||
|
||||
// Checked on the EFFECTIVE values, not the globals: an override at either level can
|
||||
// invert the relationship on one instance while the global pair stays sane, and an
|
||||
// instance whose stale window is inside its poll interval would render Stale
|
||||
// between two perfectly healthy polls.
|
||||
var timings = EffectiveTimings.Resolve(options, application, instance);
|
||||
if (timings.StaleAfter <= timings.PollInterval)
|
||||
{
|
||||
builder.Add(
|
||||
$"{path}: effective StaleAfterSeconds ({timings.StaleAfter.TotalSeconds:0.##}) must be "
|
||||
+ $"greater than effective PollIntervalSeconds ({timings.PollInterval.TotalSeconds:0.##}) "
|
||||
+ "— otherwise this instance renders Stale between two healthy polls");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static void RequirePositive(ValidationBuilder builder, int value, string field) =>
|
||||
builder.RequireThat(value > 0, $"{field} must be greater than 0");
|
||||
|
||||
private static void RequirePositiveIfSet(ValidationBuilder builder, int? value, string field)
|
||||
{
|
||||
if (value is { } set)
|
||||
RequirePositive(builder, set, field);
|
||||
}
|
||||
|
||||
private static void RequireAbsoluteHttpUrl(
|
||||
ValidationBuilder builder, string? value, string field, bool required)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(value))
|
||||
{
|
||||
if (required)
|
||||
builder.Add($"{field} is required");
|
||||
return;
|
||||
}
|
||||
|
||||
// Absolute + http(s) explicitly: a relative URL binds without complaint and only fails at
|
||||
// the first poll, as an opaque HttpClient error against a dashboard that has no base
|
||||
// address of its own.
|
||||
if (!Uri.TryCreate(value, UriKind.Absolute, out var uri)
|
||||
|| (uri.Scheme != Uri.UriSchemeHttp && uri.Scheme != Uri.UriSchemeHttps))
|
||||
{
|
||||
builder.Add($"{field} must be an absolute http:// or https:// URL (was '{value}')");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk.Web">
|
||||
|
||||
<!--
|
||||
The family overview dashboard: a Blazor Server app that polls every registered instance's
|
||||
anonymous /health/ready + /health/active endpoints and renders one cached page.
|
||||
|
||||
Inline pinned versions, deliberately NO central package management: that is the app
|
||||
convention in this family (mxgw and HistorianGateway carry inline pins too, and a
|
||||
CPM-shaped sweep silently reports such repos as empty).
|
||||
|
||||
NO Auth / Audit / Secrets packages. The dashboard is anonymous and read-only by
|
||||
requirement — there is no login, no mutating action to audit, and nothing secret in the
|
||||
registry. That is a requirement, not an omission; do not add them "while at it".
|
||||
-->
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<RootNamespace>ZB.MOM.WW.Overview</RootNamespace>
|
||||
<UserSecretsId>zb-mom-ww-overview</UserSecretsId>
|
||||
</PropertyGroup>
|
||||
|
||||
<!-- Exposes the polling internals (snapshot store, status derivation, leader resolution)
|
||||
to the test project so they are testable without going through the UI. -->
|
||||
<ItemGroup>
|
||||
<InternalsVisibleTo Include="ZB.MOM.WW.Overview.Tests" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="ZB.MOM.WW.Theme" Version="0.3.1" />
|
||||
<PackageReference Include="ZB.MOM.WW.Health" Version="0.2.0" />
|
||||
<PackageReference Include="ZB.MOM.WW.Telemetry" Version="0.1.0" />
|
||||
<PackageReference Include="ZB.MOM.WW.Telemetry.Serilog" Version="0.1.0" />
|
||||
<PackageReference Include="ZB.MOM.WW.Configuration" Version="0.1.0" />
|
||||
<PackageReference Include="Serilog.AspNetCore" Version="10.0.0" />
|
||||
<PackageReference Include="Serilog.Sinks.Console" Version="6.1.1" />
|
||||
<PackageReference Include="Serilog.Sinks.File" Version="7.0.0" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
@@ -0,0 +1,89 @@
|
||||
{
|
||||
"Logging": {
|
||||
"LogLevel": {
|
||||
"Default": "Information",
|
||||
"Microsoft.AspNetCore": "Warning"
|
||||
}
|
||||
},
|
||||
|
||||
// Kestrel endpoint config, NOT ASPNETCORE_URLS. Setting both puts Kestrel into
|
||||
// explicit-endpoints mode and silently discards one of them — the family's recurring
|
||||
// URLs-override trap.
|
||||
"Kestrel": {
|
||||
"Endpoints": {
|
||||
"Http": {
|
||||
"Url": "http://localhost:5320"
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
// Development registry for `dotnet run` ON THE MAC HOST. This is deliberately a SUBSET of the
|
||||
// fleet: the dev rigs publish only some node ports to the host, so these are the instances a
|
||||
// host-run dashboard can genuinely reach. Verified against the running containers 2026-07-24.
|
||||
//
|
||||
// ScadaBridge central pair localhost:9001 / :9002 (container :5000 published)
|
||||
// ScadaBridge site nodes NOT published — their health port (8084) is container-internal
|
||||
// OtOpcUa (all six nodes) NOT published — only Traefik's load-balanced :9200, which
|
||||
// round-robins the central pair and so cannot identify a node
|
||||
// HistorianGateway localhost:5220
|
||||
// MxAccessGateway windev 10.100.0.48:5130 (needs the VPN up)
|
||||
//
|
||||
// Run the containerised dashboard (docker/, environment "Docker") to see the whole fleet: it
|
||||
// joins the rig networks and addresses every node by container DNS name.
|
||||
"Overview": {
|
||||
"PollIntervalSeconds": 10,
|
||||
"TimeoutSeconds": 3,
|
||||
"StaleAfterSeconds": 45,
|
||||
"Applications": [
|
||||
{
|
||||
"Name": "ScadaBridge",
|
||||
"ManagementLabel": "CentralUI",
|
||||
"Instances": [
|
||||
{
|
||||
"Name": "central-a",
|
||||
"Group": "central",
|
||||
"BaseUrl": "http://localhost:9001",
|
||||
"ManagementUrl": "http://localhost:9001/",
|
||||
"HasActiveRole": true
|
||||
},
|
||||
{
|
||||
"Name": "central-b",
|
||||
"Group": "central",
|
||||
"BaseUrl": "http://localhost:9002",
|
||||
"ManagementUrl": "http://localhost:9002/",
|
||||
"HasActiveRole": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"Name": "HistorianGateway",
|
||||
"ManagementLabel": "Dashboard",
|
||||
"Instances": [
|
||||
{
|
||||
"Name": "histgw",
|
||||
"BaseUrl": "http://localhost:5220",
|
||||
"ManagementUrl": "http://localhost:5220/",
|
||||
"HasActiveRole": false,
|
||||
|
||||
// Its readiness probe reaches the real historian over the VPN; when that is down the
|
||||
// probe hangs rather than refusing, so it needs more room than the 3s default before
|
||||
// it is honestly called unreachable.
|
||||
"TimeoutSeconds": 10
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"Name": "MxAccessGateway",
|
||||
"ManagementLabel": "Dashboard",
|
||||
"Instances": [
|
||||
{
|
||||
"Name": "windev",
|
||||
"BaseUrl": "http://10.100.0.48:5130",
|
||||
"ManagementUrl": "http://10.100.0.48:5130/",
|
||||
"HasActiveRole": false
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,163 @@
|
||||
{
|
||||
"Logging": {
|
||||
"LogLevel": {
|
||||
"Default": "Information",
|
||||
"Microsoft.AspNetCore": "Warning"
|
||||
}
|
||||
},
|
||||
|
||||
"Kestrel": {
|
||||
"Endpoints": {
|
||||
"Http": {
|
||||
"Url": "http://+:5320"
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
// Full-fleet registry for the CONTAINERISED dashboard, which joins the rig networks
|
||||
// (otopcua-dev_default, scadabridge-net, zb-historiangw_default) and therefore reaches every
|
||||
// node by container DNS name — including the ones whose ports are never published to the host.
|
||||
//
|
||||
// Ports below were verified by probing the running rigs on 2026-07-24, not read off the compose
|
||||
// files, because the two disagree:
|
||||
// OtOpcUa central-1/2 :9000 (ASPNETCORE_URLS)
|
||||
// OtOpcUa site-* :8080 (NOT 9000 — the site nodes' explicit Kestrel listeners for
|
||||
// LocalDb sync re-bind the primary HTTP port)
|
||||
// ScadaBridge central :5000
|
||||
// ScadaBridge site :8084 (MetricsPort; MapZbHealth lands here from the site-health work)
|
||||
//
|
||||
// ManagementUrl values stay host-relative (localhost:*) on purpose: they are followed by the
|
||||
// OPERATOR'S BROWSER, not by this container, so container DNS names would be dead links.
|
||||
"Overview": {
|
||||
"PollIntervalSeconds": 10,
|
||||
"TimeoutSeconds": 3,
|
||||
"StaleAfterSeconds": 45,
|
||||
"Applications": [
|
||||
{
|
||||
"Name": "ScadaBridge",
|
||||
"ManagementLabel": "CentralUI",
|
||||
"Instances": [
|
||||
{
|
||||
"Name": "central-a",
|
||||
"Group": "central",
|
||||
"BaseUrl": "http://scadabridge-central-a:5000",
|
||||
"ManagementUrl": "http://localhost:9001/",
|
||||
"HasActiveRole": true
|
||||
},
|
||||
{
|
||||
"Name": "central-b",
|
||||
"Group": "central",
|
||||
"BaseUrl": "http://scadabridge-central-b:5000",
|
||||
"ManagementUrl": "http://localhost:9002/",
|
||||
"HasActiveRole": true
|
||||
},
|
||||
{
|
||||
"Name": "site-a-a",
|
||||
"Group": "site-a",
|
||||
"BaseUrl": "http://scadabridge-site-a-a:8084",
|
||||
"HasActiveRole": true
|
||||
},
|
||||
{
|
||||
"Name": "site-a-b",
|
||||
"Group": "site-a",
|
||||
"BaseUrl": "http://scadabridge-site-a-b:8084",
|
||||
"HasActiveRole": true
|
||||
},
|
||||
{
|
||||
"Name": "site-b-a",
|
||||
"Group": "site-b",
|
||||
"BaseUrl": "http://scadabridge-site-b-a:8084",
|
||||
"HasActiveRole": true
|
||||
},
|
||||
{
|
||||
"Name": "site-b-b",
|
||||
"Group": "site-b",
|
||||
"BaseUrl": "http://scadabridge-site-b-b:8084",
|
||||
"HasActiveRole": true
|
||||
},
|
||||
{
|
||||
"Name": "site-c-a",
|
||||
"Group": "site-c",
|
||||
"BaseUrl": "http://scadabridge-site-c-a:8084",
|
||||
"HasActiveRole": true
|
||||
},
|
||||
{
|
||||
"Name": "site-c-b",
|
||||
"Group": "site-c",
|
||||
"BaseUrl": "http://scadabridge-site-c-b:8084",
|
||||
"HasActiveRole": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"Name": "OtOpcUa",
|
||||
"ManagementLabel": "AdminUI",
|
||||
"Instances": [
|
||||
{
|
||||
"Name": "central-1",
|
||||
"Group": "MAIN",
|
||||
"BaseUrl": "http://central-1:9000",
|
||||
"ManagementUrl": "http://localhost:9200/",
|
||||
"HasActiveRole": true
|
||||
},
|
||||
{
|
||||
"Name": "central-2",
|
||||
"Group": "MAIN",
|
||||
"BaseUrl": "http://central-2:9000",
|
||||
"ManagementUrl": "http://localhost:9200/",
|
||||
"HasActiveRole": true
|
||||
},
|
||||
{
|
||||
"Name": "site-a-1",
|
||||
"Group": "SITE-A",
|
||||
"BaseUrl": "http://site-a-1:8080",
|
||||
"HasActiveRole": true
|
||||
},
|
||||
{
|
||||
"Name": "site-a-2",
|
||||
"Group": "SITE-A",
|
||||
"BaseUrl": "http://site-a-2:8080",
|
||||
"HasActiveRole": true
|
||||
},
|
||||
{
|
||||
"Name": "site-b-1",
|
||||
"Group": "SITE-B",
|
||||
"BaseUrl": "http://site-b-1:8080",
|
||||
"HasActiveRole": true
|
||||
},
|
||||
{
|
||||
"Name": "site-b-2",
|
||||
"Group": "SITE-B",
|
||||
"BaseUrl": "http://site-b-2:8080",
|
||||
"HasActiveRole": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"Name": "HistorianGateway",
|
||||
"ManagementLabel": "Dashboard",
|
||||
"Instances": [
|
||||
{
|
||||
"Name": "histgw",
|
||||
"BaseUrl": "http://zb-historiangw:5220",
|
||||
"ManagementUrl": "http://localhost:5220/",
|
||||
"HasActiveRole": false,
|
||||
"TimeoutSeconds": 10
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"Name": "MxAccessGateway",
|
||||
"ManagementLabel": "Dashboard",
|
||||
"Instances": [
|
||||
{
|
||||
"Name": "windev",
|
||||
"BaseUrl": "http://10.100.0.48:5130",
|
||||
"ManagementUrl": "http://10.100.0.48:5130/",
|
||||
"HasActiveRole": false
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
{
|
||||
"Logging": {
|
||||
"LogLevel": {
|
||||
"Default": "Information",
|
||||
"Microsoft.AspNetCore": "Warning"
|
||||
}
|
||||
},
|
||||
"AllowedHosts": "*",
|
||||
|
||||
// AddZbSerilog drives its sinks entirely from this section (ReadFrom.Configuration) and adds
|
||||
// none of its own. An empty "Serilog": {} therefore produces an app that logs NOWHERE — verified
|
||||
// against the running HistorianGateway container, whose log stream is completely empty for
|
||||
// exactly this reason. Declaring the sinks here is what keeps this app diagnosable.
|
||||
"Serilog": {
|
||||
// Every probe emits four Information-level HttpClient lines. At 16 instances on a 10s cadence
|
||||
// that is several lines a second of pure noise from an app whose entire job is polling — it
|
||||
// would bury anything worth reading. Note this override, not Logging:LogLevel, is what governs
|
||||
// once Serilog takes over.
|
||||
"MinimumLevel": {
|
||||
"Default": "Information",
|
||||
"Override": {
|
||||
"Microsoft.AspNetCore": "Warning",
|
||||
"System.Net.Http.HttpClient": "Warning"
|
||||
}
|
||||
},
|
||||
"Using": [
|
||||
"Serilog.Sinks.Console",
|
||||
"Serilog.Sinks.File"
|
||||
],
|
||||
"WriteTo": [
|
||||
{ "Name": "Console" },
|
||||
{
|
||||
"Name": "File",
|
||||
"Args": {
|
||||
"path": "logs/overview-.log",
|
||||
"rollingInterval": "Day"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
|
||||
"Telemetry": {},
|
||||
|
||||
// Tuning defaults only. `Applications` is deliberately EMPTY here: the registry is a
|
||||
// per-deployment fact, and shipping a placeholder fleet would let a misconfigured deployment
|
||||
// boot and render instances nobody asked about. With no registry supplied, ConfigPreflight
|
||||
// fails startup and names the missing key. See docker/README.md for the full shape.
|
||||
"Overview": {
|
||||
"PollIntervalSeconds": 10,
|
||||
"TimeoutSeconds": 3,
|
||||
"StaleAfterSeconds": 45,
|
||||
"Applications": []
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,112 @@
|
||||
/* ═══════════════════════════════════════════════════════════════════════════
|
||||
ZB.MOM.WW.Overview — app-local styles.
|
||||
|
||||
Everything here is what the shared ZB.MOM.WW.Theme kit does NOT already
|
||||
provide. The kit owns the design tokens, the shell, and the generic
|
||||
vocabulary this file builds on: .chip-*, .panel, .panel-foot, .kv, .agg-*,
|
||||
.card-grid, .conn-pill, .rail-btn, .mono, .rise. None of those are
|
||||
redefined here — this file adds only the dashboard-specific anatomy from
|
||||
docs/mockups/overview-dashboard-mockup.html (page-head, app/group headers,
|
||||
the instance card, the check-detail list).
|
||||
═══════════════════════════════════════════════════════════════════════════ */
|
||||
|
||||
/* ── Page header ────────────────────────────────────────────────────────── */
|
||||
.page-head { display: flex; align-items: baseline; gap: 1rem; flex-wrap: wrap; margin: 0 0 1rem; }
|
||||
.page-head h1 { font-size: 1.25rem; font-weight: 600; letter-spacing: 0.01em; margin: 0; text-wrap: balance; }
|
||||
.page-head .spacer { flex: 1; }
|
||||
.page-meta { font-family: var(--mono); font-size: 0.78rem; color: var(--ink-soft); }
|
||||
|
||||
/* KPI variants the kit does not carry (it ships .alert / .caution only). */
|
||||
.agg-card.good .agg-value { color: var(--ok); }
|
||||
.agg-card.quiet .agg-value { color: var(--idle); }
|
||||
.kv .v.idle { color: var(--idle); }
|
||||
|
||||
/* ── Application sections ───────────────────────────────────────────────── */
|
||||
.app-section { margin-bottom: 1.5rem; }
|
||||
.app-head {
|
||||
display: flex; align-items: baseline; gap: 0.6rem; flex-wrap: wrap;
|
||||
padding-bottom: 0.4rem; border-bottom: 1px solid var(--rule-strong); margin-bottom: 0.75rem;
|
||||
}
|
||||
.app-head h2 { font-size: 1.02rem; font-weight: 600; margin: 0; }
|
||||
.app-head .spacer { flex: 1; }
|
||||
.app-meta { font-family: var(--mono); font-size: 0.76rem; color: var(--ink-faint); }
|
||||
|
||||
.group { margin-bottom: 0.9rem; }
|
||||
.group-head { display: flex; align-items: center; gap: 0.55rem; flex-wrap: wrap; margin-bottom: 0.5rem; }
|
||||
.group-name {
|
||||
font-size: 0.74rem; font-weight: 600; text-transform: uppercase;
|
||||
letter-spacing: 0.07em; color: var(--ink-soft);
|
||||
}
|
||||
.group-meta { font-family: var(--mono); font-size: 0.74rem; color: var(--ink-faint); }
|
||||
|
||||
/* ── Instance card ──────────────────────────────────────────────────────── */
|
||||
.inst {
|
||||
background: var(--card); border: 1px solid var(--rule); border-radius: 8px;
|
||||
overflow: hidden; display: flex; flex-direction: column;
|
||||
}
|
||||
|
||||
/* Status is carried by the top stripe. Unreachable additionally goes DASHED:
|
||||
colour alone cannot distinguish it from Down, and the two send an operator
|
||||
to completely different places (network vs. the application itself). */
|
||||
.inst.up { border-top: 3px solid var(--ok); }
|
||||
.inst.degraded { border-top: 3px solid var(--warn); }
|
||||
.inst.down { border-top: 3px solid var(--bad); }
|
||||
.inst.unreachable { border-top: 3px solid var(--bad); border-style: dashed; border-top-style: solid; }
|
||||
.inst.pending { border-top: 3px solid var(--rule-strong); }
|
||||
.inst.stale { border-top: 3px solid var(--idle); }
|
||||
.inst.stale .inst-body,
|
||||
.inst.stale .inst-head .inst-host { opacity: 0.55; }
|
||||
|
||||
.inst-head {
|
||||
display: flex; align-items: flex-start; justify-content: space-between;
|
||||
gap: 0.6rem; padding: 0.6rem 0.9rem 0.45rem;
|
||||
}
|
||||
.inst-name { font-weight: 600; font-size: 0.92rem; }
|
||||
.inst-host {
|
||||
display: block; font-family: var(--mono); font-size: 0.72rem;
|
||||
color: var(--ink-faint); margin-top: 0.15rem;
|
||||
}
|
||||
.inst-chips { display: flex; gap: 0.3rem; flex-wrap: wrap; justify-content: flex-end; }
|
||||
|
||||
.inst .panel-foot {
|
||||
margin-top: auto; display: flex; align-items: center;
|
||||
justify-content: space-between; gap: 0.6rem;
|
||||
}
|
||||
.foot-note { font-family: var(--mono); font-size: 0.72rem; color: var(--ink-faint); }
|
||||
|
||||
/* ── Check detail (native <details>, no JS) ─────────────────────────────── */
|
||||
details.checks { border-top: 1px solid var(--rule); }
|
||||
details.checks summary {
|
||||
list-style: none; cursor: pointer; padding: 0.45rem 0.9rem; font-size: 0.72rem;
|
||||
font-weight: 600; text-transform: uppercase; letter-spacing: 0.06em;
|
||||
color: var(--ink-faint); user-select: none;
|
||||
}
|
||||
details.checks summary::-webkit-details-marker { display: none; }
|
||||
details.checks summary::before { content: '\25B6'; font-size: 0.55rem; margin-right: 0.4rem; }
|
||||
details.checks[open] summary::before { content: '\25BC'; }
|
||||
details.checks summary:hover { color: var(--ink); }
|
||||
|
||||
.check-row { display: flex; align-items: baseline; gap: 0.5rem; padding: 0.28rem 0.9rem; font-size: 0.8rem; }
|
||||
.check-row:nth-child(even) { background: var(--zebra-bg); }
|
||||
.check-name { font-family: var(--mono); font-size: 0.78rem; min-width: 9.5rem; }
|
||||
.check-desc { color: var(--ink-soft); font-size: 0.78rem; }
|
||||
.check-dot { width: 8px; height: 8px; border-radius: 2px; flex: 0 0 8px; align-self: center; }
|
||||
.check-dot.ok { background: var(--ok); }
|
||||
.check-dot.warn { background: var(--warn); }
|
||||
.check-dot.bad { background: var(--bad); }
|
||||
.check-dot.idle { background: var(--idle); }
|
||||
|
||||
.check-data {
|
||||
margin: 0.35rem 0.9rem 0.6rem; padding: 0.4rem 0.6rem; background: var(--zebra-bg);
|
||||
border: 1px solid var(--rule); border-radius: 4px;
|
||||
font-family: var(--mono); font-size: 0.72rem; color: var(--ink-soft); line-height: 1.6;
|
||||
/* Scrolls inside its own box: an Akka address plus member counts is wider than a
|
||||
310px card, and letting it widen the card would break the whole grid. */
|
||||
overflow-x: auto; white-space: nowrap;
|
||||
}
|
||||
.check-data b { color: var(--accent-deep); font-weight: 500; }
|
||||
|
||||
.rail-note {
|
||||
padding: 0 0.6rem 0.6rem; font-family: var(--mono); font-size: 0.72rem;
|
||||
color: var(--ink-faint); line-height: 1.5;
|
||||
}
|
||||
+6
File diff suppressed because one or more lines are too long
+7
File diff suppressed because one or more lines are too long
@@ -0,0 +1,193 @@
|
||||
using System.Net;
|
||||
using Microsoft.AspNetCore.Mvc.Testing;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using ZB.MOM.WW.Overview.Observability;
|
||||
using ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Boots the REAL <c>Program.cs</c> and asks the questions no in-process test can: does every
|
||||
/// endpoint answer without a login, and does a bad registry stop the host rather than produce a
|
||||
/// dashboard full of nothing.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The registry is supplied through environment variables rather than
|
||||
/// <c>ConfigureAppConfiguration</c>. <c>ConfigPreflight</c> runs against
|
||||
/// <c>builder.Configuration</c> in the top-level statements, before <c>Build()</c> — which is when
|
||||
/// the factory's configuration callbacks are applied — so a callback-supplied registry would arrive
|
||||
/// too late for the very check these tests are about. Environment variables are read by
|
||||
/// <c>CreateBuilder</c> itself, so they are in place from the first line.
|
||||
///
|
||||
/// <para>
|
||||
/// Environment variables are process-global, so every test here lives in one class: xunit runs a
|
||||
/// class's tests sequentially, which is what keeps two hosts from fighting over the same keys.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class BootTests : IDisposable
|
||||
{
|
||||
private readonly List<string> _keys = [];
|
||||
|
||||
private void Set(string key, string? value)
|
||||
{
|
||||
_keys.Add(key);
|
||||
Environment.SetEnvironmentVariable(key, value);
|
||||
}
|
||||
|
||||
/// <summary>Restores the environment so a later-running test class sees a clean process.</summary>
|
||||
public void Dispose()
|
||||
{
|
||||
foreach (var key in _keys)
|
||||
Environment.SetEnvironmentVariable(key, null);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A registry with one instance pointing at a port nothing listens on. The dashboard must boot
|
||||
/// and serve regardless of whether anything it watches is up — that is the entire premise of
|
||||
/// rendering from cache.
|
||||
/// </summary>
|
||||
private void SetValidRegistry()
|
||||
{
|
||||
Set("ASPNETCORE_ENVIRONMENT", "Testing");
|
||||
Set("Overview__PollIntervalSeconds", "10");
|
||||
Set("Overview__TimeoutSeconds", "1");
|
||||
Set("Overview__StaleAfterSeconds", "45");
|
||||
Set("Overview__Applications__0__Name", "TestApp");
|
||||
Set("Overview__Applications__0__ManagementLabel", "UI");
|
||||
Set("Overview__Applications__0__Instances__0__Name", "node-a");
|
||||
Set("Overview__Applications__0__Instances__0__BaseUrl", "http://127.0.0.1:1");
|
||||
Set("Overview__Applications__0__Instances__0__HasActiveRole", "false");
|
||||
}
|
||||
|
||||
private static WebApplicationFactory<Program> Factory() => new();
|
||||
|
||||
[Fact]
|
||||
public async Task Page_IsServedAnonymously()
|
||||
{
|
||||
SetValidRegistry();
|
||||
using var factory = Factory();
|
||||
|
||||
// AllowAutoRedirect off so an auth redirect would surface as a 302 here rather than being
|
||||
// silently followed to a login page that then returns 200.
|
||||
using var client = factory.CreateClient(new WebApplicationFactoryClientOptions
|
||||
{
|
||||
AllowAutoRedirect = false,
|
||||
});
|
||||
|
||||
using var response = await client.GetAsync("/");
|
||||
|
||||
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
|
||||
Assert.Contains("SCADA Overview", await response.Content.ReadAsStringAsync(), StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Page_RendersEvenThoughNothingItWatchesIsReachable()
|
||||
{
|
||||
// Cache-first, stated as a test: the registry above points at a dead port, and the page
|
||||
// must still paint its full layout instead of waiting on a sweep.
|
||||
SetValidRegistry();
|
||||
using var factory = Factory();
|
||||
using var client = factory.CreateClient();
|
||||
|
||||
var html = await client.GetStringAsync("/");
|
||||
|
||||
Assert.Contains("data-testid=\"instance-card\"", html, StringComparison.Ordinal);
|
||||
Assert.Contains("node-a", html, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("/health/ready")]
|
||||
[InlineData("/health/active")]
|
||||
[InlineData("/healthz")]
|
||||
[InlineData("/metrics")]
|
||||
public async Task HealthAndMetrics_AreServedAnonymously(string path)
|
||||
{
|
||||
SetValidRegistry();
|
||||
using var factory = Factory();
|
||||
using var client = factory.CreateClient(new WebApplicationFactoryClientOptions
|
||||
{
|
||||
AllowAutoRedirect = false,
|
||||
});
|
||||
|
||||
using var response = await client.GetAsync(path);
|
||||
|
||||
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Ready_ReportsTheDashboardsOwnChecks()
|
||||
{
|
||||
SetValidRegistry();
|
||||
using var factory = Factory();
|
||||
using var client = factory.CreateClient();
|
||||
|
||||
var body = await client.GetStringAsync("/health/ready");
|
||||
|
||||
Assert.Contains(ObservabilityServiceCollectionExtensions.RegistryLoadedCheckName, body, StringComparison.Ordinal);
|
||||
Assert.Contains(ObservabilityServiceCollectionExtensions.PollCycleCheckName, body, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Metrics_ExportsTheAppsOwnMeter()
|
||||
{
|
||||
// The Meters allowlist in AddZbTelemetry is silent when it is wrong: a missing name drops
|
||||
// the app's whole instrument surface from /metrics with no warning anywhere. Only an
|
||||
// end-to-end scrape catches that.
|
||||
SetValidRegistry();
|
||||
using var factory = Factory();
|
||||
using var client = factory.CreateClient();
|
||||
|
||||
// Force a sweep so the observable gauge has a probed instance to report. The instance is
|
||||
// unreachable, which is a perfectly good status to publish.
|
||||
var poller = factory.Services.GetServices<IHostedService>().OfType<OverviewPollerService>().Single();
|
||||
await poller.SweepAsync();
|
||||
|
||||
var body = await client.GetStringAsync("/metrics");
|
||||
|
||||
Assert.Contains("overview_instance_status", body, StringComparison.Ordinal);
|
||||
Assert.Contains("node=\"node-a\"", body, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task MissingRegistry_FailsToBootAndNamesTheKey()
|
||||
{
|
||||
// appsettings.json ships an EMPTY Applications list precisely so this is the outcome when a
|
||||
// deployment forgets its registry — an empty dashboard that boots would look like a healthy
|
||||
// fleet of nothing.
|
||||
Set("ASPNETCORE_ENVIRONMENT", "Testing");
|
||||
|
||||
using var factory = Factory();
|
||||
|
||||
var error = await Record.ExceptionAsync(() => factory.CreateClient().GetAsync("/"));
|
||||
|
||||
Assert.NotNull(error);
|
||||
Assert.Contains("Overview:Applications:0:Name", Flatten(error), StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task StaleWindowInsideThePollInterval_FailsToBootWithTheValidatorMessage()
|
||||
{
|
||||
// Past preflight and into the validator: a staleness window shorter than the poll interval
|
||||
// would mark every instance Stale between two perfectly healthy polls.
|
||||
SetValidRegistry();
|
||||
Set("Overview__StaleAfterSeconds", "5");
|
||||
|
||||
using var factory = Factory();
|
||||
|
||||
var error = await Record.ExceptionAsync(() => factory.CreateClient().GetAsync("/"));
|
||||
|
||||
Assert.NotNull(error);
|
||||
Assert.Contains("StaleAfterSeconds", Flatten(error), StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
private static string Flatten(Exception exception)
|
||||
{
|
||||
var text = new System.Text.StringBuilder();
|
||||
|
||||
for (var current = exception; current is not null; current = current.InnerException)
|
||||
text.AppendLine(current.Message);
|
||||
|
||||
return text.ToString();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,249 @@
|
||||
using Bunit;
|
||||
using ZB.MOM.WW.Overview.Components.Widgets;
|
||||
using ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// The instance card is where a reading becomes a decision, so the tests pin what an operator can
|
||||
/// actually distinguish at a glance: which of the four states is shown, whether it is still
|
||||
/// current, and whether the raw evidence behind it is visible.
|
||||
/// </summary>
|
||||
public class InstanceCardTests : TestContext
|
||||
{
|
||||
private static readonly DateTimeOffset Now = new(2026, 7, 24, 12, 0, 0, TimeSpan.Zero);
|
||||
|
||||
private static InstanceSnapshot Card(
|
||||
InstanceStatus? status = InstanceStatus.Up,
|
||||
ActiveState active = ActiveState.NotApplicable,
|
||||
DateTimeOffset? lastPolled = null,
|
||||
DateTimeOffset? lastReachable = null,
|
||||
TimeSpan? staleAfter = null,
|
||||
string? error = null,
|
||||
ZbHealthReport? report = null,
|
||||
string? managementUrl = null) =>
|
||||
new()
|
||||
{
|
||||
Application = "ScadaBridge",
|
||||
Instance = "site-a-a",
|
||||
Group = "site-a",
|
||||
BaseUrl = "http://site-a-a:8084",
|
||||
ManagementUrl = managementUrl,
|
||||
ManagementLabel = "AdminUI",
|
||||
HasActiveRole = active != ActiveState.NotApplicable,
|
||||
Status = status,
|
||||
Active = active,
|
||||
LastPolledUtc = lastPolled ?? Now,
|
||||
LastReachableUtc = lastReachable ?? Now,
|
||||
Latency = TimeSpan.FromMilliseconds(18),
|
||||
Error = error,
|
||||
Report = report,
|
||||
StaleAfter = staleAfter ?? TimeSpan.FromSeconds(45),
|
||||
};
|
||||
|
||||
private IRenderedComponent<InstanceCard> Render(InstanceSnapshot instance, bool isLeader = false) =>
|
||||
RenderComponent<InstanceCard>(p => p
|
||||
.Add(c => c.Instance, instance)
|
||||
.Add(c => c.Now, Now)
|
||||
.Add(c => c.IsLeader, isLeader));
|
||||
|
||||
[Theory]
|
||||
[InlineData(InstanceStatus.Up, "up", "Up")]
|
||||
[InlineData(InstanceStatus.Degraded, "degraded", "Degraded")]
|
||||
[InlineData(InstanceStatus.Down, "down", "Down")]
|
||||
[InlineData(InstanceStatus.Unreachable, "unreachable", "Unreachable")]
|
||||
public void Card_CarriesItsStatusAsBothClassAndLabel(InstanceStatus status, string cssClass, string label)
|
||||
{
|
||||
var card = Render(Card(status));
|
||||
|
||||
Assert.Contains(cssClass, card.Find("article").ClassList, StringComparer.Ordinal);
|
||||
Assert.Contains(label, card.Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void UnreachableAndDown_AreVisuallyDistinct_NotJustDifferentWords()
|
||||
{
|
||||
// The CSS gives Unreachable a dashed border precisely because both are red. If the class
|
||||
// did not differ, the two would be indistinguishable on a wall display.
|
||||
var down = Render(Card(InstanceStatus.Down)).Find("article").ClassList;
|
||||
var unreachable = Render(Card(InstanceStatus.Unreachable)).Find("article").ClassList;
|
||||
|
||||
Assert.NotEqual(down, unreachable);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void NeverPolled_RendersPending_NotHealthy()
|
||||
{
|
||||
var card = Render(Card(status: null, lastPolled: null));
|
||||
|
||||
Assert.Contains("pending", card.Find("article").ClassList, StringComparer.Ordinal);
|
||||
Assert.Contains("Pending", card.Markup, StringComparison.Ordinal);
|
||||
Assert.DoesNotContain(">Up<", card.Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void StaleCard_IsMarkedStaleButKeepsItsLastKnownStatusClass()
|
||||
{
|
||||
// Both facts matter: the reading was Up, and it is no longer current. Dropping either one
|
||||
// loses information the operator needs.
|
||||
var card = Render(Card(lastPolled: Now.AddMinutes(-5)));
|
||||
|
||||
var classes = card.Find("article").ClassList;
|
||||
Assert.Contains("stale", classes, StringComparer.Ordinal);
|
||||
Assert.Contains("up", classes, StringComparer.Ordinal);
|
||||
Assert.Contains("Stale", card.Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void StaleCard_CountsFromLastContact_NotFromTheLastPollAttempt()
|
||||
{
|
||||
// The poller kept trying for ten minutes and kept failing. "Stale since 5 s ago" would
|
||||
// suggest the instance was fine moments ago; it has actually been gone for ten minutes.
|
||||
var card = Render(Card(
|
||||
status: InstanceStatus.Unreachable,
|
||||
lastPolled: Now.AddSeconds(-5),
|
||||
lastReachable: Now.AddMinutes(-10),
|
||||
staleAfter: TimeSpan.FromSeconds(1)));
|
||||
|
||||
Assert.Contains("Stale since", card.Markup, StringComparison.Ordinal);
|
||||
Assert.Contains("10 min ago", card.Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(ActiveState.Active, "Active")]
|
||||
[InlineData(ActiveState.Standby, "Standby")]
|
||||
[InlineData(ActiveState.Unknown, "Role ?")]
|
||||
public void ActiveState_IsBadged(ActiveState state, string expected)
|
||||
{
|
||||
Assert.Contains(expected, Render(Card(active: state)).Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void NoActiveRole_ShowsNoActivityBadgeAtAll()
|
||||
{
|
||||
// A single-instance gateway has no active/standby concept; a badge would invent one.
|
||||
var markup = Render(Card(active: ActiveState.NotApplicable)).Markup;
|
||||
|
||||
Assert.DoesNotContain("Standby", markup, StringComparison.Ordinal);
|
||||
Assert.DoesNotContain("Role ?", markup, StringComparison.Ordinal);
|
||||
Assert.DoesNotContain("· active", markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void LeaderBadge_IsShownOnlyWhenTheCardIsTheLeader()
|
||||
{
|
||||
Assert.Contains("Leader", Render(Card(), isLeader: true).Markup, StringComparison.Ordinal);
|
||||
Assert.DoesNotContain("Leader", Render(Card(), isLeader: false).Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Checks_AreListedWithAHealthyCount()
|
||||
{
|
||||
var card = Render(Card(report: Report(("localdb", "Healthy", null), ("akka-cluster", "Unhealthy", null))));
|
||||
|
||||
Assert.Contains("1 / 2 healthy", card.Markup, StringComparison.Ordinal);
|
||||
Assert.Equal(2, card.FindAll(".check-row").Count);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ClusterData_IsRenderedAsRawEvidence()
|
||||
{
|
||||
var card = Render(Card(report: Report(("akka-cluster", "Healthy", "akka.tcp://scadabridge@site-a-a:8082"))));
|
||||
|
||||
var data = card.Find(".check-data");
|
||||
Assert.Contains("akka.tcp://scadabridge@site-a-a:8082", data.TextContent, StringComparison.Ordinal);
|
||||
Assert.Contains("leader", data.TextContent, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ClusterData_FromAnInstanceWithoutIt_IsOmittedEntirely()
|
||||
{
|
||||
// Every pre-0.2.0 node, and every check that publishes no data. The card must simply not
|
||||
// show the line rather than showing an empty one.
|
||||
var card = Render(Card(report: Report(("localdb", "Healthy", null))));
|
||||
|
||||
Assert.Empty(card.FindAll(".check-data"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ClusterData_IsHtmlEscaped()
|
||||
{
|
||||
// The value is remote input from another application. Operationally trusted, but a health
|
||||
// check that ever emits markup must not be able to inject it into this page.
|
||||
var card = Render(Card(report: Report(("akka-cluster", "Healthy", "<img src=x onerror=alert(1)>"))));
|
||||
|
||||
var data = card.Find(".check-data");
|
||||
Assert.Empty(data.QuerySelectorAll("img"));
|
||||
Assert.Contains("<img src=x onerror=alert(1)>", data.TextContent, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ManagementLink_OpensInANewTabWithNoopener()
|
||||
{
|
||||
var link = Render(Card(managementUrl: "http://site-a-a:9000/")).Find(".panel-foot a");
|
||||
|
||||
Assert.Equal("http://site-a-a:9000/", link.GetAttribute("href"));
|
||||
Assert.Equal("_blank", link.GetAttribute("target"));
|
||||
Assert.Equal("noopener", link.GetAttribute("rel"));
|
||||
Assert.Contains("AdminUI", link.TextContent, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void NoManagementUrl_RendersNoDeadLink()
|
||||
{
|
||||
var card = Render(Card(managementUrl: null));
|
||||
|
||||
Assert.Empty(card.FindAll(".panel-foot a"));
|
||||
Assert.Contains("no management link", card.Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(InstanceStatus.Up, ActiveState.Active, "ready 200 · active 200")]
|
||||
[InlineData(InstanceStatus.Up, ActiveState.Standby, "ready 200 · active 503")]
|
||||
[InlineData(InstanceStatus.Down, ActiveState.NotApplicable, "ready 503")]
|
||||
[InlineData(InstanceStatus.Unreachable, ActiveState.NotApplicable, "ready fail")]
|
||||
public void RawSignalLine_ShowsWhatWasActuallyReceived(
|
||||
InstanceStatus status, ActiveState active, string expected)
|
||||
{
|
||||
// The footer is the escape hatch from every derived label above it: when the dashboard and
|
||||
// a curl disagree, this line is what reconciles them.
|
||||
Assert.Contains(
|
||||
expected,
|
||||
Render(Card(status, active)).Find("[data-testid=raw-signals]").TextContent,
|
||||
StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ProbeError_IsSurfacedOnTheCard()
|
||||
{
|
||||
var card = Render(Card(InstanceStatus.Unreachable, error: "Connection refused"));
|
||||
|
||||
Assert.Contains("Connection refused", card.Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Host_IsShownWithoutTheScheme()
|
||||
{
|
||||
Assert.Contains("site-a-a:8084", Render(Card()).Find(".inst-host").TextContent, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(4, "4 s ago")]
|
||||
[InlineData(90, "2 min ago")]
|
||||
[InlineData(7200, "2 h ago")]
|
||||
[InlineData(172800, "2 d ago")]
|
||||
public void Relative_ScalesTheUnit(int seconds, string expected)
|
||||
{
|
||||
Assert.Equal(expected, InstanceCard.Relative(TimeSpan.FromSeconds(seconds)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Relative_NegativeAge_ReadsAsNow_NotAsTheFuture()
|
||||
{
|
||||
// Clock skew between this host and a probed one is routine; "-3 s ago" is nonsense.
|
||||
Assert.Equal("0 s ago", InstanceCard.Relative(TimeSpan.FromSeconds(-3)));
|
||||
}
|
||||
|
||||
private static ZbHealthReport Report(params (string Name, string Status, string? Leader)[] entries) =>
|
||||
System.Text.Json.JsonSerializer.Deserialize<ZbHealthReport>(HealthBody.Ready("Healthy", entries))!;
|
||||
}
|
||||
@@ -0,0 +1,283 @@
|
||||
using ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Leader aggregation and the split-brain flag. The disagreement rule is the one output an operator
|
||||
/// would be woken by, so both directions are pinned: it must fire when two members genuinely claim
|
||||
/// different leaders, and it must NOT fire for the ordinary cases that superficially resemble that
|
||||
/// — a node that stopped answering, a node running a pre-0.2.0 Health package.
|
||||
/// </summary>
|
||||
public class LeaderResolverTests
|
||||
{
|
||||
private const string LeaderA = "akka.tcp://scadabridge@site-a-a:8082";
|
||||
private const string LeaderB = "akka.tcp://scadabridge@site-a-b:8082";
|
||||
|
||||
[Fact]
|
||||
public void Resolve_AgreeingPair_NamesTheLeaderInstance()
|
||||
{
|
||||
var groups = LeaderResolver.Resolve(
|
||||
[
|
||||
Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084", leader: LeaderA),
|
||||
Snapshots.Instance("site-a-b", "site-a", "http://site-a-b:8084", leader: LeaderA),
|
||||
]);
|
||||
|
||||
var group = Assert.Single(groups);
|
||||
Assert.Equal("site-a", group.Name);
|
||||
Assert.Equal("site-a-a", group.LeaderInstance);
|
||||
Assert.Equal(LeaderA, group.LeaderAddress);
|
||||
Assert.False(group.Disagreement);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Resolve_MembersClaimDifferentLeaders_FlagsDisagreement()
|
||||
{
|
||||
var groups = LeaderResolver.Resolve(
|
||||
[
|
||||
Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084", leader: LeaderA),
|
||||
Snapshots.Instance("site-a-b", "site-a", "http://site-a-b:8084", leader: LeaderB),
|
||||
]);
|
||||
|
||||
var group = Assert.Single(groups);
|
||||
Assert.True(group.Disagreement);
|
||||
Assert.Equal([LeaderA, LeaderB], group.ReportedLeaders);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(InstanceStatus.Down)]
|
||||
[InlineData(InstanceStatus.Unreachable)]
|
||||
public void Resolve_FailedMemberDoesNotVote_SoAFailoverIsNotMisreadAsSplitBrain(InstanceStatus failed)
|
||||
{
|
||||
// Mid-failover the departed node's last body still names the old leader. Counting it would
|
||||
// raise a split-brain warning during the most normal event in a redundant pair's life.
|
||||
var groups = LeaderResolver.Resolve(
|
||||
[
|
||||
Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084", status: failed, leader: LeaderA),
|
||||
Snapshots.Instance("site-a-b", "site-a", "http://site-a-b:8084", leader: LeaderB),
|
||||
]);
|
||||
|
||||
var group = Assert.Single(groups);
|
||||
Assert.False(group.Disagreement);
|
||||
Assert.Equal("site-a-b", group.LeaderInstance);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Resolve_DegradedMemberStillVotes()
|
||||
{
|
||||
// Degraded means answering with a full cluster view — its opinion on the leader is current.
|
||||
var groups = LeaderResolver.Resolve(
|
||||
[
|
||||
Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084", status: InstanceStatus.Degraded, leader: LeaderA),
|
||||
Snapshots.Instance("site-a-b", "site-a", "http://site-a-b:8084", leader: LeaderB),
|
||||
]);
|
||||
|
||||
Assert.True(Assert.Single(groups).Disagreement);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Resolve_NoMemberPublishesLeaderData_YieldsNoLeaderAndNoWarning()
|
||||
{
|
||||
// The whole-fleet-on-Health-0.1.0 case: no data key anywhere. It must degrade to "leader
|
||||
// unknown", never to a false split-brain warning.
|
||||
var groups = LeaderResolver.Resolve(
|
||||
[
|
||||
Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084"),
|
||||
Snapshots.Instance("site-a-b", "site-a", "http://site-a-b:8084"),
|
||||
]);
|
||||
|
||||
var group = Assert.Single(groups);
|
||||
Assert.Null(group.LeaderAddress);
|
||||
Assert.Null(group.LeaderInstance);
|
||||
Assert.False(group.Disagreement);
|
||||
Assert.Empty(group.ReportedLeaders);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Resolve_OneMemberOnOldHealthPackage_UsesTheOneThatDoesReport()
|
||||
{
|
||||
// A partly-bumped fleet is the expected state during rollout, and one silent member is not
|
||||
// a disagreement.
|
||||
var groups = LeaderResolver.Resolve(
|
||||
[
|
||||
Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084", leader: LeaderA),
|
||||
Snapshots.Instance("site-a-b", "site-a", "http://site-a-b:8084"),
|
||||
]);
|
||||
|
||||
var group = Assert.Single(groups);
|
||||
Assert.Equal("site-a-a", group.LeaderInstance);
|
||||
Assert.False(group.Disagreement);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Resolve_UnresolvableAddress_KeepsTheRawAddress()
|
||||
{
|
||||
// A leader on a host that is not in the registry (a node nobody added) still tells the
|
||||
// operator something useful — showing nothing would hide the discovery.
|
||||
var groups = LeaderResolver.Resolve(
|
||||
[
|
||||
Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084", leader: "akka.tcp://scadabridge@ghost:8082"),
|
||||
]);
|
||||
|
||||
var group = Assert.Single(groups);
|
||||
Assert.Null(group.LeaderInstance);
|
||||
Assert.Equal("akka.tcp://scadabridge@ghost:8082", group.LeaderAddress);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Resolve_MatchesOnHostOnly_BecauseAkkaAndHttpPortsDiffer()
|
||||
{
|
||||
// The registry knows the HTTP port (8084); the leader address carries the Akka remoting
|
||||
// port (8082). Matching on authority instead of host would never resolve anything.
|
||||
var name = LeaderResolver.ResolveToInstanceName(
|
||||
LeaderA,
|
||||
[Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084")]);
|
||||
|
||||
Assert.Equal("site-a-a", name);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Resolve_GroupWithoutActiveRole_IsNotAGroupAtAll()
|
||||
{
|
||||
// Instances grouped only for layout have no leader; a chip there would invent a concept.
|
||||
var groups = LeaderResolver.Resolve(
|
||||
[
|
||||
Snapshots.Instance("gw-1", "gateways", "http://gw-1:5120", hasActiveRole: false),
|
||||
Snapshots.Instance("gw-2", "gateways", "http://gw-2:5120", hasActiveRole: false),
|
||||
]);
|
||||
|
||||
Assert.Empty(groups);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Resolve_UngroupedInstances_ProduceNoGroups()
|
||||
{
|
||||
Assert.Empty(LeaderResolver.Resolve([Snapshots.Instance("solo")]));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Resolve_SeparateGroups_AreScopedIndependently()
|
||||
{
|
||||
// Two site pairs, each with its own leader: per-Cluster election is the family's actual
|
||||
// topology, so a leader from one group must never be attributed to another.
|
||||
var groups = LeaderResolver.Resolve(
|
||||
[
|
||||
Snapshots.Instance("site-a-a", "site-a", "http://site-a-a:8084", leader: LeaderA),
|
||||
Snapshots.Instance("site-b-a", "site-b", "http://site-b-a:8084", leader: "akka.tcp://scadabridge@site-b-a:8082"),
|
||||
]);
|
||||
|
||||
Assert.Equal(2, groups.Count);
|
||||
Assert.Equal("site-a-a", groups[0].LeaderInstance);
|
||||
Assert.Equal("site-b-a", groups[1].LeaderInstance);
|
||||
Assert.All(groups, g => Assert.False(g.Disagreement));
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("akka.tcp://scadabridge@site-a-a:8082", "site-a-a")]
|
||||
[InlineData("akka.tcp://otopcua@10.100.0.35:4053", "10.100.0.35")]
|
||||
[InlineData("akka.tcp://sys@host", "host")]
|
||||
[InlineData("not-an-address", null)]
|
||||
[InlineData("", null)]
|
||||
[InlineData(null, null)]
|
||||
public void HostOf_ParsesAkkaAddresses(string? address, string? expected)
|
||||
{
|
||||
Assert.Equal(expected, LeaderResolver.HostOf(address));
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("akka-cluster")] // ScadaBridge
|
||||
[InlineData("akka")] // OtOpcUa
|
||||
[InlineData("cluster-view")] // any future app
|
||||
public void Leader_IsFoundWhateverTheCheckIsCalled(string checkName)
|
||||
{
|
||||
const string NodeALeader = "akka.tcp://otopcua@node-a:4053";
|
||||
|
||||
// The two apps register the SAME shared AkkaClusterHealthCheck under different names, so a
|
||||
// resolver keyed on one name renders no leader chip at all for the other — silently, since
|
||||
// a group with no votes is a legitimate state. That is exactly how this shipped past
|
||||
// review and unit tests and was only caught on a live rig.
|
||||
var groups = LeaderResolver.Resolve(
|
||||
[
|
||||
Snapshots.Instance("node-a", "pair", "http://node-a:8080", leader: NodeALeader, clusterCheckName: checkName),
|
||||
Snapshots.Instance("node-b", "pair", "http://node-b:8080", leader: NodeALeader, clusterCheckName: checkName),
|
||||
]);
|
||||
|
||||
var group = Assert.Single(groups);
|
||||
Assert.Equal("node-a", group.LeaderInstance);
|
||||
Assert.False(group.Disagreement);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Leader_IgnoresChecksThatPublishDataWithoutALeader()
|
||||
{
|
||||
// ScadaBridge's site nodes publish a localdb check whose data carries replication counters
|
||||
// and no leader. Scanning entries must pick the one that actually answers the question,
|
||||
// not merely the first one carrying a data object.
|
||||
var body = HealthBody.Ready(
|
||||
"Healthy",
|
||||
("localdb", "Healthy", null),
|
||||
("akka", "Healthy", "akka.tcp://otopcua@node-a:4053"));
|
||||
var report = System.Text.Json.JsonSerializer.Deserialize<ZbHealthReport>(body)!;
|
||||
var instance = Snapshots.Instance("node-a", "pair", "http://node-a:8080") with { Report = report };
|
||||
|
||||
Assert.Equal("akka.tcp://otopcua@node-a:4053", LeaderResolver.LeaderReportedBy(instance));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ActiveNode_WinsOverLeader_WhenTheyDisagree()
|
||||
{
|
||||
// The two are DIFFERENT NODES after any restart: leader is the lowest-addressed Up member,
|
||||
// the active node is the oldest. Observed live on the rebuilt OtOpcUa rig — leader central-1,
|
||||
// active central-2 — so ranking leader first would put the Active chip on the standby, which
|
||||
// is the same class of error as lmxopcua#494 rendered in the UI instead of in routing.
|
||||
var body = HealthBody.WithData(
|
||||
"Healthy",
|
||||
("akka", "Healthy", new Dictionary<string, object>
|
||||
{
|
||||
["leader"] = "akka.tcp://otopcua@central-1:4053",
|
||||
}),
|
||||
("cluster-primary", "Healthy", new Dictionary<string, object>
|
||||
{
|
||||
["activeNode"] = "akka.tcp://otopcua@central-2:4053",
|
||||
}));
|
||||
var report = System.Text.Json.JsonSerializer.Deserialize<ZbHealthReport>(body)!;
|
||||
var instance = Snapshots.Instance("central-1", "MAIN", "http://central-1:8080") with { Report = report };
|
||||
|
||||
Assert.Equal("akka.tcp://otopcua@central-2:4053", LeaderResolver.LeaderReportedBy(instance));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ActiveNode_ReportedByAStandby_NamesTheActiveNode_NotItself()
|
||||
{
|
||||
// Every member publishes who IS active, so a standby votes for the same address the active
|
||||
// node does. That is what lets the group resolve with one unanimous chip instead of a
|
||||
// manufactured disagreement, and it is why the standby's payload alone is enough.
|
||||
var body = HealthBody.WithData(
|
||||
"Unhealthy",
|
||||
("cluster-primary", "Unhealthy", new Dictionary<string, object>
|
||||
{
|
||||
["activeNode"] = "akka.tcp://otopcua@site-a-1:4053",
|
||||
["selfAddress"] = "akka.tcp://otopcua@site-a-2:4053",
|
||||
}));
|
||||
var report = System.Text.Json.JsonSerializer.Deserialize<ZbHealthReport>(body)!;
|
||||
var instance = Snapshots.Instance("site-a-2", "SITE-A", "http://site-a-2:8080") with { Report = report };
|
||||
|
||||
Assert.Equal("akka.tcp://otopcua@site-a-1:4053", LeaderResolver.LeaderReportedBy(instance));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Leader_IsStillUsed_WhenNoActiveNodeIsPublished()
|
||||
{
|
||||
// An instance on Health < 0.3.0, or one whose active tier was not probed, publishes only
|
||||
// `leader`. Showing it beats showing nothing, and it keeps the pre-0.3.0 behaviour intact.
|
||||
var body = HealthBody.WithData(
|
||||
"Healthy",
|
||||
("akka-cluster", "Healthy", new Dictionary<string, object>
|
||||
{
|
||||
["leader"] = "akka.tcp://scadabridge@central-a:4053",
|
||||
}));
|
||||
var report = System.Text.Json.JsonSerializer.Deserialize<ZbHealthReport>(body)!;
|
||||
var instance = Snapshots.Instance("central-a", "central", "http://central-a:8080") with { Report = report };
|
||||
|
||||
Assert.Equal("akka.tcp://scadabridge@central-a:4053", LeaderResolver.LeaderReportedBy(instance));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,309 @@
|
||||
using System.Diagnostics.Metrics;
|
||||
using Microsoft.Extensions.Diagnostics.HealthChecks;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.Overview.Observability;
|
||||
using ZB.MOM.WW.Overview.Polling;
|
||||
using ZB.MOM.WW.Overview.Registry;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// The dashboard watching itself: the gauge it exports about the fleet, and the two checks that
|
||||
/// decide whether its own <c>/health/ready</c> is telling the truth.
|
||||
/// </summary>
|
||||
public class ObservabilityTests
|
||||
{
|
||||
private static readonly DateTimeOffset Start = new(2026, 7, 24, 12, 0, 0, TimeSpan.Zero);
|
||||
|
||||
private static OverviewOptions Registry(
|
||||
int pollIntervalSeconds = 10,
|
||||
int timeoutSeconds = 3,
|
||||
params ApplicationEntry[] applications) =>
|
||||
new()
|
||||
{
|
||||
PollIntervalSeconds = pollIntervalSeconds,
|
||||
TimeoutSeconds = timeoutSeconds,
|
||||
StaleAfterSeconds = 45,
|
||||
Applications = applications.Length > 0
|
||||
? applications
|
||||
: [new ApplicationEntry
|
||||
{
|
||||
Name = "App",
|
||||
Instances = [new InstanceEntry { Name = "a", BaseUrl = "http://a:8080" }],
|
||||
}],
|
||||
};
|
||||
|
||||
private static HealthCheckContext Context() => new();
|
||||
|
||||
// ---------------------------------------------------------------- registry-loaded
|
||||
|
||||
[Fact]
|
||||
public async Task RegistryLoaded_ReportsTheCountsItBound()
|
||||
{
|
||||
var options = Registry(applications:
|
||||
[
|
||||
new ApplicationEntry
|
||||
{
|
||||
Name = "ScadaBridge",
|
||||
Instances =
|
||||
[
|
||||
new InstanceEntry { Name = "central-a", BaseUrl = "http://central-a:9000" },
|
||||
new InstanceEntry { Name = "central-b", BaseUrl = "http://central-b:9100" },
|
||||
],
|
||||
},
|
||||
new ApplicationEntry
|
||||
{
|
||||
Name = "OtOpcUa",
|
||||
Instances = [new InstanceEntry { Name = "admin", BaseUrl = "http://admin:9200" }],
|
||||
},
|
||||
]);
|
||||
|
||||
var result = await new RegistryLoadedHealthCheck(Options.Create(options))
|
||||
.CheckHealthAsync(Context());
|
||||
|
||||
Assert.Equal(HealthStatus.Healthy, result.Status);
|
||||
Assert.Equal(2, result.Data["applications"]);
|
||||
Assert.Equal(3, result.Data["instances"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RegistryLoaded_EmptyRegistry_IsUnhealthy()
|
||||
{
|
||||
// Unreachable through the real startup path — preflight and the validator both refuse it —
|
||||
// but the check must still be able to say so, or it is only ever decorative.
|
||||
var options = new OverviewOptions { Applications = [] };
|
||||
|
||||
var result = await new RegistryLoadedHealthCheck(Options.Create(options))
|
||||
.CheckHealthAsync(Context());
|
||||
|
||||
Assert.Equal(HealthStatus.Unhealthy, result.Status);
|
||||
Assert.Equal(0, result.Data["instances"]);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- grace window
|
||||
|
||||
[Fact]
|
||||
public void Grace_IsTwiceTheFastestIntervalPlusTheSlowestTimeout()
|
||||
{
|
||||
var options = Registry(pollIntervalSeconds: 10, timeoutSeconds: 3);
|
||||
|
||||
Assert.Equal(TimeSpan.FromSeconds(26), PollCycleHealthCheck.GraceFor(options));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Grace_HonoursPerInstanceOverrides()
|
||||
{
|
||||
// The sweep loop ticks at the FASTEST configured interval, and the slowest timeout bounds
|
||||
// how long any one sweep can take — so the window has to be derived from both extremes,
|
||||
// not from the global defaults.
|
||||
var options = Registry(pollIntervalSeconds: 30, timeoutSeconds: 3, applications:
|
||||
[
|
||||
new ApplicationEntry
|
||||
{
|
||||
Name = "App",
|
||||
Instances =
|
||||
[
|
||||
new InstanceEntry { Name = "fast", BaseUrl = "http://fast:8080", PollIntervalSeconds = 5 },
|
||||
new InstanceEntry { Name = "slow", BaseUrl = "http://slow:8080", TimeoutSeconds = 20 },
|
||||
],
|
||||
},
|
||||
]);
|
||||
|
||||
Assert.Equal(TimeSpan.FromSeconds(50), PollCycleHealthCheck.GraceFor(options));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- last-poll-cycle-completed
|
||||
|
||||
[Fact]
|
||||
public async Task PollCycle_NoCycleYetButStillStarting_IsDegradedNotUnhealthy()
|
||||
{
|
||||
// Every deployment passes through this state. Failing readiness here would make the
|
||||
// container look broken for the first seconds of its life, every single restart.
|
||||
var time = new FakeTimeProvider(Start);
|
||||
var check = new PollCycleHealthCheck(new PollCycleHeartbeat(time), Options.Create(Registry()), time);
|
||||
|
||||
time.Advance(TimeSpan.FromSeconds(5));
|
||||
var result = await check.CheckHealthAsync(Context());
|
||||
|
||||
Assert.Equal(HealthStatus.Degraded, result.Status);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task PollCycle_NeverSwept_EventuallyGoesUnhealthy()
|
||||
{
|
||||
// A poller that never started must not hide behind "still starting" forever.
|
||||
var time = new FakeTimeProvider(Start);
|
||||
var check = new PollCycleHealthCheck(new PollCycleHeartbeat(time), Options.Create(Registry()), time);
|
||||
|
||||
time.Advance(TimeSpan.FromMinutes(5));
|
||||
var result = await check.CheckHealthAsync(Context());
|
||||
|
||||
Assert.Equal(HealthStatus.Unhealthy, result.Status);
|
||||
Assert.Contains("since startup", result.Description, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task PollCycle_RecentSweep_IsHealthyAndPublishesItsAge()
|
||||
{
|
||||
var time = new FakeTimeProvider(Start);
|
||||
var heartbeat = new PollCycleHeartbeat(time);
|
||||
var check = new PollCycleHealthCheck(heartbeat, Options.Create(Registry()), time);
|
||||
|
||||
heartbeat.RecordCycleCompleted(Start);
|
||||
time.Advance(TimeSpan.FromSeconds(4));
|
||||
var result = await check.CheckHealthAsync(Context());
|
||||
|
||||
Assert.Equal(HealthStatus.Healthy, result.Status);
|
||||
Assert.Equal(4d, result.Data["ageSeconds"]);
|
||||
Assert.Equal(26d, result.Data["graceSeconds"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task PollCycle_StoppedSweeping_IsUnhealthy()
|
||||
{
|
||||
// The failure this check exists for: Kestrel is fine, the page still renders, and every
|
||||
// card on it is quietly frozen.
|
||||
var time = new FakeTimeProvider(Start);
|
||||
var heartbeat = new PollCycleHeartbeat(time);
|
||||
var check = new PollCycleHealthCheck(heartbeat, Options.Create(Registry()), time);
|
||||
|
||||
heartbeat.RecordCycleCompleted(Start);
|
||||
time.Advance(TimeSpan.FromMinutes(2));
|
||||
var result = await check.CheckHealthAsync(Context());
|
||||
|
||||
Assert.Equal(HealthStatus.Unhealthy, result.Status);
|
||||
Assert.Contains("stale", result.Description, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Heartbeat_ReportsNothingUntilACycleCompletes()
|
||||
{
|
||||
var heartbeat = new PollCycleHeartbeat(new FakeTimeProvider(Start));
|
||||
|
||||
Assert.Null(heartbeat.LastCompletedUtc);
|
||||
Assert.Equal(Start, heartbeat.StartedUtc);
|
||||
|
||||
heartbeat.RecordCycleCompleted(Start.AddSeconds(7));
|
||||
|
||||
Assert.Equal(Start.AddSeconds(7), heartbeat.LastCompletedUtc);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- the fleet gauge
|
||||
|
||||
[Fact]
|
||||
public void Gauge_ReportsOneMeasurementPerProbedInstance()
|
||||
{
|
||||
var store = new OverviewSnapshotStore();
|
||||
store.Publish(SnapshotOf(
|
||||
Snapshots.Instance("central-a", group: "central", status: InstanceStatus.Up),
|
||||
Snapshots.Instance("central-b", group: "central", status: InstanceStatus.Degraded),
|
||||
Snapshots.Instance("site-a-a", group: "site-a", status: InstanceStatus.Unreachable)));
|
||||
|
||||
using var factory = new DummyMeterFactory();
|
||||
using var metrics = new OverviewMetrics(factory, store);
|
||||
|
||||
var measured = CollectStatusGauge(factory);
|
||||
|
||||
Assert.Equal(3, measured.Count);
|
||||
Assert.Equal(0, measured.Single(m => m.Tags["node"] as string == "central-a").Value);
|
||||
Assert.Equal(1, measured.Single(m => m.Tags["node"] as string == "central-b").Value);
|
||||
Assert.Equal(3, measured.Single(m => m.Tags["node"] as string == "site-a-a").Value);
|
||||
Assert.Equal("central", measured.First().Tags["group"]);
|
||||
Assert.Equal("App", measured.First().Tags["application"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Gauge_SkipsInstancesThatHaveNeverBeenProbed()
|
||||
{
|
||||
// The enum's zero ordinal is Up, so emitting anything for an unprobed instance would
|
||||
// publish the single most reassuring answer about a node nobody has heard from.
|
||||
var store = new OverviewSnapshotStore();
|
||||
store.Publish(SnapshotOf(
|
||||
Snapshots.Instance("probed", status: InstanceStatus.Down),
|
||||
Snapshots.Instance("never-probed", status: null)));
|
||||
|
||||
using var factory = new DummyMeterFactory();
|
||||
using var metrics = new OverviewMetrics(factory, store);
|
||||
|
||||
var measured = CollectStatusGauge(factory);
|
||||
|
||||
Assert.Equal("probed", Assert.Single(measured).Tags["node"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Gauge_TracksTheStoreRatherThanASnapshotTakenAtConstruction()
|
||||
{
|
||||
// It is an OBSERVABLE gauge: read at scrape time from whatever the store currently holds,
|
||||
// so the metric and the page can never disagree about what the dashboard believes.
|
||||
var store = new OverviewSnapshotStore();
|
||||
store.Publish(SnapshotOf(Snapshots.Instance("a", status: InstanceStatus.Up)));
|
||||
|
||||
using var factory = new DummyMeterFactory();
|
||||
using var metrics = new OverviewMetrics(factory, store);
|
||||
|
||||
Assert.Equal(0, Assert.Single(CollectStatusGauge(factory)).Value);
|
||||
|
||||
store.Publish(SnapshotOf(Snapshots.Instance("a", status: InstanceStatus.Down)));
|
||||
|
||||
Assert.Equal(2, Assert.Single(CollectStatusGauge(factory)).Value);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_StampsTheHeartbeatSoTheSelfCheckCanSeeIt()
|
||||
{
|
||||
// Ties the two halves together: without this the health check would be measuring a value
|
||||
// nothing ever writes, and would report a healthy poller forever.
|
||||
var time = new FakeTimeProvider(Start);
|
||||
var store = new OverviewSnapshotStore();
|
||||
var heartbeat = new PollCycleHeartbeat(time);
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", System.Net.HttpStatusCode.OK, HealthBody.Ready("Healthy"));
|
||||
|
||||
using var factory = new DummyMeterFactory();
|
||||
var poller = new OverviewPollerService(
|
||||
Options.Create(Registry()),
|
||||
new StubHttpClientFactory(handler),
|
||||
store,
|
||||
time,
|
||||
new OverviewMetrics(factory, store),
|
||||
heartbeat,
|
||||
Microsoft.Extensions.Logging.Abstractions.NullLogger<OverviewPollerService>.Instance);
|
||||
|
||||
Assert.Null(heartbeat.LastCompletedUtc);
|
||||
|
||||
await poller.SweepAsync();
|
||||
|
||||
Assert.Equal(Start, heartbeat.LastCompletedUtc);
|
||||
}
|
||||
|
||||
private static OverviewSnapshot SnapshotOf(params InstanceSnapshot[] instances) =>
|
||||
new(Start, [new ApplicationSnapshot("App", instances, [])]);
|
||||
|
||||
private static List<(int Value, Dictionary<string, object?> Tags)> CollectStatusGauge(
|
||||
DummyMeterFactory factory)
|
||||
{
|
||||
var measured = new List<(int, Dictionary<string, object?>)>();
|
||||
|
||||
using var listener = new MeterListener
|
||||
{
|
||||
InstrumentPublished = (instrument, l) =>
|
||||
{
|
||||
// Scoped to THIS test's factory: meter names are process-global, so matching on
|
||||
// the name alone would also collect instruments from tests running in parallel.
|
||||
if (ReferenceEquals(instrument.Meter.Scope, factory) &&
|
||||
instrument.Name == OverviewMetrics.InstanceStatusInstrument)
|
||||
{
|
||||
l.EnableMeasurementEvents(instrument);
|
||||
}
|
||||
},
|
||||
};
|
||||
|
||||
listener.SetMeasurementEventCallback<int>((_, value, tags, _) =>
|
||||
measured.Add((value, tags.ToArray().ToDictionary(t => t.Key, t => t.Value, StringComparer.Ordinal))));
|
||||
|
||||
listener.Start();
|
||||
listener.RecordObservableInstruments();
|
||||
|
||||
return measured;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,226 @@
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.Overview.Registry;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Registry validation. A malformed registry must fail the host at startup with a message naming
|
||||
/// the offending key — the alternative is a dashboard that boots and quietly shows nothing, or one
|
||||
/// that throws on its first poll where nobody is looking.
|
||||
/// </summary>
|
||||
public class OverviewOptionsValidatorTests
|
||||
{
|
||||
private static readonly OverviewOptionsValidator Validator = new();
|
||||
|
||||
private static OverviewOptions Valid(Action<OverviewOptions>? mutate = null)
|
||||
{
|
||||
var options = new OverviewOptions
|
||||
{
|
||||
PollIntervalSeconds = 10,
|
||||
TimeoutSeconds = 3,
|
||||
StaleAfterSeconds = 45,
|
||||
Applications =
|
||||
{
|
||||
new ApplicationEntry
|
||||
{
|
||||
Name = "OtOpcUa",
|
||||
ManagementLabel = "AdminUI",
|
||||
Instances =
|
||||
{
|
||||
new InstanceEntry
|
||||
{
|
||||
Name = "central-1",
|
||||
Group = "central",
|
||||
BaseUrl = "http://otopcua-central-1:9000",
|
||||
ManagementUrl = "http://otopcua-central-1:9000/",
|
||||
HasActiveRole = true,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
mutate?.Invoke(options);
|
||||
return options;
|
||||
}
|
||||
|
||||
private static ValidateOptionsResult Run(OverviewOptions options) => Validator.Validate(null, options);
|
||||
|
||||
private static void AssertFailsWith(OverviewOptions options, string expectedFragment)
|
||||
{
|
||||
var result = Run(options);
|
||||
|
||||
Assert.True(result.Failed, "expected validation to fail");
|
||||
Assert.Contains(
|
||||
result.Failures,
|
||||
f => f.Contains(expectedFragment, StringComparison.OrdinalIgnoreCase));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ValidRegistry_Succeeds()
|
||||
{
|
||||
// Positive control: without this, every "fails with X" assertion below could be passing
|
||||
// because the fixture itself is malformed for some unrelated reason.
|
||||
Assert.True(Run(Valid()).Succeeded);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void EmptyRegistry_IsRejected()
|
||||
{
|
||||
AssertFailsWith(Valid(o => o.Applications.Clear()), "Overview:Applications");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ApplicationWithNoInstances_IsRejected()
|
||||
{
|
||||
AssertFailsWith(Valid(o => o.Applications[0].Instances.Clear()), "Instances");
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("")]
|
||||
[InlineData(" ")]
|
||||
public void ApplicationWithoutName_IsRejected(string name)
|
||||
{
|
||||
AssertFailsWith(Valid(o => o.Applications[0].Name = name), "Applications:0:Name");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DuplicateApplicationNames_AreRejected()
|
||||
{
|
||||
AssertFailsWith(
|
||||
Valid(o => o.Applications.Add(new ApplicationEntry
|
||||
{
|
||||
Name = "OtOpcUa",
|
||||
Instances = { new InstanceEntry { Name = "other", BaseUrl = "http://other:9000" } },
|
||||
})),
|
||||
"duplicated");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DuplicateInstanceNamesWithinAnApplication_AreRejected()
|
||||
{
|
||||
AssertFailsWith(
|
||||
Valid(o => o.Applications[0].Instances.Add(new InstanceEntry
|
||||
{
|
||||
Name = "central-1",
|
||||
BaseUrl = "http://otopcua-central-2:9000",
|
||||
})),
|
||||
"duplicated");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SameInstanceNameInDifferentApplications_IsAllowed()
|
||||
{
|
||||
// "central-1" under OtOpcUa and "central-1" under ScadaBridge are different machines with
|
||||
// the same conventional name — uniqueness is per application, not global.
|
||||
var options = Valid(o => o.Applications.Add(new ApplicationEntry
|
||||
{
|
||||
Name = "ScadaBridge",
|
||||
Instances = { new InstanceEntry { Name = "central-1", BaseUrl = "http://sb-central-a:5000" } },
|
||||
}));
|
||||
|
||||
Assert.True(Run(options).Succeeded);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("")]
|
||||
[InlineData("otopcua-central-1:9000")] // no scheme
|
||||
[InlineData("/health/ready")] // relative
|
||||
[InlineData("ftp://otopcua-central-1")] // wrong scheme
|
||||
public void NonAbsoluteHttpBaseUrl_IsRejected(string baseUrl)
|
||||
{
|
||||
AssertFailsWith(Valid(o => o.Applications[0].Instances[0].BaseUrl = baseUrl), "BaseUrl");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RelativeManagementUrl_IsRejected()
|
||||
{
|
||||
AssertFailsWith(Valid(o => o.Applications[0].Instances[0].ManagementUrl = "/admin"), "ManagementUrl");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void OmittedManagementUrl_IsAllowed()
|
||||
{
|
||||
Assert.True(Run(Valid(o => o.Applications[0].Instances[0].ManagementUrl = null)).Succeeded);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(0)]
|
||||
[InlineData(-1)]
|
||||
public void NonPositiveGlobalIntervals_AreRejected(int value)
|
||||
{
|
||||
AssertFailsWith(Valid(o => o.PollIntervalSeconds = value), "PollIntervalSeconds");
|
||||
AssertFailsWith(Valid(o => o.TimeoutSeconds = value), "TimeoutSeconds");
|
||||
AssertFailsWith(Valid(o => o.StaleAfterSeconds = value), "StaleAfterSeconds");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void StaleWindowNotGreaterThanPollInterval_IsRejected()
|
||||
{
|
||||
AssertFailsWith(Valid(o => o.StaleAfterSeconds = o.PollIntervalSeconds), "StaleAfterSeconds");
|
||||
AssertFailsWith(Valid(o => o.StaleAfterSeconds = o.PollIntervalSeconds - 1), "StaleAfterSeconds");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void StaleWindowIsCheckedOnEffectiveValues_NotGlobals()
|
||||
{
|
||||
// The globals here are perfectly sane (10 / 45). Only the INSTANCE override inverts the
|
||||
// relationship, which a globals-only check would wave straight through.
|
||||
AssertFailsWith(
|
||||
Valid(o => o.Applications[0].Instances[0].PollIntervalSeconds = 120),
|
||||
"effective StaleAfterSeconds");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ApplicationLevelOverrideCanSatisfyTheStaleRule()
|
||||
{
|
||||
// The mirror of the previous test: an override that RESTORES the relationship must pass, so
|
||||
// the rule is not just "any override fails".
|
||||
var options = Valid(o =>
|
||||
{
|
||||
o.Applications[0].Instances[0].PollIntervalSeconds = 120;
|
||||
o.Applications[0].Instances[0].StaleAfterSeconds = 300;
|
||||
});
|
||||
|
||||
Assert.True(Run(options).Succeeded);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void AllFailuresAreAccumulated_NotJustTheFirst()
|
||||
{
|
||||
// OptionsValidatorBase accumulates; an operator fixing a registry one boot at a time is the
|
||||
// failure mode this avoids.
|
||||
var result = Run(Valid(o =>
|
||||
{
|
||||
o.Applications[0].Name = string.Empty;
|
||||
o.Applications[0].Instances[0].BaseUrl = "not-a-url";
|
||||
o.PollIntervalSeconds = 0;
|
||||
}));
|
||||
|
||||
Assert.True(result.Failed);
|
||||
Assert.True(result.Failures.Count() >= 3, $"expected >= 3 failures, got {result.Failures.Count()}");
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(null, null, null, 10, 3, 45)] // all defaults
|
||||
[InlineData(20, null, null, 20, 3, 45)] // application override
|
||||
[InlineData(20, 30, null, 30, 3, 45)] // instance beats application
|
||||
[InlineData(null, 30, 90, 30, 90, 45)] // independent fields resolve independently
|
||||
public void EffectiveTimings_ResolveInstanceOverApplicationOverGlobal(
|
||||
int? applicationPoll, int? instancePoll, int? instanceTimeout,
|
||||
int expectedPoll, int expectedTimeout, int expectedStale)
|
||||
{
|
||||
var options = Valid(o =>
|
||||
{
|
||||
o.Applications[0].PollIntervalSeconds = applicationPoll;
|
||||
o.Applications[0].Instances[0].PollIntervalSeconds = instancePoll;
|
||||
o.Applications[0].Instances[0].TimeoutSeconds = instanceTimeout;
|
||||
});
|
||||
|
||||
var timings = EffectiveTimings.Resolve(options, options.Applications[0], options.Applications[0].Instances[0]);
|
||||
|
||||
Assert.Equal(TimeSpan.FromSeconds(expectedPoll), timings.PollInterval);
|
||||
Assert.Equal(TimeSpan.FromSeconds(expectedTimeout), timings.Timeout);
|
||||
Assert.Equal(TimeSpan.FromSeconds(expectedStale), timings.StaleAfter);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,251 @@
|
||||
using Bunit;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// The page as a whole: the KPI strip an operator scans first, the group headers that carry the
|
||||
/// split-brain warning, and the pause control — which has to freeze the view without pretending
|
||||
/// the view is still live.
|
||||
/// </summary>
|
||||
public class OverviewPageTests : TestContext
|
||||
{
|
||||
private static readonly DateTimeOffset Now = new(2026, 7, 24, 12, 0, 0, TimeSpan.Zero);
|
||||
|
||||
private readonly OverviewSnapshotStore _store = new();
|
||||
private readonly FakeTimeProvider _time = new(Now);
|
||||
|
||||
public OverviewPageTests()
|
||||
{
|
||||
Services.AddSingleton(_store);
|
||||
Services.AddSingleton<TimeProvider>(_time);
|
||||
}
|
||||
|
||||
private static InstanceSnapshot Node(
|
||||
string name,
|
||||
InstanceStatus? status,
|
||||
string? group = null,
|
||||
string? leader = null,
|
||||
DateTimeOffset? lastPolled = null) =>
|
||||
new()
|
||||
{
|
||||
Application = "ScadaBridge",
|
||||
Instance = name,
|
||||
Group = group,
|
||||
BaseUrl = $"http://{name}:8084",
|
||||
HasActiveRole = group is not null,
|
||||
Status = status,
|
||||
LastPolledUtc = lastPolled ?? Now,
|
||||
LastReachableUtc = lastPolled ?? Now,
|
||||
StaleAfter = TimeSpan.FromSeconds(45),
|
||||
Report = leader is null
|
||||
? null
|
||||
: System.Text.Json.JsonSerializer.Deserialize<ZbHealthReport>(
|
||||
HealthBody.Ready("Healthy", ("akka-cluster", "Healthy", leader))),
|
||||
};
|
||||
|
||||
private static OverviewSnapshot Snapshot(params InstanceSnapshot[] instances)
|
||||
{
|
||||
var applications = instances
|
||||
.GroupBy(i => i.Application, StringComparer.Ordinal)
|
||||
.Select(g => new ApplicationSnapshot(g.Key, g.ToList(), LeaderResolver.Resolve(g.ToList())))
|
||||
.ToList();
|
||||
|
||||
return new OverviewSnapshot(Now, applications);
|
||||
}
|
||||
|
||||
private IRenderedComponent<Components.Pages.Overview> RenderPage() =>
|
||||
RenderComponent<Components.Pages.Overview>();
|
||||
|
||||
[Fact]
|
||||
public void EmptyStore_RendersAnExplicitEmptyState()
|
||||
{
|
||||
// Before the first sweep. It must say so rather than render a KPI strip full of zeroes,
|
||||
// which would read as "the whole fleet is registered and nothing is up".
|
||||
var page = RenderPage();
|
||||
|
||||
Assert.NotNull(page.Find("[data-testid=empty-registry]"));
|
||||
Assert.Empty(page.FindAll(".agg-grid"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void KpiStrip_CountsEachStatus()
|
||||
{
|
||||
_store.Publish(Snapshot(
|
||||
Node("a", InstanceStatus.Up),
|
||||
Node("b", InstanceStatus.Up),
|
||||
Node("c", InstanceStatus.Degraded),
|
||||
Node("d", InstanceStatus.Down),
|
||||
Node("e", InstanceStatus.Unreachable)));
|
||||
|
||||
var page = RenderPage();
|
||||
|
||||
Assert.Equal("2", page.Find("[data-testid=kpi-up]").TextContent);
|
||||
Assert.Equal("1", page.Find("[data-testid=kpi-degraded]").TextContent);
|
||||
Assert.Equal("1", page.Find("[data-testid=kpi-down]").TextContent);
|
||||
Assert.Equal("1", page.Find("[data-testid=kpi-unreachable]").TextContent);
|
||||
Assert.Equal("0", page.Find("[data-testid=kpi-stale]").TextContent);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void StaleInstances_CountAsStaleAndNotAlsoAsUp()
|
||||
{
|
||||
// Double-counting would make the KPI totals exceed the instance count and would let a
|
||||
// frozen dashboard keep reporting a healthy fleet.
|
||||
_store.Publish(Snapshot(
|
||||
Node("a", InstanceStatus.Up),
|
||||
Node("b", InstanceStatus.Up, lastPolled: Now.AddMinutes(-5))));
|
||||
|
||||
var page = RenderPage();
|
||||
|
||||
Assert.Equal("1", page.Find("[data-testid=kpi-up]").TextContent);
|
||||
Assert.Equal("1", page.Find("[data-testid=kpi-stale]").TextContent);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void GroupHeader_NamesTheLeader()
|
||||
{
|
||||
const string leader = "akka.tcp://scadabridge@a:8082";
|
||||
_store.Publish(Snapshot(
|
||||
Node("a", InstanceStatus.Up, "site-a", leader),
|
||||
Node("b", InstanceStatus.Up, "site-a", leader)));
|
||||
|
||||
Assert.Contains("Leader · a", RenderPage().Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void GroupHeader_WarnsOnDisagreement()
|
||||
{
|
||||
// The one output on this page that means "go look right now".
|
||||
_store.Publish(Snapshot(
|
||||
Node("a", InstanceStatus.Up, "site-a", "akka.tcp://scadabridge@a:8082"),
|
||||
Node("b", InstanceStatus.Up, "site-a", "akka.tcp://scadabridge@b:8082")));
|
||||
|
||||
Assert.Contains("Split brain", RenderPage().Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void AgreeingGroup_ShowsNoWarning()
|
||||
{
|
||||
const string leader = "akka.tcp://scadabridge@a:8082";
|
||||
_store.Publish(Snapshot(
|
||||
Node("a", InstanceStatus.Up, "site-a", leader),
|
||||
Node("b", InstanceStatus.Up, "site-a", leader)));
|
||||
|
||||
Assert.DoesNotContain("Split brain", RenderPage().Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void PublishedSnapshot_ReRendersThePage()
|
||||
{
|
||||
_store.Publish(Snapshot(Node("a", InstanceStatus.Up)));
|
||||
var page = RenderPage();
|
||||
Assert.Equal("1", page.Find("[data-testid=kpi-up]").TextContent);
|
||||
|
||||
_store.Publish(Snapshot(Node("a", InstanceStatus.Down)));
|
||||
|
||||
page.WaitForAssertion(() => Assert.Equal("0", page.Find("[data-testid=kpi-up]").TextContent));
|
||||
Assert.Equal("1", page.Find("[data-testid=kpi-down]").TextContent);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Paused_StopsAdoptingNewSnapshots()
|
||||
{
|
||||
_store.Publish(Snapshot(Node("a", InstanceStatus.Up)));
|
||||
var page = RenderPage();
|
||||
|
||||
page.Find("[data-testid=pause-toggle]").Click();
|
||||
_store.Publish(Snapshot(Node("a", InstanceStatus.Down)));
|
||||
|
||||
Assert.Equal("1", page.Find("[data-testid=kpi-up]").TextContent);
|
||||
Assert.Contains("Resume auto-refresh", page.Markup, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Resumed_AdoptsWhateverTheStoreHoldsNow()
|
||||
{
|
||||
// Resume must not replay the snapshots missed while paused — it must jump to current.
|
||||
_store.Publish(Snapshot(Node("a", InstanceStatus.Up)));
|
||||
var page = RenderPage();
|
||||
page.Find("[data-testid=pause-toggle]").Click();
|
||||
_store.Publish(Snapshot(Node("a", InstanceStatus.Down)));
|
||||
|
||||
page.Find("[data-testid=pause-toggle]").Click();
|
||||
|
||||
Assert.Equal("1", page.Find("[data-testid=kpi-down]").TextContent);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void PausedView_AgesIntoStaleOnItsOwn()
|
||||
{
|
||||
// This is why pause freezes adoption rather than stopping the clock: a paused dashboard
|
||||
// greys out by itself, so it cannot be mistaken for a live one.
|
||||
_store.Publish(Snapshot(Node("a", InstanceStatus.Up)));
|
||||
var page = RenderPage();
|
||||
page.Find("[data-testid=pause-toggle]").Click();
|
||||
|
||||
_time.Advance(TimeSpan.FromMinutes(5));
|
||||
page.Render();
|
||||
|
||||
Assert.Equal("1", page.Find("[data-testid=kpi-stale]").TextContent);
|
||||
Assert.Equal("0", page.Find("[data-testid=kpi-up]").TextContent);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Disposal_UnsubscribesFromTheStore()
|
||||
{
|
||||
// A leaked handler would call StateHasChanged on a disposed renderer every sweep, for the
|
||||
// life of the process — one leak per page load.
|
||||
_store.Publish(Snapshot(Node("a", InstanceStatus.Up)));
|
||||
var page = RenderPage();
|
||||
|
||||
page.Instance.Dispose();
|
||||
|
||||
var exception = Record.Exception(() => _store.Publish(Snapshot(Node("a", InstanceStatus.Down))));
|
||||
Assert.Null(exception);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ApplicationSection_CarriesAnAnchorMatchingTheRailLink()
|
||||
{
|
||||
// The rail's application links are fragment anchors; a mismatch makes every one dead.
|
||||
_store.Publish(Snapshot(Node("a", InstanceStatus.Up)));
|
||||
|
||||
var section = RenderPage().Find("[data-testid=application]");
|
||||
|
||||
Assert.Equal(
|
||||
ZB.MOM.WW.Overview.Components.Layout.MainLayout.Slug("ScadaBridge"),
|
||||
section.GetAttribute("id"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void OneCardPerRegisteredInstance()
|
||||
{
|
||||
_store.Publish(Snapshot(
|
||||
Node("a", InstanceStatus.Up, "site-a"),
|
||||
Node("b", InstanceStatus.Up, "site-a"),
|
||||
Node("gw", InstanceStatus.Up)));
|
||||
|
||||
Assert.Equal(3, RenderPage().FindAll("[data-testid=instance-card]").Count);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void UngroupedInstances_RenderAfterTheClusterGroups()
|
||||
{
|
||||
_store.Publish(Snapshot(
|
||||
Node("gw", InstanceStatus.Up),
|
||||
Node("a", InstanceStatus.Up, "site-a")));
|
||||
|
||||
// Enumerated, not indexed: bunit 1.40 was built against AngleSharp 1.2 and its
|
||||
// RefreshableElementCollection indexer calls an IHtmlCollection member whose signature
|
||||
// changed by 1.5.2 — the version this project pins for GHSA reasons. LINQ goes through
|
||||
// IEnumerable and is unaffected.
|
||||
var cards = RenderPage()
|
||||
.FindAll("[data-testid=instance-card]")
|
||||
.Select(c => c.GetAttribute("data-instance"))
|
||||
.ToList();
|
||||
|
||||
Assert.Equal(["a", "gw"], cards);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,433 @@
|
||||
using System.Net;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.Overview.Observability;
|
||||
using ZB.MOM.WW.Overview.Polling;
|
||||
using ZB.MOM.WW.Overview.Registry;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Sweep behaviour, driven directly rather than through the timer: the rules worth pinning are what
|
||||
/// a sweep probes, what it does with the answers, and when it declines to probe at all.
|
||||
/// </summary>
|
||||
public class OverviewPollerServiceTests
|
||||
{
|
||||
private static readonly DateTimeOffset Start = new(2026, 7, 24, 12, 0, 0, TimeSpan.Zero);
|
||||
|
||||
private static OverviewOptions Registry(params ApplicationEntry[] applications) =>
|
||||
new()
|
||||
{
|
||||
PollIntervalSeconds = 10,
|
||||
TimeoutSeconds = 3,
|
||||
StaleAfterSeconds = 45,
|
||||
Applications = applications,
|
||||
};
|
||||
|
||||
private static ApplicationEntry App(string name, params InstanceEntry[] instances) =>
|
||||
new() { Name = name, ManagementLabel = "AdminUI", Instances = instances };
|
||||
|
||||
private static InstanceEntry Node(
|
||||
string name,
|
||||
string baseUrl,
|
||||
string? group = null,
|
||||
bool hasActiveRole = false,
|
||||
int? pollIntervalSeconds = null) =>
|
||||
new()
|
||||
{
|
||||
Name = name,
|
||||
BaseUrl = baseUrl,
|
||||
Group = group,
|
||||
HasActiveRole = hasActiveRole,
|
||||
PollIntervalSeconds = pollIntervalSeconds,
|
||||
};
|
||||
|
||||
private static (OverviewPollerService Poller, OverviewSnapshotStore Store) Build(
|
||||
OverviewOptions options,
|
||||
HttpMessageHandler handler,
|
||||
TimeProvider timeProvider)
|
||||
{
|
||||
var store = new OverviewSnapshotStore();
|
||||
var poller = new OverviewPollerService(
|
||||
Options.Create(options),
|
||||
new StubHttpClientFactory(handler),
|
||||
store,
|
||||
timeProvider,
|
||||
new OverviewMetrics(new DummyMeterFactory(), store),
|
||||
new PollCycleHeartbeat(timeProvider),
|
||||
NullLogger<OverviewPollerService>.Instance);
|
||||
|
||||
return (poller, store);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_ProbesReadyForEveryInstance()
|
||||
{
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"))
|
||||
.Json("http://b:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"));
|
||||
var (poller, _) = Build(
|
||||
Registry(App("App", Node("a", "http://a:8080"), Node("b", "http://b:8080"))),
|
||||
handler,
|
||||
new FakeTimeProvider(Start));
|
||||
|
||||
var snapshot = await poller.SweepAsync();
|
||||
|
||||
Assert.Equal(2, snapshot.AllInstances.Count());
|
||||
Assert.All(snapshot.AllInstances, i => Assert.Equal(InstanceStatus.Up, i.Status));
|
||||
Assert.Equal(2, handler.Requests.Count);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_TrailingSlashInBaseUrl_DoesNotDoubleTheSeparator()
|
||||
{
|
||||
// "//health/ready" would 404 on every instance — a registry typo class that must not matter.
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"));
|
||||
var (poller, _) = Build(Registry(App("App", Node("a", "http://a:8080/"))), handler, new FakeTimeProvider(Start));
|
||||
|
||||
var snapshot = await poller.SweepAsync();
|
||||
|
||||
Assert.Equal(InstanceStatus.Up, Assert.Single(snapshot.AllInstances).Status);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_ActiveTierProbedOnlyForInstancesThatHaveTheRole()
|
||||
{
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"))
|
||||
.Status("http://a:8080/health/active", HttpStatusCode.OK)
|
||||
.Json("http://gw:5120/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"));
|
||||
var (poller, _) = Build(
|
||||
Registry(App("App", Node("a", "http://a:8080", "pair", hasActiveRole: true), Node("gw", "http://gw:5120"))),
|
||||
handler,
|
||||
new FakeTimeProvider(Start));
|
||||
|
||||
var snapshot = await poller.SweepAsync();
|
||||
|
||||
Assert.DoesNotContain("http://gw:5120/health/active", handler.Requests, StringComparer.Ordinal);
|
||||
var byName = snapshot.AllInstances.ToDictionary(i => i.Instance, StringComparer.Ordinal);
|
||||
Assert.Equal(ActiveState.Active, byName["a"].Active);
|
||||
Assert.Equal(ActiveState.NotApplicable, byName["gw"].Active);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_StandbyAnswers503OnActive_AndIsStillUp()
|
||||
{
|
||||
// 503 on the ACTIVE tier is the standby's correct answer and says nothing about health —
|
||||
// conflating the two tiers would paint every healthy standby in the fleet as Down.
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://b:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"))
|
||||
.Status("http://b:8080/health/active", HttpStatusCode.ServiceUnavailable);
|
||||
var (poller, _) = Build(
|
||||
Registry(App("App", Node("b", "http://b:8080", "pair", hasActiveRole: true))),
|
||||
handler,
|
||||
new FakeTimeProvider(Start));
|
||||
|
||||
var instance = Assert.Single((await poller.SweepAsync()).AllInstances);
|
||||
|
||||
Assert.Equal(InstanceStatus.Up, instance.Status);
|
||||
Assert.Equal(ActiveState.Standby, instance.Active);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_UnreachableInstance_SkipsTheActiveProbe()
|
||||
{
|
||||
// Spending a second full timeout to confirm what the first one established would double
|
||||
// the worst-case sweep duration for every dead node in the registry.
|
||||
var handler = new RoutingHandler().Fails("http://a:8080/health/ready", "Connection refused");
|
||||
var (poller, _) = Build(
|
||||
Registry(App("App", Node("a", "http://a:8080", "pair", hasActiveRole: true))),
|
||||
handler,
|
||||
new FakeTimeProvider(Start));
|
||||
|
||||
await poller.SweepAsync();
|
||||
|
||||
Assert.DoesNotContain("http://a:8080/health/active", handler.Requests, StringComparer.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_SickInstanceStillGetsItsActiveProbe()
|
||||
{
|
||||
// A 503 on ready means the app is talking. Whether the sick node is the ACTIVE half is
|
||||
// precisely what decides whether this is an outage or a standby nobody is using.
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.ServiceUnavailable, HealthBody.Ready("Unhealthy"))
|
||||
.Status("http://a:8080/health/active", HttpStatusCode.OK);
|
||||
var (poller, _) = Build(
|
||||
Registry(App("App", Node("a", "http://a:8080", "pair", hasActiveRole: true))),
|
||||
handler,
|
||||
new FakeTimeProvider(Start));
|
||||
|
||||
var instance = Assert.Single((await poller.SweepAsync()).AllInstances);
|
||||
|
||||
Assert.Equal(InstanceStatus.Down, instance.Status);
|
||||
Assert.Equal(ActiveState.Active, instance.Active);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_FirstFailureAfterAGoodPoll_IsDampedThenAdopted()
|
||||
{
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"));
|
||||
var time = new FakeTimeProvider(Start);
|
||||
var (poller, _) = Build(Registry(App("App", Node("a", "http://a:8080"))), handler, time);
|
||||
|
||||
Assert.Equal(InstanceStatus.Up, Assert.Single((await poller.SweepAsync()).AllInstances).Status);
|
||||
|
||||
handler.Fails("http://a:8080/health/ready", "Connection refused");
|
||||
|
||||
time.Advance(TimeSpan.FromSeconds(10));
|
||||
var damped = Assert.Single((await poller.SweepAsync()).AllInstances);
|
||||
Assert.Equal(InstanceStatus.Up, damped.Status);
|
||||
Assert.Equal(1, damped.ConsecutiveFailures);
|
||||
|
||||
time.Advance(TimeSpan.FromSeconds(10));
|
||||
var adopted = Assert.Single((await poller.SweepAsync()).AllInstances);
|
||||
Assert.Equal(InstanceStatus.Unreachable, adopted.Status);
|
||||
Assert.Equal(2, adopted.ConsecutiveFailures);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_LastReachableSurvivesAFailedPoll()
|
||||
{
|
||||
// "last seen" must keep pointing at the last real contact, not reset to null the moment
|
||||
// the instance goes away — that timestamp is how an operator judges how long it has been out.
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"));
|
||||
var time = new FakeTimeProvider(Start);
|
||||
var (poller, _) = Build(Registry(App("App", Node("a", "http://a:8080"))), handler, time);
|
||||
|
||||
await poller.SweepAsync();
|
||||
handler.Fails("http://a:8080/health/ready", "Connection refused");
|
||||
time.Advance(TimeSpan.FromSeconds(10));
|
||||
|
||||
var instance = Assert.Single((await poller.SweepAsync()).AllInstances);
|
||||
|
||||
Assert.Equal(Start, instance.LastReachableUtc);
|
||||
Assert.Equal(Start.AddSeconds(10), instance.LastPolledUtc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_BeforeAnInstanceIsDue_LeavesItAlone()
|
||||
{
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"));
|
||||
var time = new FakeTimeProvider(Start);
|
||||
var (poller, _) = Build(Registry(App("App", Node("a", "http://a:8080"))), handler, time);
|
||||
|
||||
await poller.SweepAsync();
|
||||
time.Advance(TimeSpan.FromSeconds(1));
|
||||
await poller.SweepAsync();
|
||||
|
||||
Assert.Single(handler.Requests);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_PerInstanceInterval_IsHonoured()
|
||||
{
|
||||
// The override is only real if a slow instance actually skips ticks while a fast one runs.
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://fast:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"))
|
||||
.Json("http://slow:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"));
|
||||
var time = new FakeTimeProvider(Start);
|
||||
var (poller, _) = Build(
|
||||
Registry(App(
|
||||
"App",
|
||||
Node("fast", "http://fast:8080", pollIntervalSeconds: 5),
|
||||
Node("slow", "http://slow:8080", pollIntervalSeconds: 30))),
|
||||
handler,
|
||||
time);
|
||||
|
||||
for (var i = 0; i < 4; i++)
|
||||
{
|
||||
await poller.SweepAsync();
|
||||
time.Advance(TimeSpan.FromSeconds(5));
|
||||
}
|
||||
|
||||
Assert.Equal(4, handler.Requests.Count(r => r.Contains("fast", StringComparison.Ordinal)));
|
||||
Assert.Equal(1, handler.Requests.Count(r => r.Contains("slow", StringComparison.Ordinal)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_DueCheckToleratesAnEarlyTick()
|
||||
{
|
||||
// A tick arriving a few ms early must not push the instance to half its configured rate.
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"));
|
||||
var time = new FakeTimeProvider(Start);
|
||||
var (poller, _) = Build(Registry(App("App", Node("a", "http://a:8080"))), handler, time);
|
||||
|
||||
await poller.SweepAsync();
|
||||
time.Advance(TimeSpan.FromMilliseconds(9_900));
|
||||
await poller.SweepAsync();
|
||||
|
||||
Assert.Equal(2, handler.Requests.Count);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_PublishesToTheStore()
|
||||
{
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"));
|
||||
var (poller, store) = Build(Registry(App("App", Node("a", "http://a:8080"))), handler, new FakeTimeProvider(Start));
|
||||
OverviewSnapshot? notified = null;
|
||||
store.Changed += s => notified = s;
|
||||
|
||||
var returned = await poller.SweepAsync();
|
||||
|
||||
Assert.Same(returned, store.Current);
|
||||
Assert.Same(returned, notified);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_CarriesRegistryMetadataOntoEveryCard()
|
||||
{
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"))
|
||||
.Status("http://a:8080/health/active", HttpStatusCode.OK);
|
||||
var application = App("ScadaBridge", Node("central-a", "http://a:8080", "central", hasActiveRole: true));
|
||||
application.Instances[0].ManagementUrl = "http://a:9000/";
|
||||
var (poller, _) = Build(Registry(application), handler, new FakeTimeProvider(Start));
|
||||
|
||||
var instance = Assert.Single((await poller.SweepAsync()).AllInstances);
|
||||
|
||||
Assert.Equal("ScadaBridge", instance.Application);
|
||||
Assert.Equal("central", instance.Group);
|
||||
Assert.Equal("AdminUI", instance.ManagementLabel);
|
||||
Assert.Equal("http://a:9000/", instance.ManagementUrl);
|
||||
Assert.Equal(TimeSpan.FromSeconds(45), instance.StaleAfter);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_ResolvesGroupLeadersIntoTheSnapshot()
|
||||
{
|
||||
const string leader = "akka.tcp://scadabridge@a:8082";
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8084/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy", ("akka-cluster", "Healthy", leader)))
|
||||
.Status("http://a:8084/health/active", HttpStatusCode.OK)
|
||||
.Json("http://b:8084/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy", ("akka-cluster", "Healthy", leader)))
|
||||
.Status("http://b:8084/health/active", HttpStatusCode.ServiceUnavailable);
|
||||
var (poller, _) = Build(
|
||||
Registry(App(
|
||||
"ScadaBridge",
|
||||
Node("site-a-a", "http://a:8084", "site-a", hasActiveRole: true),
|
||||
Node("site-a-b", "http://b:8084", "site-a", hasActiveRole: true))),
|
||||
handler,
|
||||
new FakeTimeProvider(Start));
|
||||
|
||||
var application = Assert.Single((await poller.SweepAsync()).Applications);
|
||||
var group = Assert.Single(application.Groups);
|
||||
|
||||
Assert.Equal("site-a", group.Name);
|
||||
Assert.Equal("site-a-a", group.LeaderInstance);
|
||||
Assert.False(group.Disagreement);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_WorstStatusRollsUpAcrossTheApplication()
|
||||
{
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"))
|
||||
.Json("http://b:8080/health/ready", HttpStatusCode.ServiceUnavailable, HealthBody.Ready("Unhealthy"));
|
||||
var (poller, _) = Build(
|
||||
Registry(App("App", Node("a", "http://a:8080"), Node("b", "http://b:8080"))),
|
||||
handler,
|
||||
new FakeTimeProvider(Start));
|
||||
|
||||
var application = Assert.Single((await poller.SweepAsync()).Applications);
|
||||
|
||||
Assert.Equal(InstanceStatus.Down, application.Worst);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_OneSlowInstanceDoesNotDelayTheOthers()
|
||||
{
|
||||
// The fan-out must be genuinely parallel: a serial sweep over a fleet with several dead
|
||||
// nodes would take timeout × dead-node-count and starve the whole dashboard.
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"))
|
||||
.Json("http://b:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"));
|
||||
var options = Registry(App("App", Node("a", "http://a:8080"), Node("b", "http://b:8080")));
|
||||
var gate = new SemaphoreSlim(0);
|
||||
var (poller, _) = Build(options, new GatedHandler(handler, gate), new FakeTimeProvider(Start));
|
||||
|
||||
var sweep = poller.SweepAsync();
|
||||
Assert.False(sweep.IsCompleted);
|
||||
|
||||
// Both probes must already be in flight; releasing one at a time proves neither waited.
|
||||
gate.Release(2);
|
||||
var snapshot = await sweep.WaitAsync(TimeSpan.FromSeconds(10));
|
||||
|
||||
Assert.All(snapshot.AllInstances, i => Assert.Equal(InstanceStatus.Up, i.Status));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_UnregisteredInstances_RenderAsPendingNotAsHealthy()
|
||||
{
|
||||
// Between host start and the first probe the registry is already published so the page has
|
||||
// its layout. Those cards must read "pending", never an unmeasured "Up".
|
||||
var handler = new RoutingHandler();
|
||||
var (poller, store) = Build(Registry(App("App", Node("a", "http://a:8080"))), handler, new FakeTimeProvider(Start));
|
||||
|
||||
Assert.Empty(store.Current.Applications);
|
||||
|
||||
handler.Fails("http://a:8080/health/ready", "Connection refused");
|
||||
var instance = Assert.Single((await poller.SweepAsync()).AllInstances);
|
||||
|
||||
Assert.Equal(InstanceStatus.Unreachable, instance.Status);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void PendingSnapshot_IsNeverStale()
|
||||
{
|
||||
// Staleness on a card that has never been polled would be meaningless and alarming.
|
||||
var pending = Snapshots.Instance("a", status: null);
|
||||
|
||||
Assert.False(pending.IsStale(Start.AddYears(1)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Snapshot_GoesStaleOnceItsWindowElapses()
|
||||
{
|
||||
var polled = Snapshots.Instance("a") with { LastPolledUtc = Start, StaleAfter = TimeSpan.FromSeconds(45) };
|
||||
|
||||
Assert.False(polled.IsStale(Start.AddSeconds(45)));
|
||||
Assert.True(polled.IsStale(Start.AddSeconds(46)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_HostShutdown_AbandonsTheSweepInsteadOfRecordingFailures()
|
||||
{
|
||||
// Writing "Unreachable" across the registry on the way out would leave a lie in the store.
|
||||
var handler = new RoutingHandler()
|
||||
.Json("http://a:8080/health/ready", HttpStatusCode.OK, HealthBody.Ready("Healthy"));
|
||||
var (poller, store) = Build(Registry(App("App", Node("a", "http://a:8080"))), handler, new FakeTimeProvider(Start));
|
||||
using var shutdown = new CancellationTokenSource();
|
||||
await shutdown.CancelAsync();
|
||||
|
||||
await Assert.ThrowsAnyAsync<OperationCanceledException>(() => poller.SweepAsync(shutdown.Token));
|
||||
|
||||
Assert.Empty(store.Current.Applications);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task EmptyRegistry_StartsAndStopsWithoutProbing()
|
||||
{
|
||||
var (poller, _) = Build(Registry(), new RoutingHandler(), new FakeTimeProvider(Start));
|
||||
|
||||
await poller.StartAsync(CancellationToken.None);
|
||||
await poller.StopAsync(CancellationToken.None);
|
||||
}
|
||||
|
||||
/// <summary>Holds every request until the test releases it, proving the fan-out is concurrent.</summary>
|
||||
private sealed class GatedHandler(RoutingHandler inner, SemaphoreSlim gate) : HttpMessageHandler
|
||||
{
|
||||
protected override async Task<HttpResponseMessage> SendAsync(
|
||||
HttpRequestMessage request, CancellationToken cancellationToken)
|
||||
{
|
||||
await gate.WaitAsync(cancellationToken).ConfigureAwait(false);
|
||||
return await inner.HandleAsync(request).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,97 @@
|
||||
using ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// The store is the cache-first seam: a page reads it and never probes. These tests pin that it
|
||||
/// always has something to serve and that subscribers hear about replacements.
|
||||
/// </summary>
|
||||
public class OverviewSnapshotStoreTests
|
||||
{
|
||||
private static OverviewSnapshot SnapshotAt(int second) =>
|
||||
new(new DateTimeOffset(2026, 7, 24, 12, 0, second, TimeSpan.Zero), []);
|
||||
|
||||
[Fact]
|
||||
public void Current_BeforeAnyPublish_IsEmptyNotNull()
|
||||
{
|
||||
// A page can render before the poller's first sweep finishes; a null here would be a
|
||||
// NullReferenceException on the landing page of a monitoring dashboard.
|
||||
var store = new OverviewSnapshotStore();
|
||||
|
||||
Assert.NotNull(store.Current);
|
||||
Assert.Empty(store.Current.Applications);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Publish_ReplacesCurrent()
|
||||
{
|
||||
var store = new OverviewSnapshotStore();
|
||||
var snapshot = SnapshotAt(1);
|
||||
|
||||
store.Publish(snapshot);
|
||||
|
||||
Assert.Same(snapshot, store.Current);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Publish_NotifiesSubscribersWithTheNewSnapshot()
|
||||
{
|
||||
var store = new OverviewSnapshotStore();
|
||||
OverviewSnapshot? received = null;
|
||||
store.Changed += s => received = s;
|
||||
|
||||
var snapshot = SnapshotAt(2);
|
||||
store.Publish(snapshot);
|
||||
|
||||
Assert.Same(snapshot, received);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Publish_AfterUnsubscribe_DoesNotNotify()
|
||||
{
|
||||
// Blazor circuits come and go; a handler that outlives its component would call
|
||||
// StateHasChanged on a disposed renderer on every sweep.
|
||||
var store = new OverviewSnapshotStore();
|
||||
var calls = 0;
|
||||
void Handler(OverviewSnapshot _) => calls++;
|
||||
|
||||
store.Changed += Handler;
|
||||
store.Publish(SnapshotAt(1));
|
||||
store.Changed -= Handler;
|
||||
store.Publish(SnapshotAt(2));
|
||||
|
||||
Assert.Equal(1, calls);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Publish_Null_Throws()
|
||||
{
|
||||
Assert.Throws<ArgumentNullException>(() => new OverviewSnapshotStore().Publish(null!));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Current_ReadDuringPublish_ObservesOneWholeSnapshot()
|
||||
{
|
||||
// The tearing guarantee: a reader always sees a complete graph, never a half-swapped one.
|
||||
var store = new OverviewSnapshotStore();
|
||||
var a = SnapshotAt(1);
|
||||
var b = SnapshotAt(2);
|
||||
store.Publish(a);
|
||||
|
||||
using var stop = new CancellationTokenSource();
|
||||
var reader = Task.Run(() =>
|
||||
{
|
||||
while (!stop.IsCancellationRequested)
|
||||
{
|
||||
var seen = store.Current;
|
||||
Assert.True(ReferenceEquals(seen, a) || ReferenceEquals(seen, b));
|
||||
}
|
||||
});
|
||||
|
||||
for (var i = 0; i < 1_000; i++)
|
||||
store.Publish(i % 2 == 0 ? b : a);
|
||||
|
||||
await stop.CancelAsync();
|
||||
await reader;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,197 @@
|
||||
using System.Diagnostics.Metrics;
|
||||
using System.Net;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
using ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Tests;
|
||||
|
||||
/// <summary>A meter factory with no listener attached, for tests that only need the poller to run.</summary>
|
||||
internal sealed class DummyMeterFactory : IMeterFactory
|
||||
{
|
||||
private readonly List<Meter> _meters = [];
|
||||
|
||||
public Meter Create(MeterOptions options)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
// Stamp ourselves as the scope, exactly as the real factory does. Meter names are global,
|
||||
// so a MeterListener that matched on name alone would pick up meters created by tests
|
||||
// running in parallel; the scope is what makes one test's instruments identifiable.
|
||||
options.Scope = this;
|
||||
|
||||
var meter = new Meter(options);
|
||||
_meters.Add(meter);
|
||||
return meter;
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
foreach (var meter in _meters)
|
||||
meter.Dispose();
|
||||
|
||||
_meters.Clear();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>A clock the test drives by hand.</summary>
|
||||
/// <remarks>
|
||||
/// Only <see cref="GetUtcNow"/> is overridden: the poller's sweep — the part with rules worth
|
||||
/// pinning — is exercised directly, so no test here needs a fake timer.
|
||||
/// </remarks>
|
||||
internal sealed class FakeTimeProvider(DateTimeOffset start) : TimeProvider
|
||||
{
|
||||
private DateTimeOffset _now = start;
|
||||
|
||||
public override DateTimeOffset GetUtcNow() => _now;
|
||||
|
||||
public void Advance(TimeSpan by) => _now += by;
|
||||
}
|
||||
|
||||
/// <summary>Serves canned responses per absolute URL and counts what was requested.</summary>
|
||||
internal sealed class RoutingHandler : HttpMessageHandler
|
||||
{
|
||||
private readonly Dictionary<string, Func<HttpResponseMessage>> _routes = new(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
public List<string> Requests { get; } = [];
|
||||
|
||||
public RoutingHandler Json(string url, HttpStatusCode status, string body)
|
||||
{
|
||||
_routes[url] = () => new HttpResponseMessage(status)
|
||||
{
|
||||
Content = new StringContent(body, Encoding.UTF8, "application/json"),
|
||||
};
|
||||
return this;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A status-code-only response with an EMPTY body — what the active tier actually returns when
|
||||
/// it is a bare gate endpoint rather than a full health report.
|
||||
/// </summary>
|
||||
public RoutingHandler Status(string url, HttpStatusCode status)
|
||||
{
|
||||
_routes[url] = () => new HttpResponseMessage(status)
|
||||
{
|
||||
Content = new StringContent(string.Empty, Encoding.UTF8, "application/json"),
|
||||
};
|
||||
return this;
|
||||
}
|
||||
|
||||
public RoutingHandler Fails(string url, string message)
|
||||
{
|
||||
_routes[url] = () => throw new HttpRequestException(message);
|
||||
return this;
|
||||
}
|
||||
|
||||
/// <summary>Lets a wrapping handler delegate here without re-entering the protected member.</summary>
|
||||
public Task<HttpResponseMessage> HandleAsync(HttpRequestMessage request) =>
|
||||
SendAsync(request, CancellationToken.None);
|
||||
|
||||
protected override Task<HttpResponseMessage> SendAsync(
|
||||
HttpRequestMessage request, CancellationToken cancellationToken)
|
||||
{
|
||||
var url = request.RequestUri!.ToString();
|
||||
|
||||
lock (Requests)
|
||||
Requests.Add(url);
|
||||
|
||||
if (!_routes.TryGetValue(url, out var factory))
|
||||
return Task.FromException<HttpResponseMessage>(new HttpRequestException($"no route for {url}"));
|
||||
|
||||
try
|
||||
{
|
||||
return Task.FromResult(factory());
|
||||
}
|
||||
catch (HttpRequestException ex)
|
||||
{
|
||||
return Task.FromException<HttpResponseMessage>(ex);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Hands every named client the same test handler.</summary>
|
||||
internal sealed class StubHttpClientFactory(HttpMessageHandler handler) : IHttpClientFactory
|
||||
{
|
||||
public HttpClient CreateClient(string name) =>
|
||||
new(handler, disposeHandler: false) { Timeout = Timeout.InfiniteTimeSpan };
|
||||
}
|
||||
|
||||
/// <summary>Builds canonical health bodies so tests state intent, not JSON.</summary>
|
||||
internal static class HealthBody
|
||||
{
|
||||
public static string Ready(string status, params (string Name, string Status, string? Leader)[] entries)
|
||||
{
|
||||
var payload = new
|
||||
{
|
||||
status,
|
||||
totalDurationMs = 1.5,
|
||||
entries = entries.ToDictionary(
|
||||
e => e.Name,
|
||||
e => (object)new
|
||||
{
|
||||
status = e.Status,
|
||||
description = $"{e.Name} says {e.Status}",
|
||||
durationMs = 0.5,
|
||||
data = e.Leader is null ? null : new Dictionary<string, object> { ["leader"] = e.Leader },
|
||||
},
|
||||
StringComparer.Ordinal),
|
||||
};
|
||||
|
||||
return JsonSerializer.Serialize(payload);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Builds a body whose entries carry arbitrary <c>data</c>, for the cases where the key under
|
||||
/// test is not <c>leader</c> — notably the active tier's <c>activeNode</c>.
|
||||
/// </summary>
|
||||
public static string WithData(string status, params (string Name, string Status, Dictionary<string, object>? Data)[] entries)
|
||||
{
|
||||
var payload = new
|
||||
{
|
||||
status,
|
||||
totalDurationMs = 1.5,
|
||||
entries = entries.ToDictionary(
|
||||
e => e.Name,
|
||||
e => (object)new
|
||||
{
|
||||
status = e.Status,
|
||||
description = $"{e.Name} says {e.Status}",
|
||||
durationMs = 0.5,
|
||||
data = e.Data,
|
||||
},
|
||||
StringComparer.Ordinal),
|
||||
};
|
||||
|
||||
return JsonSerializer.Serialize(payload);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Builds <see cref="InstanceSnapshot"/> values for the pure-logic tests.</summary>
|
||||
internal static class Snapshots
|
||||
{
|
||||
public static InstanceSnapshot Instance(
|
||||
string name,
|
||||
string? group = null,
|
||||
string baseUrl = "http://host:8080",
|
||||
bool hasActiveRole = true,
|
||||
InstanceStatus? status = InstanceStatus.Up,
|
||||
string? leader = null,
|
||||
string clusterCheckName = "akka-cluster") =>
|
||||
new()
|
||||
{
|
||||
Application = "App",
|
||||
Instance = name,
|
||||
Group = group,
|
||||
BaseUrl = baseUrl,
|
||||
HasActiveRole = hasActiveRole,
|
||||
Status = status,
|
||||
Report = leader is null ? null : ReportWithLeader(leader, clusterCheckName),
|
||||
StaleAfter = TimeSpan.FromSeconds(45),
|
||||
};
|
||||
|
||||
private static ZbHealthReport ReportWithLeader(string leader, string clusterCheckName)
|
||||
{
|
||||
var body = HealthBody.Ready("Healthy", (clusterCheckName, "Healthy", leader));
|
||||
return JsonSerializer.Deserialize<ZbHealthReport>(body)!;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,155 @@
|
||||
using ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Status derivation and two-strike flap damping. These rules decide what an operator sees, so
|
||||
/// every branch is pinned: the difference between Down and Unreachable sends someone to a
|
||||
/// different place entirely, and damping that fires the wrong way either hides a real outage or
|
||||
/// cries wolf on every dropped packet.
|
||||
/// </summary>
|
||||
public class StatusDerivationTests
|
||||
{
|
||||
private static HealthProbeResult Reachable(string reportStatus, int statusCode) =>
|
||||
new(true, statusCode, new ZbHealthReport { Status = reportStatus }, TimeSpan.FromMilliseconds(5), null);
|
||||
|
||||
private static HealthProbeResult Unreachable(string error) =>
|
||||
new(false, null, null, TimeSpan.FromMilliseconds(5), error);
|
||||
|
||||
[Theory]
|
||||
[InlineData("Healthy", 200, InstanceStatus.Up)]
|
||||
[InlineData("Degraded", 200, InstanceStatus.Degraded)]
|
||||
[InlineData("Unhealthy", 503, InstanceStatus.Down)]
|
||||
public void Observe_MapsCanonicalStatuses(string reportStatus, int code, InstanceStatus expected)
|
||||
{
|
||||
Assert.Equal(expected, StatusDerivation.Observe(Reachable(reportStatus, code)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Observe_TransportFailure_IsUnreachable_NotDown()
|
||||
{
|
||||
// The distinction is the whole point: Down means "the app answered and said it is sick",
|
||||
// Unreachable means "nothing answered" — usually VPN, firewall or a registry typo.
|
||||
Assert.Equal(InstanceStatus.Unreachable, StatusDerivation.Observe(Unreachable("Connection refused")));
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(null)]
|
||||
[InlineData("")]
|
||||
[InlineData("Weird")]
|
||||
public void Observe_UnrecognisedStatus_IsUnreachable(string? reportStatus)
|
||||
{
|
||||
// Parsed as JSON but not this contract — something else is on that port.
|
||||
Assert.Equal(InstanceStatus.Unreachable, StatusDerivation.Observe(Reachable(reportStatus!, 200)));
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(false, 200, ActiveState.NotApplicable)]
|
||||
[InlineData(false, 503, ActiveState.NotApplicable)]
|
||||
public void ObserveActive_WithoutActiveRole_IsNotApplicable(bool hasRole, int code, ActiveState expected)
|
||||
{
|
||||
Assert.Equal(expected, StatusDerivation.ObserveActive(hasRole, Reachable("Healthy", code)));
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(200, ActiveState.Active)]
|
||||
[InlineData(503, ActiveState.Standby)]
|
||||
[InlineData(404, ActiveState.Unknown)]
|
||||
public void ObserveActive_MapsStatusCodes(int code, ActiveState expected)
|
||||
{
|
||||
Assert.Equal(expected, StatusDerivation.ObserveActive(true, Reachable("Healthy", code)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ObserveActive_NoAnswer_IsUnknown_NotStandby()
|
||||
{
|
||||
// Guessing Standby on a failed probe would let a pair render with zero Active nodes during
|
||||
// a blip, or — worse, if guessed the other way — with two.
|
||||
Assert.Equal(ActiveState.Unknown, StatusDerivation.ObserveActive(true, Unreachable("timed out")));
|
||||
Assert.Equal(ActiveState.Unknown, StatusDerivation.ObserveActive(true, probe: null));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Damp_FirstFailureFromGoodState_HoldsThePreviousStatus()
|
||||
{
|
||||
var (status, failures) = StatusDerivation.Damp(InstanceStatus.Up, 0, InstanceStatus.Unreachable);
|
||||
|
||||
Assert.Equal(InstanceStatus.Up, status);
|
||||
Assert.Equal(1, failures);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Damp_SecondConsecutiveFailure_Flips()
|
||||
{
|
||||
var (status, failures) = StatusDerivation.Damp(InstanceStatus.Up, 1, InstanceStatus.Unreachable);
|
||||
|
||||
Assert.Equal(InstanceStatus.Unreachable, status);
|
||||
Assert.Equal(2, failures);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Damp_RecoveryIsImmediate_EvenAfterManyFailures()
|
||||
{
|
||||
var (status, failures) = StatusDerivation.Damp(InstanceStatus.Down, 27, InstanceStatus.Up);
|
||||
|
||||
Assert.Equal(InstanceStatus.Up, status);
|
||||
Assert.Equal(0, failures);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Damp_FirstEverPollThatFails_IsAdoptedImmediately()
|
||||
{
|
||||
// No previous state means nothing to protect and nothing better to show. Holding "Up" here
|
||||
// would be inventing a healthy reading that was never observed.
|
||||
var (status, failures) = StatusDerivation.Damp(previous: null, 0, InstanceStatus.Unreachable);
|
||||
|
||||
Assert.Equal(InstanceStatus.Unreachable, status);
|
||||
Assert.Equal(1, failures);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Damp_AlternatingFailures_NeverFlip()
|
||||
{
|
||||
// The flapping link this exists for: fail, recover, fail, recover. The counter must reset
|
||||
// on every good poll, so the card stays Up instead of oscillating.
|
||||
InstanceStatus? status = InstanceStatus.Up;
|
||||
var failures = 0;
|
||||
|
||||
foreach (var observed in new[]
|
||||
{
|
||||
InstanceStatus.Unreachable, InstanceStatus.Up,
|
||||
InstanceStatus.Unreachable, InstanceStatus.Up,
|
||||
InstanceStatus.Unreachable, InstanceStatus.Up,
|
||||
})
|
||||
{
|
||||
(var next, failures) = StatusDerivation.Damp(status, failures, observed);
|
||||
status = next;
|
||||
}
|
||||
|
||||
Assert.Equal(InstanceStatus.Up, status);
|
||||
Assert.Equal(0, failures);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Damp_DegradedIsNotAFailure_AndIsAdoptedImmediately()
|
||||
{
|
||||
// Degraded is a real answer from a live app, not a failed probe: it must show at once, and
|
||||
// must not accumulate strikes toward Down.
|
||||
var (status, failures) = StatusDerivation.Damp(InstanceStatus.Up, 1, InstanceStatus.Degraded);
|
||||
|
||||
Assert.Equal(InstanceStatus.Degraded, status);
|
||||
Assert.Equal(0, failures);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Damp_TwoDifferentFailureKindsInARow_StillFlips()
|
||||
{
|
||||
// Down then Unreachable is two consecutive failures, not one of each kind restarting the
|
||||
// count — the instance is not answering healthily either way.
|
||||
var (first, failures) = StatusDerivation.Damp(InstanceStatus.Up, 0, InstanceStatus.Down);
|
||||
Assert.Equal(InstanceStatus.Up, first);
|
||||
|
||||
var (second, _) = StatusDerivation.Damp(first, failures, InstanceStatus.Unreachable);
|
||||
Assert.Equal(InstanceStatus.Unreachable, second);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<!--
|
||||
Test stack mirrors ZB.MOM.WW.Theme's (xunit 2.9.3 + bunit), so component tests here read the
|
||||
same way as the kit's own.
|
||||
-->
|
||||
<PropertyGroup>
|
||||
<IsPackable>false</IsPackable>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.14.1" />
|
||||
<PackageReference Include="xunit" Version="2.9.3" />
|
||||
<PackageReference Include="xunit.runner.visualstudio" Version="3.1.4" />
|
||||
<PackageReference Include="bunit" Version="1.40.0" />
|
||||
<!-- Transitive-security pin, test-only: bunit 1.40.0 pulls AngleSharp 1.2.0, which carries
|
||||
NU1902 (GHSA-pgww-w46g-26qg) and therefore breaks the warnings-as-errors gate. Pinning the
|
||||
patched version here is the same fix HistorianGateway applied (6bc005d); AngleSharp never
|
||||
reaches the shipped app, only the bUnit renderer. -->
|
||||
<PackageReference Include="AngleSharp" Version="1.5.2" />
|
||||
<PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="10.0.7" />
|
||||
<PackageReference Include="coverlet.collector" Version="6.0.4" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<Using Include="Xunit" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\..\src\ZB.MOM.WW.Overview\ZB.MOM.WW.Overview.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
@@ -0,0 +1,252 @@
|
||||
using System.Net;
|
||||
using System.Text;
|
||||
using ZB.MOM.WW.Overview.Polling;
|
||||
|
||||
namespace ZB.MOM.WW.Overview.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Golden-payload tests for the health client. The fixtures are hand-written to the exact shape
|
||||
/// <c>ZbHealthWriter</c> emits — including an entry WITH <c>data</c> (0.2.0+) and one WITHOUT
|
||||
/// (every check that publishes none, and every pre-0.2.0 node), because tolerating both is what
|
||||
/// lets the dashboard run against a fleet that is only partly bumped.
|
||||
/// </summary>
|
||||
public class ZbHealthReportClientTests
|
||||
{
|
||||
private const string ReadyHealthyWithData = """
|
||||
{
|
||||
"status": "Healthy",
|
||||
"totalDurationMs": 12.34,
|
||||
"entries": {
|
||||
"akka-cluster": {
|
||||
"status": "Healthy",
|
||||
"description": "Akka cluster member status: Up",
|
||||
"durationMs": 1.23,
|
||||
"data": {
|
||||
"selfAddress": "akka.tcp://scadabridge@site-a-a:8082",
|
||||
"selfRoles": ["Site", "site-a"],
|
||||
"memberCount": 2,
|
||||
"unreachableCount": 0,
|
||||
"leader": "akka.tcp://scadabridge@site-a-a:8082"
|
||||
}
|
||||
},
|
||||
"localdb": {
|
||||
"status": "Healthy",
|
||||
"description": "Site LocalDb reachable.",
|
||||
"durationMs": 0.45
|
||||
}
|
||||
}
|
||||
}
|
||||
""";
|
||||
|
||||
private const string ReadyUnhealthyNoData = """
|
||||
{
|
||||
"status": "Unhealthy",
|
||||
"totalDurationMs": 3001.5,
|
||||
"entries": {
|
||||
"database": {
|
||||
"status": "Unhealthy",
|
||||
"description": null,
|
||||
"durationMs": 3000.1
|
||||
}
|
||||
}
|
||||
}
|
||||
""";
|
||||
|
||||
private static ZbHealthReportClient ClientReturning(
|
||||
HttpStatusCode statusCode,
|
||||
string body,
|
||||
string contentType = "application/json") =>
|
||||
new(new HttpClient(new StubHandler((statusCode, body, contentType))));
|
||||
|
||||
private static readonly TimeSpan ProbeTimeout = TimeSpan.FromSeconds(3);
|
||||
|
||||
[Fact]
|
||||
public async Task Probe_HealthyBody_ParsesEnvelopeAndEntries()
|
||||
{
|
||||
var client = ClientReturning(HttpStatusCode.OK, ReadyHealthyWithData);
|
||||
|
||||
var result = await client.ProbeAsync("http://node/health/ready", ProbeTimeout);
|
||||
|
||||
Assert.True(result.Reachable);
|
||||
Assert.Equal(200, result.StatusCode);
|
||||
Assert.Equal("Healthy", result.Report!.Status);
|
||||
Assert.Equal(12.34, result.Report.TotalDurationMs);
|
||||
Assert.Equal(new[] { "akka-cluster", "localdb" }, result.Report.Entries.Keys.OrderBy(k => k, StringComparer.Ordinal));
|
||||
Assert.Equal("Site LocalDb reachable.", result.Report.Entries["localdb"].Description);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Probe_EntryWithData_ExposesLeaderAndCounts()
|
||||
{
|
||||
var client = ClientReturning(HttpStatusCode.OK, ReadyHealthyWithData);
|
||||
|
||||
var result = await client.ProbeAsync("http://node/health/ready", ProbeTimeout);
|
||||
var entry = result.Report!.Entries["akka-cluster"];
|
||||
|
||||
Assert.Equal("akka.tcp://scadabridge@site-a-a:8082", entry.DataString("leader"));
|
||||
Assert.Equal(2, entry.DataInt("memberCount"));
|
||||
Assert.Equal(0, entry.DataInt("unreachableCount"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Probe_EntryWithoutData_ReadsAsNullNotAsAnError()
|
||||
{
|
||||
// The 0.1.0-era / no-data case. It must degrade to "no leader shown", never to a parse
|
||||
// failure — that tolerance is what keeps the dashboard usable mid-rollout.
|
||||
var client = ClientReturning(HttpStatusCode.OK, ReadyHealthyWithData);
|
||||
|
||||
var result = await client.ProbeAsync("http://node/health/ready", ProbeTimeout);
|
||||
var entry = result.Report!.Entries["localdb"];
|
||||
|
||||
Assert.True(result.Reachable);
|
||||
Assert.Null(entry.Data);
|
||||
Assert.Null(entry.DataString("leader"));
|
||||
Assert.Null(entry.DataInt("memberCount"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Probe_503_IsAParseableAnswer_NotAFailedProbe()
|
||||
{
|
||||
// 503 carries a full body naming the failing check — the single most useful payload the
|
||||
// dashboard receives. Treating it as a transport failure would throw that away.
|
||||
var client = ClientReturning(HttpStatusCode.ServiceUnavailable, ReadyUnhealthyNoData);
|
||||
|
||||
var result = await client.ProbeAsync("http://node/health/ready", ProbeTimeout);
|
||||
|
||||
Assert.True(result.Reachable);
|
||||
Assert.Equal(503, result.StatusCode);
|
||||
Assert.Equal("Unhealthy", result.Report!.Status);
|
||||
Assert.Null(result.Report.Entries["database"].Description);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("<html><body>502 Bad Gateway</body></html>")]
|
||||
[InlineData("not json at all")]
|
||||
public async Task Probe_NonJsonBody_IsUnreachable(string body)
|
||||
{
|
||||
var client = ClientReturning(HttpStatusCode.OK, body, contentType: "text/html");
|
||||
|
||||
var result = await client.ProbeAsync("http://node/health/ready", ProbeTimeout);
|
||||
|
||||
Assert.False(result.Reachable);
|
||||
Assert.Null(result.Report);
|
||||
Assert.NotNull(result.Error);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Probe_ConnectionRefused_IsUnreachableAndDoesNotThrow()
|
||||
{
|
||||
var client = new ZbHealthReportClient(
|
||||
new HttpClient(new ThrowingHandler(new HttpRequestException("Connection refused"))));
|
||||
|
||||
var result = await client.ProbeAsync("http://node/health/ready", ProbeTimeout);
|
||||
|
||||
Assert.False(result.Reachable);
|
||||
Assert.Contains("refused", result.Error, StringComparison.OrdinalIgnoreCase);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Probe_Timeout_IsUnreachableAndReportsTheTimeout()
|
||||
{
|
||||
var client = new ZbHealthReportClient(new HttpClient(new HangingHandler()));
|
||||
|
||||
var result = await client.ProbeAsync("http://node/health/ready", TimeSpan.FromMilliseconds(50));
|
||||
|
||||
Assert.False(result.Reachable);
|
||||
Assert.Contains("timed out", result.Error, StringComparison.OrdinalIgnoreCase);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Probe_HostShutdown_PropagatesCancellation()
|
||||
{
|
||||
// A shutdown must NOT be recorded as a timed-out instance — the sweep is being abandoned,
|
||||
// and writing "Unreachable" for every node on the way out would be a lie in the snapshot.
|
||||
using var shutdown = new CancellationTokenSource();
|
||||
await shutdown.CancelAsync();
|
||||
var client = new ZbHealthReportClient(new HttpClient(new HangingHandler()));
|
||||
|
||||
await Assert.ThrowsAnyAsync<OperationCanceledException>(
|
||||
() => client.ProbeAsync("http://node/health/ready", ProbeTimeout, shutdown.Token));
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(HttpStatusCode.OK, 200)]
|
||||
[InlineData(HttpStatusCode.ServiceUnavailable, 503)]
|
||||
public async Task ProbeStatus_EmptyBody_IsStillAnAnswer(HttpStatusCode status, int expected)
|
||||
{
|
||||
// The active tier answers with a status code; the body is not part of that contract. An
|
||||
// empty 200/503 is the endpoint telling us the role plainly, and insisting on parseable
|
||||
// JSON here reports every such node as "role unknown".
|
||||
var client = ClientReturning(status, string.Empty);
|
||||
|
||||
var result = await client.ProbeStatusAsync("http://node/health/active", ProbeTimeout);
|
||||
|
||||
Assert.True(result.Reachable);
|
||||
Assert.Equal(expected, result.StatusCode);
|
||||
Assert.Null(result.Report);
|
||||
Assert.Null(result.Error);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ProbeStatus_StillParsesABodyWhenOneIsThere()
|
||||
{
|
||||
var client = ClientReturning(HttpStatusCode.OK, ReadyHealthyWithData);
|
||||
|
||||
var result = await client.ProbeStatusAsync("http://node/health/active", ProbeTimeout);
|
||||
|
||||
Assert.Equal("Healthy", result.Report!.Status);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ProbeStatus_TransportFailure_IsStillUnreachable()
|
||||
{
|
||||
// Relaxing the body requirement must not relax what "reachable" means.
|
||||
var client = new ZbHealthReportClient(
|
||||
new HttpClient(new ThrowingHandler(new HttpRequestException("Connection refused"))));
|
||||
|
||||
var result = await client.ProbeStatusAsync("http://node/health/active", ProbeTimeout);
|
||||
|
||||
Assert.False(result.Reachable);
|
||||
Assert.Null(result.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Probe_ReadyTierStillDemandsAParseableBody()
|
||||
{
|
||||
// The ready tier's body IS the payload — the per-check list and the leader data come from
|
||||
// it. An empty 200 there means something other than a family app is on that port.
|
||||
var client = ClientReturning(HttpStatusCode.OK, string.Empty);
|
||||
|
||||
var result = await client.ProbeAsync("http://node/health/ready", ProbeTimeout);
|
||||
|
||||
Assert.False(result.Reachable);
|
||||
}
|
||||
|
||||
private sealed class StubHandler((HttpStatusCode Status, string Body, string ContentType) response)
|
||||
: HttpMessageHandler
|
||||
{
|
||||
protected override Task<HttpResponseMessage> SendAsync(
|
||||
HttpRequestMessage request, CancellationToken cancellationToken) =>
|
||||
Task.FromResult(new HttpResponseMessage(response.Status)
|
||||
{
|
||||
Content = new StringContent(response.Body, Encoding.UTF8, response.ContentType),
|
||||
});
|
||||
}
|
||||
|
||||
private sealed class ThrowingHandler(Exception exception) : HttpMessageHandler
|
||||
{
|
||||
protected override Task<HttpResponseMessage> SendAsync(
|
||||
HttpRequestMessage request, CancellationToken cancellationToken) =>
|
||||
Task.FromException<HttpResponseMessage>(exception);
|
||||
}
|
||||
|
||||
private sealed class HangingHandler : HttpMessageHandler
|
||||
{
|
||||
protected override async Task<HttpResponseMessage> SendAsync(
|
||||
HttpRequestMessage request, CancellationToken cancellationToken)
|
||||
{
|
||||
await Task.Delay(Timeout.InfiniteTimeSpan, cancellationToken).ConfigureAwait(false);
|
||||
return new HttpResponseMessage(HttpStatusCode.OK);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,531 @@
|
||||
# Akka.NET HA, Failover & Singletons — ScadaBridge + OtOpcUa Deep Dive
|
||||
|
||||
*Written 2026-07-22 against ScadaBridge `main @ 69b3ccfc` and OtOpcUa (`lmxopcua`) `master @ 0de1352e`.
|
||||
Both trees contain the 2026-07-21 downing/election corrections (ScadaBridge `cf3bd52f`; OtOpcUa `2964361a` auto-down + `50b55ee4` oldest-driver election — note `0de1352e` itself is the adjacent NU1903 dependency pin, not the redundancy fix).*
|
||||
|
||||
**Design priority (stated 2026-07-22):** for the 2-node site pairs, a network-partition
|
||||
split-brain is *less* risky than failing to bring the second node up quickly. HA goal:
|
||||
either node can be launched first, automatic failover, no manual intervention. The
|
||||
architecture below is evaluated against that priority throughout.
|
||||
|
||||
---
|
||||
|
||||
## 1. How Akka.NET handles failover — the fundamentals both apps build on
|
||||
|
||||
The deployment unit at every site is a **2-node Akka.NET cluster** (one node per Windows
|
||||
VM). Central is also a 2-node cluster, with SQL Server; sites have no SQL Server.
|
||||
Everything HA-related decomposes into five Akka mechanisms:
|
||||
|
||||
### 1.1 Membership and seed nodes
|
||||
A cluster forms by gossip. Each node lists `seed-nodes`; a booting node sends `InitJoin`
|
||||
to the seeds. **Only the FIRST seed in the list may join itself to form a *new* cluster** —
|
||||
any other node waits until some existing member answers. Both apps set
|
||||
`min-nr-of-members = 1`, so a single node (if it's allowed to form) goes `Up` alone and
|
||||
hosts all singletons — the pair does not need both members to become operational.
|
||||
|
||||
### 1.2 Failure detection (the heartbeat)
|
||||
Every node heartbeats its peers and feeds the intervals into a **Phi Accrual failure
|
||||
detector**. When the suspicion level crosses the threshold, the peer is marked
|
||||
**Unreachable** — a suspicion, not a removal. Both apps run the same values:
|
||||
|
||||
| Setting | Value (both apps) | Meaning |
|
||||
|---|---|---|
|
||||
| `failure-detector.heartbeat-interval` | **2 s** | Cluster-membership heartbeat rate |
|
||||
| `failure-detector.threshold` | 10.0 | Phi threshold (suspicion sensitivity) |
|
||||
| `failure-detector.acceptable-heartbeat-pause` | **10 s** | Tolerated silence before Unreachable |
|
||||
| `transport-failure-detector` (remoting layer) | 2 s / 10 s pause | Detects dead TCP associations |
|
||||
|
||||
So a hard-killed peer is declared Unreachable roughly **10–12 s** after death. These are
|
||||
exactly the "heartbeat with a small adjustable timeout" — in ScadaBridge they are bound
|
||||
from config (`Cluster:HeartbeatInterval`, `Cluster:FailureDetectionThreshold` →
|
||||
`ClusterOptions`, rendered into HOCON by `AkkaHostedService.BuildHocon`); in OtOpcUa they
|
||||
live in `akka.conf` (lines 68–72) in the Cluster library.
|
||||
|
||||
### 1.3 Downing (the decision to act)
|
||||
Unreachable alone changes nothing — the leader cannot make membership decisions while any
|
||||
member is unreachable, singletons do not move, the cluster is wedged. Something must
|
||||
**Down** the unreachable member. This is the strategy choice, and it is the whole
|
||||
availability-vs-partition-safety trade:
|
||||
|
||||
- **Auto-downing** (`Akka.Cluster.AutoDowning` + `auto-down-unreachable-after`): after the
|
||||
window, the leader *among the reachable members* downs the unreachable one. In a 1-vs-1
|
||||
split, **each side downs the other** — the crash case fails over perfectly; a genuine
|
||||
partition produces dual-active.
|
||||
- **Split Brain Resolver `keep-oldest` (+ `down-if-alone`)**: keeps the side containing
|
||||
the oldest member. Partition-safe, but **fatally broken for a 2-node pair**: in
|
||||
Akka.NET 1.5.62, `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side
|
||||
holding ≥ 2 members (`otherSide == 1 && thisSide >= 2`). A 1-vs-1 survivor whose peer
|
||||
(the oldest) crashed takes `DownReachable` and **downs itself** — proven live:
|
||||
`SBR took decision Akka.Cluster.SBR.DownReachable … including myself, [1] unreachable of [2] members`.
|
||||
Net effect: the pair could not survive a crash of the oldest node — total outage.
|
||||
|
||||
**Both apps switched their default from `keep-oldest` to `auto-down` on 2026-07-21**
|
||||
(ScadaBridge `cf3bd52f`, decision record
|
||||
`ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md`; OtOpcUa
|
||||
`2964361a`, same fix ported). The owner rationale recorded in the decision doc is
|
||||
verbatim the priority stated above: *"network partitions are less of a risk than if this
|
||||
stops working."*
|
||||
|
||||
Rejected alternatives (from the decision record): `static-quorum(1)` → `DownAll` (worse);
|
||||
`static-quorum(2)` → survivor self-downs; `keep-majority` → keeps the lowest-address side,
|
||||
same fatal crash re-keyed; `lease-majority` → needs a shared lease store (no SQL/K8s at
|
||||
sites); a third arbiter node → no third VM at sites; a custom downing provider → would
|
||||
reimplement AutoDowning.
|
||||
|
||||
### 1.4 Cluster singletons (what actually "fails over")
|
||||
`ClusterSingletonManager` runs on every node of the (optionally role-filtered) cluster
|
||||
and guarantees exactly one instance of the actor, hosted on the **oldest member** of that
|
||||
role. `ClusterSingletonProxy` routes to wherever it currently lives. Failover semantics:
|
||||
|
||||
- **Graceful shutdown** (service stop, redeploy): hand-over protocol — the old host
|
||||
transfers to the new oldest with no dead window beyond hand-over latency. Both apps
|
||||
hook `CoordinatedShutdown` (`run-by-clr-shutdown-hook = on`; ScadaBridge additionally
|
||||
drains each singleton via a `PhaseClusterLeave` `GracefulStop` task, 10 s per singleton,
|
||||
`phases.cluster-leave.timeout = 15s`).
|
||||
- **Hard crash**: the singleton is simply gone until the dead node is **Downed**; then the
|
||||
new oldest member's manager instantiates a fresh instance. This is why downing is the
|
||||
linchpin — no downing decision, no singleton recovery, ever.
|
||||
|
||||
### 1.5 Oldest-member semantics — the family-wide rule
|
||||
Both codebases converged on the same insight: **Akka's "leader" (lowest address) and
|
||||
"oldest member" (singleton placement) diverge permanently after any node restart** — the
|
||||
restarted node rejoins as a *new incarnation*, becoming the youngest, but may still have
|
||||
the lowest address and thus be leader. Any "active node" decision keyed on
|
||||
leader/lowest-address can therefore point at a node that is *not* hosting the singletons.
|
||||
Both apps now derive "active/primary" strictly from **oldest Up member** (ScadaBridge
|
||||
`ActiveNodeEvaluator.SelfIsOldestUp`; OtOpcUa `RedundancyStateActor.SelectDriverPrimary`
|
||||
with `Member.AgeOrdering`), which by construction matches singleton placement.
|
||||
|
||||
---
|
||||
|
||||
## 2. The failover timeline (and the knobs that tune it)
|
||||
|
||||
End-to-end failover after a hard node death, with current production values:
|
||||
|
||||
```
|
||||
t=0 node dies (process crash / VM loss)
|
||||
t≈10-12s failure detector: peer marked Unreachable ← acceptable-heartbeat-pause (10s)
|
||||
t≈25-27s auto-down window expires; survivor downs peer ← auto-down-unreachable-after (15s)
|
||||
t≈25-30s survivor is now oldest; singletons instantiate;
|
||||
primary/active gates flip; ServiceLevel 250 moves
|
||||
```
|
||||
|
||||
Measured, not theoretical: ScadaBridge's 2026-07-21 docker drill measured **28 s**
|
||||
(active-node crash → standby takeover) and **27 s** (standby crash → removal, zero routing
|
||||
blips); the in-process `FailoverTimingTests` measured 33.7 s for a full cycle.
|
||||
|
||||
**Tuning for faster failover** (per the availability-first priority):
|
||||
|
||||
| Knob | Where | Current | Effect of lowering |
|
||||
|---|---|---|---|
|
||||
| `acceptable-heartbeat-pause` | SB: `Cluster:FailureDetectionThreshold`; OtOpcUa: `akka.conf` | 10 s | Faster Unreachable, but < ~5 s risks false positives from GC pauses / Windows VM scheduling stalls / snapshot freezes |
|
||||
| `heartbeat-interval` | SB: `Cluster:HeartbeatInterval`; OtOpcUa: `akka.conf` | 2 s | More samples → detector converges faster; cheap to lower to 1 s |
|
||||
| Auto-down window | SB: `Cluster:StableAfter`; OtOpcUa: `DowningStableAfter` const (15 s) + emitted HOCON | 15 s | Direct 1:1 reduction of failover time. Every second cut is a second of dual-active risk during a *transient* blip (link flap, VM pause) — the window exists to let transients heal before the irreversible Down |
|
||||
| `down-removal-margin` (OtOpcUa akka.conf) | 15 s | Delay before rebalancing after removal | Keep ≈ downing window |
|
||||
|
||||
Constraint pinned by an OtOpcUa test (`Downing_window_exceeds_the_acceptable_heartbeat_pause`):
|
||||
the downing window **must exceed** `acceptable-heartbeat-pause`, otherwise a node can be
|
||||
downed on a pause the detector was configured to tolerate. A realistic aggressive profile
|
||||
for the site pairs would be ~5 s pause + ~7–8 s window ≈ **12–13 s total failover**,
|
||||
at the cost of downing a peer that stalls (not dies) for > ~12 s — acceptable per the
|
||||
stated priority, but worth validating against real Windows VM GC/backup/snapshot pauses
|
||||
before rollout. In ScadaBridge these are plain per-deployment config values; in OtOpcUa
|
||||
the 15 s window is currently a constant in `ServiceCollectionExtensions.cs` (only the
|
||||
*strategy* is config-selectable), so making the window configurable would be a small
|
||||
change there.
|
||||
|
||||
**What happens to the dead node's side:** both apps set
|
||||
`run-coordinated-shutdown-when-down = on` — a node that gets Downed (e.g. it was merely
|
||||
partitioned and the other side downed it) terminates its own ActorSystem. Both apps run a
|
||||
termination watchdog (ScadaBridge in `AkkaHostedService`; OtOpcUa
|
||||
`ActorSystemTerminationWatchdog`) that distinguishes intentional stop from self-down and
|
||||
calls `StopApplication()`, so the **process exits and the service supervisor restarts it**
|
||||
(Windows: `sc.exe failure … restart/5000/restart/30000/restart/60000`; Docker:
|
||||
`restart: unless-stopped`). The restarted process rejoins as a fresh incarnation —
|
||||
youngest member, i.e. it comes back as the **standby**, never usurping the survivor.
|
||||
|
||||
---
|
||||
|
||||
## 3. ScadaBridge
|
||||
|
||||
### 3.1 Cluster formation
|
||||
All HOCON is generated in code — `AkkaHostedService.BuildHocon(...)`
|
||||
(`src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs:258–329`) from
|
||||
`ClusterOptions`/`NodeOptions` (`src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs`).
|
||||
Key facts:
|
||||
|
||||
- **Seed nodes**: central docker pair → `scadabridge-central-a:8081` then
|
||||
`…-central-b:8081` (both nodes list `central-a` first). Site pairs analogous on 8082.
|
||||
- **`MinNrOfMembers = 1`** (validator *requires* exactly 1) — one node forms the cluster
|
||||
and brings up all singletons.
|
||||
- **Roles**: central node = `["Central"]`; site node = `["Site", "site-{SiteId}"]`
|
||||
(`BuildRoles`, line 406). The role drives the entire composition branch in `Program.cs`.
|
||||
- **Ports**: central remoting 8081; site remoting 8082, gRPC 8083, metrics 8084.
|
||||
|
||||
### 3.2 Downing configuration
|
||||
Configurable via **`Cluster:SplitBrainResolverStrategy`**, allowed values `auto-down` |
|
||||
`keep-oldest` (case-insensitive; validator `ClusterOptionsValidator`). **Default and all
|
||||
16 appsettings files: `auto-down`.** The HOCON branch (`AkkaHostedService.cs:275–286`):
|
||||
|
||||
```csharp
|
||||
var downingBlock = string.Equals(strategy, "auto-down", OrdinalIgnoreCase)
|
||||
? $@"downing-provider-class = ""Akka.Cluster.AutoDowning, Akka.Cluster""
|
||||
auto-down-unreachable-after = {DurationHocon(clusterOptions.StableAfter)}"
|
||||
: /* SBR provider + split-brain-resolver { active-strategy, stable-after,
|
||||
keep-oldest { down-if-alone } } */;
|
||||
```
|
||||
|
||||
`StableAfter = 00:00:15` doubles as the SBR `stable-after` and the auto-down window.
|
||||
`DownIfAlone` is now validated only under `keep-oldest` (inert under auto-down).
|
||||
`keep-oldest` remains supported and SBR-path tests still pin it
|
||||
(`SbrFailoverTests.HardCrashOfYoungerNode_SbrDownsIt_AndOldestKeepsSingleton`).
|
||||
|
||||
### 3.3 Cluster singletons
|
||||
All registered through `SingletonRegistrar.Start(...)` (names `{name}-singleton` /
|
||||
`{name}-proxy`, PoisonPill termination, optional role, per-singleton
|
||||
`PhaseClusterLeave` drain task of 10 s).
|
||||
|
||||
**Central — 7 singletons** (role not further scoped; central nodes are all `Central`):
|
||||
|
||||
| Singleton | Actor | Purpose |
|
||||
|---|---|---|
|
||||
| `notification-outbox` | `NotificationOutboxActor` | Notification ingest/dispatch/purge |
|
||||
| `audit-log-ingest` | `AuditLogIngestActor` | Central audit ingest from sites |
|
||||
| `site-call-audit` | `SiteCallAuditActor` | SiteCalls upserts + central→site retry/discard relay |
|
||||
| `audit-log-purge` | `AuditLogPurgeActor` | Daily partition-switch purge |
|
||||
| `site-audit-reconciliation` | `SiteAuditReconciliationActor` | Per-site audit reconciliation pull |
|
||||
| `kpi-history-recorder` | `KpiHistoryRecorderActor` | KPI sampling (not readiness-gated) |
|
||||
| `pending-deployment-purge` | `PendingDeploymentPurgeActor` | Expired staging-row reclaim (not readiness-gated) |
|
||||
|
||||
`RequiredSingletonsHealthCheck` probes the first five via bounded `Identify` for
|
||||
`/health/ready`; the last two are deliberately best-effort.
|
||||
|
||||
**Site — 2 singletons**, role-scoped to `site-{SiteId}` so each site pair elects its own:
|
||||
|
||||
| Singleton | Actor | Purpose |
|
||||
|---|---|---|
|
||||
| `deployment-manager` | `DeploymentManagerActor` | Deployment lifecycle; drains in-flight SQLite writes on hand-over |
|
||||
| `event-log-handler` | `EventLogHandlerActor` | Event-log queries always served by the active node (event log is node-local, not replicated) |
|
||||
|
||||
Deliberately **not** singletons: `dcl-manager` (DataConnectionManagerActor) and
|
||||
`CertStoreActor` run on **every** site node — the deployment singleton fans PKI trust
|
||||
changes to both nodes' cert stores so a failover never finds a stale trust store.
|
||||
|
||||
### 3.4 Active-node gating
|
||||
One definition of "active": `ActiveNodeEvaluator.SelfIsOldestUp`
|
||||
(`src/ZB.MOM.WW.ScadaBridge.Communication/ClusterState/ActiveNodeEvaluator.cs`) — oldest
|
||||
Up member, explicitly **not** `cluster.State.Leader` (the XML docs spell out the
|
||||
leader-vs-oldest divergence after restarts). Consumers:
|
||||
|
||||
- **Central inbound API**: `IActiveNodeGate` → `InboundApiEndpointFilter` returns
|
||||
**HTTP 503 `{code:"STANDBY_NODE"}`** on the standby; Traefik routes on
|
||||
`/health/active` (`OldestNodeActiveHealthCheck` + `DatabaseHealthCheck`), so the proxy
|
||||
and the API agree on which node is active.
|
||||
- **Site store-and-forward**: `storeAndForwardService.SetDeliveryGate(() =>
|
||||
clusterNodeProvider.SelfIsPrimary)` — only the active node runs the delivery sweep;
|
||||
without the gate both nodes would deliver the same replicated `sf_messages` rows
|
||||
(systematic duplicate external calls). Re-evaluated per sweep tick, so delivery resumes
|
||||
within one `RetryTimerInterval` after failover.
|
||||
- **Heartbeat to central**: `SiteCommunicationActor` stamps `HeartbeatMessage.IsActive`
|
||||
from the same evaluator (exception-safe, defaults to `false`).
|
||||
|
||||
Takeover is purely membership-driven: peer Downed → survivor becomes oldest →
|
||||
`SelfIsOldestUp` flips → singletons, S&F gate, `/health/active`, heartbeat all follow.
|
||||
|
||||
### 3.5 Site state without SQL Server — LocalDb
|
||||
Sites persist everything in **one consolidated SQLite file** (`LocalDb:Path`, required,
|
||||
on the data volume) via `ZB.MOM.WW.LocalDb` — 10 replicated tables: `OperationTracking`,
|
||||
`site_events`, `sf_messages` (the store-and-forward buffer), and 7 config tables
|
||||
(deployed configurations, static overrides, shared scripts, external systems, DB
|
||||
connections, DCL definitions, native alarm state).
|
||||
|
||||
**Pair replication** (default OFF; opt-in `LocalDb:Replication:PeerAddress` + matching
|
||||
`ApiKey` both sides; fail-closed auth interceptor; rides the existing gRPC 8083 listener):
|
||||
CDC triggers capture row mutations, shipped bidirectionally under HLC last-writer-wins;
|
||||
snapshot resync merges per row and never deletes. **This is what makes site failover
|
||||
real**: the standby already holds the S&F buffer and the full config, so on takeover it
|
||||
just opens its delivery gate over data it already has — zero cross-node fetch (the old
|
||||
notify-and-fetch deploy path is deleted).
|
||||
|
||||
Not replicated (by design): the event log (hence the `event-log-handler` singleton),
|
||||
in-memory alarm state (re-evaluated from incoming values on the survivor), and
|
||||
`notification_lists`/`smtp_configurations` (would ship plaintext SMTP passwords).
|
||||
|
||||
**Operational constraints:**
|
||||
- **Stop/start a site pair TOGETHER.** Rolling upgrades are unsupported: Phase 2 deleted
|
||||
the legacy replicator and its `SfBufferSnapshot` compat handler, so a mixed-version
|
||||
pair runs but silently does not converge.
|
||||
- A node offline longer than `TombstoneRetention` (7 d) can **resurrect deleted rows** on
|
||||
rejoin; within the window rejoin is safe and self-correcting.
|
||||
- `MaxBatchSize` counts **rows, not bytes** — rig pins 16 (the 500 default could build a
|
||||
~35 MB batch against the 4 MB gRPC cap).
|
||||
|
||||
### 3.6 Verified behavior + remaining gaps
|
||||
- Auto-down failover drill (docker `failover-drill.sh`, SIGKILL): active-crash takeover
|
||||
28 s, standby-crash 27 s, 0 routing blips; both drill modes now *assert* recovery.
|
||||
- After a real partition the two mutually-downed halves **do not merge** (quarantined
|
||||
association) — an operator restarts one side, which rejoins as standby. This is the
|
||||
accepted dual-active trade.
|
||||
- **The old "restart central-a and central-b together (SBR self-down)" gotcha is closed**
|
||||
by auto-down — but a related constraint survives with a different mechanism: the
|
||||
**seed-node bootstrap constraint** (§6.1).
|
||||
- ⚠ Stale doc: `docs/deployment/topology-guide.md:101` still says keep-oldest +
|
||||
down-if-alone. Authoritative: `Component-ClusterInfrastructure.md`, the auto-down
|
||||
decision record, `docker/README.md`.
|
||||
|
||||
---
|
||||
|
||||
## 4. OtOpcUa
|
||||
|
||||
### 4.1 Cluster formation
|
||||
Base HOCON in the library (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf`):
|
||||
system `otopcua`, remoting port 4053, `min-nr-of-members = 1`, empty seeds/roles overlaid
|
||||
at runtime from `AkkaClusterOptions` (`Cluster:*` config; roles restricted to
|
||||
`admin` | `driver` | `dev` by `RoleParser`). Bootstrap via Akka.Hosting:
|
||||
`WithOtOpcUaClusterBootstrap` (remoting + clustering + the downing HOCON, Prepended).
|
||||
Note the dual role plumbing: `OTOPCUA_ROLES` drives DI composition (`hasAdmin`/`hasDriver`
|
||||
branches in `Program.cs`), while `Cluster__Roles__*` drives the actual Akka member roles —
|
||||
docker-dev sets both (a past production incident came from setting only `OTOPCUA_ROLES`,
|
||||
leaving the Akka member role-less).
|
||||
|
||||
**Current mesh shape (important):** today OtOpcUa runs **one Akka mesh for the whole
|
||||
fleet** — docker-dev is six nodes (central-1/2 = `admin,driver`; site-a-1/2, site-b-1/2 =
|
||||
`driver`) all seeded by `central-1`, with logical MAIN/SITE-A/SITE-B separation done by
|
||||
`ServerCluster.ClusterId` rows in the shared ConfigDb, not by separate meshes. The
|
||||
per-cluster mesh design (§4.4) will change this to one mesh per 2-node pair — the shape
|
||||
that matches your site deployment.
|
||||
|
||||
### 4.2 Downing configuration
|
||||
Same key as ScadaBridge: **`Cluster:SplitBrainResolverStrategy`**, default **`auto-down`**,
|
||||
allowed `auto-down` | `keep-oldest`, anything else **fails the host at startup**
|
||||
(`ArgumentOutOfRangeException` — no silent fallback; pinned by
|
||||
`Unknown_strategy_throws_rather_than_silently_falling_back`). `BuildDowningHocon` emits:
|
||||
|
||||
```
|
||||
akka.cluster {
|
||||
downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster"
|
||||
auto-down-unreachable-after = 15000ms
|
||||
}
|
||||
```
|
||||
|
||||
(the comment notes `auto-down-unreachable-after` defaults to `off` upstream and is
|
||||
load-bearing — AutoDowning without it never downs anything). Under `keep-oldest` the
|
||||
typed `KeepOldestOption { DownIfAlone = true }` installs the SBR instead; the
|
||||
`split-brain-resolver` block in akka.conf is inert under the default. Tests read the
|
||||
**effective config off a running ActorSystem** (not the input HOCON) — provider class,
|
||||
15 s window, case-insensitivity, and the window-exceeds-heartbeat-pause invariant.
|
||||
|
||||
**Live gate still open:** the 1-vs-1 oldest-crash pathology cannot be exercised on the
|
||||
current six-node docker-dev mesh; the crash-the-oldest drill is deferred to the
|
||||
per-cluster mesh work (Phase 6/7). Supporting live evidence exists from a 2-container
|
||||
kill test (`archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md`) and
|
||||
from ScadaBridge's identical fix. Cautionary note recorded in `docs/Redundancy.md`:
|
||||
the pre-existing `HardKillFailoverTests` stayed green under broken keep-oldest because it
|
||||
shut down the *transport* (ActorSystem alive) rather than killing the process — "a
|
||||
standing example of a green test over a fatal defect."
|
||||
|
||||
### 4.3 Redundancy primary election + ServiceLevel
|
||||
- **`RedundancyStateActor`** (an `admin`-role singleton) computes the fleet snapshot:
|
||||
`SelectDriverPrimary` = oldest **Up** member carrying the `driver` role
|
||||
(`Member.AgeOrdering`; `Leaving` excluded — a leaving node is handing its singletons
|
||||
over and must not be Primary). Debounced 250 ms, republished on a 10 s heartbeat over
|
||||
distributed pub/sub topic `redundancy-state` (DPS doesn't replay to late subscribers).
|
||||
- **`NodeRedundancyState`** carries `RedundancyRole { Primary, Secondary, Detached }` and
|
||||
`IsDriverPrimary` (renamed from `IsRoleLeaderForDriver` — "the name now describes what
|
||||
the value means rather than how it used to be computed").
|
||||
- **ServiceLevel** (`ServiceLevelCalculator`): health basis 240 (healthy) / 200 (stale) /
|
||||
100 (critically degraded) / 0 (down), **+10 if driver Primary**, so a healthy pair
|
||||
reads **250 / 240**. Published into the OPC UA `Server.ServiceLevel` variable via
|
||||
`SdkServiceLevelPublisher` (deferred publisher until the SDK server exists; first
|
||||
computed value always published so no node lingers at the SDK default 255).
|
||||
- **Client-visible failover is OPC UA non-transparent redundancy**: each node has its own
|
||||
`ApplicationUri`; clients read `Server.ServerArray` (self + peers, from
|
||||
`PeerApplicationUris`) and pick the endpoint with the highest ServiceLevel. The shipped
|
||||
client (`OpcUaClientService`) does this: failover URL list, KeepAlive-failure trigger,
|
||||
single-flight failover guard, data + alarm subscription replay after reconnect.
|
||||
**Consumer note:** because Primary = oldest (not lowest address), after any node
|
||||
restart a ServiceLevel-selecting client may legitimately prefer a different node than
|
||||
before — that's correct behavior, not a bug.
|
||||
|
||||
**Primary gating of the data plane (`PrimaryGatePolicy`)** — the S4 default-deny fix:
|
||||
|
||||
```csharp
|
||||
localRole switch {
|
||||
Primary => true,
|
||||
Secondary or Detached => false,
|
||||
_ /* unknown */ => driverMemberCount <= 1, // multi-driver: DENY until proven
|
||||
};
|
||||
```
|
||||
|
||||
During the boot window (role not yet known) a multi-driver node **rejects device writes**
|
||||
with `not primary (role unknown)` (denial meter `reason=role-unknown`); a single-driver
|
||||
node stays default-allow. Gated surfaces: device writes, native-alarm acks, alarm-alert
|
||||
fan-out. A deliberately *different* policy governs alarm-history drain: unknown → drain
|
||||
anyway ("a duplicate row beats a silent gap"), keyed on whether the Primary shares this
|
||||
node's queue (pair-local). Live-proven on a 2-node docker rig (R2-04 T15): ServiceLevel
|
||||
250/240, boot-window write rejected, steady-state secondary rejected + reverted, primary
|
||||
write reaches the device.
|
||||
|
||||
Failover sequence when the primary dies: peer downed (auto-down) → survivor is oldest
|
||||
driver → `SelectDriverPrimary` flips → snapshot republished → gates open on the survivor →
|
||||
ServiceLevel 250 moves → OPC UA clients see the old endpoint die (KeepAlive) and re-select
|
||||
by ServiceLevel. Singletons migrate to the same node the election names, because both
|
||||
derive from member age.
|
||||
|
||||
### 4.4 Singletons and the per-cluster mesh gap
|
||||
Five control-plane singletons, all role-pinned to `admin`:
|
||||
|
||||
| Singleton | Actor | Purpose |
|
||||
|---|---|---|
|
||||
| `config-publish` | `ConfigPublishCoordinator` | Persists per-node deploy ACKs |
|
||||
| `admin-operations` | `AdminOperationsActor` | Operator ack/shelve; TagConfig deploy gate |
|
||||
| `audit-writer` | `AuditWriterActor` | Audit persistence |
|
||||
| `fleet-status` | `FleetStatusBroadcaster` | AdminUI fleet panel |
|
||||
| `redundancy-state` | `RedundancyStateActor` | Primary election + DPS publisher |
|
||||
|
||||
**KNOWN LIMITATION:** the Primary election is scoped **per Akka cluster, not per
|
||||
application Cluster**. On the one-mesh fleet, `SelectDriverPrimary` names exactly ONE
|
||||
Primary among *all* driver nodes (MAIN + SITE-A + SITE-B), while every gated resource is
|
||||
pair-local. The per-cluster mesh design
|
||||
(`OtOpcUa/docs/plans/2026-07-21-per-cluster-mesh-design.md`) fixes this by construction:
|
||||
one Akka mesh per application Cluster (two nodes max), central↔cluster joined by explicit
|
||||
transports (modeled on ScadaBridge's ClusterClient/gRPC/HTTP trio), so "am I the driver
|
||||
Primary?" and "the resource I'm gating" have the same scope. Status: Phases 0a (auto-down)
|
||||
and 0b (oldest-Up election) DONE and merged; Phases 1–7 (ClusterNode columns, comm actors,
|
||||
config fetch-and-cache, cut driver-side ConfigDb, gRPC stream contract, mesh partition,
|
||||
failover drill/live gate) not started — **but now sequenced by a program plan
|
||||
(`OtOpcUa/docs/plans/2026-07-22-per-cluster-mesh-program.md`, 2026-07-22), driven by the
|
||||
decision that OtOpcUa pairs will run on the SAME Windows VMs as the ScadaBridge nodes** —
|
||||
two independent, identically-postured 2-node clusters per site, and driver nodes off the
|
||||
ConfigDb entirely (sites have no SQL Server).
|
||||
|
||||
### 4.5 Surviving central-SQL loss — availability guarantees
|
||||
Two mechanisms make a driver pair independent of central SQL Server availability:
|
||||
|
||||
- **Address-space availability guarantee (#485/#486):** an artifact the node cannot read
|
||||
is *no answer*, never *an empty configuration*. Previously a transient ConfigDb error
|
||||
mid-deploy parsed zero bytes into an empty composition → the planner diffed it as a
|
||||
`PureRemove` of every node → served address space emptied, drivers stopped,
|
||||
subscriptions cleared, deploy still reported Applied. Now `OpcUaPublishActor` /
|
||||
`DriverHostActor` hold last-known-good address space, drivers, and subscriptions, and
|
||||
report the deploy **Failed** without advancing the revision so the retry lands.
|
||||
- **LocalDb Phase 1 boot-from-cache:** every driver node caches the chunked, SHA-256
|
||||
verified deployed-configuration artifact in a pair-local SQLite DB; a node restarting
|
||||
into a central-SQL outage boots from the last artifact it applied. With opt-in pair
|
||||
replication (`deployment_artifacts` + `deployment_pointer`, HLC last-writer-wins,
|
||||
fail-closed auth) either node holds a byte-identical copy even if it never applied that
|
||||
deploy itself. Scope: config artifact only — no live tags/alarms/historian data.
|
||||
|
||||
---
|
||||
|
||||
## 5. Dual-active: what it concretely means here
|
||||
|
||||
Under auto-down, a *genuine* partition (both nodes alive, link cut) ends with each side
|
||||
downing the other and running active until an operator restarts one side (the mutual
|
||||
downing quarantines the association — the sides will not self-merge). Accepted per the
|
||||
availability-first decision; concrete blast radius:
|
||||
|
||||
**ScadaBridge site pair:** both nodes' `SelfIsOldestUp` = true → both open the S&F
|
||||
delivery gate → **duplicate external deliveries** of the same replicated `sf_messages`
|
||||
rows for the partition's duration; both stamp `IsActive` heartbeats. LocalDb replication
|
||||
itself is partition-tolerant (HLC last-writer-wins reconverges on heal, after the
|
||||
operator restart). Central pair: both serve `/health/active` = 200 and run duplicate
|
||||
singleton sets against the same SQL Server — duplicated sweeps/purges (mostly idempotent
|
||||
upserts), and Traefik may alternate between two "active" backends.
|
||||
|
||||
**OtOpcUa pair:** both nodes elect themselves driver Primary → both publish
|
||||
**ServiceLevel 250** → clients on either side keep their session; both sides accept
|
||||
device writes → **two masters against the same field devices** for the duration. This is
|
||||
the classic OT dual-primary risk — mitigated by rarity (2 VMs, one LAN segment, real
|
||||
partitions ≈ switch failure) and by the alternative (keep-oldest) having been proven to
|
||||
turn a plain crash into a total outage.
|
||||
|
||||
Detection/ops: the survivor logs the downing decision; OtOpcUa emits
|
||||
`otopcua.redundancy.service_level_change` and `primary_gate_denied` meters; two
|
||||
simultaneous 250s / two 200-responses on `/health/active` are the alarm signal. Recovery
|
||||
is always "restart one side; it rejoins as youngest/standby."
|
||||
|
||||
---
|
||||
|
||||
## 6. Gaps vs the stated HA goal ("either node first, automatic failover")
|
||||
|
||||
### 6.1 The seed-node bootstrap constraint — CLOSED 2026-07-22 (differently per repo)
|
||||
**Automatic failover: achieved.** Either node crashing is survivable, unattended, in
|
||||
~27–30 s. The remaining gap — a non-first seed cold-starting alone loops on `InitJoin`
|
||||
forever (the "registered outage gap") — was **closed in both repos on 2026-07-22**, but
|
||||
execution overturned the original design and the two repos legitimately diverged:
|
||||
|
||||
**⚠️ The original design's safety claim was FALSE — proven independently in both repos.**
|
||||
The planned `SelfFormAfter` watchdog assumed "window expires mid-join-handshake → benign,
|
||||
Akka ignores `Join` once joined." Wrong: the watchdog sits *outside* Akka's join
|
||||
handshake and cannot distinguish *no seed answered* from *a seed answered and the join is
|
||||
in flight*. On a routine restart into a live peer, the join stalls briefly behind removal
|
||||
of the node's own stale incarnation; a `Join(self)` fired in that window **abandons the
|
||||
in-flight join and forms a second cluster at the same address — a permanent island**.
|
||||
ScadaBridge proved it with a written test (split still unhealed after 90 s); OtOpcUa hit
|
||||
it live when its manual-failover drill bounced a node (InitJoinAck received, no Welcome
|
||||
inside the window, fallback fired → second cluster).
|
||||
|
||||
**ScadaBridge's fix — self-first seed ordering (no runtime code).** Every node lists
|
||||
ITSELF as `seed-nodes[0]`, partner second. Akka's own `FirstSeedNodeProcess` then
|
||||
implements exactly the intended semantics *inside* the handshake (InitJoin the other
|
||||
seeds; self-join only if nobody answers) — race-free by construction. A `StartupValidator`
|
||||
rule hard-gates the ordering at boot. Live-proven behaviors: lone cold-start self-joins
|
||||
after ~5 s (`seed-node-timeout`); restart into a live peer rejoins (never islands);
|
||||
simultaneous reachable cold-start converges to ONE cluster (the old "two clusters that
|
||||
never merge" objection was disproved by test); a genuine boot partition dual-forms — the
|
||||
accepted auto-down trade. There is **no `SelfFormAfter` option in ScadaBridge** — the
|
||||
whole watchdog was rejected in code review.
|
||||
|
||||
**OtOpcUa's fix — converged on self-first ordering the same day.** OtOpcUa first shipped
|
||||
the watchdog and hit the race live (its manual-failover drill bounced a node; the fallback
|
||||
islanded it), patched it with a TCP reachability guard — and then **retired the whole
|
||||
watchdog hours later** (`lmxopcua master @ 3f24d4d6`, a breaking change):
|
||||
`ClusterBootstrapFallback` and `Cluster:SelfFormAfter` are **deleted**, docker-dev
|
||||
`central-2` now lists itself as `SeedNodes__0`, and a new `AkkaClusterOptionsValidator`
|
||||
enforces the ordering at boot. Two subtleties its rule handles that ScadaBridge's didn't
|
||||
need: the invariant is **conditional** (self must be `seed[0]` only IF self appears in
|
||||
the list at all — site nodes seed off central only and are deliberately not seeds), and
|
||||
identity compares on **`PublicHostname` + port** (the address Akka puts in `SelfAddress`),
|
||||
because matching the `0.0.0.0` bind address would leave the rule silently inert on the
|
||||
whole rig. The retirement was proven by a falsifiability control: the old peer-first
|
||||
ordering test failed at 11 s against the watchdog — demonstrating the watchdog could
|
||||
never usefully fire for the nodes that needed it.
|
||||
|
||||
**Result: both repos now share the identical mechanism** — self-first seed ordering,
|
||||
validator-enforced, no bootstrap runtime code. The mesh program's Phase 6 convergence
|
||||
item is already done; what remains there is only the per-pair seed topology itself.
|
||||
|
||||
**Ops action (outstanding):** the gitignored `ScadaBridge/deploy/wonder-app-vd03/`
|
||||
overlay must have its `SeedNodes` reordered self-first before its next deploy — the new
|
||||
validator rule is a deliberate hard boot gate.
|
||||
|
||||
### 6.2 Other open items
|
||||
- **OtOpcUa auto-down live gate** still open (needs a real 1-vs-1 rig; deferred to
|
||||
per-cluster mesh Phase 6/7). ScadaBridge's identical mechanism *is* drill-proven.
|
||||
- **OtOpcUa election scope** is cluster-wide until per-cluster mesh Phases 1–7 land
|
||||
(§4.4) — only relevant while multiple pairs share one mesh.
|
||||
- **OtOpcUa's 15 s downing window is a code constant**, not config — worth making
|
||||
tunable alongside any heartbeat-timeout reduction (ScadaBridge's already is:
|
||||
`Cluster:StableAfter`).
|
||||
- **ScadaBridge site pairs: stop/start together** for upgrades (no mixed-version LocalDb
|
||||
replication); ≥ 7-day-offline nodes risk tombstone resurrection.
|
||||
- **Stale doc**: `ScadaBridge/docs/deployment/topology-guide.md:101` still describes
|
||||
keep-oldest.
|
||||
|
||||
---
|
||||
|
||||
## 7. Quick reference — file map
|
||||
|
||||
| Concern | ScadaBridge | OtOpcUa |
|
||||
|---|---|---|
|
||||
| HOCON / downing emit | `Host/Actors/AkkaHostedService.cs:258–329` | `Cluster/ServiceCollectionExtensions.cs:119–146` + `Cluster/Resources/akka.conf` |
|
||||
| Options + validation | `ClusterInfrastructure/ClusterOptions.cs`, `ClusterOptionsValidator.cs` | `Cluster/AkkaClusterOptions.cs`, `RoleParser.cs` |
|
||||
| Strategy key | `Cluster:SplitBrainResolverStrategy` (default `auto-down`) | same key, same default; unknown value = startup failure |
|
||||
| Active/primary definition | `Communication/ClusterState/ActiveNodeEvaluator.cs` (oldest Up) | `ControlPlane/Redundancy/RedundancyStateActor.cs` (oldest Up `driver`) |
|
||||
| Data-plane gating | `InboundApiEndpointFilter` (503 STANDBY_NODE), S&F `SetDeliveryGate` | `Runtime/Drivers/PrimaryGatePolicy.cs` + `DriverHostActor` gates |
|
||||
| Singleton registration | `Host/Actors/SingletonRegistrar.cs` (7 central + 2 site) | `ControlPlane/ServiceCollectionExtensions.cs:44–94` (5, role `admin`) |
|
||||
| Client-visible failover | Traefik on `/health/active`; gRPC A/B flip | OPC UA ServiceLevel 250/240 + `Server.ServerArray`; client KeepAlive failover |
|
||||
| Self-down recovery | watchdog in `AkkaHostedService` → process exit → supervisor | `Host/ActorSystemTerminationWatchdog.cs` → same |
|
||||
| Decision records | `docs/plans/2026-07-21-auto-down-availability-decision.md`, `docs/requirements/Component-ClusterInfrastructure.md` | `docs/Redundancy.md` §"Split-brain / downing", `docs/plans/2026-07-21-per-cluster-mesh-design.md` |
|
||||
| Failover drills / tests | `docker/failover-drill.sh` (28 s/27 s), `SbrFailoverTests`, `FailoverTimingTests` | `SplitBrainResolverActivationTests`, `RedundancyPrimaryElectionTests`, R2-04 T13/T15 live gate |
|
||||
@@ -282,6 +282,52 @@ Mostly no — mapping the family's gRPC surfaces:
|
||||
The "big messages so we don't need side channels" hope is really an Artery feature (dedicated
|
||||
chunked large-message streams), not a QUIC-transport feature.
|
||||
|
||||
### 8a. The overlooked 1.5-era tool: Akka.Cluster.Chunking (added 2026-07-22)
|
||||
|
||||
One mechanism was missing from the original analysis — Petabridge's
|
||||
**[Akka.Cluster.Chunking](https://github.com/petabridge/Akka.Cluster.Chunking)** (Apache-2.0),
|
||||
which exists precisely to bridge the pre-Artery gap this report is about: sending large messages
|
||||
(5 MB+) over Akka.Remote **without raising `maximum-frame-size` and without head-of-line
|
||||
blocking**. It chunks at the application layer on top of `Akka.Delivery`'s reliable-delivery
|
||||
machinery (ordered, ack'd, resend-on-loss), so every chunk is an ordinary sub-frame-cap remoting
|
||||
message and heartbeats/system messages interleave between chunks — directly defusing the §2
|
||||
guardrail concern with zero transport work.
|
||||
|
||||
Usage is pleasantly transparent: `ChunkingManager.For(system)` then
|
||||
`chunker.DeliverChunked(message, recipient, sender)` (Akka.Hosting: `.AddChunkingManager()`);
|
||||
config via `akka.cluster.chunking.chunk-size` (default 64 kB), `outbound-queue-capacity`
|
||||
(default 20), `request-timeout` (default 5 s). Unlike raw `Akka.Delivery`, no per-flow
|
||||
`ProducerController`/`ConsumerController` pair has to be hand-wired — the manager runs the
|
||||
delivery channels per node pair.
|
||||
|
||||
The honest caveats for our context:
|
||||
|
||||
- **Maturity.** v0.4.0 (Aug 2025), and the README says plainly it is "currently an experiment and
|
||||
is available on an as-is basis." For an OT product that's a real adoption bar; budget for
|
||||
owning/forking it if Petabridge doesn't graduate it.
|
||||
- **Acceptance ≠ delivery.** `DeliverChunked`'s task completes when the message is *accepted* for
|
||||
delivery, not when the far side has it — callers needing end-to-end confirmation still build
|
||||
their own ack on top.
|
||||
- **Requires Akka.Cluster, in-cluster only.** It rides remoting between cluster members. It does
|
||||
not cross the ScadaBridge site↔central boundary — two separate Akka clusters bridged by
|
||||
ClusterClient/gRPC — which is exactly where the family's big payloads actually flow.
|
||||
**Nor does ClusterClient itself chunk** (the tunneled payload + receptionist envelope must fit
|
||||
one frame). It *could* be bolted on — ClusterClient replies carry real IActorRefs, so raw
|
||||
`Akka.Delivery` could then run point-to-point over a direct cross-cluster remoting association —
|
||||
but that reopens exactly the direct-remoting coupling (firewall surface, quarantine interplay)
|
||||
the seed-partitioned boundary exists to avoid; large cross-boundary payloads stay on the gRPC
|
||||
side-channels.
|
||||
- **Correctness tool, not throughput tool** — shallow queues and the reliable-delivery handshake
|
||||
make it slower than plain tells; it removes a ceiling, it doesn't add speed.
|
||||
|
||||
Re-mapping §8 with this in hand changes no conclusion: mxaccessgw/HistorianGateway peers aren't
|
||||
cluster members; LocalDb sync is independent of cluster state *by design*; the ScadaBridge
|
||||
bulk/artifact side channels are cross-cluster, out of reach. Where it **is** the right answer is
|
||||
any *future within-cluster* large-message need (e.g. a big payload between the members of one
|
||||
pair): reach for Akka.Cluster.Chunking (or raw `Akka.Delivery` with
|
||||
`ProducerController.Settings.ChunkLargeMessagesBytes`, the in-box primitive it wraps) instead of
|
||||
raising the frame cap — until 1.6's Artery large-message lanes make both workarounds unnecessary.
|
||||
|
||||
## 9. Options
|
||||
|
||||
**A. Don't build it; tune what we have; track 1.6 (recommended).** Best-practice fit:
|
||||
@@ -289,8 +335,9 @@ transport-layer plumbing is exactly the code an OT product wants to own least, a
|
||||
design explicitly targets the wishlist (large-message channels, quarantine-bypass control channel —
|
||||
the latter directly relevant to the 2-node availability-first failover posture). Concretely: bump
|
||||
1.5.62 → 1.5.70 (eight releases of remoting/streams fixes, including a cluster-wedging
|
||||
consistent-hash-router bug), and if specific pains exist now, raise `maximum-frame-size` and/or
|
||||
enable the new dot-netty mTLS. Subscribe to #7466; when real alphas ship, the pair-restart
|
||||
consistent-hash-router bug), and if specific pains exist now, prefer Akka.Cluster.Chunking /
|
||||
`Akka.Delivery` chunking (§8a) over raising `maximum-frame-size` for any within-cluster large
|
||||
message, and/or enable the new dot-netty mTLS. Subscribe to #7466; when real alphas ship, the pair-restart
|
||||
discipline makes adoption cheap.
|
||||
|
||||
**B. Build a classic-SPI QUIC transport now.** ~3–5 weeks to production quality per §3. Choose this
|
||||
@@ -306,7 +353,8 @@ all get exercised) without owning it in production. A reasonable middle path.
|
||||
**Recommendation: A**, with **C** if hands-on evidence is wanted — the honest summary is that for
|
||||
this workload, an MsQuic transport on Akka 1.5 buys security and handshake ergonomics that are
|
||||
mostly available from config today, while the features that would actually reduce the gRPC
|
||||
side-channels only arrive with the Artery pipeline rewrite, which upstream hasn't started coding.
|
||||
side-channels only arrive with the Artery pipeline rewrite — now in active development upstream
|
||||
(see the §1 status update) — with Akka.Cluster.Chunking (§8a) as the interim in-cluster answer.
|
||||
|
||||
## Sources
|
||||
|
||||
@@ -317,3 +365,5 @@ side-channels only arrive with the Artery pipeline rewrite, which upstream hasn'
|
||||
- [Akka.NET remoting transports documentation](https://getakka.net/articles/remoting/transports.html)
|
||||
- [Akka.NET PR #7851 — mutual TLS for DotNetty transport](https://github.com/akkadotnet/akka.net/pull/7851)
|
||||
- [QUIC support in .NET (System.Net.Quic)](https://learn.microsoft.com/en-us/dotnet/fundamentals/networking/quic/quic-overview)
|
||||
- [petabridge/Akka.Cluster.Chunking](https://github.com/petabridge/Akka.Cluster.Chunking) — v0.4.0, Apache-2.0, experimental
|
||||
- [Akka.NET reliable delivery (Akka.Delivery)](https://getakka.net/articles/actors/reliable-delivery.html)
|
||||
|
||||
@@ -17,6 +17,28 @@ reach the shared `ZB.MOM.WW.Health` library. Status legend: ⛔ gap · 🟡 part
|
||||
> until someone re-walks them. They also predate HistorianGateway, which is absent from every
|
||||
> table below.
|
||||
|
||||
> **⚠️ Gaps L1/L2 were closed the WRONG WAY, then reopened and closed properly at `0.3.0`
|
||||
> (2026-07-24).** Everything below about the active-node check describes the leader-based design and
|
||||
> is **superseded** — read [`spec/SPEC.md`](spec/SPEC.md) §2.3 for the current rule.
|
||||
>
|
||||
> L1 shipped `ActiveNodeHealthCheck` selecting by `ClusterState.Leader` / `RoleLeader`, and L2 backed
|
||||
> `AkkaActiveNodeGate` the same way. Leadership is **address**-ordered (host, then port) with no
|
||||
> relationship to time, while cluster-singleton placement is **age**-ordered. The two agree on a
|
||||
> freshly-formed cluster — so every unit test and happy-path rig passed — and diverge permanently
|
||||
> after any restart: the restarted node rejoins youngest but keeps its address, so if it holds the
|
||||
> lower address it becomes leader while the singletons stay on the other node. During a partition
|
||||
> both sides also compute themselves leader.
|
||||
>
|
||||
> Both apps adopted L1/L2 as written, both hit the defect on live rigs, and both **reverted to
|
||||
> private copies** (ScadaBridge `OldestNodeActiveHealthCheck`, OtOpcUa `ClusterPrimaryHealthCheck`).
|
||||
> The result was that the package's whole active-node surface had **zero consumers** — avoided, not
|
||||
> unused. `0.3.0` promoted those two copies into one `ClusterActiveNode` primitive, rebuilt the check
|
||||
> and the gate on it, and both apps migrated back and deleted their copies.
|
||||
>
|
||||
> **The lesson worth keeping:** a shared component can be adopted, green, and still wrong, when the
|
||||
> only fixture that distinguishes right from wrong is a cluster whose oldest member is not its
|
||||
> lowest-addressed one. The library now pins exactly that fixture.
|
||||
|
||||
## Divergence vs spec
|
||||
|
||||
### §1 Endpoint tiers
|
||||
|
||||
@@ -10,7 +10,7 @@ exist: OtOpcUa `Health/` (three-tier + probes), ScadaBridge `Health/` (inline pr
|
||||
|
||||
```
|
||||
ZB.MOM.WW.Health # core: tier convention, response writer, IActiveNodeGate, GrpcDependencyHealthCheck
|
||||
ZB.MOM.WW.Health.Akka # AkkaClusterHealthCheck, ActiveNodeHealthCheck, AkkaActiveNodeGate
|
||||
ZB.MOM.WW.Health.Akka # AkkaClusterHealthCheck, ClusterActiveNode, ActiveNodeHealthCheck, AkkaActiveNodeGate
|
||||
ZB.MOM.WW.Health.EntityFrameworkCore # DatabaseHealthCheck<TContext>
|
||||
```
|
||||
|
||||
@@ -79,7 +79,7 @@ public static class ZbHealthWriter
|
||||
public static Task WriteJsonAsync(HttpContext context, HealthReport report);
|
||||
}
|
||||
|
||||
/// Single-property seam: is this node the active/leader node?
|
||||
/// Single-property seam: is this node the active node (oldest Up member)?
|
||||
/// Attach to route groups via RequireActiveNode(). Implement with AkkaActiveNodeGate (Health.Akka)
|
||||
/// or a project-specific implementation for non-Akka nodes.
|
||||
public interface IActiveNodeGate
|
||||
@@ -162,49 +162,77 @@ public sealed class AkkaClusterStatusPolicy
|
||||
public static AkkaClusterStatusPolicy OtOpcUaCompat { get; }
|
||||
}
|
||||
|
||||
/// Checks whether this node is the designated leader / active node.
|
||||
/// Optional role parameter scopes the check to nodes carrying that role.
|
||||
/// Register to tag ZbHealthTags.Active.
|
||||
/// THE family-wide definition of "the active node": the OLDEST Up member, optionally scoped to a
|
||||
/// role — the member ClusterSingletonManager places singletons on. Never cluster leadership, which is
|
||||
/// address-ordered and diverges from singleton placement permanently after any restart (0.3.0).
|
||||
public static class ClusterActiveNode
|
||||
{
|
||||
/// The oldest Up member, optionally scoped to a role. Null when none is Up.
|
||||
public static Member? OldestUpMember(IEnumerable<Member> members, string? role = null);
|
||||
public static Member? OldestUpMember(Cluster cluster, string? role = null);
|
||||
|
||||
/// True when the local member is Up, carries the role (if given), and no eligible member is
|
||||
/// older. Identity compares UniqueAddress, so a node restarted on the same host:port is a
|
||||
/// different member during the overlap.
|
||||
public static bool SelfIsActive(Cluster cluster, string? role = null);
|
||||
|
||||
/// The first role in rolePreference that selfRoles carries, else null. Lets a fused node answer
|
||||
/// for the role its singletons are pinned to (e.g. ["admin", "driver"]).
|
||||
public static string? ResolveActiveRole(IEnumerable<string> selfRoles, IReadOnlyList<string> rolePreference);
|
||||
}
|
||||
|
||||
/// Configures which role's active node a node competes to be.
|
||||
public sealed class ActiveNodeHealthCheckOptions
|
||||
{
|
||||
/// Candidate roles in descending precedence. Empty (default) = unscoped: oldest Up member of the
|
||||
/// whole cluster.
|
||||
public IReadOnlyList<string> RolePreference { get; init; }
|
||||
|
||||
/// Status when RolePreference is non-empty and the node carries none of those roles — it owns no
|
||||
/// active work. Defaults to Unhealthy: the active tier exists so a load balancer can pick exactly
|
||||
/// one node, and "not applicable => Healthy" made the tier answer 200 on every node (lmxopcua#494).
|
||||
public HealthStatus NoActiveRoleStatus { get; init; }
|
||||
}
|
||||
|
||||
/// Active-tier probe: Healthy on the one node in charge, Unhealthy (503) on its standby.
|
||||
/// Register to tag ZbHealthTags.Active. Selects via ClusterActiveNode (oldest Up member).
|
||||
/// Result data carries selfAddress, activeRole (when scoped) and activeNode, so a standby reports
|
||||
/// WHO is active.
|
||||
/// <remarks>
|
||||
/// The ActorSystem is resolved lazily from the service provider. If the ActorSystem is not yet
|
||||
/// available (e.g. during startup before Akka is initialised), the check returns Degraded rather
|
||||
/// than throwing. This makes the check startup-safe.
|
||||
/// The ActorSystem is resolved lazily from the service provider. If the ActorSystem or cluster is not
|
||||
/// yet available (e.g. during startup before Akka is initialised), the check returns Degraded rather
|
||||
/// than throwing. This makes the check startup-safe. A reachable cluster with no Up member in scope
|
||||
/// is Unhealthy, not Degraded — a cluster that never forms must not sit at 200.
|
||||
/// </remarks>
|
||||
public sealed class ActiveNodeHealthCheck : IHealthCheck
|
||||
{
|
||||
/// Role-less constructor: Healthy = node is Up AND cluster leader (ScadaBridge ActiveNode pattern).
|
||||
/// Returns Degraded when ActorSystem/cluster is not yet ready.
|
||||
/// <param name="serviceProvider">
|
||||
/// The application service provider. ActorSystem is resolved lazily so the check is
|
||||
/// startup-safe: if no ActorSystem is registered yet the result is Degraded.
|
||||
/// </param>
|
||||
/// Unscoped: the active node is the oldest Up member of the whole cluster.
|
||||
public ActiveNodeHealthCheck(IServiceProvider serviceProvider);
|
||||
|
||||
/// Role-filtered constructor: Healthy = (node lacks the role) OR (node carries role AND is role-singleton leader).
|
||||
/// Degraded = node carries role but is not the role-singleton leader (OtOpcUa AdminRoleLeader pattern).
|
||||
/// Returns Degraded when ActorSystem/cluster is not yet ready.
|
||||
/// <param name="serviceProvider">
|
||||
/// The application service provider. ActorSystem is resolved lazily so the check is
|
||||
/// startup-safe: if no ActorSystem is registered yet the result is Degraded.
|
||||
/// </param>
|
||||
/// Role-scoped: the active node is the oldest Up member carrying `role`.
|
||||
public ActiveNodeHealthCheck(IServiceProvider serviceProvider, string role);
|
||||
|
||||
/// Configured, including a multi-role preference order.
|
||||
public ActiveNodeHealthCheck(IServiceProvider serviceProvider, ActiveNodeHealthCheckOptions options);
|
||||
|
||||
public Task<HealthCheckResult> CheckHealthAsync(
|
||||
HealthCheckContext context,
|
||||
CancellationToken cancellationToken = default);
|
||||
}
|
||||
|
||||
/// IActiveNodeGate implementation that computes IsActiveNode directly from the ActorSystem
|
||||
/// (SelfMember Up + cluster leader), null-guarded for startup safety.
|
||||
/// IActiveNodeGate implementation that computes IsActiveNode directly from the ActorSystem via
|
||||
/// ClusterActiveNode (oldest Up member, optional role scope), null-guarded for startup safety.
|
||||
/// Shares its rule with ActiveNodeHealthCheck, so a gated endpoint and /health/active agree.
|
||||
/// Register as a singleton. Does NOT resolve ActiveNodeHealthCheck from DI.
|
||||
public sealed class AkkaActiveNodeGate : IActiveNodeGate
|
||||
{
|
||||
/// <param name="serviceProvider">
|
||||
/// The application service provider. ActorSystem is resolved lazily; if not yet available
|
||||
/// IsActiveNode returns false (safe default during startup).
|
||||
/// The gate checks SelfMember.Status == Up AND cluster.State.Leader == self.Address directly.
|
||||
/// The gate delegates to ClusterActiveNode.SelfIsActive(cluster, role) directly.
|
||||
/// </param>
|
||||
public AkkaActiveNodeGate(IServiceProvider serviceProvider);
|
||||
/// <param name="role">Optional role scope; null (default) considers every Up member.</param>
|
||||
public AkkaActiveNodeGate(IServiceProvider serviceProvider, string? role = null);
|
||||
|
||||
public bool IsActiveNode { get; }
|
||||
}
|
||||
|
||||
@@ -10,8 +10,10 @@ three code-verified current-state docs (`../current-state/`). Goal is *path to s
|
||||
`/healthz`) with canonical tags `ready` / `active` / `live` and their semantics; the canonical
|
||||
JSON response shape; the `IActiveNodeGate` request-gating seam; a configurable
|
||||
`AkkaClusterHealthCheck` with two named policy presets that reconcile the diverging Akka logic in
|
||||
OtOpcUa and ScadaBridge; a role-filtered `ActiveNodeHealthCheck` that unifies OtOpcUa's
|
||||
`AdminRoleLeaderHealthCheck` and ScadaBridge's `ActiveNodeHealthCheck`; a generic
|
||||
OtOpcUa and ScadaBridge; the `ClusterActiveNode` oldest-`Up`-member rule and the
|
||||
`ActiveNodeHealthCheck` over it, which unify OtOpcUa's `ClusterPrimaryHealthCheck` and ScadaBridge's
|
||||
`OldestNodeActiveHealthCheck` — the two private replacements each app wrote after the original
|
||||
leader-based check proved wrong on a live rig; a generic
|
||||
`DatabaseHealthCheck<TContext>` that covers both apps' EF Core probe patterns; a
|
||||
`GrpcDependencyHealthCheck` for downstream gRPC reachability.
|
||||
|
||||
@@ -29,7 +31,7 @@ Three tiers, always served in this order, each filtered to a named tag:
|
||||
| Tier | Endpoint | Tag | Semantics | Healthy→ | Degraded→ | Unhealthy→ |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Ready | `/health/ready` | `ready` | Can this node serve its dependencies? Fails if a DB, gRPC dependency, or cluster membership check is unhealthy. Orchestrators use this to gate traffic. | 200 | 200 | 503 |
|
||||
| Active | `/health/active` | `active` | Is this the leader / active node? Fails (503) on a standby or role-member-but-not-leader node. Used to route write traffic or admin requests to exactly one node. | 200 | 200 | 503 |
|
||||
| Active | `/health/active` | `active` | Is this the active node — the **oldest `Up` member**, optionally within a role? Fails (503) on a standby, and on a node that owns no active work. Used to route write traffic or admin requests to exactly one node. | 200 | 200 | 503 |
|
||||
| Live | `/healthz` | `live` | Bare process liveness — is the process alive and not deadlocked? **No probes registered to this tag** (predicate `_ => false`). Always 200 as long as the process can handle HTTP. | 200 | 200 | 200 |
|
||||
|
||||
Notes:
|
||||
@@ -77,21 +79,45 @@ rather than Unhealthy. Registered to the `ready` tag.
|
||||
> `OtOpcUaCompat` column refer to states that collapse into Degraded via the member-scan result,
|
||||
> not to an explicit policy match.
|
||||
|
||||
### 2.3 Active / leader probe — `ActiveNodeHealthCheck`
|
||||
### 2.3 Active-node probe — `ActiveNodeHealthCheck`
|
||||
|
||||
Checks whether this node is the designated leader (active node). Accepts an optional Akka
|
||||
cluster role name that scopes the check to nodes carrying that role.
|
||||
Checks whether this node is the **active node**: the **oldest `Up` member**, optionally scoped to a
|
||||
role. Registered to the `active` tag. The rule itself is `ClusterActiveNode`, which is also what the
|
||||
`IActiveNodeGate` implementation and each app's own primary/standby logic call, so an endpoint gate
|
||||
and the probe an orchestrator routes by cannot disagree.
|
||||
|
||||
**Two behaviors unify the existing divergence:**
|
||||
**Oldest, never leader.** Cluster leadership (`ClusterState.Leader` / `RoleLeader`) is
|
||||
*address*-ordered — host, then port, with no relationship to time — while singleton placement is
|
||||
*age*-ordered. The two agree on a freshly-formed cluster and diverge permanently after any restart:
|
||||
the restarted node rejoins as the youngest but keeps its address, so if it holds the lower address it
|
||||
becomes leader while the singletons, and all the work they own, stay on the other node. A leader-based
|
||||
active tier therefore routes traffic to a node that is not hosting the work. During a partition both
|
||||
sides also compute themselves leader, so both answer 200.
|
||||
|
||||
| Mode | Role param | Origin | Healthy | Degraded | Unhealthy |
|
||||
|---|---|---|---|---|---|
|
||||
| Role-less | `null` | ScadaBridge `ActiveNodeHealthCheck` | Node is `Up` **and** cluster leader | — | Otherwise |
|
||||
| Role-filtered | e.g. `"admin"` | OtOpcUa `AdminRoleLeaderHealthCheck` | Node does **not** carry the role (not a participant — ignore it) **or** node carries the role and is the role-singleton leader | Carries the role but is **not** the role-singleton leader (role member, not leader) | — |
|
||||
| Mode | `RolePreference` | Healthy | Unhealthy |
|
||||
|---|---|---|---|
|
||||
| Unscoped | *(empty)* | Node is the oldest `Up` member of the cluster | Otherwise |
|
||||
| Role-scoped | e.g. `["admin", "driver"]` | Node is the oldest `Up` member carrying the first listed role it holds | It is not — **or** it holds none of the listed roles (see below) |
|
||||
|
||||
The role-filtered variant maps "not a member of the role" to Healthy (transparent — the probe
|
||||
is irrelevant for this node). This is the correct behavior for heterogeneous clusters where not
|
||||
every node carries every role. Registered to the `active` tag.
|
||||
`RolePreference` is an ordered list so a fused node answers for the role its singletons are pinned to
|
||||
(`admin` before `driver`), while a single-role node answers for its own role.
|
||||
|
||||
**A node holding none of the listed roles is `Unhealthy` by default** (`NoActiveRoleStatus`). This is
|
||||
deliberately *not* the "not applicable ⇒ Healthy" convention used by readiness probes: the active tier
|
||||
exists so a load balancer can pick exactly one node, and a node owning no active work must not be in
|
||||
that pool. The earlier role-filtered mode mapped that case to Healthy, which made the tier answer 200
|
||||
on **every** node and silently broke leader-pinning ([lmxopcua#494]).
|
||||
|
||||
Startup-safe: `Degraded` (a 200) while the `ActorSystem` or cluster is not yet available, so a booting
|
||||
node is not reported as a failed standby and pulled from rotation on the way up. But once the cluster
|
||||
is reachable and simply has no `Up` member in scope, the answer is `Unhealthy` — a cluster that never
|
||||
forms must not sit at 200 forever.
|
||||
|
||||
Results carry `selfAddress`, `activeRole` (when scoped) and `activeNode` in the per-entry `data`
|
||||
object, so a standby reports **who** is active and a dashboard can render a pair from either node's
|
||||
payload alone.
|
||||
|
||||
[lmxopcua#494]: https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/494
|
||||
|
||||
### 2.4 gRPC dependency probe — `GrpcDependencyHealthCheck`
|
||||
|
||||
@@ -151,9 +177,11 @@ plug it into `MapHealthChecks` options and also call it from custom endpoints.
|
||||
`IActiveNodeGate` is a single-property interface (`bool IsActiveNode { get; }`) that expresses
|
||||
whether the current node should accept write / active-role requests. The default implementation,
|
||||
`AkkaActiveNodeGate`, reads cluster state **directly**: `IsActiveNode` returns `true` iff the
|
||||
`ActorSystem` is available, `SelfMember.Status == Up`, and the node is the cluster leader. It is
|
||||
null-guarded and returns `false` when the `ActorSystem` is not yet ready (safe default during
|
||||
startup). It does **not** resolve `ActiveNodeHealthCheck` from DI. A `RequireActiveNode()` extension on
|
||||
`ActorSystem` is available and the node is the **oldest `Up` member**, optionally within a role — the
|
||||
same `ClusterActiveNode` rule §2.3 describes, so a gated endpoint and the `/health/active` probe an
|
||||
orchestrator routes by cannot disagree. It is null-guarded and returns `false` when the `ActorSystem`
|
||||
or cluster is not yet ready (safe default during startup, matching the standby case). It does **not**
|
||||
resolve `ActiveNodeHealthCheck` from DI. A `RequireActiveNode()` extension on
|
||||
`IEndpointConventionBuilder` attaches a policy that short-circuits with `503 Service Unavailable`
|
||||
on standby nodes.
|
||||
|
||||
@@ -180,6 +208,13 @@ predicates and response writer.
|
||||
|
||||
## 6. Migration notes
|
||||
|
||||
> **Historical.** This table records the pre-adoption state (2026-06) and is kept for provenance.
|
||||
> Adoption is complete. Two rows carry guidance that was later proven wrong: "replace with shared
|
||||
> `ActiveNodeHealthCheck(role: "admin")`" and "replace inline `ActiveNodeGate` with
|
||||
> `AkkaActiveNodeGate`" both pointed at the leader-based selection. Both apps followed it, both hit
|
||||
> the defect on a live rig, and both reverted to private copies until 0.3.0 rebuilt the shared check
|
||||
> on oldest-`Up`-member semantics (§2.3) — which is what they now use.
|
||||
|
||||
| Project | Current state | Gap | What normalizes |
|
||||
|---|---|---|---|
|
||||
| **OtOpcUa** | All three tiers present (`/health/ready`, `/health/active`, `/healthz`); `DatabaseHealthCheck`, `AkkaClusterHealthCheck`, `AdminRoleLeaderHealthCheck` inline. | Inline probes diverge from the shared policy model; no `IActiveNodeGate`. | Replace inline `AkkaClusterHealthCheck` with shared + `OtOpcUaCompat` preset; replace `AdminRoleLeaderHealthCheck` with shared `ActiveNodeHealthCheck(role: "admin")`; replace inline `DatabaseHealthCheck` with shared generic; call `app.MapZbHealth()`. |
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
# InitJoin Self-Form Fallback (`Cluster:SelfFormAfter`) + Manual Failover Control — Shared Design & Index
|
||||
|
||||
> **For Claude:** This is the family-level DESIGN + INDEX document. The executable per-repo plans live in the repos (split 2026-07-22 — the OtOpcUa plan is expected to grow and has a reserved-additions section):
|
||||
>
|
||||
> - **ScadaBridge:** `~/Desktop/ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md` (10 tasks: fallback 1–7, manual-failover UI 8–9, optional site-pair failover 10)
|
||||
> - **OtOpcUa:** `~/Desktop/OtOpcUa/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md` (7 tasks + "Reserved for additions" section)
|
||||
> - **OtOpcUa mesh-alignment PROGRAM (added 2026-07-22):** `~/Desktop/OtOpcUa/docs/plans/2026-07-22-per-cluster-mesh-program.md` — Phases 1–7 moving OtOpcUa to ScadaBridge's mesh shape (one 2-node Akka mesh per application Cluster, ClusterClient/gRPC/fetch transports, driver nodes off the ConfigDb), driven by the co-location decision: OtOpcUa pairs run on the SAME Windows VMs as the ScadaBridge nodes. The selfform plan above is its prerequisite. Design: `OtOpcUa/docs/plans/2026-07-21-per-cluster-mesh-design.md`.
|
||||
>
|
||||
> Execute each with superpowers-extended-cc:executing-plans **in that repo**. The two plans are fully independent and can run in parallel. Only Task C1 below executes from this file.
|
||||
|
||||
**Goal:** (1) Either node of a 2-node pair can cold-start alone and become operational, unattended — closing the seed-node bootstrap gap in both ScadaBridge and OtOpcUa via a configurable, fast (default 10 s) InitJoin timeout that falls back to self-forming a cluster. (2) An admin-only "Trigger failover" control on each app's cluster-status page for operator-initiated, graceful role swaps.
|
||||
|
||||
> ## ⚠️ EXECUTION OUTCOME (2026-07-22) — design partially overturned
|
||||
>
|
||||
> Both repo plans executed. **The behavior-spec row "Window expires while a join is mid-handshake → benign race: Akka ignores `Join` once joined" is FALSE** — the watchdog sits outside Akka's join handshake, and a `Join(self)` fired while a join is in flight (routine restart into a live peer, join stalled behind stale-incarnation removal) abandons the join and forms a **permanent second cluster at the same address**. Proven independently by a ScadaBridge test (split unhealed after 90 s) and by an OtOpcUa live island during the manual-failover drill.
|
||||
>
|
||||
> - **ScadaBridge** rejected the watchdog entirely and shipped **self-first seed ordering** (each node lists itself as `seed-nodes[0]`; Akka's `FirstSeedNodeProcess` provides the intended semantics race-free inside the handshake; `StartupValidator` hard-gates the ordering). No `SelfFormAfter` option exists there. See its plan's "Part 1 as executed".
|
||||
> - **OtOpcUa** first kept `SelfFormAfter` behind a TCP reachability guard, then **converged the same day** (`lmxopcua @ 3f24d4d6`): `ClusterBootstrapFallback` + `SelfFormAfter` deleted, self-first ordering shipped with a conditional `AkkaClusterOptionsValidator` (self must be `seed[0]` only IF self is a seed — site nodes exempt; identity on `PublicHostname`+port, not the 0.0.0.0 bind).
|
||||
> - **Both repos now share the identical mechanism.** Everything merged + pushed 2026-07-22, including ScadaBridge Task 10 (site-pair failover relayed from the central UI).
|
||||
> - The design sections below are retained as the decision record — read them through this banner. `akka_failover.md` §6.1 carries the corrected family-level account.
|
||||
|
||||
**Decision provenance:** `scadaproj/akka_failover.md` §6.1 (decided 2026-07-22); design priority = availability over partition safety (memory `ha-availability-over-partition-safety`). Nodes share a datacenter/LAN → a live peer answers InitJoin in milliseconds, so 10 s is generous; operators tune per site.
|
||||
|
||||
---
|
||||
|
||||
## Design
|
||||
|
||||
### The defect being fixed
|
||||
|
||||
Akka.NET only allows the **first-listed seed** to self-join and form a *new* cluster. Every other node loops on `InitJoin` forever until some existing member answers. Both repos document this as the "registered outage gap": a non-first seed cold-starting while the first seed is down never becomes Up (recovery today = manual env-var override or reviving the first seed). ScadaBridge's `ClusterOptions.SeedNodes` doc comment even claims "either can start first and form the cluster" — which the deployed configs do **not** deliver (both nodes list node-a first). The plans make the claim true and fix the comment.
|
||||
|
||||
### Behavior specification
|
||||
|
||||
| Scenario | Behavior with fallback |
|
||||
|---|---|
|
||||
| Peer alive (any boot order) | Normal seed join within ms — fallback never fires (membership arrives before the window) |
|
||||
| Lone cold-start, node IS in its own seed list | After `SelfFormAfter` (default 10 s): warn log + `Cluster.Join(SelfAddress)` → Up alone, singletons start (`min-nr-of-members=1`) |
|
||||
| Lone cold-start, node NOT in its own seed list | Fallback inert (info log at arm time); node waits for a seed — self-forming would island it. **This is the OtOpcUa site-node topology** (seeded only by `central-1`). |
|
||||
| Peer boots after survivor self-formed | Peer's `InitJoin` reaches the self-formed node (it's in the peer's seed list) → joins as youngest/standby. No island. |
|
||||
| Both cold-start simultaneously, mutually unreachable (boot-time partition) | Both self-form → two islands (same dual-active class auto-down already accepts; same recovery: restart one side). Mitigable with a staggered service start. |
|
||||
| `SelfFormAfter` = `null` or `<= 0` | Disabled — today's wait-forever behavior |
|
||||
| Window expires while a join is mid-handshake | Benign race: Akka ignores `Join` once the node has already joined a cluster |
|
||||
|
||||
### Config surface (same key both apps — family convention, like `SplitBrainResolverStrategy`)
|
||||
|
||||
- **ScadaBridge:** `ScadaBridge:Cluster:SelfFormAfter` → `ClusterOptions.SelfFormAfter` (`TimeSpan?`, default `00:00:10`), validated non-negative-or-null by `ClusterOptionsValidator`.
|
||||
- **OtOpcUa:** `Cluster:SelfFormAfter` → `AkkaClusterOptions.SelfFormAfter` (`TimeSpan?`, default `00:00:10`); disable-on-`<=0` semantics live in the fallback itself.
|
||||
- NOT a code constant (deliberate — see the `DowningStableAfter` lesson, `akka_failover.md` §6.2).
|
||||
|
||||
### Mechanism (both apps, deliberately duplicated like the termination watchdogs)
|
||||
|
||||
`ClusterBootstrapFallback.Arm(system, options, logger)`, called at ActorSystem creation (ScadaBridge: inside `AkkaHostedService.GetOrCreateActorSystem`; OtOpcUa: an Akka.Hosting `AddStartup` task inside `WithOtOpcUaClusterBootstrap`, so production and test hosts arm identically):
|
||||
|
||||
1. `SelfFormAfter` null/`≤0` → log info, return (disabled).
|
||||
2. Self address not in own `SeedNodes` (parsed via `Address.Parse`, exact equality) → log info, return (island guard).
|
||||
3. `RegisterOnMemberUp` completes a TCS; race it against `Task.Delay(window)`.
|
||||
4. On window expiry (and system not terminated): prominent warn log + `Cluster.Join(cluster.SelfAddress)`.
|
||||
|
||||
Races are benign — Akka ignores `Join` once joined this incarnation.
|
||||
|
||||
### Manual failover control (Part D design)
|
||||
|
||||
**Mechanism — graceful `Cluster.Leave`, never `Down`.** "Trigger failover" = issue `Cluster.Leave(<current active/primary member's address>)` from whichever node serves the UI (any member may initiate a leave for any member). The leaving node walks the graceful path: singleton hand-over via the cluster-leave phases (ScadaBridge additionally drains each singleton 10 s), its `CoordinatedShutdown` runs, the existing termination watchdog exits the process, the service supervisor restarts it, and it rejoins as the **youngest** member — a permanent role swap under the oldest-Up election, with no downing window and no dead interval beyond hand-over latency.
|
||||
|
||||
Shared rules:
|
||||
- **Admin-only:** button renders inside the app's admin-policy `AuthorizeView` (ScadaBridge `AuthorizationPolicies.RequireAdmin` on `/monitoring/health`; OtOpcUa the mutating-pages policy from `AdminUiPolicies` on `/clusters/{id}/redundancy`).
|
||||
- **Peer guard:** disabled (with tooltip) when fewer than 2 Up members carry the relevant role.
|
||||
- **Confirmation dialog** stating exactly what happens.
|
||||
- **Audited** through each app's existing audit seam before the Leave (`cluster.manual-failover`; ScadaBridge uses its CENTRAL audit writer, not the shared seam — dual-seam gotcha).
|
||||
- **UX note (ScadaBridge):** Traefik routes the UI to the *active* node, so triggering central failover drops the Blazor circuit; the page reconnects against the new active. The dialog warns about this.
|
||||
- **Scope note (OtOpcUa):** until the per-cluster mesh lands, the driver-Primary election is mesh-wide — the button acts on THE Primary of the whole Akka cluster; the panel says so (mirrors the KNOWN LIMITATION in `docs/Redundancy.md`).
|
||||
- **Election parity:** each service's target query is intentionally identical to the app's active/primary election (`ActiveNodeEvaluator.SelfIsOldestUp` / `RedundancyStateActor.SelectDriverPrimary`), pinned by a parity test — the node acted on must be exactly the one hosting the singletons.
|
||||
- **Self-form interplay:** none — the restarted node rejoins via its peer (alive by definition here), so `SelfFormAfter` never fires on this path.
|
||||
|
||||
### Multi-node testing assessment (getakka.net/articles/testing/multi-node-testing.html)
|
||||
|
||||
**Verdict: usable in principle, deliberately NOT adopted.** The MultiNode TestKit (`Akka.MultiNode.TestAdapter` + `Akka.Remote.TestKit` + `Akka.Cluster.TestKit`) runs each `RoleName` as a separate process under `dotnet test`, with `TestConductor` able to `Blackhole` links and `Exit` nodes, synchronized by `EnterBarrier`. Reasons not to use it here:
|
||||
|
||||
1. **Everything this feature needs is testable in-process** with real ActorSystems built from production bootstrap code (ScadaBridge `TwoNodeClusterFixture`; OtOpcUa `SplitBrainResolverActivationTests` pattern). Same fidelity, no new runner.
|
||||
2. **Heavy project-level cost:** dedicated test project, parallelization disabled assembly-wide, single-public-constructor constraints, MNTR log plumbing.
|
||||
3. **OtOpcUa xunit constraint:** the Akka TestKit family is xunit-v2-only; OtOpcUa's integration tree is xunit.v3.
|
||||
4. **The one MNTR-only scenario** (true boot-time partition → dual self-form) validates an already-documented *accepted* risk; the family's preferred proof is the docker rig drill (`failover-drill.sh` precedent, or `docker network disconnect`).
|
||||
|
||||
**Follow-up:** revisit alongside OtOpcUa per-cluster-mesh Phase 7 (failover-drill phase) if a scripted dual-island drill is ever wanted.
|
||||
|
||||
### Out of scope (recorded follow-ups)
|
||||
|
||||
- Making OtOpcUa's 15 s `DowningStableAfter` configurable — now listed in the OtOpcUa plan's "Reserved for additions".
|
||||
- Failure-detector tuning knobs for faster failover (both apps) — see `akka_failover.md` §2 tuning table; OtOpcUa side listed in its reserved-additions section.
|
||||
- Staggered Windows service start delays (pure ops; document only).
|
||||
- `deploy/wonder-app-vd03/` gitignored production overlay (ops applies `SelfFormAfter` on next deploy).
|
||||
- ScadaBridge site-pair failover from the central UI — optional Task 10 in the ScadaBridge plan (new central→site message contract; confirm with the user first).
|
||||
|
||||
---
|
||||
|
||||
## Task C1: Family docs flip (execute AFTER both repo plans complete and merge)
|
||||
|
||||
**Classification:** trivial
|
||||
**Estimated implement time:** ~2 min
|
||||
|
||||
**Files:**
|
||||
- Modify: `~/Desktop/scadaproj/akka_failover.md` §6.1 — "Status: decided, not yet implemented" → implemented (merge commit refs from both repos); leave the §6.2 `DowningStableAfter` line unless the OtOpcUa reserved-addition shipped.
|
||||
- Modify: `~/Desktop/scadaproj/CLAUDE.md` — brief addition to the ScadaBridge and OtOpcUa index rows.
|
||||
- Update memory `ha-availability-over-partition-safety.md` — mark the decision implemented.
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"planPath": "docs/plans/2026-07-22-initjoin-selfform-fallback.md",
|
||||
"note": "SPLIT 2026-07-22: executable tasks moved to the per-repo plans — ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md (10 tasks) and OtOpcUa/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md (7 tasks + reserved additions). Only C1 executes from this file, after BOTH repo plans complete and merge.",
|
||||
"tasks": [
|
||||
{"id": 0, "subject": "Task C1: Flip akka_failover.md §6.1 status + CLAUDE.md index + memory (after both repo plans merge)", "status": "pending"}
|
||||
],
|
||||
"lastUpdated": "2026-07-22T00:00:00Z"
|
||||
}
|
||||
@@ -0,0 +1,241 @@
|
||||
{
|
||||
"plan": "docs/plans/2026-07-22-overview-dashboard-impl-plan.md",
|
||||
"tasks": [
|
||||
{
|
||||
"id": "1",
|
||||
"subject": "0.1 Health writer: optional per-entry `data` object",
|
||||
"status": "completed",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": "2",
|
||||
"subject": "0.2 AkkaClusterHealthCheck: populate cluster-view data",
|
||||
"status": "completed",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": "3",
|
||||
"subject": "0.3 Health 0.2.0: version, test, pack, publish, verify",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"1",
|
||||
"2"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "4",
|
||||
"subject": "1.1 ScadaBridge: bump Health packages to 0.2.0",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"3"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "5",
|
||||
"subject": "1.2 ScadaBridge: site health checks (akka-cluster, localdb, active-node)",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"4"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "6",
|
||||
"subject": "1.3 ScadaBridge: map MapZbHealth on the site pipeline",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"5"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "7",
|
||||
"subject": "1.4 ScadaBridge: site health tests",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"6"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "8",
|
||||
"subject": "2.1 OtOpcUa: verify `akka` check is the shared AkkaClusterHealthCheck",
|
||||
"status": "completed",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": "9",
|
||||
"subject": "2.2 OtOpcUa: bump Health to 0.2.0",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"3",
|
||||
"8"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "10",
|
||||
"subject": "2.3 Alignment bumps: mxgw + HistorianGateway Health 0.2.0",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"3"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "11",
|
||||
"subject": "3.1 Overview: scaffold project",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"3"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "12",
|
||||
"subject": "3.2 Overview: registry options + validation",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"11"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "13",
|
||||
"subject": "3.3 Overview: health client + status model",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"11"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "14",
|
||||
"subject": "3.4 Overview: poller + snapshot store",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"12",
|
||||
"13"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "15",
|
||||
"subject": "3.5 Overview: leader aggregation",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"13"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "16",
|
||||
"subject": "3.6 Overview: UI (ThemeShell, Overview page, InstanceCard)",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"14",
|
||||
"15"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "17",
|
||||
"subject": "3.7 Overview: self-observability",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"14"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "18",
|
||||
"subject": "3.8 Overview: tests",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"16",
|
||||
"17"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "19",
|
||||
"subject": "3.9 Overview: config + docker",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"17"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "20",
|
||||
"subject": "4 Live acceptance (design \u00a79)",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
"7",
|
||||
"9",
|
||||
"18",
|
||||
"19"
|
||||
]
|
||||
}
|
||||
],
|
||||
"notes": {
|
||||
"deferred": [
|
||||
"Task 2.2 rig check (admin node /health/ready shows data.leader) \u2014 live docker-dev, folded into Phase 4.",
|
||||
"Phase 1 DoD rig curls (site :8084 ready/active split) \u2014 live docker rig, folded into Phase 4."
|
||||
],
|
||||
"branches": {
|
||||
"scadaproj": "feat/health-0.2.0-cluster-data (80668a0) -> feat/overview-dashboard, stacked",
|
||||
"ScadaBridge": "feat/site-node-health (commit 47850c0f)",
|
||||
"OtOpcUa": "feat/health-0.2.0-bump (commit 2e515c34, made in an isolated git worktree \u2014 feat/mesh-phase6 working tree untouched)",
|
||||
"HistorianGateway": "chore/health-0.2.0-bump (commit 51f0c2b)",
|
||||
"MxAccessGateway": "chore/health-0.2.0-bump (commit 2d54ace)"
|
||||
},
|
||||
"front_loaded": [
|
||||
"Validator / health-client / status-derivation tests from task 3.8 were written with tasks 3.2-3.3 so each batch is verified; 3.8 covers the remainder (poller, leader, store, bUnit, boot).",
|
||||
"Poller / snapshot-store / leader-resolver / bUnit InstanceCard + Overview page tests were written with tasks 3.4-3.6; task 3.8 now covers only the WebApplicationFactory boot tests."
|
||||
],
|
||||
"deviations": [
|
||||
"Task 3.6 said 'no antiforgery'. AddRazorComponents stamps antiforgery metadata on every component endpoint unconditionally and the endpoint middleware hard-fails without a matching middleware, so every page returned 500. Resolved with app.UseAntiforgery() (inert here, no forms) rather than MapRazorComponents(...).DisableAntiforgery(), which would silently expose the first form anyone adds. No authentication was added.",
|
||||
"Bootstrap IS vendored (copied from HistorianGateway, v5.3.3). Not optional: ThemeShell and TechButton are built on Bootstrap utility classes, so the shared kit does not render without it.",
|
||||
{
|
||||
"task": "3.7",
|
||||
"what": "Metric names follow components/observability/spec/METRIC-CONVENTIONS.md (dot-separated overview.instance.status / overview.poll.duration, unit s) instead of the plan's literal zb_overview_instance_status.",
|
||||
"why": "That spec is marked Standardized and authoritative for the family; the Prometheus exporter renders them as overview_instance_status and overview_poll_duration_seconds. Gauge labels are application/node/group \u2014 'node', not 'instance', because Prometheus stamps its own instance label on every scraped series and a colliding metric label is silently renamed exported_instance."
|
||||
},
|
||||
{
|
||||
"task": "3.9",
|
||||
"what": "Three registries, not one: appsettings.json ships an EMPTY Applications list, appsettings.Development.json a host-reachable subset (4 instances), appsettings.Docker.json the full 16-instance fleet.",
|
||||
"why": "The dev rigs publish only some node ports to the host. OtOpcUa publishes none per-node (only Traefik :9200, load-balanced across the central pair, so it cannot identify a node) and ScadaBridge's site nodes keep :8084 container-internal. A host-run dashboard can genuinely only see part of the fleet; the containerised one joins the rig networks and sees all of it. Empty Applications in appsettings.json keeps a registry-less production deploy failing fast at ConfigPreflight instead of booting into an empty dashboard."
|
||||
},
|
||||
{
|
||||
"task": "3.9",
|
||||
"what": "appsettings.json declares real Serilog sinks and MinimumLevel overrides rather than the family's empty \"Serilog\": {}.",
|
||||
"why": "AddZbSerilog drives sinks entirely from configuration and adds none of its own, so an empty section produces an app that logs NOWHERE \u2014 verified against the running HistorianGateway container, whose log stream is completely empty for exactly this reason (OtOpcUa, which populates the section, logs normally). The HttpClient override is needed too: 16 instances on a 10s cadence emit several Information lines a second of pure probe noise."
|
||||
},
|
||||
{
|
||||
"task": "3.9",
|
||||
"what": "Dockerfile clears ASPNETCORE_HTTP_PORTS as well as ASPNETCORE_URLS.",
|
||||
"why": "On .NET 8+ the aspnet base image declares its binding via ASPNETCORE_HTTP_PORTS=8080, not ASPNETCORE_URLS. Clearing only URLS leaves Kestrel logging 'Overriding address(es) http://*:8080' on every boot \u2014 the URLs-override trap arriving by default rather than by mistake."
|
||||
},
|
||||
{
|
||||
"task": "3.9",
|
||||
"what": "No container healthcheck.",
|
||||
"why": "aspnet:10.0 ships neither curl nor wget, and adding a --healthcheck mode to the app would put a second entry point into production code purely to satisfy Compose. The plan explicitly allows skipping."
|
||||
}
|
||||
],
|
||||
"phase4_findings": [
|
||||
"Rig ports verified by probing, not by reading compose (they disagree): OtOpcUa central-1/2 serve health on :9000 but site-* on :8080 (their explicit Kestrel listeners for LocalDb sync re-bind the primary HTTP port); ScadaBridge central on :5000, sites on :8084.",
|
||||
"ScadaBridge site nodes currently answer :8084/health/ready with 404 \u2014 MapZbHealth on sites is on the feat/site-node-health branch and the rig runs main. Phase 4 needs the rigs redeployed from the 0.2.0 branches.",
|
||||
"No leader chip renders on any group yet: the rigs run pre-0.2.0 builds whose akka check carries no data object. This is the designed graceful degradation, and it confirms Phase 4 must redeploy before checking design 9 item 2.",
|
||||
"SUSPECTED RIG SPLIT-BRAIN: scadabridge-central-a AND central-b both return /health/active 200 with 'oldest Up member (singleton host)'. Two nodes cannot both be the oldest of one cluster. Verify during Phase 4 after redeploy \u2014 this is exactly the condition the dashboard exists to surface.",
|
||||
"PRODUCT FIX during acceptance: Program.cs now calls builder.WebHost.UseStaticWebAssets() unconditionally. The host only calls it when the environment is literally 'Development', so running the build output under any other name (checking a staging registry locally) left the Theme RCL's _content assets 404 and the page rendered as a blank white screen with its markup fully present. No-op once published, so the container was never affected.",
|
||||
"RIG DEFECT (pre-existing, both products): self-first Akka seed lists mean a pair booted cold self-forms two 1-member clusters that never join. Observed fleet-wide on ScadaBridge after a --force-recreate: every node named ITSELF leader and every node answered /health/active 200. Restarting ONE node of a pair makes it join the survivor and the pair converges to one Leader + one Active + one Standby. This is the user's own open backlog item (docs/plans/2026-07-22-initjoin-selfform-fallback.md, unexecuted). The dashboard surfaced it fleet-wide in one screen, which is what it is for.",
|
||||
"Leader and Active legitimately diverge after a restart: Akka leader is lowest-address, ScadaBridge's active-node check is oldest Up member. site-c-a shows LEADER + STANDBY while site-c-b shows ACTIVE. Same divergence CLAUDE.md documents for OtOpcUa's Primary vs RoleLeader.",
|
||||
"OUTSTANDING: the OtOpcUa docker-dev rig still runs a pre-0.2.0 build. Redeploying it needs a decision because the repo is checked out on feat/mesh-phase6, which carries 4 commits and an untracked Phase 7 plan beyond the base of feat/health-0.2.0-bump.",
|
||||
"PRODUCT BUG found by the live rig: LeaderResolver and InstanceCard.ClusterDataLine both keyed the cluster view on the check NAME 'akka-cluster'. ScadaBridge registers the shared AkkaClusterHealthCheck under that name; OtOpcUa registers the SAME check as 'akka'. The result was a silent miss \u2014 OtOpcUa groups rendered with no leader chip and no evidence line, which is indistinguishable from a group that simply has not reported yet. Both now locate the entry by the DATA KEY ('leader'), which is the part of the contract the shared check owns. Regression tests cover akka-cluster / akka / an arbitrary future name, plus an entry that publishes data WITHOUT a leader (ScadaBridge's localdb check) so scanning cannot pick the wrong entry. Unit tests and review had both missed it because every fixture used ScadaBridge's name.",
|
||||
"OTOPCUA CONTRACT GAP (not a dashboard bug, worth filing): /health/active does not answer 'am I the active node'. On the standby admin (central-2) the admin-leader check returns Degraded ('Role admin member but not leader'), and ASP.NET maps Degraded to 200 \u2014 so a standby looks Active. On driver-only nodes it returns Healthy with the description 'Active for role admin (or not a role member)', i.e. it passes vacuously for every node that is not an admin at all. Meanwhile CLAUDE.md documents the real election as the oldest Up driver member per Cluster (IsDriverPrimary, ServiceLevel 250 vs 240), which this tier never consults. For the tier to mean what MapZbHealth's status mapping implies, a non-active node must return Unhealthy (503) \u2014 which is exactly what ScadaBridge's SitePairActiveNodeHealthCheck does, and why ScadaBridge reads correctly. FILED as lmxopcua#494 (https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/494), with the traefik-dynamic.yml LB-healthcheck impact flagged: docker-dev uses /health/active as its load-balancer probe, so making a standby return 503 would drop it from the UI pool.",
|
||||
"OtOpcUa rig rebuilt from a new branch feat/mesh-phase6-health-0.2.0 = feat/mesh-phase6 + a cherry-pick of the 3-line Health 0.2.0 bump (2e515c34). Checking out feat/health-0.2.0-bump directly would have regressed the rig past four commits of in-progress Phase 6/7 work. All three meshes formed cleanly (2 members, 0 unreachable, both members agreeing on one leader) \u2014 OtOpcUa's per-pair self-first seeding did NOT hit the cold-boot self-form race that ScadaBridge did.",
|
||||
"TRAEFIK LEADER-PINNING FIXED (ZB.MOM.WW.Health 0.2.1, published + restore-verified; OtOpcUa bumped at 8dd9da7d). Root cause was in the shared library: ActiveNodeDecision.Evaluate returned Degraded for role-member-but-not-leader, and MapZbHealth maps Degraded to 200 \u2014 so /health/active answered 200 on every node and OtOpcUa's Traefik kept the standby in the admin-UI pool. The shared spec had always said the tier 'Fails (503) on a standby or role-member-but-not-leader node', and OtOpcUa's own docs said '200 only on the Admin-role leader; 503 elsewhere'; the docs were right and the code was wrong. Live-verified: central-1 200/traefik UP, central-2 503/traefik DOWN; the README failover drill (which had never exercised anything) flipped Traefik to central-2 within 20s of stopping central-1 with the UI answering 200 continuously across the swap, and central-1 reclaimed the role-leader on restart. Blast radius one consumer \u2014 ScadaBridge uses its own OldestNodeActiveHealthCheck, mxgw/HG have no Akka.",
|
||||
"RIG OPERATIONS LANDMINE: the docker-dev migrator (dotnet ef database update) HANGS under x86 emulation \u2014 22 minutes at 0.00% CPU with no output, and all six node containers sit in Created because they depend_on service_completed_successfully. Worked around by stopping the migrator and docker-start-ing the nodes directly, which is safe when the change is a package-version bump with no EF model change. Worth knowing before assuming a rebuild is merely slow.",
|
||||
"lmxopcua#494 CLOSED. The site-node half needed a local ClusterPrimaryHealthCheck: this node is active iff it is the OLDEST Up member carrying its own active role (admin when present, else driver). IMPORTANT SELF-CORRECTION \u2014 the Health 0.2.1 fix alone was NOT sufficient. It made the tier a real 503 but still selected by RoleLeader (lowest address), and on the rebuilt rig the orderings diverge live: akka leader = central-1, oldest admin member = central-2 (which hosts the singletons). 0.2.1 alone would have pinned Traefik to central-1, the node NOT hosting the work \u2014 the exact failure RedundancyStateActor.SelectDriverPrimary was written to avoid, reappearing in the health tier. Only the live rig exposed it. Follow-up not filed: two of four family apps now need 'oldest Up member of role X' and both had to avoid the shared ActiveNodeHealthCheck to get it (ScadaBridge's OldestNodeActiveHealthCheck, OtOpcUa's ClusterPrimaryHealthCheck) \u2014 a candidate for promotion into ZB.MOM.WW.Health.Akka later.",
|
||||
"RIG BUILD LANDMINE: all six docker-dev node services share ONE image (otopcua-host:dev), so `docker compose build` with six service names builds the same image six times in parallel. Under x86 emulation that thrashed for 46+ minutes on `dotnet restore` with no output and looked wedged. Building a single service took 29 SECONDS for the same restore. Build one service, then recreate all six containers."
|
||||
],
|
||||
"phase4_results": {
|
||||
"1_all_render_single_active_per_pair": "PASS for both products. ScadaBridge: all four pairs one Active + one Standby. OtOpcUa: after lmxopcua#494 was fully closed (ClusterPrimaryHealthCheck, commit 45504861), exactly one 200 per mesh \u2014 MAIN central-2, SITE-A site-a-1, SITE-B site-b-1 \u2014 with the partner 503. Dashboard renders one Active + one Standby for all seven cluster groups.",
|
||||
"2_leader_chip_and_split_brain": "PASS for BOTH products after the rig redeploy and the LeaderResolver fix. All seven cluster groups (ScadaBridge central/site-a/b/c, OtOpcUa MAIN/SITE-A/SITE-B) render a Leader chip with zero split-brain flags. Verified earlier that the chip MOVES when the leader is stopped and that the split-brain flag appears while a pair is self-formed and clears once it genuinely joins.",
|
||||
"3_kill_and_restart": "PASS. Stopped scadabridge-site-c-a 12:01:27; Unreachable by 12:02:33 (two-strike damping = 2 failing sweeps). Restarted 12:04:01; Up again by 12:04:16, i.e. on the first good poll (recovery is deliberately asymmetric)",
|
||||
"4_degraded_with_failing_entry": "PASS. Canned Degraded and Unhealthy endpoints render as Degraded/Down with the failing entry and its description visible, and check-dot ok/warn/bad classes per check",
|
||||
"5_cache_first": "PASS. Five consecutive page loads in 0.00-0.02s, all showing the identical 'last sweep 12:17:44' header \u2014 the render never triggers a poll (an inline sweep could not finish in 10ms with two unreachable targets on 3s/10s timeouts)",
|
||||
"6_pause_and_stuck_poller": "PASS both halves. Pause -> both cards STALE within the 10s window, KPI stale=2, 'Stale since'; Resume -> clean recovery. Separately, SIGSTOPping the endpoints while auto-refresh stayed RUNNING froze the poller mid-sweep and the cards aged into STALE on their own \u2014 staleness is computed at render, so a stuck poller needs no explicit stall detection",
|
||||
"7_management_links_no_login": "PASS. 'Open CentralUI' opened http://localhost:9001 in a new tab (target=_blank rel=noopener); the dashboard itself never presented or redirected to a login",
|
||||
"8_malformed_registry": "PASS both branches. No registry -> ConfigPreflight names Overview:Applications:0:Name and :Instances:0:BaseUrl. Stale window inside the poll interval -> validator names the keys and the consequence",
|
||||
"9_health_and_metrics": "PASS. /health/ready, /health/active, /healthz, /metrics all 200; 16 gauge series with application/node/group labels"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,351 @@
|
||||
# ScadaBridge: migrating cross-cluster ClusterClient traffic to gRPC-only
|
||||
|
||||
**Date:** 2026-07-22. Code-verified inventory of every ClusterClient touchpoint in
|
||||
`~/Desktop/ScadaBridge`, then the migration assessment. Companion to
|
||||
`akka_msquic_transport.md` §8/§8a (why chunking can't fix this path) and
|
||||
`akka_multidc_clusters.md` (why the site↔central boundary stays two clusters).
|
||||
|
||||
## 1. What ClusterClient actually carries today
|
||||
|
||||
Everything crossing the site↔central boundary funnels through **two actors** —
|
||||
`SiteCommunicationActor` (site) and `CentralCommunicationActor` (central) — which makes the
|
||||
migration surface unusually well-bounded.
|
||||
|
||||
**Receptionist registrations (3, all in `AkkaHostedService.cs`):**
|
||||
|
||||
| Role | Actor | Path | Note |
|
||||
|---|---|---|---|
|
||||
| Central | `CentralCommunicationActor` | `/user/central-communication` (`:436`) | the real workhorse |
|
||||
| Central | `ManagementActor` | `/user/management` (`:459`) | **vestigial** — intended for an out-of-cluster CLI (REQ-HOST-6a), but the in-repo CLI has no ClusterClient; nothing in the repo sends to it |
|
||||
| Site | `SiteCommunicationActor` | `/user/site-communication` (`:935`) | central→site target |
|
||||
|
||||
**Site → central** (single `central-cluster-client`, contacts from
|
||||
`CommunicationOptions.CentralContactPoints`, `AkkaHostedService.cs:942-953`) — 7 message types,
|
||||
all Ask/reply except the heartbeat (`SiteCommunicationActor.cs`):
|
||||
|
||||
| Message | Purpose | Guarantee layered on top |
|
||||
|---|---|---|
|
||||
| `NotificationSubmit` → `NotificationSubmitAck` | S&F notification delivery (central owns SMTP) | not-accepted/timeout → rows stay buffered; active-node-gated retry sweep |
|
||||
| `NotificationStatusQuery` → response | S&F status lookup | — |
|
||||
| `IngestAuditEventsCommand` → reply | audit batch push | ack = accepted-id list; throw-on-failure keeps SQLite rows Pending |
|
||||
| `IngestCachedTelemetryCommand` → reply | audit+SiteCall dual-write push | same contract |
|
||||
| `ReconcileSiteRequest` → response | per-node startup self-heal | — |
|
||||
| `SiteHealthReport` → ack | rich periodic health → `CentralHealthAggregator` | monotonic sequence numbers + dedup; failed-send counter restore |
|
||||
| `HeartbeatMessage` (Tell) | liveness, stamps `IsActive` | fire-and-forget by design |
|
||||
|
||||
**Central → site** (one ClusterClient **per site**, contacts = NodeA/NodeB from the config DB,
|
||||
`CentralCommunicationActor.cs:469-598`): every command wrapped in
|
||||
`SiteEnvelope(siteId, message)` — **29 command types** produced by `CommunicationService`
|
||||
(instance lifecycle, artifact deploy, OPC-UA browse/verify/tag read-write, cert trust,
|
||||
event-log/parked-message queries, integration routing, `TriggerSiteFailover`).
|
||||
|
||||
**Serialization: the default Newtonsoft JSON fallback** — no `serialization-bindings` anywhere.
|
||||
This is the amplifier in the known 128 KB frame bug
|
||||
(`docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`): oversized
|
||||
ClusterClient messages are **silently dropped**, which is why deploy config already moved to
|
||||
HTTP notify-and-fetch.
|
||||
|
||||
**Not ClusterClient (stays untouched):** `DistributedPubSub` replica publishes
|
||||
(`SiteHeartbeatReplica`/`SiteHealthReportReplica`) are intra-central-cluster; all pair-internal
|
||||
Akka (singletons, membership, S&F actors) is intra-cluster.
|
||||
|
||||
## 2. Why gRPC-only is a natural completion, not a rewrite
|
||||
|
||||
The repo has already walked 3 of the 5 steps:
|
||||
|
||||
1. **`docs/plans/grpc_streams.md`** already rules ClusterClient "not built for high-throughput
|
||||
streaming" and moved all streaming values to gRPC — its stated end-state ("ClusterClient keeps
|
||||
command/control") is a waypoint, not a law.
|
||||
2. **Deploy config** already left ClusterClient for token-gated HTTP pull after the frame-size
|
||||
incident — with the rationale "fits the existing pull-based gRPC pattern".
|
||||
3. **The audit path is half-migrated already:** `sitestream.proto` *already defines*
|
||||
`IngestAuditEvents`/`IngestCachedTelemetry` with proto DTOs and an
|
||||
`ISiteStreamAuditClient` seam whose ClusterClient implementation is swappable — the contract
|
||||
and the seam exist; only the transport binding is Akka.
|
||||
4. Upstream direction agrees: JVM Akka deprecated ClusterClient in favor of gRPC for
|
||||
cross-cluster traffic; Akka.NET keeps it working but it is not where investment goes.
|
||||
5. Structural wins on arrival: the **silent-drop-over-128 KB failure class disappears** (gRPC
|
||||
default 4 MB cap + an explicit `ResourceExhausted` error instead of silence), Ask timeouts
|
||||
become real per-call deadlines, and **Akka remoting never crosses the boundary** — the
|
||||
site↔central firewall surface reduces to gRPC/HTTP ports, and the two clusters' quarantine
|
||||
lifecycles fully decouple (the coupling `akka_msquic_transport.md` §8a warns about).
|
||||
|
||||
## 3. What the migration takes
|
||||
|
||||
### 3.1 Topology decision — who serves what
|
||||
|
||||
Today the **site** is the only gRPC server (`SiteStreamService` on `:8083` h2c) and central the
|
||||
only client; but both sides already dial each other at the Akka level, and sites already dial
|
||||
central over HTTP for deploy fetch. So the clean shape is symmetric, no new firewall direction:
|
||||
|
||||
- **New `CentralControlService`** (gRPC) hosted on central's existing Kestrel (same host as the
|
||||
deploy-fetch endpoint): the 7 site→central RPCs.
|
||||
- **Extended site service** (grow `SiteStreamService` or a sibling `SiteCommandService` on the
|
||||
same `:8083` listener): the central→site command envelope.
|
||||
|
||||
(The alternative — a central-initiated duplex stream tunneling site→central pushes — avoids the
|
||||
central listener but re-implements connection management the receptionist gave for free; not
|
||||
worth it when sites already reach central over HTTP.)
|
||||
|
||||
### 3.2 Contract work
|
||||
|
||||
- **Site→central (small):** 7 RPCs. Audit/telemetry are a near-free move — proto DTOs exist and
|
||||
the swap is a new `ISiteStreamAuditClient` implementation (the `NoOp`/`ClusterClient`
|
||||
implementations prove the seam). Notifications, health report, reconcile, heartbeat need
|
||||
proto messages written (they're modest DTOs today).
|
||||
- **Central→site (the bulk):** 29 `SiteEnvelope` commands. Don't write 30 RPCs and don't smuggle
|
||||
JSON in one generic RPC — group by domain into a few RPCs with **`oneof` request/response
|
||||
envelopes** (lifecycle, deploy, OPC-UA browse/tag, queries, failover). Mirrors `SiteEnvelope`
|
||||
semantics with wire-level typing; mapper + golden tests are the real cost here.
|
||||
- **Transport seams:** the site side has precedent (`AkkaHealthReportTransport`,
|
||||
`ISiteStreamAuditClient`); `CommunicationService` (central) needs an `ISiteCommandTransport`
|
||||
seam so the 29 call sites don't change — the seam is introduced first, Akka behind it, then
|
||||
the gRPC implementation swaps in per domain group.
|
||||
|
||||
### 3.3 Semantics to preserve (all transport-agnostic, all carry over)
|
||||
|
||||
- S&F transient-failure contract: gRPC `Unavailable`/`DeadlineExceeded` → rows stay buffered —
|
||||
identical to today's not-accepted/timeout handling, with better error taxonomy.
|
||||
- Health sequence numbers + dedup + counter-restore: unchanged (they live above the transport).
|
||||
- Heartbeat stays fire-and-forget (unary, errors logged not thrown).
|
||||
- Dual-central routing: the receptionist let a site reach "whichever central answers"; the gRPC
|
||||
client needs try-in-order failover across the central A/B endpoints (`CentralContactPoints`
|
||||
becomes `CentralGrpcEndpoints`). The intra-central replica pub/sub already reconciles which
|
||||
central saw the report — unchanged.
|
||||
- Per-site channel management on central: the DB-driven contact cache refresh loop
|
||||
(`CentralCommunicationActor.cs:532-598`) carries over as a `GrpcChannel`-per-site factory —
|
||||
which `grpc_streams.md` (§167, §810) already models for the streaming path.
|
||||
|
||||
### 3.4 Work you must do anyway: auth — preshared key from the Secrets store (decided 2026-07-22)
|
||||
|
||||
`SiteStreamService` today has **no auth at all** — plaintext h2c, no interceptor (only the
|
||||
LocalDb sync path is ApiKey-gated, and only for its own service). This gap exists today
|
||||
regardless of migration — it just becomes indefensible after it. **Decision: the gRPC channels
|
||||
authenticate with a preshared key stored in `ZB.MOM.WW.Secrets`** (which ScadaBridge already
|
||||
adopted, `${secret:NAME}` resolution, fail-closed). Design:
|
||||
|
||||
- **Key scope: one PSK per site relationship** (e.g. secret name `SB-GRPC-PSK-SITE-A`), covering
|
||||
both directions of that pair's traffic (central→site commands AND that site's calls to
|
||||
`CentralControlService`). Central's store holds one secret per site; each site's local SQLite
|
||||
store holds only its own. Blast radius: a compromised site never yields another site's key.
|
||||
(A single fleet-wide PSK is simpler to seed but is rejected for exactly that reason.)
|
||||
- **Client side:** resolve `${secret:SB-GRPC-PSK-<site>}` at startup **fail-closed** (the
|
||||
Layer-B pattern the family already uses for driver secrets — boot fails loudly on a missing/
|
||||
undecryptable ref, no silent unauthenticated fallback) and attach it per-call as a bearer
|
||||
metadata header via a `CallCredentials`/interceptor on the channel pair from §3.5.
|
||||
- **Server side:** a fail-closed interceptor modeled on `LocalDbSyncAuthInterceptor` (the
|
||||
in-repo precedent) on **both** services — constant-time comparison, applied to the SiteStream/
|
||||
SiteCommand services on `:8083` and to `CentralControlService` on central. The LocalDb sync
|
||||
path keeps its own existing key; no sharing between the two mechanisms.
|
||||
- **Seeding/ops:** the `secret` CLI's *Reference audit & seed* flow (interactive console 0.3.0)
|
||||
is the deployment-setup tool — it classifies every `${secret:}` ref and seeds the gaps per
|
||||
node. Rotation: set the new value on both sides' stores, then restart the pair (pairs restart
|
||||
together by established discipline); if zero-blip rotation is ever needed, the server
|
||||
interceptor can accept old+new during a configured window.
|
||||
- **Transport honesty:** a bearer PSK over plaintext h2c is readable/replayable by anyone on
|
||||
path — today that's accepted via network isolation (same posture as the unauthenticated status
|
||||
quo, but now with authentication). TLS on these listeners is the follow-on hardening; the PSK
|
||||
design doesn't change if/when TLS lands.
|
||||
|
||||
### 3.5 Active-node resolution — does the gRPC client need to find the pair's active node?
|
||||
|
||||
**No — and this is load-bearing good news, verified in code.** Inbound commands on a site node
|
||||
are not executed locally: `SiteCommunicationActor` `Forward`s essentially everything to the
|
||||
**Deployment Manager cluster-singleton proxy** (`SiteCommunicationActor.cs:101-181`), and the
|
||||
code comments are explicit — the singleton "always lands on the active site node"; DCL
|
||||
connections/browse sessions "only exist on the singleton's node," so commands hop through the
|
||||
proxy by design (`:153-178`). The same shape holds centralward: `CentralCommunicationActor` runs
|
||||
on both central nodes, and heartbeat/health state is reconciled across the pair via the
|
||||
intra-cluster `DistributedPubSub` replicas. In other words, **each cluster already solves
|
||||
active-node routing internally with singleton proxies + pub/sub — the cross-cluster transport
|
||||
never had to know, and ClusterClient never did** (the receptionist delivers to whichever node
|
||||
answers).
|
||||
|
||||
A gRPC server on either pair node therefore keeps the exact same contract: accept the call, hand
|
||||
it to the same actor, let the singleton proxy route. What the migration must provide instead is
|
||||
cheap **either-node failover at the client**:
|
||||
|
||||
**Client failover/failback policy (per pair — applies to central's per-site clients AND the
|
||||
site's central client alike):**
|
||||
|
||||
- **Two channels, one preference.** Hold a `GrpcChannel` per endpoint (NodeA, NodeB) rather than
|
||||
one channel with re-dial logic; mark one *preferred* (config order, or `/health/active` probe
|
||||
once site-node health lands — Overview plan Phase 1). All calls go to the current channel.
|
||||
- **Failover:** on `Unavailable`/connect failure/readiness rejection
|
||||
(`SiteStreamGrpcServer` already returns `StatusCode.Unavailable` when not ready — the one
|
||||
state where the singleton proxy can't route), flip to the secondary **if available** and stay
|
||||
there (sticky-until-failure, so a flapping primary doesn't ping-pong every call).
|
||||
- **Failback:** a background probe (30–60 s cadence) checks the preferred endpoint
|
||||
(`grpc.health` check or `/health/ready`); when it answers again, new calls return to it —
|
||||
in-flight calls and open streams finish where they are. Failback is an optimization, never a
|
||||
correctness requirement: either node accepts and singleton-routes (§ above).
|
||||
- **Retry safety:** automatic cross-node retry ONLY for failures that provably never reached a
|
||||
server (connect refused, `Unavailable` before response headers). **Never blind-retry
|
||||
`DeadlineExceeded` on non-idempotent commands** (`WriteTag`, `DeployArtifacts`,
|
||||
`TriggerSiteFailover`) — ClusterClient's Ask had the same ambiguity and the layers above
|
||||
already tolerate it (S&F dedup, sequence numbers), but the migration must not *add* an eager
|
||||
retry that turns one write into two. Queries/browses are freely retryable.
|
||||
- **Streams don't fail over transparently:** an open `SubscribeInstance`/`SubscribeSite` stream
|
||||
dies with its channel; the subscriber re-subscribes via the policy above (the
|
||||
`grpc_streams.md` design already owns re-subscribe semantics — reuse, don't reinvent).
|
||||
- Cost of landing on the standby is one intra-pair remoting hop (as today).
|
||||
- One verification item for Phase 2: confirm none of the 29 command handlers bypasses the proxy
|
||||
and assumes local state (the audit/parked-message query paths are the ones to eyeball).
|
||||
|
||||
### 3.6 Deletions at the end
|
||||
|
||||
`ClusterClient`/`ClusterClientReceptionist` creation + registrations, `DefaultSiteClientFactory`,
|
||||
the `SiteEnvelope` Akka path, `CommunicationOptions.CentralContactPoints`. The
|
||||
`/user/management` registration is **deletable today** independent of everything else (no sender
|
||||
exists in the repo). `Akka.Cluster.Tools` stays for ClusterSingleton.
|
||||
|
||||
### 3.7 The management CLI is unaffected (deliberately out of scope)
|
||||
|
||||
`scadabridge.exe` never used ClusterClient: it speaks HTTP Basic to a Central's `/management`
|
||||
endpoints, which ask `ManagementActor` **in-process** (`ManagementEndpoints.cs:117` via
|
||||
`ManagementActorHolder`) — the `/user/management` receptionist registration was built for an
|
||||
out-of-cluster CLI that was never wired (REQ-HOST-6a) and has no sender in the repo. So this
|
||||
migration requires **zero CLI changes**; the CLI's transport is already boundary-independent.
|
||||
Retargeting the CLI onto the new gRPC services later is a separate, optional decision — it would
|
||||
buy one consistent management wire, but costs the CLI a gRPC stack and auth story for no current
|
||||
functional gain, so it is explicitly **not** part of this migration's scope or estimate.
|
||||
|
||||
## 4. Phasing and effort
|
||||
|
||||
| Phase | Content | Effort |
|
||||
|---|---|---|
|
||||
| 0 | Delete vestigial `ManagementActor` registration; add the §3.4 PSK-from-Secrets auth interceptor to `SiteStreamService` (standalone hardening win — central client attaches the PSK it resolves from its store) | days |
|
||||
| 1 | `CentralControlService` + the 7 site→central RPCs behind existing seams; audit swap; heartbeat/health/notifications/reconcile | ~1–2 wks |
|
||||
| 2 | `ISiteCommandTransport` seam + `oneof` command envelopes; migrate the 29 commands by domain group (deploy+lifecycle first, queries last) | ~2–3 wks |
|
||||
| 3 | Remove ClusterClient/receptionist + config cutover; live gate on the docker rig: kill central-a mid-S&F drain, site-node failover, deploy, `TriggerSiteFailover`, dual-central failover of site→central calls | ~1 wk |
|
||||
|
||||
Total ≈ **4–6 weeks** with live gating. Both transports can coexist per-path throughout (the
|
||||
seams make each path a one-line flip), so there is no big-bang cutover; rollback per phase is a
|
||||
config/DI revert.
|
||||
|
||||
**Risks, honestly:** the `oneof` envelope mappers are tedious and where bugs will hide (golden
|
||||
tests against captured payloads); ordering assumptions are low-risk (everything is request/reply
|
||||
or sequence-numbered, verify none of the 30 commands relies on ClusterClient's per-connection
|
||||
ordering across *different* commands); Ask-sender-forwarding tricks disappear (fine — RPC replies
|
||||
are direct, but check `SiteCallAuditActor`'s relay paths); the auth addition must not break the
|
||||
rig's current unauthenticated compose setup (env-gated like LocalDb's).
|
||||
|
||||
## 5. The other knob: raising the 128 KB frame cap
|
||||
|
||||
Since the frame limit motivates part of this migration, the obvious question — why not just
|
||||
raise `akka.remote.dot-netty.tcp.maximum-frame-size`? It is a legitimate knob, and if
|
||||
ClusterClient stays (Option B) it is worth doing deliberately, but know exactly what it buys:
|
||||
|
||||
- **How:** set `maximum-frame-size` (default 128000b) in the generated HOCON
|
||||
(`AkkaHostedService.BuildHocon` — today it never overrides it) and raise
|
||||
`send-buffer-size`/`receive-buffer-size` (default 256000b) to at least the new frame size.
|
||||
**The limit is enforced by the receiver**, so it must be raised on **every node of both
|
||||
clusters and rolled out everywhere before any sender relies on it** — a half-rolled-out raise
|
||||
reproduces the silent-drop bug in one direction. The pair-restart discipline makes the rollout
|
||||
cheap; the coordination requirement is the trap.
|
||||
- **What it buys:** headroom for today's fat-ish commands (browse results, query replies) with
|
||||
zero code change; the deploy-config class of payload would fit at, say, 2–4 MB.
|
||||
- **What it does not fix:** (1) the **silent-drop failure class survives** — the threshold moves,
|
||||
it doesn't disappear; anything over the new cap still vanishes without an error. (2)
|
||||
**Head-of-line gets worse, not better:** all ClusterClient traffic, heartbeats, and
|
||||
failure-detector signals share one ordered dot-netty channel, so a single 4 MB frame stalls
|
||||
everything behind it — on a 2-node availability-first posture with tight failure-detector
|
||||
timeouts, big frames are exactly what you don't want ahead of heartbeats (this is why the
|
||||
deploy fix chose notify-and-fetch over a cap raise, and why `akka_msquic_transport.md` §2
|
||||
treats the default cap as a guardrail, not a bug). (3) The default Newtonsoft serializer's
|
||||
double-JSON-escaping roughly **halves the effective budget** for JSON-bearing payloads. (4) It
|
||||
is a dead-end investment: Artery (1.6) replaces the pipeline, and Option A removes the
|
||||
constraint entirely (gRPC's 4 MB default cap fails loudly with `ResourceExhausted`, and is
|
||||
itself configurable per-call-type).
|
||||
|
||||
Verdict: as a *stopgap under Option B*, a modest raise (e.g. 512 KB–1 MB, buffers to match,
|
||||
cluster-wide) plus an explicit size check before `ClusterClient.Send` (log-and-reject instead of
|
||||
silent drop) is defensible. As part of Option A it's unnecessary — don't spend the rollout.
|
||||
|
||||
## 6. Options
|
||||
|
||||
**A. Full migration, phased as §4 (recommended).** Best-practice fit: it completes a direction
|
||||
the repo already chose twice under pressure (streaming, deploy), eliminates the silent-drop
|
||||
failure class on the control path, closes the unauthenticated-control-channel gap as part of the
|
||||
work, decouples the clusters' failure domains entirely, and lands on the transport the whole
|
||||
family already operates (gRPC + Kestrel + interceptors). Aligns with upstream's own deprecation
|
||||
of ClusterClient.
|
||||
|
||||
**B. Status quo per `grpc_streams.md` (ClusterClient keeps command/control).** Defensible if no
|
||||
current pain: the frame-size hazard is mostly defused (big payloads already moved) and the
|
||||
receptionist's discovery/failover is free; pair it with the deliberate cap raise + pre-send size
|
||||
check from §5. But it leaves two cross-cluster transports to operate,
|
||||
the silent-drop class alive for any future fat command, an unauthenticated gRPC service, and a
|
||||
dependency on the least-invested-in corner of Akka.NET. If choosing B, still do Phase 0 (auth +
|
||||
vestigial-registration deletion) — it stands alone.
|
||||
|
||||
**C. Wait for Akka.NET 1.6.** Not a reason to wait: Artery changes intra-cluster transport;
|
||||
ClusterClient's cross-cluster story doesn't improve. Deferring buys nothing here.
|
||||
|
||||
**Recommendation: A** — with Phase 0 done immediately regardless of the rest.
|
||||
|
||||
## 7. Design-completion deep dive (2026-07-22) — corrections and refinements
|
||||
|
||||
A three-track code deep dive (full message inventory; hosting/config/Secrets/tests; existing
|
||||
gRPC machinery) closed the remaining unknowns. The implementation plan lives at
|
||||
`ScadaBridge/docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`; corrections that supersede
|
||||
earlier sections:
|
||||
|
||||
1. **The seam is smaller than §3.2 assumed — one choke point per direction.** Every central→site
|
||||
producer (all of `CommunicationService` AND `SiteCallAuditActor`'s two relays) funnels
|
||||
`SiteEnvelope` through **`CentralCommunicationActor`**, which unwraps and routes via the
|
||||
per-site ClusterClient; every site→central path funnels through **`SiteCommunicationActor`**.
|
||||
So the transport swap happens *inside the two communication actors* (an injected
|
||||
`ISiteCommandTransport` / `ICentralTransport` behind a config flag) — `CommunicationService`
|
||||
needs **no interface extraction** and its ~20 consumers are untouched. (It has no interface
|
||||
today and is DI-registered concrete; extracting one is unnecessary.)
|
||||
2. **Exact command count: 29, and one is dead.** 27 from `CommunicationService` + 2
|
||||
`SiteCallAuditActor` relays. **Finding: `IntegrationCallRequest` is unwired in production** —
|
||||
`RegisterLocalHandler(Integration, …)` exists only in tests, so the command always answers
|
||||
"Integration handler not available." It is excluded from the gRPC contract (28 commands
|
||||
migrate) and filed as its own issue rather than silently ported or fixed.
|
||||
3. **§3.5 verification item RESOLVED.** Of the 29: 22 are singleton-/cluster-routed (incl.
|
||||
artifact deploy and event-log query, whose "local handler" slots are actually registered to
|
||||
singleton proxies — the event-log one deliberately, per wiring comments); `TriggerSiteFailover`
|
||||
resolves its target from cluster state, node-agnostic. The **only node-local target is the
|
||||
parked-message handler** (5 command types) — and it is standby-safe *by replication*, not
|
||||
routing: it reads/mutates the LocalDb-replicated S&F store, and the delivery sweep is
|
||||
active-gated. The gRPC design must preserve exactly that (dispatch on the receiving node, no
|
||||
forced active-node routing) — which either-node gRPC does.
|
||||
4. **Central hosting is net-new work.** Central has **no** `AddGrpc`/`MapGrpcService` and no
|
||||
explicit Kestrel listener (env-driven `:5000`, Traefik in front on HTTP/1). The migration adds
|
||||
an explicit HTTP/2 (h2c) listener on central (port `8083`, symmetric with sites; rig publishes
|
||||
it), `AddGrpc` + the PSK interceptor, and `CentralControlService`. Sites also get a **new**
|
||||
standing config `ScadaBridge:Communication:CentralGrpcEndpoints` — today the only central URL
|
||||
a site sees arrives *inside* `RefreshDeploymentCommand` (`CentralFetchBaseUrl`), so gRPC
|
||||
endpoint discovery genuinely needs new config.
|
||||
5. **PSK mechanics, concretized from the in-repo templates.** Server side copies
|
||||
`LocalDbSyncAuthInterceptor` exactly (service-prefix match, `authorization: Bearer`,
|
||||
`CryptographicOperations.FixedTimeEquals`, fail-closed on unset key, `PermissionDenied`);
|
||||
clients attach the key as call-level `Metadata` (the LocalDb sync client pattern). Key
|
||||
sourcing: **site nodes** read a static config key supplied as `${secret:SB-GRPC-PSK-<site>}`
|
||||
via the existing pre-host `SecretReferenceExpander` (`Program.cs:56-65`); **central** — whose
|
||||
site set is dynamic from the DB — resolves `SB-GRPC-PSK-{siteId}` at channel-build time via
|
||||
the runtime `ISecretResolver` (the MxGateway G-3 pattern), cached, fail-closed. Sites identify
|
||||
themselves to `CentralControlService` with an `x-scadabridge-site` metadata header so central
|
||||
verifies against that site's key. The rig mirrors the LocalDb dev pattern (same literal on
|
||||
both pair nodes; prod uses `${secret:}`).
|
||||
6. **Reuse map for §3.5's client policy:** per-site `GrpcNodeAAddress`/`GrpcNodeBAddress` columns
|
||||
already exist on the `Site` entity (streaming path reads them); primary/fallback single-retry
|
||||
exists in the pull clients; sticky flip + stability-window retry-budget exists in
|
||||
`DebugStreamBridgeActor`/`SiteAlarmAggregatorActor`. The plan extracts this into one shared
|
||||
`SitePairChannelProvider` rather than a fourth hand-rolled copy. **Caution — doc-vs-code gap:**
|
||||
`grpc_streams.md` names `CentralCommunicationActor.LoadSiteAddressesFromDb()` and
|
||||
singleton-aware node selection; neither exists in code (selection is blind A-then-B flip).
|
||||
7. **Deadlines are net-new.** No gRPC call in the repo sets a deadline or retry policy today; the
|
||||
new services introduce per-call deadlines mirroring the eight `CommunicationOptions` Ask
|
||||
timeouts (`DeploymentTimeout` 2 m … `HealthReportTimeout` 10 s), preserving the documented
|
||||
inner-before-outer timeout ordering (e.g. `SiteCallAuditOptions.RelayTimeout` 10 s <
|
||||
`QueryTimeout` 30 s).
|
||||
8. **Proto codegen is checked-in, not build-time** (protoc segfaults in the linux_arm64 build
|
||||
image): new protos follow the sitestream recipe — commented-out `<Protobuf>` item, generate on
|
||||
macOS, commit the generated C#, `docker/regen-proto.sh`.
|
||||
9. **Test assets to extend:** `Communication.Tests` already covers both communication actors with
|
||||
Akka TestKit (xunit **v2** — TestKit.Xunit2 compatible) + NSubstitute;
|
||||
`DirectActorSiteStreamAuditClient` is an explicitly-reusable wire-bypassing double for
|
||||
ingest-path integration tests.
|
||||
Reference in New Issue
Block a user