a2f0086fa1
ROW 80 (observability correctness). An unreplicated site published
LocalDbReplicationConnected=false / LocalDbOplogBacklog=0, contradicting the
nullable semantics every doc and the proto already state ("null means no
reading, NOT disconnected with an empty backlog").
Root cause: the replication engine is registered unconditionally and idles
without a peer, so ISyncStatus answers false/0 on a node that was never meant to
replicate, and LocalDbReplicationStatusReporter passed that straight through.
Two consequences: an operator read connected:false on a healthy site as a fault,
and central's null branches — row 33's conditional KPI emission and
sustained-badge anchor exclusion — never received the null they exist for, so
rig site-b/site-c emitted a permanently misleading
localDbReplicationConnected=0 series.
Fix mirrors the SiteEventLogReplicationCheck precedent: the Host evaluates
SiteLocalDbSetup.ReplicationIsConfigured ONCE and passes it to
AddLocalDbReplicationHealthBridge as a third provider; unconfigured
short-circuits the probe to (null, null). HealthMonitoring never reads LocalDb
config itself, and the same captured value now also feeds
SiteEventLogReplicationCheck so the two surfaces cannot disagree.
ISiteHealthCollector.SetLocalDbReplicationStatus widens to bool? — the
non-nullable bool was the type-level root cause (null was inexpressible).
false now means exactly one thing: configured, peer session down = a real fault.
Wire/central/UI needed no change: the proto already carries BoolValue/Int64Value
wrappers and the aggregator + KPI source already branch on null. This makes code
match doc, so no doc claim was weakened.
Watch the trap that caused this: configured-ness is PeerAddress OR ApiKey (only
the dialling half sets a peer address). The pre-existing SiteLocalDbWiringTests
fixture sets ApiKey and IS a replicating node, so its false/0 assertion is
correct and is kept, with a comment saying so; the new
UnreplicatedSiteLocalDbHealthTests fixture is the genuinely-unreplicated one.
ROW 73 (silent loss of self-heal). AkkaHostedService skips
SiteReconciliationActor when no IDeploymentConfigFetcher is resolvable, and that
skip was a lone boot Warning — a node with no self-heal path at all went on
reporting itself perfectly healthy forever.
Adds tri-state SiteHealthReport.StartupReconcilerActive (null = not reported,
i.e. the central self-report or a pre-row-73 site build; true = running;
false = booted without it), latched once at boot by the Host in both branches.
Additive proto field 29 (BoolValue, vendored C# regenerated via
docker/regen-proto.sh centralcontrol; csproj restored, verified empty diff).
The skip branch also raises its log to Error — no configuration makes the
fetcher legitimately absent on a site node — and writes a best-effort
deployment/Error site event so the evidence outlives the boot log.
Central treatment: a "No self-heal" WARNING badge on the site tile, not a
failure. Config still reaches the node via LocalDb replication while it is up;
what it lost is recovery from a deploy that landed while it was down — degraded,
not offline. Only the explicit false is badged (badging null would light up the
whole fleet the moment central is upgraded ahead of the sites), and offline
sites are skipped, matching the replication-backlog badge.
Fail-before evidence: with the row-80 gate behaviourally reverted,
UnreplicatedSite_HealthReport_ReportsNullForBothLocalDbFields fails
"Expected: null / Actual: False" and 3 of the 8 new reporter unit tests fail.
Tests: HealthMonitoring 128, Host 518, CentralUI 1051, Communication 717,
Commons 715, SiteEventLogging 81 — all green.
183 lines
7.6 KiB
C#
183 lines
7.6 KiB
C#
using Microsoft.Extensions.Logging.Abstractions;
|
|
|
|
namespace ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests;
|
|
|
|
/// <summary>
|
|
/// Deferred-work register row 80 — <see cref="LocalDbReplicationStatusReporter"/> must
|
|
/// distinguish "replication is not configured on this node" (null, no reading) from
|
|
/// "replication is configured and the peer session is down" (false, a genuine fault).
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// The replication engine is registered unconditionally on every site node and sits inert
|
|
/// without a peer, so <c>ISyncStatus</c> answers <c>Connected=false</c> / <c>OplogBacklog=0</c>
|
|
/// on a site that was never meant to replicate. Passing that straight through made rig
|
|
/// site-b/site-c publish "disconnected, empty backlog" — a healthy unreplicated site reading
|
|
/// as a broken replicated one, and central's null branches (conditional KPI emission,
|
|
/// sustained-badge anchor exclusion, per register row 33) never receiving the null they exist
|
|
/// for. The gate is supplied by the Host from <c>SiteLocalDbSetup.ReplicationIsConfigured</c>,
|
|
/// mirroring <c>SiteEventLogReplicationCheck</c>.
|
|
/// </remarks>
|
|
public class LocalDbReplicationStatusReporterTests
|
|
{
|
|
private static LocalDbReplicationStatusReporter Build(
|
|
SiteHealthCollector collector,
|
|
bool replicationConfigured,
|
|
bool connected = false,
|
|
long? backlog = 0L) =>
|
|
new(
|
|
connectedProvider: () => connected,
|
|
oplogBacklogProvider: () => backlog,
|
|
replicationConfiguredProvider: () => replicationConfigured,
|
|
collector: collector,
|
|
logger: NullLogger<LocalDbReplicationStatusReporter>.Instance,
|
|
refreshInterval: TimeSpan.FromHours(1));
|
|
|
|
[Fact]
|
|
public void Probe_ReplicationUnconfigured_PublishesNullForBothFields()
|
|
{
|
|
// The row-80 regression. The inert engine's false/0 must NOT reach the report.
|
|
var collector = new SiteHealthCollector();
|
|
using var reporter = Build(collector, replicationConfigured: false);
|
|
|
|
reporter.Probe();
|
|
var report = collector.CollectReport("TestSite");
|
|
|
|
Assert.Null(report.LocalDbReplicationConnected);
|
|
Assert.Null(report.LocalDbOplogBacklog);
|
|
}
|
|
|
|
[Fact]
|
|
public void Probe_ReplicationUnconfigured_DoesNotReadTheSyncStatusProviders()
|
|
{
|
|
// The gate short-circuits rather than reading and discarding: an unconfigured node
|
|
// should not be polling a replication engine it never uses. Also guards against a
|
|
// future provider that throws when replication is off.
|
|
var connectedReads = 0;
|
|
var backlogReads = 0;
|
|
var collector = new SiteHealthCollector();
|
|
using var reporter = new LocalDbReplicationStatusReporter(
|
|
connectedProvider: () => { connectedReads++; return false; },
|
|
oplogBacklogProvider: () => { backlogReads++; return 0L; },
|
|
replicationConfiguredProvider: () => false,
|
|
collector: collector,
|
|
logger: NullLogger<LocalDbReplicationStatusReporter>.Instance,
|
|
refreshInterval: TimeSpan.FromHours(1));
|
|
|
|
reporter.Probe();
|
|
|
|
Assert.Equal(0, connectedReads);
|
|
Assert.Equal(0, backlogReads);
|
|
}
|
|
|
|
[Fact]
|
|
public void Probe_ReplicationConfiguredButDisconnected_PublishesFalse_NotNull()
|
|
{
|
|
// The other half of the contract, and the reason the fix is a gate rather than a
|
|
// blanket "never report false": on a node that IS configured to replicate, a down
|
|
// peer session is a real fault and must stay visible as false.
|
|
var collector = new SiteHealthCollector();
|
|
using var reporter = Build(
|
|
collector, replicationConfigured: true, connected: false, backlog: 4200L);
|
|
|
|
reporter.Probe();
|
|
var report = collector.CollectReport("TestSite");
|
|
|
|
Assert.False(report.LocalDbReplicationConnected);
|
|
Assert.Equal(4200L, report.LocalDbOplogBacklog);
|
|
}
|
|
|
|
[Fact]
|
|
public void Probe_ReplicationConfiguredAndConnected_PublishesTrue()
|
|
{
|
|
var collector = new SiteHealthCollector();
|
|
using var reporter = Build(
|
|
collector, replicationConfigured: true, connected: true, backlog: 0L);
|
|
|
|
reporter.Probe();
|
|
var report = collector.CollectReport("TestSite");
|
|
|
|
Assert.True(report.LocalDbReplicationConnected);
|
|
Assert.Equal(0L, report.LocalDbOplogBacklog);
|
|
}
|
|
|
|
[Fact]
|
|
public void Probe_ReplicationConfigured_PreservesANullBacklogAsUnknown()
|
|
{
|
|
// A failed oplog poll on a REPLICATED node is "unknown", never 0 — rendering it as a
|
|
// healthy empty backlog would report a pair that cannot read its own oplog as fine.
|
|
// Connected stays false (the session really is down); only the backlog is unknown.
|
|
var collector = new SiteHealthCollector();
|
|
using var reporter = Build(
|
|
collector, replicationConfigured: true, connected: false, backlog: null);
|
|
|
|
reporter.Probe();
|
|
var report = collector.CollectReport("TestSite");
|
|
|
|
Assert.False(report.LocalDbReplicationConnected);
|
|
Assert.Null(report.LocalDbOplogBacklog);
|
|
}
|
|
|
|
[Fact]
|
|
public void Probe_ProviderThrows_LeavesThePreviousSnapshotIntact()
|
|
{
|
|
// Health reporting must never take the node down, and a failed probe must not
|
|
// silently downgrade a good reading to null.
|
|
var shouldThrow = false;
|
|
var collector = new SiteHealthCollector();
|
|
using var reporter = new LocalDbReplicationStatusReporter(
|
|
connectedProvider: () => shouldThrow
|
|
? throw new InvalidOperationException("sync status unavailable")
|
|
: true,
|
|
oplogBacklogProvider: () => 7L,
|
|
replicationConfiguredProvider: () => true,
|
|
collector: collector,
|
|
logger: NullLogger<LocalDbReplicationStatusReporter>.Instance,
|
|
refreshInterval: TimeSpan.FromHours(1));
|
|
|
|
reporter.Probe();
|
|
shouldThrow = true;
|
|
reporter.Probe();
|
|
|
|
var report = collector.CollectReport("TestSite");
|
|
Assert.True(report.LocalDbReplicationConnected);
|
|
Assert.Equal(7L, report.LocalDbOplogBacklog);
|
|
}
|
|
|
|
[Fact]
|
|
public void Probe_GateFlippedOn_StopsReportingNull()
|
|
{
|
|
// Turning replication on for a previously-unreplicated site is a both-nodes-together
|
|
// restart in practice, but the gate is read per probe, so pin that it is not latched.
|
|
var configured = false;
|
|
var collector = new SiteHealthCollector();
|
|
using var reporter = new LocalDbReplicationStatusReporter(
|
|
connectedProvider: () => true,
|
|
oplogBacklogProvider: () => 12L,
|
|
replicationConfiguredProvider: () => configured,
|
|
collector: collector,
|
|
logger: NullLogger<LocalDbReplicationStatusReporter>.Instance,
|
|
refreshInterval: TimeSpan.FromHours(1));
|
|
|
|
reporter.Probe();
|
|
Assert.Null(collector.CollectReport("TestSite").LocalDbReplicationConnected);
|
|
|
|
configured = true;
|
|
reporter.Probe();
|
|
|
|
var report = collector.CollectReport("TestSite");
|
|
Assert.True(report.LocalDbReplicationConnected);
|
|
Assert.Equal(12L, report.LocalDbOplogBacklog);
|
|
}
|
|
|
|
[Fact]
|
|
public void Constructor_NullReplicationConfiguredProvider_Throws()
|
|
{
|
|
Assert.Throws<ArgumentNullException>(() => new LocalDbReplicationStatusReporter(
|
|
connectedProvider: () => false,
|
|
oplogBacklogProvider: () => 0L,
|
|
replicationConfiguredProvider: null!,
|
|
collector: new SiteHealthCollector(),
|
|
logger: NullLogger<LocalDbReplicationStatusReporter>.Instance));
|
|
}
|
|
}
|