a2f0086fa1
ROW 80 (observability correctness). An unreplicated site published
LocalDbReplicationConnected=false / LocalDbOplogBacklog=0, contradicting the
nullable semantics every doc and the proto already state ("null means no
reading, NOT disconnected with an empty backlog").
Root cause: the replication engine is registered unconditionally and idles
without a peer, so ISyncStatus answers false/0 on a node that was never meant to
replicate, and LocalDbReplicationStatusReporter passed that straight through.
Two consequences: an operator read connected:false on a healthy site as a fault,
and central's null branches — row 33's conditional KPI emission and
sustained-badge anchor exclusion — never received the null they exist for, so
rig site-b/site-c emitted a permanently misleading
localDbReplicationConnected=0 series.
Fix mirrors the SiteEventLogReplicationCheck precedent: the Host evaluates
SiteLocalDbSetup.ReplicationIsConfigured ONCE and passes it to
AddLocalDbReplicationHealthBridge as a third provider; unconfigured
short-circuits the probe to (null, null). HealthMonitoring never reads LocalDb
config itself, and the same captured value now also feeds
SiteEventLogReplicationCheck so the two surfaces cannot disagree.
ISiteHealthCollector.SetLocalDbReplicationStatus widens to bool? — the
non-nullable bool was the type-level root cause (null was inexpressible).
false now means exactly one thing: configured, peer session down = a real fault.
Wire/central/UI needed no change: the proto already carries BoolValue/Int64Value
wrappers and the aggregator + KPI source already branch on null. This makes code
match doc, so no doc claim was weakened.
Watch the trap that caused this: configured-ness is PeerAddress OR ApiKey (only
the dialling half sets a peer address). The pre-existing SiteLocalDbWiringTests
fixture sets ApiKey and IS a replicating node, so its false/0 assertion is
correct and is kept, with a comment saying so; the new
UnreplicatedSiteLocalDbHealthTests fixture is the genuinely-unreplicated one.
ROW 73 (silent loss of self-heal). AkkaHostedService skips
SiteReconciliationActor when no IDeploymentConfigFetcher is resolvable, and that
skip was a lone boot Warning — a node with no self-heal path at all went on
reporting itself perfectly healthy forever.
Adds tri-state SiteHealthReport.StartupReconcilerActive (null = not reported,
i.e. the central self-report or a pre-row-73 site build; true = running;
false = booted without it), latched once at boot by the Host in both branches.
Additive proto field 29 (BoolValue, vendored C# regenerated via
docker/regen-proto.sh centralcontrol; csproj restored, verified empty diff).
The skip branch also raises its log to Error — no configuration makes the
fetcher legitimately absent on a site node — and writes a best-effort
deployment/Error site event so the evidence outlives the boot log.
Central treatment: a "No self-heal" WARNING badge on the site tile, not a
failure. Config still reaches the node via LocalDb replication while it is up;
what it lost is recovery from a deploy that landed while it was down — degraded,
not offline. Only the explicit false is badged (badging null would light up the
whole fleet the moment central is upgraded ahead of the sites), and offline
sites are skipped, matching the replication-backlog badge.
Fail-before evidence: with the row-80 gate behaviourally reverted,
UnreplicatedSite_HealthReport_ReportsNullForBothLocalDbFields fails
"Expected: null / Actual: False" and 3 of the 8 new reporter unit tests fail.
Tests: HealthMonitoring 128, Host 518, CentralUI 1051, Communication 717,
Commons 715, SiteEventLogging 81 — all green.
135 lines
5.8 KiB
C#
135 lines
5.8 KiB
C#
using Microsoft.AspNetCore.Builder;
|
|
using Microsoft.Extensions.Configuration;
|
|
using Microsoft.Extensions.DependencyInjection;
|
|
using Microsoft.Extensions.Hosting;
|
|
using ZB.MOM.WW.LocalDb.Replication;
|
|
using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
|
|
using ZB.MOM.WW.ScadaBridge.Host.Actors;
|
|
|
|
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
|
|
|
|
/// <summary>
|
|
/// Deferred-work register row 80 — a site node with replication UNCONFIGURED must report
|
|
/// <c>LocalDbReplicationConnected</c> and <c>LocalDbOplogBacklog</c> as <c>null</c>.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// <para>
|
|
/// The sibling <see cref="SiteLocalDbWiringTests"/> fixture is a <em>replicating</em> node
|
|
/// (it sets <c>LocalDb:Replication:ApiKey</c>, modelling the passive half of the rig's
|
|
/// site-a pair), so false/0 is the right answer there. This fixture is the other posture —
|
|
/// the rig's site-b/site-c, and the shipping default — with NEITHER <c>PeerAddress</c> nor
|
|
/// <c>ApiKey</c>.
|
|
/// </para>
|
|
/// <para>
|
|
/// The replication engine is registered unconditionally and idles, so <c>ISyncStatus</c>
|
|
/// still answers <c>Connected=false</c> with a real <c>0</c> backlog. Publishing that pair
|
|
/// is the row-80 bug: a dashboard reader sees <c>connected: false</c> on a perfectly healthy
|
|
/// unreplicated site and reads a fault, and central's null branches — the conditional KPI
|
|
/// emission and the sustained-badge anchor exclusion built for register row 33 — never
|
|
/// receive the null they exist for, so unreplicated sites emitted a misleading
|
|
/// <c>localDbReplicationConnected=0</c> series.
|
|
/// </para>
|
|
/// <para>
|
|
/// Built from the REAL composition root for the same reason the sibling fixture is: the
|
|
/// defect is in the wiring, and a hand-assembled <c>ServiceCollection</c> would prove
|
|
/// nothing about what the Host actually passes.
|
|
/// </para>
|
|
/// </remarks>
|
|
public class UnreplicatedSiteLocalDbHealthTests : IDisposable
|
|
{
|
|
private readonly WebApplication _host;
|
|
private readonly string _tempDbPath;
|
|
private readonly string _tempSiteDbPath;
|
|
private readonly string _tempTrackingDbPath;
|
|
|
|
public UnreplicatedSiteLocalDbHealthTests()
|
|
{
|
|
var stamp = Guid.NewGuid();
|
|
_tempDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_unrep_localdb_{stamp}.db");
|
|
_tempSiteDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_unrep_site_{stamp}.db");
|
|
_tempTrackingDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_unrep_track_{stamp}.db");
|
|
|
|
var builder = WebApplication.CreateBuilder();
|
|
builder.Configuration.Sources.Clear();
|
|
builder.Configuration.AddInMemoryCollection(new Dictionary<string, string?>
|
|
{
|
|
["ScadaBridge:Node:Role"] = "Site",
|
|
["ScadaBridge:Node:NodeName"] = "node-a",
|
|
["ScadaBridge:Node:NodeHostname"] = "test-site-b",
|
|
["ScadaBridge:Node:SiteId"] = "TestSiteB",
|
|
["ScadaBridge:Node:RemotingPort"] = "0",
|
|
["ScadaBridge:Node:GrpcPort"] = "0",
|
|
["ScadaBridge:Database:SiteDbPath"] = _tempSiteDbPath,
|
|
["ScadaBridge:OperationTracking:ConnectionString"] = $"Data Source={_tempTrackingDbPath}",
|
|
["ScadaBridge:Cluster:SeedNodes:0"] = "akka.tcp://scadabridge@localhost:2551",
|
|
["ScadaBridge:Cluster:SeedNodes:1"] = "akka.tcp://scadabridge@localhost:2552",
|
|
|
|
// The point of this fixture: LocalDb:Path only. No PeerAddress, no ApiKey —
|
|
// SiteLocalDbSetup.ReplicationIsConfigured is false, matching rig site-b/site-c.
|
|
["LocalDb:Path"] = _tempDbPath,
|
|
});
|
|
|
|
builder.Services.AddGrpc();
|
|
builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer>();
|
|
|
|
SiteServiceRegistration.Configure(builder.Services, builder.Configuration);
|
|
|
|
AkkaHostedServiceRemover.RemoveAkkaHostedServiceOnly(builder.Services);
|
|
|
|
_host = builder.Build();
|
|
}
|
|
|
|
public void Dispose()
|
|
{
|
|
(_host as IDisposable)?.Dispose();
|
|
foreach (var path in new[] { _tempDbPath, _tempSiteDbPath, _tempTrackingDbPath })
|
|
{
|
|
try { File.Delete(path); } catch { /* best effort */ }
|
|
}
|
|
|
|
GC.SuppressFinalize(this);
|
|
}
|
|
|
|
[Fact]
|
|
public void UnreplicatedSite_HealthReport_ReportsNullForBothLocalDbFields()
|
|
{
|
|
// The row-80 regression, end-to-end through the real container and the real
|
|
// collector. Before the fix this produced false/0.
|
|
var reporter = _host.Services.GetServices<IHostedService>()
|
|
.OfType<LocalDbReplicationStatusReporter>()
|
|
.Single();
|
|
var collector = _host.Services.GetRequiredService<ISiteHealthCollector>();
|
|
|
|
reporter.Probe();
|
|
var report = collector.CollectReport("TestSiteB");
|
|
|
|
Assert.Null(report.LocalDbReplicationConnected);
|
|
Assert.Null(report.LocalDbOplogBacklog);
|
|
}
|
|
|
|
[Fact]
|
|
public void UnreplicatedSite_StillRegistersTheHealthBridge()
|
|
{
|
|
// The fix is a GATE on what the reporter publishes, not a skipped registration.
|
|
// Keeping the hosted service present means turning replication on is a config
|
|
// change rather than a different service graph, and it keeps one code path for
|
|
// both postures.
|
|
var hosted = _host.Services.GetServices<IHostedService>();
|
|
|
|
Assert.Contains(hosted, h => h is LocalDbReplicationStatusReporter);
|
|
}
|
|
|
|
[Fact]
|
|
public void UnreplicatedSite_ReplicationEngine_StillResolvesAndIdles()
|
|
{
|
|
// Establishes the premise the row-80 fix rests on: the engine really does answer
|
|
// false/0 here. If this ever starts returning null on its own, the gate becomes
|
|
// redundant rather than wrong — but the test above still pins the observable
|
|
// contract either way.
|
|
var status = _host.Services.GetRequiredService<ISyncStatus>();
|
|
|
|
Assert.False(status.Connected);
|
|
Assert.Equal(0L, status.OplogBacklog);
|
|
}
|
|
}
|