fix(health): rows 80+73 — unconfigured replication reports null, reconciler presence surfaced

ROW 80 (observability correctness). An unreplicated site published LocalDbReplicationConnected=false / LocalDbOplogBacklog=0, contradicting the nullable semantics every doc and the proto already state ("null means no reading, NOT disconnected with an empty backlog"). Root cause: the replication engine is registered unconditionally and idles without a peer, so ISyncStatus answers false/0 on a node that was never meant to replicate, and LocalDbReplicationStatusReporter passed that straight through. Two consequences: an operator read connected:false on a healthy site as a fault, and central's null branches — row 33's conditional KPI emission and sustained-badge anchor exclusion — never received the null they exist for, so rig site-b/site-c emitted a permanently misleading localDbReplicationConnected=0 series. Fix mirrors the SiteEventLogReplicationCheck precedent: the Host evaluates SiteLocalDbSetup.ReplicationIsConfigured ONCE and passes it to AddLocalDbReplicationHealthBridge as a third provider; unconfigured short-circuits the probe to (null, null). HealthMonitoring never reads LocalDb config itself, and the same captured value now also feeds SiteEventLogReplicationCheck so the two surfaces cannot disagree. ISiteHealthCollector.SetLocalDbReplicationStatus widens to bool? — the non-nullable bool was the type-level root cause (null was inexpressible). false now means exactly one thing: configured, peer session down = a real fault. Wire/central/UI needed no change: the proto already carries BoolValue/Int64Value wrappers and the aggregator + KPI source already branch on null. This makes code match doc, so no doc claim was weakened. Watch the trap that caused this: configured-ness is PeerAddress OR ApiKey (only the dialling half sets a peer address). The pre-existing SiteLocalDbWiringTests fixture sets ApiKey and IS a replicating node, so its false/0 assertion is correct and is kept, with a comment saying so; the new UnreplicatedSiteLocalDbHealthTests fixture is the genuinely-unreplicated one. ROW 73 (silent loss of self-heal). AkkaHostedService skips SiteReconciliationActor when no IDeploymentConfigFetcher is resolvable, and that skip was a lone boot Warning — a node with no self-heal path at all went on reporting itself perfectly healthy forever. Adds tri-state SiteHealthReport.StartupReconcilerActive (null = not reported, i.e. the central self-report or a pre-row-73 site build; true = running; false = booted without it), latched once at boot by the Host in both branches. Additive proto field 29 (BoolValue, vendored C# regenerated via docker/regen-proto.sh centralcontrol; csproj restored, verified empty diff). The skip branch also raises its log to Error — no configuration makes the fetcher legitimately absent on a site node — and writes a best-effort deployment/Error site event so the evidence outlives the boot log. Central treatment: a "No self-heal" WARNING badge on the site tile, not a failure. Config still reaches the node via LocalDb replication while it is up; what it lost is recovery from a deploy that landed while it was down — degraded, not offline. Only the explicit false is badged (badging null would light up the whole fleet the moment central is upgraded ahead of the sites), and offline sites are skipped, matching the replication-backlog badge. Fail-before evidence: with the row-80 gate behaviourally reverted, UnreplicatedSite_HealthReport_ReportsNullForBothLocalDbFields fails "Expected: null / Actual: False" and 3 of the 8 new reporter unit tests fail. Tests: HealthMonitoring 128, Host 518, CentralUI 1051, Communication 717, Commons 715, SiteEventLogging 81 — all green.
2026-08-15 21:40:48 -04:00
parent 3dc38fb674
commit a2f0086fa1
18 changed files with 921 additions and 67 deletions
@@ -1029,6 +1029,14 @@ akka {{
        // singleton, which starts the Instance Actor if the boot-time load missed it. Going
        // through the PROXY (not a local ActorOf) keeps Instance Actors under the singleton
        // regardless of which node's reconcile pass found the gap.
+        //
+        // Whichever branch is taken is LATCHED onto the site health report as
+        // StartupReconcilerActive (deferred-work register row 73). Before that, the skip
+        // was a single boot Warning and nothing else: a node that silently lost its entire
+        // self-heal path went on reporting itself perfectly healthy forever, and the only
+        // evidence was one line scrolled off the top of a log. Central badges the false
+        // case at Warning rather than failing the site — replication is still the primary
+        // path, so the node is degraded, not down.
        if (deploymentConfigFetcher != null)
        {
            var reconcileLogger = _serviceProvider.GetRequiredService<ILoggerFactory>()
@@ -1045,15 +1053,50 @@ akka {{
                    null,
                    dmProxy)),
                "site-reconciliation");
+            siteHealthCollector?.SetStartupReconcilerActive(true);
            _logger.LogInformation(
                "SiteReconciliationActor created (per-node startup self-heal) for site {SiteId} node {Node}",
                _nodeOptions.SiteId, _nodeOptions.NodeName);
        }
        else
        {
-            _logger.LogWarning(
-                "No IDeploymentConfigFetcher available; SiteReconciliationActor not created — "
-                + "startup self-heal disabled (replication remains the primary path)");
+            siteHealthCollector?.SetStartupReconcilerActive(false);
+
+            // Raised from Warning to Error (register row 73). This is a misconfiguration,
+            // not a degraded-but-expected state: no configuration makes the fetcher
+            // legitimately absent on a site node, and the consequence is that neither the
+            // file heal nor the row-67 runtime heal will ever run on this node.
+            _logger.LogError(
+                "No IDeploymentConfigFetcher available; SiteReconciliationActor not created for "
+                + "site {SiteId} node {Node} — startup self-heal is DISABLED (replication "
+                + "remains the primary path). Surfaced centrally as StartupReconcilerActive=false.",
+                _nodeOptions.SiteId, _nodeOptions.NodeName);
+
+            // Also record it where an operator reviewing the site after the fact will find
+            // it — the boot log line is gone by then, but the site event log is queryable
+            // from the Central UI. Best-effort: a failed event write must never keep the
+            // node from finishing startup, which is why this is fire-and-forget with an
+            // observed continuation rather than an await.
+            var bootEventLogger = _serviceProvider
+                .GetService<ZB.MOM.WW.ScadaBridge.SiteEventLogging.ISiteEventLogger>();
+            _ = bootEventLogger?.LogEventAsync(
+                    "deployment",
+                    "Error",
+                    null,
+                    "AkkaHostedService",
+                    "Startup self-heal disabled: SiteReconciliationActor was not created "
+                    + "because no IDeploymentConfigFetcher is registered.",
+                    $"Node {_nodeOptions.NodeName} of site {_nodeOptions.SiteId} cannot "
+                    + "reconcile its deployed inventory at boot. Config deploys still reach "
+                    + "this node via LocalDb replication when it is up; a deploy that lands "
+                    + "while it is DOWN will not be recovered on restart.")
+                .ContinueWith(
+                    t => _logger.LogWarning(
+                        t.Exception,
+                        "Failed to write the startup-self-heal-disabled site event."),
+                    CancellationToken.None,
+                    TaskContinuationOptions.OnlyOnFaulted,
+                    TaskScheduler.Default);
        }

        // Audit Log — site-side telemetry actor that drains the SQLite
@@ -137,14 +137,27 @@ public static class SiteServiceRegistration
            sp => () => sp.GetRequiredService<ISiteEventLogger>().FailedWriteCount);

        // LocalDb replication — bridge ISyncStatus onto the site health report. Registered
-        // unconditionally alongside the engine: on a node with no peer this reports
-        // Connected=false with a real 0 backlog, which is the healthy default-OFF state.
-        // OplogBacklog is passed through NULLABLE on purpose — null means the poll failed
-        // or no provider is wired, and flattening it to 0 would render a replication pair
-        // that cannot read its own oplog as perfectly healthy.
+        // unconditionally alongside the engine, but what it PUBLISHES is gated on whether
+        // this node is configured to replicate at all (register row 80).
+        //
+        // The engine is inert without a peer, so ISyncStatus answers Connected=false with a
+        // real 0 backlog on a site that was never meant to replicate. Publishing that pair
+        // contradicts the documented nullable semantics ("null means no reading, NOT
+        // disconnected with an empty backlog"): a dashboard reader sees connected:false on a
+        // healthy unreplicated site and reads a fault, and central's null-branch machinery
+        // (conditional KPI emission, sustained-badge anchor exclusion) never receives the
+        // null it was built for — so rig site-b/site-c emitted a misleading
+        // localDbReplicationConnected=0 series. Unconfigured now publishes null/null.
+        //
+        // Delegating to SiteLocalDbSetup.ReplicationIsConfigured keeps the
+        // PeerAddress-OR-ApiKey rule in ONE place and mirrors SiteEventLogReplicationCheck
+        // below — HealthMonitoring must not learn to read LocalDb configuration itself.
+        // Read once here (configuration is fixed for the process lifetime) and captured.
+        var localDbReplicationConfigured = SiteLocalDbSetup.ReplicationIsConfigured(config);
        services.AddLocalDbReplicationHealthBridge(
            sp => () => sp.GetRequiredService<ISyncStatus>().Connected,
-            sp => () => sp.GetRequiredService<ISyncStatus>().OplogBacklog);
+            sp => () => sp.GetRequiredService<ISyncStatus>().OplogBacklog,
+            _ => () => localDbReplicationConfigured);

        // Audit Log — site-side hot-path writer + telemetry collaborators.
        // The SiteAuditTelemetryActor itself is registered by AkkaHostedService
@@ -217,9 +230,9 @@ public static class SiteServiceRegistration
        // backlog genuinely rises at purge time and reads like a replication fault to anyone who
        // does not know a purge just ran. Delegating to SiteLocalDbSetup.ReplicationIsConfigured
        // keeps the PeerAddress-OR-ApiKey rule in one place — SiteEventLogging must not learn to
-        // read LocalDb configuration itself.
-        var replicationConfigured = SiteLocalDbSetup.ReplicationIsConfigured(config);
-        SiteEventLogReplicationCheck replicationCheck = () => replicationConfigured;
+        // read LocalDb configuration itself. Shares the single read taken for the LocalDb
+        // replication health bridge above, so the two surfaces can never disagree.
+        SiteEventLogReplicationCheck replicationCheck = () => localDbReplicationConfigured;
        services.AddSingleton(replicationCheck);

        // Health checks — the shared ZB.MOM.WW.Health probes, mapped by MapZbHealth on the site's