fix(localdb): allowlist the replication meter + record the Task 12 live gate
Task 12 (live gate on the docker rig) found one real defect, which this fixes.
THE DEFECT. The plan asserted LocalDb metrics "need no work - the
ZB.MOM.WW.LocalDb.Replication meter flows through the existing AddZbTelemetry
Prometheus export". It does not. ZbTelemetryOptions.Meters is an ALLOWLIST: an
unlisted meter's instruments are never observed, with no error, no warning, and
no missing-metric signal anywhere. The rig's /metrics scrape returned 301 lines
and zero localdb_* series. Only looking for them found it.
Fixed by hoisting the allowlist to SiteServiceRegistration.ObservedMeters and
adding LocalDbMetrics.MeterName. The pin test asserts on that constant rather
than on OTel's observed set, because AddZbTelemetry applies its options to a
local instance and never registers IOptions - there is nothing resolvable to
interrogate. The end-to-end proof is the live scrape below; the test exists to
stop the entry being dropped again. Called out as such in the test.
LIVE GATE EVIDENCE (docker rig, 8 nodes, rebuilt via docker/deploy.sh):
1. Consolidated DB created on every site node at /app/data/site-localdb.db -
inside the mounted volume, unlike the legacy CWD-relative databases.
2. Replication proven REAL, not two independent local writes. Both site-a nodes
hold the identical row and, decisively, the identical originating node_id and
HLC in __localdb_row_version:
site_events|{"id":"8b06b37c..."}|116946936661147648|65b00319-...|0
__localdb_peer_state shows a completed handshake in both directions.
3. Convergence: 5/5 events on both nodes, __localdb_row_version byte-identical
across the pair, oplog drained to 0, dead_letter 0 on both.
4. SITE failover drill. Note the repo's failover-drill.sh targets CENTRAL nodes
and does not exercise this claim, so the site-pair flip was run directly:
stopped site-a-a (the node holding the history); site-a-b survived with the
complete pre-flip history (3/3 events) plus its own takeover event. Restarted
site-a-a; it caught up on what it missed and both nodes reconverged to the
same 5 ids. This is the failure Phase 1 exists to prevent.
5. localdb_* now exported after the fix:
localdb_sync_reconnects_total{...} 1
localdb_oplog_depth{...} 0
6. DEFAULT-OFF pin, live and side-by-side: site-b (no Replication section) boots
identically, creates its consolidated DB, registers the engine - and stays
inert. Zero replication log lines, no reconnect counter at all, so it never
dialled anything.
One transient WRN at startup on site-a-a ("Connection refused", reconnecting) is
node A dialling before node B's listener was up; it reconnected within a second.
Expected for a pair that starts together.
Verified: build 0 warnings; Host wiring tests 11/11.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
@@ -27,6 +27,24 @@ namespace ZB.MOM.WW.ScadaBridge.Host;
|
||||
/// </summary>
|
||||
public static class SiteServiceRegistration
|
||||
{
|
||||
/// <summary>
|
||||
/// Every <see cref="System.Diagnostics.Metrics.Meter"/> OpenTelemetry is told to observe.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <c>ZbTelemetryOptions.Meters</c> is an ALLOWLIST, and an unlisted meter's instruments
|
||||
/// are simply never observed — no error, no warning, no missing-metric signal anywhere.
|
||||
/// The LocalDb replication meter was silently absent from the rig's <c>/metrics</c>
|
||||
/// scrape until the Phase 1 live gate went looking for it (the plan had assumed it
|
||||
/// flowed through automatically). Hoisted to a named constant so the list is
|
||||
/// assertable; adding a meter anywhere in the product means adding it here.
|
||||
/// </remarks>
|
||||
public static readonly string[] ObservedMeters =
|
||||
[
|
||||
ScadaBridgeTelemetry.MeterName,
|
||||
// Emitted only on site nodes; harmless on central, which never records against it.
|
||||
LocalDbMetrics.MeterName,
|
||||
];
|
||||
|
||||
/// <summary>Registers all DI services required for the site role.</summary>
|
||||
/// <param name="services">The service collection to register into.</param>
|
||||
/// <param name="config">Application configuration for options binding.</param>
|
||||
@@ -262,7 +280,7 @@ public static class SiteServiceRegistration
|
||||
o.ServiceName = "scadabridge";
|
||||
o.SiteId = config["ScadaBridge:Node:SiteId"] ?? "central";
|
||||
o.NodeRole = config["ScadaBridge:Node:Role"];
|
||||
o.Meters = [ScadaBridgeTelemetry.MeterName];
|
||||
o.Meters = ObservedMeters;
|
||||
if (Enum.TryParse<ZbExporter>(config["ScadaBridge:Telemetry:Exporter"], ignoreCase: true, out var exporter))
|
||||
o.Exporter = exporter;
|
||||
var otlp = config["ScadaBridge:Telemetry:OtlpEndpoint"];
|
||||
|
||||
@@ -3,7 +3,9 @@ using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
|
||||
using ZB.MOM.WW.Telemetry;
|
||||
using ZB.MOM.WW.LocalDb;
|
||||
using ZB.MOM.WW.LocalDb.Replication;
|
||||
using ZB.MOM.WW.ScadaBridge.Host;
|
||||
@@ -196,6 +198,23 @@ public class SiteLocalDbWiringTests : IDisposable
|
||||
Assert.Null(report.LocalDbOplogBacklog);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Site_Telemetry_Allowlists_TheLocalDbReplicationMeter()
|
||||
{
|
||||
// ZbTelemetryOptions.Meters is an ALLOWLIST: an unlisted Meter's instruments are
|
||||
// never observed, with no error and no missing-metric warning anywhere. The
|
||||
// replication meter was silently absent from the rig's /metrics scrape until the
|
||||
// live gate looked for it, so pin the allowlist rather than trusting the next
|
||||
// reader to remember.
|
||||
//
|
||||
// This asserts on the allowlist constant, not on OTel's observed-meter set:
|
||||
// AddZbTelemetry applies its options to a local instance and never registers
|
||||
// IOptions, so there is nothing resolvable to interrogate. The end-to-end proof
|
||||
// that localdb_* actually reaches /metrics is the Task 12 live gate; this test
|
||||
// exists to stop the entry being dropped again.
|
||||
Assert.Contains(LocalDbMetrics.MeterName, SiteServiceRegistration.ObservedMeters);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Site_LocalDb_CreatesTheConfiguredFile()
|
||||
{
|
||||
|
||||
Reference in New Issue
Block a user