feat(localdb): fail-closed auth on the sync endpoint + replication health signal

Tasks 8 and 9 of the LocalDb Phase 1 adoption plan.

Task 8 - LocalDbSyncAuthInterceptor. The replication library's LocalDbSyncService
verifies nothing; inbound auth is explicitly the host's job. Without this,
anything able to reach a site node's gRPC port could stream arbitrary rows into
the consolidated site database - including OperationTracking, which central
reconciles from.

Scoped strictly to /localdb_sync.v1.LocalDbSync/; SiteStream shares the same
AddGrpc pipeline and passes through untouched. Fail-closed: with no
LocalDb:Replication:ApiKey configured NO sync stream is accepted, authenticated
or not. That is deliberate - "no key" is the default every site node ships with,
so treating it as "no auth required" would expose the endpoint on precisely the
most common configuration. Comparison is FixedTimeEquals over UTF-8 bytes.

All four server handler shapes are gated, not just unary: the sync RPC is a
bidirectional stream, so gating only the unary path would leave the real endpoint
open while every unary test still passed. There is a test for that.

Deviation from the plan: it specified Grpc.Core.Testing for the fake
ServerCallContext. That type ships in the retired native Grpc.Core package and
does not exist on the grpc-dotnet stack this solution uses; a minimal
FakeServerCallContext in the test file was the better trade than adding a dead
dependency.

Task 9 - ISyncStatus onto the site health report as LocalDbReplicationConnected
and LocalDbOplogBacklog, via a delegate-seam hosted service following the
AddSiteEventLogHealthMetricsBridge precedent (HealthMonitoring takes no reference
on the replication library). Both are additive init properties, so the
Akka-remoted SiteHealthReport constructor signature is untouched.

Both fields are nullable and the distinction is load-bearing:
  - null = the reporter has not run / replication is not wired ("no data");
  - false/0 = a real reading. On a node with no peer that IS the healthy
    default-OFF state, not an outage.
OplogBacklog is passed through nullable end-to-end because ISyncStatus returns
null when the poll fails - flattening it to 0 would report a replication pair
that cannot read its own oplog as perfectly healthy. The collector stores both
values as one tuple and CollectReport reads it once, so a torn read cannot pair a
fresh Connected with a stale backlog.

Verified: build 0 warnings; Host 307/307 (8 interceptor + 3 health tests new),
HealthMonitoring 97/97, Commons 684/684.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
2026-07-19 09:29:22 -04:00
parent e62b076f2e
commit 59c695191c
10 changed files with 637 additions and 5 deletions
@@ -18,6 +18,10 @@ public class SiteHealthCollector : ISiteHealthCollector
private int _auditRedactionFailures;
private volatile SiteAuditBacklogSnapshot? _siteAuditBacklog;
private long _siteEventLogWriteFailures;
// One volatile tuple rather than two independent fields: a torn read that paired a
// fresh Connected with a stale backlog would be indistinguishable from a real state.
// Null = the reporter has not yet run (or replication is not wired).
private volatile Tuple<bool, long?>? _localDbReplicationStatus;
private readonly ConcurrentDictionary<string, ConnectionHealth> _connectionStatuses = new();
private readonly ConcurrentDictionary<string, TagResolutionStatus> _tagResolutionCounts = new();
private readonly ConcurrentDictionary<string, string> _connectionEndpoints = new();
@@ -94,6 +98,12 @@ public class SiteHealthCollector : ISiteHealthCollector
Interlocked.Exchange(ref _siteEventLogWriteFailures, count);
}
/// <inheritdoc />
public void SetLocalDbReplicationStatus(bool connected, long? oplogBacklog)
{
_localDbReplicationStatus = Tuple.Create(connected, oplogBacklog);
}
/// <inheritdoc />
public void UpdateConnectionHealth(string connectionName, ConnectionHealth health)
{
@@ -219,6 +229,9 @@ public class SiteHealthCollector : ISiteHealthCollector
var deadLetters = Interlocked.Exchange(ref _deadLetterCount, 0);
var siteAuditWriteFailures = Interlocked.Exchange(ref _siteAuditWriteFailures, 0);
var auditRedactionFailures = Interlocked.Exchange(ref _auditRedactionFailures, 0);
// Single read of the volatile tuple — two reads could straddle a reporter tick and
// pair a fresh Connected with a stale backlog.
var localDbReplication = _localDbReplicationStatus;
// Snapshot current connection and tag resolution state
var connectionStatuses = new Dictionary<string, ConnectionHealth>(_connectionStatuses);
@@ -259,7 +272,12 @@ public class SiteHealthCollector : ISiteHealthCollector
{
ScriptQueueDepth = Interlocked.CompareExchange(ref _scriptQueueDepth, 0, 0),
ScriptBusyThreads = Interlocked.CompareExchange(ref _scriptBusyThreads, 0, 0),
ScriptOldestBusyAgeSeconds = ReadScriptOldestBusyAgeSeconds()
ScriptOldestBusyAgeSeconds = ReadScriptOldestBusyAgeSeconds(),
// Both fields come from the ONE snapshot read above. Null (the reporter has
// not run) leaves both report fields null — "no data", not "disconnected
// with an empty backlog".
LocalDbReplicationConnected = localDbReplication?.Item1,
LocalDbOplogBacklog = localDbReplication?.Item2
};
}
}