feat(localdb): fail-closed auth on the sync endpoint + replication health signal

Tasks 8 and 9 of the LocalDb Phase 1 adoption plan.

Task 8 - LocalDbSyncAuthInterceptor. The replication library's LocalDbSyncService
verifies nothing; inbound auth is explicitly the host's job. Without this,
anything able to reach a site node's gRPC port could stream arbitrary rows into
the consolidated site database - including OperationTracking, which central
reconciles from.

Scoped strictly to /localdb_sync.v1.LocalDbSync/; SiteStream shares the same
AddGrpc pipeline and passes through untouched. Fail-closed: with no
LocalDb:Replication:ApiKey configured NO sync stream is accepted, authenticated
or not. That is deliberate - "no key" is the default every site node ships with,
so treating it as "no auth required" would expose the endpoint on precisely the
most common configuration. Comparison is FixedTimeEquals over UTF-8 bytes.

All four server handler shapes are gated, not just unary: the sync RPC is a
bidirectional stream, so gating only the unary path would leave the real endpoint
open while every unary test still passed. There is a test for that.

Deviation from the plan: it specified Grpc.Core.Testing for the fake
ServerCallContext. That type ships in the retired native Grpc.Core package and
does not exist on the grpc-dotnet stack this solution uses; a minimal
FakeServerCallContext in the test file was the better trade than adding a dead
dependency.

Task 9 - ISyncStatus onto the site health report as LocalDbReplicationConnected
and LocalDbOplogBacklog, via a delegate-seam hosted service following the
AddSiteEventLogHealthMetricsBridge precedent (HealthMonitoring takes no reference
on the replication library). Both are additive init properties, so the
Akka-remoted SiteHealthReport constructor signature is untouched.

Both fields are nullable and the distinction is load-bearing:
  - null = the reporter has not run / replication is not wired ("no data");
  - false/0 = a real reading. On a node with no peer that IS the healthy
    default-OFF state, not an outage.
OplogBacklog is passed through nullable end-to-end because ISyncStatus returns
null when the poll fails - flattening it to 0 would report a replication pair
that cannot read its own oplog as perfectly healthy. The collector stores both
values as one tuple and CollectReport reads it once, so a torn read cannot pair a
fresh Connected with a stale backlog.

Verified: build 0 warnings; Host 307/307 (8 interceptor + 3 health tests new),
HealthMonitoring 97/97, Commons 684/684.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
2026-07-19 09:29:22 -04:00
parent e62b076f2e
commit 59c695191c
10 changed files with 637 additions and 5 deletions
@@ -73,6 +73,37 @@ public record SiteHealthReport(
/// indicates a stuck/blocked script holding a dedicated thread.
/// </summary>
public double? ScriptOldestBusyAgeSeconds { get; init; }
// LocalDb 2-node replication of the consolidated site database (Phase 1).
// Additive init properties for the same reason as the scheduler gauges above:
// the positional constructor stays untouched. Refreshed on the site by
// LocalDbReplicationStatusReporter.
/// <summary>
/// Whether a replication sync session is currently running with the peer site node,
/// or <see langword="null"/> when replication is not wired on this node.
/// </summary>
/// <remarks>
/// Nullable on purpose, and <see langword="false"/> is NOT the same as null.
/// Replication ships default-OFF, so a site node with no peer configured reports
/// <see langword="false"/> — that is a healthy, expected state, not an outage. Only
/// a node that was configured with a peer and is reporting <see langword="false"/>
/// is degraded, and distinguishing those two cases is the operator's job, not this
/// field's.
/// </remarks>
public bool? LocalDbReplicationConnected { get; init; }
/// <summary>
/// Unacked replication oplog entries waiting to reach the peer, or
/// <see langword="null"/> when unknown.
/// </summary>
/// <remarks>
/// <b>Null means unknown, never zero.</b> The underlying provider returns null when
/// no backlog source is wired or the poll failed, and that must survive to the
/// operator: a failed read rendered as "0 backlog" would report a broken replication
/// pair as perfectly healthy. Collapsing null to 0 anywhere on this path is a bug.
/// </remarks>
public long? LocalDbOplogBacklog { get; init; }
}
/// <summary>