feat(localdb): fail-closed auth on the sync endpoint + replication health signal

Tasks 8 and 9 of the LocalDb Phase 1 adoption plan.

Task 8 - LocalDbSyncAuthInterceptor. The replication library's LocalDbSyncService
verifies nothing; inbound auth is explicitly the host's job. Without this,
anything able to reach a site node's gRPC port could stream arbitrary rows into
the consolidated site database - including OperationTracking, which central
reconciles from.

Scoped strictly to /localdb_sync.v1.LocalDbSync/; SiteStream shares the same
AddGrpc pipeline and passes through untouched. Fail-closed: with no
LocalDb:Replication:ApiKey configured NO sync stream is accepted, authenticated
or not. That is deliberate - "no key" is the default every site node ships with,
so treating it as "no auth required" would expose the endpoint on precisely the
most common configuration. Comparison is FixedTimeEquals over UTF-8 bytes.

All four server handler shapes are gated, not just unary: the sync RPC is a
bidirectional stream, so gating only the unary path would leave the real endpoint
open while every unary test still passed. There is a test for that.

Deviation from the plan: it specified Grpc.Core.Testing for the fake
ServerCallContext. That type ships in the retired native Grpc.Core package and
does not exist on the grpc-dotnet stack this solution uses; a minimal
FakeServerCallContext in the test file was the better trade than adding a dead
dependency.

Task 9 - ISyncStatus onto the site health report as LocalDbReplicationConnected
and LocalDbOplogBacklog, via a delegate-seam hosted service following the
AddSiteEventLogHealthMetricsBridge precedent (HealthMonitoring takes no reference
on the replication library). Both are additive init properties, so the
Akka-remoted SiteHealthReport constructor signature is untouched.

Both fields are nullable and the distinction is load-bearing:
  - null = the reporter has not run / replication is not wired ("no data");
  - false/0 = a real reading. On a node with no peer that IS the healthy
    default-OFF state, not an outage.
OplogBacklog is passed through nullable end-to-end because ISyncStatus returns
null when the poll fails - flattening it to 0 would report a replication pair
that cannot read its own oplog as perfectly healthy. The collector stores both
values as one tuple and CollectReport reads it once, so a torn read cannot pair a
fresh Connected with a stale backlog.

Verified: build 0 warnings; Host 307/307 (8 interceptor + 3 health tests new),
HealthMonitoring 97/97, Commons 684/684.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts

src/ZB.MOM.WW.ScadaBridge.Host/LocalDbSyncAuthInterceptor.cs

@@ -0,0 +1,158 @@
+using System.Security.Cryptography;
+using System.Text;
+using Grpc.Core;
+using Grpc.Core.Interceptors;
+using Microsoft.Extensions.Logging;
+using Microsoft.Extensions.Options;
+using ZB.MOM.WW.LocalDb.Replication;
+
+namespace ZB.MOM.WW.ScadaBridge.Host;
+
+/// <summary>
+/// Gates the LocalDb passive sync endpoint. The replication library deliberately leaves
+/// inbound authentication to the host — its <c>LocalDbSyncService</c> verifies nothing —
+/// so without this interceptor anything that can reach the site node's gRPC port could
+/// stream arbitrary rows into the consolidated site database.
+/// </summary>
+/// <remarks>
+/// <para>
+/// <b>Scoped by method path.</b> Only calls under
+/// <c>/localdb_sync.v1.LocalDbSync/</c> are gated; every other method — notably the
+/// existing <c>SiteStream</c> service sharing this listener — passes through untouched.
+/// </para>
+/// <para>
+/// <b>Fail-closed.</b> With no <c>LocalDb:Replication:ApiKey</c> configured, NO sync
+/// stream is accepted, authenticated or not. That is the deliberate choice: the
+/// alternative — treating "no key" as "no auth required" — would silently expose the
+/// endpoint on exactly the default configuration every site node ships with. An operator
+/// enabling replication must set the same key on both nodes, which is already required
+/// for the initiator to dial out (<c>SyncBackgroundService</c> sends
+/// <c>Authorization: Bearer &lt;key&gt;</c>).
+/// </para>
+/// <para>
+/// Comparison is <see cref="CryptographicOperations.FixedTimeEquals"/> over UTF-8 bytes,
+/// so a wrong key cannot be recovered byte-by-byte from response timing. Length differences
+/// are unavoidably observable and are not sensitive.
+/// </para>
+/// </remarks>
+public sealed class LocalDbSyncAuthInterceptor : Interceptor
+{
+    private const string ServicePrefix = "/localdb_sync.v1.LocalDbSync/";
+    private const string AuthorizationHeader = "authorization";
+    private const string BearerPrefix = "Bearer ";
+
+    private readonly IOptions<ReplicationOptions> _options;
+    private readonly ILogger<LocalDbSyncAuthInterceptor> _logger;
+
+    /// <summary>Creates the interceptor.</summary>
+    /// <param name="options">Replication options; <c>ApiKey</c> is the expected bearer token.</param>
+    /// <param name="logger">Logger for denial diagnostics.</param>
+    public LocalDbSyncAuthInterceptor(
+        IOptions<ReplicationOptions> options,
+        ILogger<LocalDbSyncAuthInterceptor> logger)
+    {
+        ArgumentNullException.ThrowIfNull(options);
+        ArgumentNullException.ThrowIfNull(logger);
+
+        _options = options;
+        _logger = logger;
+    }
+
+    /// <inheritdoc />
+    public override Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
+        TRequest request,
+        ServerCallContext context,
+        UnaryServerMethod<TRequest, TResponse> continuation)
+    {
+        Authorize(context);
+        return continuation(request, context);
+    }
+
+    /// <inheritdoc />
+    public override Task DuplexStreamingServerHandler<TRequest, TResponse>(
+        IAsyncStreamReader<TRequest> requestStream,
+        IServerStreamWriter<TResponse> responseStream,
+        ServerCallContext context,
+        DuplexStreamingServerMethod<TRequest, TResponse> continuation)
+    {
+        Authorize(context);
+        return continuation(requestStream, responseStream, context);
+    }
+
+    /// <inheritdoc />
+    public override Task<TResponse> ClientStreamingServerHandler<TRequest, TResponse>(
+        IAsyncStreamReader<TRequest> requestStream,
+        ServerCallContext context,
+        ClientStreamingServerMethod<TRequest, TResponse> continuation)
+    {
+        Authorize(context);
+        return continuation(requestStream, context);
+    }
+
+    /// <inheritdoc />
+    public override Task ServerStreamingServerHandler<TRequest, TResponse>(
+        TRequest request,
+        IServerStreamWriter<TResponse> responseStream,
+        ServerCallContext context,
+        ServerStreamingServerMethod<TRequest, TResponse> continuation)
+    {
+        Authorize(context);
+        return continuation(request, responseStream, context);
+    }
+
+    /// <summary>
+    /// Throws <see cref="RpcException"/> with <see cref="StatusCode.PermissionDenied"/> if
+    /// this is a sync call that does not carry the configured bearer token. Non-sync calls
+    /// return immediately.
+    /// </summary>
+    private void Authorize(ServerCallContext context)
+    {
+        if (!context.Method.StartsWith(ServicePrefix, StringComparison.Ordinal))
+            return;
+
+        var expected = _options.Value.ApiKey;
+        if (string.IsNullOrEmpty(expected))
+        {
+            _logger.LogWarning(
+                "Rejected a LocalDb sync call to {Method}: no LocalDb:Replication:ApiKey is configured, " +
+                "so the passive sync endpoint is closed. Configure the same key on both nodes of the pair.",
+                context.Method);
+            throw new RpcException(new Status(
+                StatusCode.PermissionDenied,
+                "LocalDb sync is not accepting connections: no API key is configured on this node."));
+        }
+
+        var presented = ExtractBearerToken(context.RequestHeaders);
+        if (presented is null || !FixedTimeEquals(presented, expected))
+        {
+            _logger.LogWarning(
+                "Rejected a LocalDb sync call to {Method}: {Reason}.",
+                context.Method,
+                presented is null ? "no bearer token presented" : "bearer token did not match");
+            throw new RpcException(new Status(
+                StatusCode.PermissionDenied,
+                "LocalDb sync authentication failed."));
+        }
+    }
+
+    private static string? ExtractBearerToken(Metadata headers)
+    {
+        // gRPC lowercases header keys on the wire; compare case-insensitively anyway so a
+        // hand-built Metadata in a test behaves the same as a real request.
+        foreach (var entry in headers)
+        {
+            if (!string.Equals(entry.Key, AuthorizationHeader, StringComparison.OrdinalIgnoreCase))
+                continue;
+
+            var value = entry.Value;
+            if (value is not null && value.StartsWith(BearerPrefix, StringComparison.OrdinalIgnoreCase))
+                return value[BearerPrefix.Length..];
+        }
+
+        return null;
+    }
+
+    private static bool FixedTimeEquals(string presented, string expected)
+        => CryptographicOperations.FixedTimeEquals(
+            Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expected));
+}

src/ZB.MOM.WW.ScadaBridge.Host/Program.cs

@@ -514,7 +514,11 @@ try
         });
 
         // gRPC server registration
-        builder.Services.AddGrpc();
+        // The interceptor gates ONLY /localdb_sync.v1.LocalDbSync/ — SiteStream calls on
+        // this same pipeline pass through untouched. It is fail-closed: with no
+        // LocalDb:Replication:ApiKey configured, no sync stream is accepted at all.
+        builder.Services.AddGrpc(options =>
+            options.Interceptors.Add<LocalDbSyncAuthInterceptor>());
         builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer>();
 
         // Existing site service registrations (this is also where LocalDb and its
@@ -540,9 +544,8 @@ try
         // The passive half of LocalDb replication: the peer node dials THIS endpoint.
         // It shares the Kestrel h2c listener the site gRPC server already uses, so no
         // listener or port changes are needed. Mapping it is harmless with no peer
-        // configured — nothing dials it. Inbound auth is the host's job and lands in
-        // Task 8; until then this endpoint is unauthenticated, which is why
-        // replication ships default-OFF.
+        // configured — nothing dials it, and LocalDbSyncAuthInterceptor (registered on
+        // AddGrpc above) rejects anything that tries until an ApiKey is configured.
         app.MapZbLocalDbSync();
 
         // Site-shutdown ordering. ApplicationStopping

src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs

@@ -100,6 +100,16 @@ public static class SiteServiceRegistration
         services.AddSiteEventLogHealthMetricsBridge(
             sp => () => sp.GetRequiredService<ISiteEventLogger>().FailedWriteCount);
 
+        // LocalDb replication — bridge ISyncStatus onto the site health report. Registered
+        // unconditionally alongside the engine: on a node with no peer this reports
+        // Connected=false with a real 0 backlog, which is the healthy default-OFF state.
+        // OplogBacklog is passed through NULLABLE on purpose — null means the poll failed
+        // or no provider is wired, and flattening it to 0 would render a replication pair
+        // that cannot read its own oplog as perfectly healthy.
+        services.AddLocalDbReplicationHealthBridge(
+            sp => () => sp.GetRequiredService<ISyncStatus>().Connected,
+            sp => () => sp.GetRequiredService<ISyncStatus>().OplogBacklog);
+
         // Audit Log — site-side hot-path writer + telemetry collaborators.
         // The SiteAuditTelemetryActor itself is registered by AkkaHostedService
         // in the site-role block; this call wires every DI dependency it (and