feat(secrets): Grpc hub client — pull-only sweep service + options + DI

Client half of the central-hosted secrets hub: a site node follows central and
never pushes.

- `ISecretsHubReader` + `GrpcSecretsHubClient` — the read half of the hub as a
  follower sees it (manifest + bulk fetch, nothing else). Bearer credential and a
  per-call deadline are attached per call; responses convert through
  `WireToDomain`. Faults are deliberately not caught here, so an unreachable hub
  can never be mistaken for an empty one.
- `WireToDomain` — strict, per-row proto -> domain parse mirroring the Akka
  replicator's `SecretRowDto`: a name the domain would reject, a content type
  naming no enum member, or a non-round-trip timestamp is refused at the boundary.
  Explicit proto3 presence keeps null distinguishable from empty.
- `GrpcSecretSyncService` — `SqlServerSecretSyncService` with the push half
  removed. One `ReconcileAsync` per tick (pull-only by the reconciler's own
  contract, and batched, so a cold-start pull set cannot exceed gRPC's 4 MB cap),
  never throws out of `ExecuteAsync`, warns once and retries on the next interval.
  The local store is left completely undecorated — the sweep is its only hub-ward
  writer.
- `SecretsGrpcHubClientOptions` — fail-closed `Validate()` naming the exact
  configuration key, including the section a caller actually bound. The bearer
  token is never echoed by any message.
- `AddZbSecretsGrpcHubClient` — binds, eagerly validates, and registers the
  channel under a service KEY so it cannot lose a `TryAdd` race to an
  application's own `GrpcChannel` and dial the wrong endpoint with the hub
  credential attached.

Tests 52/52 (red-first), including a source-scan guard pinning that no client file
names `ComputePushSet`, `ReadLocalAsync`, or `ApplyReplicatedAsync`, and a
reflection pin that the reader seam exposes exactly two read operations.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
This commit is contained in:
Joseph Doherty
2026-08-07 06:12:03 -04:00
parent cc95590196
commit c5ea3a7b59
8 changed files with 1429 additions and 0 deletions
@@ -0,0 +1,148 @@
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.Secrets.Abstractions;
using ZB.MOM.WW.Secrets.Replication;
namespace ZB.MOM.WW.Secrets.Replicator.Grpc;
/// <summary>
/// Keeps a site node's local secret store converged with the central-hosted hub by running a
/// <b>pull-only</b> anti-entropy sweep on an interval.
/// </summary>
/// <remarks>
/// <para>
/// This is the SQL-Server package's sync service with one half removed, and the removal is the
/// design. In this topology every update originates at central and a site node is a strict follower,
/// so there is no push set to compute and no local write to propagate outward. The wire contract has
/// no write RPC to push with either — the narrow contract IS the enforcement, and this service is
/// kept honest to match it so the invariant reads the same from both ends.
/// </para>
/// <para>
/// The local store is left completely untouched as a write target for ordinary local use: nothing
/// here decorates it, so local reads and any local writes behave exactly as they did before
/// replication was switched on. This sweep is the only writer that carries hub state into it.
/// </para>
/// <para>
/// The sweep never throws out of <see cref="ExecuteAsync"/>: a hub outage must degrade the node to
/// "serving its last-known-good local secrets" rather than crash the host. Failures are logged once
/// and retried on the next tick.
/// </para>
/// </remarks>
internal sealed partial class GrpcSecretSyncService : BackgroundService
{
private readonly SecretReplicationReconciler _reconciler;
private readonly ISecretsHubReader _hub;
private readonly SecretsGrpcHubClientOptions _options;
private readonly ILogger<GrpcSecretSyncService> _logger;
private readonly TimeProvider _timeProvider;
/// <summary>Creates the sync service.</summary>
/// <param name="local">The node's local store — the only thing this service writes to.</param>
/// <param name="hub">The central hub, read-only.</param>
/// <param name="cacheInvalidator">Resolver-cache seam, evicted for every applied row.</param>
/// <param name="options">Hub-client options.</param>
/// <param name="logger">Logger for sweep outcomes and failures.</param>
/// <param name="timeProvider">Clock, injectable so tests need not wait a real interval.</param>
public GrpcSecretSyncService(
ISecretStore local,
ISecretsHubReader hub,
ISecretCacheInvalidator? cacheInvalidator,
IOptions<SecretsGrpcHubClientOptions> options,
ILogger<GrpcSecretSyncService> logger,
TimeProvider? timeProvider = null)
{
ArgumentNullException.ThrowIfNull(options);
ArgumentNullException.ThrowIfNull(local);
_hub = hub ?? throw new ArgumentNullException(nameof(hub));
_options = options.Value;
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_timeProvider = timeProvider ?? TimeProvider.System;
_reconciler = new SecretReplicationReconciler(
local,
cacheInvalidator,
onRowFailed: (name, ex) => LogRowDiscarded(ex, name.Value));
}
/// <inheritdoc />
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
if (_options.SyncOnStartup)
{
await SweepSafelyAsync(stoppingToken).ConfigureAwait(false);
}
using PeriodicTimer timer = new(_options.SyncInterval, _timeProvider);
while (await timer.WaitForNextTickAsync(stoppingToken).ConfigureAwait(false))
{
await SweepSafelyAsync(stoppingToken).ConfigureAwait(false);
}
}
/// <summary>
/// Runs one pull-only reconcile against the hub.
/// </summary>
/// <param name="ct">A token to cancel the sweep.</param>
/// <returns>How many rows were pulled from the hub.</returns>
internal async Task<int> SweepAsync(CancellationToken ct)
{
IReadOnlyList<SecretManifestEntry> hubManifest =
await _hub.GetManifestAsync(ct).ConfigureAwait(false);
// The reconciler's pull path is exactly the required algorithm — diff the hub's manifest
// against the local one, fetch what the hub holds newer, apply last-writer-wins, evict the
// resolver cache per applied row — and it is documented as "a pull, never a push". Calling
// it rather than re-deriving the diff here also keeps the batching: a cold-starting node's
// pull set is central's ENTIRE inventory, and an unchunked GetSecrets would exceed gRPC's
// 4 MB default message cap and then fail identically on every interval forever.
return await _reconciler
.ReconcileAsync(hubManifest, (names, token) => _hub.GetManyAsync(names, token), ct)
.ConfigureAwait(false);
}
private async Task SweepSafelyAsync(CancellationToken ct)
{
try
{
int pulled = await SweepAsync(ct).ConfigureAwait(false);
if (pulled > 0)
{
LogSweepConverged(pulled);
}
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
// Host shutdown — not a failure.
}
catch (Exception ex)
{
// Deliberately catch-all. Everything reachable from here — an unreachable hub, a
// rejected credential, a deadline, a row the hub sent that this node's parse refuses —
// has the same correct response: keep serving the local store and try again. Letting any
// of it escape would fault the BackgroundService and take the host down over a transient
// central outage, which is precisely the availability the pull-only topology exists for.
LogSweepFailed(ex);
}
}
[LoggerMessage(
EventId = 1,
Level = LogLevel.Information,
Message = "Secret hub sync converged: pulled {Pulled} row(s) from the central hub.")]
private partial void LogSweepConverged(int pulled);
[LoggerMessage(
EventId = 2,
Level = LogLevel.Warning,
Message = "Secret hub sync failed; the node continues serving its local store and will retry on the next interval.")]
private partial void LogSweepFailed(Exception exception);
[LoggerMessage(
EventId = 3,
Level = LogLevel.Warning,
Message = "Discarding replicated secret row {SecretName} that could not be applied locally; the rest of the batch continues.")]
private partial void LogRowDiscarded(Exception exception, string secretName);
}