feat(secrets): cluster replication via SQL Server and Akka.NET (G-7, 0.2.0)

Secrets were per-node SQLite, so a secret written on one node was invisible to
the rest of a cluster. G-7's design resolved the "shared SQL store vs Akka
replicator" fork to build only the former; both are built here so the choice is
a deployment decision (availability vs partition tolerance) rather than a
library limitation.

Two new packages — ZB.MOM.WW.Secrets.Replicator.SqlServer (shared store, plus a
local-store-with-hub mode) and .Replicator.AkkaDotNet (peer-to-peer over
distributed pub/sub). Core gains ISecretsStoreMigrator, one shared
SecretLastWriterWins predicate so no two stores can disagree on a tie, the
transport-agnostic reconciler, and ReplicatingSecretStore — which closes a real
gap: nothing had ever called ISecretReplicator.PublishAsync, so the seam was
inert and local writes would not have propagated at all.

Verified 182 pass / 1 skip / 0 warnings, including 15 live tests against a real
SQL Server 2022 (the SQLite suite ported case-for-case, so any behavioural
divergence between the stores fails) and a 9-test in-process 2-node Akka
cluster over real remoting. A post-build review caught six defects, all fixed
and now covered: both replication modes could not resolve from the container
(no test had built one), an unbounded fetch that broke past SQL Server's
2100-parameter cap, a poison row that aborted the rest of its batch forever,
Enum.Parse on peer input that could restart the actor in a loop, null crypto
blobs crossing the trust boundary, and a silently dropped pull-read failure.

Packed at 0.2.0 and vulnerability-scanned clean; not yet published to the feed.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
2026-07-18 04:08:23 -04:00
parent e46060fada
commit dd0a846b64
55 changed files with 4848 additions and 51 deletions
@@ -0,0 +1,141 @@
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.DependencyInjection.Extensions;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.Secrets.Abstractions;
using ZB.MOM.WW.Secrets.Replication;
using ZB.MOM.WW.Secrets.DependencyInjection;
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer.DependencyInjection;
/// <summary>
/// Dependency-injection entry points for backing a cluster's secrets with SQL Server, in either of
/// the two supported topologies.
/// </summary>
/// <remarks>
/// <para>
/// <b>Both require the same KEK on every node.</b> The database stores ciphertext only, so a node
/// whose master key differs cannot decrypt rows another node wrote — it fails closed on resolve with
/// a <c>kek_id</c> mismatch, which reads like data corruption but is a deployment error. Mount the
/// same key file or supply the same key environment variable everywhere.
/// </para>
/// </remarks>
public static class SqlServerSecretsServiceCollectionExtensions
{
/// <summary>
/// <b>Shared-store mode.</b> Points every node's <see cref="ISecretStore"/> at one shared
/// SQL-Server database. There is exactly one copy of each row, so there is nothing to replicate
/// and nothing to reconcile — no split-brain, no anti-entropy lag, no clock skew.
/// </summary>
/// <remarks>
/// Prefer this unless a node must keep resolving secrets while <em>partitioned</em> from the
/// shared database. The trade is availability: a node cut off from the database falls back only
/// to the resolver's short in-memory cache. If that is unacceptable, use
/// <see cref="AddZbSecretsSqlServerReplication"/> (local store + hub) or the Akka package
/// (peer-to-peer, no shared database at all).
/// </remarks>
/// <param name="services">The service collection to add to.</param>
/// <param name="config">The application configuration.</param>
/// <param name="secretsSectionPath">Configuration section holding the core secrets options.</param>
/// <param name="sqlServerSectionPath">Configuration section holding the SQL-Server options.</param>
/// <returns>The same <paramref name="services"/> instance, for chaining.</returns>
public static IServiceCollection AddZbSecretsSqlServerStore(
this IServiceCollection services,
IConfiguration config,
string secretsSectionPath = "Secrets",
string sqlServerSectionPath = "Secrets:SqlServer")
{
RegisterSqlServerCore(services, config, sqlServerSectionPath);
// Registered BEFORE AddZbSecrets so its TryAdd calls find these already present and leave
// them alone — that is the whole mechanism by which the SQL-Server store displaces SQLite.
services.TryAddSingleton<ISecretStore>(sp => sp.GetRequiredService<SqlServerSecretStore>());
services.TryAddSingleton<ISecretsStoreMigrator>(sp =>
sp.GetRequiredService<SqlServerSecretsStoreMigrator>());
services.AddZbSecrets(config, secretsSectionPath);
return services;
}
/// <summary>
/// <b>Hub-replication mode.</b> Each node keeps its own local store (SQLite by default) and
/// stays converged with a shared SQL-Server hub: local writes publish to the hub immediately, and
/// a background sweep reconciles both directions on an interval.
/// </summary>
/// <remarks>
/// Choose this over <see cref="AddZbSecretsSqlServerStore"/> when a node must keep resolving
/// secrets while cut off from the hub — reads are served from local state, so a hub outage
/// degrades propagation rather than availability. The cost is real distributed-systems
/// behaviour: rows converge eventually (bounded by the sweep interval), conflicting concurrent
/// writes to the same secret resolve last-writer-wins, and KEK rotation must be run against the
/// hub <em>and</em> each node's local store, because a re-wrap deliberately does not replicate.
/// </remarks>
/// <param name="services">The service collection to add to.</param>
/// <param name="config">The application configuration.</param>
/// <param name="secretsSectionPath">Configuration section holding the core secrets options.</param>
/// <param name="sqlServerSectionPath">Configuration section holding the SQL-Server options.</param>
/// <returns>The same <paramref name="services"/> instance, for chaining.</returns>
public static IServiceCollection AddZbSecretsSqlServerReplication(
this IServiceCollection services,
IConfiguration config,
string secretsSectionPath = "Secrets",
string sqlServerSectionPath = "Secrets:SqlServer")
{
RegisterSqlServerCore(services, config, sqlServerSectionPath);
services.TryAddSingleton<ISecretReplicationHub>(sp => sp.GetRequiredService<SqlServerSecretStore>());
services.TryAddSingleton<ISecretReplicator>(sp =>
new SqlServerSecretReplicator(sp.GetRequiredService<ISecretReplicationHub>()));
// The local store + its migrator come from AddZbSecrets (SQLite by default).
services.AddZbSecrets(config, secretsSectionPath);
// Decorate the local store so every write publishes. Resolved from the concrete SQLite store
// rather than ISecretStore, because ISecretStore is what we are replacing — asking the
// container for it here would resolve this very decorator and recurse forever.
services.AddSingleton<ISecretStore>(sp => new ReplicatingSecretStore(
sp.GetRequiredService<Secrets.Sqlite.SqliteSecretStore>(),
sp.GetRequiredService<ISecretReplicator>(),
sp.GetRequiredService<ILogger<ReplicatingSecretStore>>()));
// The hub's schema must exist too — the node's own migrator only provisions the local store.
services.AddHostedService<SqlServerHubMigrationHostedService>();
services.AddHostedService<SqlServerSecretSyncService>();
return services;
}
// Registers the pieces both modes need: validated options, connection factory, hub store, and
// hub migrator.
private static void RegisterSqlServerCore(
IServiceCollection services,
IConfiguration config,
string sqlServerSectionPath)
{
ArgumentNullException.ThrowIfNull(services);
ArgumentNullException.ThrowIfNull(config);
ArgumentException.ThrowIfNullOrWhiteSpace(sqlServerSectionPath);
IConfigurationSection section = config.GetSection(sqlServerSectionPath);
services.Configure<SqlServerSecretsOptions>(section);
// Bind and validate NOW as well as through the options pipeline: a missing connection string
// or a bad schema name should fail at startup with a clear message, not on the first resolve.
var eager = new SqlServerSecretsOptions();
section.Bind(eager);
eager.Validate();
services.TryAddSingleton(sp =>
new SecretsSqlServerConnectionFactory(
sp.GetRequiredService<IOptions<SqlServerSecretsOptions>>().Value));
services.TryAddSingleton(sp =>
new SqlServerSecretStore(sp.GetRequiredService<SecretsSqlServerConnectionFactory>()));
services.TryAddSingleton(sp =>
new SqlServerSecretsStoreMigrator(sp.GetRequiredService<SecretsSqlServerConnectionFactory>()));
}
}