feat(secrets): cluster replication via SQL Server and Akka.NET (G-7, 0.2.0)

Secrets were per-node SQLite, so a secret written on one node was invisible to
the rest of a cluster. G-7's design resolved the "shared SQL store vs Akka
replicator" fork to build only the former; both are built here so the choice is
a deployment decision (availability vs partition tolerance) rather than a
library limitation.

Two new packages — ZB.MOM.WW.Secrets.Replicator.SqlServer (shared store, plus a
local-store-with-hub mode) and .Replicator.AkkaDotNet (peer-to-peer over
distributed pub/sub). Core gains ISecretsStoreMigrator, one shared
SecretLastWriterWins predicate so no two stores can disagree on a tie, the
transport-agnostic reconciler, and ReplicatingSecretStore — which closes a real
gap: nothing had ever called ISecretReplicator.PublishAsync, so the seam was
inert and local writes would not have propagated at all.

Verified 182 pass / 1 skip / 0 warnings, including 15 live tests against a real
SQL Server 2022 (the SQLite suite ported case-for-case, so any behavioural
divergence between the stores fails) and a 9-test in-process 2-node Akka
cluster over real remoting. A post-build review caught six defects, all fixed
and now covered: both replication modes could not resolve from the container
(no test had built one), an unbounded fetch that broke past SQL Server's
2100-parameter cap, a poison row that aborted the rest of its batch forever,
Enum.Parse on peer input that could restart the actor in a loop, null crypto
blobs crossing the trust boundary, and a silently dropped pull-read failure.

Packed at 0.2.0 and vulnerability-scanned clean; not yet published to the feed.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
2026-07-18 04:08:23 -04:00
parent e46060fada
commit dd0a846b64
55 changed files with 4848 additions and 51 deletions
@@ -0,0 +1,141 @@
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.DependencyInjection.Extensions;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.Secrets.Abstractions;
using ZB.MOM.WW.Secrets.Replication;
using ZB.MOM.WW.Secrets.DependencyInjection;
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer.DependencyInjection;
/// <summary>
/// Dependency-injection entry points for backing a cluster's secrets with SQL Server, in either of
/// the two supported topologies.
/// </summary>
/// <remarks>
/// <para>
/// <b>Both require the same KEK on every node.</b> The database stores ciphertext only, so a node
/// whose master key differs cannot decrypt rows another node wrote — it fails closed on resolve with
/// a <c>kek_id</c> mismatch, which reads like data corruption but is a deployment error. Mount the
/// same key file or supply the same key environment variable everywhere.
/// </para>
/// </remarks>
public static class SqlServerSecretsServiceCollectionExtensions
{
/// <summary>
/// <b>Shared-store mode.</b> Points every node's <see cref="ISecretStore"/> at one shared
/// SQL-Server database. There is exactly one copy of each row, so there is nothing to replicate
/// and nothing to reconcile — no split-brain, no anti-entropy lag, no clock skew.
/// </summary>
/// <remarks>
/// Prefer this unless a node must keep resolving secrets while <em>partitioned</em> from the
/// shared database. The trade is availability: a node cut off from the database falls back only
/// to the resolver's short in-memory cache. If that is unacceptable, use
/// <see cref="AddZbSecretsSqlServerReplication"/> (local store + hub) or the Akka package
/// (peer-to-peer, no shared database at all).
/// </remarks>
/// <param name="services">The service collection to add to.</param>
/// <param name="config">The application configuration.</param>
/// <param name="secretsSectionPath">Configuration section holding the core secrets options.</param>
/// <param name="sqlServerSectionPath">Configuration section holding the SQL-Server options.</param>
/// <returns>The same <paramref name="services"/> instance, for chaining.</returns>
public static IServiceCollection AddZbSecretsSqlServerStore(
this IServiceCollection services,
IConfiguration config,
string secretsSectionPath = "Secrets",
string sqlServerSectionPath = "Secrets:SqlServer")
{
RegisterSqlServerCore(services, config, sqlServerSectionPath);
// Registered BEFORE AddZbSecrets so its TryAdd calls find these already present and leave
// them alone — that is the whole mechanism by which the SQL-Server store displaces SQLite.
services.TryAddSingleton<ISecretStore>(sp => sp.GetRequiredService<SqlServerSecretStore>());
services.TryAddSingleton<ISecretsStoreMigrator>(sp =>
sp.GetRequiredService<SqlServerSecretsStoreMigrator>());
services.AddZbSecrets(config, secretsSectionPath);
return services;
}
/// <summary>
/// <b>Hub-replication mode.</b> Each node keeps its own local store (SQLite by default) and
/// stays converged with a shared SQL-Server hub: local writes publish to the hub immediately, and
/// a background sweep reconciles both directions on an interval.
/// </summary>
/// <remarks>
/// Choose this over <see cref="AddZbSecretsSqlServerStore"/> when a node must keep resolving
/// secrets while cut off from the hub — reads are served from local state, so a hub outage
/// degrades propagation rather than availability. The cost is real distributed-systems
/// behaviour: rows converge eventually (bounded by the sweep interval), conflicting concurrent
/// writes to the same secret resolve last-writer-wins, and KEK rotation must be run against the
/// hub <em>and</em> each node's local store, because a re-wrap deliberately does not replicate.
/// </remarks>
/// <param name="services">The service collection to add to.</param>
/// <param name="config">The application configuration.</param>
/// <param name="secretsSectionPath">Configuration section holding the core secrets options.</param>
/// <param name="sqlServerSectionPath">Configuration section holding the SQL-Server options.</param>
/// <returns>The same <paramref name="services"/> instance, for chaining.</returns>
public static IServiceCollection AddZbSecretsSqlServerReplication(
this IServiceCollection services,
IConfiguration config,
string secretsSectionPath = "Secrets",
string sqlServerSectionPath = "Secrets:SqlServer")
{
RegisterSqlServerCore(services, config, sqlServerSectionPath);
services.TryAddSingleton<ISecretReplicationHub>(sp => sp.GetRequiredService<SqlServerSecretStore>());
services.TryAddSingleton<ISecretReplicator>(sp =>
new SqlServerSecretReplicator(sp.GetRequiredService<ISecretReplicationHub>()));
// The local store + its migrator come from AddZbSecrets (SQLite by default).
services.AddZbSecrets(config, secretsSectionPath);
// Decorate the local store so every write publishes. Resolved from the concrete SQLite store
// rather than ISecretStore, because ISecretStore is what we are replacing — asking the
// container for it here would resolve this very decorator and recurse forever.
services.AddSingleton<ISecretStore>(sp => new ReplicatingSecretStore(
sp.GetRequiredService<Secrets.Sqlite.SqliteSecretStore>(),
sp.GetRequiredService<ISecretReplicator>(),
sp.GetRequiredService<ILogger<ReplicatingSecretStore>>()));
// The hub's schema must exist too — the node's own migrator only provisions the local store.
services.AddHostedService<SqlServerHubMigrationHostedService>();
services.AddHostedService<SqlServerSecretSyncService>();
return services;
}
// Registers the pieces both modes need: validated options, connection factory, hub store, and
// hub migrator.
private static void RegisterSqlServerCore(
IServiceCollection services,
IConfiguration config,
string sqlServerSectionPath)
{
ArgumentNullException.ThrowIfNull(services);
ArgumentNullException.ThrowIfNull(config);
ArgumentException.ThrowIfNullOrWhiteSpace(sqlServerSectionPath);
IConfigurationSection section = config.GetSection(sqlServerSectionPath);
services.Configure<SqlServerSecretsOptions>(section);
// Bind and validate NOW as well as through the options pipeline: a missing connection string
// or a bad schema name should fail at startup with a clear message, not on the first resolve.
var eager = new SqlServerSecretsOptions();
section.Bind(eager);
eager.Validate();
services.TryAddSingleton(sp =>
new SecretsSqlServerConnectionFactory(
sp.GetRequiredService<IOptions<SqlServerSecretsOptions>>().Value));
services.TryAddSingleton(sp =>
new SqlServerSecretStore(sp.GetRequiredService<SecretsSqlServerConnectionFactory>()));
services.TryAddSingleton(sp =>
new SqlServerSecretsStoreMigrator(sp.GetRequiredService<SecretsSqlServerConnectionFactory>()));
}
}
@@ -0,0 +1,34 @@
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer;
/// <summary>
/// The subset of store operations hub replication needs from the shared database: compare
/// inventories, fetch rows in bulk, and accept a row verbatim.
/// </summary>
/// <remarks>
/// Narrower than <see cref="ISecretStore"/> on purpose. The hub is never resolved from, never
/// written through <see cref="ISecretStore.UpsertAsync"/> (which would stamp the hub's own revision
/// and bounce the row back), and never rewrapped by a syncing node — so exposing those operations
/// would only create ways to use it wrongly. It also makes the sync sweep testable against an
/// in-process fake instead of requiring a live SQL Server for every convergence case.
/// </remarks>
public interface ISecretReplicationHub
{
/// <summary>Returns the hub's manifest — names, revisions, timestamps, tombstone flags.</summary>
/// <param name="ct">A token to cancel the operation.</param>
/// <returns>The hub's manifest entries.</returns>
Task<IReadOnlyList<SecretManifestEntry>> GetManifestAsync(CancellationToken ct);
/// <summary>Fetches several encrypted rows by name; names absent from the hub are omitted.</summary>
/// <param name="names">The names to fetch.</param>
/// <param name="ct">A token to cancel the operation.</param>
/// <returns>The matching encrypted rows.</returns>
Task<IReadOnlyList<StoredSecret>> GetManyAsync(IReadOnlyList<SecretName> names, CancellationToken ct);
/// <summary>Applies a row to the hub verbatim, last-writer-wins.</summary>
/// <param name="row">The encrypted row.</param>
/// <param name="ct">A token to cancel the operation.</param>
/// <returns>A task that completes when the row has been applied or ignored.</returns>
Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct);
}
@@ -0,0 +1,57 @@
using Microsoft.Data.SqlClient;
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer;
/// <summary>
/// Opens connections to the shared SQL-Server secret database and carries the settings every
/// command needs (schema name, command timeout).
/// </summary>
/// <param name="options">The validated SQL-Server store options.</param>
public sealed class SecretsSqlServerConnectionFactory(SqlServerSecretsOptions options)
{
private readonly SqlServerSecretsOptions _options =
options ?? throw new ArgumentNullException(nameof(options));
/// <summary>The bracket-quoted, allow-list-validated schema prefix for object names.</summary>
public string SchemaName => _options.SchemaName;
/// <summary>Command timeout, in whole seconds, for every command issued against this store.</summary>
public int CommandTimeoutSeconds => Math.Max(1, (int)_options.CommandTimeout.TotalSeconds);
/// <summary>Opens a connection to the shared secret database.</summary>
/// <param name="cancellationToken">A token to cancel the connect.</param>
/// <returns>An opened connection. The caller owns disposal.</returns>
public async Task<SqlConnection> OpenConnectionAsync(CancellationToken cancellationToken)
{
var connection = new SqlConnection(_options.ConnectionString);
try
{
await connection.OpenAsync(cancellationToken).ConfigureAwait(false);
return connection;
}
catch
{
await connection.DisposeAsync().ConfigureAwait(false);
throw;
}
}
/// <summary>
/// Creates a command on <paramref name="connection"/> with the configured timeout applied.
/// </summary>
/// <param name="connection">The open connection.</param>
/// <param name="commandText">The command text.</param>
/// <param name="transaction">The ambient transaction, if any.</param>
/// <returns>The configured command. The caller owns disposal.</returns>
public SqlCommand CreateCommand(
SqlConnection connection, string commandText, SqlTransaction? transaction = null)
{
ArgumentNullException.ThrowIfNull(connection);
SqlCommand command = connection.CreateCommand();
command.CommandText = commandText;
command.CommandTimeout = CommandTimeoutSeconds;
command.Transaction = transaction;
return command;
}
}
@@ -0,0 +1,33 @@
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Options;
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer;
/// <summary>
/// Provisions the shared hub schema at startup in hub-replication mode.
/// </summary>
/// <remarks>
/// Separate from the core migration hosted service on purpose: in hub mode there are <em>two</em>
/// stores to provision — the node's local one (handled by the core service through the
/// <c>ISecretsStoreMigrator</c> seam) and the shared hub, which no node "owns". Every node runs this
/// idempotently at startup, so the hub exists as soon as any node has started, with no separate
/// provisioning step for an operator to forget.
/// </remarks>
/// <param name="migrator">The hub migrator.</param>
/// <param name="options">Core secrets options, for the shared startup-migration switch.</param>
internal sealed class SqlServerHubMigrationHostedService(
SqlServerSecretsStoreMigrator migrator,
IOptions<SecretsOptions> options) : IHostedService
{
/// <inheritdoc />
public async Task StartAsync(CancellationToken cancellationToken)
{
if (options.Value.RunMigrationsOnStartup)
{
await migrator.MigrateAsync(cancellationToken).ConfigureAwait(false);
}
}
/// <inheritdoc />
public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask;
}
@@ -0,0 +1,26 @@
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer;
/// <summary>
/// Publishes local writes to the shared SQL-Server hub, so a secret written on one node is
/// immediately visible to every other node that syncs against the same hub.
/// </summary>
/// <remarks>
/// The row is written through <see cref="ISecretStore.ApplyReplicatedAsync"/>, not
/// <see cref="ISecretStore.UpsertAsync"/> — the hub must record the row <em>verbatim</em>, keeping
/// the originating node's revision and <c>updated_utc</c>. An upsert would stamp the hub's own
/// revision and timestamp, and the row would then look newer than the writer's own copy, so it
/// would bounce straight back on the next sync.
/// </remarks>
/// <param name="hub">The shared hub store.</param>
public sealed class SqlServerSecretReplicator(ISecretReplicationHub hub) : ISecretReplicator
{
/// <inheritdoc />
public Task PublishAsync(StoredSecret row, CancellationToken ct)
{
ArgumentNullException.ThrowIfNull(row);
return hub.ApplyReplicatedAsync(row, ct);
}
}
@@ -0,0 +1,398 @@
using System.Data;
using System.Globalization;
using Microsoft.Data.SqlClient;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer;
/// <summary>
/// SQL-Server-backed <see cref="ISecretStore"/>. Holds the same envelope-encrypted
/// <see cref="StoredSecret"/> rows as the SQLite store, in a database every node of a cluster can
/// reach — so one write is visible everywhere with no replication at all.
/// </summary>
/// <remarks>
/// <para>
/// The database holds <b>ciphertext only</b>. The KEK stays per-node and out of the database, so a
/// DBA, a backup tape, or a replica cannot decrypt anything. The corollary is a hard operational
/// constraint: every node must resolve the <em>same</em> KEK, because a row whose <c>kek_id</c> does
/// not match the local provider fails closed on resolve.
/// </para>
/// <para>
/// Semantics are deliberately identical to <c>SqliteSecretStore</c> — revision bump on upsert,
/// created stamps preserved on overwrite, verbatim last-writer-wins apply, compare-and-swap rewrap.
/// A cluster can run both stores at once (nodes on local SQLite replicating against a shared hub),
/// so a divergence between the two would be a silent convergence bug.
/// </para>
/// </remarks>
/// <param name="connectionFactory">Factory for connections to the shared secret database.</param>
public sealed class SqlServerSecretStore(SecretsSqlServerConnectionFactory connectionFactory)
: ISecretStore, ISecretReplicationHub
{
// All columns of the secret table, in schema order, for full-row reads.
private const string AllColumns =
"name, description, content_type, ciphertext, nonce, tag, wrapped_dek, wrap_nonce, wrap_tag, " +
"kek_id, revision, is_deleted, deleted_utc, created_utc, updated_utc, created_by, updated_by";
// The safe metadata projection — deliberately excludes every ciphertext / crypto BLOB column.
private const string MetadataColumns =
"name, description, content_type, kek_id, revision, is_deleted, created_utc, updated_utc, created_by, updated_by";
private string Secret => $"[{connectionFactory.SchemaName}].[{SqlServerSecretsSchema.SecretTable}]";
/// <inheritdoc />
public async Task<StoredSecret?> GetAsync(SecretName name, CancellationToken ct)
{
await using SqlConnection connection =
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
await using SqlCommand command = connectionFactory.CreateCommand(
connection, $"SELECT {AllColumns} FROM {Secret} WHERE name = @name;");
command.Parameters.AddWithValue("@name", name.Value);
await using SqlDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false);
return await reader.ReadAsync(ct).ConfigureAwait(false)
? ReadStoredSecret(reader)
: null;
}
/// <inheritdoc />
public async Task UpsertAsync(StoredSecret row, CancellationToken ct)
{
ArgumentNullException.ThrowIfNull(row);
await using SqlConnection connection =
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
string now = DateTimeOffset.UtcNow.ToString("O");
// MERGE establishes revision 0 / created==updated==now on insert; on match it overwrites the
// crypto material in place, bumps the revision, refreshes updated_utc/updated_by, and clears
// any tombstone — but deliberately preserves created_utc / created_by (the SQLite store's
// ON CONFLICT semantics, expressed in T-SQL).
//
// HOLDLOCK is not optional: without it MERGE takes only an update lock on the key range and
// two concurrent nodes upserting the same new name can both miss the match and both INSERT,
// producing a primary-key violation instead of one insert + one update.
await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
MERGE {Secret} WITH (HOLDLOCK) AS target
USING (SELECT @name AS name) AS source
ON target.name = source.name
WHEN MATCHED THEN UPDATE SET
description = @description,
content_type = @content_type,
ciphertext = @ciphertext,
nonce = @nonce,
tag = @tag,
wrapped_dek = @wrapped_dek,
wrap_nonce = @wrap_nonce,
wrap_tag = @wrap_tag,
kek_id = @kek_id,
revision = target.revision + 1,
updated_utc = @now,
updated_by = @updated_by,
is_deleted = 0,
deleted_utc = NULL
WHEN NOT MATCHED THEN INSERT (
name, description, content_type, ciphertext, nonce, tag,
wrapped_dek, wrap_nonce, wrap_tag, kek_id, revision,
is_deleted, deleted_utc, created_utc, updated_utc, created_by, updated_by)
VALUES (
@name, @description, @content_type, @ciphertext, @nonce, @tag,
@wrapped_dek, @wrap_nonce, @wrap_tag, @kek_id, 0,
0, NULL, @now, @now, @created_by, @updated_by);
""");
BindCryptoColumns(command, row);
command.Parameters.AddWithValue("@now", now);
command.Parameters.AddWithValue("@created_by", (object?)row.CreatedBy ?? DBNull.Value);
command.Parameters.AddWithValue("@updated_by", (object?)row.UpdatedBy ?? DBNull.Value);
await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
}
/// <inheritdoc />
public async Task<bool> DeleteAsync(SecretName name, string? actor, CancellationToken ct)
{
await using SqlConnection connection =
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
string now = DateTimeOffset.UtcNow.ToString("O");
await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
UPDATE {Secret} SET
is_deleted = 1,
deleted_utc = @now,
revision = revision + 1,
updated_utc = @now,
updated_by = @actor
WHERE name = @name AND is_deleted = 0;
""");
command.Parameters.AddWithValue("@now", now);
command.Parameters.AddWithValue("@actor", (object?)actor ?? DBNull.Value);
command.Parameters.AddWithValue("@name", name.Value);
int rowsAffected = await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
return rowsAffected > 0;
}
/// <inheritdoc />
public async Task<IReadOnlyList<SecretMetadata>> ListAsync(bool includeDeleted, CancellationToken ct)
{
await using SqlConnection connection =
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
await using SqlCommand command = connectionFactory.CreateCommand(
connection,
$"SELECT {MetadataColumns} FROM {Secret} WHERE (@include_deleted = 1 OR is_deleted = 0) ORDER BY name;");
command.Parameters.AddWithValue("@include_deleted", includeDeleted ? 1 : 0);
var results = new List<SecretMetadata>();
await using SqlDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false);
while (await reader.ReadAsync(ct).ConfigureAwait(false))
{
results.Add(new SecretMetadata
{
Name = new SecretName(reader.GetString(0)),
Description = reader.IsDBNull(1) ? null : reader.GetString(1),
ContentType = Enum.Parse<SecretContentType>(reader.GetString(2)),
KekId = reader.GetString(3),
Revision = reader.GetInt64(4),
IsDeleted = reader.GetBoolean(5),
CreatedUtc = ParseUtc(reader.GetString(6)),
UpdatedUtc = ParseUtc(reader.GetString(7)),
CreatedBy = reader.IsDBNull(8) ? null : reader.GetString(8),
UpdatedBy = reader.IsDBNull(9) ? null : reader.GetString(9),
});
}
return results;
}
/// <inheritdoc />
public async Task<IReadOnlyList<SecretManifestEntry>> GetManifestAsync(CancellationToken ct)
{
await using SqlConnection connection =
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
await using SqlCommand command = connectionFactory.CreateCommand(
connection, $"SELECT name, revision, updated_utc, is_deleted FROM {Secret} ORDER BY name;");
var results = new List<SecretManifestEntry>();
await using SqlDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false);
while (await reader.ReadAsync(ct).ConfigureAwait(false))
{
results.Add(new SecretManifestEntry
{
Name = new SecretName(reader.GetString(0)),
Revision = reader.GetInt64(1),
UpdatedUtc = ParseUtc(reader.GetString(2)),
IsDeleted = reader.GetBoolean(3),
});
}
return results;
}
/// <inheritdoc />
public async Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct)
{
ArgumentNullException.ThrowIfNull(row);
await using SqlConnection connection =
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
await using SqlTransaction transaction = (SqlTransaction)
await connection.BeginTransactionAsync(IsolationLevel.Serializable, ct).ConfigureAwait(false);
// Read the local (updated_utc, revision) so we can apply last-writer-wins. UPDLOCK/HOLDLOCK
// keeps another node from slipping a newer row in between this read and the write below —
// Serializable alone would not lock a key range that currently has no row.
await using (SqlCommand read = connectionFactory.CreateCommand(
connection,
$"SELECT updated_utc, revision FROM {Secret} WITH (UPDLOCK, HOLDLOCK) WHERE name = @name;",
transaction))
{
read.Parameters.AddWithValue("@name", row.Name.Value);
await using SqlDataReader reader = await read.ExecuteReaderAsync(ct).ConfigureAwait(false);
if (await reader.ReadAsync(ct).ConfigureAwait(false))
{
DateTimeOffset localUpdated = ParseUtc(reader.GetString(0));
long localRevision = reader.GetInt64(1);
// Routed through the shared predicate so every store agrees on ties.
if (!SecretLastWriterWins.IsNewer(row.UpdatedUtc, row.Revision, localUpdated, localRevision))
{
await reader.CloseAsync().ConfigureAwait(false);
await transaction.CommitAsync(ct).ConfigureAwait(false);
return;
}
}
}
// Apply the incoming row VERBATIM — its own revision, timestamps, tombstone flag, and crypto
// material — with no revision bump (this is the replication path, not a local write).
await using (SqlCommand upsert = connectionFactory.CreateCommand(connection, $"""
MERGE {Secret} WITH (HOLDLOCK) AS target
USING (SELECT @name AS name) AS source
ON target.name = source.name
WHEN MATCHED THEN UPDATE SET
description = @description,
content_type = @content_type,
ciphertext = @ciphertext,
nonce = @nonce,
tag = @tag,
wrapped_dek = @wrapped_dek,
wrap_nonce = @wrap_nonce,
wrap_tag = @wrap_tag,
kek_id = @kek_id,
revision = @revision,
is_deleted = @is_deleted,
deleted_utc = @deleted_utc,
created_utc = @created_utc,
updated_utc = @updated_utc,
created_by = @created_by,
updated_by = @updated_by
WHEN NOT MATCHED THEN INSERT (
name, description, content_type, ciphertext, nonce, tag,
wrapped_dek, wrap_nonce, wrap_tag, kek_id, revision,
is_deleted, deleted_utc, created_utc, updated_utc, created_by, updated_by)
VALUES (
@name, @description, @content_type, @ciphertext, @nonce, @tag,
@wrapped_dek, @wrap_nonce, @wrap_tag, @kek_id, @revision,
@is_deleted, @deleted_utc, @created_utc, @updated_utc, @created_by, @updated_by);
""", transaction))
{
BindCryptoColumns(upsert, row);
upsert.Parameters.AddWithValue("@revision", row.Revision);
upsert.Parameters.AddWithValue("@is_deleted", row.IsDeleted ? 1 : 0);
upsert.Parameters.AddWithValue("@deleted_utc", (object?)row.DeletedUtc?.ToString("O") ?? DBNull.Value);
upsert.Parameters.AddWithValue("@created_utc", row.CreatedUtc.ToString("O"));
upsert.Parameters.AddWithValue("@updated_utc", row.UpdatedUtc.ToString("O"));
upsert.Parameters.AddWithValue("@created_by", (object?)row.CreatedBy ?? DBNull.Value);
upsert.Parameters.AddWithValue("@updated_by", (object?)row.UpdatedBy ?? DBNull.Value);
await upsert.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
}
await transaction.CommitAsync(ct).ConfigureAwait(false);
}
/// <inheritdoc />
public async Task<bool> ApplyRewrapAsync(
StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct)
{
ArgumentNullException.ThrowIfNull(rewrappedRow);
ArgumentNullException.ThrowIfNull(expectedCurrentWrappedDek);
await using SqlConnection connection =
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
// Overwrite ONLY the KEK-wrap envelope + kek_id. Revision, updated_utc/by, the sealed body,
// and the tombstone state are deliberately untouched — a re-wrap is not a logical change, so
// it must not bump the revision or updated timestamp (see ISecretStore.ApplyRewrapAsync).
//
// Compare-and-swap on wrapped_dek: a concurrent set/rotate generates a fresh (random) DEK
// wrap, so this matches 0 rows instead of pairing this stale re-wrap with a newer body —
// which would leave the row permanently undecryptable. The caller re-processes a 0-row miss.
await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
UPDATE {Secret} SET
wrapped_dek = @wrapped_dek,
wrap_nonce = @wrap_nonce,
wrap_tag = @wrap_tag,
kek_id = @kek_id
WHERE name = @name AND wrapped_dek = @expected_wrapped_dek;
""");
command.Parameters.AddWithValue("@wrapped_dek", rewrappedRow.WrappedDek);
command.Parameters.AddWithValue("@wrap_nonce", rewrappedRow.WrapNonce);
command.Parameters.AddWithValue("@wrap_tag", rewrappedRow.WrapTag);
command.Parameters.AddWithValue("@kek_id", rewrappedRow.KekId);
command.Parameters.AddWithValue("@name", rewrappedRow.Name.Value);
command.Parameters.AddWithValue("@expected_wrapped_dek", expectedCurrentWrappedDek);
int rowsAffected = await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
return rowsAffected > 0;
}
/// <summary>
/// Fetches several rows by name in one round trip — the replication fetch path. Names not
/// present are simply absent from the result.
/// </summary>
/// <param name="names">The names to fetch.</param>
/// <param name="ct">A token to cancel the operation.</param>
/// <returns>The matching encrypted rows.</returns>
public async Task<IReadOnlyList<StoredSecret>> GetManyAsync(
IReadOnlyList<SecretName> names, CancellationToken ct)
{
ArgumentNullException.ThrowIfNull(names);
if (names.Count == 0)
{
return [];
}
await using SqlConnection connection =
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
// One parameter per name — still fully parameterized, no name text is interpolated.
string[] placeholders = [.. names.Select((_, i) => $"@n{i.ToString(CultureInfo.InvariantCulture)}")];
await using SqlCommand command = connectionFactory.CreateCommand(
connection,
$"SELECT {AllColumns} FROM {Secret} WHERE name IN ({string.Join(", ", placeholders)});");
for (int i = 0; i < names.Count; i++)
{
command.Parameters.AddWithValue(placeholders[i], names[i].Value);
}
var results = new List<StoredSecret>(names.Count);
await using SqlDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false);
while (await reader.ReadAsync(ct).ConfigureAwait(false))
{
results.Add(ReadStoredSecret(reader));
}
return results;
}
// Binds the identity, description, content-type, KEK id, and all six crypto BLOB columns
// shared by every insert path.
private static void BindCryptoColumns(SqlCommand command, StoredSecret row)
{
command.Parameters.AddWithValue("@name", row.Name.Value);
command.Parameters.AddWithValue("@description", (object?)row.Description ?? DBNull.Value);
command.Parameters.AddWithValue("@content_type", row.ContentType.ToString());
command.Parameters.AddWithValue("@ciphertext", row.Ciphertext);
command.Parameters.AddWithValue("@nonce", row.Nonce);
command.Parameters.AddWithValue("@tag", row.Tag);
command.Parameters.AddWithValue("@wrapped_dek", row.WrappedDek);
command.Parameters.AddWithValue("@wrap_nonce", row.WrapNonce);
command.Parameters.AddWithValue("@wrap_tag", row.WrapTag);
command.Parameters.AddWithValue("@kek_id", row.KekId);
}
private static StoredSecret ReadStoredSecret(SqlDataReader reader) => new()
{
Name = new SecretName(reader.GetString(0)),
Description = reader.IsDBNull(1) ? null : reader.GetString(1),
ContentType = Enum.Parse<SecretContentType>(reader.GetString(2)),
Ciphertext = reader.GetFieldValue<byte[]>(3),
Nonce = reader.GetFieldValue<byte[]>(4),
Tag = reader.GetFieldValue<byte[]>(5),
WrappedDek = reader.GetFieldValue<byte[]>(6),
WrapNonce = reader.GetFieldValue<byte[]>(7),
WrapTag = reader.GetFieldValue<byte[]>(8),
KekId = reader.GetString(9),
Revision = reader.GetInt64(10),
IsDeleted = reader.GetBoolean(11),
DeletedUtc = reader.IsDBNull(12) ? null : ParseUtc(reader.GetString(12)),
CreatedUtc = ParseUtc(reader.GetString(13)),
UpdatedUtc = ParseUtc(reader.GetString(14)),
CreatedBy = reader.IsDBNull(15) ? null : reader.GetString(15),
UpdatedBy = reader.IsDBNull(16) ? null : reader.GetString(16),
};
private static DateTimeOffset ParseUtc(string value) =>
DateTimeOffset.Parse(value, CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind);
}
@@ -0,0 +1,158 @@
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.Secrets.Abstractions;
using ZB.MOM.WW.Secrets.Replication;
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer;
/// <summary>
/// Keeps a node's local secret store converged with the shared SQL-Server hub, in hub-replication
/// mode. Runs a bidirectional anti-entropy sweep on an interval: pulls rows the hub holds newer,
/// pushes rows the node holds newer.
/// </summary>
/// <remarks>
/// <para>
/// This is the durability guarantee behind best-effort publishing. Live publishes make a write
/// visible in milliseconds; this sweep is what makes it <em>certain</em>. A node that was offline
/// while a peer wrote catches up on its next sweep, and a write made while the hub was unreachable
/// is pushed on the next sweep instead of being lost.
/// </para>
/// <para>
/// The sweep never throws out of <see cref="ExecuteAsync"/>: a hub outage must degrade the node to
/// "serving its last-known-good local secrets" rather than crash the host. Failures are logged and
/// retried on the next tick.
/// </para>
/// </remarks>
public sealed partial class SqlServerSecretSyncService : BackgroundService
{
private readonly SecretReplicationReconciler _reconciler;
private readonly ISecretStore _local;
private readonly ISecretReplicationHub _hub;
private readonly SqlServerSecretsOptions _options;
private readonly ILogger<SqlServerSecretSyncService> _logger;
private readonly TimeProvider _timeProvider;
/// <summary>Creates the sync service.</summary>
/// <param name="local">The node's local store.</param>
/// <param name="hub">The shared hub store.</param>
/// <param name="cacheInvalidator">Resolver-cache seam, evicted for every applied row.</param>
/// <param name="options">SQL-Server replication options.</param>
/// <param name="logger">Logger for sweep outcomes and failures.</param>
/// <param name="timeProvider">Clock, injectable so tests need not wait a real interval.</param>
public SqlServerSecretSyncService(
ISecretStore local,
ISecretReplicationHub hub,
ISecretCacheInvalidator? cacheInvalidator,
IOptions<SqlServerSecretsOptions> options,
ILogger<SqlServerSecretSyncService> logger,
TimeProvider? timeProvider = null)
{
ArgumentNullException.ThrowIfNull(options);
_local = local ?? throw new ArgumentNullException(nameof(local));
_hub = hub ?? throw new ArgumentNullException(nameof(hub));
_options = options.Value;
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_timeProvider = timeProvider ?? TimeProvider.System;
_reconciler = new SecretReplicationReconciler(
_local,
cacheInvalidator,
onRowFailed: (name, ex) => LogRowDiscarded(ex, name.Value));
}
/// <inheritdoc />
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
if (_options.SyncOnStartup)
{
await SweepSafelyAsync(stoppingToken).ConfigureAwait(false);
}
using PeriodicTimer timer = new(_options.SyncInterval, _timeProvider);
while (await timer.WaitForNextTickAsync(stoppingToken).ConfigureAwait(false))
{
await SweepSafelyAsync(stoppingToken).ConfigureAwait(false);
}
}
/// <summary>
/// Runs one bidirectional reconcile against the hub.
/// </summary>
/// <param name="ct">A token to cancel the sweep.</param>
/// <returns>How many rows were pulled from and pushed to the hub.</returns>
internal async Task<(int Pulled, int Pushed)> SweepAsync(CancellationToken ct)
{
IReadOnlyList<SecretManifestEntry> hubManifest =
await _hub.GetManifestAsync(ct).ConfigureAwait(false);
// Pull first: adopting the hub's newer rows before computing the push set means a row the hub
// already had newer is not then pushed straight back at it.
int pulled = await _reconciler
.ReconcileAsync(hubManifest, (names, token) => _hub.GetManyAsync(names, token), ct)
.ConfigureAwait(false);
IReadOnlyList<SecretManifestEntry> localManifest =
await _local.GetManifestAsync(ct).ConfigureAwait(false);
IReadOnlyList<SecretName> push =
SecretReplicationReconciler.ComputePushSet(localManifest, hubManifest);
int pushed = 0;
if (push.Count > 0)
{
IReadOnlyList<StoredSecret> rows =
await _reconciler.ReadLocalAsync(push, ct).ConfigureAwait(false);
foreach (StoredSecret row in rows)
{
// Verbatim, like the replicator — the hub records the originating node's revision.
await _hub.ApplyReplicatedAsync(row, ct).ConfigureAwait(false);
pushed++;
}
}
return (pulled, pushed);
}
private async Task SweepSafelyAsync(CancellationToken ct)
{
try
{
(int pulled, int pushed) = await SweepAsync(ct).ConfigureAwait(false);
if (pulled > 0 || pushed > 0)
{
LogSweepConverged(pulled, pushed);
}
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
// Host shutdown — not a failure.
}
catch (Exception ex)
{
LogSweepFailed(ex);
}
}
[LoggerMessage(
EventId = 1,
Level = LogLevel.Information,
Message = "Secret hub sync converged: pulled {Pulled} row(s) from the hub, pushed {Pushed} row(s) to it.")]
private partial void LogSweepConverged(int pulled, int pushed);
[LoggerMessage(
EventId = 2,
Level = LogLevel.Warning,
Message = "Secret hub sync failed; the node continues serving its local store and will retry on the next interval.")]
private partial void LogSweepFailed(Exception exception);
[LoggerMessage(
EventId = 3,
Level = LogLevel.Warning,
Message = "Discarding replicated secret row {SecretName} that could not be applied locally; the rest of the batch continues.")]
private partial void LogRowDiscarded(Exception exception, string secretName);
}
@@ -0,0 +1,121 @@
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer;
/// <summary>
/// Configuration for the shared SQL-Server secret store and its hub replication. Bound from a
/// configuration section by the <c>AddZbSecretsSqlServer*</c> extensions.
/// </summary>
public sealed class SqlServerSecretsOptions
{
private string _schemaName = SqlServerSecretsSchema.DefaultSchemaName;
/// <summary>
/// Connection string for the shared secret database. Required. Supply it through a
/// <c>${secret:}</c> / <c>secret:</c> reference like every other connection string in the family
/// — the shared store holds ciphertext only, but its credentials are still credentials.
/// </summary>
public string ConnectionString { get; set; } = string.Empty;
/// <summary>
/// SQL schema the secret tables live in. Defaults to <c>zbsecrets</c> so the secret tables never
/// collide with an application's own <c>dbo</c> objects when they share a database.
/// </summary>
/// <exception cref="ArgumentException">
/// The value is not a plain identifier (letters, digits, and underscore, not starting with a
/// digit, at most 128 characters).
/// </exception>
/// <remarks>
/// Validated on assignment rather than at use: the schema name is interpolated into DDL and
/// query text (T-SQL cannot parameterize an object name), so the allow-list is the injection
/// boundary and it must be impossible to get an unvalidated value into the property at all.
/// </remarks>
public string SchemaName
{
get => _schemaName;
set => _schemaName = ValidateIdentifier(value);
}
/// <summary>
/// How often the sync service reconciles the local store against the hub, in hub-replication
/// mode. Defaults to 30 seconds. Ignored in shared-store mode (there is nothing to reconcile).
/// </summary>
public TimeSpan SyncInterval { get; set; } = TimeSpan.FromSeconds(30);
/// <summary>
/// When <see langword="true"/> (the default), hub-replication mode runs a reconcile at startup
/// before the first interval elapses, so a node that was down while a peer wrote does not serve
/// stale secrets for a whole interval after it comes back.
/// </summary>
public bool SyncOnStartup { get; set; } = true;
/// <summary>
/// Command timeout applied to every store operation. Defaults to 30 seconds — long enough for a
/// contended shared database, short enough that a hung hub surfaces as an error rather than
/// stalling a resolve indefinitely.
/// </summary>
public TimeSpan CommandTimeout { get; set; } = TimeSpan.FromSeconds(30);
/// <summary>
/// Throws if the options are unusable. Called by the DI extensions at registration time so a
/// misconfigured node fails at startup rather than on the first secret resolve.
/// </summary>
/// <exception cref="InvalidOperationException">The connection string is missing, or an interval is not positive.</exception>
public void Validate()
{
if (string.IsNullOrWhiteSpace(ConnectionString))
{
throw new InvalidOperationException(
"SQL-Server secret store requires a ConnectionString (Secrets:SqlServer:ConnectionString).");
}
if (SyncInterval <= TimeSpan.Zero)
{
throw new InvalidOperationException(
$"SQL-Server secret replication SyncInterval must be positive (was {SyncInterval}).");
}
if (CommandTimeout <= TimeSpan.Zero)
{
throw new InvalidOperationException(
$"SQL-Server secret store CommandTimeout must be positive (was {CommandTimeout}).");
}
}
/// <summary>
/// Strict allow-list for a SQL identifier that will be interpolated into DDL or query text.
/// Rejects everything that is not <c>[A-Za-z_][A-Za-z0-9_]{0,127}</c> — no brackets, quotes,
/// dots, or spaces — so no bracket-escape (<c>]]</c>) trick can break out of the quoting applied
/// at the use site. Uses the ASCII-only character tests deliberately, so homoglyph and
/// full-width look-alikes are rejected rather than accepted as "letters".
/// </summary>
/// <param name="value">The candidate identifier.</param>
/// <returns>The same value, once validated.</returns>
/// <exception cref="ArgumentException">The value is not a plain identifier.</exception>
internal static string ValidateIdentifier(string value)
{
ArgumentException.ThrowIfNullOrWhiteSpace(value);
if (value.Length > 128)
{
throw new ArgumentException(
$"SQL schema name must be at most 128 characters (was {value.Length}).", nameof(value));
}
if (!char.IsAsciiLetter(value[0]) && value[0] != '_')
{
throw new ArgumentException(
$"SQL schema name must start with a letter or underscore (was '{value}').", nameof(value));
}
foreach (char c in value)
{
if (!char.IsAsciiLetterOrDigit(c) && c != '_')
{
throw new ArgumentException(
$"SQL schema name may contain only letters, digits, and underscore (was '{value}').",
nameof(value));
}
}
return value;
}
}
@@ -0,0 +1,100 @@
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer;
/// <summary>
/// Schema constants and DDL for the shared SQL-Server secret store. Mirrors
/// <c>ZB.MOM.WW.Secrets.Sqlite.SqliteSecretsSchema</c> column-for-column so a row means exactly the
/// same thing in either store.
/// </summary>
/// <remarks>
/// <para>
/// <b>Timestamps are ISO-8601 <c>nvarchar</c>, not <c>datetimeoffset</c>.</b> This looks wrong for a
/// SQL-Server schema and is deliberate: the same <see cref="ZB.MOM.WW.Secrets.Abstractions.StoredSecret"/>
/// rows replicate between this store and SQLite (which stores <c>"O"</c>-format text), and
/// last-writer-wins compares those timestamps for equality. <c>datetimeoffset</c> rounds to 100ns
/// while .NET <c>DateTimeOffset</c> ticks are also 100ns but the round-trip through the driver is
/// not guaranteed bit-identical across providers — a sub-tick difference would turn a tie into a
/// spurious "newer", and two nodes would then overwrite each other forever. Storing the exact
/// round-trippable text both providers already agree on removes the failure mode entirely.
/// </para>
/// <para>
/// The cost is that <c>ORDER BY updated_utc</c> is a lexicographic sort. That is still chronological
/// for <c>"O"</c>-format UTC strings (fixed-width, zero-padded, always <c>+00:00</c>), which is what
/// the store writes.
/// </para>
/// </remarks>
public static class SqlServerSecretsSchema
{
/// <summary>
/// The schema version this build creates and supports. The migrator refuses a database whose
/// recorded version is <em>newer</em> than this.
/// </summary>
public const int CurrentVersion = 1;
/// <summary>Default schema the secret tables live in.</summary>
public const string DefaultSchemaName = "zbsecrets";
/// <summary>Name of the single-row table tracking the applied schema version.</summary>
public const string SchemaVersionTable = "schema_version";
/// <summary>Name of the table storing envelope-encrypted secret records.</summary>
public const string SecretTable = "secret";
/// <summary>
/// Returns the idempotent DDL that provisions the schema, both tables, and the supporting index
/// in <paramref name="schemaName"/>.
/// </summary>
/// <param name="schemaName">The already-validated SQL schema name to create the objects in.</param>
/// <returns>The DDL batch.</returns>
/// <remarks>
/// <paramref name="schemaName"/> is interpolated, not parameterized — T-SQL has no parameter form
/// for an object name. It is re-validated here against the same strict identifier allow-list
/// <see cref="SqlServerSecretsOptions.SchemaName"/> enforces, and bracket-quoted below. The
/// duplicate check is deliberate: this method is <c>public</c>, so it must defend itself rather
/// than trust that every caller routed through the options type.
/// </remarks>
/// <exception cref="ArgumentException"><paramref name="schemaName"/> is not a plain identifier.</exception>
public static string CreateSchemaDdl(string schemaName)
{
SqlServerSecretsOptions.ValidateIdentifier(schemaName);
return $"""
IF NOT EXISTS (SELECT 1 FROM sys.schemas WHERE name = N'{schemaName}')
EXEC(N'CREATE SCHEMA [{schemaName}]');
IF OBJECT_ID(N'[{schemaName}].[{SchemaVersionTable}]', N'U') IS NULL
CREATE TABLE [{schemaName}].[{SchemaVersionTable}] (
id INT NOT NULL PRIMARY KEY CHECK (id = 1),
version INT NOT NULL,
applied_utc NVARCHAR(33) NOT NULL
);
IF OBJECT_ID(N'[{schemaName}].[{SecretTable}]', N'U') IS NULL
CREATE TABLE [{schemaName}].[{SecretTable}] (
name NVARCHAR(450) NOT NULL PRIMARY KEY,
description NVARCHAR(MAX) NULL,
content_type NVARCHAR(64) NOT NULL,
ciphertext VARBINARY(MAX) NOT NULL,
nonce VARBINARY(MAX) NOT NULL,
tag VARBINARY(MAX) NOT NULL,
wrapped_dek VARBINARY(MAX) NOT NULL,
wrap_nonce VARBINARY(MAX) NOT NULL,
wrap_tag VARBINARY(MAX) NOT NULL,
kek_id NVARCHAR(256) NOT NULL,
revision BIGINT NOT NULL CONSTRAINT DF_secret_revision DEFAULT 0,
is_deleted BIT NOT NULL CONSTRAINT DF_secret_is_deleted DEFAULT 0,
deleted_utc NVARCHAR(33) NULL,
created_utc NVARCHAR(33) NOT NULL,
updated_utc NVARCHAR(33) NOT NULL,
created_by NVARCHAR(256) NULL,
updated_by NVARCHAR(256) NULL
);
IF NOT EXISTS (
SELECT 1 FROM sys.indexes
WHERE name = N'ix_secret_updated_utc'
AND object_id = OBJECT_ID(N'[{schemaName}].[{SecretTable}]'))
CREATE INDEX ix_secret_updated_utc
ON [{schemaName}].[{SecretTable}] (updated_utc);
""";
}
}
@@ -0,0 +1,125 @@
using System.Data;
using System.Globalization;
using Microsoft.Data.SqlClient;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer;
/// <summary>
/// Creates the shared SQL-Server secret schema and records the applied version. Idempotent: safe to
/// run on every node at every startup, including concurrently. Refuses to run against a database
/// whose recorded version is newer than this build supports.
/// </summary>
/// <param name="connectionFactory">Factory for connections to the shared secret database.</param>
public sealed class SqlServerSecretsStoreMigrator(SecretsSqlServerConnectionFactory connectionFactory)
: ISecretsStoreMigrator
{
// Concurrent Serializable DDL against the system catalogs can pick a deadlock victim (error
// 1205) even though the IF NOT EXISTS guards make the logic itself safe. Every node runs this at
// every startup, so a virgin database being provisioned by N nodes at once is the NORMAL case,
// not a rare race — and an unretried victim aborts that node's whole host at startup. Retrying
// is correct precisely because the migration is idempotent.
private const int MaxDeadlockRetries = 5;
private const int SqlDeadlockVictimErrorNumber = 1205;
/// <inheritdoc />
/// <exception cref="SecretStoreMigrationException">
/// The recorded schema version is newer than <see cref="SqlServerSecretsSchema.CurrentVersion"/>.
/// </exception>
public async Task MigrateAsync(CancellationToken cancellationToken)
{
for (int attempt = 1; ; attempt++)
{
try
{
await MigrateOnceAsync(cancellationToken).ConfigureAwait(false);
return;
}
catch (SqlException ex)
when (ex.Number == SqlDeadlockVictimErrorNumber && attempt < MaxDeadlockRetries)
{
// Staggered back-off so two victims do not retry in lockstep and deadlock again.
await Task.Delay(TimeSpan.FromMilliseconds(100 * attempt), cancellationToken)
.ConfigureAwait(false);
}
}
}
private async Task MigrateOnceAsync(CancellationToken cancellationToken)
{
string schema = connectionFactory.SchemaName;
await using SqlConnection connection =
await connectionFactory.OpenConnectionAsync(cancellationToken).ConfigureAwait(false);
// Serializable: unlike the single-writer SQLite store, EVERY node in the cluster runs this
// migration at startup, so concurrent first-boots are the normal case rather than a rare
// race. The isolation level plus the IF NOT EXISTS guards make a simultaneous provision by
// two nodes converge on one schema instead of one of them failing with "object exists".
await using SqlTransaction transaction = (SqlTransaction)
await connection.BeginTransactionAsync(IsolationLevel.Serializable, cancellationToken)
.ConfigureAwait(false);
int existingVersion =
await ReadExistingSchemaVersionAsync(connection, transaction, schema, cancellationToken)
.ConfigureAwait(false);
if (existingVersion > SqlServerSecretsSchema.CurrentVersion)
{
throw new SecretStoreMigrationException(
$"Shared secret database schema version {existingVersion} is newer than supported version " +
$"{SqlServerSecretsSchema.CurrentVersion}. Upgrade this node before it writes to the shared store.");
}
await using (SqlCommand ddl = connectionFactory.CreateCommand(
connection, SqlServerSecretsSchema.CreateSchemaDdl(schema), transaction))
{
await ddl.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false);
}
await using (SqlCommand version = connectionFactory.CreateCommand(
connection,
$"""
MERGE [{schema}].[{SqlServerSecretsSchema.SchemaVersionTable}] AS target
USING (SELECT 1 AS id, @version AS version, @applied_utc AS applied_utc) AS source
ON target.id = source.id
WHEN MATCHED THEN
UPDATE SET version = source.version, applied_utc = source.applied_utc
WHEN NOT MATCHED THEN
INSERT (id, version, applied_utc) VALUES (1, source.version, source.applied_utc);
""",
transaction))
{
version.Parameters.AddWithValue("@version", SqlServerSecretsSchema.CurrentVersion);
version.Parameters.AddWithValue("@applied_utc", DateTimeOffset.UtcNow.ToString("O"));
await version.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false);
}
await transaction.CommitAsync(cancellationToken).ConfigureAwait(false);
}
private static async Task<int> ReadExistingSchemaVersionAsync(
SqlConnection connection,
SqlTransaction transaction,
string schema,
CancellationToken cancellationToken)
{
await using SqlCommand command = connection.CreateCommand();
command.Transaction = transaction;
// OBJECT_ID returns NULL for an absent table, so this reads 0 on a virgin database without
// needing the table to exist first.
command.CommandText = $"""
IF OBJECT_ID(N'[{schema}].[{SqlServerSecretsSchema.SchemaVersionTable}]', N'U') IS NULL
SELECT 0;
ELSE
SELECT ISNULL((SELECT version FROM [{schema}].[{SqlServerSecretsSchema.SchemaVersionTable}] WHERE id = 1), 0);
""";
object? version = await command.ExecuteScalarAsync(cancellationToken).ConfigureAwait(false);
return version is null || version == DBNull.Value
? 0
: Convert.ToInt32(version, CultureInfo.InvariantCulture);
}
}
@@ -0,0 +1,39 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<GenerateDocumentationFile>true</GenerateDocumentationFile>
</PropertyGroup>
<PropertyGroup>
<IsPackable>true</IsPackable>
<PackageId>ZB.MOM.WW.Secrets.Replicator.SqlServer</PackageId>
<Authors>ZB.MOM.WW</Authors>
<Description>Shared SQL-Server secret store and hub replication for the ZB.MOM.WW SCADA family — cluster-wide, ciphertext-only secrets.</Description>
<PackageProjectUrl>https://gitea.dohertylan.com/dohertj2/zb-mom-ww-secrets</PackageProjectUrl>
<RepositoryUrl>https://gitea.dohertylan.com/dohertj2/zb-mom-ww-secrets</RepositoryUrl>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\ZB.MOM.WW.Secrets\ZB.MOM.WW.Secrets.csproj" />
<ProjectReference Include="..\ZB.MOM.WW.Secrets.Abstractions\ZB.MOM.WW.Secrets.Abstractions.csproj" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Data.SqlClient" />
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" />
<PackageReference Include="Microsoft.Extensions.Options" />
<PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions" />
<PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" />
<PackageReference Include="Microsoft.Extensions.Hosting.Abstractions" />
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" />
</ItemGroup>
<ItemGroup>
<InternalsVisibleTo Include="ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests" />
</ItemGroup>
</Project>