fix(secrets): make concurrent SQLite store initialisation safe; bump to 0.6.4 (unpublished) (scadaproj#25)

Several setups migrating one SQLite secret store at once (mxaccessgw's parallel test hosts on the
per-process store) failed intermittently on Windows with "cannot start a transaction within a
transaction" from SqliteSecretsStoreMigrator.MigrateAsync. That message means the BEGIN ran on a
handle that was already in a transaction: the pinned Microsoft.Data.Sqlite 10.0.7 can hand one
pooled handle to two owners (dotnet/efcore#39009, fixed in 10.0.13) and no 10.0.x refuses to
re-pool a handle whose rollback failed (dotnet/efcore#39082).

The migrator now serialises migrations of one store per process behind a gate keyed on the
normalised path, runs each attempt on a dedicated unpooled connection inside BEGIN IMMEDIATE, and
retries SQLITE_BUSY/SQLITE_LOCKED (another process past the busy timeout) with equal-jitter
exponential backoff. Store operations and the Microsoft.Data.Sqlite pin are unchanged.

Tests: concurrent-init (16 hosts x 20 rounds on fresh and existing stores, two path spellings,
16 AddZbSecrets containers running the pre-host migrator + hosted service), gate sharing/waiting,
BUSY retry past the 5 s busy timeout, transient-code classification.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 13:57:53 -05:00
parent 01c6dc1d02
commit f21f2e551d
7 changed files with 420 additions and 9 deletions
+23
View File
@@ -5,6 +5,29 @@ follows [Keep a Changelog](https://keepachangelog.com/); versions are shared acr
package in the family (see `Directory.Build.props`) even when a given release only touches
one of them.
## 0.6.4 — unreleased (NOT published to the feed)
### Fixed
- **SQLite store initialisation is safe when several setups run at once on one store**
(scadaproj#25). mxaccessgw's parallel test hosts, each migrating the same per-process store,
failed intermittently on the Windows nightly with `SqliteException: cannot start a transaction
within a transaction` from `SqliteSecretsStoreMigrator.MigrateAsync`. The migrator now:
- serialises migrations of one store within a process behind a process-wide gate keyed on the
normalised store path (`Path.GetFullPath`; case-insensitive on Windows and macOS), however
many factories, migrators or DI containers point at it;
- runs each attempt on a dedicated **unpooled** connection inside `BEGIN IMMEDIATE`, so it can
neither be handed a pooled handle another owner is still using nor leave one behind in a
transaction. The pinned Microsoft.Data.Sqlite `10.0.7` can hand one pooled handle to two
owners (dotnet/efcore#39009, fixed in `10.0.13`), and no `10.0.x` refuses to re-pool a handle
whose rollback failed (dotnet/efcore#39082); both surface as exactly this error on the next
`BEGIN`;
- retries `SQLITE_BUSY` / `SQLITE_LOCKED` (another process holding the write lock past the 5 s
busy timeout) up to 6 attempts on a fresh connection with equal-jitter exponential backoff.
No schema, wire or configuration change; store operations other than the migration are unchanged.
The Microsoft.Data.Sqlite pin stays at `10.0.7` so this patch does not raise the consumer floor.
## 0.6.3 — 2026-09-28
### Added
+2
View File
@@ -4,6 +4,8 @@ Shared encrypted secret store + `${secret:}` resolution for the **ZB.MOM.WW SCAD
**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — currently #7 (ignitionoee router pins), #8 (Replicator.Grpc / `0.6.3` fate), #18 (migrator retry jitter). The history below is a record, not a status tracker; file new follow-ups as issues.
**`0.6.4` — built, NOT published (2026-11-09, scadaproj#25):** `Directory.Build.props` says `0.6.4` but the feed's newest is still `0.6.3`. Concurrent SQLite migration of one store is now gated per normalised path, runs on an unpooled `BEGIN IMMEDIATE` connection and retries `SQLITE_BUSY`/`SQLITE_LOCKED` (see `CHANGELOG.md`). Tests: `tests/ZB.MOM.WW.Secrets.Tests/Sqlite/SqliteSecretsStoreConcurrentInitTests.cs` + the gate/retry cases in `SqliteSecretsStoreMigratorTests.cs`; also run on windev.
## Index history (migrated from the scadaproj root CLAUDE.md)
> Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.**
+1 -1
View File
@@ -5,7 +5,7 @@
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<LangVersion>latest</LangVersion>
<Version>0.6.3</Version>
<Version>0.6.4</Version>
<ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
<PackageReadmeFile>README.md</PackageReadmeFile>
</PropertyGroup>
@@ -24,12 +24,21 @@ public sealed class SecretsSqliteConnectionFactory
_sqlitePath = sqlitePath;
}
/// <summary>
/// The store path made absolute and canonical (<c>..</c> segments resolved), so two factories
/// built from different spellings of one file agree on its identity. Used to key the
/// process-wide migration gate.
/// </summary>
internal string NormalizedPath => Path.GetFullPath(_sqlitePath);
/// <summary>
/// Creates an unopened SQLite connection (Mode=ReadWriteCreate). Prefer
/// <see cref="OpenConnectionAsync"/>, which also applies WAL journaling and the
/// busy timeout.
/// </summary>
public SqliteConnection CreateConnection()
public SqliteConnection CreateConnection() => CreateConnection(pooling: true);
private SqliteConnection CreateConnection(bool pooling)
{
string? directory = Path.GetDirectoryName(_sqlitePath);
@@ -42,7 +51,7 @@ public sealed class SecretsSqliteConnectionFactory
{
DataSource = _sqlitePath,
Mode = SqliteOpenMode.ReadWriteCreate,
Pooling = true,
Pooling = pooling,
DefaultTimeout = (int)BusyTimeout.TotalSeconds,
};
@@ -56,9 +65,25 @@ public sealed class SecretsSqliteConnectionFactory
/// </summary>
/// <param name="cancellationToken">Cancellation token for the operation.</param>
/// <returns>An opened and configured SQLite connection.</returns>
public async Task<SqliteConnection> OpenConnectionAsync(CancellationToken cancellationToken)
public Task<SqliteConnection> OpenConnectionAsync(CancellationToken cancellationToken) =>
OpenAsync(pooling: true, cancellationToken);
/// <summary>
/// Opens a connection that never comes from, and never returns to, the connection pool. The
/// migrator uses it so schema creation cannot be handed a pooled handle another owner is still
/// using, nor leave one behind (scadaproj#25). Microsoft.Data.Sqlite before 10.0.13 can hand one
/// pooled handle to two owners at once (dotnet/efcore#39009), and no 10.0.x release refuses to
/// pool a handle whose rollback failed mid-transaction (dotnet/efcore#39082); either surfaces as
/// "cannot start a transaction within a transaction" on the next <c>BEGIN</c>.
/// </summary>
/// <param name="cancellationToken">Cancellation token for the operation.</param>
/// <returns>An opened, configured, unpooled SQLite connection.</returns>
internal Task<SqliteConnection> OpenUnpooledConnectionAsync(CancellationToken cancellationToken) =>
OpenAsync(pooling: false, cancellationToken);
private async Task<SqliteConnection> OpenAsync(bool pooling, CancellationToken cancellationToken)
{
SqliteConnection connection = CreateConnection();
SqliteConnection connection = CreateConnection(pooling);
try
{
await connection.OpenAsync(cancellationToken).ConfigureAwait(false);
@@ -1,3 +1,4 @@
using System.Collections.Concurrent;
using System.Globalization;
using Microsoft.Data.Sqlite;
using ZB.MOM.WW.Secrets.Abstractions;
@@ -9,9 +10,33 @@ namespace ZB.MOM.WW.Secrets.Sqlite;
/// safe to run repeatedly. Refuses to run against a database whose on-disk version is
/// newer than this build supports.
/// </summary>
/// <remarks>
/// Safe when several initialisations of one store run at once (scadaproj#25 — mxaccessgw's parallel
/// test hosts migrating one per-process store failed on Windows with
/// "cannot start a transaction within a transaction"):
/// <list type="bullet">
/// <item>Migrations of one store inside one process are serialised by a process-wide gate keyed on the
/// normalised store path, however many factories, migrators or DI containers point at it.</item>
/// <item>Each attempt runs on a dedicated, unpooled connection inside <c>BEGIN IMMEDIATE</c>, so it takes the
/// write lock up front and can neither inherit nor leave behind a pooled handle in a transaction.</item>
/// <item>Contention with another process (<c>SQLITE_BUSY</c> / <c>SQLITE_LOCKED</c> after the busy timeout)
/// is retried on a fresh connection with jittered exponential backoff.</item>
/// </list>
/// </remarks>
public sealed class SqliteSecretsStoreMigrator(SecretsSqliteConnectionFactory connectionFactory)
: ISecretsStoreMigrator
{
/// <summary>Total attempts (first try plus retries) on SQLITE_BUSY / SQLITE_LOCKED.</summary>
internal const int MaxAttempts = 6;
private const int SqliteBusy = 5;
private const int SqliteLocked = 6;
private static readonly ConcurrentDictionary<string, SemaphoreSlim> Gates = new(
OperatingSystem.IsWindows() || OperatingSystem.IsMacOS()
? StringComparer.OrdinalIgnoreCase
: StringComparer.Ordinal);
/// <summary>Applies the schema migration to the secret store.</summary>
/// <param name="cancellationToken">Cancellation token.</param>
/// <exception cref="SecretStoreMigrationException">
@@ -19,11 +44,56 @@ public sealed class SqliteSecretsStoreMigrator(SecretsSqliteConnectionFactory co
/// </exception>
public async Task MigrateAsync(CancellationToken cancellationToken)
{
await using SqliteConnection connection =
await connectionFactory.OpenConnectionAsync(cancellationToken).ConfigureAwait(false);
SemaphoreSlim gate = GateFor(connectionFactory.NormalizedPath);
await gate.WaitAsync(cancellationToken).ConfigureAwait(false);
try
{
for (int attempt = 1; ; attempt++)
{
try
{
await MigrateOnceAsync(cancellationToken).ConfigureAwait(false);
return;
}
catch (SqliteException ex) when (IsTransient(ex) && attempt < MaxAttempts)
{
await Task.Delay(Backoff(attempt), cancellationToken).ConfigureAwait(false);
}
}
}
finally
{
gate.Release();
}
}
await using SqliteTransaction transaction =
(SqliteTransaction)await connection.BeginTransactionAsync(System.Data.IsolationLevel.Serializable, cancellationToken).ConfigureAwait(false);
/// <summary>
/// The gate serialising migrations of the store at <paramref name="normalizedPath"/>; exposed so a
/// test can prove two spellings of one path share it.
/// </summary>
internal static SemaphoreSlim GateFor(string normalizedPath) =>
Gates.GetOrAdd(normalizedPath, static _ => new SemaphoreSlim(1, 1));
internal static bool IsTransient(SqliteException ex) =>
(ex.SqliteErrorCode & 0xFF) is SqliteBusy or SqliteLocked;
// Equal-jitter exponential backoff: 100, 200, 400, 800, 1600 ms ceilings, at least half of each,
// so concurrent processes that collided once do not retry in lockstep.
private static TimeSpan Backoff(int attempt)
{
int ceiling = 100 * (1 << Math.Min(attempt - 1, 4));
return TimeSpan.FromMilliseconds((ceiling / 2) + Random.Shared.Next(ceiling / 2 + 1));
}
private async Task MigrateOnceAsync(CancellationToken cancellationToken)
{
await using SqliteConnection connection =
await connectionFactory.OpenUnpooledConnectionAsync(cancellationToken).ConfigureAwait(false);
// deferred: false => BEGIN IMMEDIATE: take the write lock before reading the version, so two
// migrators can never both read "no schema" and then race to create it.
await using SqliteTransaction transaction = connection.BeginTransaction(
System.Data.IsolationLevel.Serializable, deferred: false);
int existingVersion =
await ReadExistingSchemaVersionAsync(connection, transaction, cancellationToken).ConfigureAwait(false);
@@ -0,0 +1,217 @@
using Microsoft.Data.Sqlite;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using ZB.MOM.WW.Secrets.Abstractions;
using ZB.MOM.WW.Secrets.DependencyInjection;
using ZB.MOM.WW.Secrets.Sqlite;
namespace ZB.MOM.WW.Secrets.Tests.Sqlite;
/// <summary>
/// scadaproj#25: several hosts in one process initialising the SAME SQLite secret store at once
/// (mxaccessgw's parallel test classes, each building a host whose migration hosted service and
/// store point at the per-process default path). On Windows this surfaced as
/// "cannot start a transaction within a transaction". Each simulated host owns its own
/// connection factory, migrator and store, exactly as separate DI containers would.
/// </summary>
public sealed class SqliteSecretsStoreConcurrentInitTests : IDisposable
{
private const int Hosts = 16;
private const int Rounds = 20;
private readonly string _root =
Path.Combine(Path.GetTempPath(), $"zb-secrets-concurrent-{Guid.NewGuid():N}");
[Fact]
public async Task ConcurrentInitialisation_OfOneFreshStore_AllHostsSucceed()
{
for (int round = 0; round < Rounds; round++)
{
string path = Path.Combine(_root, $"round-{round}", "secrets.db");
await RunHostsAsync(path, round);
await AssertStoreAsync(path, expectedSecrets: Hosts);
}
}
[Fact]
public async Task ConcurrentInitialisation_OfOneExistingStore_AllHostsSucceed()
{
string path = Path.Combine(_root, "existing", "secrets.db");
await new SqliteSecretsStoreMigrator(new SecretsSqliteConnectionFactory(path))
.MigrateAsync(CancellationToken.None);
for (int round = 0; round < Rounds; round++)
{
await RunHostsAsync(path, round);
}
await AssertStoreAsync(path, expectedSecrets: Hosts * Rounds);
}
[Fact]
public async Task ConcurrentInitialisation_ThroughDifferentSpellingsOfOnePath_AllHostsSucceed()
{
// The per-path gate keys on the NORMALISED path: a relative spelling and a "..\" detour
// must land on the same gate as the canonical absolute path.
string dir = Path.Combine(_root, "spellings");
string canonical = Path.Combine(dir, "secrets.db");
string detour = Path.Combine(dir, "sub", "..", "secrets.db");
string[] spellings = [canonical, detour];
Task[] hosts = new Task[Hosts];
TaskCompletionSource start = new(TaskCreationOptions.RunContinuationsAsynchronously);
for (int i = 0; i < Hosts; i++)
{
string spelling = spellings[i % spellings.Length];
int host = i;
hosts[i] = Task.Run(async () =>
{
await start.Task;
await RunOneHostAsync(spelling, $"spell/{host}");
});
}
start.SetResult();
await Task.WhenAll(hosts);
await AssertStoreAsync(canonical, expectedSecrets: Hosts);
}
[Fact]
public async Task ConcurrentInitialisation_ThroughSeparateAddZbSecretsContainers_AllSucceed()
{
// The exact mxaccessgw shape: every host builds its own container with AddZbSecrets, runs the
// pre-host migrator, then starts the SecretsMigrationHostedService — all on one store path.
string path = Path.Combine(_root, "di", "secrets.db");
IConfiguration configuration = new ConfigurationBuilder()
.AddInMemoryCollection(new Dictionary<string, string?>
{
["Secrets:SqlitePath"] = path,
["Secrets:RunMigrationsOnStartup"] = "true",
})
.Build();
TaskCompletionSource start = new(TaskCreationOptions.RunContinuationsAsynchronously);
Task[] hosts = new Task[Hosts];
for (int i = 0; i < Hosts; i++)
{
hosts[i] = Task.Run(async () =>
{
await using ServiceProvider provider = new ServiceCollection()
.AddZbSecrets(configuration, "Secrets")
.BuildServiceProvider();
await start.Task;
provider.GetRequiredService<SqliteSecretsStoreMigrator>()
.MigrateAsync(default).GetAwaiter().GetResult();
foreach (IHostedService hosted in provider.GetServices<IHostedService>())
{
if (hosted.GetType().Name == "SecretsMigrationHostedService")
{
await hosted.StartAsync(CancellationToken.None);
}
}
_ = await provider.GetRequiredService<ISecretStore>().ListAsync(includeDeleted: true, CancellationToken.None);
});
}
start.SetResult();
await Task.WhenAll(hosts);
await AssertStoreAsync(path, expectedSecrets: 0);
}
private static async Task RunHostsAsync(string path, int round)
{
TaskCompletionSource start = new(TaskCreationOptions.RunContinuationsAsynchronously);
Task[] hosts = new Task[Hosts];
for (int i = 0; i < Hosts; i++)
{
int host = i;
hosts[i] = Task.Run(async () =>
{
await start.Task;
await RunOneHostAsync(path, $"r{round}/h{host}");
});
}
// Release every host at once so their migrations genuinely overlap.
start.SetResult();
await Task.WhenAll(hosts);
}
private static async Task RunOneHostAsync(string path, string secretName)
{
// One "host": its own factory, migrator and store, as a separate DI container would build.
SecretsSqliteConnectionFactory factory = new(path);
await new SqliteSecretsStoreMigrator(factory).MigrateAsync(CancellationToken.None);
SqliteSecretStore store = new(factory);
StoredSecret row = MakeSecret(secretName);
await store.UpsertAsync(row, CancellationToken.None);
await store.ApplyReplicatedAsync(row with { Revision = 5, UpdatedUtc = row.UpdatedUtc.AddSeconds(1) }, CancellationToken.None);
Assert.NotNull(await store.GetAsync(row.Name, CancellationToken.None));
_ = await store.ListAsync(includeDeleted: false, CancellationToken.None);
}
private static async Task AssertStoreAsync(string path, int expectedSecrets)
{
SecretsSqliteConnectionFactory factory = new(path);
await using SqliteConnection connection = await factory.OpenConnectionAsync(CancellationToken.None);
await using SqliteCommand version = connection.CreateCommand();
version.CommandText = "SELECT version FROM schema_version WHERE id = 1;";
Assert.Equal((long)SqliteSecretsSchema.CurrentVersion, Convert.ToInt64(await version.ExecuteScalarAsync()));
await using SqliteCommand count = connection.CreateCommand();
count.CommandText = "SELECT COUNT(*) FROM secret;";
Assert.Equal((long)expectedSecrets, Convert.ToInt64(await count.ExecuteScalarAsync()));
// No pooled connection may be handed back with a transaction still open: a fresh
// BEGIN on a pooled connection is exactly what failed on Windows.
await using SqliteTransaction tx = (SqliteTransaction)
await connection.BeginTransactionAsync(System.Data.IsolationLevel.Serializable);
await tx.CommitAsync();
}
private static StoredSecret MakeSecret(string name) => new()
{
Name = new SecretName(name),
Description = "concurrent-init",
ContentType = SecretContentType.ConnectionString,
Ciphertext = [1, 2, 3],
Nonce = [4, 5, 6],
Tag = [7, 8, 9],
WrappedDek = [10, 11, 12],
WrapNonce = [13, 14, 15],
WrapTag = [16, 17, 18],
KekId = "kek-1",
Revision = 0,
IsDeleted = false,
DeletedUtc = null,
CreatedUtc = DateTimeOffset.UtcNow,
UpdatedUtc = DateTimeOffset.UtcNow,
CreatedBy = "concurrent",
UpdatedBy = "concurrent",
};
public void Dispose()
{
SqliteConnection.ClearAllPools();
try
{
if (Directory.Exists(_root))
{
Directory.Delete(_root, recursive: true);
}
}
catch (IOException)
{
// Best-effort temp cleanup.
}
catch (UnauthorizedAccessException)
{
// Best-effort temp cleanup.
}
}
}
@@ -78,6 +78,80 @@ public sealed class SqliteSecretsStoreMigratorTests : IDisposable
() => migrator.MigrateAsync(CancellationToken.None));
}
[Fact]
public void NormalizedPath_DifferentSpellingsOfOneFile_ShareOneMigrationGate()
{
string dir = Path.GetDirectoryName(_dbPath)!;
string detour = Path.Combine(dir, "zb-detour", "..", Path.GetFileName(_dbPath));
var canonical = new SecretsSqliteConnectionFactory(_dbPath);
var other = new SecretsSqliteConnectionFactory(detour);
Assert.Equal(canonical.NormalizedPath, other.NormalizedPath);
Assert.Same(
SqliteSecretsStoreMigrator.GateFor(canonical.NormalizedPath),
SqliteSecretsStoreMigrator.GateFor(other.NormalizedPath));
Assert.NotSame(
SqliteSecretsStoreMigrator.GateFor(canonical.NormalizedPath),
SqliteSecretsStoreMigrator.GateFor(canonical.NormalizedPath + ".other"));
}
[Fact]
public async Task Migrate_WaitsForTheProcessWideGateOfItsStore()
{
var factory = new SecretsSqliteConnectionFactory(_dbPath);
SemaphoreSlim gate = SqliteSecretsStoreMigrator.GateFor(factory.NormalizedPath);
await gate.WaitAsync();
Task migration;
try
{
// A second factory/migrator pair, as a separate DI container would build.
migration = new SqliteSecretsStoreMigrator(new SecretsSqliteConnectionFactory(_dbPath))
.MigrateAsync(CancellationToken.None);
await Task.Delay(300);
Assert.False(migration.IsCompleted, "migration ran while another migration of the same store held the gate");
}
finally
{
gate.Release();
}
await migration.WaitAsync(TimeSpan.FromSeconds(30));
Assert.Equal(SqliteSecretsSchema.CurrentVersion, await ReadSchemaVersionAsync(factory));
}
[Fact]
public async Task Migrate_RetriesWhenAnotherWriterHoldsTheStorePastTheBusyTimeout()
{
var factory = new SecretsSqliteConnectionFactory(_dbPath);
// Another "process": an unpooled connection holding the write lock (BEGIN IMMEDIATE) for longer
// than the 5 s busy timeout, so the first migration attempt surfaces SQLITE_BUSY.
await using SqliteConnection holder = await factory.OpenUnpooledConnectionAsync(CancellationToken.None);
SqliteTransaction lockTx = holder.BeginTransaction(System.Data.IsolationLevel.Serializable, deferred: false);
Task migration = new SqliteSecretsStoreMigrator(factory).MigrateAsync(CancellationToken.None);
await Task.Delay(TimeSpan.FromSeconds(6.5));
Assert.False(migration.IsCompleted, "migration finished while the write lock was still held");
lockTx.Rollback();
lockTx.Dispose();
await migration.WaitAsync(TimeSpan.FromSeconds(30));
Assert.Equal(SqliteSecretsSchema.CurrentVersion, await ReadSchemaVersionAsync(factory));
}
[Theory]
[InlineData(5, true)] // SQLITE_BUSY
[InlineData(6, true)] // SQLITE_LOCKED
[InlineData(261, true)] // SQLITE_BUSY_RECOVERY (extended code of BUSY)
[InlineData(1, false)] // SQLITE_ERROR, e.g. "cannot start a transaction within a transaction"
[InlineData(11, false)] // SQLITE_CORRUPT
public void IsTransient_OnlyBusyAndLockedAreRetried(int errorCode, bool expected) =>
Assert.Equal(expected, SqliteSecretsStoreMigrator.IsTransient(new SqliteException("x", errorCode)));
private static async Task<long> ReadSchemaVersionAsync(SecretsSqliteConnectionFactory factory)
{
await using SqliteConnection connection = await factory.OpenConnectionAsync(CancellationToken.None);