fix(localdb): registration follows replication config — baseline on enable, deregister self-heal on disable (pair-symmetric)

LocalDbSetup.OnReady registered all four replicated tables unconditionally on
every driver node, replicating or not. That was not merely wasteful (an
unreplicated node paid two INSERTs plus a json_object of the whole row inside
every write transaction, for an oplog nothing ever drained) — it made
baselineExistingRows unusable, so enabling replication on a long-running
default-OFF node converged on SUBSEQUENT WRITES ONLY and the rows already in the
file stayed invisible to the peer, silently and forever.

Registration now follows one predicate, LocalDbSetup.ReplicationIsConfigured:
LocalDb:Replication:PeerAddress OR :ApiKey non-blank. The OR is load-bearing —
only the dialling half of a pair sets PeerAddress, so keying on it alone would
strip capture from every passive node and the pair would converge in one
direction only. SyncListenPort deliberately does not join it: the passive
endpoint is fail-closed on the key, so a port with no ApiKey syncs with nobody.

  configured   => RegisterReplicated(table, baselineExistingRows: true), per
                  table, in the unchanged load-bearing DDL->register order
  unconfigured => DeregisterReplicated(table), an idempotent boot self-heal that
                  drops the capture triggers an earlier build left behind and
                  prunes that table's oplog/row-version rows. One Information
                  line, only when it actually cleaned something.

PAIR SYMMETRY: the sync handshake compares registered-table digests fail-closed,
so flipping either key is a both-nodes-together change — set it on both and
restart both. Documented in CLAUDE.md, docs/Configuration.md and the replication
runbook (which claimed "this repo never calls DeregisterReplicated").

Verified against the pinned LocalDb 0.2.1: RegisterReplicated(string, bool) and
DeregisterReplicated(string) -> bool both exist; baselining seeds
__localdb_row_version at HLC 0 and the deregister return value distinguishes
"cleaned" from "nothing to clean". No pin bump needed.

Tests: LocalDbSetupTests gains pins for both branches (configured via
PeerAddress alone and via ApiKey alone; unconfigured => no tables, zero
triggers; late-enable => an HLC-0 baseline ledger entry; the log line fires once
and only when something was cleaned) — four went red against the old
unconditional code, and all four go red again under a constant-true predicate.
LocalDbWiringTests splits its exact-set pin into a replicating graph and a
default-OFF one. LocalDbPairHarness and AlarmSfLegacyMigratorTests built their
database from LocalDb:Path alone while configuring the replication engine from a
second config — production reads one configuration, so the two are now aligned.
Host.IntegrationTests: 224 passed, 0 failed, 5 skipped.
This commit is contained in:
Joseph Doherty
2026-08-18 07:25:01 -04:00
parent e29fab19f6
commit 3e62f1617b
8 changed files with 542 additions and 93 deletions
@@ -194,8 +194,17 @@ public sealed class AlarmSfLegacyMigratorTests : IDisposable
private ILocalDb BuildDb()
{
// The sync ApiKey is what makes OnReady register the tables for capture at all — since
// 2026-08-18 registration follows the replication config rather than being unconditional
// (see LocalDbSetup.ReplicationIsConfigured). Without it these databases would have no
// capture triggers and Migrated_rows_enter_the_oplog would pin nothing. A passive node
// carries exactly this and no PeerAddress, so it is also the realistic shape.
var configuration = new ConfigurationBuilder()
.AddInMemoryCollection(new Dictionary<string, string?> { ["LocalDb:Path"] = _dbPath })
.AddInMemoryCollection(new Dictionary<string, string?>
{
["LocalDb:Path"] = _dbPath,
["LocalDb:Replication:ApiKey"] = "test-sync-key",
})
.Build();
_provider = new ServiceCollection()
@@ -91,8 +91,8 @@ public sealed class LocalDbPairHarness : IAsyncDisposable
_apiKeyA = apiKeyA ?? DefaultApiKey;
_apiKeyB = apiKeyB ?? DefaultApiKey;
_dbProviderA = BuildDatabaseProvider(_pathA);
_dbProviderB = BuildDatabaseProvider(_pathB);
_dbProviderA = BuildDatabaseProvider(_pathA, _apiKeyA);
_dbProviderB = BuildDatabaseProvider(_pathB, _apiKeyB);
}
/// <summary>Node A — the initiator, which dials the peer.</summary>
@@ -148,7 +148,7 @@ public sealed class LocalDbPairHarness : IAsyncDisposable
DeleteDatabaseFiles(_pathB);
_pathB = Path.Combine(Path.GetTempPath(), $"otopcua-pairB-{Guid.NewGuid():N}.db");
_dbProviderB = BuildDatabaseProvider(_pathB);
_dbProviderB = BuildDatabaseProvider(_pathB, _apiKeyB);
_ = B;
}
@@ -258,10 +258,20 @@ public sealed class LocalDbPairHarness : IAsyncDisposable
/// A provider owning one deployment-cache database, initialised through the host's own
/// <see cref="LocalDbSetup.OnReady"/> — same schema, same registration order.
/// </summary>
private static ServiceProvider BuildDatabaseProvider(string path)
private static ServiceProvider BuildDatabaseProvider(string path, string apiKey)
{
// The ApiKey is not decoration here. Since 2026-08-18 OnReady registers the tables for
// change capture only when this node is CONFIGURED to replicate — PeerAddress or ApiKey
// (LocalDbSetup.ReplicationIsConfigured) — so a database built from `LocalDb:Path` alone
// would come up with no capture triggers and every convergence scenario below would time
// out with nothing to send. Production reads one configuration; this harness happens to
// build the database and the replication engine from two, and they must agree.
var config = new ConfigurationBuilder()
.AddInMemoryCollection(new Dictionary<string, string?> { ["LocalDb:Path"] = path })
.AddInMemoryCollection(new Dictionary<string, string?>
{
["LocalDb:Path"] = path,
["LocalDb:Replication:ApiKey"] = apiKey,
})
.Build();
return new ServiceCollection()
@@ -1,6 +1,8 @@
using Microsoft.Data.Sqlite;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Serilog.Core;
using Serilog.Events;
using Shouldly;
using Xunit;
using ZB.MOM.WW.LocalDb;
@@ -20,49 +22,153 @@ namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
/// every schema-shape test while shipping a replication path that quietly moves nothing.
/// </para>
/// <para>
/// <b>BEHAVIOUR CHANGE (2026-08-18): registration is CONDITIONAL on this node actually
/// being configured to replicate.</b> Until now <c>OnReady</c> registered all four tables
/// unconditionally, so every test here could build a database from <c>LocalDb:Path</c>
/// alone and still find them registered. They now pass a replication section, because that
/// is what makes registration happen at all — see
/// <see cref="OnReady_WhenReplicationIsNotConfigured_RegistersNothingAndInstallsNoTriggers"/>
/// for the other branch. The registered-table-set pin is therefore now a pin on
/// <b>configured</b> nodes; the unconfigured node's pin is the deregistration self-heal.
/// </para>
/// <para>
/// Built against a real temp-file <see cref="ILocalDb"/> rather than a hand-written schema:
/// the library has no in-memory mode, and asserting against a schema the test itself wrote
/// would only prove the test agrees with itself.
/// would only prove the test agrees with itself. That is doubly true here — the baseline and
/// deregistration behaviours live entirely inside the library, so only a real file can show
/// them.
/// </para>
/// </remarks>
public sealed class LocalDbSetupTests : IDisposable
{
/// <summary>
/// The initiating half of a replicating pair: it dials, so it carries <c>PeerAddress</c>.
/// </summary>
private static readonly KeyValuePair<string, string?>[] InitiatorReplication =
[
new("LocalDb:Replication:PeerAddress", "http://peer:9001"),
new("LocalDb:Replication:ApiKey", "test-sync-key"),
];
/// <summary>
/// The passive half of the same pair. It never dials, so it has <b>no</b> <c>PeerAddress</c>
/// — only the shared key (plus a listen port, which is Program.cs's concern). Keying the
/// predicate on <c>PeerAddress</c> alone would strip capture from this node and leave the
/// pair converging in one direction only.
/// </summary>
private static readonly KeyValuePair<string, string?>[] PassiveReplication =
[
new("LocalDb:Replication:ApiKey", "test-sync-key"),
];
private readonly string _dbPath =
Path.Combine(Path.GetTempPath(), $"otopcua-localdb-setup-{Guid.NewGuid():N}.db");
private ServiceProvider? _provider;
private readonly List<ServiceProvider> _providers = [];
private ILocalDb BuildDb()
/// <summary>
/// Boots <see cref="LocalDbSetup.OnReady"/> over the test's database file.
/// </summary>
/// <remarks>
/// Successive calls reopen the <b>same</b> file, which is how the self-heal and late-opt-in
/// cases are expressed: boot once under one replication configuration, write rows, then boot
/// again under another. That is exactly the sequence an operator performs by editing config
/// and restarting the node.
/// </remarks>
private ILocalDb Boot(
KeyValuePair<string, string?>[]? replication = null,
Serilog.ILogger? logger = null)
{
// A previous boot's provider still holds the file open; the library has no second-handle
// mode. Closing it (and the pool) is what makes a reopen behave like a process restart.
DisposeProviders();
var settings = new List<KeyValuePair<string, string?>>
{
new("LocalDb:Path", _dbPath),
};
settings.AddRange(replication ?? []);
var configuration = new ConfigurationBuilder()
.AddInMemoryCollection(new Dictionary<string, string?> { ["LocalDb:Path"] = _dbPath })
.AddInMemoryCollection(settings)
.Build();
_provider = new ServiceCollection()
.AddZbLocalDb(configuration, db => LocalDbSetup.OnReady(db, configuration))
var provider = new ServiceCollection()
.AddZbLocalDb(configuration, db => LocalDbSetup.OnReady(db, configuration, logger))
.BuildServiceProvider();
return _provider.GetRequiredService<ILocalDb>();
_providers.Add(provider);
return provider.GetRequiredService<ILocalDb>();
}
private static async Task<int> ScalarAsync(ILocalDb db, string sql) =>
(await db.QueryAsync(sql, r => r.GetInt32(0), parameters: null, TestContext.Current.CancellationToken))[0];
private static Task InsertPointerAsync(ILocalDb db, string clusterId) =>
db.ExecuteAsync(
"""
INSERT INTO deployment_pointer
(cluster_id, deployment_id, revision_hash, artifact_sha256, applied_at_utc)
VALUES (@ClusterId, @DeploymentId, @RevisionHash, @Sha, @AppliedAtUtc)
""",
new
{
ClusterId = clusterId,
DeploymentId = "0123456789abcdef0123456789abcdef",
RevisionHash = new string('a', 64),
Sha = new string('b', 64),
AppliedAtUtc = "2026-07-20T00:00:00.0000000Z",
},
TestContext.Current.CancellationToken);
[Fact]
public void OnReady_RegistersExactlyTheFourReplicatedTables()
public void OnReady_WhenReplicationIsConfigured_RegistersExactlyTheFourReplicatedTables()
{
// Both directions are load-bearing. "No fewer" catches a dropped RegisterReplicated call
// (that table then never replicates). "No more" catches an accidental registration —
// every replicated table costs three triggers and oplog volume on every write.
var db = BuildDb();
var db = Boot(InitiatorReplication);
db.ReplicatedTables.Keys.OrderBy(k => k, StringComparer.Ordinal)
.ShouldBe(["alarm_condition_state", "alarm_sf_events", "deployment_artifacts", "deployment_pointer"]);
}
[Fact]
public void OnReady_WhenOnlyTheSyncApiKeyIsSet_StillRegistersTheFourTables()
{
// The passive half of a pair carries the key and nothing else. If the predicate were
// "PeerAddress is set", this node would install no capture triggers and its OWN writes
// would never reach the initiator — a pair that replicates in one direction only, with
// nothing logged anywhere. This test is the falsifier for that narrower predicate.
var db = Boot(PassiveReplication);
db.ReplicatedTables.Keys.OrderBy(k => k, StringComparer.Ordinal)
.ShouldBe(["alarm_condition_state", "alarm_sf_events", "deployment_artifacts", "deployment_pointer"]);
}
[Fact]
public async Task OnReady_WhenReplicationIsNotConfigured_RegistersNothingAndInstallsNoTriggers()
{
// The default-OFF node. It has no peer, so capture buys it nothing and costs it two INSERTs
// plus a json_object of the whole row inside every write transaction — for an oplog that is
// never drained. Asserting the trigger count as well as ReplicatedTables matters: the
// dictionary is in-process bookkeeping, the triggers are what actually run.
var db = Boot();
db.ReplicatedTables.ShouldBeEmpty();
// None of the three application schemas defines a trigger of its own, so every trigger in
// this file would be a capture trigger.
(await ScalarAsync(db, "SELECT COUNT(*) FROM sqlite_master WHERE type = 'trigger'")).ShouldBe(0);
}
[Fact]
public void DeploymentArtifacts_PkIsDeploymentIdPlusChunkIndex()
{
// The composite PK is what makes an artifact chunkable. A single-column PK here would
// make every chunk of a deployment collide onto one row under LWW.
var db = BuildDb();
var db = Boot(InitiatorReplication);
db.ReplicatedTables["deployment_artifacts"].PkColumns
.ShouldBe(["deployment_id", "chunk_index"]);
@@ -73,7 +179,7 @@ public sealed class LocalDbSetupTests : IDisposable
{
// One current-deployment pointer per cluster: the pair's two nodes converge on the same
// row rather than each keeping its own.
var db = BuildDb();
var db = Boot(InitiatorReplication);
db.ReplicatedTables["deployment_pointer"].PkColumns.ShouldBe(["cluster_id"]);
}
@@ -84,7 +190,7 @@ public sealed class LocalDbSetupTests : IDisposable
// A single TEXT PK the application mints. The legacy queue this table replaces keyed on
// `RowId INTEGER PRIMARY KEY AUTOINCREMENT`, which LWW cannot replicate: two nodes would
// independently allocate rowid 7 for different alarms and silently overwrite each other.
var db = BuildDb();
var db = Boot(InitiatorReplication);
db.ReplicatedTables["alarm_sf_events"].PkColumns.ShouldBe(["id"]);
}
@@ -94,7 +200,7 @@ public sealed class LocalDbSetupTests : IDisposable
{
// One condition-state row per alarm identity. The pair's two nodes converge on the same row
// under LWW rather than each keeping its own, and Save is an unconditional upsert onto it.
var db = BuildDb();
var db = Boot(InitiatorReplication);
db.ReplicatedTables["alarm_condition_state"].PkColumns.ShouldBe(["scripted_alarm_id"]);
}
@@ -106,7 +212,7 @@ public sealed class LocalDbSetupTests : IDisposable
// Worth its own case because the alarm table is registered by a different call site, and a
// registration added ahead of its DDL would fail loudly while one added after the migrator
// would fail silently.
var db = BuildDb();
var db = Boot(InitiatorReplication);
await db.ExecuteAsync(
"""
@@ -123,13 +229,8 @@ public sealed class LocalDbSetupTests : IDisposable
},
TestContext.Current.CancellationToken);
var oplogRows = await db.QueryAsync(
"SELECT COUNT(*) FROM __localdb_oplog WHERE table_name = 'alarm_sf_events'",
r => r.GetInt32(0),
parameters: null,
TestContext.Current.CancellationToken);
oplogRows[0].ShouldBe(1);
(await ScalarAsync(db, "SELECT COUNT(*) FROM __localdb_oplog WHERE table_name = 'alarm_sf_events'"))
.ShouldBe(1);
}
[Fact]
@@ -138,40 +239,114 @@ public sealed class LocalDbSetupTests : IDisposable
// THE ordering assertion. If DDL and RegisterReplicated were swapped — or a write were
// added to OnReady ahead of registration — the triggers would not exist at write time and
// this count would stay 0 while every other test in this file still passed.
var db = BuildDb();
var db = Boot(InitiatorReplication);
await db.ExecuteAsync(
"""
INSERT INTO deployment_pointer
(cluster_id, deployment_id, revision_hash, artifact_sha256, applied_at_utc)
VALUES (@ClusterId, @DeploymentId, @RevisionHash, @Sha, @AppliedAtUtc)
""",
new
{
ClusterId = "SITE-A",
DeploymentId = "0123456789abcdef0123456789abcdef",
RevisionHash = new string('a', 64),
Sha = new string('b', 64),
AppliedAtUtc = "2026-07-20T00:00:00.0000000Z",
},
TestContext.Current.CancellationToken);
await InsertPointerAsync(db, "SITE-A");
var oplogRows = await db.QueryAsync(
"SELECT COUNT(*) FROM __localdb_oplog",
r => r.GetInt32(0),
parameters: null,
TestContext.Current.CancellationToken);
(await ScalarAsync(db, "SELECT COUNT(*) FROM __localdb_oplog")).ShouldBeGreaterThanOrEqualTo(1);
}
oplogRows[0].ShouldBeGreaterThanOrEqualTo(1);
[Fact]
public async Task EnablingReplicationLater_BaselinesTheRowsAlreadyInTheFile()
{
// The defect this whole change exists for. Capture is change-data-capture and the snapshot
// streamer pages from __localdb_row_version, so rows written while replication was off are
// in neither — a late opt-in used to converge on subsequent writes only, silently and
// forever. RegisterReplicated(table, baselineExistingRows: true) seeds the ledger at the LWW
// floor and flags a resync so they actually travel.
//
// The assertion is on the ledger entry's HLC, not merely on its existence. A row captured by
// a trigger carries a genuine (large, non-zero) stamp; a BASELINED row is seeded at the LWW
// floor of HLC 0, which is what makes it lose to any real remote write of the same key and
// win only where the peer holds no version at all. Asserting existence alone would pass
// against the old unconditional registration, where the trigger simply captured the write —
// i.e. it would prove nothing about the flag.
var unreplicated = Boot();
await InsertPointerAsync(unreplicated, "SITE-A");
var replicated = Boot(InitiatorReplication);
(await ScalarAsync(
replicated,
"""
SELECT COUNT(*) FROM __localdb_row_version
WHERE table_name = 'deployment_pointer' AND hlc = 0
"""))
.ShouldBe(1);
}
[Fact]
public async Task DisablingReplication_DropsTheCaptureTriggersAnEarlierBootInstalled()
{
// The self-heal half. Merely declining to register would leave a file registered by an
// earlier build capturing forever — on precisely the node that has no peer to capture for.
// DeregisterReplicated drops the triggers and prunes that table's oplog and row-version
// rows.
var replicated = Boot(InitiatorReplication);
await InsertPointerAsync(replicated, "SITE-A");
(await ScalarAsync(replicated, "SELECT COUNT(*) FROM sqlite_master WHERE type = 'trigger'"))
.ShouldBe(12);
var unreplicated = Boot();
unreplicated.ReplicatedTables.ShouldBeEmpty();
(await ScalarAsync(unreplicated, "SELECT COUNT(*) FROM sqlite_master WHERE type = 'trigger'"))
.ShouldBe(0);
}
[Fact]
public async Task DisablingReplication_LogsOnceWhenItActuallyCleanedSomething()
{
// An operator-visible record that a node just stopped capturing, and how much it dropped.
// Silent trigger removal on a node someone MEANT to replicate is the failure this line
// exists to make diagnosable.
var replicated = Boot(InitiatorReplication);
await InsertPointerAsync(replicated, "SITE-A");
var sink = new CollectingSink();
Boot(replication: null, logger: BuildLogger(sink));
sink.Events.Count(e => e.Level == LogEventLevel.Information).ShouldBe(1);
}
[Fact]
public void DisablingReplication_SaysNothingWhenThereWasNothingToClean()
{
// The steady state of every default-OFF node in the fleet: it boots unreplicated every day
// and has nothing to deregister. A log line here would be a daily false alarm, which is how
// the real one gets ignored.
var sink = new CollectingSink();
Boot(replication: null, logger: BuildLogger(sink));
sink.Events.ShouldBeEmpty();
}
private static Serilog.ILogger BuildLogger(ILogEventSink sink) =>
new Serilog.LoggerConfiguration().MinimumLevel.Verbose().WriteTo.Sink(sink).CreateLogger();
/// <summary>Captures whatever <see cref="LocalDbSetup.OnReady"/> logs.</summary>
private sealed class CollectingSink : ILogEventSink
{
public List<LogEvent> Events { get; } = [];
public void Emit(LogEvent logEvent) => Events.Add(logEvent);
}
private void DisposeProviders()
{
foreach (var provider in _providers)
provider.Dispose();
_providers.Clear();
// No in-memory mode, so these are real files. Pooled connections keep a handle open past
// dispose; clearing the pools first is what lets a reopen (or the delete below) succeed.
SqliteConnection.ClearAllPools();
}
public void Dispose()
{
_provider?.Dispose();
// No in-memory mode, so these are real files. Pooled connections keep a handle open past
// dispose; clearing the pools first is what makes the delete actually succeed.
SqliteConnection.ClearAllPools();
DisposeProviders();
foreach (var path in new[] { _dbPath, $"{_dbPath}-wal", $"{_dbPath}-shm" })
{
@@ -45,16 +45,27 @@ public sealed class LocalDbWiringTests : IDisposable
/// <see cref="LocalDbRegistration.AddOtOpcUaLocalDb"/>, health, and observability methods,
/// over a temp-file <c>LocalDb:Path</c>.
/// </summary>
private IServiceProvider BuildDriverGraph()
/// <param name="replicating">
/// Whether this node carries a replication configuration. Since 2026-08-18 that is what
/// decides whether <c>LocalDbSetup.OnReady</c> registers the tables for change capture at
/// all, so the two graphs are genuinely different nodes and both are pinned below. Most of
/// the fleet is the default-OFF one.
/// </param>
private IServiceProvider BuildDriverGraph(bool replicating = false)
{
var dbPath = Path.Combine(Path.GetTempPath(), $"otopcua-wiring-{Guid.NewGuid():N}.db");
_dbPaths.Add(dbPath);
var builder = WebApplication.CreateBuilder(new WebApplicationOptions { Args = [] });
builder.Configuration.AddInMemoryCollection(new Dictionary<string, string?>
var settings = new Dictionary<string, string?>
{
["LocalDb:Path"] = dbPath,
});
};
if (replicating)
settings["LocalDb:Replication:ApiKey"] = "wiring-test-sync-key";
var builder = WebApplication.CreateBuilder(new WebApplicationOptions { Args = [] });
builder.Configuration.AddInMemoryCollection(settings);
// The exact LocalDb composition Program.cs runs under hasDriver.
builder.Services.AddOtOpcUaLocalDb(builder.Configuration);
@@ -86,9 +97,9 @@ public sealed class LocalDbWiringTests : IDisposable
}
[Fact]
public void DriverGraph_LocalDbResolvesAsASingletonWithEveryReplicatedTableRegistered()
public void ReplicatingDriverGraph_LocalDbResolvesAsASingletonWithEveryReplicatedTableRegistered()
{
var sp = BuildDriverGraph();
var sp = BuildDriverGraph(replicating: true);
var first = sp.GetService<ILocalDb>();
first.ShouldNotBeNull();
@@ -100,6 +111,22 @@ public sealed class LocalDbWiringTests : IDisposable
.ShouldBe(["alarm_condition_state", "alarm_sf_events", "deployment_artifacts", "deployment_pointer"]);
}
[Fact]
public void DefaultOffDriverGraph_LocalDbStillResolvesButRegistersNoTables()
{
// BEHAVIOUR CHANGE (2026-08-18): registration follows the replication config. This graph is
// most of the fleet — central and site-b are default-OFF — and it must still get a working
// local database (the artifact cache and the alarm buffer both depend on it) while paying
// for none of the change-capture machinery it has no peer to send to.
var sp = BuildDriverGraph();
var db = sp.GetService<ILocalDb>();
db.ShouldNotBeNull();
sp.GetService<IDeploymentArtifactCache>().ShouldNotBeNull();
db.ReplicatedTables.ShouldBeEmpty();
}
[Fact]
public void DriverGraph_ArtifactCacheResolvesToTheLocalDbBackedImplementation()
{