dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(localdb): migrate legacy store-and-forward.db into the consolidated DB

Browse Source 
Task 8. Adds ResolveStoreAndForwardPath + MigrateStoreAndForward, wired as a
third call in Migrate alongside the two Phase 1 files.

Unlike the Phase 1 paths, the store-and-forward default sits INSIDE the data
volume, so a real deployment has a real file here holding undelivered
messages. This migration genuinely moves data; losing it would discard exactly
the buffered calls store-and-forward exists to protect.

No id synthesis: sf_messages.id is already a caller-assigned TEXT primary key,
so INSERT OR IGNORE is idempotent across a crash-then-rerun.

The copy intersects the legacy column set with the current one rather than
naming all 16 columns outright. A file from an older build predates
execution_id / parent_execution_id / last_attempt_at_ms, and naming a missing
column throws 'no such column' — which the existing reader treats as an
unrecognised shape and silently discards every row. A required-column (PK)
guard keeps that tolerance from degrading into copying NULL-keyed rows.

Four tests: copy+rename, crash-before-rename idempotence, the __localdb_oplog
assertion that catches migrate-before-register, and the older-column-set case.
All four verified to fail without the Migrate call.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
2026-07-20 03:32:10 -04:00
parent f8aa02e2a9
commit bdc0dffea2
2 changed files with 301 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs
+165 -3
View File
@@ -5,8 +5,9 @@ using ZB.MOM.WW.LocalDb;
namespace ZB.MOM.WW.ScadaBridge.Host;
/// <summary>
/// One-time copy of the pre-Phase-1 site databases (<c>site-tracking.db</c> and
/// <c>site_events.db</c>) into the consolidated <c>ZB.MOM.WW.LocalDb</c> database.
/// One-time copy of the pre-consolidation site databases — Phase 1's
/// <c>site-tracking.db</c> and <c>site_events.db</c>, and Phase 2's
/// <c>store-and-forward.db</c> — into the consolidated <c>ZB.MOM.WW.LocalDb</c> database.
/// </summary>
/// <remarks>
/// <para>
@@ -31,13 +32,20 @@ namespace ZB.MOM.WW.ScadaBridge.Host;
/// half-migrated state to reason about. A second boot sees the renamed file and no-ops.
/// </para>
/// <para>
/// <b>Finding nothing is the expected case on the docker rig.</b> Neither legacy config key
/// <b>Finding nothing is the expected case on the docker rig for the two Phase 1 files.</b>
/// Neither legacy config key
/// is set in any rig appsettings, so both fall back to CWD-relative code defaults
/// (<c>/app/site-tracking.db</c>, <c>/app/site_events.db</c>) that sit OUTSIDE the mounted
/// data volume — meaning they were already being discarded on every container recreate.
/// Phase 1 incidentally fixes that data-loss bug by consolidating into
/// <c>/app/data/site-localdb.db</c>. A no-op here is a legitimate result, not a failure.
/// </para>
/// <para>
/// <b>Store-and-forward is the exception.</b> Its default path <i>is</i> inside the data
/// volume (<c>./data/store-and-forward.db</c>), so a real deployment has a real file there
/// holding undelivered messages. That migration genuinely moves data, and dropping it would
/// silently discard exactly the buffered calls store-and-forward exists to protect.
/// </para>
/// </remarks>
public static class SiteLocalDbLegacyMigrator
{
@@ -49,6 +57,22 @@ public static class SiteLocalDbLegacyMigrator
/// <summary>Default legacy event-log path (<c>SiteEventLogOptions.DatabasePath</c>).</summary>
private const string DefaultEventLogPath = "site_events.db";
/// <summary>Default legacy store-and-forward path (<c>StoreAndForwardOptions.SqliteDbPath</c>).</summary>
private const string DefaultStoreAndForwardPath = "./data/store-and-forward.db";
/// <summary>
/// Every column of the current <c>sf_messages</c> schema, in a fixed order. Columns
/// absent from an older legacy file are dropped from the copy rather than failing it —
/// see <see cref="PresentColumns"/>.
/// </summary>
private static readonly string[] StoreAndForwardColumns =
[
"id", "category", "target", "payload_json",
"retry_count", "max_retries", "retry_interval_ms",
"created_at", "last_attempt_at", "status", "last_error", "origin_instance",
"execution_id", "source_script", "parent_execution_id", "last_attempt_at_ms",
];
/// <summary>
/// Copies any legacy site databases into <paramref name="db"/>, then renames them.
/// </summary>
@@ -63,6 +87,7 @@ public static class SiteLocalDbLegacyMigrator
MigrateTracking(db, ResolveTrackingPath(config));
MigrateEvents(db, ResolveEventLogPath(config), nodeName);
MigrateStoreAndForward(db, ResolveStoreAndForwardPath(config));
}
/// <summary>
@@ -113,6 +138,86 @@ public static class SiteLocalDbLegacyMigrator
return Path.GetFullPath(path);
}
/// <summary>
/// Resolves the legacy store-and-forward database path from the OLD key, falling back
/// to the code default. Unlike the two Phase 1 paths, this default sits INSIDE the
/// mounted data volume (<c>./data/</c>), so on the docker rig there is a real file here
/// with real buffered messages — this migration is not the usual no-op.
/// </summary>
internal static string ResolveStoreAndForwardPath(IConfiguration config)
{
var path = config["ScadaBridge:StoreAndForward:SqliteDbPath"] ?? DefaultStoreAndForwardPath;
if (string.IsNullOrWhiteSpace(path) ||
path.Equals(":memory:", StringComparison.OrdinalIgnoreCase) ||
path.StartsWith("file:", StringComparison.OrdinalIgnoreCase))
{
return string.Empty;
}
return Path.GetFullPath(path);
}
/// <summary>
/// Copies buffered store-and-forward messages out of the legacy file.
/// </summary>
/// <remarks>
/// No id synthesis, unlike <see cref="MigrateEvents"/>: <c>sf_messages.id</c> is already
/// a caller-assigned TEXT primary key, so <c>INSERT OR IGNORE</c> is naturally idempotent
/// across a crash-then-rerun.
/// <para>
/// These are undelivered messages, so dropping them is real data loss — a buffered call
/// that never reaches its external system is exactly what store-and-forward exists to
/// prevent. That is why the copy tolerates an older column set rather than bailing.
/// </para>
/// </remarks>
private static void MigrateStoreAndForward(ILocalDb db, string legacyPath)
=> MigrateTable(db, legacyPath, "sf_messages", "id", StoreAndForwardColumns);
/// <summary>
/// Copies one table from a legacy file into the consolidated database, then renames the
/// legacy file.
/// </summary>
/// <remarks>
/// The copy is restricted to the columns the legacy table <i>actually has</i>. A file
/// written by an older build predates some columns, and naming a missing column in the
/// SELECT would throw "no such column" — which the reader treats as an unrecognised
/// shape, silently discarding every row in the table. Intersecting first means an old
/// file migrates its data and simply leaves the newer columns at their schema defaults.
/// </remarks>
/// <param name="db">The consolidated database.</param>
/// <param name="legacyPath">The legacy file, which may not exist.</param>
/// <param name="table">Table name, identical on both sides.</param>
/// <param name="requiredColumn">
/// A column without which the table is not the one we mean — normally the primary key.
/// Copying rows with a NULL PK would be worse than copying nothing.
/// </param>
/// <param name="columns">The current schema's full column list, in a fixed order.</param>
private static void MigrateTable(
ILocalDb db, string legacyPath, string table, string requiredColumn, IReadOnlyList<string> columns)
{
if (!ShouldMigrate(legacyPath)) return;
using (var legacy = OpenLegacyReadOnly(legacyPath))
{
var present = PresentColumns(legacy, table, columns);
// An absent table probes as zero columns, so this one guard covers both "old
// file predating the table" and "file we do not recognise".
if (present.Contains(requiredColumn))
{
using var connection = db.CreateConnection();
using var transaction = connection.BeginTransaction();
CopyRows(legacy, connection, transaction, table, present);
transaction.Commit();
}
}
MarkMigrated(legacyPath);
}
private static void MigrateTracking(ILocalDb db, string legacyPath)
{
if (!ShouldMigrate(legacyPath)) return;
@@ -249,6 +354,63 @@ public static class SiteLocalDbLegacyMigrator
return rows;
}
private static SqliteConnection OpenLegacyReadOnly(string legacyPath)
{
var connection = new SqliteConnection($"Data Source={legacyPath};Mode=ReadOnly");
connection.Open();
return connection;
}
/// <summary>
/// Returns the subset of <paramref name="wanted"/> that the legacy table actually has,
/// in the caller's order. An absent table yields an empty list rather than throwing.
/// </summary>
private static List<string> PresentColumns(
SqliteConnection legacy, string table, IReadOnlyList<string> wanted)
{
var present = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
using (var probe = legacy.CreateCommand())
{
// Table name is a caller-controlled constant, never user input — safe to
// interpolate (parameters are not permitted as a pragma-function argument).
probe.CommandText = $"SELECT name FROM pragma_table_info('{table}')";
using var reader = probe.ExecuteReader();
while (reader.Read()) present.Add(reader.GetString(0));
}
return [.. wanted.Where(present.Contains)];
}
/// <summary>Streams every row of <paramref name="columns"/> from the legacy table into the target.</summary>
private static void CopyRows(
SqliteConnection legacy,
SqliteConnection target,
SqliteTransaction transaction,
string table,
IReadOnlyList<string> columns)
{
var columnList = string.Join(", ", columns);
var parameterList = string.Join(", ", columns.Select((_, i) => $"$p{i}"));
using var read = legacy.CreateCommand();
read.CommandText = $"SELECT {columnList} FROM {table};";
using var reader = read.ExecuteReader();
while (reader.Read())
{
using var write = target.CreateCommand();
write.Transaction = transaction;
write.CommandText =
$"INSERT OR IGNORE INTO {table} ({columnList}) VALUES ({parameterList});";
for (var i = 0; i < columns.Count; i++)
write.Parameters.AddWithValue($"$p{i}", reader.IsDBNull(i) ? DBNull.Value : reader.GetValue(i));
write.ExecuteNonQuery();
}
}
private static void Bind(SqliteCommand cmd, object?[] row)
{
for (var i = 0; i < row.Length; i++)
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs
+136 -2
View File
@@ -40,7 +40,14 @@ public class SiteLocalDbLegacyMigratorTests : IDisposable
private string Path_(string name) => System.IO.Path.Combine(_root, name);
/// <summary>A consolidated database with both tables created and registered, as the host has it.</summary>
/// <summary>A consolidated database with the tables created and registered, as the host has it.</summary>
/// <remarks>
/// <c>sf_messages</c> is registered here even though production <c>OnReady</c> does not
/// register it until the Task 14 cutover. The oplog assertion below needs live capture
/// triggers to mean anything, and the property under test — that the migrator runs after
/// registration — is the same either way. The corollary is a real constraint on Task 14:
/// <c>Migrate</c> must stay the LAST call in <c>OnReady</c>, after every registration.
/// </remarks>
private ILocalDb CreateConsolidated(IConfiguration config)
{
var provider = new ServiceCollection()
@@ -49,8 +56,10 @@ public class SiteLocalDbLegacyMigratorTests : IDisposable
using var connection = db.CreateConnection();
ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingSchema.Apply(connection);
ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogSchema.Apply(connection);
ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardSchema.Apply(connection);
db.RegisterReplicated("OperationTracking");
db.RegisterReplicated("site_events");
db.RegisterReplicated("sf_messages");
})
.BuildServiceProvider();
_providers.Add(provider);
@@ -59,12 +68,22 @@ public class SiteLocalDbLegacyMigratorTests : IDisposable
}
private IConfiguration Config(
string? trackingPath = null, string? eventsPath = null, string nodeName = "node-a")
string? trackingPath = null,
string? eventsPath = null,
string nodeName = "node-a",
string? storeAndForwardPath = null)
{
var values = new Dictionary<string, string?>
{
["LocalDb:Path"] = Path_("consolidated.db"),
["ScadaBridge:Node:NodeName"] = nodeName,
// Always pinned inside the test's own directory, even when a test does not care
// about it. Left unset, the resolver falls back to the CWD-relative code default
// "./data/store-and-forward.db" — and the test run's CWD is the test binary's
// output directory, so an unlucky run could migrate (and RENAME) a real file.
["ScadaBridge:StoreAndForward:SqliteDbPath"] =
storeAndForwardPath ?? Path_("absent-store-and-forward.db"),
};
if (trackingPath is not null)
@@ -75,6 +94,28 @@ public class SiteLocalDbLegacyMigratorTests : IDisposable
return new ConfigurationBuilder().AddInMemoryCollection(values).Build();
}
/// <summary>Seeds a legacy store-and-forward file with the CURRENT column set.</summary>
private static void SeedLegacyStoreAndForward(string path, params string[] ids)
{
using var connection = new SqliteConnection($"Data Source={path}");
connection.Open();
ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardSchema.Apply(connection);
foreach (var id in ids)
{
using var cmd = connection.CreateCommand();
cmd.CommandText = """
INSERT INTO sf_messages (
id, category, target, payload_json, retry_count, max_retries,
retry_interval_ms, created_at, status, execution_id)
VALUES ($id, 0, 'ERP', '{"order":1}', 0, 50, 30000, $now, 0, 'exec-1');
""";
cmd.Parameters.AddWithValue("$id", id);
cmd.Parameters.AddWithValue("$now", DateTime.UtcNow.ToString("o"));
cmd.ExecuteNonQuery();
}
}
private static void SeedLegacyTracking(string path, params string[] ids)
{
using var connection = new SqliteConnection($"Data Source={path}");
@@ -261,6 +302,99 @@ public class SiteLocalDbLegacyMigratorTests : IDisposable
Assert.Equal(2L, (long)cmd.ExecuteScalar()!);
}
[Fact]
public void SfMessages_AreCopiedFromTheLegacyFile()
{
// Phase 2 (Task 8). Unlike the two Phase 1 files, the store-and-forward default path
// is inside the data volume, so on a real deployment this actually moves data:
// undelivered messages that a lost migration would silently discard.
var sfPath = Path_("store-and-forward.db");
SeedLegacyStoreAndForward(sfPath, "msg-1", "msg-2", "msg-3");
var config = Config(storeAndForwardPath: sfPath);
var db = CreateConsolidated(config);
SiteLocalDbLegacyMigrator.Migrate(db, config);
Assert.Equal(3, CountRows(db, "sf_messages"));
Assert.Equal(["msg-1", "msg-2", "msg-3"], SelectIds(db, "sf_messages", "id"));
// Renamed, not deleted — same contract as the Phase 1 files.
Assert.False(File.Exists(sfPath));
Assert.True(File.Exists(sfPath + ".migrated"));
}
[Fact]
public void SfMessages_Migration_IsIdempotent_WhenRerunAfterACrashBeforeRename()
{
// The crash window: copy committed, rename did not. sf_messages.id is a natural
// TEXT key, so INSERT OR IGNORE absorbs the re-copy with no id synthesis needed.
var sfPath = Path_("store-and-forward.db");
SeedLegacyStoreAndForward(sfPath, "msg-1", "msg-2");
var config = Config(storeAndForwardPath: sfPath);
var db = CreateConsolidated(config);
SiteLocalDbLegacyMigrator.Migrate(db, config);
File.Move(sfPath + ".migrated", sfPath);
SiteLocalDbLegacyMigrator.Migrate(db, config);
Assert.Equal(2, CountRows(db, "sf_messages"));
}
[Fact]
public void MigratedSfMessages_EnterTheOplog_SoTheyActuallyReplicate()
{
// Same ordering trap as the Phase 1 tables: migrate before RegisterReplicated and
// the rows never enter the oplog, never reach the peer, and nothing errors. Only an
// assertion on __localdb_oplog catches the ordering being reversed.
var sfPath = Path_("store-and-forward.db");
SeedLegacyStoreAndForward(sfPath, "msg-1", "msg-2");
var config = Config(storeAndForwardPath: sfPath);
var db = CreateConsolidated(config);
SiteLocalDbLegacyMigrator.Migrate(db, config);
using var connection = db.CreateConnection();
using var cmd = connection.CreateCommand();
cmd.CommandText = "SELECT COUNT(*) FROM __localdb_oplog WHERE table_name = 'sf_messages'";
Assert.Equal(2L, (long)cmd.ExecuteScalar()!);
}
[Fact]
public void SfMessages_FromAnOlderBuild_MigrateDespiteMissingColumns()
{
// A legacy file written before execution_id / parent_execution_id / last_attempt_at_ms
// existed. Naming a missing column in the SELECT throws "no such column", and the
// reader treats that as an unrecognised shape — which would silently discard every
// buffered message. The copy intersects the column sets instead.
var sfPath = Path_("store-and-forward.db");
using (var legacy = new SqliteConnection($"Data Source={sfPath}"))
{
legacy.Open();
using var ddl = legacy.CreateCommand();
ddl.CommandText = """
CREATE TABLE sf_messages (
id TEXT PRIMARY KEY, category INTEGER NOT NULL, target TEXT NOT NULL,
payload_json TEXT NOT NULL, retry_count INTEGER NOT NULL DEFAULT 0,
max_retries INTEGER NOT NULL DEFAULT 50,
retry_interval_ms INTEGER NOT NULL DEFAULT 30000,
created_at TEXT NOT NULL, last_attempt_at TEXT,
status INTEGER NOT NULL DEFAULT 0, last_error TEXT, origin_instance TEXT
);
INSERT INTO sf_messages (id, category, target, payload_json, created_at)
VALUES ('old-1', 0, 'ERP', '{}', '2026-01-01T00:00:00Z');
""";
ddl.ExecuteNonQuery();
}
var config = Config(storeAndForwardPath: sfPath);
var db = CreateConsolidated(config);
SiteLocalDbLegacyMigrator.Migrate(db, config);
Assert.Equal(1, CountRows(db, "sf_messages"));
Assert.Equal(["old-1"], SelectIds(db, "sf_messages", "id"));
}
[Fact]
public void LegacyFileWithoutTheExpectedTable_IsTreatedAsEmpty_NotAFailure()
{