fix(siteeventlog)+fix(host): purge startup yield + auto_vacuum reality (arch-review 2026-08-19 P4-5 + P4-6)

P4-5: EventLogPurgeService.ExecuteAsync now yields (Task.Yield()) before running the
initial RunPurge(), making "the purge never runs synchronously on the caller's thread"
a property of the method itself rather than an artifact of .NET 8+'s BackgroundService
scheduling (verified empirically not to reproduce the literal "StartAsync blocks" symptom
on this repo's net10.0 target — a 1.5s synchronous Thread.Sleep in ExecuteAsync still lets
StartAsync return in ~2ms). Proven instead via a reflection-invoked ExecuteAsync test that
bypasses BackgroundService's Task.Run wrapper, which is timing-independent. Also: a
throwing isActive predicate now fails CLOSED (logged as inactive, at Warning) instead of
silently claiming "active" — the predicate only affects the tick's log line since P4-4,
but a throw must not masquerade as a successful check.

P4-6: PRAGMA auto_vacuum = INCREMENTAL only takes effect on a table-less database, and
LocalDbSchema.EnsureCreated (inside SqliteLocalDb's constructor) always creates LocalDb's
own bookkeeping tables before SiteEventLogSchema.Apply ever gets a connection — so the
pragma there had never once taken effect, in any real call path. Real fix per plan
decision 2: SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile pre-creates a genuinely fresh
LocalDb file with the pragma set, called from SiteServiceRegistration.Configure BEFORE
AddZbLocalDb's lazy singleton ever constructs SqliteLocalDb — no LocalDb API change
needed. Only helps NEW site databases; an existing file (every node on the docker rig
today) is left untouched rather than VACUUM'd, per the plan's explicit rejection of an
unconditional VACUUM against a live ~1 GB file. The now-dead pragma call in
SiteEventLogSchema.Apply is removed and its comment corrected.

Tests proven to fail against the unfixed code (git-diff-and-restore, not stash — this repo
shares one stash ref across all worktrees): ExecuteAsync_YieldsBeforeRunningTheInitial-
PurgeSynchronously and RunPurge_ActiveNodeCheckThrows_TickLogsAsInactive_NotActive fail
without the EventLogPurgeService fix; SiteLocalDbAutoVacuumTests fails to compile without
EnsureAutoVacuumOnFreshFile. SiteEventLogging.Tests 112/112 and Host.Tests 526/526 green
with the fix applied.
This commit is contained in:
Joseph Doherty
2026-08-19 16:31:57 -04:00
parent 99da32d635
commit 11675285af
6 changed files with 387 additions and 15 deletions
@@ -1,3 +1,4 @@
using Microsoft.Data.Sqlite;
using Microsoft.Extensions.Configuration;
using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.ScadaBridge.SiteEventLogging;
@@ -234,4 +235,69 @@ public static class SiteLocalDbSetup
return !string.IsNullOrWhiteSpace(section["PeerAddress"])
|| !string.IsNullOrWhiteSpace(section["ApiKey"]);
}
/// <summary>
/// Sets <c>auto_vacuum = INCREMENTAL</c> on the consolidated LocalDb file — arch-review
/// 2026-08-19 P4-6 — but ONLY when that file does not exist yet.
/// </summary>
/// <remarks>
/// <para>
/// <b>The defect this closes.</b> SQLite only honors a change to <c>auto_vacuum</c> while
/// a database is still completely table-less; once one table exists the pragma is a
/// silent no-op. <c>SqliteLocalDb</c>'s constructor runs <c>LocalDbSchema.EnsureCreated</c>
/// on its own master connection before <see cref="OnReady"/> — or anything else — ever
/// gets a connection to the file, so the pragma <see cref="SiteEventLogSchema.Apply"/>
/// used to issue had never once taken effect, in any real call path in this repo: by the
/// time it ran, LocalDb's own bookkeeping tables already existed.
/// </para>
/// <para>
/// <b>The fix, and why it needs no LocalDb API change.</b> <c>AddZbLocalDb</c> registers
/// <see cref="ILocalDb"/> as a LAZY singleton — nothing touches the file until it is
/// first resolved. This method is called from <c>SiteServiceRegistration.Configure</c>,
/// synchronously, BEFORE <c>services.AddZbLocalDb(...)</c> is even reached in that method
/// — i.e. well before the host is built and anything could resolve <see cref="ILocalDb"/>
/// — so it is free to open its own short-lived connection to the not-yet-existing file and
/// set the pragma there. Empirically verified (not merely inferred from the pragma's
/// documented "unless a subsequent operation modifies the database" caveat): opening a
/// connection to a fresh path and executing this single PRAGMA is sufficient by itself —
/// Microsoft.Data.Sqlite persists it to the file header on close, with no throwaway
/// CREATE/DROP TABLE and no VACUUM required. A LATER session — <c>SqliteLocalDb</c>'s own
/// constructor, opening the SAME path — sees the flag already baked in and every table it
/// (and everyone downstream) subsequently creates lives under incremental-vacuum mode.
/// </para>
/// <para>
/// <b>Only helps FRESH files — say so honestly, always.</b> A file that already exists
/// already has tables, so this method deliberately does nothing to it: retrofitting
/// auto_vacuum onto a populated database needs SQLite's own <c>VACUUM</c> rebuild, which
/// plan decision 2 (2026-08-19) explicitly rejects running unconditionally against a live
/// ~1 GB file on every boot. Consequence: every node on the docker rig today, and any
/// site database created before this fix shipped, stays permanently
/// <c>auto_vacuum=NONE</c> — <c>PRAGMA incremental_vacuum</c>, the per-trim call in
/// <see cref="ZB.MOM.WW.ScadaBridge.SiteEventLogging.EventLogPurgeService.PurgeByStorageCap"/>,
/// remains a harmless no-op there until an operator opts into a one-time offline VACUUM.
/// Only a NEWLY created site database benefits, from the moment this ships onward.
/// </para>
/// </remarks>
/// <param name="config">Configuration, read for <c>LocalDb:Path</c> only.</param>
internal static void EnsureAutoVacuumOnFreshFile(IConfiguration config)
{
var path = config["LocalDb:Path"];
if (string.IsNullOrWhiteSpace(path))
return; // AddZbLocalDb's own options validation reports a missing path loudly; nothing to pre-create here.
if (File.Exists(path))
return; // Already has tables (or is at least a pre-existing file) — the pragma cannot help it now.
var directory = Path.GetDirectoryName(path);
if (!string.IsNullOrEmpty(directory))
Directory.CreateDirectory(directory);
var connectionString = new SqliteConnectionStringBuilder { DataSource = path }.ToString();
using var connection = new SqliteConnection(connectionString);
connection.Open();
using var pragmaCmd = connection.CreateCommand();
pragmaCmd.CommandText = "PRAGMA auto_vacuum = INCREMENTAL";
pragmaCmd.ExecuteNonQuery();
}
}
@@ -91,6 +91,16 @@ public static class SiteServiceRegistration
// failure, and the default site path is the relative "./data/site-localdb.db".
// The shim is gone; SiteLocalDbDirectoryTests still pins the outcome here, since
// what this host needs is the guarantee, not any particular owner of it.
//
// arch-review 2026-08-19 P4-6: MUST run before AddZbLocalDb — its singleton
// factory is lazy, but the ordering still has to be enforced here rather than
// relied upon, because it is what makes "PRAGMA auto_vacuum = INCREMENTAL" take
// effect at all. SQLite only honors that pragma on a table-less database, and
// AddZbLocalDb's own SqliteLocalDb constructor creates LocalDb's bookkeeping
// tables the moment it is first resolved — see
// SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile for the full mechanism, and note
// it only helps a file that does not exist yet.
SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile(config);
services.AddZbLocalDb(config, db => SiteLocalDbSetup.OnReady(db, config));
// The replication engine, likewise unconditional but INERT by default: with no
@@ -164,6 +164,19 @@ public class EventLogPurgeService : BackgroundService
_options.RetentionDays, _options.MaxStorageMb, _options.PurgeInterval,
_options.MaxPurgedRowsPerTick > 0 ? _options.MaxPurgedRowsPerTick.ToString() : "unlimited");
// arch-review 2026-08-19 P4-5: yield BEFORE the initial (potentially ~17s
// ceiling-limited, ~285s unpaced worst case) RunPurge() rather than calling it
// as the first synchronous statement. .NET 8+'s BackgroundService already invokes
// ExecuteAsync via Task.Run, so a synchronous prelude here does not actually block
// IHostedService.StartAsync on this repo's net10.0 target (verified empirically —
// it did not reproduce). This Task.Yield() is still the correct fix rather than a
// no-op: it makes the "purge never blocks the calling thread" contract a property of
// THIS method, not an artifact of the host's scheduling internals — reflection-invoking
// ExecuteAsync directly (as EventLogPurgeServiceTests does, bypassing BackgroundService
// entirely) would otherwise still run the initial purge fully synchronously, and any
// future caller that awaits ExecuteAsync directly (not through StartAsync) would block.
await Task.Yield();
// Run an initial purge on startup
var ceilingHit = RunPurge();
@@ -213,9 +226,18 @@ public class EventLogPurgeService : BackgroundService
}
catch (Exception checkEx)
{
_logger.LogDebug(checkEx,
"Active-node check threw during purge tick; treating as active for logging purposes");
isActive = true;
// arch-review 2026-08-19 P4-5: fail CLOSED, not "active". This predicate no
// longer gates any work (P4-4 — both nodes purge their own file every tick
// regardless), so a wrong value here cannot skip a purge; but silently
// reporting "active" for a check that just threw is dishonest, and an
// operator correlating this line against the real active-node state would be
// misled into thinking the check succeeded. Warning (not Debug) because the
// check itself is broken and is worth noticing on its own.
_logger.LogWarning(checkEx,
"Active-node check threw during purge tick; logging this tick as inactive " +
"rather than claiming an active status the check could not confirm. The purge " +
"itself is unaffected — every node purges its own file regardless of this flag");
isActive = false;
}
_logger.LogDebug("Running event log purge tick (active={IsActive})", isActive);
@@ -513,9 +535,16 @@ public class EventLogPurgeService : BackgroundService
""";
var rows = cmd.ExecuteNonQuery();
// Reclaim free pages so page_count/freelist measurement reflects the
// delete. Effective because auto_vacuum = INCREMENTAL is set at schema
// creation; harmless otherwise.
// Reclaim free pages so page_count/freelist measurement reflects the delete.
// arch-review 2026-08-19 P4-6: this is effective ONLY on a consolidated
// LocalDb file that was FRESH when SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile
// ran (Host/SiteLocalDbSetup.cs) — auto_vacuum can only be enabled before any
// table exists, and SqliteLocalDb's constructor creates LocalDb's own
// bookkeeping tables before this component ever gets a connection. Every file
// that predates that fix — including every node on the docker rig today —
// stays permanently auto_vacuum=NONE, and this PRAGMA is a harmless, silent
// no-op on it (retrofitting one requires an offline VACUUM, deliberately not
// attempted here against a live ~1 GB file — plan decision 2, 2026-08-19).
using var vacuumCmd = connection.CreateCommand();
vacuumCmd.CommandText = "PRAGMA incremental_vacuum";
vacuumCmd.ExecuteNonQuery();
@@ -36,15 +36,24 @@ public static class SiteEventLogSchema
{
ArgumentNullException.ThrowIfNull(connection);
// auto_vacuum must be set before any table is created for it to take effect
// on a fresh database. With INCREMENTAL mode, PRAGMA incremental_vacuum can
// later reclaim free pages so the storage-cap purge can shrink the file.
using (var pragmaCmd = connection.CreateCommand())
{
pragmaCmd.CommandText = "PRAGMA auto_vacuum = INCREMENTAL";
pragmaCmd.ExecuteNonQuery();
}
// arch-review 2026-08-19 P4-6: this method used to also set
// "PRAGMA auto_vacuum = INCREMENTAL" here, on the theory that setting it right
// before CREATE TABLE would make it take effect on a fresh database. That was
// wrong in every real call path: SQLite only honors an auto_vacuum change on a
// database that has NO tables yet, and by the time this method runs — via the
// Host's AddZbLocalDb onReady callback (SiteLocalDbSetup.OnReady) OR via
// SiteEventLogger's own InitializeSchema fallback for directly-constructed
// instances — the consolidated file already holds LocalDb's own bookkeeping
// tables (SqliteLocalDb's constructor runs LocalDbSchema.EnsureCreated on its
// master connection before either caller ever gets a connection). So the pragma
// here had never once taken effect; it was a silent, permanent no-op. The real
// fix now lives in Host/SiteLocalDbSetup.cs
// (EnsureAutoVacuumOnFreshFile), which pre-creates a genuinely fresh LocalDb file
// with the pragma set BEFORE AddZbLocalDb's lazy singleton ever constructs
// SqliteLocalDb — and, being a Host-layer, one-time, pre-file-existence step, it
// helps ONLY brand-new site databases; every file created before this fix shipped
// stays auto_vacuum=NONE forever (see the PurgeByStorageCap remarks in
// EventLogPurgeService for the operational consequence).
using var cmd = connection.CreateCommand();
cmd.CommandText = """
CREATE TABLE IF NOT EXISTS site_events (