LocalDb adoption Phase 1 + 2: consolidate the site database, delete the bespoke replicators #23
Assignees
dohertj2 (dohertj2)
Clear assignees
dohertj2 (dohertj2)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: dohertj2/ScadaBridge#23
Reference in New Issue
Block a user
Delete Branch "feat/localdb-phase2"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Completes the ScadaBridge LocalDb adoption. Phase 1 consolidates
OperationTracking+site_eventsinto oneZB.MOM.WW.LocalDbdatabase; Phase 2 moves insf_messagesand the seven site config tables, then deletes the bespoke replication machinery —SiteReplicationActor,ReplicationMessages, StoreAndForward'sReplicationService,StoreAndForwardStorage.ReplaceAllAsync, and the whole notify-and-fetch exchange. Ten tables now replicate via LocalDb CDC.Three things worth a reviewer's attention:
ReplaceAllAsyncwas unsafe to keep, not merely unused. It was a destructive delete-all-then-insert-all resync; a mass DELETE on a now-replicated table would be captured by CDC and shipped to the peer. LocalDb's snapshot resync merges per row under LWW and never deletes — which is also why the old N1 directional-authority guard is gone rather than ported: there is no wipe left to gate.notification_listsandsmtp_configurationsare created but deliberately NEVER registered. They are permanently empty on a site, and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Pinned by a security-named test and verified live (no CDC triggers on either rig node).037798b3).Test Plan
TreatWarningsAsErrorson)RegisterReplicatedcalls commented out, all four Task 18 scenarios go red (4 failed / 0 passed); restored, 20/20docs/plans/2026-07-19-localdb-phase2-live-gate.mdLive-gate highlights:
Caveats recorded rather than glossed
native_alarm_stateleg was empty live and is covered only offline.Operational constraints (new, documented in
docs/deployment/)LocalDb:Replication:TombstoneRetention(7 d) can resurrect deleted rows on rejoin.LocalDb:Replication:MaxBatchSizebatches by row count against a 4 MB gRPC cap — rig pins 16; the 500 default would allow ~35 MB.Replication remains default-OFF. It is enabled on the docker rig's site-a pair only; no production deployment has it enabled.
Depends on
ZB.MOM.WW.LocalDb0.1.1, published during this work to fix a latent boot defect (the library never created the parent directory ofLocalDb:Path, and opens the file eagerly — a missing directory was a hardSQLite Error 14failure).https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Execution review against the code found the integer site_events.id is load-bearing in three places the original Task 5 did not list: - keyset cursor EventLogQueryService.cs:70,141,153 (id > $afterId) - storage-cap purge EventLogPurgeService.cs:164 (ORDER BY id ASC) - wire contracts EventLogEntry.Id, both ContinuationTokens (long) cross the site<->central Akka boundary A GUID PK against the existing cursor silently drops rows, and against the purge deletes arbitrary rows instead of the oldest. Task 5 is therefore split into 5a (writer, standard) and 5b (read path + DTOs, high-risk), with 5b's full file set spelled out and Task 10 gated on it. Also recorded: Task 4's real registration site, and that both legacy DB paths are CWD-relative and unset in docker today (so they live outside the data volume and are already lost on container recreate - Phase 1 fixes that). Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdtsForced by ZB.MOM.WW.LocalDb.Replication 0.1.0, whose nuspec floors Grpc.AspNetCore / Grpc.Net.Client / Grpc.Core.Api / Grpc.Tools at 2.76.0 and Google.Protobuf at 3.34.1. Restore fails NU1605 (package downgrade) below that floor, so LocalDb adoption cannot proceed without it. Landed as its own commit deliberately. The SQLitePCLRaw note in Directory.Packages.props deferred precisely this bump as belonging in "their own reviewed commit - not smuggled in under a SQLite security fix"; this honors that. Scope kept minimal: - Grpc.Core.Api pinned explicitly so the family stays on one version instead of floating in transitively. - Microsoft.Data.SqlClient and Newtonsoft.Json, also named in that note, are NOT bumped - nothing forces them and they carry their own risk profile. Direct consumer surface is two projects: Grpc.AspNetCore in Host, Grpc.Net.Client in Communication. Verified: dotnet build ZB.MOM.WW.ScadaBridge.slnx -> 0 Error(s), 0 Warning(s) Communication.Tests -> 312 passed, 0 failed Transport.IntegrationTests -> 103 passed, 0 failed Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdtsTask 1 of the Phase 1 adoption plan. Wiring only - no behavior change yet. - Directory.Packages.props: LocalDb / .Replication / .Contracts @ 0.1.0 - NuGet.config: ZB.MOM.WW.LocalDb + ZB.MOM.WW.LocalDb.* mapped to the dohertj2-gitea feed (central package management requires explicit mapping) - SiteRuntime + SiteEventLogging: PackageReference ZB.MOM.WW.LocalDb - Host: ZB.MOM.WW.LocalDb + ZB.MOM.WW.LocalDb.Replication Required two prerequisite commits, both landed ahead of this one:ecf6b628- AngleSharp pin (pre-existing break; the solution would not build)4715f2f9- gRPC 2.71.0 -> 2.76.0 (LocalDb.Replication's nuspec floor) SQLitePCLRaw stays pinned at 2.1.12 (GHSA-2m69-gcr7-jv3q) - verified intact. Verified: dotnet build ZB.MOM.WW.ScadaBridge.slnx -> 0 Error(s), 0 Warning(s) all three LocalDb packages restored from the Gitea feed Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdtsecf6b628) f056b67e9aTasks 2 + 5a-writer + 5b of the LocalDb Phase 1 plan, landed together because they are one indivisible change: the schema, the writer that fills it, and every consumer that assumed the old id semantics. WHY the id changes. Site pairs will replicate site_events with last-writer-wins on the primary key. With INTEGER PRIMARY KEY AUTOINCREMENT both nodes independently mint 1, 2, 3... for unrelated events, so sync would treat them as the same row and silently overwrite. A GUID makes the event log a pure union across the pair. Task 2 - schema extracted so the Host's AddZbLocalDb onReady can create the tables before RegisterReplicated installs capture triggers (pre-registration rows are never captured): - new OperationTrackingSchema.Apply / SiteEventLogSchema.Apply, plain Microsoft.Data.Sqlite, no LocalDb dependency - both stores delegate their InitializeSchema to them (idempotent, so a directly-constructed store still works) - OperationTracking is unchanged - it already had a TEXT PK and replicates as-is Task 5a - SiteEventLogger mints Guid.NewGuid("N") per event and inserts it. Task 5b - three consumers assumed a monotonic integer id. All three move to timestamp ordering; leaving any one behind would be a live bug: - EventLogQueryService: "id > $afterId" would return an ARBITRARY subset of a GUID-keyed table and SILENTLY DROP ROWS from page-through. Now a composite (timestamp, id) keyset cursor with an opaque string token; timestamps are not unique, so id is the tie-break that guarantees exactly-once paging. - EventLogPurgeService: "ORDER BY id ASC LIMIT 1000" would delete a RANDOM batch instead of the oldest. Now orders by timestamp. - EventLogEntry.Id and both ContinuationTokens: long -> string. WIRE COMPATIBILITY. Those DTOs cross the site<->central Akka boundary (SiteCommunicationActor -> CommunicationService -> ManagementActor / CentralUI). No rolling-upgrade shim is needed because both sides ship in the same deployable and the rig redeploys as a unit. Checked: no Akka serializer binding pins these types by name. A stale numeric token degrades to "start from the beginning" (a visible repeat) rather than throwing or losing rows. Tests: the two that encoded the old semantics were rewritten to guard the new invariant rather than deleted - uniqueness instead of monotonicity, and oldest-purged-first keyed on timestamp. That second test also exposed a latent weakness: the bulk seed stamped every row with the same UtcNow, so "oldest" was never actually well-defined; rows now get distinct increasing timestamps, kept inside the retention window so the retention purge does not eat them first. Verified: dotnet build ZB.MOM.WW.ScadaBridge.slnx -> 0 Error(s), 0 Warning(s) SiteEventLogging.Tests -> 70 passed (12 new schema tests, red-first) CentralUI.Tests -> 925 passed Commons.Tests -> 684 passed SiteRuntime.Tests -> 529 passed, 1 pre-existing flaky failure (InstanceActorChildAttributeRaceTests - passes 3/3 in isolation with and without this change) Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdtsTask 7 of the LocalDb Phase 1 adoption plan. AddZbLocalDbReplication registers the engine; MapZbLocalDbSync exposes the passive endpoint the peer node dials. Both are unconditional but INERT by default: with no LocalDb:Replication:PeerAddress the initiating SyncBackgroundService starts and immediately idles, and nothing dials the endpoint. A site pair replicates only once an operator sets a peer on BOTH nodes. The endpoint shares the Kestrel h2c listener the site gRPC server already uses, so there are no listener or port changes. Deviation from the plan: it put AddZbLocalDbReplication in Program.cs. Doing so would have left it untested - the composition-root tests build SiteServiceRegistration's graph, not Program.cs - so a registration that silently went missing would still show green. It now sits next to AddZbLocalDb in SiteServiceRegistration, where the DI-graph tests actually cover it. MapZbLocalDbSync stays in Program.cs because it needs the WebApplication. Two pins added to SiteLocalDbWiringTests, which configures no peer: - the engine resolves and is idle (not connected, no peer, never synced) - default-off means inert, not unregistered; - OplogBacklog is 0, not null. It is deliberately nullable so a failed poll reads as "unknown" instead of a healthy zero; a real 0 proves the provider is wired to the oplog, which Task 9's health signal depends on. Inbound auth on the sync endpoint is Task 8. Until that lands the endpoint is unauthenticated - which is precisely why replication ships default-OFF and no config in this repo sets a peer. Verified: build 0 warnings; Host 296/296, SiteEventLogging 70/70, SiteRuntime 530/530, Commons 684/684, Communication 312/312. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdtsTasks 8 and 9 of the LocalDb Phase 1 adoption plan. Task 8 - LocalDbSyncAuthInterceptor. The replication library's LocalDbSyncService verifies nothing; inbound auth is explicitly the host's job. Without this, anything able to reach a site node's gRPC port could stream arbitrary rows into the consolidated site database - including OperationTracking, which central reconciles from. Scoped strictly to /localdb_sync.v1.LocalDbSync/; SiteStream shares the same AddGrpc pipeline and passes through untouched. Fail-closed: with no LocalDb:Replication:ApiKey configured NO sync stream is accepted, authenticated or not. That is deliberate - "no key" is the default every site node ships with, so treating it as "no auth required" would expose the endpoint on precisely the most common configuration. Comparison is FixedTimeEquals over UTF-8 bytes. All four server handler shapes are gated, not just unary: the sync RPC is a bidirectional stream, so gating only the unary path would leave the real endpoint open while every unary test still passed. There is a test for that. Deviation from the plan: it specified Grpc.Core.Testing for the fake ServerCallContext. That type ships in the retired native Grpc.Core package and does not exist on the grpc-dotnet stack this solution uses; a minimal FakeServerCallContext in the test file was the better trade than adding a dead dependency. Task 9 - ISyncStatus onto the site health report as LocalDbReplicationConnected and LocalDbOplogBacklog, via a delegate-seam hosted service following the AddSiteEventLogHealthMetricsBridge precedent (HealthMonitoring takes no reference on the replication library). Both are additive init properties, so the Akka-remoted SiteHealthReport constructor signature is untouched. Both fields are nullable and the distinction is load-bearing: - null = the reporter has not run / replication is not wired ("no data"); - false/0 = a real reading. On a node with no peer that IS the healthy default-OFF state, not an outage. OplogBacklog is passed through nullable end-to-end because ISyncStatus returns null when the poll fails - flattening it to 0 would report a replication pair that cannot read its own oplog as perfectly healthy. The collector stores both values as one tuple and CollectReport reads it once, so a torn read cannot pair a fresh Connected with a stale backlog. Verified: build 0 warnings; Host 307/307 (8 interceptor + 3 health tests new), HealthMonitoring 97/97, Commons 684/684. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdtsTask 6 of the LocalDb Phase 1 adoption plan. Copies site-tracking.db and site_events.db into the consolidated database, then renames each source to <name>.migrated. Runs AFTER RegisterReplicated, deliberately: capture is trigger-based, so rows inserted before registration would never enter the oplog and would never reach the peer - silently. There is a test asserting migrated rows land in __localdb_oplog, because every other test in the file would still pass with that ordering reversed. Idempotent twice over. Tracking rows keep their existing TEXT primary key, so INSERT OR IGNORE dedupes naturally. Legacy events had autoincrement integer ids, which the consolidated schema replaced with GUIDs; they get DETERMINISTIC ids of the form mig-{NodeName}-{legacyId} rather than fresh GUIDs, so a migration that crashes between commit and rename cannot duplicate history on the next boot. The NodeName prefix keeps the two nodes of a pair from colliding on their independent legacy id sequences. Both properties are covered, including an explicit crash-window test that undoes the rename and re-runs. All-or-nothing: the copy commits in one transaction and the rename happens only afterwards, so a failure throws out of OnReady, fails host startup, and leaves the legacy files untouched. Deviation from the plan: it specified BeginTransactionAsync, but OnReady is a synchronous Action<ILocalDb>. A raw transaction on a CreateConnection() connection gives identical atomicity and identical trigger capture without blocking on async in a startup callback. Legacy paths come from the OLD config keys and fall back to the CWD-relative code defaults, because that is where the old code actually wrote them. Note this means both legacy databases have always lived outside the mounted data volume and were already discarded on every container recreate - so on the rig this migrator will usually find nothing, which is a legitimate no-op rather than a failure. Phase 1 incidentally fixes that data-loss bug. Non-failure cases are covered too: missing files, a legacy file with no such table, and in-memory legacy connection strings from test/dev configs. Verified: build 0 warnings; Host 316/316 (9 migrator tests new). Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdtsTask 11. site-a-a is the initiator (dials scadabridge-site-a-b:8083, the existing site gRPC h2c listener - no new port); site-a-b is passive. Both carry the SAME dev ApiKey, which is load-bearing: LocalDbSyncAuthInterceptor is fail-closed, so a mismatch does not degrade to unauthenticated replication, it rejects every stream and the pair silently stops converging. site-b and site-c stay unreplicated on purpose, so the default-OFF posture is proven side-by-side on one rig rather than asserted. Plain dev key matches the rig's existing posture; production uses a ${secret:...} reference through the pre-host expander. Noted in both files. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdtsTask 12 (live gate on the docker rig) found one real defect, which this fixes. THE DEFECT. The plan asserted LocalDb metrics "need no work - the ZB.MOM.WW.LocalDb.Replication meter flows through the existing AddZbTelemetry Prometheus export". It does not. ZbTelemetryOptions.Meters is an ALLOWLIST: an unlisted meter's instruments are never observed, with no error, no warning, and no missing-metric signal anywhere. The rig's /metrics scrape returned 301 lines and zero localdb_* series. Only looking for them found it. Fixed by hoisting the allowlist to SiteServiceRegistration.ObservedMeters and adding LocalDbMetrics.MeterName. The pin test asserts on that constant rather than on OTel's observed set, because AddZbTelemetry applies its options to a local instance and never registers IOptions - there is nothing resolvable to interrogate. The end-to-end proof is the live scrape below; the test exists to stop the entry being dropped again. Called out as such in the test. LIVE GATE EVIDENCE (docker rig, 8 nodes, rebuilt via docker/deploy.sh): 1. Consolidated DB created on every site node at /app/data/site-localdb.db - inside the mounted volume, unlike the legacy CWD-relative databases. 2. Replication proven REAL, not two independent local writes. Both site-a nodes hold the identical row and, decisively, the identical originating node_id and HLC in __localdb_row_version: site_events|{"id":"8b06b37c..."}|116946936661147648|65b00319-...|0 __localdb_peer_state shows a completed handshake in both directions. 3. Convergence: 5/5 events on both nodes, __localdb_row_version byte-identical across the pair, oplog drained to 0, dead_letter 0 on both. 4. SITE failover drill. Note the repo's failover-drill.sh targets CENTRAL nodes and does not exercise this claim, so the site-pair flip was run directly: stopped site-a-a (the node holding the history); site-a-b survived with the complete pre-flip history (3/3 events) plus its own takeover event. Restarted site-a-a; it caught up on what it missed and both nodes reconverged to the same 5 ids. This is the failure Phase 1 exists to prevent. 5. localdb_* now exported after the fix: localdb_sync_reconnects_total{...} 1 localdb_oplog_depth{...} 0 6. DEFAULT-OFF pin, live and side-by-side: site-b (no Replication section) boots identically, creates its consolidated DB, registers the engine - and stays inert. Zero replication log lines, no reconnect counter at all, so it never dialled anything. One transient WRN at startup on site-a-a ("Connection refused", reconnecting) is node A dialling before node B's listener was up; it reconnected within a second. Expected for a pair that starts together. Verified: build 0 warnings; Host wiring tests 11/11. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdtsTask 3 of the Phase 2 plan. Mirrors OperationTrackingSchema: the sf_messages DDL, the four additive ALTERs, the last_attempt_at_ms backfill and the due-index move verbatim into a static sync Apply(SqliteConnection), depending only on Microsoft.Data.Sqlite. The Host needs to apply this to a LocalDb-managed connection before RegisterReplicated installs the capture triggers; the store still calls it so a directly-constructed store stays self-sufficient. Two deviations from the plan as written, both deliberate: 1. The PRAGMA journal_mode=WAL in InitializeAsync STAYS. The plan's Step 4 snippet drops it, but Task 4 explicitly says not to move the equivalent pragma ("LocalDb owns the connection's pragmas") — the two tasks contradict each other. Keeping it preserves behaviour for the intermediate commits and for directly-constructed stores; it becomes moot in Task 5 when the store stops opening its own connections. Dropping it now would quietly regress the concurrent-writer support the pragma's own comment documents. 2. Added a second test beyond the plan's. The specified test asserts only against a freshly-created table, where CREATE TABLE already lists all 16 columns — it would pass with every ALTER deleted. The added test starts from the pre-upgrade 12-column shape with a row in it and proves the upgrade path runs and preserves data. That second test also surfaced that the backfill lands 1 ms low: julianday()'s double day-fraction cannot represent every millisecond. Pre-existing behaviour, carried over verbatim, asserted with a 1 ms tolerance rather than pinning a precision the implementation never had. Suite: 154 passed, 0 failed. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdtsTask 4 of the Phase 2 plan. All nine table definitions plus MigrateSchemaAsync and TryAddColumnAsync move into a static sync Apply(SqliteConnection) depending only on Microsoft.Data.Sqlite, so the Host can apply the DDL to a LocalDb-managed connection before RegisterReplicated installs the capture triggers. Per the plan, PRAGMA journal_mode=WAL stays in InitializeAsync — LocalDb owns the connection's pragmas, and the service still opens its own connections until Task 6. One deviation: TryAddColumnAsync's `catch (SqliteException) when (ex.Message.Contains("duplicate column"))` becomes a PRAGMA table_info probe, matching OperationTrackingSchema. The message-matching form depends on an error string that is not part of SQLite's contract, and it swallowed every SqliteException whose message happened to contain that substring. The probe also drops the ILogger dependency, which is what let the class become static. Cost: the per-column "Migrated: added column" info log is gone — it fired once per column per legacy database and nothing consumes it. Also added a test beyond the plan's specified one, for the same reason as Task 3: the specified test asserts against a freshly-created database, where CREATE TABLE already lists the migration columns, so it would pass with every ALTER deleted. The added test starts from the pre-migration shapes with a row present and proves the migration path runs and preserves data across a NOT NULL DEFAULT add. SiteRuntime suite 532 passed / 0 failed; full solution build 0 warnings. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdtsTasks 5 and 6 of the Phase 2 plan, committed together because their test fallout is entangled — several fixtures construct both stores. StoreAndForwardStorage and SiteStorageService now take ILocalDb. Connections come from ILocalDb.CreateConnection(), which hands out an already-open, pragma-configured connection carrying the zb_hlc_next() UDF the capture triggers call; a raw connection would lack the UDF and every write to a replicated table would fail closed. Deleted with the connection strings: S&F's EnsureDatabaseDirectoryExists and its per-open busy_timeout pragma, and the site service's BusyTimeoutFloorSeconds normalization — LocalDb owns all of it now. DI: AddSiteRuntime's string overload is gone (nothing left to supply), so the Host calls the no-arg form. ScadaBridge:Database:SiteDbPath and StoreAndForwardOptions.SqliteDbPath survive only as the migrator's source locations in Tasks 8/9. Two things the plan did not anticipate, both worth reading: 1. FOUND A REAL LATENT DEFECT, from Phase 1, now fixed. The plan assumed directory creation simply moved to LocalDb along with file ownership. It did not: the LocalDb library never creates the parent directory, and SqliteLocalDb opens the file eagerly in its constructor — so a missing directory is a hard boot failure ("SQLite Error 14: unable to open database file"), not a degraded start. The default site config points at the RELATIVE path ./data/site-localdb.db, so any site node without a pre-existing data/ directory fails to boot. The docker rig escapes only because its volume mount happens to create /app/data — a coincidence that would have hidden this until a bare-metal or fresh deployment. This has been latent since Phase 1 made LocalDb:Path required; deleting S&F's EnsureDatabaseDirectoryExists here would have widened it. Re-established the guarantee at the layer that now owns the path (SiteLocalDbDirectory.Ensure, called before AddZbLocalDb) and pinned it with SiteLocalDbDirectoryTests. Non-vacuity is not assumed: two tests written against the wrong assumption failed with exactly this SQLite Error 14 before the fix existed. 2. Test fallout was ~7x the plan's estimate. The plan named "fixtures" in one project; the constructor change actually reaches 40 files across 7 test projects, and most used Mode=Memory;Cache=Shared — which LocalDb has no equivalent for, so every one had to move to a real temp file. Rather than copy the Phase 1 TestLocalDb fixture into 7 projects, added a shared tests/ZB.MOM.WW.ScadaBridge.TestSupport library (not a test project) so the WAL-sidecar cleanup and the "real, not stubbed" rationale live in one place. Retargeted rather than deleted, in both directions: the S&F WAL test now asserts against the LocalDb-backed store (WAL genuinely is LocalDb's job), while the directory-creation test moved to Host.Tests (that guarantee is NOT LocalDb's). SiteStorageServiceTests.Initialize_EnablesWalJournalMode got the same treatment. DeploymentManagerMediumFindingsTests induced a persistence failure via an unopenable path, which no longer reaches the assertion since the fixture now throws first; it induces the same failure shape via an uninitialized store. Verified: full solution build 0 warnings; SiteRuntime 532, Host 318, AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97, StoreAndForward 153 — 1597 passed, 0 failed. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdtsAn audit row whose tracking snapshot cannot be resolved is skipped and left Pending, on the reasoning that central reconciliation will pick it up. Nothing removes it from the local drain queue, so the next tick re-reads it, fails identically, and logs again — forever. Measured on the rig at ~2,800 warnings/minute, sustained across both a process restart and a container restart, until the audit database itself was discarded. The code comment already names the cause ("the tracking store was reset"), so this is an anticipated input, not an exotic one. Two production triggers need no operator error: the tracking retention window elapsing before the drain catches up (long central outage, large backlog), or restoring one store independently of the other. The two stores are easy to desync because they live in different places — audit in auditlog.db, which on the docker rig is INSIDE the container, and tracking in the bind-mounted LocalDb database. Skipping the row is right; leaving it Pending with no other state change is not. There is no attempt counter, no backoff, no terminal state, and one Warning per row per pass. Suggested fixes ranked by effort, the cheapest being to rate-limit the warning the way MaintenanceBackgroundService already does for oplog caps. No data loss — the audit rows are intact and still reach central by the reconciliation path. Found while cleaning the rig after the Phase 2 live gate. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts