ScadaBridge/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json

{
  "planPath": "docs/plans/2026-07-19-localdb-adoption-phase2.md",
  "execution": {
    "mode": "parallel-waves",
    "implementerModel": "opus",
    "isolation": "worktree",
    "branch": "feat/localdb-phase2",
    "baseBranch": "feat/localdb-phase1",
    "note": "Phase 1's branch is NOT merged/pushed, so phase 2 branches from it. Dispatch every unblocked task concurrently per the wave table in the plan. Parallel implementers MUST use worktree isolation - concurrent git in one worktree races destructively."
  },
  "scopeDecision": {
    "date": "2026-07-19",
    "by": "user",
    "choice": "Full scope as designed",
    "note": "User chose both surfaces (config tables + sf_messages) in one phase, deleting both bespoke mechanisms together, over the recommended split of S&F-first. The four open design questions are therefore resolved INSIDE this plan as D1-D6 rather than deferred."
  },
  "reviewPass": {
    "date": "2026-07-19",
    "note": "Plan verified against actual code (3 verification sweeps + LocalDb library source) and corrected in place. Headline corrections: D1 (StoreDeployedConfigIfNewerAsync has a SECOND surviving caller, SiteReconciliationActor.cs:166 - the method and guard STAY; original Task 13 would also not have compiled, deleting a method whose caller dies only in Task 15), D3 (the active node ALREADY purges at DeploymentManagerActor.cs:1921 - Task 12 became a pin test, nothing is re-homed), D6 added (4 MB gRPC receive cap x row-count-only MaxBatchSize batching; config_json > 128 KB documented - measure in Task 1, size MaxBatchSize in Task 19, single row near 4 MB = stop/lib work), Task 1 rewritten (Phase 2 tables are NOT registered on the Phase 1 rig so driven churn never reaches __localdb_oplog - measure legacy-DB write rates + arithmetic; metrics port 8084 not 8080; real metric names are localdb_oplog_depth / localdb_sync_*, the plan's localdb_oplog_backlog/replication_dead_letters/sync_connected never existed; containers have no sqlite3 - sample via throwaway copies of the DB triplet, NEVER host-side sqlite3 against the live files [2026-07-20: that poisons the container's WAL state - see docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md]; branch already exists, no checkout -b), Task 14 (register 8 tables NOT 11 - notification_lists/smtp_configurations deliberately unregistered, reversing the original instruction; keep Migrate LAST in OnReady)."
  },
  "decisions": [
    {
      "id": "D1",
      "subject": "Config moves to CDC; notify-and-fetch is DELETED - but the guarded write STAYS",
      "evidence": "SiteReplicationActor sends id+fetch-coords only because the config blob exceeds Akka's 128KB frame (docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md, already marked RESOLVED by the notify-and-fetch rework). LocalDb sync is gRPC - no such limit (but see D6). CORRECTED: StoreDeployedConfigIfNewerAsync (SiteStorageService.cs:301-336, guard at :325) has TWO production callers - SiteReplicationActor.cs:375 (dies in Task 15) AND SiteReconciliationActor.cs:166 (per-node startup self-heal vs central, SURVIVES Phase 2, stale-fetch race still real). Method + guard stay; reconcile becomes a benign second CDC writer. Do NOT reproduce the guard on top of LWW - deployed_at and HLC are different clocks and mixing them is non-convergent."
    },
    {
      "id": "D2",
      "subject": "ReplaceAllAsync deleted; the N1 directional guard becomes unnecessary",
      "evidence": "LocalDb's snapshot resync MERGES per-row LWW and never WIPES: SnapshotApplier.OnBeginAsync (SnapshotStreamer.cs:163-170) resets counters only; OnBatchAsync (:172-186) routes snapshot rows through the same LwwApplier as deltas; LwwApplier.cs:69-78 discards an incoming row whose HLC is lower. Row-level deletes DO replicate (delete-trigger tombstones, streamed by SnapshotStreamer, applied as real DELETEs by LwwApplier) - only the destructive whole-table replace is gone. Caveat: tombstones pruned after TombstoneRetention (default 7d); a node offline longer can resurrect deleted rows (runbook, Task 21). SEMANTIC CHANGE: the standby is convergent, no longer byte-identical."
    },
    {
      "id": "D3",
      "subject": "CORRECTED: the SMTP purge already runs on the active node - pin it, don't move it",
      "evidence": "PurgeCentralOnlyNotificationConfigAsync (SiteStorageService.cs:811-821) has TWO callers: DeploymentManagerActor.cs:1921 (ACTIVE node's HandleDeployArtifacts, :1864-1963) and SiteReplicationActor.cs:456 (standby copy, dies with the actor). The purge never lapses; the original 're-home before any deletion' premise was false. Task 12 = pin test only (ArtifactStorageTests covers the storage method, not the actor call site Task 16 edits). No site writer to notification_lists/smtp_configurations since 2026-07-10 (verified: only test seeding inserts exist) + migrator skips them => permanently empty in the consolidated DB => Task 14 does NOT register them."
    },
    {
      "id": "D4",
      "subject": "native_alarm_state volume is MEASURED, not assumed",
      "evidence": "scadabridge.db is not only config - native_alarm_state mirrors live A&C conditions (NativeAlarmActor.cs:504) and is the highest-volume table in either DB. sf_messages worst case ~50 row-writes/sec. Task 1 measures both AT THE LEGACY-DB SOURCE (they are not in the Phase 1 oplog - see reviewPass) and sets MaxOplogRows/MaxOplogAge arithmetically; Task 20 evidence 10 does the empirical post-cutover drain check. If growth is monotonic, STOP: keyed-instances escape hatch = ~/Desktop/scadaproj/docs/plans/2026-07-19-scadabridge-localdb-design.md:139-141 (adoption design doc, NOT the 07-17 lib doc), a scadaproj library effort that would suspend this plan."
    },
    {
      "id": "D5",
      "subject": "Cutover forecloses rolling site upgrades",
      "evidence": "SiteReplicationActor retains a legacy monolithic SfBufferSnapshot handler for rolling upgrades. With no dual-mechanism period, one node would speak a protocol the other no longer implements. Both nodes of a site must be stopped and started together. Task 21 puts this in the deployment docs (installation-guide.md/topology-guide.md - no file literally named runbook)."
    },
    {
      "id": "D6",
      "subject": "NEW (review pass): 4 MB gRPC message cap replaces the 128 KB Akka frame as the size ceiling",
      "evidence": "Neither side configures gRPC message sizes (ScadaBridge AddGrpc at Program.cs:519-521 sets only the auth interceptor; the lib's initiator channel is bare GrpcChannel.ForAddress) => 4 MB default receive limit both directions. Batching is row-count-only (MaxBatchSize default 500; SyncSession.cs:227, SnapshotStreamer.cs:55; no byte-aware chunking). deployed_configurations.config_json documented >128 KB/row; a few dozen such rows in one batch exceeds 4 MB and wedges the stream on a poison batch. Task 1 measures max/avg config_json bytes; Task 19 sets LocalDb:Replication:MaxBatchSize so max-row-bytes x MaxBatchSize << 4 MB; any single row near 4 MB = STOP (needs byte-aware batching or size knobs in the LocalDb lib - scadaproj effort)."
    }
  ],
  "reconFindings": [
    "The gate doc named 2 test files as the specification; the real spec is 5 - it missed StoreAndForwardReplicationTests.cs (incl. the only Requeue coverage), ReplicationWireSerializationPinTests.cs, ResyncWireSerializationPinTests.cs, and SfBufferResyncPredicateTests.cs (the N1 Critical regression test).",
    "sf_messages has NO version column today - ON CONFLICT(id) DO UPDATE has no comparison predicate (StoreAndForwardStorage.cs:331-345). 'Newest wins' is bare arrival order. LWW-by-HLC is an IMPROVEMENT here, not a regression.",
    "No autoincrement-integer PKs exist anywhere in Phase 2 scope - all 9 config tables use natural TEXT or composite TEXT keys, sf_messages is TEXT. LocalDb RegisterReplicated SUPPORTS composite PKs (ordered pk ordinals) and rejects BLOB columns - no Phase 2 table has one (verified). Phase 1's site_events GUID conversion has no Phase 2 analogue.",
    "SiteStorageService has NO foreign keys. RemoveDeployedConfigAsync (:343-376) is a manual 3-statement cascade in one transaction. Under CDC these become three independent delete streams that LWW may reorder - the most likely real defect in the plan (Task 18 scenario 2).",
    "DeploymentManagerActor's replicationActor is an OPTIONAL POSITIONAL parameter at :169 (ctor :161-175; :184 is the field ASSIGNMENT). The same-typed IActorRef? optional dclManager sits immediately BEFORE it at :168 - that's the real silent-shift hazard. Props.Create passes it positionally at AkkaHostedService.cs:810. Check every call site by hand (Task 16).",
    "ActiveNodeEvaluator must NOT be deleted - the S&F delivery gate still uses it (AkkaHostedService.cs:866 SetDeliveryGate -> SelfIsPrimary -> SelfIsOldestUp). Doc comment :14/:16 mentions replication; SiteReplicationActor.cs:288 calls it directly (dies with the actor).",
    "ConfigFetchRetryCount's ONLY production reader is SiteReplicationActor.cs:157 (verified) - dead after Task 15; removed in Task 17 (removing it in Task 13, before the actor deletion, would not compile). IDeploymentConfigFetcher is KEPT: DeploymentManagerActor refresh path + SiteReconciliationActor + DI at ServiceCollectionExtensions.cs:84.",
    "notification_lists and smtp_configurations are deliberately NOT migrated (Task 9) AND NOT registered (Task 14, corrected) - migrating or replicating them would resurrect/ship plaintext SMTP passwords; they are permanently empty by design (writers removed 2026-07-10; verified only test-seeding inserts exist).",
    "REVIEW-PASS ADDITIONS: SiteReconciliationActor (runs on EVERY node at startup, best-effort self-heal vs central) is the second caller of both StoreDeployedConfigIfNewerAsync and IDeploymentConfigFetcher - it survives Phase 2 and constrains Tasks 11/13/20 (a startup fetch on the standby is legitimate; zero-fetch assertions must scope to the deploy window).",
    "SiteStorageService has 21 (not 22) inline connection+OpenAsync pairs: 60,206,248,311,345,386,414,443,470,498,538,583,608,640,663,692,730,769,813,837,868. CreateConnection():51's only repository consumer is SiteExternalSystemRepository.",
    "StoreAndForwardService: :39 is the _replication FIELD, :243 the ctor param; exactly 6 emission sites (:654,805,836,872,1120,1146). ServiceCollectionExtensions.cs:32 also resolves ReplicationService inside the StoreAndForwardService factory - must go in Task 14.",
    "ReplicationMessages.cs holds ONLY the 10 Replicate*/Apply* records; the four SfBuffer resync records live at the bottom of SiteReplicationActor.cs (:678-707) and die with the actor file.",
    "10 (not 9) appsettings.Site.json files set the legacy paths - deploy/wonder-app-vd03/appsettings.Site.json was missed (also sets ReplicationEnabled:false).",
    "Rig facts: site metrics on port 8084 (AnyIP in-container, NOT published to host; 8080 is Traefik); real metric names localdb_oplog_depth / localdb_sync_* (meter ZB.MOM.WW.LocalDb.Replication); containers (aspnet:10.0) have NO sqlite3 - data dirs are host bind mounts but NEVER run sqlite3 host-side against the live files (poisons the container's WAL state, root cause of the 2026-07-20 disk-I/O-error incident): cp the .db/-wal/-shm triplet and query the copy (see the snap() helper in the plan); ReplicationOptions bind at LocalDb:Replication:* (lib binds the section).",
    "Phase 1's LocalDbSitePairConvergenceTests uses ONE shared API key (:47) - it never did a mismatched-key non-vacuity run; wrong-key denial is unit-covered in Host.Tests/LocalDbSyncAuthInterceptorTests.cs. Task 18's non-vacuity check must be done directly for the new scenarios.",
    "MigrateEvents synthesizes deterministic 'mig-{node}-{legacyId}' ids (NOT fresh GUIDs - crash-rerun idempotency). Migration runs AFTER RegisterReplicated in OnReady (order is load-bearing); Task 8's oplog pin test must register sf_messages on its own TestLocalDb since production doesn't register it until Task 14."
  ],
  "tasks": [
    {"id": 1, "subject": "Task 1: Rig soak - measure legacy-DB write rates + config_json sizes (corrected method)", "status": "completed", "classification": "high-risk", "note": "GATE CLOSED - verdict PROCEED (2026-07-20). NEVER sample host-side sqlite3 on live bind-mounted WAL files (that method poisoned the first run - use the copy-based snap() helper). Clean re-run: sf_messages 0.80 rows/sec insert, 0/sec retry-UPDATE, oplog 0, alarms 0, max payload_json 76 B, max config_json 721 B (NOT representative), zero SQLite errors in 30 min, both nodes converged. No stop condition met. Binding output: MaxBatchSize 500 -> 16 (Task 19). Empirical drain check remains Task 20 evidence 10."},
    {"id": 2, "subject": "Task 2: Decision record - close the phase 2 gate", "status": "completed", "classification": "trivial", "blockedBy": [1], "note": "Gate doc status flipped NOT STARTED -> CLOSED; all 5 SS5 questions answered inline (questions kept, not deleted); SS2's N5 duplicate-bound requirement routed explicitly to Task 21."},
    {"id": 3, "subject": "Task 3: Extract StoreAndForwardSchema.Apply", "status": "completed", "classification": "small", "blockedBy": [2], "note": "DEVIATION: kept PRAGMA journal_mode=WAL in InitializeAsync - the plan Step 4 snippet drops it but Task 4 says not to move the equivalent pragma; contradictory, and dropping it would regress documented concurrent-writer support before Task 5 makes it moot. Added a legacy-upgrade test beyond the plan's (the specified test asserts against a FRESH table where CREATE TABLE already lists all 16 columns - it would pass with every ALTER deleted). That test found the last_attempt_at_ms backfill lands 1 ms low: julianday() double day-fraction rounding, pre-existing, carried verbatim, asserted with 1 ms tolerance. Suite 154/154."}, 
    {"id": 4, "subject": "Task 4: Extract SiteStorageSchema.Apply", "status": "completed", "classification": "small", "blockedBy": [2], "note": "DEVIATION: TryAddColumnAsync's catch-on-message-text ('duplicate column') became a PRAGMA table_info probe matching OperationTrackingSchema - the message form depends on a non-contractual error string and swallowed unrelated SqliteExceptions; also dropped the ILogger dep, which is what let the class be static. Cost: the per-column 'Migrated: added column' info log is gone (nothing consumes it). PRAGMA journal_mode=WAL stays in InitializeAsync per plan. Added a legacy-upgrade test for the same reason as Task 3. SiteRuntime 532/532, full solution build 0 warnings."}, 
    {"id": 5, "subject": "Task 5: Rewire StoreAndForwardStorage onto ILocalDb", "status": "pending", "classification": "high-risk", "blockedBy": [3]},
    {"id": 6, "subject": "Task 6: Rewire SiteStorageService onto ILocalDb", "status": "pending", "classification": "high-risk", "blockedBy": [4]},
    {"id": 7, "subject": "Task 7: Extend SiteLocalDbSetup with the new DDL (not yet registered)", "status": "pending", "classification": "standard", "blockedBy": [5, 6]},
    {"id": 8, "subject": "Task 8: Extend the legacy migrator for sf_messages", "status": "pending", "classification": "high-risk", "blockedBy": [7], "note": "Oplog pin test registers sf_messages on its own TestLocalDb - production OnReady doesn't register it until Task 14."},
    {"id": 9, "subject": "Task 9: Extend the legacy migrator for the config tables (skip notification/smtp)", "status": "pending", "classification": "high-risk", "blockedBy": [7]},
    {"id": 10, "subject": "Task 10: Port the S&F replication test intents as CDC specs", "status": "pending", "classification": "standard", "blockedBy": [8]},
    {"id": 11, "subject": "Task 11: Port the resync + directional-authority tests", "status": "pending", "classification": "standard", "blockedBy": [9], "note": "Zero-fetch assertion scoped to deploy convergence - SiteReconciliationActor's startup fetch is legitimate and survives."},
    {"id": 12, "subject": "Task 12: Pin the active-node SMTP purge (corrected: already exists at DeploymentManagerActor.cs:1921)", "status": "pending", "classification": "standard", "blockedBy": [10, 11], "note": "Pin test only - no production change. Guards Task 16's actor edits from silently dropping the purge call."},
    {"id": 13, "subject": "Task 13: Notify-and-fetch scope check - guarded write STAYS for SiteReconciliationActor", "status": "pending", "classification": "standard", "blockedBy": [12], "note": "CORRECTED: do NOT delete StoreDeployedConfigIfNewerAsync (second caller SiteReconciliationActor.cs:166 survives; deleting here wouldn't compile anyway). Doc-comment update only; ConfigFetchRetryCount removal moved to Task 17."},
    {"id": 14, "subject": "Task 14: Register the 8 Phase 2 tables and delete ReplicationService", "status": "pending", "classification": "high-risk", "blockedBy": [13], "note": "THE CUTOVER. Register 7 config tables + sf_messages - NOT notification_lists/smtp_configurations (reversed from original; see D3). Keep Migrate LAST in OnReady. Registration + bespoke deletion in ONE commit."},
    {"id": 15, "subject": "Task 15: Delete SiteReplicationActor and its messages", "status": "pending", "classification": "high-risk", "blockedBy": [14], "note": "Resync records live at SiteReplicationActor.cs:678-707, not ReplicationMessages.cs."},
    {"id": 16, "subject": "Task 16: Clean up DeploymentManagerActor and AkkaHostedService", "status": "pending", "classification": "standard", "blockedBy": [15], "note": "Ctor param at :169; same-typed IActorRef? dclManager at :168 BEFORE it is the silent-shift hazard; Props.Create positional at AkkaHostedService.cs:810."},
    {"id": 17, "subject": "Task 17: Config-key cleanup (incl. ConfigFetchRetryCount; 10 appsettings files)", "status": "pending", "classification": "standard", "blockedBy": [16]},
    {"id": 18, "subject": "Task 18: Two-node convergence suite for the Phase 2 tables", "status": "pending", "classification": "high-risk", "blockedBy": [17], "note": "Non-vacuity must be proven directly (Phase 1 never did the mismatched-key run the original plan claimed)."},
    {"id": 19, "subject": "Task 19: Rig configuration (MaxOplogRows/MaxOplogAge + MaxBatchSize per D6)", "status": "pending", "classification": "small", "blockedBy": [17]},
    {"id": 20, "subject": "Task 20: Live gate on the docker rig (10 evidence items)", "status": "pending", "classification": "high-risk", "blockedBy": [18, 19], "note": "Includes the post-cutover oplog drain-under-churn check (evidence 10) that Task 1 cannot measure pre-cutover. Sample DBs via throwaway copies (snap() helper), never host-side sqlite3 on live files; metrics on :8084 in-container."},
    {"id": 21, "subject": "Task 21: Documentation truth pass", "status": "pending", "classification": "standard", "blockedBy": [20]}
  ],
  "knownFlakes": [
    {
      "test": "SiteRuntime.Tests InstanceActorChildAttributeRaceTests.ChildActors_AreSeededFromAnIsolatedCopy_NotTheLiveAttributesDictionary",
      "note": "Intermittent ActorNotFoundException under full-suite load; passes in isolation. Pre-existing, carried over from Phase 1."
    },
    {
      "test": "AuditLog.Tests ParentExecutionIdCorrelationTests.InboundRoutedRun_AllRoutedRows_CarryInboundExecutionId_AsParentExecutionId",
      "note": "Cold-MSSQL-fixture timing: ~91s and AwaitAssert-times-out cold, ~1s warm. Re-run before investigating."
    }
  ],
  "lastUpdated": "2026-07-20",
  "phase2Status": "UNBLOCKED - Task 1 gate CLOSED (verdict PROCEED) and Task 2 DONE, 2026-07-20. Task 1's original STOP verdict is SUPERSEDED: the 'Phase 1 disk I/O defect' was OBSERVER-INDUCED (host-side sqlite3 against live bind-mounted WAL files resets the WAL across virtiofs and permanently poisons the container's connections) - NOT a product defect. See docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md. Both earlier isolation claims were confounded: one sampling pass had already poisoned BOTH nodes, and a poisoned standby looks healthy only because it issues almost no statements; 'LocalDb-specific' was sampling-selection bias (only the LocalDb file had ever been host-read). Clean re-run 2026-07-20 with the copy-based snap() helper on restarted nodes, 6 consecutive 60s intervals: sf_messages 48 rows/min = 0.80 rows/sec dead steady, retry-UPDATE rate 0/sec (SUM(retry_count) flat at 200), oplog 0, native_alarm_state 0, max payload_json 76 B, max config_json 721 B, ZERO SQLite errors across 30 min, both site-a nodes converged at 3564 rows. Honest gap: 0.80/s is ~1.6% of the 50/s ceiling and the retry-UPDATE path was never exercised - acceptable because that ceiling is structural (SweepBatchLimit/RetryTimerInterval), not empirical. Rig config rows (721 B) are NOT representative; D6 sizing rests on the documented ~60-70 KB production config_json. Plan-premise corrections stand: D6 (MaxBatchSize 500 -> 16, the one firmly evidence-backed number; Task 19 sets it), D4 (alarm writes bounded by per-SourceReference coalescing at a 100 ms flush, NOT unbounded), sf_messages hard ceiling 50 rows/sec, oplog cap overrun = graceful snapshot resync (needs_snapshot), not data loss. NEXT: Wave 1 = Tasks 3 + 4, dispatchable in parallel (disjoint Files blocks). Tasks 3-21 untouched; no plan code written yet. Rig cleanup still owed before Task 20: restore ExternalSystemDefinitions id 1 to http://scadabridge-restapi:5200 (currently http://127.0.0.1:9), and remove SoakGenerator template 2021 + instances soakgen-1..4 (ids 5-8), still deployed and generating load."
}