docs(localdb): phase 2 plan review pass — corrections from code verification

Three verification sweeps over SiteRuntime, StoreAndForward, and Host/rig/docs
plus the LocalDb library source. Load-bearing corrections: D1 (the guarded write
has a second surviving caller, SiteReconciliationActor), D3 (the active node
already purges — Task 12 becomes a pin), D6 (new: the 4 MB gRPC cap vs
row-count-only batching), Task 1 (rewritten method — the Phase 2 tables are not
in the Phase 1 oplog), and Task 14 (do not register the notification/SMTP tables).

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts

docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json

@@ -12,66 +12,83 @@
     "date": "2026-07-19",
     "by": "user",
     "choice": "Full scope as designed",
-    "note": "User chose both surfaces (config tables + sf_messages) in one phase, deleting both bespoke mechanisms together, over the recommended split of S&F-first. The four open design questions are therefore resolved INSIDE this plan as D1-D5 rather than deferred."
+    "note": "User chose both surfaces (config tables + sf_messages) in one phase, deleting both bespoke mechanisms together, over the recommended split of S&F-first. The four open design questions are therefore resolved INSIDE this plan as D1-D6 rather than deferred."
+  },
+  "reviewPass": {
+    "date": "2026-07-19",
+    "note": "Plan verified against actual code (3 verification sweeps + LocalDb library source) and corrected in place. Headline corrections: D1 (StoreDeployedConfigIfNewerAsync has a SECOND surviving caller, SiteReconciliationActor.cs:166 - the method and guard STAY; original Task 13 would also not have compiled, deleting a method whose caller dies only in Task 15), D3 (the active node ALREADY purges at DeploymentManagerActor.cs:1921 - Task 12 became a pin test, nothing is re-homed), D6 added (4 MB gRPC receive cap x row-count-only MaxBatchSize batching; config_json > 128 KB documented - measure in Task 1, size MaxBatchSize in Task 19, single row near 4 MB = stop/lib work), Task 1 rewritten (Phase 2 tables are NOT registered on the Phase 1 rig so driven churn never reaches __localdb_oplog - measure legacy-DB write rates + arithmetic; metrics port 8084 not 8080; real metric names are localdb_oplog_depth / localdb_sync_*, the plan's localdb_oplog_backlog/replication_dead_letters/sync_connected never existed; containers have no sqlite3 - use host-side sqlite3 on the bind mounts; branch already exists, no checkout -b), Task 14 (register 8 tables NOT 11 - notification_lists/smtp_configurations deliberately unregistered, reversing the original instruction; keep Migrate LAST in OnReady)."
   },
   "decisions": [
     {
       "id": "D1",
-      "subject": "Config moves to CDC; notify-and-fetch is DELETED, not preserved",
-      "evidence": "SiteReplicationActor sends id+fetch-coords only because the config blob exceeds Akka's 128KB frame (docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md). LocalDb sync is gRPC - no such limit. StoreDeployedConfigIfNewerAsync's guard (SiteStorageService.cs:325, `WHERE excluded.deployed_at > deployed_configurations.deployed_at`) protects only against a stale FETCH racing; with no fetch there is no race. Do NOT reproduce the guard on top of LWW - deployed_at and HLC are different clocks and mixing them is non-convergent."
+      "subject": "Config moves to CDC; notify-and-fetch is DELETED - but the guarded write STAYS",
+      "evidence": "SiteReplicationActor sends id+fetch-coords only because the config blob exceeds Akka's 128KB frame (docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md, already marked RESOLVED by the notify-and-fetch rework). LocalDb sync is gRPC - no such limit (but see D6). CORRECTED: StoreDeployedConfigIfNewerAsync (SiteStorageService.cs:301-336, guard at :325) has TWO production callers - SiteReplicationActor.cs:375 (dies in Task 15) AND SiteReconciliationActor.cs:166 (per-node startup self-heal vs central, SURVIVES Phase 2, stale-fetch race still real). Method + guard stay; reconcile becomes a benign second CDC writer. Do NOT reproduce the guard on top of LWW - deployed_at and HLC are different clocks and mixing them is non-convergent."
     },
     {
       "id": "D2",
       "subject": "ReplaceAllAsync deleted; the N1 directional guard becomes unnecessary",
-      "evidence": "LocalDb's snapshot resync MERGES per-row LWW and never deletes: SnapshotApplier.OnBeginAsync (SnapshotStreamer.cs:163-170) resets counters only, no DELETE anywhere in the class; OnBatchAsync (:172-186) routes snapshot rows through the same LwwApplier as deltas; LwwApplier.cs:69-78 discards an incoming row whose HLC is lower. So a stale peer cannot wipe a live buffer - structurally impossible, not guarded. SEMANTIC CHANGE: the standby is convergent, no longer byte-identical."
+      "evidence": "LocalDb's snapshot resync MERGES per-row LWW and never WIPES: SnapshotApplier.OnBeginAsync (SnapshotStreamer.cs:163-170) resets counters only; OnBatchAsync (:172-186) routes snapshot rows through the same LwwApplier as deltas; LwwApplier.cs:69-78 discards an incoming row whose HLC is lower. Row-level deletes DO replicate (delete-trigger tombstones, streamed by SnapshotStreamer, applied as real DELETEs by LwwApplier) - only the destructive whole-table replace is gone. Caveat: tombstones pruned after TombstoneRetention (default 7d); a node offline longer can resurrect deleted rows (runbook, Task 21). SEMANTIC CHANGE: the standby is convergent, no longer byte-identical."
     },
     {
       "id": "D3",
-      "subject": "The SMTP purge is re-homed BEFORE anything is deleted",
-      "evidence": "SiteReplicationActor.HandleApplyArtifacts calls PurgeCentralOnlyNotificationConfigAsync (SiteStorageService.cs:811-821), deleting notification_lists + smtp_configurations incl. plaintext SMTP passwords from pre-2026-07-10 deployments. A security cleanup riding the replication path; CDC will not reproduce it. Task 12 moves it to the active node's deploy path before any deletion, so it never lapses."
+      "subject": "CORRECTED: the SMTP purge already runs on the active node - pin it, don't move it",
+      "evidence": "PurgeCentralOnlyNotificationConfigAsync (SiteStorageService.cs:811-821) has TWO callers: DeploymentManagerActor.cs:1921 (ACTIVE node's HandleDeployArtifacts, :1864-1963) and SiteReplicationActor.cs:456 (standby copy, dies with the actor). The purge never lapses; the original 're-home before any deletion' premise was false. Task 12 = pin test only (ArtifactStorageTests covers the storage method, not the actor call site Task 16 edits). No site writer to notification_lists/smtp_configurations since 2026-07-10 (verified: only test seeding inserts exist) + migrator skips them => permanently empty in the consolidated DB => Task 14 does NOT register them."
     },
     {
       "id": "D4",
       "subject": "native_alarm_state volume is MEASURED, not assumed",
-      "evidence": "scadabridge.db is not only config - native_alarm_state mirrors live A&C conditions (NativeAlarmActor.cs:504) and is the highest-volume table in either DB. sf_messages worst case ~50 row-writes/sec. Task 1 is a rig soak that measures both and sets MaxOplogRows/MaxOplogAge. If backlog grows monotonically, STOP: the design doc's keyed-instances escape hatch (line 140) is a scadaproj library effort that would suspend this plan."
+      "evidence": "scadabridge.db is not only config - native_alarm_state mirrors live A&C conditions (NativeAlarmActor.cs:504) and is the highest-volume table in either DB. sf_messages worst case ~50 row-writes/sec. Task 1 measures both AT THE LEGACY-DB SOURCE (they are not in the Phase 1 oplog - see reviewPass) and sets MaxOplogRows/MaxOplogAge arithmetically; Task 20 evidence 10 does the empirical post-cutover drain check. If growth is monotonic, STOP: keyed-instances escape hatch = ~/Desktop/scadaproj/docs/plans/2026-07-19-scadabridge-localdb-design.md:139-141 (adoption design doc, NOT the 07-17 lib doc), a scadaproj library effort that would suspend this plan."
     },
     {
       "id": "D5",
       "subject": "Cutover forecloses rolling site upgrades",
-      "evidence": "SiteReplicationActor retains a legacy monolithic SfBufferSnapshot handler for rolling upgrades. With no dual-mechanism period, one node would speak a protocol the other no longer implements. Both nodes of a site must be stopped and started together. Task 21 puts this in the deployment runbook."
+      "evidence": "SiteReplicationActor retains a legacy monolithic SfBufferSnapshot handler for rolling upgrades. With no dual-mechanism period, one node would speak a protocol the other no longer implements. Both nodes of a site must be stopped and started together. Task 21 puts this in the deployment docs (installation-guide.md/topology-guide.md - no file literally named runbook)."
+    },
+    {
+      "id": "D6",
+      "subject": "NEW (review pass): 4 MB gRPC message cap replaces the 128 KB Akka frame as the size ceiling",
+      "evidence": "Neither side configures gRPC message sizes (ScadaBridge AddGrpc at Program.cs:519-521 sets only the auth interceptor; the lib's initiator channel is bare GrpcChannel.ForAddress) => 4 MB default receive limit both directions. Batching is row-count-only (MaxBatchSize default 500; SyncSession.cs:227, SnapshotStreamer.cs:55; no byte-aware chunking). deployed_configurations.config_json documented >128 KB/row; a few dozen such rows in one batch exceeds 4 MB and wedges the stream on a poison batch. Task 1 measures max/avg config_json bytes; Task 19 sets LocalDb:Replication:MaxBatchSize so max-row-bytes x MaxBatchSize << 4 MB; any single row near 4 MB = STOP (needs byte-aware batching or size knobs in the LocalDb lib - scadaproj effort)."
     }
   ],
   "reconFindings": [
     "The gate doc named 2 test files as the specification; the real spec is 5 - it missed StoreAndForwardReplicationTests.cs (incl. the only Requeue coverage), ReplicationWireSerializationPinTests.cs, ResyncWireSerializationPinTests.cs, and SfBufferResyncPredicateTests.cs (the N1 Critical regression test).",
     "sf_messages has NO version column today - ON CONFLICT(id) DO UPDATE has no comparison predicate (StoreAndForwardStorage.cs:331-345). 'Newest wins' is bare arrival order. LWW-by-HLC is an IMPROVEMENT here, not a regression.",
-    "No autoincrement-integer PKs exist anywhere in Phase 2 scope - all 9 config tables use natural TEXT or composite TEXT keys, sf_messages is TEXT. Phase 1's site_events GUID conversion has no Phase 2 analogue.",
+    "No autoincrement-integer PKs exist anywhere in Phase 2 scope - all 9 config tables use natural TEXT or composite TEXT keys, sf_messages is TEXT. LocalDb RegisterReplicated SUPPORTS composite PKs (ordered pk ordinals) and rejects BLOB columns - no Phase 2 table has one (verified). Phase 1's site_events GUID conversion has no Phase 2 analogue.",
     "SiteStorageService has NO foreign keys. RemoveDeployedConfigAsync (:343-376) is a manual 3-statement cascade in one transaction. Under CDC these become three independent delete streams that LWW may reorder - the most likely real defect in the plan (Task 18 scenario 2).",
-    "DeploymentManagerActor's replicationActor is an OPTIONAL POSITIONAL parameter (:184). Removing it silently shifts every argument after it. Compilers will not catch a shift between two same-typed optional params - check every call site by hand (Task 16).",
-    "ActiveNodeEvaluator must NOT be deleted - the S&F delivery gate still uses it. Only its doc comment mentions replication.",
-    "ConfigFetchRetryCount may not be fully dead: IDeploymentConfigFetcher serves BOTH the standby replication path and the active singleton's RefreshDeploymentCommand path. Verify before removing (Task 13 step 2).",
-    "notification_lists and smtp_configurations are deliberately NOT migrated (Task 9) - migrating them would resurrect plaintext SMTP passwords from a pre-fix legacy file into a replicated table."
+    "DeploymentManagerActor's replicationActor is an OPTIONAL POSITIONAL parameter at :169 (ctor :161-175; :184 is the field ASSIGNMENT). The same-typed IActorRef? optional dclManager sits immediately BEFORE it at :168 - that's the real silent-shift hazard. Props.Create passes it positionally at AkkaHostedService.cs:810. Check every call site by hand (Task 16).",
+    "ActiveNodeEvaluator must NOT be deleted - the S&F delivery gate still uses it (AkkaHostedService.cs:866 SetDeliveryGate -> SelfIsPrimary -> SelfIsOldestUp). Doc comment :14/:16 mentions replication; SiteReplicationActor.cs:288 calls it directly (dies with the actor).",
+    "ConfigFetchRetryCount's ONLY production reader is SiteReplicationActor.cs:157 (verified) - dead after Task 15; removed in Task 17 (removing it in Task 13, before the actor deletion, would not compile). IDeploymentConfigFetcher is KEPT: DeploymentManagerActor refresh path + SiteReconciliationActor + DI at ServiceCollectionExtensions.cs:84.",
+    "notification_lists and smtp_configurations are deliberately NOT migrated (Task 9) AND NOT registered (Task 14, corrected) - migrating or replicating them would resurrect/ship plaintext SMTP passwords; they are permanently empty by design (writers removed 2026-07-10; verified only test-seeding inserts exist).",
+    "REVIEW-PASS ADDITIONS: SiteReconciliationActor (runs on EVERY node at startup, best-effort self-heal vs central) is the second caller of both StoreDeployedConfigIfNewerAsync and IDeploymentConfigFetcher - it survives Phase 2 and constrains Tasks 11/13/20 (a startup fetch on the standby is legitimate; zero-fetch assertions must scope to the deploy window).",
+    "SiteStorageService has 21 (not 22) inline connection+OpenAsync pairs: 60,206,248,311,345,386,414,443,470,498,538,583,608,640,663,692,730,769,813,837,868. CreateConnection():51's only repository consumer is SiteExternalSystemRepository.",
+    "StoreAndForwardService: :39 is the _replication FIELD, :243 the ctor param; exactly 6 emission sites (:654,805,836,872,1120,1146). ServiceCollectionExtensions.cs:32 also resolves ReplicationService inside the StoreAndForwardService factory - must go in Task 14.",
+    "ReplicationMessages.cs holds ONLY the 10 Replicate*/Apply* records; the four SfBuffer resync records live at the bottom of SiteReplicationActor.cs (:678-707) and die with the actor file.",
+    "10 (not 9) appsettings.Site.json files set the legacy paths - deploy/wonder-app-vd03/appsettings.Site.json was missed (also sets ReplicationEnabled:false).",
+    "Rig facts: site metrics on port 8084 (AnyIP in-container, NOT published to host; 8080 is Traefik); real metric names localdb_oplog_depth / localdb_sync_* (meter ZB.MOM.WW.LocalDb.Replication); containers (aspnet:10.0) have NO sqlite3 - data dirs are host bind mounts, run sqlite3 host-side; ReplicationOptions bind at LocalDb:Replication:* (lib binds the section).",
+    "Phase 1's LocalDbSitePairConvergenceTests uses ONE shared API key (:47) - it never did a mismatched-key non-vacuity run; wrong-key denial is unit-covered in Host.Tests/LocalDbSyncAuthInterceptorTests.cs. Task 18's non-vacuity check must be done directly for the new scenarios.",
+    "MigrateEvents synthesizes deterministic 'mig-{node}-{legacyId}' ids (NOT fresh GUIDs - crash-rerun idempotency). Migration runs AFTER RegisterReplicated in OnReady (order is load-bearing); Task 8's oplog pin test must register sf_messages on its own TestLocalDb since production doesn't register it until Task 14."
   ],
   "tasks": [
-    {"id": 1, "subject": "Task 1: Rig soak - measure oplog growth under real write rates", "status": "pending", "classification": "high-risk", "note": "GATES EVERYTHING AFTER TASK 2. Stop the plan if oplog backlog grows monotonically."},
+    {"id": 1, "subject": "Task 1: Rig soak - measure legacy-DB write rates + config_json sizes (corrected method)", "status": "pending", "classification": "high-risk", "note": "GATES EVERYTHING AFTER TASK 2. Phase 2 tables are NOT in the Phase 1 oplog - measure at the legacy DBs host-side, size caps arithmetically; empirical drain check moves to Task 20 evidence 10. Stop conditions: on-paper oplog overflow OR any config_json near 4 MB (D6)."},
     {"id": 2, "subject": "Task 2: Decision record - close the phase 2 gate", "status": "pending", "classification": "trivial", "blockedBy": [1]},
     {"id": 3, "subject": "Task 3: Extract StoreAndForwardSchema.Apply", "status": "pending", "classification": "small", "blockedBy": [2]},
     {"id": 4, "subject": "Task 4: Extract SiteStorageSchema.Apply", "status": "pending", "classification": "small", "blockedBy": [2]},
     {"id": 5, "subject": "Task 5: Rewire StoreAndForwardStorage onto ILocalDb", "status": "pending", "classification": "high-risk", "blockedBy": [3]},
     {"id": 6, "subject": "Task 6: Rewire SiteStorageService onto ILocalDb", "status": "pending", "classification": "high-risk", "blockedBy": [4]},
     {"id": 7, "subject": "Task 7: Extend SiteLocalDbSetup with the new DDL (not yet registered)", "status": "pending", "classification": "standard", "blockedBy": [5, 6]},
-    {"id": 8, "subject": "Task 8: Extend the legacy migrator for sf_messages", "status": "pending", "classification": "high-risk", "blockedBy": [7]},
-    {"id": 9, "subject": "Task 9: Extend the legacy migrator for the 9 config tables", "status": "pending", "classification": "high-risk", "blockedBy": [7]},
+    {"id": 8, "subject": "Task 8: Extend the legacy migrator for sf_messages", "status": "pending", "classification": "high-risk", "blockedBy": [7], "note": "Oplog pin test registers sf_messages on its own TestLocalDb - production OnReady doesn't register it until Task 14."},
+    {"id": 9, "subject": "Task 9: Extend the legacy migrator for the config tables (skip notification/smtp)", "status": "pending", "classification": "high-risk", "blockedBy": [7]},
     {"id": 10, "subject": "Task 10: Port the S&F replication test intents as CDC specs", "status": "pending", "classification": "standard", "blockedBy": [8]},
-    {"id": 11, "subject": "Task 11: Port the resync + directional-authority tests", "status": "pending", "classification": "standard", "blockedBy": [9]},
-    {"id": 12, "subject": "Task 12: Re-home the SMTP purge off the replication path", "status": "pending", "classification": "high-risk", "blockedBy": [10, 11], "note": "MUST land before any deletion so the security purge never lapses."},
-    {"id": 13, "subject": "Task 13: Delete the notify-and-fetch config path", "status": "pending", "classification": "high-risk", "blockedBy": [12]},
-    {"id": 14, "subject": "Task 14: Register the Phase 2 tables and delete ReplicationService", "status": "pending", "classification": "high-risk", "blockedBy": [13], "note": "THE CUTOVER. Registration + bespoke deletion in ONE commit so both mechanisms never run together."},
-    {"id": 15, "subject": "Task 15: Delete SiteReplicationActor and its messages", "status": "pending", "classification": "high-risk", "blockedBy": [14]},
-    {"id": 16, "subject": "Task 16: Clean up DeploymentManagerActor and AkkaHostedService", "status": "pending", "classification": "standard", "blockedBy": [15]},
-    {"id": 17, "subject": "Task 17: Config-key cleanup", "status": "pending", "classification": "standard", "blockedBy": [16]},
-    {"id": 18, "subject": "Task 18: Two-node convergence suite for the Phase 2 tables", "status": "pending", "classification": "high-risk", "blockedBy": [17]},
-    {"id": 19, "subject": "Task 19: Rig configuration", "status": "pending", "classification": "small", "blockedBy": [17]},
-    {"id": 20, "subject": "Task 20: Live gate on the docker rig", "status": "pending", "classification": "high-risk", "blockedBy": [18, 19]},
+    {"id": 11, "subject": "Task 11: Port the resync + directional-authority tests", "status": "pending", "classification": "standard", "blockedBy": [9], "note": "Zero-fetch assertion scoped to deploy convergence - SiteReconciliationActor's startup fetch is legitimate and survives."},
+    {"id": 12, "subject": "Task 12: Pin the active-node SMTP purge (corrected: already exists at DeploymentManagerActor.cs:1921)", "status": "pending", "classification": "standard", "blockedBy": [10, 11], "note": "Pin test only - no production change. Guards Task 16's actor edits from silently dropping the purge call."},
+    {"id": 13, "subject": "Task 13: Notify-and-fetch scope check - guarded write STAYS for SiteReconciliationActor", "status": "pending", "classification": "standard", "blockedBy": [12], "note": "CORRECTED: do NOT delete StoreDeployedConfigIfNewerAsync (second caller SiteReconciliationActor.cs:166 survives; deleting here wouldn't compile anyway). Doc-comment update only; ConfigFetchRetryCount removal moved to Task 17."},
+    {"id": 14, "subject": "Task 14: Register the 8 Phase 2 tables and delete ReplicationService", "status": "pending", "classification": "high-risk", "blockedBy": [13], "note": "THE CUTOVER. Register 7 config tables + sf_messages - NOT notification_lists/smtp_configurations (reversed from original; see D3). Keep Migrate LAST in OnReady. Registration + bespoke deletion in ONE commit."},
+    {"id": 15, "subject": "Task 15: Delete SiteReplicationActor and its messages", "status": "pending", "classification": "high-risk", "blockedBy": [14], "note": "Resync records live at SiteReplicationActor.cs:678-707, not ReplicationMessages.cs."},
+    {"id": 16, "subject": "Task 16: Clean up DeploymentManagerActor and AkkaHostedService", "status": "pending", "classification": "standard", "blockedBy": [15], "note": "Ctor param at :169; same-typed IActorRef? dclManager at :168 BEFORE it is the silent-shift hazard; Props.Create positional at AkkaHostedService.cs:810."},
+    {"id": 17, "subject": "Task 17: Config-key cleanup (incl. ConfigFetchRetryCount; 10 appsettings files)", "status": "pending", "classification": "standard", "blockedBy": [16]},
+    {"id": 18, "subject": "Task 18: Two-node convergence suite for the Phase 2 tables", "status": "pending", "classification": "high-risk", "blockedBy": [17], "note": "Non-vacuity must be proven directly (Phase 1 never did the mismatched-key run the original plan claimed)."},
+    {"id": 19, "subject": "Task 19: Rig configuration (MaxOplogRows/MaxOplogAge + MaxBatchSize per D6)", "status": "pending", "classification": "small", "blockedBy": [17]},
+    {"id": 20, "subject": "Task 20: Live gate on the docker rig (10 evidence items)", "status": "pending", "classification": "high-risk", "blockedBy": [18, 19], "note": "Includes the post-cutover oplog drain-under-churn check (evidence 10) that Task 1 cannot measure pre-cutover. Host-side sqlite3 on bind mounts; metrics on :8084 in-container."},
     {"id": 21, "subject": "Task 21: Documentation truth pass", "status": "pending", "classification": "standard", "blockedBy": [20]}
   ],
   "knownFlakes": [
@@ -85,5 +102,5 @@
     }
   ],
   "lastUpdated": "2026-07-19",
-  "phase2Status": "PLANNED - not started. Task 1 (rig soak) gates the rest."
+  "phase2Status": "PLANNED - not started. Reviewed + corrected 2026-07-19. Task 1 (rig soak) gates the rest."
 }