docs(localdb): phase 2 truth pass across both repos

Normative first: Component-StoreAndForward.md:83 specified the whole chunked, ack-confirmed SfBufferSnapshotChunk resync protocol, which Phase 2 deleted. It is rewritten rather than removed — the failure modes it reasoned about still exist, they are just bounded differently — and it now states the duplicate-delivery bound explicitly: a message can be delivered twice only when the OLD primary delivered it and the status change had not yet replicated when the gate flipped. One flush interval plus the in-flight ack, and unlike the old model it does NOT grow with backlog depth or with how long a node was absent. The N1 directional-authority and N5 orphan-row hazards are recorded as structurally gone, not merely unguarded. Frame-size known-issue amended: its 2026-06-26 resolution replaced the intra-site hop with notify-and-fetch; Phase 2 then deleted notify-and-fetch itself, so the 128 KB Akka frame constraint no longer applies to that hop in any form. Successor ceiling recorded (4 MB gRPC cap via MaxBatchSize, which batches by ROW COUNT), including that the failure mode differs — an oversized gRPC message is rejected, not silently dropped. Deployment docs gain the two operational constraints that have no home in code: a site pair must be stopped and started TOGETHER (the SfBufferSnapshot compat handler that made a mixed-version pair converge went with the replicator, and a mixed pair now diverges silently), and a node offline beyond TombstoneRetention can resurrect deleted rows on rejoin. Both CLAUDE.md files corrected — each still said Phase 2 was NOT started. Definition of done closed: build 0 warnings, all 10 suites green (3509 tests, 0 failures, 0 skips). Two DoD items needed amending rather than ticking: the stale-symbol grep still matches 4 lines, all deliberate comment prose recording what was deleted (a literal zero would delete the explanations that stop the old design coming back), and the live gate has 10 evidence items, not the 9 the checklist claimed. Deletes the phase2 resume-state scratch doc, which said to delete it on landing. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-20 05:37:48 -04:00
parent 158e79bb50
commit 7b5a5a6f34
11 changed files with 89 additions and 219 deletions
@@ -10,6 +10,27 @@ Fixed via the **notify-and-fetch** rework (the primary recommendation below), no
 - **Plan:** [`docs/plans/2026-06-26-deploy-config-notify-and-fetch.md`](../plans/2026-06-26-deploy-config-notify-and-fetch.md)
 - **Validated:** live docker-cluster smoke — a previously-hanging deploy now completes in ~0.11 s; reconciliation heals single-node and concurrent-both-missing gaps.

+## Amendment (2026-07-20) — LocalDb Phase 2 removed the second hop entirely
+
+The resolution above fixed the intra-site hop by replacing it with notify-and-fetch. LocalDb
+Phase 2 then deleted **notify-and-fetch itself**, along with `SiteReplicationActor`: the site's
+`deployed_configurations` table is now replicated by CDC, so the config reaches the standby as an
+ordinary row change over the gRPC sync stream. There is no intra-site Akka hop carrying config any
+more, so the 128 000-byte frame constraint does not apply to it in any form.
+
+The central→site hop is unchanged — it still sends a small `RefreshDeploymentCommand` and the site
+still fetches over HTTP, so that half of the original fix stands.
+
+**The successor ceiling is different in kind.** The gRPC sync stream has a 4 MB default receive
+limit, and LocalDb batches by ROW COUNT (`LocalDb:Replication:MaxBatchSize`, default 500), not by
+bytes. A ~70 KB `config_json` — the largest measured in production — times 500 rows is ~35 MB,
+which would exceed the limit. The rig therefore pins `MaxBatchSize` to **16** (~1.1 MB worst case).
+Any deployment replicating wide rows must size that key deliberately; see the Phase 2 plan (D6) and
+`docs/plans/2026-07-19-localdb-phase2-live-gate.md`.
+
+Note the failure mode differs from the one documented below: an oversized gRPC message is
+**rejected**, not silently dropped.
+
 The diagnosis below is retained as the historical record of how the bug was found and reasoned about.

 ## Summary