docs(plans): close residual #9 (R7 merged); register rows 33-34 for R7's findings (oplog backlog observability, purge-burst pacing)

2026-08-15 03:28:23 -04:00
parent 0f455d0f1c
commit 267d774bfe
2 changed files with 13 additions and 5 deletions
@@ -26,6 +26,8 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
 | 30 | **WP-4 residual scope: `[xc-3]` site-event-log retention volume, `[xc-4]` audit-load-vs-central degradation, `[xc-5]` large-template flattening time, `[xc-8]` UI responsiveness under load** | `docs/plans/2026-08-15-target-scale-load-test-results.md` §6 + row 25 closure note | The row-25 harness is a site-runtime rig; these four WP-4 sub-criteria are central-cluster/browser concerns it deliberately does not cover. Recorded so the row-25 closure never reads as "WP-4 fully done" — phase-8 WP-4 is closed for the site-runtime data path only. | Before declaring phase-8 WP-4 complete, or production go-live at target scale |
 | 31 | **Unpaced-burst absorption is bounded by the shared publish stage, not per-subscriber buffers (load-test finding F3 caveat)** | `docs/plans/2026-08-15-target-scale-load-test-results.md` F3 | Measurement-backed design note, no defect: slow-subscriber isolation is structural and total (a stalled reader loses events only in its own bounded `DropOldest` channel; healthy subscribers delivered 100.00%), but the publish `Source.ActorRef` **upstream** of the `BroadcastHub` is shared, so an unpaced burst degrades every subscriber equally — that stage, not per-subscriber buffering, is the burst ceiling. | First field report of event loss on *healthy* subscribers during a burst, or any redesign of the site stream publish stage |
 | 32 | **`QueueDepthGaugeTests` full-suite flake — shared static gauge (load-test finding F4)** | `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/QueueDepthGaugeTests.cs`; surfaced by the row-25 verification runs (`docs/plans/2026-08-15-target-scale-load-test-results.md` F4) | Pre-existing, test-only: fails when the full suite runs concurrently, passes in isolation — the gauge under test is process-global static state shared across test classes. Zero `src/` involvement; needs test isolation (per-test meter/collector instance), not a product fix. | Next test-infra session, or the first time it fails a CI/full-suite run that matters |
+| 33 | **`LocalDbOplogBacklog` has no threshold, KPI history, or dashboard treatment** | Found by R7 (`docs/plans/2026-08-15-residual-remediation-plan.md`) while documenting the purge burst | `SiteAuditBacklog` drives a configurable threshold warning on the site tile, but the LocalDb backlog is reported and Prometheus-exported with nothing consuming it for alerting or trending — it is not among the metrics `SiteHealthKpiSampleSource` samples into KPI history, so the purge-burst spike R7 documents cannot be reviewed after the fact; an operator must be watching live. Candidate: add it to the KPI sample source and/or a sustained-backlog threshold badge. Pairs naturally with the R1 per-table-snapshot work, which changes snapshot/backlog behavior. | Next health-dashboard session, or first operator question about a backlog spike they didn't see live |
+| 34 | **`site_events` purge burst is unbounded in the oplog and unmeasured at scale** | Found by R7 | Retention deletes are sliced at 1000 rows/statement, but the whole expired set is deleted in one tick, so a replicated site's oplog gains one row per deleted event with no pacing — a first purge after enabling replication (or after a long low-purge period) queues the entire batch at once. The wire path is bounded (`MaxBatchBytes`), but the oplog write itself is not paced, and no measurement exists for batch size or drain time at target scale. Candidate: measure on the rig or in the load harness; consider a per-tick delete ceiling if drain time is material. | Before enabling replication on a site with a large existing `site_events` backlog, or the next load-harness session |

 ## Resolved (verified against the code 2026-07-10)
 Rows removed from the Deferred table above once confirmed shipped. Kept here for traceability.
@@ -131,11 +131,17 @@ Deliberately not fixed in this program — each has a stated reason, not an over
   are central-cluster/browser concerns outside this harness and remain open scope.
 8. **Playwright 14 pre-existing env failures.** Present on `main` too, rig-state related, not
   introduced by this branch.
-9. **`site_events` retention purge still oplog-visible.** WP3.2's sliced retention DELETE is a row
-   change like any other and is captured by CDC on a replicated site (site-a) — correct per the
-   "CDC does all three jobs" design (no separate resync path to gate), but means a purge burst is
-   visible in the oplog/backlog metrics; not a correctness issue, just a metrics-reading note for
-   operators watching `LocalDbOplogBacklog` during a purge window.
+9. ~~**`site_events` retention purge still oplog-visible.**~~ **RESOLVED 2026-08-15 (documented
+   by design, `9d2834e3`, R7).** The CDC capture of retention/cap deletes is correct and unchanged
+   — there is deliberately no purge-exemption path. The operator-facing gap is closed three ways:
+   `docs/deployment/topology-guide.md` → *Reading the replication backlog* (burst shape, 24 h
+   cadence anchored to active-node process start, healthy-vs-faulty signature),
+   `Component-SiteEventLogging.md` → Storage, and `Component-HealthMonitoring.md`, which gained
+   the two previously-undocumented `LocalDbReplicationConnected`/`LocalDbOplogBacklog` metric
+   rows. `EventLogPurgeService` now emits one Information line correlating the spike to the purge
+   on a replication-enabled node (both the retention and the storage-cap delete paths), via a
+   Host-supplied `SiteEventLogReplicationCheck` delegate sourced from the single
+   `SiteLocalDbSetup.ReplicationIsConfigured` predicate. No behavior change; 5 unit tests.

 ## Docs propagated