From 267d774bfe669aafef63628b49bc2202f8624383 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sat, 15 Aug 2026 03:28:23 -0400 Subject: [PATCH] docs(plans): close residual #9 (R7 merged); register rows 33-34 for R7's findings (oplog backlog observability, purge-burst pacing) --- docs/plans/2026-07-08-deferred-work-register.md | 2 ++ ...8-15-arch-review-remediation-execution-log.md | 16 +++++++++++----- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/docs/plans/2026-07-08-deferred-work-register.md b/docs/plans/2026-07-08-deferred-work-register.md index 213c44ca..fe4af06a 100644 --- a/docs/plans/2026-07-08-deferred-work-register.md +++ b/docs/plans/2026-07-08-deferred-work-register.md @@ -26,6 +26,8 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round | 30 | **WP-4 residual scope: `[xc-3]` site-event-log retention volume, `[xc-4]` audit-load-vs-central degradation, `[xc-5]` large-template flattening time, `[xc-8]` UI responsiveness under load** | `docs/plans/2026-08-15-target-scale-load-test-results.md` §6 + row 25 closure note | The row-25 harness is a site-runtime rig; these four WP-4 sub-criteria are central-cluster/browser concerns it deliberately does not cover. Recorded so the row-25 closure never reads as "WP-4 fully done" — phase-8 WP-4 is closed for the site-runtime data path only. | Before declaring phase-8 WP-4 complete, or production go-live at target scale | | 31 | **Unpaced-burst absorption is bounded by the shared publish stage, not per-subscriber buffers (load-test finding F3 caveat)** | `docs/plans/2026-08-15-target-scale-load-test-results.md` F3 | Measurement-backed design note, no defect: slow-subscriber isolation is structural and total (a stalled reader loses events only in its own bounded `DropOldest` channel; healthy subscribers delivered 100.00%), but the publish `Source.ActorRef` **upstream** of the `BroadcastHub` is shared, so an unpaced burst degrades every subscriber equally — that stage, not per-subscriber buffering, is the burst ceiling. | First field report of event loss on *healthy* subscribers during a burst, or any redesign of the site stream publish stage | | 32 | **`QueueDepthGaugeTests` full-suite flake — shared static gauge (load-test finding F4)** | `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/QueueDepthGaugeTests.cs`; surfaced by the row-25 verification runs (`docs/plans/2026-08-15-target-scale-load-test-results.md` F4) | Pre-existing, test-only: fails when the full suite runs concurrently, passes in isolation — the gauge under test is process-global static state shared across test classes. Zero `src/` involvement; needs test isolation (per-test meter/collector instance), not a product fix. | Next test-infra session, or the first time it fails a CI/full-suite run that matters | +| 33 | **`LocalDbOplogBacklog` has no threshold, KPI history, or dashboard treatment** | Found by R7 (`docs/plans/2026-08-15-residual-remediation-plan.md`) while documenting the purge burst | `SiteAuditBacklog` drives a configurable threshold warning on the site tile, but the LocalDb backlog is reported and Prometheus-exported with nothing consuming it for alerting or trending — it is not among the metrics `SiteHealthKpiSampleSource` samples into KPI history, so the purge-burst spike R7 documents cannot be reviewed after the fact; an operator must be watching live. Candidate: add it to the KPI sample source and/or a sustained-backlog threshold badge. Pairs naturally with the R1 per-table-snapshot work, which changes snapshot/backlog behavior. | Next health-dashboard session, or first operator question about a backlog spike they didn't see live | +| 34 | **`site_events` purge burst is unbounded in the oplog and unmeasured at scale** | Found by R7 | Retention deletes are sliced at 1000 rows/statement, but the whole expired set is deleted in one tick, so a replicated site's oplog gains one row per deleted event with no pacing — a first purge after enabling replication (or after a long low-purge period) queues the entire batch at once. The wire path is bounded (`MaxBatchBytes`), but the oplog write itself is not paced, and no measurement exists for batch size or drain time at target scale. Candidate: measure on the rig or in the load harness; consider a per-tick delete ceiling if drain time is material. | Before enabling replication on a site with a large existing `site_events` backlog, or the next load-harness session | ## Resolved (verified against the code 2026-07-10) Rows removed from the Deferred table above once confirmed shipped. Kept here for traceability. diff --git a/docs/plans/2026-08-15-arch-review-remediation-execution-log.md b/docs/plans/2026-08-15-arch-review-remediation-execution-log.md index f80f44f6..f80fa5bb 100644 --- a/docs/plans/2026-08-15-arch-review-remediation-execution-log.md +++ b/docs/plans/2026-08-15-arch-review-remediation-execution-log.md @@ -131,11 +131,17 @@ Deliberately not fixed in this program — each has a stated reason, not an over are central-cluster/browser concerns outside this harness and remain open scope. 8. **Playwright 14 pre-existing env failures.** Present on `main` too, rig-state related, not introduced by this branch. -9. **`site_events` retention purge still oplog-visible.** WP3.2's sliced retention DELETE is a row - change like any other and is captured by CDC on a replicated site (site-a) — correct per the - "CDC does all three jobs" design (no separate resync path to gate), but means a purge burst is - visible in the oplog/backlog metrics; not a correctness issue, just a metrics-reading note for - operators watching `LocalDbOplogBacklog` during a purge window. +9. ~~**`site_events` retention purge still oplog-visible.**~~ **RESOLVED 2026-08-15 (documented + by design, `9d2834e3`, R7).** The CDC capture of retention/cap deletes is correct and unchanged + — there is deliberately no purge-exemption path. The operator-facing gap is closed three ways: + `docs/deployment/topology-guide.md` → *Reading the replication backlog* (burst shape, 24 h + cadence anchored to active-node process start, healthy-vs-faulty signature), + `Component-SiteEventLogging.md` → Storage, and `Component-HealthMonitoring.md`, which gained + the two previously-undocumented `LocalDbReplicationConnected`/`LocalDbOplogBacklog` metric + rows. `EventLogPurgeService` now emits one Information line correlating the spike to the purge + on a replication-enabled node (both the retention and the storage-cap delete paths), via a + Host-supplied `SiteEventLogReplicationCheck` delegate sourced from the single + `SiteLocalDbSetup.ReplicationIsConfigured` predicate. No behavior change; 5 unit tests. ## Docs propagated