docs+log(siteeventlogging): explain the site_events purge oplog-backlog burst (R7)

The daily site_events retention purge (and the storage-cap trim) is CDC-captured
on a replication-enabled site node exactly like any other write — correct by
design, since LocalDb Phase 2 deliberately has no purge-exemption path — so the
backlog jumps by the deleted batch size at purge time. LocalDbOplogBacklog /
localdb_oplog_depth spike, drain, and an operator watching the gauge with no
context reads it as a replication fault.

Documentation + one log line, no behaviour change:

- topology-guide.md gains "Reading the replication backlog — the daily
  site_events purge burst": when it fires (PurgeInterval 24h, anchored to the
  active node's PROCESS START, not a wall-clock hour, so it moves after every
  failover), where it shows (replicated nodes only — not rig site-b/site-c),
  the healthy signature (LocalDbReplicationConnected stays true, backlog
  returns to ~0) and what a genuine fault looks like instead.
- Component-SiteEventLogging.md Storage records the same under retention/purge;
  Component-HealthMonitoring.md gains the two previously-undocumented
  LocalDbReplicationConnected / LocalDbOplogBacklog metric rows carrying the
  caveat, with cross-references both ways.
- EventLogPurgeService emits one Information line naming the row count and the
  expected transient backlog when a purge deleted rows on a replication-enabled
  node, so the spike is correlatable in the log. Replication-awareness comes in
  as a Host-supplied SiteEventLogReplicationCheck delegate, mirroring the
  existing SiteEventLogActiveNodeCheck seam: SiteLocalDbSetup.ReplicationIsConfigured
  goes internal so the PeerAddress-OR-ApiKey rule stays in one place and
  SiteEventLogging never learns to read LocalDb config. Unregistered ⇒ no note,
  matching the default that replication is opt-in and off.

Both delete paths carry the note (a cap trim is usually the larger burst); the
predicate is try/caught since a log-wording check must never break the purge.

Tests: 5 new EventLogPurgeServiceTests cases (replicated logs it, unreplicated
does not, zero-rows does not, cap purge logs it, throwing predicate still purges
and swallows) via a local capturing ILogger. SiteEventLogging 81/81 green,
Host 490/490 green, full solution build clean (0 warnings).
This commit is contained in:
Joseph Doherty
2026-08-15 03:26:30 -04:00
parent 2b74851f96
commit 9d2834e30a
9 changed files with 322 additions and 9 deletions
@@ -37,6 +37,8 @@ Site clusters (metric collection and reporting). Central cluster (aggregation an
| `SiteAuditBacklog` | Audit Log (site) | Count of `Pending` rows in the site-local `AuditLog` plus oldest-pending-age plus on-disk bytes. A configurable threshold drives a Health dashboard warning on the affected site tile. |
| `SiteAuditWriteFailures` | Audit Log (site) | Count of failed hot-path audit appends at the site since the last health report. |
| `AuditRedactionFailure` | Audit Log (central) | Count of payload redactor errors (over-redacted payloads, safety-net hit) since the last interval. |
| `LocalDbReplicationConnected` | Consolidated site LocalDb | Whether a peer sync session is currently running on this node. **Nullable — null means "no reading", not "disconnected".** Reported only by a node with replication configured. |
| `LocalDbOplogBacklog` | Consolidated site LocalDb | Unacked oplog depth (Prometheus `localdb_oplog_depth`). **Nullable — null means "no reading", NOT "connected with an empty backlog"** (a failed poll rendered as 0 would report a pair that cannot read its own oplog as perfectly healthy). **Expected transient spikes:** the daily `site_events` retention purge and the storage-cap trim are CDC-captured like any other write, so the backlog jumps by the batch size at purge time and drains as the peer acks it — see `Component-SiteEventLogging.md` → Storage and `docs/deployment/topology-guide.md`*Reading the replication backlog* for the healthy-versus-faulty signature. |
## Reporting Protocol
@@ -113,6 +115,7 @@ These tiles are **point-in-time** like the Notification Outbox and Site Call Aud
- **Cluster Infrastructure (site)**: Provides node role status.
- **Notification Outbox (central)**: Provides central-computed outbox KPIs — queue depth, stuck count, parked count — for the headline dashboard tiles.
- **Site Call Audit (central)**: Provides central-computed cached-call KPIs — buffered count, parked count, failed/delivered (last interval), oldest pending age, stuck count — for the headline dashboard tiles.
- **Site Event Logging (site)**: Provides `SiteEventLogWriteFailures`. Its daily retention purge and storage-cap trim are also the expected cause of transient `LocalDbOplogBacklog` spikes on a replicated node — see [Component-SiteEventLogging.md](Component-SiteEventLogging.md) → Storage.
- **Audit Log (#23)**: Provides the site-reported `SiteAuditBacklog` / `SiteAuditWriteFailures` metrics (via the site health report) and the central-computed `AuditRedactionFailure` metric, plus the central audit-row rate feeding the **Audit** dashboard tile group (Audit volume, Audit error rate, Audit backlog).
## Interactions