docs+log(siteeventlogging): explain the site_events purge oplog-backlog burst (R7)
The daily site_events retention purge (and the storage-cap trim) is CDC-captured on a replication-enabled site node exactly like any other write — correct by design, since LocalDb Phase 2 deliberately has no purge-exemption path — so the backlog jumps by the deleted batch size at purge time. LocalDbOplogBacklog / localdb_oplog_depth spike, drain, and an operator watching the gauge with no context reads it as a replication fault. Documentation + one log line, no behaviour change: - topology-guide.md gains "Reading the replication backlog — the daily site_events purge burst": when it fires (PurgeInterval 24h, anchored to the active node's PROCESS START, not a wall-clock hour, so it moves after every failover), where it shows (replicated nodes only — not rig site-b/site-c), the healthy signature (LocalDbReplicationConnected stays true, backlog returns to ~0) and what a genuine fault looks like instead. - Component-SiteEventLogging.md Storage records the same under retention/purge; Component-HealthMonitoring.md gains the two previously-undocumented LocalDbReplicationConnected / LocalDbOplogBacklog metric rows carrying the caveat, with cross-references both ways. - EventLogPurgeService emits one Information line naming the row count and the expected transient backlog when a purge deleted rows on a replication-enabled node, so the spike is correlatable in the log. Replication-awareness comes in as a Host-supplied SiteEventLogReplicationCheck delegate, mirroring the existing SiteEventLogActiveNodeCheck seam: SiteLocalDbSetup.ReplicationIsConfigured goes internal so the PeerAddress-OR-ApiKey rule stays in one place and SiteEventLogging never learns to read LocalDb config. Unregistered ⇒ no note, matching the default that replication is opt-in and off. Both delete paths carry the note (a cap trim is usually the larger burst); the predicate is try/caught since a log-wording check must never break the purge. Tests: 5 new EventLogPurgeServiceTests cases (replicated logs it, unreplicated does not, zero-rows does not, cap purge logs it, throwing predicate still purges and swallows) via a local capturing ILogger. SiteEventLogging 81/81 green, Host 490/490 green, full solution build clean (0 warnings).
This commit is contained in:
@@ -92,8 +92,27 @@ Each event entry contains:
|
||||
On a site pair running **without** replication configured (by deliberate choice, e.g. the
|
||||
rig's site-b/site-c), the log stays node-local and a failover does start it fresh — the
|
||||
documented trade of not configuring a peer, unchanged by this policy.
|
||||
- **Retention**: 30 days. A **daily background job** runs on the active node and deletes all events older than 30 days. Hard delete — no archival. (Today, retention/cap deletes on a replicated node are captured by CDC like any other write — see the Volume Policy section above for why this is a small residual cost now that per-run rows are off by default.)
|
||||
- **Retention**: 30 days. A **daily background job** runs on the active node and deletes all events older than 30 days. Hard delete — no archival. Deletes are sliced into bounded 1000-row batches per DELETE statement rather than one unbounded statement, so a large expired backlog does not hold the write lock (and every concurrent recorder flush) for the duration of the purge.
|
||||
- **Storage cap**: A configurable maximum database size (default: 1 GB) is enforced. If the storage cap is reached before the 30-day retention window, the oldest events are purged first. This prevents disk exhaustion from alarm storms, script failure loops, or connection flapping.
|
||||
- **Purge deletes are CDC-captured on a replicated site — the resulting backlog spike is expected.**
|
||||
Retention and cap deletes are ordinary row changes on a replicated table; there is deliberately no
|
||||
purge-exemption path (LocalDb Phase 2 — CDC does all three jobs, and a table-wide exemption is the
|
||||
same mechanism that made `ReplaceAllAsync` unsafe). One oplog row is queued per deleted event, so
|
||||
the site health report's `LocalDbOplogBacklog` (Prometheus `localdb_oplog_depth`) jumps by the size
|
||||
of the batch at purge time and drains as the peer acks it. **Healthy signature:**
|
||||
`LocalDbReplicationConnected` stays true across the spike and the backlog returns to ~0; a backlog
|
||||
that keeps climbing, or climbs while `LocalDbReplicationConnected` is false, is a genuine
|
||||
replication fault and not the purge. The purge logs an Information line naming the row count and
|
||||
the expected transient backlog whenever it deletes rows on a replication-enabled node, so an
|
||||
operator can correlate a spike with the purge that caused it — a spike with no such line nearby is
|
||||
not the purge. Node-local wiring: the Host supplies the replication predicate
|
||||
(`SiteLocalDbSetup.ReplicationIsConfigured` — the `PeerAddress`-OR-`ApiKey` rule) as
|
||||
`SiteEventLogReplicationCheck`; this component never reads LocalDb configuration itself, and with
|
||||
no predicate registered the note is suppressed. Operator detail —
|
||||
including that the purge timer is anchored to the active node's process start, not a wall-clock
|
||||
hour — is in `docs/deployment/topology-guide.md` → *Reading the replication backlog*. See also the
|
||||
Volume Policy section above for why this residual cost is small now that per-run script rows are
|
||||
off by default.
|
||||
|
||||
## Central Access
|
||||
|
||||
@@ -121,4 +140,4 @@ Each event entry contains:
|
||||
- **Communication Layer**: Receives remote queries from central and returns results.
|
||||
- **Central UI**: Site Event Log Viewer displays queried events.
|
||||
- **Store-and-Forward Engine**: Its notification path (the site→central forward of script-generated notifications) reports forward failures and long-buffered notifications as Notification-category events. Routine enqueue and forward-success events are deliberately not logged — central's authoritative `Notifications` table (owned by the Notification Outbox component) is the audit record of record; site-side logging covers only the in-transit blind spot when central is unreachable.
|
||||
- **Health Monitoring**: Script error rates and alarm evaluation error rates can be derived from event log data.
|
||||
- **Health Monitoring**: Script error rates and alarm evaluation error rates can be derived from event log data; this component also reports `SiteEventLogWriteFailures`, and its purge is the expected cause of the transient `LocalDbOplogBacklog` spikes described in [Component-HealthMonitoring.md](Component-HealthMonitoring.md) → Monitored Metrics.
|
||||
|
||||
Reference in New Issue
Block a user