docs+log(siteeventlogging): explain the site_events purge oplog-backlog burst (R7)
The daily site_events retention purge (and the storage-cap trim) is CDC-captured on a replication-enabled site node exactly like any other write — correct by design, since LocalDb Phase 2 deliberately has no purge-exemption path — so the backlog jumps by the deleted batch size at purge time. LocalDbOplogBacklog / localdb_oplog_depth spike, drain, and an operator watching the gauge with no context reads it as a replication fault. Documentation + one log line, no behaviour change: - topology-guide.md gains "Reading the replication backlog — the daily site_events purge burst": when it fires (PurgeInterval 24h, anchored to the active node's PROCESS START, not a wall-clock hour, so it moves after every failover), where it shows (replicated nodes only — not rig site-b/site-c), the healthy signature (LocalDbReplicationConnected stays true, backlog returns to ~0) and what a genuine fault looks like instead. - Component-SiteEventLogging.md Storage records the same under retention/purge; Component-HealthMonitoring.md gains the two previously-undocumented LocalDbReplicationConnected / LocalDbOplogBacklog metric rows carrying the caveat, with cross-references both ways. - EventLogPurgeService emits one Information line naming the row count and the expected transient backlog when a purge deleted rows on a replication-enabled node, so the spike is correlatable in the log. Replication-awareness comes in as a Host-supplied SiteEventLogReplicationCheck delegate, mirroring the existing SiteEventLogActiveNodeCheck seam: SiteLocalDbSetup.ReplicationIsConfigured goes internal so the PeerAddress-OR-ApiKey rule stays in one place and SiteEventLogging never learns to read LocalDb config. Unregistered ⇒ no note, matching the default that replication is opt-in and off. Both delete paths carry the note (a cap trim is usually the larger burst); the predicate is try/caught since a log-wording check must never break the purge. Tests: 5 new EventLogPurgeServiceTests cases (replicated logs it, unreplicated does not, zero-rows does not, cap purge logs it, throwing predicate still purges and swallows) via a local capturing ILogger. SiteEventLogging 81/81 green, Host 490/490 green, full solution build clean (0 warnings).
This commit is contained in:
@@ -198,6 +198,50 @@ that drops a table its peer still replicates stops syncing with a schema-mismatc
|
||||
diverging silently. Turning it on again later re-baselines, which is what makes the ledger prune on
|
||||
deregistration safe.
|
||||
|
||||
#### Reading the replication backlog — the daily `site_events` purge burst
|
||||
|
||||
The site health report carries `LocalDbReplicationConnected` and `LocalDbOplogBacklog` (nullable —
|
||||
**null means "no reading", not "disconnected with an empty backlog"**), and the same numbers export
|
||||
as the `localdb_*` Prometheus series (`localdb_oplog_depth` is the backlog gauge). A healthy pair
|
||||
sits at a backlog of roughly zero, so a sudden spike naturally reads as a replication problem.
|
||||
|
||||
**One expected spike is not a problem: the daily `site_events` retention purge.** `site_events` is
|
||||
one of the ten replicated tables, and its retention DELETE is captured by CDC exactly like an
|
||||
ordinary write — by design, since LocalDb Phase 2 there is deliberately no purge-exemption path
|
||||
(the same property that makes a mass DELETE dangerous, which is why `ReplaceAllAsync` was deleted
|
||||
rather than reinstated). One oplog row is therefore queued per deleted event, and the backlog jumps
|
||||
by the size of the day's expired batch.
|
||||
|
||||
- **When.** Every `ScadaBridge:SiteEventLog:PurgeInterval` (default **24 h**), plus once at
|
||||
startup. The timer is anchored to the **active node's process start**, not to a wall-clock hour,
|
||||
so the burst lands at a different time of day after each failover or restart — do not expect it
|
||||
at a fixed hour. The storage-cap trim (default 1 GB) can produce the same shape off-schedule, and
|
||||
is usually the larger of the two.
|
||||
- **Where it shows.** Only on a node with replication configured — the rig's site-a. site-b/site-c
|
||||
have no capture triggers at all and report no backlog for a purge.
|
||||
- **What healthy looks like.** `LocalDbReplicationConnected` stays **true** across the spike, and
|
||||
the backlog drains back to ~0 as the peer acks the batch — within seconds to a couple of minutes
|
||||
depending on batch size (delta messages are bounded by `LocalDb:Replication:MaxBatchBytes`,
|
||||
default 2 MB, and secondarily by `MaxBatchSize`). No dead letters, no schema-mismatch errors.
|
||||
- **What is actually wrong.** `LocalDbReplicationConnected` **false** while the backlog climbs, a
|
||||
backlog that keeps rising across successive readings rather than draining, or a backlog that
|
||||
never returns near zero between bursts. Those point at the sync stream — an ApiKey mismatch
|
||||
(fail-closed: the pair simply stops converging), an unreachable peer, or an asymmetric
|
||||
registered-table set.
|
||||
|
||||
**Correlating it in the log.** When a purge on a replication-enabled node actually deletes rows it
|
||||
logs an Information line next to the purge count:
|
||||
|
||||
```
|
||||
Purged 41230 events older than 30 days
|
||||
Purged 41230 site_events rows on a replication-enabled node — a transient LocalDb oplog backlog
|
||||
is expected while the deletes replicate to the peer. It drains on its own;
|
||||
LocalDbReplicationConnected staying true with the backlog returning to ~0 is the healthy signature.
|
||||
```
|
||||
|
||||
An unreplicated node logs only the first line. If a backlog spike has no such line near it in the
|
||||
active node's log, the purge is *not* the explanation and the spike is worth investigating.
|
||||
|
||||
### Site Pair Upgrades — stop and start BOTH nodes together
|
||||
|
||||
**A rolling upgrade of a site pair, one node at a time, is no longer supported.** It worked while
|
||||
|
||||
Reference in New Issue
Block a user