docs+log(siteeventlogging): explain the site_events purge oplog-backlog burst (R7)
The daily site_events retention purge (and the storage-cap trim) is CDC-captured on a replication-enabled site node exactly like any other write — correct by design, since LocalDb Phase 2 deliberately has no purge-exemption path — so the backlog jumps by the deleted batch size at purge time. LocalDbOplogBacklog / localdb_oplog_depth spike, drain, and an operator watching the gauge with no context reads it as a replication fault. Documentation + one log line, no behaviour change: - topology-guide.md gains "Reading the replication backlog — the daily site_events purge burst": when it fires (PurgeInterval 24h, anchored to the active node's PROCESS START, not a wall-clock hour, so it moves after every failover), where it shows (replicated nodes only — not rig site-b/site-c), the healthy signature (LocalDbReplicationConnected stays true, backlog returns to ~0) and what a genuine fault looks like instead. - Component-SiteEventLogging.md Storage records the same under retention/purge; Component-HealthMonitoring.md gains the two previously-undocumented LocalDbReplicationConnected / LocalDbOplogBacklog metric rows carrying the caveat, with cross-references both ways. - EventLogPurgeService emits one Information line naming the row count and the expected transient backlog when a purge deleted rows on a replication-enabled node, so the spike is correlatable in the log. Replication-awareness comes in as a Host-supplied SiteEventLogReplicationCheck delegate, mirroring the existing SiteEventLogActiveNodeCheck seam: SiteLocalDbSetup.ReplicationIsConfigured goes internal so the PeerAddress-OR-ApiKey rule stays in one place and SiteEventLogging never learns to read LocalDb config. Unregistered ⇒ no note, matching the default that replication is opt-in and off. Both delete paths carry the note (a cap trim is usually the larger burst); the predicate is try/caught since a log-wording check must never break the purge. Tests: 5 new EventLogPurgeServiceTests cases (replicated logs it, unreplicated does not, zero-rows does not, cap purge logs it, throwing predicate still purges and swallows) via a local capturing ILogger. SiteEventLogging 81/81 green, Host 490/490 green, full solution build clean (0 warnings).
This commit is contained in:
@@ -83,7 +83,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
|||||||
- `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs.
|
- `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs.
|
||||||
- `ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath` and — as of Phase 2 — `ScadaBridge:StoreAndForward:SqliteDbPath` + `ScadaBridge:Database:SiteDbPath` are all **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated.
|
- `ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath` and — as of Phase 2 — `ScadaBridge:StoreAndForward:SqliteDbPath` + `ScadaBridge:Database:SiteDbPath` are all **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated.
|
||||||
- This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate.
|
- This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate.
|
||||||
- **Replication is default-OFF and opt-in** via `LocalDb:Replication:PeerAddress` + a matching `ApiKey` on both nodes. `LocalDbSyncAuthInterceptor` is **fail-closed**: no configured key means no sync stream is accepted at all, so a key typo does not degrade to unauthenticated replication — the pair simply stops converging. The sync endpoint shares the existing site gRPC h2c listener (8083); no new port. Rig posture: **site-a replicated, site-b/site-c deliberately not**, so both states are proven side-by-side. Status surfaces on the site health report as `LocalDbReplicationConnected` / `LocalDbOplogBacklog` (both nullable — null means "no data", NOT "disconnected with an empty backlog") and as `localdb_*` Prometheus series. Note `ZbTelemetryOptions.Meters` is an **allowlist** (`SiteServiceRegistration.ObservedMeters`); an unlisted meter exports nothing, silently.
|
- **Replication is default-OFF and opt-in** via `LocalDb:Replication:PeerAddress` + a matching `ApiKey` on both nodes. `LocalDbSyncAuthInterceptor` is **fail-closed**: no configured key means no sync stream is accepted at all, so a key typo does not degrade to unauthenticated replication — the pair simply stops converging. The sync endpoint shares the existing site gRPC h2c listener (8083); no new port. Rig posture: **site-a replicated, site-b/site-c deliberately not**, so both states are proven side-by-side. Status surfaces on the site health report as `LocalDbReplicationConnected` / `LocalDbOplogBacklog` (both nullable — null means "no data", NOT "disconnected with an empty backlog") and as `localdb_*` Prometheus series. **A transient backlog spike is expected daily and is not a fault:** the `site_events` retention purge (and the storage-cap trim) is CDC-captured like any other write — there is deliberately no purge-exemption path — so the backlog jumps by the deleted batch size and drains as the peer acks it; the purge logs an Information note saying so on a replication-enabled node (`SiteEventLogReplicationCheck`, supplied by the Host from `SiteLocalDbSetup.ReplicationIsConfigured` so SiteEventLogging never reads LocalDb config itself). Healthy = `LocalDbReplicationConnected` stays true and the backlog returns to ~0; see `docs/deployment/topology-guide.md` → *Reading the replication backlog*. Note `ZbTelemetryOptions.Meters` is an **allowlist** (`SiteServiceRegistration.ObservedMeters`); an unlisted meter exports nothing, silently.
|
||||||
- Design: **scadaproj** `docs/plans/2026-07-19-scadabridge-localdb-design.md` (that doc lives in the umbrella repo, not here); Phase 1 plan + Phase 2 gate are here under `docs/plans/`. Phase 2 **deleted** `SiteReplicationActor`, its `ReplicationMessages`, StoreAndForward's `ReplicationService`, and `StoreAndForwardStorage.ReplaceAllAsync` — do not reintroduce them:
|
- Design: **scadaproj** `docs/plans/2026-07-19-scadabridge-localdb-design.md` (that doc lives in the umbrella repo, not here); Phase 1 plan + Phase 2 gate are here under `docs/plans/`. Phase 2 **deleted** `SiteReplicationActor`, its `ReplicationMessages`, StoreAndForward's `ReplicationService`, and `StoreAndForwardStorage.ReplaceAllAsync` — do not reintroduce them:
|
||||||
|
|
||||||
- CDC replication does all three jobs now: config deploys reach the standby as ordinary row changes — **the standby makes no fetch at all** during a deploy (`SiteReconciliationActor`'s node-STARTUP fetch when central reports gaps is a different, surviving path) — and buffer mutations replicate via triggers on `sf_messages`. `ReplaceAllAsync` was a destructive delete-all-then-insert-all resync and is **unsafe to reintroduce**: a mass DELETE on a replicated table would be captured and shipped to the peer. LocalDb's snapshot resync merges per row under LWW and never deletes, which is also why the old N1 directional-authority guard is gone — there is no wipe left to gate.
|
- CDC replication does all three jobs now: config deploys reach the standby as ordinary row changes — **the standby makes no fetch at all** during a deploy (`SiteReconciliationActor`'s node-STARTUP fetch when central reports gaps is a different, surviving path) — and buffer mutations replicate via triggers on `sf_messages`. `ReplaceAllAsync` was a destructive delete-all-then-insert-all resync and is **unsafe to reintroduce**: a mass DELETE on a replicated table would be captured and shipped to the peer. LocalDb's snapshot resync merges per row under LWW and never deletes, which is also why the old N1 directional-authority guard is gone — there is no wipe left to gate.
|
||||||
|
|||||||
@@ -198,6 +198,50 @@ that drops a table its peer still replicates stops syncing with a schema-mismatc
|
|||||||
diverging silently. Turning it on again later re-baselines, which is what makes the ledger prune on
|
diverging silently. Turning it on again later re-baselines, which is what makes the ledger prune on
|
||||||
deregistration safe.
|
deregistration safe.
|
||||||
|
|
||||||
|
#### Reading the replication backlog — the daily `site_events` purge burst
|
||||||
|
|
||||||
|
The site health report carries `LocalDbReplicationConnected` and `LocalDbOplogBacklog` (nullable —
|
||||||
|
**null means "no reading", not "disconnected with an empty backlog"**), and the same numbers export
|
||||||
|
as the `localdb_*` Prometheus series (`localdb_oplog_depth` is the backlog gauge). A healthy pair
|
||||||
|
sits at a backlog of roughly zero, so a sudden spike naturally reads as a replication problem.
|
||||||
|
|
||||||
|
**One expected spike is not a problem: the daily `site_events` retention purge.** `site_events` is
|
||||||
|
one of the ten replicated tables, and its retention DELETE is captured by CDC exactly like an
|
||||||
|
ordinary write — by design, since LocalDb Phase 2 there is deliberately no purge-exemption path
|
||||||
|
(the same property that makes a mass DELETE dangerous, which is why `ReplaceAllAsync` was deleted
|
||||||
|
rather than reinstated). One oplog row is therefore queued per deleted event, and the backlog jumps
|
||||||
|
by the size of the day's expired batch.
|
||||||
|
|
||||||
|
- **When.** Every `ScadaBridge:SiteEventLog:PurgeInterval` (default **24 h**), plus once at
|
||||||
|
startup. The timer is anchored to the **active node's process start**, not to a wall-clock hour,
|
||||||
|
so the burst lands at a different time of day after each failover or restart — do not expect it
|
||||||
|
at a fixed hour. The storage-cap trim (default 1 GB) can produce the same shape off-schedule, and
|
||||||
|
is usually the larger of the two.
|
||||||
|
- **Where it shows.** Only on a node with replication configured — the rig's site-a. site-b/site-c
|
||||||
|
have no capture triggers at all and report no backlog for a purge.
|
||||||
|
- **What healthy looks like.** `LocalDbReplicationConnected` stays **true** across the spike, and
|
||||||
|
the backlog drains back to ~0 as the peer acks the batch — within seconds to a couple of minutes
|
||||||
|
depending on batch size (delta messages are bounded by `LocalDb:Replication:MaxBatchBytes`,
|
||||||
|
default 2 MB, and secondarily by `MaxBatchSize`). No dead letters, no schema-mismatch errors.
|
||||||
|
- **What is actually wrong.** `LocalDbReplicationConnected` **false** while the backlog climbs, a
|
||||||
|
backlog that keeps rising across successive readings rather than draining, or a backlog that
|
||||||
|
never returns near zero between bursts. Those point at the sync stream — an ApiKey mismatch
|
||||||
|
(fail-closed: the pair simply stops converging), an unreachable peer, or an asymmetric
|
||||||
|
registered-table set.
|
||||||
|
|
||||||
|
**Correlating it in the log.** When a purge on a replication-enabled node actually deletes rows it
|
||||||
|
logs an Information line next to the purge count:
|
||||||
|
|
||||||
|
```
|
||||||
|
Purged 41230 events older than 30 days
|
||||||
|
Purged 41230 site_events rows on a replication-enabled node — a transient LocalDb oplog backlog
|
||||||
|
is expected while the deletes replicate to the peer. It drains on its own;
|
||||||
|
LocalDbReplicationConnected staying true with the backlog returning to ~0 is the healthy signature.
|
||||||
|
```
|
||||||
|
|
||||||
|
An unreplicated node logs only the first line. If a backlog spike has no such line near it in the
|
||||||
|
active node's log, the purge is *not* the explanation and the spike is worth investigating.
|
||||||
|
|
||||||
### Site Pair Upgrades — stop and start BOTH nodes together
|
### Site Pair Upgrades — stop and start BOTH nodes together
|
||||||
|
|
||||||
**A rolling upgrade of a site pair, one node at a time, is no longer supported.** It worked while
|
**A rolling upgrade of a site pair, one node at a time, is no longer supported.** It worked while
|
||||||
|
|||||||
@@ -37,6 +37,8 @@ Site clusters (metric collection and reporting). Central cluster (aggregation an
|
|||||||
| `SiteAuditBacklog` | Audit Log (site) | Count of `Pending` rows in the site-local `AuditLog` plus oldest-pending-age plus on-disk bytes. A configurable threshold drives a Health dashboard warning on the affected site tile. |
|
| `SiteAuditBacklog` | Audit Log (site) | Count of `Pending` rows in the site-local `AuditLog` plus oldest-pending-age plus on-disk bytes. A configurable threshold drives a Health dashboard warning on the affected site tile. |
|
||||||
| `SiteAuditWriteFailures` | Audit Log (site) | Count of failed hot-path audit appends at the site since the last health report. |
|
| `SiteAuditWriteFailures` | Audit Log (site) | Count of failed hot-path audit appends at the site since the last health report. |
|
||||||
| `AuditRedactionFailure` | Audit Log (central) | Count of payload redactor errors (over-redacted payloads, safety-net hit) since the last interval. |
|
| `AuditRedactionFailure` | Audit Log (central) | Count of payload redactor errors (over-redacted payloads, safety-net hit) since the last interval. |
|
||||||
|
| `LocalDbReplicationConnected` | Consolidated site LocalDb | Whether a peer sync session is currently running on this node. **Nullable — null means "no reading", not "disconnected".** Reported only by a node with replication configured. |
|
||||||
|
| `LocalDbOplogBacklog` | Consolidated site LocalDb | Unacked oplog depth (Prometheus `localdb_oplog_depth`). **Nullable — null means "no reading", NOT "connected with an empty backlog"** (a failed poll rendered as 0 would report a pair that cannot read its own oplog as perfectly healthy). **Expected transient spikes:** the daily `site_events` retention purge and the storage-cap trim are CDC-captured like any other write, so the backlog jumps by the batch size at purge time and drains as the peer acks it — see `Component-SiteEventLogging.md` → Storage and `docs/deployment/topology-guide.md` → *Reading the replication backlog* for the healthy-versus-faulty signature. |
|
||||||
|
|
||||||
## Reporting Protocol
|
## Reporting Protocol
|
||||||
|
|
||||||
@@ -113,6 +115,7 @@ These tiles are **point-in-time** like the Notification Outbox and Site Call Aud
|
|||||||
- **Cluster Infrastructure (site)**: Provides node role status.
|
- **Cluster Infrastructure (site)**: Provides node role status.
|
||||||
- **Notification Outbox (central)**: Provides central-computed outbox KPIs — queue depth, stuck count, parked count — for the headline dashboard tiles.
|
- **Notification Outbox (central)**: Provides central-computed outbox KPIs — queue depth, stuck count, parked count — for the headline dashboard tiles.
|
||||||
- **Site Call Audit (central)**: Provides central-computed cached-call KPIs — buffered count, parked count, failed/delivered (last interval), oldest pending age, stuck count — for the headline dashboard tiles.
|
- **Site Call Audit (central)**: Provides central-computed cached-call KPIs — buffered count, parked count, failed/delivered (last interval), oldest pending age, stuck count — for the headline dashboard tiles.
|
||||||
|
- **Site Event Logging (site)**: Provides `SiteEventLogWriteFailures`. Its daily retention purge and storage-cap trim are also the expected cause of transient `LocalDbOplogBacklog` spikes on a replicated node — see [Component-SiteEventLogging.md](Component-SiteEventLogging.md) → Storage.
|
||||||
- **Audit Log (#23)**: Provides the site-reported `SiteAuditBacklog` / `SiteAuditWriteFailures` metrics (via the site health report) and the central-computed `AuditRedactionFailure` metric, plus the central audit-row rate feeding the **Audit** dashboard tile group (Audit volume, Audit error rate, Audit backlog).
|
- **Audit Log (#23)**: Provides the site-reported `SiteAuditBacklog` / `SiteAuditWriteFailures` metrics (via the site health report) and the central-computed `AuditRedactionFailure` metric, plus the central audit-row rate feeding the **Audit** dashboard tile group (Audit volume, Audit error rate, Audit backlog).
|
||||||
|
|
||||||
## Interactions
|
## Interactions
|
||||||
|
|||||||
@@ -92,8 +92,27 @@ Each event entry contains:
|
|||||||
On a site pair running **without** replication configured (by deliberate choice, e.g. the
|
On a site pair running **without** replication configured (by deliberate choice, e.g. the
|
||||||
rig's site-b/site-c), the log stays node-local and a failover does start it fresh — the
|
rig's site-b/site-c), the log stays node-local and a failover does start it fresh — the
|
||||||
documented trade of not configuring a peer, unchanged by this policy.
|
documented trade of not configuring a peer, unchanged by this policy.
|
||||||
- **Retention**: 30 days. A **daily background job** runs on the active node and deletes all events older than 30 days. Hard delete — no archival. (Today, retention/cap deletes on a replicated node are captured by CDC like any other write — see the Volume Policy section above for why this is a small residual cost now that per-run rows are off by default.)
|
- **Retention**: 30 days. A **daily background job** runs on the active node and deletes all events older than 30 days. Hard delete — no archival. Deletes are sliced into bounded 1000-row batches per DELETE statement rather than one unbounded statement, so a large expired backlog does not hold the write lock (and every concurrent recorder flush) for the duration of the purge.
|
||||||
- **Storage cap**: A configurable maximum database size (default: 1 GB) is enforced. If the storage cap is reached before the 30-day retention window, the oldest events are purged first. This prevents disk exhaustion from alarm storms, script failure loops, or connection flapping.
|
- **Storage cap**: A configurable maximum database size (default: 1 GB) is enforced. If the storage cap is reached before the 30-day retention window, the oldest events are purged first. This prevents disk exhaustion from alarm storms, script failure loops, or connection flapping.
|
||||||
|
- **Purge deletes are CDC-captured on a replicated site — the resulting backlog spike is expected.**
|
||||||
|
Retention and cap deletes are ordinary row changes on a replicated table; there is deliberately no
|
||||||
|
purge-exemption path (LocalDb Phase 2 — CDC does all three jobs, and a table-wide exemption is the
|
||||||
|
same mechanism that made `ReplaceAllAsync` unsafe). One oplog row is queued per deleted event, so
|
||||||
|
the site health report's `LocalDbOplogBacklog` (Prometheus `localdb_oplog_depth`) jumps by the size
|
||||||
|
of the batch at purge time and drains as the peer acks it. **Healthy signature:**
|
||||||
|
`LocalDbReplicationConnected` stays true across the spike and the backlog returns to ~0; a backlog
|
||||||
|
that keeps climbing, or climbs while `LocalDbReplicationConnected` is false, is a genuine
|
||||||
|
replication fault and not the purge. The purge logs an Information line naming the row count and
|
||||||
|
the expected transient backlog whenever it deletes rows on a replication-enabled node, so an
|
||||||
|
operator can correlate a spike with the purge that caused it — a spike with no such line nearby is
|
||||||
|
not the purge. Node-local wiring: the Host supplies the replication predicate
|
||||||
|
(`SiteLocalDbSetup.ReplicationIsConfigured` — the `PeerAddress`-OR-`ApiKey` rule) as
|
||||||
|
`SiteEventLogReplicationCheck`; this component never reads LocalDb configuration itself, and with
|
||||||
|
no predicate registered the note is suppressed. Operator detail —
|
||||||
|
including that the purge timer is anchored to the active node's process start, not a wall-clock
|
||||||
|
hour — is in `docs/deployment/topology-guide.md` → *Reading the replication backlog*. See also the
|
||||||
|
Volume Policy section above for why this residual cost is small now that per-run script rows are
|
||||||
|
off by default.
|
||||||
|
|
||||||
## Central Access
|
## Central Access
|
||||||
|
|
||||||
@@ -121,4 +140,4 @@ Each event entry contains:
|
|||||||
- **Communication Layer**: Receives remote queries from central and returns results.
|
- **Communication Layer**: Receives remote queries from central and returns results.
|
||||||
- **Central UI**: Site Event Log Viewer displays queried events.
|
- **Central UI**: Site Event Log Viewer displays queried events.
|
||||||
- **Store-and-Forward Engine**: Its notification path (the site→central forward of script-generated notifications) reports forward failures and long-buffered notifications as Notification-category events. Routine enqueue and forward-success events are deliberately not logged — central's authoritative `Notifications` table (owned by the Notification Outbox component) is the audit record of record; site-side logging covers only the in-transit blind spot when central is unreachable.
|
- **Store-and-Forward Engine**: Its notification path (the site→central forward of script-generated notifications) reports forward failures and long-buffered notifications as Notification-category events. Routine enqueue and forward-success events are deliberately not logged — central's authoritative `Notifications` table (owned by the Notification Outbox component) is the audit record of record; site-side logging covers only the in-transit blind spot when central is unreachable.
|
||||||
- **Health Monitoring**: Script error rates and alarm evaluation error rates can be derived from event log data.
|
- **Health Monitoring**: Script error rates and alarm evaluation error rates can be derived from event log data; this component also reports `SiteEventLogWriteFailures`, and its purge is the expected cause of the transient `LocalDbOplogBacklog` spikes described in [Component-HealthMonitoring.md](Component-HealthMonitoring.md) → Monitored Metrics.
|
||||||
|
|||||||
@@ -195,6 +195,14 @@ public static class SiteLocalDbSetup
|
|||||||
/// replicates.
|
/// replicates.
|
||||||
/// </para>
|
/// </para>
|
||||||
/// <para>
|
/// <para>
|
||||||
|
/// <b><c>internal</c>, not private, so there is exactly one copy of this rule.</b>
|
||||||
|
/// <c>SiteServiceRegistration</c> reuses it to supply SiteEventLogging's
|
||||||
|
/// <c>SiteEventLogReplicationCheck</c> — the predicate that decides whether the daily
|
||||||
|
/// <c>site_events</c> purge adds its "expect a transient oplog backlog" operator note.
|
||||||
|
/// A second hand-rolled PeerAddress-OR-ApiKey test would be free to drift out of step
|
||||||
|
/// with the one that actually installs the triggers.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
/// <b>Both directions of a change to this predicate are now handled at boot</b> (LocalDb
|
/// <b>Both directions of a change to this predicate are now handled at boot</b> (LocalDb
|
||||||
/// 0.2.0). Flipping it to false deregisters, so a file registered by an older build stops
|
/// 0.2.0). Flipping it to false deregisters, so a file registered by an older build stops
|
||||||
/// capturing on the next start instead of paying for triggers forever; flipping it to true
|
/// capturing on the next start instead of paying for triggers forever; flipping it to true
|
||||||
@@ -208,7 +216,7 @@ public static class SiteLocalDbSetup
|
|||||||
/// <c>docs/deployment/topology-guide.md</c>.
|
/// <c>docs/deployment/topology-guide.md</c>.
|
||||||
/// </para>
|
/// </para>
|
||||||
/// </remarks>
|
/// </remarks>
|
||||||
private static bool ReplicationIsConfigured(IConfiguration config)
|
internal static bool ReplicationIsConfigured(IConfiguration config)
|
||||||
{
|
{
|
||||||
var section = config.GetSection("LocalDb:Replication");
|
var section = config.GetSection("LocalDb:Replication");
|
||||||
|
|
||||||
|
|||||||
@@ -210,6 +210,18 @@ public static class SiteServiceRegistration
|
|||||||
return () => nodeProvider.SelfIsPrimary;
|
return () => nodeProvider.SelfIsPrimary;
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// Wording-only companion to the gate above: when a purge on a replication-enabled node
|
||||||
|
// actually deletes rows, it appends an operator note explaining the LocalDbOplogBacklog
|
||||||
|
// spike that follows. site_events is a replicated table and a retention/cap DELETE is
|
||||||
|
// CDC-captured like any other write (LocalDb Phase 2 — no purge-exemption path), so the
|
||||||
|
// backlog genuinely rises at purge time and reads like a replication fault to anyone who
|
||||||
|
// does not know a purge just ran. Delegating to SiteLocalDbSetup.ReplicationIsConfigured
|
||||||
|
// keeps the PeerAddress-OR-ApiKey rule in one place — SiteEventLogging must not learn to
|
||||||
|
// read LocalDb configuration itself.
|
||||||
|
var replicationConfigured = SiteLocalDbSetup.ReplicationIsConfigured(config);
|
||||||
|
SiteEventLogReplicationCheck replicationCheck = () => replicationConfigured;
|
||||||
|
services.AddSingleton(replicationCheck);
|
||||||
|
|
||||||
// Health checks — the shared ZB.MOM.WW.Health probes, mapped by MapZbHealth on the site's
|
// Health checks — the shared ZB.MOM.WW.Health probes, mapped by MapZbHealth on the site's
|
||||||
// HTTP/1.1 listener (Program.cs). Site nodes served no health endpoints before this; the
|
// HTTP/1.1 listener (Program.cs). Site nodes served no health endpoints before this; the
|
||||||
// family overview dashboard probes every node the same way, so a site node has to answer
|
// family overview dashboard probes every node the same way, so a site node has to answer
|
||||||
|
|||||||
@@ -20,6 +20,32 @@ namespace ZB.MOM.WW.ScadaBridge.SiteEventLogging;
|
|||||||
/// <returns><c>true</c> if this node is the active site member and should run the purge; <c>false</c> to skip.</returns>
|
/// <returns><c>true</c> if this node is the active site member and should run the purge; <c>false</c> to skip.</returns>
|
||||||
public delegate bool SiteEventLogActiveNodeCheck();
|
public delegate bool SiteEventLogActiveNodeCheck();
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Predicate the <see cref="EventLogPurgeService"/> consults when a purge actually deleted
|
||||||
|
/// rows, to decide whether to add the "expect a transient replication backlog" operator note
|
||||||
|
/// to its purge log line.
|
||||||
|
///
|
||||||
|
/// <para>
|
||||||
|
/// <c>site_events</c> is one of LocalDb's replicated tables, and CDC captures a retention
|
||||||
|
/// DELETE exactly like any other row change — by design, since Phase 2 there is deliberately
|
||||||
|
/// no purge-exemption path. So on a replication-enabled node the purge briefly inflates the
|
||||||
|
/// oplog (health field <c>LocalDbOplogBacklog</c>, Prometheus <c>localdb_oplog_depth</c>),
|
||||||
|
/// which reads like a replication fault to an operator who does not know a purge just ran.
|
||||||
|
/// The log line is the breadcrumb that ties the two together; it is a message-wording choice
|
||||||
|
/// only and changes no purge behaviour.
|
||||||
|
/// </para>
|
||||||
|
///
|
||||||
|
/// <para>
|
||||||
|
/// Registration is the Host's responsibility — SiteEventLogging has no view of
|
||||||
|
/// <c>LocalDb:Replication</c> and must not grow one; the Host already owns that predicate
|
||||||
|
/// (<c>SiteLocalDbSetup.ReplicationIsConfigured</c>, the <c>PeerAddress</c>-OR-<c>ApiKey</c>
|
||||||
|
/// rule) and passes it in. When no implementation is registered the note is suppressed, which
|
||||||
|
/// matches the product default: replication is opt-in and off unless configured.
|
||||||
|
/// </para>
|
||||||
|
/// </summary>
|
||||||
|
/// <returns><c>true</c> if this node has LocalDb replication configured; <c>false</c> otherwise.</returns>
|
||||||
|
public delegate bool SiteEventLogReplicationCheck();
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Background service that periodically purges old events from the SQLite event log.
|
/// Background service that periodically purges old events from the SQLite event log.
|
||||||
/// Enforces both time-based retention (default 30 days) and storage cap (default 1GB).
|
/// Enforces both time-based retention (default 30 days) and storage cap (default 1GB).
|
||||||
@@ -37,6 +63,7 @@ public class EventLogPurgeService : BackgroundService
|
|||||||
private readonly SiteEventLogOptions _options;
|
private readonly SiteEventLogOptions _options;
|
||||||
private readonly ILogger<EventLogPurgeService> _logger;
|
private readonly ILogger<EventLogPurgeService> _logger;
|
||||||
private readonly SiteEventLogActiveNodeCheck _isActiveNode;
|
private readonly SiteEventLogActiveNodeCheck _isActiveNode;
|
||||||
|
private readonly SiteEventLogReplicationCheck _isReplicationConfigured;
|
||||||
|
|
||||||
/// <summary>Initializes a new instance of <see cref="EventLogPurgeService"/>.</summary>
|
/// <summary>Initializes a new instance of <see cref="EventLogPurgeService"/>.</summary>
|
||||||
/// <param name="eventLogger">The concrete event logger providing lock-guarded database access.</param>
|
/// <param name="eventLogger">The concrete event logger providing lock-guarded database access.</param>
|
||||||
@@ -49,11 +76,18 @@ public class EventLogPurgeService : BackgroundService
|
|||||||
/// the Host on a site node — each tick early-exits on the standby so the
|
/// the Host on a site node — each tick early-exits on the standby so the
|
||||||
/// daily purge runs only on the active node, matching the design.
|
/// daily purge runs only on the active node, matching the design.
|
||||||
/// </param>
|
/// </param>
|
||||||
|
/// <param name="isReplicationConfigured">
|
||||||
|
/// Optional LocalDb-replication check. When <c>null</c> — non-clustered hosts,
|
||||||
|
/// unit tests — the purge log omits the replication-backlog operator note, matching
|
||||||
|
/// the product default that replication is opt-in and off unless configured.
|
||||||
|
/// See <see cref="SiteEventLogReplicationCheck"/>.
|
||||||
|
/// </param>
|
||||||
public EventLogPurgeService(
|
public EventLogPurgeService(
|
||||||
SiteEventLogger eventLogger,
|
SiteEventLogger eventLogger,
|
||||||
IOptions<SiteEventLogOptions> options,
|
IOptions<SiteEventLogOptions> options,
|
||||||
ILogger<EventLogPurgeService> logger,
|
ILogger<EventLogPurgeService> logger,
|
||||||
SiteEventLogActiveNodeCheck? isActiveNode = null)
|
SiteEventLogActiveNodeCheck? isActiveNode = null,
|
||||||
|
SiteEventLogReplicationCheck? isReplicationConfigured = null)
|
||||||
{
|
{
|
||||||
// Depend on the concrete recorder directly: purge must funnel database access
|
// Depend on the concrete recorder directly: purge must funnel database access
|
||||||
// through its lock-guarded WithConnection. Taking ISiteEventLogger and
|
// through its lock-guarded WithConnection. Taking ISiteEventLogger and
|
||||||
@@ -62,6 +96,7 @@ public class EventLogPurgeService : BackgroundService
|
|||||||
_options = options.Value;
|
_options = options.Value;
|
||||||
_logger = logger;
|
_logger = logger;
|
||||||
_isActiveNode = isActiveNode ?? (static () => true);
|
_isActiveNode = isActiveNode ?? (static () => true);
|
||||||
|
_isReplicationConfigured = isReplicationConfigured ?? (static () => false);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
@@ -161,9 +196,49 @@ public class EventLogPurgeService : BackgroundService
|
|||||||
if (totalDeleted > 0)
|
if (totalDeleted > 0)
|
||||||
{
|
{
|
||||||
_logger.LogInformation("Purged {Count} events older than {Days} days", totalDeleted, _options.RetentionDays);
|
_logger.LogInformation("Purged {Count} events older than {Days} days", totalDeleted, _options.RetentionDays);
|
||||||
|
LogReplicationBacklogNote(totalDeleted);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Emits the operator breadcrumb that explains the oplog backlog spike a purge produces on a
|
||||||
|
/// replication-enabled node. No-op when this node has no replication peer.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// <c>site_events</c> is a replicated table, and a retention/cap DELETE is captured by CDC
|
||||||
|
/// exactly like an ordinary write — there is deliberately no purge-exemption path (LocalDb
|
||||||
|
/// Phase 2: CDC does all three jobs). The daily purge therefore pushes one oplog row per
|
||||||
|
/// deleted event, and the site health report's <c>LocalDbOplogBacklog</c> (Prometheus
|
||||||
|
/// <c>localdb_oplog_depth</c>) spikes until the peer acks them. That is expected and drains
|
||||||
|
/// on its own; without this line an operator correlating the spike has nothing in the log to
|
||||||
|
/// tie it to. Information level because it is only interesting next to the purge line it
|
||||||
|
/// follows — the spike itself is not a fault.
|
||||||
|
/// </remarks>
|
||||||
|
/// <param name="deletedRows">Number of rows the purge just deleted; always > 0 at the call sites.</param>
|
||||||
|
private void LogReplicationBacklogNote(int deletedRows)
|
||||||
|
{
|
||||||
|
bool replicated;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
replicated = _isReplicationConfigured();
|
||||||
|
}
|
||||||
|
catch (Exception checkEx)
|
||||||
|
{
|
||||||
|
// A log-wording predicate must never break the purge loop.
|
||||||
|
_logger.LogDebug(checkEx, "Replication check threw while composing the purge log note; note suppressed");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!replicated)
|
||||||
|
return;
|
||||||
|
|
||||||
|
_logger.LogInformation(
|
||||||
|
"Purged {Count} site_events rows on a replication-enabled node — a transient LocalDb oplog backlog " +
|
||||||
|
"is expected while the deletes replicate to the peer. It drains on its own; " +
|
||||||
|
"LocalDbReplicationConnected staying true with the backlog returning to ~0 is the healthy signature.",
|
||||||
|
deletedRows);
|
||||||
|
}
|
||||||
|
|
||||||
private void PurgeByStorageCap()
|
private void PurgeByStorageCap()
|
||||||
{
|
{
|
||||||
var capBytes = (long)_options.MaxStorageMb * 1024 * 1024;
|
var capBytes = (long)_options.MaxStorageMb * 1024 * 1024;
|
||||||
@@ -180,6 +255,8 @@ public class EventLogPurgeService : BackgroundService
|
|||||||
// The loop also stops if the on-disk size fails to decrease across an
|
// The loop also stops if the on-disk size fails to decrease across an
|
||||||
// iteration (e.g. if vacuum cannot reclaim space), so a cap that can never
|
// iteration (e.g. if vacuum cannot reclaim space), so a cap that can never
|
||||||
// be met does not silently empty the entire table.
|
// be met does not silently empty the entire table.
|
||||||
|
var totalDeleted = 0;
|
||||||
|
|
||||||
while (currentSizeBytes > capBytes)
|
while (currentSizeBytes > capBytes)
|
||||||
{
|
{
|
||||||
var previousSizeBytes = currentSizeBytes;
|
var previousSizeBytes = currentSizeBytes;
|
||||||
@@ -214,6 +291,7 @@ public class EventLogPurgeService : BackgroundService
|
|||||||
if (deleted == 0)
|
if (deleted == 0)
|
||||||
break;
|
break;
|
||||||
|
|
||||||
|
totalDeleted += deleted;
|
||||||
currentSizeBytes = GetDatabaseSizeBytes();
|
currentSizeBytes = GetDatabaseSizeBytes();
|
||||||
|
|
||||||
if (currentSizeBytes >= previousSizeBytes)
|
if (currentSizeBytes >= previousSizeBytes)
|
||||||
@@ -228,6 +306,13 @@ public class EventLogPurgeService : BackgroundService
|
|||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Same CDC cost as the retention purge above — a cap-driven trim is usually the
|
||||||
|
// larger of the two, so it gets the same operator breadcrumb.
|
||||||
|
if (totalDeleted > 0)
|
||||||
|
{
|
||||||
|
LogReplicationBacklogNote(totalDeleted);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
|
|||||||
@@ -30,11 +30,18 @@ public static class ServiceCollectionExtensions
|
|||||||
// it unregistered, and the purge defaults to "always run" (the
|
// it unregistered, and the purge defaults to "always run" (the
|
||||||
// pre-fix behaviour). Building the service via a factory so the
|
// pre-fix behaviour). Building the service via a factory so the
|
||||||
// optional delegate flows from DI rather than the constructor default.
|
// optional delegate flows from DI rather than the constructor default.
|
||||||
|
//
|
||||||
|
// SiteEventLogReplicationCheck flows the same way and is likewise Host-supplied:
|
||||||
|
// it only selects the wording of the purge log line (whether to add the
|
||||||
|
// "expect a transient LocalDb oplog backlog" note), and this component has no
|
||||||
|
// view of LocalDb:Replication. Unregistered ⇒ no note, matching the default
|
||||||
|
// that replication is opt-in and off.
|
||||||
services.AddHostedService(sp => new EventLogPurgeService(
|
services.AddHostedService(sp => new EventLogPurgeService(
|
||||||
sp.GetRequiredService<SiteEventLogger>(),
|
sp.GetRequiredService<SiteEventLogger>(),
|
||||||
sp.GetRequiredService<IOptions<SiteEventLogOptions>>(),
|
sp.GetRequiredService<IOptions<SiteEventLogOptions>>(),
|
||||||
sp.GetRequiredService<ILogger<EventLogPurgeService>>(),
|
sp.GetRequiredService<ILogger<EventLogPurgeService>>(),
|
||||||
sp.GetService<SiteEventLogActiveNodeCheck>()));
|
sp.GetService<SiteEventLogActiveNodeCheck>(),
|
||||||
|
sp.GetService<SiteEventLogReplicationCheck>()));
|
||||||
return services;
|
return services;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1,3 +1,4 @@
|
|||||||
|
using Microsoft.Extensions.Logging;
|
||||||
using Microsoft.Extensions.Logging.Abstractions;
|
using Microsoft.Extensions.Logging.Abstractions;
|
||||||
using Microsoft.Extensions.Options;
|
using Microsoft.Extensions.Options;
|
||||||
|
|
||||||
@@ -45,14 +46,17 @@ public class EventLogPurgeServiceTests : IDisposable
|
|||||||
|
|
||||||
private EventLogPurgeService CreatePurgeService(
|
private EventLogPurgeService CreatePurgeService(
|
||||||
SiteEventLogOptions? optionsOverride = null,
|
SiteEventLogOptions? optionsOverride = null,
|
||||||
SiteEventLogActiveNodeCheck? isActiveNode = null)
|
SiteEventLogActiveNodeCheck? isActiveNode = null,
|
||||||
|
SiteEventLogReplicationCheck? isReplicationConfigured = null,
|
||||||
|
ILogger<EventLogPurgeService>? logger = null)
|
||||||
{
|
{
|
||||||
var opts = optionsOverride ?? _options;
|
var opts = optionsOverride ?? _options;
|
||||||
return new EventLogPurgeService(
|
return new EventLogPurgeService(
|
||||||
_eventLogger,
|
_eventLogger,
|
||||||
Options.Create(opts),
|
Options.Create(opts),
|
||||||
NullLogger<EventLogPurgeService>.Instance,
|
logger ?? NullLogger<EventLogPurgeService>.Instance,
|
||||||
isActiveNode);
|
isActiveNode,
|
||||||
|
isReplicationConfigured);
|
||||||
}
|
}
|
||||||
|
|
||||||
private void InsertEventWithTimestamp(DateTimeOffset timestamp)
|
private void InsertEventWithTimestamp(DateTimeOffset timestamp)
|
||||||
@@ -455,4 +459,135 @@ public class EventLogPurgeServiceTests : IDisposable
|
|||||||
|
|
||||||
Assert.Equal(0, GetEventCount());
|
Assert.Equal(0, GetEventCount());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── R7: replication-backlog operator note on the purge log line ──
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Fragment unique to the R7 note, matched against the rendered log message.
|
||||||
|
/// Deliberately not the whole sentence — the wording is operator prose and may be
|
||||||
|
/// reworded; what must not regress is that the note fires (or does not) per the
|
||||||
|
/// replication predicate.
|
||||||
|
/// </summary>
|
||||||
|
private const string BacklogNoteFragment = "transient LocalDb oplog backlog";
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void PurgeByRetention_OnReplicatedNode_LogsTheOplogBacklogNote()
|
||||||
|
{
|
||||||
|
// R7: site_events is a replicated table and the retention DELETE is CDC-captured
|
||||||
|
// like any other write — no purge-exemption path by design — so the purge inflates
|
||||||
|
// LocalDbOplogBacklog / localdb_oplog_depth until the peer acks. The Information
|
||||||
|
// line is the breadcrumb that lets an operator tie the spike to the purge instead
|
||||||
|
// of reading it as a replication fault.
|
||||||
|
InsertEventWithTimestamp(DateTimeOffset.UtcNow.AddDays(-31));
|
||||||
|
|
||||||
|
var logger = new CapturingLogger();
|
||||||
|
var purge = CreatePurgeService(isReplicationConfigured: () => true, logger: logger);
|
||||||
|
purge.RunPurge();
|
||||||
|
|
||||||
|
Assert.Equal(0, GetEventCount());
|
||||||
|
var note = Assert.Single(logger.Entries, e => e.Message.Contains(BacklogNoteFragment, StringComparison.Ordinal));
|
||||||
|
Assert.Equal(LogLevel.Information, note.Level);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void PurgeByRetention_OnUnreplicatedNode_DoesNotLogTheNote()
|
||||||
|
{
|
||||||
|
// The other half of the gate: site-b/site-c on the rig run with no replication
|
||||||
|
// configured, have no CDC triggers at all, and therefore no backlog to explain.
|
||||||
|
// Emitting the note there would be noise pointing at a metric that reads null.
|
||||||
|
InsertEventWithTimestamp(DateTimeOffset.UtcNow.AddDays(-31));
|
||||||
|
|
||||||
|
var logger = new CapturingLogger();
|
||||||
|
var purge = CreatePurgeService(isReplicationConfigured: () => false, logger: logger);
|
||||||
|
purge.RunPurge();
|
||||||
|
|
||||||
|
Assert.Equal(0, GetEventCount());
|
||||||
|
Assert.DoesNotContain(logger.Entries, e => e.Message.Contains(BacklogNoteFragment, StringComparison.Ordinal));
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void PurgeByRetention_WithNoRowsDeleted_DoesNotLogTheNote()
|
||||||
|
{
|
||||||
|
// The note is a companion to a purge line, not a tick heartbeat: a daily tick that
|
||||||
|
// deleted nothing produces no oplog rows and so must stay silent, otherwise the
|
||||||
|
// breadcrumb loses all correlating value.
|
||||||
|
InsertEventWithTimestamp(DateTimeOffset.UtcNow);
|
||||||
|
|
||||||
|
var logger = new CapturingLogger();
|
||||||
|
var purge = CreatePurgeService(isReplicationConfigured: () => true, logger: logger);
|
||||||
|
purge.RunPurge();
|
||||||
|
|
||||||
|
Assert.Equal(1, GetEventCount());
|
||||||
|
Assert.DoesNotContain(logger.Entries, e => e.Message.Contains(BacklogNoteFragment, StringComparison.Ordinal));
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void PurgeByStorageCap_OnReplicatedNode_LogsTheOplogBacklogNote()
|
||||||
|
{
|
||||||
|
// A cap-driven trim deletes rows through the same replicated table and is usually
|
||||||
|
// the larger of the two bursts, so it carries the same note.
|
||||||
|
for (int i = 0; i < 100; i++)
|
||||||
|
{
|
||||||
|
InsertEventWithTimestamp(DateTimeOffset.UtcNow);
|
||||||
|
}
|
||||||
|
|
||||||
|
var capOptions = new SiteEventLogOptions
|
||||||
|
{
|
||||||
|
DatabasePath = _dbPath,
|
||||||
|
RetentionDays = 30,
|
||||||
|
MaxStorageMb = 0 // 0 MB cap forces the cap purge; retention deletes nothing here
|
||||||
|
};
|
||||||
|
|
||||||
|
var logger = new CapturingLogger();
|
||||||
|
var purge = CreatePurgeService(capOptions, isReplicationConfigured: () => true, logger: logger);
|
||||||
|
purge.RunPurge();
|
||||||
|
|
||||||
|
Assert.Equal(0, GetEventCount());
|
||||||
|
var note = Assert.Single(logger.Entries, e => e.Message.Contains(BacklogNoteFragment, StringComparison.Ordinal));
|
||||||
|
Assert.Equal(LogLevel.Information, note.Level);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void RunPurge_WhenReplicationCheckThrows_StillPurgesAndSwallows()
|
||||||
|
{
|
||||||
|
// Defensive: the replication predicate only selects log wording. A throw from it
|
||||||
|
// must never escape the purge — the rows are already deleted by the time it is
|
||||||
|
// consulted, and an exception here would surface as "Error during event log purge"
|
||||||
|
// for a purge that in fact succeeded.
|
||||||
|
InsertEventWithTimestamp(DateTimeOffset.UtcNow.AddDays(-31));
|
||||||
|
|
||||||
|
var logger = new CapturingLogger();
|
||||||
|
var purge = CreatePurgeService(
|
||||||
|
isReplicationConfigured: () => throw new InvalidOperationException("boom"),
|
||||||
|
logger: logger);
|
||||||
|
purge.RunPurge();
|
||||||
|
|
||||||
|
Assert.Equal(0, GetEventCount());
|
||||||
|
Assert.DoesNotContain(logger.Entries, e => e.Level == LogLevel.Error);
|
||||||
|
Assert.DoesNotContain(logger.Entries, e => e.Message.Contains(BacklogNoteFragment, StringComparison.Ordinal));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Minimal <see cref="ILogger{TCategoryName}"/> that records level + rendered message.
|
||||||
|
/// The purge service has no other observable output for a log-only change, and the
|
||||||
|
/// suite has no shared capturing logger to reuse.
|
||||||
|
/// </summary>
|
||||||
|
private sealed class CapturingLogger : ILogger<EventLogPurgeService>
|
||||||
|
{
|
||||||
|
public List<(LogLevel Level, string Message)> Entries { get; } = [];
|
||||||
|
|
||||||
|
public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
|
||||||
|
|
||||||
|
public bool IsEnabled(LogLevel logLevel) => true;
|
||||||
|
|
||||||
|
public void Log<TState>(
|
||||||
|
LogLevel logLevel,
|
||||||
|
EventId eventId,
|
||||||
|
TState state,
|
||||||
|
Exception? exception,
|
||||||
|
Func<TState, Exception?, string> formatter)
|
||||||
|
{
|
||||||
|
Entries.Add((logLevel, formatter(state, exception)));
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user