ScadaBridge/docs/requirements/Component-SiteEventLogging.md

# Component: Site Event Logging

## Purpose

The Site Event Logging component records operational events at each site cluster, providing a local audit trail of runtime activity. Events are queryable from the central UI for remote troubleshooting.

## Location

Site clusters (event recording and storage). Central cluster (remote query access via UI).

## Responsibilities

- Record operational events from all site subsystems.
- Persist events to local SQLite.
- Enforce 30-day retention policy with automatic purging.
- Respond to remote queries from central for event log data.

## Events Logged

| Category | Events |
|----------|--------|
| Script Executions | Script started, completed *(sampled/opt-in — see Volume Policy below)*; failed, timed out, recursion limit exceeded (with error details; always logged) |
| Script Run Summary | Interval aggregate replacing the per-run rows by default — one Info row per flush interval summarizing every instance script run since the last flush (see Volume Policy) |
| Alarm Events | Alarm activated, alarm cleared (which alarm, which instance), alarm evaluation error |
| Deployment Events | Configuration received from central, scripts compiled, applied successfully, apply failed |
| Data Connection Status | Connected, disconnected, reconnected (per connection) |
| Store-and-Forward | Message queued, delivered, retried, parked |
| Instance Lifecycle | Instance enabled, disabled, deleted |
| Notification | Site→central forward failure, long-buffered notification (still in the site buffer past a threshold) |

## Volume Policy — Script Run Events (WP3.2, 2026-08-15)

Per-run instance-script Started/Completed Info events were, before this policy, the dominant
`site_events` writer: two rows per run, translating to roughly a dozen physical row-writes
across a replicated site pair once CDC oplog/row-version rows and their eventual purge
tombstones are counted — for content nobody looks at unless something went wrong. The policy
below (design memo: `docs/plans/2026-08-15-site-events-policy-design.md`) keeps the signal
while removing the volume.

- **Off by default.** `ScadaBridge:SiteRuntime:PerRunScriptEvents` (default `false`) is the
  global switch back to the legacy per-run Started/Completed rows for every script.
- **Interval aggregate instead.** `ScriptRunSummaryRecorder` accumulates per-`(instance,
  script)` counters (started, completed, failed, timed out, total/max duration) from the same
  call sites that used to emit the per-run rows. A hosted flush service emits **one** "script"
  Info row per interval — `ScadaBridge:SiteRuntime:ScriptRunSummaryIntervalSeconds` (default
  `300`, i.e. 5 minutes; `0` disables the flush) — with headline totals in the message (e.g.
  "1,243 runs: 1,240 completed, 2 failed, 1 timed out across 17 scripts") and a per-script
  breakdown in `Details`, capped at the **top 50 scripts by run activity** with an "others"
  rollup entry so a pathological deployment cannot mint an oversized row. **An interval with
  zero activity emits no row** — the standby, which never runs scripts, therefore never
  produces a summary row naturally. Source is `"ScriptRunSummary"`; Instance ID is `NULL`
  (the row spans every instance).
- **Per-script opt-in for debugging.** `ScadaBridge:SiteRuntime:PerRunScriptEventScripts`
  (default empty) restores the legacy per-run rows for named scripts only, without paying the
  volume cost for the rest of the deployment. Entries are `"{InstanceName}/{ScriptName}"` for
  an exact match or `"{InstanceName}/*"` for every script on that instance. Effective only
  while the global switch is off.
- **Hot-togglable.** Both keys are read from the live options snapshot at the top of every
  run (`IOptionsMonitor<SiteRuntimeOptions>.CurrentValue`), so an operator can flip either one
  — including narrowing an opt-in to a single misbehaving script — without a restart or a
  redeploy.
- **Error-level events are unconditional.** Timeout, failure, stuck-watchdog, and
  recursion-limit Error rows are unaffected by any of the above — they always fire, regardless
  of `PerRunScriptEvents` or the opt-in list.

## Event Entry Schema

Each event entry contains:
- **Timestamp**: When the event occurred.
- **Event Type**: Category of the event (script, alarm, deployment, connection, store-and-forward, instance-lifecycle, notification).
- **Severity**: Info, Warning, or Error.
- **Instance ID** *(optional)*: The instance associated with the event (if applicable).
- **Source**: The subsystem that generated the event (e.g., "ScriptActor:MonitorSpeed", "AlarmActor:OverTemp", "DataConnection:PLC1").
- **Message**: Human-readable description of the event.
- **Details** *(optional)*: Additional structured data (e.g., exception stack trace, alarm name, message ID, compilation errors).

## Storage

- Events are stored in the **consolidated site LocalDb** SQLite file (`site_events` table;
  see CLAUDE.md → Consolidated site database). The active node generates most events (a
  standby runs no scripts, subsystems, or deployments to log), but DataConnectionActor
  runs on **both** site nodes and writes its own `site_events` rows regardless of
  active/standby status (register rows 89/90/97) — see *Retention* below for the
  purge consequence.
- **`site_events` IS replicated** (`SiteLocalDbSetup.ReplicatedTables`) on a site pair with
  replication configured — this reverses the component's pre-LocalDb-Phase-1 behavior, and the
  reason is **failover history continuity, not either-node queries**: the query handler
  (`EventLogHandlerActor`) is a **cluster singleton**, so a central query always routes to the
  actively-written copy regardless of which node answers the gRPC call — replication is never
  needed to make queries *work*. What replication buys is **content to read after a failover**:
  when the active node dies, the singleton restarts on the survivor, and its query service reads
  the survivor's *local* copy — which has history *only because it was replicated*. Without
  replication, a failover would start the event log empty at precisely the moment an operator
  is investigating an incident (WP3.2 design memo, Decision (b): `docs/plans/2026-08-15-site-events-policy-design.md`).
  On a site pair running **without** replication configured (by deliberate choice, e.g. the
  rig's site-b/site-c), the log stays node-local and a failover does start it fresh — the
  documented trade of not configuring a peer, unchanged by this policy.
- **Retention**: 30 days. A **daily background job** runs on **every** site node (arch-review 2026-08-19 P4-4 — previously active-node-only, but the standby writes its own `site_events` rows too, so an unreplicated standby's file grew unbounded under the old active-only gate) and deletes that node's own events older than 30 days. Hard delete — no archival. Deletes are sliced into bounded 1000-row batches per DELETE statement rather than one unbounded statement, so a large expired backlog does not hold the write lock (and every concurrent recorder flush) for the duration of the purge.
- **Storage cap**: A configurable maximum size (default: 1 GB) is enforced. If the cap is reached
  before the 30-day retention window, the oldest events are purged first. This prevents disk
  exhaustion from alarm storms, script failure loops, or connection flapping.
- **The cap measures `site_events` and its four indexes — NOT the database file** (register row 81;
  decision record `docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md`). This matters
  because the event log shares one SQLite file with nine other tables plus LocalDb's
  `__localdb_oplog` / `__localdb_row_version` / `__localdb_snapshot_state` bookkeeping. The cap used
  to compare `MaxStorageMb` against `page_count − freelist_count` over that whole file, which on a
  replication-enabled node made the trim **unwinnable**: each 1000-row delete freed ~390 KB of event
  pages while adding ~1000 oplog rows and ~1000 tombstones to the same file, the measured size did
  not fall, and the progress guard stopped the trim — measured 2.5x over a 64 MB cap with 226 k rows
  still present. A node whose peer was down could not enforce its cap at all. Two consequences of
  the new scoping are worth internalising:
  - **The cap is now independent of replication health.** It was previously a function of the
    *peer's* liveness, and in the wrong direction — an inflated oplog is a replication fault, and
    letting it drive event deletion would destroy operational history in response to one.
  - **`MaxStorageMb` no longer bounds the data volume on its own.** LocalDb bookkeeping is separate
    overhead on the same disk, bounded separately by `LocalDb:Replication:MaxOplogRows` and
    `TombstoneRetention`; sizing a site's volume means adding the two. Managing that bookkeeping is
    deliberately **not** this component's job — see the decision record §6.
- **The measurement is modelled, not exact.** SQLite can report per-table page usage precisely via
  the `dbstat` virtual table, and that was the intended fix — but `dbstat` is a compile-time option
  and the bundled `SQLitePCLRaw.lib.e_sqlite3` native library (SQLite 3.53.3) does **not** enable it;
  neither `dbstat` nor `sqlite_dbpage` exists, verified by probe and pinned by
  `SqliteCapabilityTests`. The table's footprint is therefore estimated from sampled row width
  (500 rows from each end of the table, measured with `octet_length` so multi-byte UTF-8 is not
  under-counted) fed into a page-layout model; index pages count against the cap, deliberately,
  because they are ~34 % of the table's real footprint and the 1 GB default was sized against a
  bytes/row figure that included them. Measured error is **−0.8 % to +12.8 %** across representative
  workloads, biased high on purpose — a cap that fires slightly early forfeits budget, whereas one
  that reads low fails at the only job it has. The estimate is additionally clamped by the
  whole-file logical size, which is a true upper bound on any one table.
- **Two wipe-prevention backstops** guard the cap trim, replacing the single "on-disk size did not
  decrease" check. (1) If row width cannot be sampled, the trim **refuses to run** and warns —
  checked before the cap comparison, because a zero row width makes "are we over cap" unanswerable
  rather than false, and the old loop's answer to an unanswerable cap was to keep deleting. (2) If
  the *estimated table size* fails to fall across a batch, the trim stops and warns. Guard (2) is the
  old check re-based onto the table-scoped metric, and its meaning is now unambiguous: replication
  bookkeeping can no longer trip it, so it fires only when new events are arriving at least as fast
  as the trim removes them. For that to be observable, the loop re-reads the row count from the
  database after every batch, while bytes-per-row is sampled **once per tick and held fixed** —
  re-sampling per batch would make the estimate jitter as the trim eats the head of the table and
  could stall the loop through guard (2), which is the same self-defeating-measurement failure as
  row 81 itself.
- **Per-tick delete ceiling** (`MaxPurgedRowsPerTick`, default **250 000**; `0` = unlimited): one
  purge tick deletes at most this many rows, **shared** by the retention purge and the storage-cap
  trim — two independently-capped phases would let a tick emit twice the intended burst. A tick that
  stops at the ceiling schedules the next tick after `PurgeCatchUpInterval` (default **5 minutes**)
  instead of `PurgeInterval`, so retention still converges: a worst-case backlog drains over
  ~11 accelerated ticks (≈55 minutes) rather than 11 days at the daily cadence, with every
  individual burst bounded. The ceiling exists because the purge is CDC-captured (next bullet) and
  an unbounded tick could exceed LocalDb's oplog budget; measured evidence and the chosen defaults
  are in `docs/plans/2026-08-15-site-events-purge-burst-measurement.md`. A ceiling-limited tick logs
  a Warning (rate-limited to one line per 15 minutes) and increments
  `scadabridge.site.event_log.purge_ceiling_hits` — the counter is the unthrottled series that
  distinguishes a backlog converging from a site whose event rate outruns one paced tick.
  Consequence of the accelerated retry: the purge loop is a delay loop, not a `PeriodicTimer`, so
  the cadence is anchored to the **end** of the previous purge rather than to process start.
- **Purge deletes are CDC-captured on a replicated site — the resulting backlog spike is expected.**
  Retention and cap deletes are ordinary row changes on a replicated table; there is deliberately no
  purge-exemption path (LocalDb Phase 2 — CDC does all three jobs, and a table-wide exemption is the
  same mechanism that made `ReplaceAllAsync` unsafe). One oplog row is queued per deleted event, so
  the site health report's `LocalDbOplogBacklog` (Prometheus `localdb_oplog_depth`) jumps by the size
  of the batch at purge time and drains as the peer acks it. **The height of that jump is bounded by
  `MaxPurgedRowsPerTick`, not by how much history expired** — measured (register row 34): a full 1 GB
  event log holds ~2.73 million rows at 393 bytes/row, and purging it unbounded queued 2 731 947
  oplog rows, **2.7x LocalDb's `MaxOplogRows` default of 1 000 000**, which would make the
  maintenance tick prune the oplog and flag a full ten-table snapshot resync. How much of that is
  *resident* depends on the peer — the oplog drains concurrently, and a loopback peer held the same
  burst to a 391 000 peak — so the overrun is conditional on peer throughput and unconditional when
  the peer is down. Bounding the tick is what makes the outcome independent of the peer. Pacing the
  tick, not exempting the purge from capture, is the fix — the exemption remains ruled out.
  **Healthy signature:**
  `LocalDbReplicationConnected` stays true across the spike and the backlog returns to ~0; a backlog
  that keeps climbing, or climbs while `LocalDbReplicationConnected` is false, is a genuine
  replication fault and not the purge. The purge logs an Information line naming the row count and
  the expected transient backlog whenever it deletes rows on a replication-enabled node, so an
  operator can correlate a spike with the purge that caused it — a spike with no such line nearby is
  not the purge. That line is emitted **once per tick**, carrying the retention purge's and the cap
  trim's deletes combined (register row 84 — it used to fire once per phase, so a tick in which both
  deleted produced two identical notes; the tick total is also the more useful number to put beside
  an oplog spike). Node-local wiring: the Host supplies the replication predicate
  (`SiteLocalDbSetup.ReplicationIsConfigured` — the `PeerAddress`-OR-`ApiKey` rule) as
  `SiteEventLogReplicationCheck`; this component never reads LocalDb configuration itself, and with
  no predicate registered the note is suppressed. Operator detail —
  including that the purge timer is anchored to the active node's process start, not a wall-clock
  hour — is in `docs/deployment/topology-guide.md` → *Reading the replication backlog*. See also the
  Volume Policy section above for why this residual cost is small now that per-run script rows are
  off by default.

## Central Access

- The central UI can query site event logs remotely via the Communication Layer.
- Queries support filtering by:
  - Event type / category
  - Time range
  - Instance ID
  - Severity
  - **Keyword search**: Free-text search on message and source fields (SQLite LIKE query). Useful for finding events by script name, alarm name, or error message across all instances.
- Results are **paginated** with a configurable page size (default: 500 events). Each response includes a continuation token for fetching additional pages. This prevents broad queries from overwhelming the communication channel.
- The site processes the query locally against SQLite and returns matching results to central.

## Dependencies

- **SQLite**: Local storage on each site node.
- **Communication Layer**: Handles remote query requests from central.
- **Site Runtime**: Generates script execution events, alarm events, deployment application events, and instance lifecycle events.
- **Data Connection Layer**: Generates connection status events.
- **Store-and-Forward Engine**: Generates buffer activity events, including notification-category forward failures and long-buffered notifications on the site→central notification path.

## Interactions

- **All site subsystems**: Event logging is a cross-cutting concern — any subsystem that produces notable events calls the Event Logging service.
- **Communication Layer**: Receives remote queries from central and returns results.
- **Central UI**: Site Event Log Viewer displays queried events.
- **Store-and-Forward Engine**: Its notification path (the site→central forward of script-generated notifications) reports forward failures and long-buffered notifications as Notification-category events. Routine enqueue and forward-success events are deliberately not logged — central's authoritative `Notifications` table (owned by the Notification Outbox component) is the audit record of record; site-side logging covers only the in-transit blind spot when central is unreachable.
- **Health Monitoring**: Script error rates and alarm evaluation error rates can be derived from event log data; this component also reports `SiteEventLogWriteFailures`, and its purge is the expected cause of the transient `LocalDbOplogBacklog` spikes described in [Component-HealthMonitoring.md](Component-HealthMonitoring.md) → Monitored Metrics.

Rig settings no longer advertise PurgeScheduleCron. Retention uses PurgeInterval after the previous purge completes, so it is interval-based rather than a fixed wall-clock schedule.