ScadaBridge/docs/requirements/Component-SiteEventLogging.md

# Component: Site Event Logging

## Purpose

The Site Event Logging component records operational events at each site cluster, providing a local audit trail of runtime activity. Events are queryable from the central UI for remote troubleshooting.

## Location

Site clusters (event recording and storage). Central cluster (remote query access via UI).

## Responsibilities

- Record operational events from all site subsystems.
- Persist events to local SQLite.
- Enforce 30-day retention policy with automatic purging.
- Respond to remote queries from central for event log data.

## Events Logged

| Category | Events |
|----------|--------|
| Script Executions | Script started, completed *(sampled/opt-in — see Volume Policy below)*; failed, timed out, recursion limit exceeded (with error details; always logged) |
| Script Run Summary | Interval aggregate replacing the per-run rows by default — one Info row per flush interval summarizing every instance script run since the last flush (see Volume Policy) |
| Alarm Events | Alarm activated, alarm cleared (which alarm, which instance), alarm evaluation error |
| Deployment Events | Configuration received from central, scripts compiled, applied successfully, apply failed |
| Data Connection Status | Connected, disconnected, reconnected (per connection) |
| Store-and-Forward | Message queued, delivered, retried, parked |
| Instance Lifecycle | Instance enabled, disabled, deleted |
| Notification | Site→central forward failure, long-buffered notification (still in the site buffer past a threshold) |

## Volume Policy — Script Run Events (WP3.2, 2026-08-15)

Per-run instance-script Started/Completed Info events were, before this policy, the dominant
`site_events` writer: two rows per run, translating to roughly a dozen physical row-writes
across a replicated site pair once CDC oplog/row-version rows and their eventual purge
tombstones are counted — for content nobody looks at unless something went wrong. The policy
below (design memo: `docs/plans/2026-08-15-site-events-policy-design.md`) keeps the signal
while removing the volume.

- **Off by default.** `ScadaBridge:SiteRuntime:PerRunScriptEvents` (default `false`) is the
  global switch back to the legacy per-run Started/Completed rows for every script.
- **Interval aggregate instead.** `ScriptRunSummaryRecorder` accumulates per-`(instance,
  script)` counters (started, completed, failed, timed out, total/max duration) from the same
  call sites that used to emit the per-run rows. A hosted flush service emits **one** "script"
  Info row per interval — `ScadaBridge:SiteRuntime:ScriptRunSummaryIntervalSeconds` (default
  `300`, i.e. 5 minutes; `0` disables the flush) — with headline totals in the message (e.g.
  "1,243 runs: 1,240 completed, 2 failed, 1 timed out across 17 scripts") and a per-script
  breakdown in `Details`, capped at the **top 50 scripts by run activity** with an "others"
  rollup entry so a pathological deployment cannot mint an oversized row. **An interval with
  zero activity emits no row** — the standby, which never runs scripts, therefore never
  produces a summary row naturally. Source is `"ScriptRunSummary"`; Instance ID is `NULL`
  (the row spans every instance).
- **Per-script opt-in for debugging.** `ScadaBridge:SiteRuntime:PerRunScriptEventScripts`
  (default empty) restores the legacy per-run rows for named scripts only, without paying the
  volume cost for the rest of the deployment. Entries are `"{InstanceName}/{ScriptName}"` for
  an exact match or `"{InstanceName}/*"` for every script on that instance. Effective only
  while the global switch is off.
- **Hot-togglable.** Both keys are read from the live options snapshot at the top of every
  run (`IOptionsMonitor<SiteRuntimeOptions>.CurrentValue`), so an operator can flip either one
  — including narrowing an opt-in to a single misbehaving script — without a restart or a
  redeploy.
- **Error-level events are unconditional.** Timeout, failure, stuck-watchdog, and
  recursion-limit Error rows are unaffected by any of the above — they always fire, regardless
  of `PerRunScriptEvents` or the opt-in list.

## Event Entry Schema

Each event entry contains:
- **Timestamp**: When the event occurred.
- **Event Type**: Category of the event (script, alarm, deployment, connection, store-and-forward, instance-lifecycle, notification).
- **Severity**: Info, Warning, or Error.
- **Instance ID** *(optional)*: The instance associated with the event (if applicable).
- **Source**: The subsystem that generated the event (e.g., "ScriptActor:MonitorSpeed", "AlarmActor:OverTemp", "DataConnection:PLC1").
- **Message**: Human-readable description of the event.
- **Details** *(optional)*: Additional structured data (e.g., exception stack trace, alarm name, message ID, compilation errors).

## Storage

- Events are stored in the **consolidated site LocalDb** SQLite file (`site_events` table;
  see CLAUDE.md → Consolidated site database). Only the **active node** generates and stores
  events (a standby runs no scripts, subsystems, or deployments to log).
- **`site_events` IS replicated** (`SiteLocalDbSetup.ReplicatedTables`) on a site pair with
  replication configured — this reverses the component's pre-LocalDb-Phase-1 behavior, and the
  reason is **failover history continuity, not either-node queries**: the query handler
  (`EventLogHandlerActor`) is a **cluster singleton**, so a central query always routes to the
  actively-written copy regardless of which node answers the gRPC call — replication is never
  needed to make queries *work*. What replication buys is **content to read after a failover**:
  when the active node dies, the singleton restarts on the survivor, and its query service reads
  the survivor's *local* copy — which has history *only because it was replicated*. Without
  replication, a failover would start the event log empty at precisely the moment an operator
  is investigating an incident (WP3.2 design memo, Decision (b): `docs/plans/2026-08-15-site-events-policy-design.md`).
  On a site pair running **without** replication configured (by deliberate choice, e.g. the
  rig's site-b/site-c), the log stays node-local and a failover does start it fresh — the
  documented trade of not configuring a peer, unchanged by this policy.
- **Retention**: 30 days. A **daily background job** runs on the active node and deletes all events older than 30 days. Hard delete — no archival. Deletes are sliced into bounded 1000-row batches per DELETE statement rather than one unbounded statement, so a large expired backlog does not hold the write lock (and every concurrent recorder flush) for the duration of the purge.
- **Storage cap**: A configurable maximum database size (default: 1 GB) is enforced. If the storage cap is reached before the 30-day retention window, the oldest events are purged first. This prevents disk exhaustion from alarm storms, script failure loops, or connection flapping.
- **Purge deletes are CDC-captured on a replicated site — the resulting backlog spike is expected.**
  Retention and cap deletes are ordinary row changes on a replicated table; there is deliberately no
  purge-exemption path (LocalDb Phase 2 — CDC does all three jobs, and a table-wide exemption is the
  same mechanism that made `ReplaceAllAsync` unsafe). One oplog row is queued per deleted event, so
  the site health report's `LocalDbOplogBacklog` (Prometheus `localdb_oplog_depth`) jumps by the size
  of the batch at purge time and drains as the peer acks it. **Healthy signature:**
  `LocalDbReplicationConnected` stays true across the spike and the backlog returns to ~0; a backlog
  that keeps climbing, or climbs while `LocalDbReplicationConnected` is false, is a genuine
  replication fault and not the purge. The purge logs an Information line naming the row count and
  the expected transient backlog whenever it deletes rows on a replication-enabled node, so an
  operator can correlate a spike with the purge that caused it — a spike with no such line nearby is
  not the purge. Node-local wiring: the Host supplies the replication predicate
  (`SiteLocalDbSetup.ReplicationIsConfigured` — the `PeerAddress`-OR-`ApiKey` rule) as
  `SiteEventLogReplicationCheck`; this component never reads LocalDb configuration itself, and with
  no predicate registered the note is suppressed. Operator detail —
  including that the purge timer is anchored to the active node's process start, not a wall-clock
  hour — is in `docs/deployment/topology-guide.md` → *Reading the replication backlog*. See also the
  Volume Policy section above for why this residual cost is small now that per-run script rows are
  off by default.

## Central Access

- The central UI can query site event logs remotely via the Communication Layer.
- Queries support filtering by:
  - Event type / category
  - Time range
  - Instance ID
  - Severity
  - **Keyword search**: Free-text search on message and source fields (SQLite LIKE query). Useful for finding events by script name, alarm name, or error message across all instances.
- Results are **paginated** with a configurable page size (default: 500 events). Each response includes a continuation token for fetching additional pages. This prevents broad queries from overwhelming the communication channel.
- The site processes the query locally against SQLite and returns matching results to central.

## Dependencies

- **SQLite**: Local storage on each site node.
- **Communication Layer**: Handles remote query requests from central.
- **Site Runtime**: Generates script execution events, alarm events, deployment application events, and instance lifecycle events.
- **Data Connection Layer**: Generates connection status events.
- **Store-and-Forward Engine**: Generates buffer activity events, including notification-category forward failures and long-buffered notifications on the site→central notification path.

## Interactions

- **All site subsystems**: Event logging is a cross-cutting concern — any subsystem that produces notable events calls the Event Logging service.
- **Communication Layer**: Receives remote queries from central and returns results.
- **Central UI**: Site Event Log Viewer displays queried events.
- **Store-and-Forward Engine**: Its notification path (the site→central forward of script-generated notifications) reports forward failures and long-buffered notifications as Notification-category events. Routine enqueue and forward-success events are deliberately not logged — central's authoritative `Notifications` table (owned by the Notification Outbox component) is the audit record of record; site-side logging covers only the in-transit blind spot when central is unreachable.
- **Health Monitoring**: Script error rates and alarm evaluation error rates can be derived from event log data; this component also reports `SiteEventLogWriteFailures`, and its purge is the expected cause of the transient `LocalDbOplogBacklog` spikes described in [Component-HealthMonitoring.md](Component-HealthMonitoring.md) → Monitored Metrics.