ScadaBridge/docs/requirements/Component-SiteEventLogging.md

# Component: Site Event Logging

## Purpose

The Site Event Logging component records operational events at each site cluster, providing a local audit trail of runtime activity. Events are queryable from the central UI for remote troubleshooting.

## Location

Site clusters (event recording and storage). Central cluster (remote query access via UI).

## Responsibilities

- Record operational events from all site subsystems.
- Persist events to local SQLite.
- Enforce 30-day retention policy with automatic purging.
- Respond to remote queries from central for event log data.

## Events Logged

| Category | Events |
|----------|--------|
| Script Executions | Script started, completed *(sampled/opt-in — see Volume Policy below)*; failed, timed out, recursion limit exceeded (with error details; always logged) |
| Script Run Summary | Interval aggregate replacing the per-run rows by default — one Info row per flush interval summarizing every instance script run since the last flush (see Volume Policy) |
| Alarm Events | Alarm activated, alarm cleared (which alarm, which instance), alarm evaluation error |
| Deployment Events | Configuration received from central, scripts compiled, applied successfully, apply failed |
| Data Connection Status | Connected, disconnected, reconnected (per connection) |
| Store-and-Forward | Message queued, delivered, retried, parked |
| Instance Lifecycle | Instance enabled, disabled, deleted |
| Notification | Site→central forward failure, long-buffered notification (still in the site buffer past a threshold) |

## Volume Policy — Script Run Events (WP3.2, 2026-08-15)

Per-run instance-script Started/Completed Info events were, before this policy, the dominant
`site_events` writer: two rows per run, translating to roughly a dozen physical row-writes
across a replicated site pair once CDC oplog/row-version rows and their eventual purge
tombstones are counted — for content nobody looks at unless something went wrong. The policy
below (design memo: `docs/plans/2026-08-15-site-events-policy-design.md`) keeps the signal
while removing the volume.

- **Off by default.** `ScadaBridge:SiteRuntime:PerRunScriptEvents` (default `false`) is the
  global switch back to the legacy per-run Started/Completed rows for every script.
- **Interval aggregate instead.** `ScriptRunSummaryRecorder` accumulates per-`(instance,
  script)` counters (started, completed, failed, timed out, total/max duration) from the same
  call sites that used to emit the per-run rows. A hosted flush service emits **one** "script"
  Info row per interval — `ScadaBridge:SiteRuntime:ScriptRunSummaryIntervalSeconds` (default
  `300`, i.e. 5 minutes; `0` disables the flush) — with headline totals in the message (e.g.
  "1,243 runs: 1,240 completed, 2 failed, 1 timed out across 17 scripts") and a per-script
  breakdown in `Details`, capped at the **top 50 scripts by run activity** with an "others"
  rollup entry so a pathological deployment cannot mint an oversized row. **An interval with
  zero activity emits no row** — the standby, which never runs scripts, therefore never
  produces a summary row naturally. Source is `"ScriptRunSummary"`; Instance ID is `NULL`
  (the row spans every instance).
- **Per-script opt-in for debugging.** `ScadaBridge:SiteRuntime:PerRunScriptEventScripts`
  (default empty) restores the legacy per-run rows for named scripts only, without paying the
  volume cost for the rest of the deployment. Entries are `"{InstanceName}/{ScriptName}"` for
  an exact match or `"{InstanceName}/*"` for every script on that instance. Effective only
  while the global switch is off.
- **Hot-togglable.** Both keys are read from the live options snapshot at the top of every
  run (`IOptionsMonitor<SiteRuntimeOptions>.CurrentValue`), so an operator can flip either one
  — including narrowing an opt-in to a single misbehaving script — without a restart or a
  redeploy.
- **Error-level events are unconditional.** Timeout, failure, stuck-watchdog, and
  recursion-limit Error rows are unaffected by any of the above — they always fire, regardless
  of `PerRunScriptEvents` or the opt-in list.

## Event Entry Schema

Each event entry contains:
- **Timestamp**: When the event occurred.
- **Event Type**: Category of the event (script, alarm, deployment, connection, store-and-forward, instance-lifecycle, notification).
- **Severity**: Info, Warning, or Error.
- **Instance ID** *(optional)*: The instance associated with the event (if applicable).
- **Source**: The subsystem that generated the event (e.g., "ScriptActor:MonitorSpeed", "AlarmActor:OverTemp", "DataConnection:PLC1").
- **Message**: Human-readable description of the event.
- **Details** *(optional)*: Additional structured data (e.g., exception stack trace, alarm name, message ID, compilation errors).

## Storage

- Events are stored in the **consolidated site LocalDb** SQLite file (`site_events` table;
  see CLAUDE.md → Consolidated site database). Only the **active node** generates and stores
  events (a standby runs no scripts, subsystems, or deployments to log).
- **`site_events` IS replicated** (`SiteLocalDbSetup.ReplicatedTables`) on a site pair with
  replication configured — this reverses the component's pre-LocalDb-Phase-1 behavior, and the
  reason is **failover history continuity, not either-node queries**: the query handler
  (`EventLogHandlerActor`) is a **cluster singleton**, so a central query always routes to the
  actively-written copy regardless of which node answers the gRPC call — replication is never
  needed to make queries *work*. What replication buys is **content to read after a failover**:
  when the active node dies, the singleton restarts on the survivor, and its query service reads
  the survivor's *local* copy — which has history *only because it was replicated*. Without
  replication, a failover would start the event log empty at precisely the moment an operator
  is investigating an incident (WP3.2 design memo, Decision (b): `docs/plans/2026-08-15-site-events-policy-design.md`).
  On a site pair running **without** replication configured (by deliberate choice, e.g. the
  rig's site-b/site-c), the log stays node-local and a failover does start it fresh — the
  documented trade of not configuring a peer, unchanged by this policy.
- **Retention**: 30 days. A **daily background job** runs on the active node and deletes all events older than 30 days. Hard delete — no archival. (Today, retention/cap deletes on a replicated node are captured by CDC like any other write — see the Volume Policy section above for why this is a small residual cost now that per-run rows are off by default.)
- **Storage cap**: A configurable maximum database size (default: 1 GB) is enforced. If the storage cap is reached before the 30-day retention window, the oldest events are purged first. This prevents disk exhaustion from alarm storms, script failure loops, or connection flapping.

## Central Access

- The central UI can query site event logs remotely via the Communication Layer.
- Queries support filtering by:
  - Event type / category
  - Time range
  - Instance ID
  - Severity
  - **Keyword search**: Free-text search on message and source fields (SQLite LIKE query). Useful for finding events by script name, alarm name, or error message across all instances.
- Results are **paginated** with a configurable page size (default: 500 events). Each response includes a continuation token for fetching additional pages. This prevents broad queries from overwhelming the communication channel.
- The site processes the query locally against SQLite and returns matching results to central.

## Dependencies

- **SQLite**: Local storage on each site node.
- **Communication Layer**: Handles remote query requests from central.
- **Site Runtime**: Generates script execution events, alarm events, deployment application events, and instance lifecycle events.
- **Data Connection Layer**: Generates connection status events.
- **Store-and-Forward Engine**: Generates buffer activity events, including notification-category forward failures and long-buffered notifications on the site→central notification path.

## Interactions

- **All site subsystems**: Event logging is a cross-cutting concern — any subsystem that produces notable events calls the Event Logging service.
- **Communication Layer**: Receives remote queries from central and returns results.
- **Central UI**: Site Event Log Viewer displays queried events.
- **Store-and-Forward Engine**: Its notification path (the site→central forward of script-generated notifications) reports forward failures and long-buffered notifications as Notification-category events. Routine enqueue and forward-success events are deliberately not logged — central's authoritative `Notifications` table (owned by the Notification Outbox component) is the audit record of record; site-side logging covers only the in-transit blind spot when central is unreachable.
- **Health Monitoring**: Script error rates and alarm evaluation error rates can be derived from event log data.