dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(auditlog): reconciliation pull serves only unacked rows (PushAbandoned/Rejected push-exit states), central pages within a tick with set-based inserts; refused push rows leave the queue after N attempts (review 2026-10-21 06/PERF-1, 06/COR-3, 06/IMPL-1)

Browse Source
This commit is contained in:
Joseph Doherty
2026-10-22 06:38:02 -04:00
parent 1da5d0bac4
commit 9e7e19c3c1
13 changed files with 873 additions and 116 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/requirements/Component-AuditLog.md
+61 -20
View File
@@ -255,13 +255,21 @@ has no anchor and falls back to the unbounded scan (correctness over speed).
A SQLite database file on each site node, alongside the Store-and-Forward
buffer. Same schema as central minus `IngestedAtUtc` (irrelevant at the source),
plus a `ForwardState` column with values `Pending | Forwarded | Reconciled` that
drives the telemetry loop and reconciliation pull. `SourceNode` is stamped by the
drives the telemetry loop and reconciliation pull — and, since review 2026-10-21
(06/PERF-1, 06/IMPL-1), two site-only **push-exit** states, `PushAbandoned` and
`Rejected`, for rows that left the telemetry push queue *without* a central ack.
`Forwarded` means "central acked it"; `Pending`, `PushAbandoned` and `Rejected`
together are **owed** to central: the reconciliation pull serves exactly those,
a covering cursor retires them to `Reconciled`, and the retention purge never
drops them. The extra states are strings in the site sidecar only (no schema
change, never on the wire); a downgraded node ignores rows in a state it does not
know. `SourceNode` is stamped by the
writing node itself (`node-a` / `node-b`) at append time and travels with the row
through telemetry and reconciliation unchanged.
**Site SQLite retention rule (hard invariant):**
> A row is eligible for purge only when both `OccurredAtUtc < retention threshold` AND `ForwardState IN ('Forwarded', 'Reconciled')`. Pending rows are never purged.
> A row is eligible for purge only when both `OccurredAtUtc < retention threshold` AND `ForwardState IN ('Forwarded', 'Reconciled')`. Pending rows — and the owed push-exit states `PushAbandoned` / `Rejected` — are never purged.
A prolonged central outage will grow the site audit table indefinitely until
central is reachable again. This is intentional — losing audit rows to make
@@ -302,8 +310,21 @@ round-trip on the hot path.
A `SiteAuditTelemetryActor` singleton drives the forwarding loop: select up to
N `Pending` rows ordered by `OccurredAtUtc`, batch-send them to central via the
existing `SiteStream` gRPC channel as `IngestAuditEvents(events)`, and on
central-ack flip `ForwardState = 'Forwarded'` for accepted IDs. Rejected IDs
stay `Pending` for the next sweep. Cadence is short (default 5 s) when
central-ack flip `ForwardState = 'Forwarded'` for accepted IDs. IDs central
answered for but did not accept stay `Pending` for the next sweep, with
`AttemptCount` / `LastAttemptUtc` bumped on every such refusal; after
`AuditLog:SiteTelemetry:MaxRejectedPushAttempts` (default 10) refusals a row
leaves the push queue as `Rejected` and bumps `SiteAuditWriteFailures`
(review 2026-10-21 06/IMPL-1). The drain reads oldest-first, so before this one
row central can never insert (e.g. an over-width column) sat at the head of every
batch forever, failing each set-based insert at central into the per-row fallback.
`Rejected` is not a drop: the reconciliation pull still serves the row, so a
transient refusal still lands, and a genuinely poison row is recorded centrally as
a `ReconciliationAbandoned` row before its cursor retires it. A transport fault (no
answer) is never counted as a refusal. A cached-lifecycle row whose tracking
snapshot is gone past `CachedTrackingGraceSeconds` leaves the cached drain as
`PushAbandoned` (it used to be marked `Forwarded`, which only worked because the
pull re-served every `Forwarded` row). Cadence is short (default 5 s) when
non-empty, longer (default 30 s) when idle; telemetry runs on a dedicated
dispatcher.
@@ -349,10 +370,27 @@ still replied while the outer Asks are still waiting.
### Reconciliation pull (self-healing for missed telemetry)
A central `SiteAuditReconciliationActor` periodically (default 5 min per site)
asks each site for its oldest `Pending` row and pending count; if backlog is
non-draining (e.g., telemetry actor wedged), central issues a
`PullAuditEvents(sinceUtc, batchSize[, afterId])` and inserts-if-not-exists.
Same self-healing pattern as Site Call Audit's reconciliation of `SiteCalls`.
issues `PullAuditEvents(sinceUtc, batchSize[, afterId])` against each site and
inserts-if-not-exists. Same self-healing pattern as Site Call Audit's
reconciliation of `SiteCalls`.
**The pull serves only rows central has not acknowledged (review 2026-10-21
06/PERF-1).** `ReadPendingSinceAsync` returns `Pending`, `PushAbandoned` and
`Rejected` rows — never `Forwarded`. It used to serve `Forwarded` rows too, so
every row the push had already delivered was re-shipped and re-inserted
(row-at-a-time) one tick later. That capped the pull at 256 rows per 5-minute tick
(≈0.85 rows/s per site, 60× below the push), so any site above that rate fell
permanently behind: a genuinely lost row waited behind days of already-forwarded
rows until the 7-day site purge, the stalled badge latched true on a healthy busy
site, and every central failover (cursor reset to `MinValue`) restarted a
multi-day replay (06/COR-3, resolved by this change — the re-pull after a reset is
now only the owed set). Central ingests each page with ONE set-based
`InsertManyIfNotExistsAsync` (per-row fallback only when the batch throws, so the
poison-row retry/abandon semantics are unchanged) and **pages within a tick** — up
to `AuditLog:Reconciliation:MaxPagesPerTick` (default 50) pulls while the site
reports `MoreAvailable` and the cursor advances, mirroring `SiteCallAuditActor`.
**Mixed versions:** a pre-PERF-1 site still serves `Forwarded` rows; central
tolerates that unchanged (idempotent on `EventId`, merely more pages).
**`PullTraceEvents` is a point read, not a reconciliation path (2026-09-05).** The
Flow Debugger's value-change run cannot get its steps inline (the device echo
@@ -407,10 +445,12 @@ the legacy inclusive `>=` read as the compatible fallback. The cursor stays in-m
and resets on singleton restart; idempotent `InsertIfNotExistsAsync` still absorbs any
re-pulled duplicates, so exactness is an optimization, not a correctness dependency.
Unlike Site Call Audit, this actor issues ONE pull per tick rather than paging within
it — a lagging drain is meant to surface as the stalled signal. The id tiebreak is
what makes that safe against a same-instant burst larger than one batch: the cursor
advances on every tick even when the timestamp cannot.
Like Site Call Audit, the actor pages within a tick (bounded by `MaxPagesPerTick`), so
the stalled signal fires only when a site still reports `MoreAvailable` after the page
cap — or its cursor is held back by a row that keeps failing — on
`StalledAfterNonDrainingCycles` consecutive ticks: a genuine non-drain, not a busy
site. The id tiebreak keeps paging safe against a same-instant burst larger than one
batch: the cursor advances on every page even when the timestamp cannot.
**Only rows that were actually served may retire.** `OccurredAtUtc` is stamped by the
caller, so a row can be *inserted* after a batch was served yet carry a timestamp
@@ -421,13 +461,14 @@ and, being `Reconciled`, purged on age. That is silent audit loss, and it is a f
mode the pre-WP2.3 explicit id-set flip could not produce, so `MarkReconciledUpToAsync`
carries a second, insertion-order bound:
> A `Pending` row retires only if its insertion order is at or below the high-water
> mark of rows this site node has actually served from `ReadPendingSinceAsync`
> (SQLite `rowid`). A `Forwarded` row is exempt — central ACKED it over the telemetry
> push path, which is proof independent of the pull.
> An owed row (`Pending` / `PushAbandoned` / `Rejected`) retires only if its insertion
> order is at or below the high-water mark of rows this site node has actually served
> from `ReadPendingSinceAsync` (SQLite `rowid`). `Forwarded` rows are not touched by the
> cursor at all — central ACKED them over the telemetry push path, they are already
> purge-eligible, and the pull never served them.
The bound is per-process, so after a site-node restart the first pull retires only
`Forwarded` rows and the pull after it resumes normal retirement — conservative in the
The bound is per-process, so after a site-node restart the first pull retires nothing
and the pull after it resumes normal retirement — conservative in the
safe direction (retirement is delayed, never a row lost). The gRPC handler's ordering
(retire, *then* read) is what keeps the bound from ever vouching for the batch it is
about to serve.
@@ -586,8 +627,8 @@ MS SQL for direct-write events). Unredacted secrets never persist.
best-effort drop with a counter, never unbounded memory (review 2026-10-21
06/MEM-1). Audit failure still never aborts the action.
- **Reconciliation as fallback.** If two consecutive reconciliation cycles
report a non-draining backlog, the supervisor restarts the telemetry actor
and a `SiteAuditTelemetryStalled` event fires.
still report a non-draining backlog after paging up to `MaxPagesPerTick`, a
`SiteAuditTelemetryStalled` event fires.
- **Permanent-abandonment record.** A reconciliation pull row that fails to
insert on every retry up to the permanent-abandon threshold is dropped and the
cursor advances past it (so one broken row can't block the site forever). The