feat(audit-log): permanently-abandoned reconciliation rows leave a durable ReconciliationAbandoned audit record

When a pulled AuditEvent fails to insert on every retry up to the permanent-
abandon threshold, the reconciliation actor now writes ONE synthetic
ReconciliationAbandoned row (new AuditKind) alongside the Critical log line —
fresh EventId, Status=Failed, channel preserved from the lost row, Extra carrying
the abandoned EventId + source site + final error — via the same
ScadaBridgeAuditEventFactory the ingest path uses. Best-effort in its own
try/catch so it never blocks the cursor twice. The permanent loss is now
queryable in the Audit Log, not only in a rotating log file.
This commit is contained in:
Joseph Doherty
2026-07-09 08:50:44 -04:00
parent 20098c6108
commit 0385942c3f
4 changed files with 207 additions and 1 deletions
+10
View File
@@ -138,6 +138,7 @@ row per lifecycle event across all channels.
| `SecuredWriteApprove` | A verifier wins the approval CAS for a pending secured write. |
| `SecuredWriteReject` | A verifier rejects a pending secured write (`Status=Discarded`). |
| `SecuredWriteExecute` | The approved write was relayed to the site MxGateway — terminal outcome (`Delivered`-equivalent on success, `Failed` on error). |
| `ReconciliationAbandoned` | Central-direct synthetic row (`Status=Failed`) written when a reconciliation pull row failed to insert on every retry up to the permanent-abandon threshold and central advanced its cursor past it. `Extra` carries the abandoned `EventId`, the source site, and the final error — so the permanent loss is queryable, not only in the Critical log line. |
Inbound API is intentionally collapsed to a single `InboundRequest` (or
`InboundAuthFailure` for auth rejections) row per request rather than a
@@ -394,6 +395,15 @@ MS SQL for direct-write events). Unredacted secrets never persist.
- **Reconciliation as fallback.** If two consecutive reconciliation cycles
report a non-draining backlog, the supervisor restarts the telemetry actor
and a `SiteAuditTelemetryStalled` event fires.
- **Permanent-abandonment record.** A reconciliation pull row that fails to
insert on every retry up to the permanent-abandon threshold is dropped and the
cursor advances past it (so one broken row can't block the site forever). The
loss is not silent: alongside the Critical log line the actor writes ONE
synthetic `ReconciliationAbandoned` audit row (fresh `EventId`, `Status=Failed`,
channel preserved from the lost row where parseable) whose `Extra` carries the
abandoned `EventId`, the source site, and the final error — so the permanent
loss is queryable in the Audit Log itself. That write is best-effort in its own
try/catch and never blocks the cursor a second time.
- **No dedup horizon.** `EventId` PK enforces uniqueness only while a row
exists. A retry that arrives after the original row is purged inserts a "new"
row — vanishingly rare and harmless.