fix(auditlog): explicit permanent rejections on the ingest ack (IngestAck.rejected_event_ids, field 2) with a dedicated counter; reconciler ignores transient faults for abandonment; persist the served high-water mark; prune only on a non-empty enumeration (review 2026-10-21 06/PERF-1, 06/IMPL-1 adversarial follow-up)

docs/requirements/Component-AuditLog.md

@@ -311,17 +311,27 @@ A `SiteAuditTelemetryActor` singleton drives the forwarding loop: select up to
 N `Pending` rows ordered by `OccurredAtUtc`, batch-send them to central via the
 existing `SiteStream` gRPC channel as `IngestAuditEvents(events)`, and on
 central-ack flip `ForwardState = 'Forwarded'` for accepted IDs. IDs central
-answered for but did not accept stay `Pending` for the next sweep, with
-`AttemptCount` / `LastAttemptUtc` bumped on every such refusal; after
-`AuditLog:SiteTelemetry:MaxRejectedPushAttempts` (default 10) refusals a row
-leaves the push queue as `Rejected` and bumps `SiteAuditWriteFailures`
-(review 2026-10-21 06/IMPL-1). The drain reads oldest-first, so before this one
+answered for but did not accept stay `Pending` for the next sweep. Central
+additionally lists, in the ack's additive `rejected_event_ids` (sitestream.proto
+`IngestAck` field 2), the ids it refused for a **permanent, row-intrinsic** SQL
+reason (string truncation 8152/2628, NOT NULL 515, conversion/overflow
+245/241/242/8114/8115/220/232, constraint 547). Only those are refusals: the site
+bumps `AttemptCount` / `LastAttemptUtc` for them and, after
+`AuditLog:SiteTelemetry:MaxRejectedPushAttempts` (default 10), moves the row out of
+the push queue as `Rejected`, counted on its own `scadabridge.site.audit.rejected_rows`
+counter (not `SiteAuditWriteFailures` — the site's write succeeded). A row merely
+absent from the accepted list — a SQL maintenance window, a timeout, an exhausted
+ingest budget — is never counted, and an older central that does not send the field
+never triggers it (review 2026-10-21 06/IMPL-1). The drain reads oldest-first, so before this one
 row central can never insert (e.g. an over-width column) sat at the head of every
 batch forever, failing each set-based insert at central into the per-row fallback.
 `Rejected` is not a drop: the reconciliation pull still serves the row, so a
 transient refusal still lands, and a genuinely poison row is recorded centrally as
 a `ReconciliationAbandoned` row before its cursor retires it. A transport fault (no
-answer) is never counted as a refusal. A cached-lifecycle row whose tracking
+answer) is never counted as a refusal. Symmetrically, the reconciliation puller counts
+only non-transient failures toward its permanent-abandon threshold: a timeout,
+connection loss, deadlock or throttling error holds the cursor and retries without
+ever abandoning the row. A cached-lifecycle row whose tracking
 snapshot is gone past `CachedTrackingGraceSeconds` leaves the cached drain as
 `PushAbandoned` (it used to be marked `Forwarded`, which only worked because the
 pull re-served every `Forwarded` row). Cadence is short (default 5 s) when
@@ -392,9 +402,23 @@ reports `MoreAvailable` and the cursor advances, mirroring `SiteCallAuditActor`.
 **Mixed versions:** a pre-PERF-1 site still serves `Forwarded` rows; central
 tolerates that unchanged (idempotent on `EventId`, merely more pages).
 
+**Rollback caveat.** Rolling a site back to a pre-PERF-1 binary strands any rows already in
+`PushAbandoned` / `Rejected` on disk: the old code neither pushes, serves, retires nor purges
+a state it does not know, so they sit in the audit file (harmless, but never delivered and
+never purged) until the node is upgraded again — at which point the pull serves them
+normally. Nothing else in the file changes shape (the `audit_site_state` table is ignored by
+an old binary).
+
+**Oversize rows (accepted, documented).** A single site audit row larger than the 32 MB gRPC
+ceiling (`GrpcMessageLimits.MaxMessageBytes`) cannot be pushed or pulled and would wedge the
+head of both paths. It is unreachable under the payload caps (8 KB / 64 KB on error by
+default) unless `DefaultCapBytes`/`ErrorCapBytes` are configured into the tens of megabytes;
+rows above the 2 MB batch budget but under the ceiling already travel alone.
+
 **Per-site state follows the site list (review 2026-10-21 06/MEM-2).** Each tick drops
 the cursor, non-draining counter and stalled latch of any site no longer returned by
-`ISiteEnumerator` (a latched site publishes `Stalled=false` as it goes), and the
+`ISiteEnumerator` (a latched site publishes `Stalled=false` as it goes; an EMPTY enumeration
+prunes nothing, since it is as likely a transient lookup glitch), and the
 `PullAuditEvents` / `PullSiteCalls` / `PullTraceEvents` invokers evict and dispose any
 cached `(site, endpoint)` gRPC channel that is not a live site's primary or fallback
 address on every enumeration. The in-memory abandoned-`EventId` short-circuit set is
@@ -476,16 +500,18 @@ carries a second, insertion-order bound:
 > cursor at all — central ACKED them over the telemetry push path, they are already
 > purge-eligible, and the pull never served them.
 
-The bound is per-process, so after a site-node restart the first pull retires nothing
-and the pull after it resumes normal retirement — conservative in the
-safe direction (retirement is delayed, never a row lost). The gRPC handler's ordering
+The bound is persisted in a node-local `audit_site_state` key/value table in the audit
+file (loaded at start-up, clamped to the live `MAX(rowid)`), so a site-node restart no longer
+resets it — before that, a healthy site whose pull serves nothing could leave already-served
+`PushAbandoned`/`Rejected` rows owed (and counted on the backlog) indefinitely. It stays
+conservative in the safe direction (retirement can be delayed, never a row lost). The gRPC handler's ordering
 (retire, *then* read) is what keeps the bound from ever vouching for the batch it is
 about to serve.
 
 **Liveness note (accepted, documented).** A row that was served but never covered by a
-later cursor — central reconciliation stopped for good, or the bound reset over a
-restart before the next cursor arrived — stays `Pending` indefinitely, and the site
-retention purge never purges `Pending`. That is the hard `ForwardState` invariant
+later cursor — central reconciliation stopped for good — stays owed (`Pending` /
+`PushAbandoned` / `Rejected`) indefinitely, and the site retention purge never purges
+an owed row. That is the hard `ForwardState` invariant
 working as intended (an unacknowledged row is not droppable), but it means the site
 store's floor depends on reconciliation actually running, not merely on the retention
 window elapsing. It is observable rather than silent: `GetBacklogStatsAsync` reports