Merge branch 'worktree-agent-a465fb3cd6ec48cd3' into arch-review-remediation

docs/requirements/Component-AuditLog.md

@@ -263,7 +263,15 @@ room is a compliance violation, not a self-healing behavior. To bound that
 growth in practice, the site emits a `SiteAuditBacklog` health metric (pending
 row count, oldest pending age, bytes on disk); crossing operator-configured
 thresholds surfaces a warning on the relevant site tile in the Health
-dashboard, mirroring the Store-and-Forward Engine's backlog metric.
+dashboard, mirroring the Store-and-Forward Engine's backlog metric, and
+`SiteAuditBacklogReporter` additionally logs a rate-limited warning once the
+oldest pending row passes 24 h.
+
+The same invariant carries a liveness dependency worth stating explicitly: the
+`Pending` floor clears only when central acknowledges the rows — a telemetry ack
+or a reconciliation cursor that covers them — never on age. See *Reconciliation
+pull* below for the served-row retirement rule that decides which rows can be
+acknowledged by a cursor at all.
 
 Central is the durable home. Site SQLite is a write-buffer with a forwarding
 guarantee.
@@ -387,6 +395,38 @@ it — a lagging drain is meant to surface as the stalled signal. The id tiebrea
 what makes that safe against a same-instant burst larger than one batch: the cursor
 advances on every tick even when the timestamp cannot.
 
+**Only rows that were actually served may retire.** `OccurredAtUtc` is stamped by the
+caller, so a row can be *inserted* after a batch was served yet carry a timestamp
+*below* central's (by then advanced) cursor — a back-dated stamp, a clock nudge, a
+write flushed late. A cursor flip that keyed on the timestamp alone retired exactly
+those rows: never served, never servable again (the keyset read has moved past them),
+and, being `Reconciled`, purged on age. That is silent audit loss, and it is a failure
+mode the pre-WP2.3 explicit id-set flip could not produce, so `MarkReconciledUpToAsync`
+carries a second, insertion-order bound:
+
+> A `Pending` row retires only if its insertion order is at or below the high-water
+> mark of rows this site node has actually served from `ReadPendingSinceAsync`
+> (SQLite `rowid`). A `Forwarded` row is exempt — central ACKED it over the telemetry
+> push path, which is proof independent of the pull.
+
+The bound is per-process, so after a site-node restart the first pull retires only
+`Forwarded` rows and the pull after it resumes normal retirement — conservative in the
+safe direction (retirement is delayed, never a row lost). The gRPC handler's ordering
+(retire, *then* read) is what keeps the bound from ever vouching for the batch it is
+about to serve.
+
+**Liveness note (accepted, documented).** A row that was served but never covered by a
+later cursor — central reconciliation stopped for good, or the bound reset over a
+restart before the next cursor arrived — stays `Pending` indefinitely, and the site
+retention purge never purges `Pending`. That is the hard `ForwardState` invariant
+working as intended (an unacknowledged row is not droppable), but it means the site
+store's floor depends on reconciliation actually running, not merely on the retention
+window elapsing. It is observable rather than silent: `GetBacklogStatsAsync` reports
+the pending count and oldest-pending instant on every site health report
+(`SiteAuditBacklog`), and `SiteAuditBacklogReporter` logs a rate-limited warning once
+the oldest pending row exceeds `StalePendingThreshold` (24 h) naming the drain and the
+reconciliation pull as the things to check.
+
 ### Central direct-write (central-originated events)
 
 Events originating at central never touch site SQLite. Inbound API writes one

docs/requirements/Component-Communication.md

@@ -70,7 +70,8 @@ Both central and site clusters. Each side has communication actors that handle m
 - **Bridge-session hardening (WP2.3):**
   - The pre-snapshot buffer is **bounded (20 000 events, drop-oldest)** and its evictions counted (`scadabridge.central.debug_view.presnapshot_dropped`). It was previously unbounded, so a session whose snapshot never arrived grew without limit on the CENTRAL node. Dropping the oldest is correct here: the snapshot that ends the buffering phase is authoritative for anything that old.
   - A **hard snapshot deadline** (`DebugStreamBridgeActor.SnapshotTimeout`, 60 s) fails the session if no `DebugViewSnapshot` arrives. Nothing else ended a session wedged in the buffering phase — a lost site reply raises no gRPC error.
-  - **Stream events no longer influence the orphan receive timeout.** The gRPC callback wraps each event in an envelope marked `INotInfluenceReceiveTimeout`, so a busy site can no longer keep an abandoned session alive forever by feeding it events. The 5-minute timeout now measures session/consumer liveness, which is what it was for.
+  - **The orphan net measures the CONSUMER, not the mailbox.** A session self-terminates after `DebugStreamBridgeActor.ConsumerIdleTimeout` (5 min) without a sign of life from its consumer, checked by a periodic self-tick (`ConsumerLivenessCheckInterval`, 30 s) against a consumer-last-seen stamp. The stamp is renewed only by `DebugStreamConsumerAlive`, which `DebugStreamService` Tells to every session still in its registry on a shared 30 s timer — holding a session there IS what "a consumer is attached" means, since both consumers (the Blazor debug view and the SignalR hub) release it on dispose/disconnect.
+    This replaced an Akka `SetReceiveTimeout(5 min)`. That version measured the MAILBOX, which conflates site chatter with consumer liveness: a busy site kept an abandoned session alive forever. Marking stream events `INotInfluenceReceiveTimeout` fixed that but left the timeout with nothing recurring to reset it — the snapshot lands once and the stability tick once — so **every healthy session self-terminated ~6 min in and the operator was told "Site disconnected"**. Both failure directions are pinned by tests: streaming events with keepalives survive many windows; an orphaned session terminates while events keep arriving.
 
 ### 6.1 Aggregated Live Alarm Stream (Site → Central)
 
@@ -80,7 +81,8 @@ Delivered 2026-07-10 (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.m
 - **Central live cache** (`ISiteAlarmLiveCache`, singleton `SiteAlarmLiveCacheService`): a DI singleton on the active central node. For each site with ≥1 active viewer it runs ONE shared, **reference-counted** per-site aggregator (`SiteAlarmAggregatorActor`); the first `Subscribe(siteId, onChanged)` starts it, the last subscriber leaving stops it after a short **linger** to avoid re-seed thrash. `GetCurrentAlarms(siteId)` returns the current immutable snapshot; `IsLive(siteId)` reports whether the aggregator has seeded **and its site-wide stream is currently up**. Liveness rides every publish from the aggregator, so a stream that faults or ends gracefully drops `IsLive` immediately rather than leaving the page grafting a freezing snapshot over fresh poll data until the next reconcile (WP2.3).
 - **Seed-then-stream** (copied from `DebugStreamBridgeActor` ordering): open the `SubscribeSite` stream first (buffer live deltas), run the snapshot fan-out once via the existing `DebugViewSnapshot` path (bounded by `LiveAlarmCacheSeedConcurrency`), flush the buffer with **dedup by `(InstanceUniqueName, AlarmName, SourceReference)`**, then live pass-through. Placeholders are seeded from the snapshot and never expected on the live stream.
 - **Alarms-only seed (WP2.3)**: the seed/reconcile fan-out sets `DebugSnapshotRequest.AlarmsOnly` (wire: `DebugSnapshotRequestDto.alarms_only`, field 3, additive), so the site builds and ships only the alarm half of the snapshot. The fan-out discarded every attribute row anyway, and an instance's attribute surface dwarfs its alarm set. A pre-WP2.3 site ignores the flag and returns the full snapshot, which reads identically.
-- **Failover & drift**: a re-seed runs **once per successful (re)connect**, not once per reconnect ATTEMPT — the connect signal is `SiteStreamGrpcClient.SubscribeSiteAsync`'s `onConnected` callback, raised when the site's response headers arrive (the site flushes them as soon as its relay is attached, so no event can be missed after it). Fanning a whole-site snapshot out per retry meant N snapshots against a site that was, by definition of the retry, unreachable. A periodic **reconcile snapshot** (default 60s, **jittered** by `LiveAlarmCacheReconcileJitterFraction` so aggregators started together do not stampede one boundary) remains the drift backstop, but it is **skipped when a fan-out already ran in that window** and **publishes only when the snapshot actually changed the cache** (a diff, not an unconditional viewer fan-out). Staleness stays bounded at two intervals: the skip consumes its flag, so the next tick always fans out. A fan-out that fails as a whole now retries on its own **backoff** timer (`reconnectDelay` doubling, capped at 8× the reconcile interval) instead of waiting a full interval. `[PERM]` (`docs/plans/2026-05-29-native-alarms-design.md`): the cache is **purely in-memory** — no EF entity/table/migration, no persisted central alarm store — so a new active node simply re-seeds from scratch.
+- **Failover & drift**: a re-seed runs **once per successful (re)connect**, not once per reconnect ATTEMPT — the connect signal is `SiteStreamGrpcClient.SubscribeSiteAsync`'s `onConnected` callback, raised when the site's response headers arrive (the site flushes them as soon as its relay is attached, so no event can be missed after it) or, for a peer that defers its headers, when the FIRST EVENT arrives — whichever comes first, and at most once. A header **timeout** is deliberately not a connect signal: an unreachable or wedged site produces exactly that shape, and reporting it as connected cleared `_streamDown`, consumed the pending re-seed and fanned a whole-site snapshot out at a site that never answered. Fanning a whole-site snapshot out per retry meant N snapshots against a site that was, by definition of the retry, unreachable. A periodic **reconcile snapshot** (default 60s, **jittered** by `LiveAlarmCacheReconcileJitterFraction` so aggregators started together do not stampede one boundary) remains the drift backstop, but it is **skipped when a CONNECT- OR FAILOVER-DRIVEN seed already ran in that window** and **publishes only when the snapshot actually changed the cache** (a diff, not an unconditional viewer fan-out). Staleness stays bounded at two intervals: the skip consumes its flag, so the next tick always fans out. A **tick's own fan-out never arms that skip** — when it did, steady state alternated fan-out/skip and the effective reconcile rate was one per TWO intervals (120 s by default), halving both the not-reporting refresh and the alarm reconcile backstop for no benefit: with no reconnect in play there is nothing duplicated to suppress. A fan-out that fails as a whole now retries on its own **backoff** timer (`reconnectDelay` doubling, capped at 8× the reconcile interval) instead of waiting a full interval. `[PERM]` (`docs/plans/2026-05-29-native-alarms-design.md`): the cache is **purely in-memory** — no EF entity/table/migration, no persisted central alarm store — so a new active node simply re-seeds from scratch.
+- **Stream terminations are a triad, and every ending hits exactly one leg.** `SiteStreamGrpcClient.ConsumeStreamAsync` classifies a stream's end as a fault (`onError`), a graceful server-side end (`onCompleted`), or our own teardown (neither). Only a cancellation **we asked for** takes the silent leg — the `RpcException(Cancelled)` filter is guarded by `cts.IsCancellationRequested`. A **foreign** `Cancelled` (the peer cancelled, or the channel was disposed underneath us) falls through to `onError`; unguarded, it fired none of the three, so the aggregator kept `_streamDown = false` — `IsLive` stuck true and the reconcile tick's reopen, which only runs on a stream known to be down, never fired.
 - **Options** (`Communication` section, `CommunicationOptions`; eagerly validated by `CommunicationOptionsValidator` / `ValidateOnStart`): `LiveAlarmCacheLinger` (default 30s), `LiveAlarmCacheReconcileInterval` (default 60s), `LiveAlarmCacheSeedConcurrency` (default 8), `LiveAlarmCacheMaxSubscribersPerSite` (default 200), `LiveAlarmCachePublishCoalesce` (default 250ms; `0` = publish per delta — legacy — batches an alarm storm into one snapshot copy + one viewer fan-out per window; arch review 02 round 2, N6), `LiveAlarmCacheReconcileJitterFraction` (default 0.2 = up to +20% per tick; `0` disables). Stream channel sizing lives alongside the other gRPC limits: `GrpcInstanceStreamChannelCapacity` (default 1000) and `GrpcSiteAlarmStreamChannelCapacity` (default 20 000).
 - **Telemetry** (`ScadaBridgeTelemetry` meter): observable gauge `scadabridge.site.alarm_cache.aggregators.active` (running per-site aggregators) and counter `scadabridge.site.alarm_cache.reconnects` (site-wide stream reconnects — a NodeA↔NodeB flip or reconcile-driven reopen; a sustained climb signals a flapping site link), and counter `scadabridge.site.alarm_cache.buffer_dropped` (deltas evicted from the aggregator's **bounded** 20 000-entry pre-seed buffer, drop-oldest — non-zero means a fan-out ran long enough for the delta storm behind it to exceed the cap; the fan-out's snapshot is authoritative for the evicted rows).
 - **Accepted limitations (arch review 02 round 2, N8):**