fix(comms): review findings — consumer-based debug orphan net, foreign-cancel triad, honest onConnected, served-row-exact retirement, full-rate reconcile

F1 (HIGH) DebugStreamBridgeActor: the 5-minute orphan net measured the MAILBOX
(SetReceiveTimeout), and once stream events were correctly marked
INotInfluenceReceiveTimeout nothing recurring reset it — the snapshot lands once
and GrpcStreamStable once — so every healthy session self-terminated at ~6 min
with a false "Site disconnected". Replaced with a periodic self-tick
(ConsumerLivenessCheckInterval, 30s) over a consumer-last-seen stamp renewed only
by DebugStreamConsumerAlive, which DebugStreamService Tells on a shared timer to
every session still in its registry (holding a session there IS "a consumer is
attached" — both the Blazor view and the SignalR hub release it on
dispose/disconnect, and it works headless). Reverting the wrapper was rejected: it
would restore the quiet-instance orphan bug.

F2 (MED) SiteStreamGrpcClient: the RpcException(Cancelled) filter now requires
cts.IsCancellationRequested. A peer-originated / channel-dispose Cancelled fired
none of onError/onCompleted/onConnected, leaving SiteAlarmAggregatorActor with
_streamDown=false forever (IsLive stuck true, reconcile reopen guard never fired).

F3 (MED) SiteStreamGrpcClient: a header TIMEOUT is no longer reported as
connected — that shape is exactly what an unreachable site produces, and it
cleared _streamDown, consumed _seedOnConnect and launched a full snapshot fan-out
at a dead site. AwaitHeadersAsync returns bool; the first received event is the
fallback connected signal, fired at most once from headers OR first event.

F4 (LOW-MED) SqliteAuditWriter.MarkReconciledUpToAsync: the blanket below-cursor
UPDATE retired late-stamped inserts that were never served (then age-purged —
silent loss). The flip is now bounded by insertion order: a Pending row retires
only if its rowid is at or below the high-water mark of rows this instance has
served from ReadPendingSinceAsync (clamped on purge, since SQLite reuses rowids);
Forwarded rows are exempt (central ACKed them over the push path). At-least-once
is unchanged.

F5 (LOW) Documented the liveness dependency (a served row never covered by a later
cursor stays Pending forever; PurgeExpiredAsync never purges Pending) in
ISiteAuditQueue + Component-AuditLog.md, and added a cheap site-health signal:
SiteAuditBacklogReporter logs a rate-limited warning when the existing
oldest-pending metric exceeds 24h.

F6 (MED) SiteAlarmAggregatorActor: _fanoutSinceLastTick was armed by the
reconcile's OWN fan-out, so steady state ran fan-out→skip→fan-out→skip — one
reconcile per 2x interval (120s), halving the not-reporting refresh and the alarm
reconcile backstop. The skip is now armed only by connect/failover-driven seeds
(initial, _seedOnConnect, and a re-seed queued behind one).

Tests: Communication.Tests 691 passed (+13), AuditLog.Tests 382 passed (+5).
This commit is contained in:
Joseph Doherty
2026-08-14 23:52:25 -04:00
parent b1de9dfdd4
commit fd5e023d08
16 changed files with 1192 additions and 84 deletions
+41 -1
View File
@@ -261,7 +261,15 @@ room is a compliance violation, not a self-healing behavior. To bound that
growth in practice, the site emits a `SiteAuditBacklog` health metric (pending
row count, oldest pending age, bytes on disk); crossing operator-configured
thresholds surfaces a warning on the relevant site tile in the Health
dashboard, mirroring the Store-and-Forward Engine's backlog metric.
dashboard, mirroring the Store-and-Forward Engine's backlog metric, and
`SiteAuditBacklogReporter` additionally logs a rate-limited warning once the
oldest pending row passes 24 h.
The same invariant carries a liveness dependency worth stating explicitly: the
`Pending` floor clears only when central acknowledges the rows — a telemetry ack
or a reconciliation cursor that covers them — never on age. See *Reconciliation
pull* below for the served-row retirement rule that decides which rows can be
acknowledged by a cursor at all.
Central is the durable home. Site SQLite is a write-buffer with a forwarding
guarantee.
@@ -367,6 +375,38 @@ it — a lagging drain is meant to surface as the stalled signal. The id tiebrea
what makes that safe against a same-instant burst larger than one batch: the cursor
advances on every tick even when the timestamp cannot.
**Only rows that were actually served may retire.** `OccurredAtUtc` is stamped by the
caller, so a row can be *inserted* after a batch was served yet carry a timestamp
*below* central's (by then advanced) cursor — a back-dated stamp, a clock nudge, a
write flushed late. A cursor flip that keyed on the timestamp alone retired exactly
those rows: never served, never servable again (the keyset read has moved past them),
and, being `Reconciled`, purged on age. That is silent audit loss, and it is a failure
mode the pre-WP2.3 explicit id-set flip could not produce, so `MarkReconciledUpToAsync`
carries a second, insertion-order bound:
> A `Pending` row retires only if its insertion order is at or below the high-water
> mark of rows this site node has actually served from `ReadPendingSinceAsync`
> (SQLite `rowid`). A `Forwarded` row is exempt — central ACKED it over the telemetry
> push path, which is proof independent of the pull.
The bound is per-process, so after a site-node restart the first pull retires only
`Forwarded` rows and the pull after it resumes normal retirement — conservative in the
safe direction (retirement is delayed, never a row lost). The gRPC handler's ordering
(retire, *then* read) is what keeps the bound from ever vouching for the batch it is
about to serve.
**Liveness note (accepted, documented).** A row that was served but never covered by a
later cursor — central reconciliation stopped for good, or the bound reset over a
restart before the next cursor arrived — stays `Pending` indefinitely, and the site
retention purge never purges `Pending`. That is the hard `ForwardState` invariant
working as intended (an unacknowledged row is not droppable), but it means the site
store's floor depends on reconciliation actually running, not merely on the retention
window elapsing. It is observable rather than silent: `GetBacklogStatsAsync` reports
the pending count and oldest-pending instant on every site health report
(`SiteAuditBacklog`), and `SiteAuditBacklogReporter` logs a rate-limited warning once
the oldest pending row exceeds `StalePendingThreshold` (24 h) naming the drain and the
reconciliation pull as the things to check.
### Central direct-write (central-originated events)
Events originating at central never touch site SQLite. Inbound API writes one
+4 -2
View File
@@ -70,7 +70,8 @@ Both central and site clusters. Each side has communication actors that handle m
- **Bridge-session hardening (WP2.3):**
- The pre-snapshot buffer is **bounded (20 000 events, drop-oldest)** and its evictions counted (`scadabridge.central.debug_view.presnapshot_dropped`). It was previously unbounded, so a session whose snapshot never arrived grew without limit on the CENTRAL node. Dropping the oldest is correct here: the snapshot that ends the buffering phase is authoritative for anything that old.
- A **hard snapshot deadline** (`DebugStreamBridgeActor.SnapshotTimeout`, 60 s) fails the session if no `DebugViewSnapshot` arrives. Nothing else ended a session wedged in the buffering phase — a lost site reply raises no gRPC error.
- **Stream events no longer influence the orphan receive timeout.** The gRPC callback wraps each event in an envelope marked `INotInfluenceReceiveTimeout`, so a busy site can no longer keep an abandoned session alive forever by feeding it events. The 5-minute timeout now measures session/consumer liveness, which is what it was for.
- **The orphan net measures the CONSUMER, not the mailbox.** A session self-terminates after `DebugStreamBridgeActor.ConsumerIdleTimeout` (5 min) without a sign of life from its consumer, checked by a periodic self-tick (`ConsumerLivenessCheckInterval`, 30 s) against a consumer-last-seen stamp. The stamp is renewed only by `DebugStreamConsumerAlive`, which `DebugStreamService` Tells to every session still in its registry on a shared 30 s timer — holding a session there IS what "a consumer is attached" means, since both consumers (the Blazor debug view and the SignalR hub) release it on dispose/disconnect.
This replaced an Akka `SetReceiveTimeout(5 min)`. That version measured the MAILBOX, which conflates site chatter with consumer liveness: a busy site kept an abandoned session alive forever. Marking stream events `INotInfluenceReceiveTimeout` fixed that but left the timeout with nothing recurring to reset it — the snapshot lands once and the stability tick once — so **every healthy session self-terminated ~6 min in and the operator was told "Site disconnected"**. Both failure directions are pinned by tests: streaming events with keepalives survive many windows; an orphaned session terminates while events keep arriving.
### 6.1 Aggregated Live Alarm Stream (Site → Central)
@@ -80,7 +81,8 @@ Delivered 2026-07-10 (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.m
- **Central live cache** (`ISiteAlarmLiveCache`, singleton `SiteAlarmLiveCacheService`): a DI singleton on the active central node. For each site with ≥1 active viewer it runs ONE shared, **reference-counted** per-site aggregator (`SiteAlarmAggregatorActor`); the first `Subscribe(siteId, onChanged)` starts it, the last subscriber leaving stops it after a short **linger** to avoid re-seed thrash. `GetCurrentAlarms(siteId)` returns the current immutable snapshot; `IsLive(siteId)` reports whether the aggregator has seeded **and its site-wide stream is currently up**. Liveness rides every publish from the aggregator, so a stream that faults or ends gracefully drops `IsLive` immediately rather than leaving the page grafting a freezing snapshot over fresh poll data until the next reconcile (WP2.3).
- **Seed-then-stream** (copied from `DebugStreamBridgeActor` ordering): open the `SubscribeSite` stream first (buffer live deltas), run the snapshot fan-out once via the existing `DebugViewSnapshot` path (bounded by `LiveAlarmCacheSeedConcurrency`), flush the buffer with **dedup by `(InstanceUniqueName, AlarmName, SourceReference)`**, then live pass-through. Placeholders are seeded from the snapshot and never expected on the live stream.
- **Alarms-only seed (WP2.3)**: the seed/reconcile fan-out sets `DebugSnapshotRequest.AlarmsOnly` (wire: `DebugSnapshotRequestDto.alarms_only`, field 3, additive), so the site builds and ships only the alarm half of the snapshot. The fan-out discarded every attribute row anyway, and an instance's attribute surface dwarfs its alarm set. A pre-WP2.3 site ignores the flag and returns the full snapshot, which reads identically.
- **Failover & drift**: a re-seed runs **once per successful (re)connect**, not once per reconnect ATTEMPT — the connect signal is `SiteStreamGrpcClient.SubscribeSiteAsync`'s `onConnected` callback, raised when the site's response headers arrive (the site flushes them as soon as its relay is attached, so no event can be missed after it). Fanning a whole-site snapshot out per retry meant N snapshots against a site that was, by definition of the retry, unreachable. A periodic **reconcile snapshot** (default 60s, **jittered** by `LiveAlarmCacheReconcileJitterFraction` so aggregators started together do not stampede one boundary) remains the drift backstop, but it is **skipped when a fan-out already ran in that window** and **publishes only when the snapshot actually changed the cache** (a diff, not an unconditional viewer fan-out). Staleness stays bounded at two intervals: the skip consumes its flag, so the next tick always fans out. A fan-out that fails as a whole now retries on its own **backoff** timer (`reconnectDelay` doubling, capped at 8× the reconcile interval) instead of waiting a full interval. `[PERM]` (`docs/plans/2026-05-29-native-alarms-design.md`): the cache is **purely in-memory** — no EF entity/table/migration, no persisted central alarm store — so a new active node simply re-seeds from scratch.
- **Failover & drift**: a re-seed runs **once per successful (re)connect**, not once per reconnect ATTEMPT — the connect signal is `SiteStreamGrpcClient.SubscribeSiteAsync`'s `onConnected` callback, raised when the site's response headers arrive (the site flushes them as soon as its relay is attached, so no event can be missed after it) or, for a peer that defers its headers, when the FIRST EVENT arrives — whichever comes first, and at most once. A header **timeout** is deliberately not a connect signal: an unreachable or wedged site produces exactly that shape, and reporting it as connected cleared `_streamDown`, consumed the pending re-seed and fanned a whole-site snapshot out at a site that never answered. Fanning a whole-site snapshot out per retry meant N snapshots against a site that was, by definition of the retry, unreachable. A periodic **reconcile snapshot** (default 60s, **jittered** by `LiveAlarmCacheReconcileJitterFraction` so aggregators started together do not stampede one boundary) remains the drift backstop, but it is **skipped when a CONNECT- OR FAILOVER-DRIVEN seed already ran in that window** and **publishes only when the snapshot actually changed the cache** (a diff, not an unconditional viewer fan-out). Staleness stays bounded at two intervals: the skip consumes its flag, so the next tick always fans out. A **tick's own fan-out never arms that skip** — when it did, steady state alternated fan-out/skip and the effective reconcile rate was one per TWO intervals (120 s by default), halving both the not-reporting refresh and the alarm reconcile backstop for no benefit: with no reconnect in play there is nothing duplicated to suppress. A fan-out that fails as a whole now retries on its own **backoff** timer (`reconnectDelay` doubling, capped at 8× the reconcile interval) instead of waiting a full interval. `[PERM]` (`docs/plans/2026-05-29-native-alarms-design.md`): the cache is **purely in-memory** — no EF entity/table/migration, no persisted central alarm store — so a new active node simply re-seeds from scratch.
- **Stream terminations are a triad, and every ending hits exactly one leg.** `SiteStreamGrpcClient.ConsumeStreamAsync` classifies a stream's end as a fault (`onError`), a graceful server-side end (`onCompleted`), or our own teardown (neither). Only a cancellation **we asked for** takes the silent leg — the `RpcException(Cancelled)` filter is guarded by `cts.IsCancellationRequested`. A **foreign** `Cancelled` (the peer cancelled, or the channel was disposed underneath us) falls through to `onError`; unguarded, it fired none of the three, so the aggregator kept `_streamDown = false``IsLive` stuck true and the reconcile tick's reopen, which only runs on a stream known to be down, never fired.
- **Options** (`Communication` section, `CommunicationOptions`; eagerly validated by `CommunicationOptionsValidator` / `ValidateOnStart`): `LiveAlarmCacheLinger` (default 30s), `LiveAlarmCacheReconcileInterval` (default 60s), `LiveAlarmCacheSeedConcurrency` (default 8), `LiveAlarmCacheMaxSubscribersPerSite` (default 200), `LiveAlarmCachePublishCoalesce` (default 250ms; `0` = publish per delta — legacy — batches an alarm storm into one snapshot copy + one viewer fan-out per window; arch review 02 round 2, N6), `LiveAlarmCacheReconcileJitterFraction` (default 0.2 = up to +20% per tick; `0` disables). Stream channel sizing lives alongside the other gRPC limits: `GrpcInstanceStreamChannelCapacity` (default 1000) and `GrpcSiteAlarmStreamChannelCapacity` (default 20 000).
- **Telemetry** (`ScadaBridgeTelemetry` meter): observable gauge `scadabridge.site.alarm_cache.aggregators.active` (running per-site aggregators) and counter `scadabridge.site.alarm_cache.reconnects` (site-wide stream reconnects — a NodeA↔NodeB flip or reconcile-driven reopen; a sustained climb signals a flapping site link), and counter `scadabridge.site.alarm_cache.buffer_dropped` (deltas evicted from the aggregator's **bounded** 20 000-entry pre-seed buffer, drop-oldest — non-zero means a fan-out ran long enough for the delta storm behind it to exceed the cap; the fan-out's snapshot is authoritative for the evicted rows).
- **Accepted limitations (arch review 02 round 2, N8):**