fix(alarms): dedup reconcile/live duplicate broadcasts on the alarm feed (NEXT-03)

A periodic reconcile can synthesize a repair transition whose matching live
transition is still buffered in the alarm lease; both then broadcast as
indistinguishable duplicates on StreamAlarms and the dashboard hub. Nothing
serializes the two paths, and a correct serialization needs a worker-side
high-water mark on QueryActiveAlarms (proto + worker change + a stall path),
so this closes the common case with a local best-effort dedup instead: both
paths already carry the same worker-derived identity — the worker stamps
record.TransitionTimestampUtc into both OnAlarmTransitionEvent's
transition_timestamp and ActiveAlarmSnapshot.last_transition_timestamp — so
ApplyTransition suppresses a live transition whose (timestamp, resulting
state) the cache already carries from a repair. The Clear leg has no cache
entry left to compare, so ApplyReconcile tombstones each synthesized Clear by
the instance's original_raise_timestamp for one reconcile generation; a
matching live Clear consumes the tombstone, while a new raise/clear cycle
carries a newer raise timestamp and passes. Suppression fires only on a
positive marker match — unset timestamps keep today's behavior — so the
documented at-least-once consumer contract stands (gateway.md, Sessions.md
updated in the same change).

Tests: two new regressions drive the exact race through the GWC-26 harness
(repair-then-buffered-live for Raise and for Clear, each with a genuine
follow-up transition proving no over-suppression); GatewayAlarmMonitor
suites 18/18.
This commit is contained in:
Joseph Doherty
2026-08-10 06:06:12 -04:00
parent 8624e21372
commit 2c03e0a684
4 changed files with 232 additions and 11 deletions
+1 -1
View File
@@ -201,7 +201,7 @@ The single worker event channel has exactly one direct reader: the `SessionEvent
The monitor takes that lease **before** it issues `SubscribeAlarms`, so no transition window is missed: the pump has been running since `MarkReady` started the dashboard mirror, and the distributor only fans to subscribers registered at fan-out time, so a lease taken after the subscribe + first-reconcile round trips would lose every transition raised inside that window. Transitions arriving while the monitor subscribes and reconciles buffer in the lease's bounded channel and are applied after the reconcile, which is order-safe because a live transition can update an alarm the reconciled snapshot already holds.
The repair transitions the monitor's reconcile broadcasts on the alarm feed (Raise/Clear presence deltas and the acked-state delta) are **at-least-once**: a reconcile reads the worker's current state while the matching live transition may still be buffered in the lease, so both can be broadcast and the duplicates are indistinguishable — alarm-feed consumers must apply transitions idempotently. This is a property of the reconcile design, not of the buffering above.
The repair transitions the monitor's reconcile broadcasts on the alarm feed (Raise/Clear presence deltas and the acked-state delta) are **at-least-once**: a reconcile reads the worker's current state while the matching live transition may still be buffered in the lease, so both can be broadcast and the duplicates are indistinguishable — alarm-feed consumers must apply transitions idempotently. This is a property of the reconcile design, not of the buffering above. The monitor dedups the common case best-effort (NEXT-03): a buffered live transition that positively matches the cache's worker timestamp and resulting state — or, for Clear, a tombstone keyed on the cleared instance's original raise timestamp — was already broadcast as a repair and is suppressed; unset timestamps never suppress, so the consumer contract is unchanged.
`AttachInternalEventSubscriber` enforces the same readiness gate as `AttachEventSubscriber` — a session (or worker) that is not `Ready` throws `SessionNotReady` *before* the distributor is constructed. A premature attach would otherwise start the pump against a source that throws, completing every subscriber with that error and latching the distributor for the session's whole lifetime.