docs: tag-subscribe in-flight guard + backoff (Lever 1) — spec, CLAUDE.md, design-doc status

2026-08-28 14:53:23 -04:00
parent ab08ecd3d3
commit 150c3f60d8
3 changed files with 6 additions and 2 deletions
@@ -70,6 +70,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
 - Site-wide Akka stream for attribute value and alarm state changes with per-subscriber buffering.
 - Instance Actors serialize all state mutations (Akka actor model); concurrent scripts produce interleaved side effects.
 - Staggered Instance Actor startup on failover to prevent reconnection storms.
+- **Instance Actor tag-subscribe retry is guarded + backed off (Lever 1, 2026-08-28).** One outstanding `SubscribeTagsRequest` per connection until `TagSubscribeInFlightCeilingMs` (45 s); the interval doubles ±20 % jitter from `TagSubscribeRetryIntervalMs` (5 s) to `TagSubscribeRetryMaxIntervalMs` (60 s), resetting on Success or `ConnectionQualityChanged(Good)`. Pre-Lever-1 the actor re-sent a full N-tag subscribe every fixed 5 s with no in-flight check — under a slow MxGateway that stacked ~6 concurrent 244-tag subscribes per 30 s and wedged the session (QA 2026-08-26). Consequence to know: a *lost* reply is now re-sent after the ceiling (45 s), not the 5 s interval — a *failed* reply still retries at the backed-off interval. Levers 2 (bulk supervisory advise, cross-repo) and 3 (session `command_timeout`) are still design-only: `docs/plans/2026-08-28-mxgateway-subscribe-storm-prevention-design.md`.
 - Explicit supervision strategies: Resume for coordinator actors, Stop for short-lived execution actors.

 ### Data & Communication
@@ -1,7 +1,7 @@
 # MxGateway subscribe-storm prevention — design

 **Date:** 2026-08-28
-**Status:** Design (not yet implemented) — **reviewed against both repos 2026-08-28** (ScadaBridge `main` @ `a9b23556`, MxAccessGateway @ `74ba889`): no part of this design has landed in either repo; corrections from that review are folded in below (Lever 3 was re-scoped — see §5).
+**Status:** Design — **Lever 1 implemented** (plan `docs/plans/2026-08-28-tag-subscribe-inflight-guard-backoff.md`, branch commits 6f7c6ad8..ab08ecd3; Levers 2–3 not yet implemented) — **reviewed against both repos 2026-08-28** (ScadaBridge `main` @ `a9b23556`, MxAccessGateway @ `74ba889`): no part of this design has landed in either repo; corrections from that review are folded in below (Lever 3 was re-scoped — see §5).
 **Scope:** ScadaBridge (`SiteRuntime`, `DataConnectionLayer`) **and** MxAccessGateway (`~/Desktop/MxAccessGateway`: `Contracts` proto, `Worker`, `Server`)
 **Motivating incident:** 2026-08-26 QA (vq03/vq04) — Z28061's 244 attributes went Bad/frozen for ~20 min. Root cause was a **transient MxAccessGateway subscribe wedge under deploy/subscribe churn**, not ScadaBridge or Galaxy. No dedicated incident record exists for 2026-08-26 yet — the analysis lives in this doc. Related prior record: `docs/deployment/deployment-records/2026-08-20-wonder-app-vq03-vq04.md` §4, the earlier **ScadaBridge DCL** hotfix `da0bfe2b` on the same connection (MxGateway adapter treating a graceful event-stream end as a disconnect — a ScadaBridge commit, not a gateway one; the 2026-08-26 freeze is a *different* failure mode, see §1).

@@ -156,7 +156,7 @@ InstanceActor.SendTagSubscribe(conn)

 Levers are independent and independently shippable; recommended order:

-1. **Lever 1 (ScadaBridge-only).** Highest value / lowest blast radius — breaks the amplifier immediately; no gateway change. Ships and deploys as a ScadaBridge binary swap.
+1. **Lever 1 (ScadaBridge-only) — DONE 2026-08-28.** Highest value / lowest blast radius — breaks the amplifier immediately; no gateway change. Ships and deploys as a ScadaBridge binary swap.
 2. **Lever 2 (gateway first, then ScadaBridge).** Deploy the gateway side (new capability + worker flag) before ScadaBridge starts requesting it — capability negotiation makes the order safe either way, but gateway-first means the optimization lights up as soon as ScadaBridge ships. Both are additive/wire-compatible.
 3. **Lever 3 (ScadaBridge-only).** Fold in with Lever 1 or Lever 2's ScadaBridge change; pure config surface.

@@ -196,6 +196,9 @@ flowchart TD
 1. Load all attribute values from the flattened configuration (static defaults).
 2. Set quality to **uncertain** for all attributes that have a data source reference. Static attributes (no data source reference) have quality **good**. The uncertain quality persists until the first value update arrives from the Data Connection Layer, distinguishing "not yet received" from "known good" or "connection lost."
 3. Register data source references with the Data Connection Layer for subscriptions.
+
+   > **Tag-subscribe retry (S4/UA6, hardened by Lever 1 of the MxGateway subscribe-storm design, 2026-08-28).** The Instance Actor groups its data-sourced tag paths by connection and sends one `SubscribeTagsRequest` per connection, arming a per-connection watchdog *on send* so both a failed reply and a lost reply are retried. **At most one request per connection is outstanding:** a watchdog tick while the in-flight request is younger than `TagSubscribeInFlightCeilingMs` (45 s, sized above the gateway's 30 s command deadline) only re-arms — it never issues a duplicate — so a gateway that is slow to answer is not handed a fresh N-tag subscribe every interval on top of the one it is still processing (the amplifier behind the 2026-08-26 QA freeze). Past the ceiling the reply is presumed lost and a new request goes out; a late reply for the superseded id is ignored. The retry interval is exponential: floor `TagSubscribeRetryIntervalMs` (5 s), ×2 with ±20 % jitter per failed/lost attempt, capped at `TagSubscribeRetryMaxIntervalMs` (60 s); a `Success` reply or the connection returning to `Good` quality resets it to the floor. All three keys are startup-validated (`SiteRuntimeOptionsValidator`). Design: `docs/plans/2026-08-28-mxgateway-subscribe-storm-prevention-design.md` §3; plan: `docs/plans/2026-08-28-tag-subscribe-inflight-guard-backoff.md`.
+
 4. Create child Script Actors (one per script defined on the instance).
 5. Create child Alarm Actors (one per alarm defined on the instance).