docs: tag-subscribe in-flight guard + backoff (Lever 1) — spec, CLAUDE.md, design-doc status

docs/plans/2026-08-28-mxgateway-subscribe-storm-prevention-design.md

@@ -1,7 +1,7 @@
 # MxGateway subscribe-storm prevention — design
 
 **Date:** 2026-08-28
-**Status:** Design (not yet implemented) — **reviewed against both repos 2026-08-28** (ScadaBridge `main` @ `a9b23556`, MxAccessGateway @ `74ba889`): no part of this design has landed in either repo; corrections from that review are folded in below (Lever 3 was re-scoped — see §5).
+**Status:** Design — **Lever 1 implemented** (plan `docs/plans/2026-08-28-tag-subscribe-inflight-guard-backoff.md`, branch commits 6f7c6ad8..ab08ecd3; Levers 2–3 not yet implemented) — **reviewed against both repos 2026-08-28** (ScadaBridge `main` @ `a9b23556`, MxAccessGateway @ `74ba889`): no part of this design has landed in either repo; corrections from that review are folded in below (Lever 3 was re-scoped — see §5).
 **Scope:** ScadaBridge (`SiteRuntime`, `DataConnectionLayer`) **and** MxAccessGateway (`~/Desktop/MxAccessGateway`: `Contracts` proto, `Worker`, `Server`)
 **Motivating incident:** 2026-08-26 QA (vq03/vq04) — Z28061's 244 attributes went Bad/frozen for ~20 min. Root cause was a **transient MxAccessGateway subscribe wedge under deploy/subscribe churn**, not ScadaBridge or Galaxy. No dedicated incident record exists for 2026-08-26 yet — the analysis lives in this doc. Related prior record: `docs/deployment/deployment-records/2026-08-20-wonder-app-vq03-vq04.md` §4, the earlier **ScadaBridge DCL** hotfix `da0bfe2b` on the same connection (MxGateway adapter treating a graceful event-stream end as a disconnect — a ScadaBridge commit, not a gateway one; the 2026-08-26 freeze is a *different* failure mode, see §1).
 
@@ -156,7 +156,7 @@ InstanceActor.SendTagSubscribe(conn)
 
 Levers are independent and independently shippable; recommended order:
 
-1. **Lever 1 (ScadaBridge-only).** Highest value / lowest blast radius — breaks the amplifier immediately; no gateway change. Ships and deploys as a ScadaBridge binary swap.
+1. **Lever 1 (ScadaBridge-only) — DONE 2026-08-28.** Highest value / lowest blast radius — breaks the amplifier immediately; no gateway change. Ships and deploys as a ScadaBridge binary swap.
 2. **Lever 2 (gateway first, then ScadaBridge).** Deploy the gateway side (new capability + worker flag) before ScadaBridge starts requesting it — capability negotiation makes the order safe either way, but gateway-first means the optimization lights up as soon as ScadaBridge ships. Both are additive/wire-compatible.
 3. **Lever 3 (ScadaBridge-only).** Fold in with Lever 1 or Lever 2's ScadaBridge change; pure config surface.
 

docs/requirements/Component-SiteRuntime.md

@@ -196,6 +196,9 @@ flowchart TD
 1. Load all attribute values from the flattened configuration (static defaults).
 2. Set quality to **uncertain** for all attributes that have a data source reference. Static attributes (no data source reference) have quality **good**. The uncertain quality persists until the first value update arrives from the Data Connection Layer, distinguishing "not yet received" from "known good" or "connection lost."
 3. Register data source references with the Data Connection Layer for subscriptions.
+
+   > **Tag-subscribe retry (S4/UA6, hardened by Lever 1 of the MxGateway subscribe-storm design, 2026-08-28).** The Instance Actor groups its data-sourced tag paths by connection and sends one `SubscribeTagsRequest` per connection, arming a per-connection watchdog *on send* so both a failed reply and a lost reply are retried. **At most one request per connection is outstanding:** a watchdog tick while the in-flight request is younger than `TagSubscribeInFlightCeilingMs` (45 s, sized above the gateway's 30 s command deadline) only re-arms — it never issues a duplicate — so a gateway that is slow to answer is not handed a fresh N-tag subscribe every interval on top of the one it is still processing (the amplifier behind the 2026-08-26 QA freeze). Past the ceiling the reply is presumed lost and a new request goes out; a late reply for the superseded id is ignored. The retry interval is exponential: floor `TagSubscribeRetryIntervalMs` (5 s), ×2 with ±20 % jitter per failed/lost attempt, capped at `TagSubscribeRetryMaxIntervalMs` (60 s); a `Success` reply or the connection returning to `Good` quality resets it to the floor. All three keys are startup-validated (`SiteRuntimeOptionsValidator`). Design: `docs/plans/2026-08-28-mxgateway-subscribe-storm-prevention-design.md` §3; plan: `docs/plans/2026-08-28-tag-subscribe-inflight-guard-backoff.md`.
+
 4. Create child Script Actors (one per script defined on the instance).
 5. Create child Alarm Actors (one per alarm defined on the instance).