ScadaBridge/docs/plans/2026-08-28-mxgateway-subscribe-storm-prevention-design.md at 17d9a4cd2cb7d5d84c49e559100854184e379885

Files

T

Joseph Doherty afb5180ce0 docs+test(site-runtime): Lever 1 final-review polish — document that the Good reset does not release an in-flight request and that the guard is inert once interval ≥ ceiling; FlappingGood asserts the retry landed during the burst instead of timing Thread.Sleep; NativeAlarmActor (same unguarded amplifier) + cancel-race residue recorded as follow-ups

Claude-Session: https://claude.ai/code/session_019ZxXT5sxZs5XrtdwYzJDPV

2026-08-28 15:10:21 -04:00

22 KiB

Raw Blame History

Date: 2026-08-28 Status: Design — Lever 1 implemented (plan docs/plans/2026-08-28-tag-subscribe-inflight-guard-backoff.md, branch commits 6f7c6ad8..ab08ecd3; Levers 2–3 not yet implemented) — reviewed against both repos 2026-08-28 (ScadaBridge main @ a9b23556, MxAccessGateway @ 74ba889): no part of this design has landed in either repo; corrections from that review are folded in below (Lever 3 was re-scoped — see §5). Scope: ScadaBridge (SiteRuntime, DataConnectionLayer) and MxAccessGateway (~/Desktop/MxAccessGateway: Contracts proto, Worker, Server) Motivating incident: 2026-08-26 QA (vq03/vq04) — Z28061's 244 attributes went Bad/frozen for ~20 min. Root cause was a transient MxAccessGateway subscribe wedge under deploy/subscribe churn, not ScadaBridge or Galaxy. No dedicated incident record exists for 2026-08-26 yet — the analysis lives in this doc. Related prior record: docs/deployment/deployment-records/2026-08-20-wonder-app-vq03-vq04.md §4, the earlier ScadaBridge DCL hotfix da0bfe2b on the same connection (MxGateway adapter treating a graceful event-stream end as a disconnect — a ScadaBridge commit, not a gateway one; the 2026-08-26 freeze is a different failure mode, see §1).

1. Problem statement

Two independent defects compound into a self-sustaining freeze:

Amplifier (ScadaBridge). InstanceActor.SendTagSubscribe arms a fixed 5 s retry timer (SiteRuntimeOptions.TagSubscribeRetryIntervalMs = 5000) on send, with no in-flight guard and no backoff. When a SubscribeBulk round-trip takes longer than 5 s — which it does the moment the gateway is under load — a new full 244-tag subscribe is issued every 5 s while the previous one is still outstanding. With two instances (Z28061 + Z28062) this stacks ~6 concurrent 244-tag subscribes per 30 s window.
- src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs:1234 (SendTagSubscribe, called from SubscribeToDcl :1224 and the retry handler), :1250-1254 (timer arm — the only "guard" is that re-arming overwrites the previous per-connection timer; nothing checks _pendingTagSubscribes before issuing), :1264 (stale-correlation drop), :1269 (cancel-on-success), :314 (RetryTagSubscribe handler), :391-394 (PostStop cancels timers). State fields _tagsByConnection/_pendingTagSubscribes/_tagSubscribeRetryTimers at :185-188. SiteRuntimeOptions.cs:120 (TagSubscribeRetryIntervalMs = 5000, doc comment explicitly says "fixed interval"); SiteRuntimeOptionsValidator.cs:77-80 checks only > 0.
Load (cross-repo). In supervisory mode (WriteUserId == 0, how QA runs), one ScadaBridge subscribe = 1 AddItemBulk RPC + ~244 per-item AdviseSupervisory RPCs (pipelined ×16). Each AdviseSupervisory is a separate STA command on the worker's single-threaded apartment. The gateway's per-command deadline is 30 s (MxGateway:Sessions:DefaultCommandTimeoutSeconds), so under contention these bulk/per-item commands hit the deadline, cascade DeadlineExceeded, and no advise is ever established → no OnDataChange reaches ScadaBridge → all tags Bad.
- ScadaBridge: src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/RealMxGatewayClient.cs:108-160 (SubscribeBulkAsync, supervisory branch).
- Gateway (canonical proto src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_gateway.proto; the copies under clients/ and build/ are generated mirrors): SubscribeBulkCommand (:325-328, server_handle=1, tag_addresses=2, no field 3) does AddItem + plain Advise per tag in one STA pass — Worker/MxAccess/MxAccessSession.cs:628-652 (SubscribeBulk, with AppendRemoveItemCleanup rollback), Worker/MxAccess/MxAccessCommandExecutor.cs:646 (ExecuteSubscribeBulk). The only supervisory entry point is the per-item AdviseSupervisoryCommand (proto :220-223, oneof 17; MxAccessSession.AdviseSupervisory :319-331). Note an AdviseItemBulkCommand (proto :312, oneof 29, repeated int32 item_handles) already exists for the plain kind — there is simply no supervisory counterpart, which is why ScadaBridge fans out per item. The STA really is one command at a time: Worker/Sta/StaRuntime.cs:58-63 (single STA thread), StaCommandDispatcher.DrainAsync :233-268 (awaits each command fully before dequeuing the next).

Proof it is not MXAccess/Galaxy: a fresh LMXProxy 32-bit STA probe did AddItem + AdviseSupervisory for every sampled tag in 0–5 ms while both worker processes sat < 5 % CPU. The bottleneck is command-path fan-out and retry stacking, not the OT layer.

Goal / non-goals

Goal: a subscribe or reconnect storm on one instance can no longer wedge the gateway session or freeze data; recovery is prompt and self-correcting.
Non-goal: changing the alarm/event streaming path (SiteStreamService, da0bfe2b), the write path, or anything about Galaxy/MXAccess itself.
Non-goal: all-or-nothing subscribe semantics; per-tag partial success is preserved.

2. Approach (chosen: full fix)

Three levers were considered; the full fix combines the two that address root cause, and folds in the third as a bounded safety knob:

Lever	What	Repo	Role
1 — retry backoff + in-flight guard	Stop re-sending a subscribe while one is in flight; exponential backoff with jitter on genuine failure	ScadaBridge	Removes the amplifier
2 — bulk supervisory-advise RPC	`SubscribeBulkCommand.supervisory` flag → worker does AddItem + `AdviseSupervisory` in one STA pass; collapses ~244 RPCs → 1	Cross-repo, capability-negotiated	Removes the load
3 — session command timeout	ScadaBridge sets an explicit `command_timeout` at `OpenSession` (the gateway's only existing override — it is per-session, not per-command; see §5)	ScadaBridge	Bounded safety margin

Levers 1 and 2 are each sufficient to break the feedback loop; together they are defense-in-depth (1 caps concurrency, 2 cuts the per-subscribe cost by ~244×). Lever 3 is a cheap knob that stops treating a merely-slow (but healthy) bulk op as a failure — with the caveat, found in review, that the gateway has no per-command override, so the knob is session-wide.

Rejected as standalone: raising DefaultCommandTimeoutSeconds globally (band-aid — lets the queue back up longer); an unnegotiated proto bump for Lever 2 (an old gateway would silently do a plain advise, breaking writes on a WriteUserId==0 connection).

3. Lever 1 — ScadaBridge retry backoff + in-flight guard

Where: InstanceActor (SiteRuntime). Per-connection subscribe state already lives here (_tagSubscribeRetryTimers, _pendingTagSubscribes).

Change:

Track an in-flight flag per connection. SendTagSubscribe becomes: if a subscribe for this connection is already outstanding and not past a hard "lost reply" ceiling, do not issue a new one — just re-arm the watchdog. This alone stops the 5 s stacking.
Replace the fixed 5 s re-arm with exponential backoff + jitter: floor TagSubscribeRetryIntervalMs (keep 5 s default), doubling to a new TagSubscribeRetryMaxIntervalMs (default 60 s), reset to the floor on a Success reply. Mirrors the existing DCL pattern for unresolved tags (DataConnectionOptions.cs:17,24 TagResolutionRetryInterval 10 s / TagResolutionRetryMaxInterval 5 min, validated floor > 0 and max >= floor at DataConnectionOptionsValidator.cs:21-22,39-41; machinery in DataConnectionActor.cs — ScheduleTagResolutionRetry :2393-2405 with its IsTimerActive guard, ResetTagResolutionBackoff :2409, BackOffTagResolution :2417-2419, pure step NextTagResolutionInterval ~:2422) — reuse the shape for consistency.
Lost-reply safety: because a reply can be genuinely lost, the in-flight guard must expire. Keep a TagSubscribeInFlightCeilingMs (default = current gateway command timeout + margin, e.g. 45 s) after which a new subscribe is allowed even if the old correlation never answered. This preserves today's "retry fires whether the response fails or is lost" guarantee without the 5 s stacking.

New options (SiteRuntimeOptions + validator):

TagSubscribeRetryMaxIntervalMs (default 60000; validate >= TagSubscribeRetryIntervalMs).
TagSubscribeInFlightCeilingMs (default 45000; validate > 0).

Invariants preserved: a superseded correlation id is still ignored (HandleSubscribeTagsResponse :1262-1264 stale-reply drop); a Success still cancels the timer (:1269) and resets backoff; PostStop still cancels all timers (:391-394).

Live gate PASS 2026-08-28 (docker rig, main @ 0589486f). OPC UA sim stopped and the instance's site node restarted so the boot-time subscribe could not be answered (failover landed rig-probe-1 on site-a-b): the log shows exactly one outstanding SubscribeTagsRequest per in-flight ceiling — unanswered for 45091ms … presuming lost, re-sending, then 47130ms — with the backoff climbing next retry in 00:00:05 → 00:00:09.415 → 00:00:20.369 (jittered), i.e. 3 requests in ~110 s where the pre-Lever-1 actor would have issued ~22. Sim restarted → DCL Entering Connected state in 4 s → all three attributes Good with fresh timestamps within ~95 s (debug snapshot --id 1492). Steady state on a healthy node: one subscribing line per (instance, connection), no still in flight / presuming lost at all. Note for future gates: an OPC UA outage alone never reaches this path — the DCL reconnects and re-subscribes transparently and the Instance Actor's subscribe stays confirmed; you must make the subscribe itself go unanswered (node restart / redeploy while the source is down).

4. Lever 2 — bulk supervisory-advise RPC (capability-negotiated)

Contract (mxaccess_gateway.proto), additive only:

message SubscribeBulkCommand {
  int32 server_handle = 1;
  repeated string tag_addresses = 2;
  bool supervisory = 3;   // NEW — advise supervisory instead of plain advise
}

Field 3 is unused today (verified); additive, field numbers never reused (house rule). Regen is the manual vendored-proto toggle both repos already use. Alternative considered: a new AdviseSupervisoryItemBulkCommand mirroring the existing plain AdviseItemBulkCommand (oneof 29) — rejected because it keeps the two-RPC AddItemBulk+AdviseBulk shape and two STA commands; the flag gives one RPC / one STA pass and reuses SubscribeBulkReply unchanged.

Capability negotiation (the back-compat guarantee): the gateway already advertises a capability list on OpenSessionReply.capabilities (field 5) — today exactly nine strings: unary-open-session, unary-close-session, unary-invoke, server-stream-events, bulk-subscribe-commands, bulk-read-commands, bulk-write-commands, unary-acknowledge-alarm, server-stream-active-alarms (Server/Grpc/MxAccessGatewayService.cs:59-67; bulk-subscribe-supervisory does not exist anywhere yet). The reply also carries gateway_protocol_version (field 8) — a capability string is still the right gate (feature-keyed, not version-keyed). Add:

reply.Capabilities.Add("bulk-subscribe-supervisory");

ScadaBridge reads the capability set at session open and only sets supervisory = true when bulk-subscribe-supervisory is present. Against an older gateway the capability is absent → ScadaBridge keeps today's AddItemBulk + per-item AdviseSupervisory path. This is the correctness gate: a plain advise on a WriteUserId==0 connection would silently break writes, so we must never send supervisory=true blind.

ScadaBridge does not currently retain the capability list in the DCL adapter — verified: RealMxGatewayClient.ConnectAsync (:83-84) does OpenSessionAsync then RegisterAsync and no Capabilities reference exists anywhere in DataConnectionLayer. This design adds: capture OpenSessionReply.capabilities into the RealMxGatewayClient at open, expose a SupportsSupervisorySubscribeBulk predicate.

Worker (Worker/MxAccess/…):

MxAccessSession.SubscribeBulk gains a supervisory parameter; when true it calls the MXAccess AdviseSupervisory (and registers MxAccessAdviceKind.Supervisory, exactly as the per-item path at :319-331 does) instead of plain Advise at :651-652 inside the same AddItem+advise loop. Per-item try/catch and AppendRemoveItemCleanup rollback are unchanged (partial success preserved). Long bulk commands hold the STA — keep the existing StaRuntime.PumpPendingMessages() (:94) behaviour the plain bulk path relies on.
ExecuteSubscribeBulk passes subscribeBulkCommand.Supervisory through (MxAccessCommandExecutor.cs:646). Reply shape (SubscribeBulkReply, per-item handles) is unchanged.

ScadaBridge (RealMxGatewayClient.SubscribeBulkAsync, :108):

Supervisory branch (UseSupervisoryAdvise :53, _writeUserId == 0): if the gateway advertises the capability, issue one SubscribeBulkAsync(_serverHandle, tagPaths, supervisory: true, ct) instead of AddItemBulkAsync (:128) + the BulkPipeline.RunAsync (:136-146) of per-item EnsureSupervisoryAdvisedAsync (:142, method :591; parallelism MxSupervisoryAdviseParallelism = 16, DataConnectionOptions.cs:73). Result mapping (MapSubscribeResults, handle bookkeeping, failed-item handle eviction) reuses the existing plain-mode path.
Fallback branch unchanged. Delete the now-superseded "Cross-repo follow-up" comment (RealMxGatewayClient.cs:126-127) since this design fulfills it.
Plain-advise mode (a configured write user) already used single-RPC SubscribeBulkAsync and is untouched.

Net effect: supervisory subscribe drops from 1 + N RPCs and N STA commands to 1 RPC / 1 STA command for N tags.

5. Lever 3 — explicit session command timeout (re-scoped in review)

Correction from the 2026-08-28 review: the gateway has no per-command timeout override. MxCommandRequest (proto :89-93) carries only session_id, client_correlation_id, command, and MxAccessGrpcRequestValidator.ValidateInvoke (:38-54) has no timeout handling. The only override is per session: OpenSessionRequest.command_timeout (field 4, google.protobuf.Duration), validated > 0 (MxAccessGrpcRequestValidator.cs:15), resolved by SessionManager.ResolveCommandTimeout (:583, called from :528) to MxGateway:Sessions:DefaultCommandTimeoutSeconds (default 30, SessionOptions.cs:8) when absent, then captured on GatewaySession.CommandTimeout (:223,293) and applied to every invoke on that session (:1151). The reply echoes the resolved value as OpenSessionReply.default_command_timeout (field 6). ScadaBridge's DCL never sets it and never reads it back (verified — no CommandTimeout anywhere in DataConnectionLayer; only _readTimeoutMs for reads).

Two ways to get a bulk-sized margin; (a) is chosen:

(a) Session-scoped, ScadaBridge-only (chosen). New DataConnectionOptions.MxCommandTimeout (default 60 s, validated > 0), sent as OpenSessionRequest.command_timeout in RealMxGatewayClient.ConnectAsync. Trade: it governs every command on the session, writes included — a wedged write would take up to 60 s instead of 30 s to fault. Acceptable: ScadaBridge's own WriteTimeout still bounds the script-facing write, and the gateway deadline is the backstop, not the primary. Log the echoed default_command_timeout at open so a gateway that clamps or ignores the request is visible.
(b) Per-invoke override (cross-repo, deferred). Additive google.protobuf.Duration command_timeout = 4 on MxCommandRequest, honoured in GatewaySession invoke resolution, capability-advertised (per-command-timeout). Cleaner but a second wire change for a margin that Lever 2 makes mostly moot (a single-pass bulk subscribe completes well under a second). Revisit only if (a)'s write-side widening proves a problem.

This remains a margin, not a fix — it only stops a merely-slow-but-alive gateway from being treated as failed and re-storming.

6. Data flow (after)

InstanceActor.SendTagSubscribe(conn)
  └─ in-flight guard: skip if a subscribe for conn is outstanding & within ceiling
  └─ else: 1 SubscribeTagsRequest ─▶ DataConnectionActor
        └─ RealMxGatewayClient.SubscribeBulkAsync(tagPaths)
              ├─ gateway advertises bulk-subscribe-supervisory?
              │     yes → 1 RPC SubscribeBulk(supervisory=true)   [session CommandTimeout = MxCommandTimeout, set at OpenSession]
              │             └─ worker: AddItem + AdviseSupervisory per tag, ONE STA pass
              │     no  → legacy AddItemBulk + N×AdviseSupervisory (fallback)
              └─ per-tag results ─▶ DCL resolves handles ─▶ OnDataChange flows
  └─ Success reply → cancel timer, reset backoff to floor
  └─ Failure/lost → backoff × jitter, re-arm (bounded by max interval)

7. Error handling & edge cases

Old gateway / capability absent: ScadaBridge falls back to today's path; no behavior change, writes stay correct.
Partial subscribe failure: unchanged — per-tag success/failure preserved end to end; failed items evict their handle mapping so a later retry re-adds cleanly.
Lost subscribe reply: in-flight ceiling releases the guard after TagSubscribeInFlightCeilingMs; a fresh subscribe then issues. No permanent stall.
Gateway restart mid-subscribe (the incident trigger): connection-level fault path (DataConnectionActor reconnect state machine) is unchanged; on reconnect the first subscribe issues immediately (backoff reset), and with the in-flight guard the reconnect storm cannot stack.
Reconnect + backoff interaction: a connection restore must reset backoff to the floor so recovery is prompt (don't inherit a stretched interval from the pre-fault failures).

8. Testing

ScadaBridge (Lever 1):

Unit (InstanceActor tests): a slow/never-answered subscribe issues exactly one in-flight request until the ceiling, not one per interval; backoff doubles on repeated failure and resets on success; a connection restore resets backoff.
Option validation: TagSubscribeRetryMaxIntervalMs >= TagSubscribeRetryIntervalMs; ceiling > 0.

Cross-repo (Lever 2):

Worker unit: SubscribeBulk(supervisory:true) calls AdviseSupervisory (fake MXAccess session asserts advise kind); partial-failure and cleanup unchanged.
Gateway: OpenSessionReply.capabilities includes bulk-subscribe-supervisory.
ScadaBridge adapter: with capability present → single SubscribeBulk(supervisory=true); absent → legacy fan-out. Handle bookkeeping identical across both paths (round-trip test).
Proto: additive-only guard (field 3 added, none renumbered).

Lever 3: RealMxGatewayClient.ConnectAsync sends OpenSessionRequest.command_timeout = MxCommandTimeout; the fake gateway asserts the value and the echoed default_command_timeout is logged; option validated > 0.

Live gate (QA rig, after deploy — not part of this doc's execution): reproduce the incident shape — redeploy Z28061 several times in quick succession while watching health site site-a; confirm successfullyResolved tracks to full without a Bad/frozen window, gateway log shows no AddItemBulk/AdviseSupervisory timed out, and (with Lever 2 active) the per-subscribe RPC count on the gateway session drops ~244×.

9. Rollout & sequencing

Levers are independent and independently shippable; recommended order:

Lever 1 (ScadaBridge-only) — DONE 2026-08-28. Highest value / lowest blast radius — breaks the amplifier immediately; no gateway change. Ships and deploys as a ScadaBridge binary swap.
Lever 2 (gateway first, then ScadaBridge). Deploy the gateway side (new capability + worker flag) before ScadaBridge starts requesting it — capability negotiation makes the order safe either way, but gateway-first means the optimization lights up as soon as ScadaBridge ships. Both are additive/wire-compatible.
Lever 3 (ScadaBridge-only). Fold in with Lever 1 or Lever 2's ScadaBridge change; pure config surface.

No EF migrations, no schema, no new ports. MxAccessGateway proto regen uses the existing manual vendored-proto toggle in both repos.

Cross-repo index: update the ScadaBridge entry in ../scadaproj/CLAUDE.md if the OPC-UA/MxGateway wire relationship summary changes (per repo rule); update MxAccessGateway's own capability list docs for bulk-subscribe-supervisory.

10. Open questions

Backoff defaults: 5 s floor → 60 s ceiling proposed; confirm the ceiling is acceptable for reconnect-recovery latency (a restore resets to floor, so steady-state recovery is unaffected).
Lever 3 value: 60 s MxCommandTimeout assumes Lever 2 lands; without Lever 2 the legacy fan-out is N separate 30 s-bounded commands, so the session timeout barely matters for subscribe — it is the write-side widening (§5 trade) that decides whether 60 s is right. Size against the fallback path if shipping Lever 3 before Lever 2.
Incident record: write docs/deployment/deployment-records/2026-08-26-wonder-app-vq03-vq04-freeze.md (or fold into the next QA deploy record) so the 2026-08-26 analysis is not only in a design doc.
NativeAlarmActor is the same amplifier, still unguarded (final review of Lever 1, 2026-08-28). NativeAlarmActor.SendSubscribe (SiteRuntime/Actors/NativeAlarmActor.cs:147-165) arms a fixed NativeAlarmRetryIntervalMs × 2 timer on send with no in-flight check and no backoff, and HandleSubscribeResponse (:319-337) never compares the correlation id — so a stale reply can cancel the timer guarding a newer request. One actor per native alarm source per instance, over the same DCL connections that wedged on 2026-08-26. Follow-up: port Lever 1's TagSubscribeState shape to it (or a Gitea issue) before the next incident finds it.
Cancel-race residue (Lever 1 M5): ICancelable.Cancel() cannot retract a RetryTagSubscribe already in the mailbox, so a tick that races a Success issues one spurious full subscribe (self-healing, bounded to one). A Confirmed flag on TagSubscribeState would close it in ~4 lines; deferred until it is observed.
Should the in-flight guard also live in DataConnectionActor (coalescing duplicate concurrent subscribe requests for the same connection across instances), as a second layer? Deferred — InstanceActor is where the storm originates and is sufficient; revisit only if multi-instance coalescing proves necessary.

22 KiB Raw Blame History Unescape Escape

MxGateway subscribe-storm prevention — design