dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

Resolve all 33 medium-severity architecture review findings

Browse Source 
Second slice of the 2026-09-09 whole-system review remediation
(docs/plans/2026-09-09-arch-review.md §10), on top of the High slice 146a1323.

Runtime/devices: SR-2 alarm on-trigger integration scope; SR-3 correlated NativeAlarmActor
subscribe retry; SR-4 immutable audited DB target; SR-5 ordered InstanceActor stop/flush;
SR-12 rehydration fence; SR-13 S&F gate recheck per delivery; DC-4 no OPC UA security
downgrade; DC-5 alarm snapshot for every new subscriber; ORCH-1 sync-dispose joins the pump.

Management/UI: MG-2 site scope on audit tree + Flow Debugger Replay; MG-3 stale-probe memo
removed; MG-4 server-owned bulk-deploy deadline; UI-5 sandbox runs in a killable child
process (new CentralUI.SandboxWorker project); UI-14 bounded sandbox output/IPC.

Edge/audit: EI-1 relay audit ownership; EI-2 SkipBodyCapture suppresses error text; EI-3/EI-11
error-field redaction + caps; EI-4 single relay deadline, pre-execution-only retry; EI-5
authenticated relay provenance; EI-10 body cap before audit read.

Persistence/contracts: DL-1 global Instance.UniqueName unique index + conflict translation;
DL-2 DataConnection protocol config encrypted at rest (migration
20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration — coordinated
central upgrade, old binaries cannot read protected rows); DL-3 bounded Replay paging;
DL-6 cross-site Overwrite of a deployed instance rejected; XC-4 proto descriptor contract test.

Config/hygiene: XC-1 central listener isolation by local port; XC-2/XC-3 inert SMTP + cron
keys removed from rig settings; CM-2 env2 bootstrap guard enabled; DC-6 deploy site exit code;
DC-7 notifier retries only explicit standby rejection; XC-5 ClusterClient purged from docs.

Gates recorded in the report: build 0/0, 9 397 passed / 0 failed, rig redeployed on all
eight nodes with the migration applied. 46 Low findings remain open.
This commit is contained in:
Joseph Doherty
2026-09-09 22:05:30 -04:00
parent 146a13238f
commit 9a87d7a328
136 changed files with 5441 additions and 648 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/requirements/Component-InboundAPI.md
+21 -8
View File
@@ -74,9 +74,9 @@ Each API method definition includes:
> **Authoring note — idempotent methods for at-least-once callers**: some external callers retry
> on their own connect-failure/timeout failover loop and cannot tell "the first attempt actually
> succeeded" from "the first attempt was lost" — e.g. the `DelmiaRecipeDownload` method backing
> `DelmiaNotifier` (see `docs/plans/2026-06-26-delmia-recipe-notifier-design.md`, "Delivery
> semantics: at-least-once"). Implementation scripts for such methods must be idempotent on their
> succeeded" from "the first attempt was lost". The updated `DelmiaNotifier` returns an uncertain
> result instead of automatically replaying such attempts. Scripts exposed to other retrying
> clients should still be idempotent on their
> natural key so a duplicate delivery returns the same result without double-applying side effects.
### Management
@@ -148,7 +148,8 @@ The `sbk_<keyId>_<secret>` design already supports zero-downtime rotation — a
| `VALIDATION_FAILED` | 400 | Parameter validation failed (missing required field, wrong type, undeclared field). |
| `BODY_TOO_LARGE` | 413 | Request body exceeded `MaxRequestBodyBytes`. |
| `STANDBY_NODE` | 503 | Request hit a standby central node that could not relay it: no `StandbyForwardPeerEndpoints` configured, or the request was already relayed once (loop guard — neither node is active, e.g. mid-failover). Carries the `X-ScadaBridge-Standby: 1` response header. |
| `ACTIVE_NODE_UNREACHABLE` | 503 | A standby tried every configured peer and none served the relay (unreachable, timed out after `StandbyForwardTimeout`, or also standby). |
| `ACTIVE_NODE_UNREACHABLE` | 503 | Every configured peer refused before execution or failed with proven name-resolution/connection-refusal errors. |
| `OUTCOME_UNKNOWN` | 502 | A relay timed out or lost its response after possible execution; it does not replay the request. Verify the operation before retrying. |
| `TIMEOUT` | 500 | The method script exceeded its execution timeout. |
| `SITE_UNREACHABLE` | 500 | The target site did not respond to the routed request within the routing timeout (no contact / down / partitioned) — typed from the routed Ask expiring (`AskTimeoutException`), never inferred from an error-message substring. A `Success=false` response means the site answered and is classified `SCRIPT_ERROR`. |
| `SCRIPT_COMPILE_FAILED` | 500 | The method's script failed to compile (returned consistently on every node until a compiling version is saved). |
@@ -362,10 +363,10 @@ Both central nodes expose `POST /api/{methodName}`, but only the active node (th
A standby now **relays** the request to the active peer instead (`InboundApiEndpointFilter` → `IStandbyRequestForwarder` / `HttpStandbyRequestForwarder`):
- **Configuration.** `ScadaBridge:InboundApi:StandbyForwardPeerEndpoints` — the HTTP base URL(s) of the *other* central node(s), e.g. `["http://scadabridge-central-b:5000"]` on central-a. Empty (default) keeps the old 503 behaviour. `StandbyForwardTimeout` (default 60 s) is the per-peer attempt deadline and must exceed the longest method timeout in use. Validated at startup (absolute http/https URLs, positive timeout). Note the standby discovers the active peer from **appsettings**, not from cluster state — the Akka member address carries the remoting port, not the HTTP port — so this is a static, restart-to-change list, like `CentralGrpcEndpoints` on sites.
- **Pure relay.** Method, path, query string, body and headers travel verbatim (hop-by-hop headers and `Host` excepted), so the **active node** authenticates the caller's own `Authorization`/`X-API-Key` credential, applies its own body cap and method timeout, and writes the audit row. The peer's status, `Content-Type` and body are replayed unchanged. The standby's `AuditWriteMiddleware` skips its own `ApiInbound` row for a relayed request (`HttpContext.Items[RelayedToActiveItemKey]`) so a call is audited once, where it ran.
- **Order of checks.** The body-size cap (413 `BODY_TOO_LARGE`) runs *before* the active-node check, so a standby never buffers or relays a body the active node would refuse. The body is buffered once on the standby (bounded by `MaxRequestBodyBytes`) so it can be re-sent to a second peer.
- **Peer selection.** The standby knows only that it is *not* active. It tries the peers in order and accepts the first response that is not a standby refusal; a peer answering 503 with `X-ScadaBridge-Standby: 1`, a connection failure, or a `StandbyForwardTimeout` expiry each move to the next peer. A 503 *without* the marker is the active node's own answer and is replayed, not retried. All peers failing → 503 `ACTIVE_NODE_UNREACHABLE`.
- **Configuration.** `ScadaBridge:InboundApi:StandbyForwardPeerEndpoints` — the HTTP base URL(s) of the *other* central node(s), e.g. `["http://scadabridge-central-b:5000"]` on central-a. Empty (default) keeps the old 503 behaviour. `StandbyForwardTimeout` (default 60 s) is the total relay budget and should exceed the longest method timeout plus network overhead. Validated at startup (absolute http/https URLs, positive timeout). Note the standby discovers the active peer from **appsettings**, not from cluster state — the Akka member address carries the remoting port, not the HTTP port — the options snapshot is read per relay.
- **Pure relay.** Method, path, query string, body and credentials travel unchanged; hop-by-hop/Host/forwarding-provenance headers are excluded and authenticated provenance is rebuilt, so the **active node** authenticates the caller's own `Authorization`/`X-API-Key` credential, applies its own body cap and method timeout, and writes the audit row. The peer's status, `Content-Type` and body are replayed unchanged. The serving node owns its `ApiInbound` row; the originating standby retains a failure row when no peer serves or the outcome is uncertain. Verified internal standby refusals are not duplicated.
- **Order of checks.** The body-size cap (HTTP 413) runs before audit reads and the active-node check, so a standby never buffers or relays a body the active node would refuse. The body is buffered once on the standby (bounded by `MaxRequestBodyBytes`) so it can be re-sent to a second peer.
- **Peer selection.** The standby knows only that it is *not* active. It tries the peers in order and accepts the first response that is not a standby refusal; a peer answering 503 with `X-ScadaBridge-Standby: 1`, proven name-resolution failure, or connection refusal permits trying the next peer. Timeout or ambiguous response loss returns 502 `OUTCOME_UNKNOWN` without replay. A 503 *without* the marker is the active node's own answer and is replayed, not retried. All peers failing → 503 `ACTIVE_NODE_UNREACHABLE`.
- **One-hop loop guard.** Every relayed request carries `X-ScadaBridge-Forwarded: <relaying host>`. A standby never relays a request that already carries it — it answers 503 `STANDBY_NODE` with the marker header — so a pair in which neither node is active (a transient during failover) bounces at most one hop and never ping-pongs. The active node ignores the header and serves the request normally.
- **Still not a retry layer.** The relay makes one HTTP attempt per peer and does not retry a peer that answered; at-least-once callers keep their own retry loop and the idempotent-method authoring note above still applies.
@@ -394,3 +395,15 @@ A standby now **relays** the request to the active peer instead (`InboundApiEndp
- **Site Runtime (Instance Actors, Script Actors)**: Routed calls execute on site Instance Actors via their Script Actors.
- **Central UI**: Administrator manages API keys; Designer manages method definitions.
- **Configuration Database (via IAuditService)**: Configuration changes are audited.
### Relay audit ownership, privacy, and request limits
The inbound body limit is installed before audit buffering. Declared oversized bodies and unknown-length HTTP/1.1 or HTTP/2 bodies are rejected with HTTP 413; enabling audit capture cannot relax the route limit. The relay also bounds its buffered copy explicitly.
A serving peer owns the inbound audit for its success or method failure. The originating standby writes a correlated failure if no peer serves the request, including an uncertain lost response. Authenticated internal standby refusals do not create duplicate inbound rows. Audit writes remain best effort; ownership is not a durable audit receipt.
Relays overwrite caller-supplied provenance with an HMAC signed envelope using a domain-separated key derived from `ApiKeyPepper`. Both centrals must share that server secret. The envelope binds the original verified remote address, execution ID, timestamp, method, path/query, content type, caller credentials, and bounded body digest. The receiver accepts it for at most five minutes and only when the request binding matches. Unsigned loop markers and forwarded-address headers confer no identity trust; missing or invalid proof uses the receiver's verified remote address. Forwarding remains one hop. Use TLS between peers when network confidentiality is required.
`StandbyForwardTimeout` is one budget across permitted attempts and response consumption. Only explicit pre-execution standby rejection, name-resolution failure, or connection refusal permits another peer. Timeout or response loss returns HTTP 502 `OUTCOME_UNKNOWN`: verify the operation before manually retrying because it may already have applied. This avoids blindly replaying non-idempotent methods.
`SkipBodyCapture` suppresses request/response summaries and all exception message/detail text, including stashed script exceptions; retain status/category and request metadata for diagnosis. Error text can quote payload values and is subject to the same privacy boundary as the body.