feat(inbound-api): standby central node relays POST /api/* to the active peer instead of 503

A caller addressing a standby central node directly (or hitting Traefik inside its
/health/active probe window after a failover) used to get 503 STANDBY_NODE. The
standby now relays the request verbatim to the active peer over HTTP
(IStandbyRequestForwarder / HttpStandbyRequestForwarder) and replays the response;
method scripts and Route.To() still execute only on the active node.

- ScadaBridge:InboundApi:StandbyForwardPeerEndpoints (static list of the OTHER
  central node's HTTP base URL; empty = old 503 behaviour) + StandbyForwardTimeout
  (60s), both startup-validated.
- Peers tried in order: peer 503 stamped X-ScadaBridge-Standby, connection failure
  or timeout → next peer; all fail → 503 ACTIVE_NODE_UNREACHABLE. A 503 without
  the marker is the active node's own answer and is replayed.
- One-hop loop guard via X-ScadaBridge-Forwarded; 413 body cap now runs BEFORE
  the active-node check so a standby never buffers/relays an oversized body.
- AuditWriteMiddleware skips its own ApiInbound row for a relayed request so a
  call is audited once, on the node that executed it.
- Rig + env2 central nodes configured; spec (Component-InboundAPI.md → Standby
  forwarding), Component-Host.md options row, CLAUDE.md updated.
- 20 new tests (filter decision + relay mechanics + validator).

Claude-Session: https://claude.ai/code/session_019ZxXT5sxZs5XrtdwYzJDPV
This commit is contained in:
Joseph Doherty
2026-08-28 13:34:20 -04:00
parent f07ba5cff5
commit a9b23556eb
16 changed files with 839 additions and 31 deletions
+1 -1
View File
@@ -59,7 +59,7 @@ The Host must bind configuration sections from `appsettings.json` to strongly-ty
| `ScadaBridge:SiteEventLog` | `SiteEventLogOptions` | Site Event Logging | RetentionDays, MaxStorageMb, PurgeScheduleCron |
| `ScadaBridge:Communication` | `CommunicationOptions` | Communication | DeploymentTimeout, LifecycleTimeout, QueryTimeout, TransportHeartbeatInterval, TransportFailureThreshold |
| `ScadaBridge:Security` | `SecurityOptions` | Security & Auth | LdapServer, LdapPort, LdapUseTls, JwtSigningKey, JwtExpiryMinutes, IdleTimeoutMinutes |
| `ScadaBridge:InboundApi` | `InboundApiOptions` | Inbound API | DefaultMethodTimeout |
| `ScadaBridge:InboundApi` | `InboundApiOptions` | Inbound API | DefaultMethodTimeout, MaxRequestBodyBytes, ApiMethodCacheTtl, StandbyForwardPeerEndpoints (HTTP base URLs of the other central node — a standby relays `POST /api/*` there instead of answering 503), StandbyForwardTimeout |
| `ScadaBridge:Notification` | `NotificationOptions` | Notification Service | (SMTP config is stored in the central config DB, not in appsettings) |
| `ScadaBridge:NotificationOutbox` | `NotificationOutboxOptions` | Notification Outbox | Dispatcher poll interval, stuck-age threshold, retention window (delivery retry settings reuse the central SMTP configuration) |
| `ScadaBridge:SiteCallAudit` | `SiteCallAuditOptions` | Site Call Audit | Reconciliation pull interval, stuck-age threshold, retention window |
+18 -4
View File
@@ -6,7 +6,7 @@ The Inbound API exposes a web API on the central cluster that external systems c
## Location
Central cluster only (active node). Not available at site clusters.
Central cluster only. Method scripts execute on the **active** central node; a **standby** central node accepts the same endpoint and relays the request to the active peer (see [Standby forwarding](#standby-forwarding)). Not available at site clusters.
## Responsibilities
@@ -16,7 +16,7 @@ Central cluster only (active node). Not available at site clusters.
- Enforce per-method API key authorization (only approved keys can call a given method).
- Execute the C# script implementation for the called method.
- Return structured responses to the caller.
- Failover: API becomes available on the new active node after central failover.
- Failover: API becomes available on the new active node after central failover; during the handover a standby relays to whichever peer is active, so a caller addressing either central node directly is served without waiting for the load balancer's health probe to catch up.
## API Key Management
@@ -122,7 +122,8 @@ The `sbk_<keyId>_<secret>` design already supports zero-downtime rotation — a
| `INVALID_JSON` | 400 | Request body was not valid JSON. |
| `VALIDATION_FAILED` | 400 | Parameter validation failed (missing required field, wrong type, undeclared field). |
| `BODY_TOO_LARGE` | 413 | Request body exceeded `MaxRequestBodyBytes`. |
| `STANDBY_NODE` | 503 | Request hit a standby central node; the inbound API serves only the active node. |
| `STANDBY_NODE` | 503 | Request hit a standby central node that could not relay it: no `StandbyForwardPeerEndpoints` configured, or the request was already relayed once (loop guard — neither node is active, e.g. mid-failover). Carries the `X-ScadaBridge-Standby: 1` response header. |
| `ACTIVE_NODE_UNREACHABLE` | 503 | A standby tried every configured peer and none served the relay (unreachable, timed out after `StandbyForwardTimeout`, or also standby). |
| `TIMEOUT` | 500 | The method script exceeded its execution timeout. |
| `SITE_UNREACHABLE` | 500 | The target site did not respond to the routed request within the routing timeout (no contact / down / partitioned) — typed from the routed Ask expiring (`AskTimeoutException`), never inferred from an error-message substring. A `Success=false` response means the site answered and is classified `SCRIPT_ERROR`. |
| `SCRIPT_COMPILE_FAILED` | 500 | The method's script failed to compile (returned consistently on every node until a compiling version is saved). |
@@ -282,6 +283,19 @@ For everything else, scripts interact with the system through the curated `Route
3. The key's provisioned scopes include the requested method name (case-sensitive).
- Failed authentication returns an appropriate HTTP error (401 Unauthorized for invalid/missing credentials; 403 Forbidden for a valid key not approved for the method). For enumeration-safety, both "method not found" and "key not in scope" return 403 with an identical body.
## Standby forwarding
Both central nodes expose `POST /api/{methodName}`, but only the active node (the oldest Up member — `ActiveNodeEvaluator.SelfIsOldestUp`, the same verdict behind `/health/active`) executes method scripts and `Route.To()` calls; running them on a standby could race the active node or read stale singleton state. Before 2026-08-28 a standby simply answered 503 `STANDBY_NODE`, and "both nodes work" relied entirely on the load balancer probing `/health/active` — a caller addressing a node directly, or hitting the balancer inside its probe interval after a failover, got the 503.
A standby now **relays** the request to the active peer instead (`InboundApiEndpointFilter` → `IStandbyRequestForwarder` / `HttpStandbyRequestForwarder`):
- **Configuration.** `ScadaBridge:InboundApi:StandbyForwardPeerEndpoints` — the HTTP base URL(s) of the *other* central node(s), e.g. `["http://scadabridge-central-b:5000"]` on central-a. Empty (default) keeps the old 503 behaviour. `StandbyForwardTimeout` (default 60 s) is the per-peer attempt deadline and must exceed the longest method timeout in use. Validated at startup (absolute http/https URLs, positive timeout). Note the standby discovers the active peer from **appsettings**, not from cluster state — the Akka member address carries the remoting port, not the HTTP port — so this is a static, restart-to-change list, like `CentralGrpcEndpoints` on sites.
- **Pure relay.** Method, path, query string, body and headers travel verbatim (hop-by-hop headers and `Host` excepted), so the **active node** authenticates the caller's own `Authorization`/`X-API-Key` credential, applies its own body cap and method timeout, and writes the audit row. The peer's status, `Content-Type` and body are replayed unchanged. The standby's `AuditWriteMiddleware` skips its own `ApiInbound` row for a relayed request (`HttpContext.Items[RelayedToActiveItemKey]`) so a call is audited once, where it ran.
- **Order of checks.** The body-size cap (413 `BODY_TOO_LARGE`) runs *before* the active-node check, so a standby never buffers or relays a body the active node would refuse. The body is buffered once on the standby (bounded by `MaxRequestBodyBytes`) so it can be re-sent to a second peer.
- **Peer selection.** The standby knows only that it is *not* active. It tries the peers in order and accepts the first response that is not a standby refusal; a peer answering 503 with `X-ScadaBridge-Standby: 1`, a connection failure, or a `StandbyForwardTimeout` expiry each move to the next peer. A 503 *without* the marker is the active node's own answer and is replayed, not retried. All peers failing → 503 `ACTIVE_NODE_UNREACHABLE`.
- **One-hop loop guard.** Every relayed request carries `X-ScadaBridge-Forwarded: <relaying host>`. A standby never relays a request that already carries it — it answers 503 `STANDBY_NODE` with the marker header — so a pair in which neither node is active (a transient during failover) bounces at most one hop and never ping-pongs. The active node ignores the header and serves the request normally.
- **Still not a retry layer.** The relay makes one HTTP attempt per peer and does not retry a peer that answered; at-least-once callers keep their own retry loop and the idempotent-method authoring note above still applies.
## Error Handling
- Invalid API key → 401 Unauthorized.
@@ -298,7 +312,7 @@ For everything else, scripts interact with the system through the curated `Route
- **Configuration Database (via IAuditService)**: All API key and method definition changes are audit logged.
- **Script Analysis (#25)**: `ForbiddenApiChecker` delegates to `ScriptTrustValidator.FindViolations` for the authoritative forbidden-API verdict during script compilation and validation.
- **Audit Log (#23)**: Every inbound API request emits an `ApiInbound.Completed` row via `ICentralAuditWriter` from request middleware (non-blocking for the HTTP response). Request and response bodies are captured in full up to `AuditLog:InboundMaxBytes` (default 1 MiB) per the Audit Log Payload Capture Policy; redaction (headers + per-target body redactors) still applies before persistence.
- **Cluster Infrastructure**: API is hosted on the active central node and fails over with it.
- **Cluster Infrastructure**: method execution is hosted on the active central node and fails over with it; `IActiveNodeGate` (Host) tells the endpoint filter whether to serve locally or relay.
## Interactions