ScadaBridge/docs/requirements/Component-InboundAPI.md

Component: Inbound API

Purpose

The Inbound API exposes a web API on the central cluster that external systems can call into. This is the reverse of the External System Gateway — where that component handles the SCADA system calling out to external systems, this component handles external systems calling in. It provides API key authentication, method-level authorization, and script-based method implementations.

Location

Central cluster only. Method scripts execute on the active central node; a standby central node accepts the same endpoint and relays the request to the active peer (see Standby forwarding). Not available at site clusters.

Responsibilities

• Host a web API endpoint on the central cluster.
• Authenticate inbound requests via API keys.
• Route requests to the appropriate API method definition.
• Enforce per-method API key authorization (only approved keys can call a given method).
• Execute the C# script implementation for the called method.
• Return structured responses to the caller.
• Failover: API becomes available on the new active node after central failover; during the handover a standby relays to whichever peer is active, so a caller addressing either central node directly is served without waiting for the load balancer's health probe to catch up.

API Key Management

Storage

• API keys are stored in the shared ZB.MOM.WW.Auth.ApiKeys library's store — SQLite per node by default, or the shared zbauth schema in the configuration database (MS SQL) on a central pair when Provider=SqlServer (see Key store provider below). Revoke/rotate/scope changes are cluster-wide immediately only under SqlServer; under Sqlite they apply to whichever node served the management call.

Key store provider

Config section ScadaBridge:InboundApi:ApiKeyStore (design 2026-09-02, register row 108):

Key	Default	Notes
Provider	Sqlite	Sqlite (per node) or SqlServer (shared across a central pair). Case-insensitive; an unrecognized value fails startup via StartupValidator.
SqlServer:ConnectionString	ScadaBridge:Database:ConfigurationDb	Must be a literal or env override, never a ${secret:} reference — the secrets store is itself SQL-backed, so resolving one through the other would be circular.
SqlServer:Schema	zbauth	Owns schema_version, api_keys, api_key_audit.
SqlServer:CommandTimeout	30 s
ImportFromSqlitePath	the library's SqlitePath default	SQLite file imported once, at boot, when Provider=SqlServer; insert-where-absent, so both nodes of a pair may import their own file and the result is the union. On success the file is renamed *.imported (plus -wal/-shm); on failure the import logs Error and the node keeps booting on the keys already in SQL.

A Central node with ≥2 seed nodes still running Provider=Sqlite logs a boot-time Warning (not a hard fail — availability-first, matching the auto-down decision) naming this design, because a failover will otherwise 401 every inbound client until keys are re-issued. Both docker rigs (docker/, docker-env2/) run Provider=SqlServer; QA/prod are a separate deployment record. See docs/operations/inbound-api-key-reissue.md → Central pairs for the reissue/migration runbook and docs/operations/sql/zbauth-schema.sql for the DDL a DBA runs by hand when the app login lacks CREATE SCHEMA/TABLE rights.

Key Properties

• Name/Label: Human-readable identifier for the key (e.g., "MES-Production", "RecipeManager-Dev").
• Key Value: The secret key string used for authentication.
• Enabled/Disabled Flag: Keys can be disabled without deletion.

Management

• Managed by users with the Administrator role via the Central UI.
• All key changes (create, enable/disable, delete) are audit logged.

API Method Definition

Properties

Each API method definition includes:

• Method Name: Unique identifier and URL path segment for the endpoint.
• Approved API Keys: List of API keys authorized to invoke this method. Requests from non-approved keys are rejected.
• Parameter Definitions: Ordered list of input parameters, each with:
  • Parameter name.
  • Data type — the extended type system (Boolean, Integer, Float, String, plus the nestable Object and List; see Extended Type System).
  • Whether the parameter is required.
• Return Value Definition: Structure of the response, with:
  • Field names and (extended-system) data types. Supports returning lists of objects and arbitrarily nested structures.
• Implementation Script: C# script that executes when the method is called. Stored inline in the method definition. Follows standard C# authoring patterns but has no template inheritance — it is a standalone script tied to this method.
• Timeout: Configurable per method. Defines the maximum time the method is allowed to execute (including any routed calls to sites) before returning a timeout error to the caller.
• TraceAllCalls (Flow Debugger, 2026-09-05): when set, every call to this method runs its routed site scripts with the trace flag on, so their ScriptRun / AttributeWrite / AttributeWait audit rows (AuditChannel.ScriptTrace) are recorded for Replay. Default off — adds roughly seven audit rows per call, so it is meant for low-volume integration methods. Set from the method form's Trace all calls checkbox or api-method create|update --trace-all-calls.

Authoring note — idempotent methods for at-least-once callers: some external callers retry on their own connect-failure/timeout failover loop and cannot tell "the first attempt actually succeeded" from "the first attempt was lost" — e.g. the DelmiaRecipeDownload method backing DelmiaNotifier (see docs/plans/2026-06-26-delmia-recipe-notifier-design.md, "Delivery semantics: at-least-once"). Implementation scripts for such methods must be idempotent on their natural key so a duplicate delivery returns the same result without double-applying side effects.

Management

• Managed by users with the Designer role via the Central UI.
• All method definition changes are audit logged.

HTTP Contract

URL Structure

• All API calls use POST /api/{methodName}.
• Method names map directly to URL path segments (e.g., method "GetProductionReport" → POST /api/GetProductionReport).
• All calls are POST — these are RPC-style script invocations, not RESTful resource operations.

Authentication Header

• The API key token (sbk_<keyId>_<secret>) is accepted from either of two headers:
  • Authorization: Bearer sbk_<keyId>_<secret> — the preferred form.
  • X-API-Key: sbk_<keyId>_<secret> — alternate form for callers that cannot set the Authorization header.
• When both headers are present, Authorization takes precedence.
• Both forms are processed by the same IApiKeyVerifier from the shared ZB.MOM.WW.Auth.ApiKeys library; the verifier strips the optional Bearer prefix and performs the same peppered-HMAC constant-time verification regardless of which header was used.

Key rotation

The sbk_<keyId>_<secret> design already supports zero-downtime rotation — a caller can hold two valid keys at once, so there is no moment where the old key must be cut off before the new one works. The procedure:

1. Create a new key with the same scope set as the key being rotated — security api-key create --name <name> --methods <methods> (CLI security api-key group). Scopes are copied at creation only: the operator must pass the same --methods list the old key was approved for; the CLI does not read the old key's scopes for you, so scope drift between the pair is an operator error, not something the system detects.
2. Deliver the new token to the caller and confirm cutover. The response from create surfaces the one-time bearer token (PrintCreatedKey — it is shown once and cannot be retrieved again, so capture it immediately). Both the old and new keys verify successfully during this window — that overlap is intended, not a bug, and gives the caller time to redeploy with the new token without a synchronized flag-day cutover. Confirm the caller has actually switched by checking the audit log's Actor field (recorded per call), which shows which key ID is serving live traffic.
3. Disable — don't delete — the old key once cutover is confirmed: security api-key update --key-id <old-key-id> --enabled false. Disabling is reversible (--enabled true re-enables it) if the cutover needs to be rolled back, whereas deletion is not.
4. Delete the old key after a quarantine period (an operator-judgment window to be sure nothing is still presenting the old token): security api-key delete --key-id <old-key-id>.

Request Format

• Content-Type: application/json.
• Parameters are top-level JSON fields in the request body matching the method's parameter definitions:

{
  "siteId": "SiteA",
  "startDate": "2026-03-01",
  "endDate": "2026-03-16"
}

Response Format

• Success (200): The response body is the method's return value as JSON, with fields matching the return value definition:

{
  "siteName": "Site Alpha",
  "totalUnits": 14250,
  "lines": [
    { "lineName": "Line-1", "units": 8200, "efficiency": 92.5 },
    { "lineName": "Line-2", "units": 6050, "efficiency": 88.1 }
  ]
}

• Failure (4xx/5xx): The response body is an error object carrying a human-readable error and a stable, machine-readable code:

{
  "error": "Site unreachable",
  "code": "SITE_UNREACHABLE"
}

• HTTP status codes distinguish success from failure — no envelope wrapper.
• Error codes. Every failure body carries a code an external integrator can branch on (the error message is for humans and may change). The two authorization negatives — unknown method and key-not-in-scope — deliberately share one code (NOT_APPROVED) and one body so a caller cannot enumerate which method names exist:

code	HTTP status	Meaning
UNAUTHORIZED	401	Missing/invalid API key (every auth-stage failure maps here — no stage leak).
NOT_APPROVED	403	Method not found or key not in scope — indistinguishable by design.
UNSUPPORTED_MEDIA_TYPE	415	Request carried a body — declared by Content-Length or Transfer-Encoding (chunked) — with a non-JSON Content-Type. A body with no Content-Type is still parsed leniently as JSON, and a chunked (no Content-Length) body is treated identically to a fixed-length one.
INVALID_JSON	400	Request body was not valid JSON.
VALIDATION_FAILED	400	Parameter validation failed (missing required field, wrong type, undeclared field).
BODY_TOO_LARGE	413	Request body exceeded MaxRequestBodyBytes.
STANDBY_NODE	503	Request hit a standby central node that could not relay it: no StandbyForwardPeerEndpoints configured, or the request was already relayed once (loop guard — neither node is active, e.g. mid-failover). Carries the X-ScadaBridge-Standby: 1 response header.
ACTIVE_NODE_UNREACHABLE	503	A standby tried every configured peer and none served the relay (unreachable, timed out after StandbyForwardTimeout, or also standby).
TIMEOUT	500	The method script exceeded its execution timeout.
SITE_UNREACHABLE	500	The target site did not respond to the routed request within the routing timeout (no contact / down / partitioned) — typed from the routed Ask expiring (AskTimeoutException), never inferred from an error-message substring. A Success=false response means the site answered and is classified SCRIPT_ERROR.
SCRIPT_COMPILE_FAILED	500	The method's script failed to compile (returned consistently on every node until a compiling version is saved).
SCRIPT_ERROR	500	Catch-all for any other script-execution failure (details are logged centrally, never leaked to the caller).

Extended Type System

• API method parameter and return type definitions support an extended type system beyond the four template attribute types (Boolean, Integer, Float, String):
  • Object: A named structure with typed fields. Supports nesting.
  • List: An ordered collection of objects or primitive types.
• This allows complex request/response structures (e.g., an object containing properties and a list of nested objects).
• Template attributes retain the simpler four-type system. The extended types apply only to Inbound API method definitions and External System Gateway method definitions.

Type Definition Format & Nested Validation

• Parameter and return type definitions are persisted as JSON Schema (the canonical format produced by the Central UI schema builder; see the MigrateParametersToJsonSchema migration). An object declares its fields via properties (+ a required array); a list declares its element type via items. The legacy flat-array form ([{name,type,required,itemType?}]) is still accepted on read for transition safety.
• Validation is recursive and type-aware for the extended types (request parameters and script return values alike, via a single shared engine so the two cannot drift):
  • Object: each declared field's value is validated against its declared (possibly nested) type; a missing required field and a present-but-wrong type are both reported.
  • List: every element is validated against the declared element type (recursing into nested objects/lists). A list whose element type is left undeclared (array without items) is shape-checked only.
  • Scalars at any depth are checked against the extended type.
  • Errors are path-qualified (e.g. order.items[2].quantity) so the caller can locate the offending field.
• Undeclared fields are rejected at every level (consistent with the top-level "unexpected parameter" rejection): an object that declares its fields rejects any field not in its properties, so a typo'd field name surfaces as a 400/error rather than being silently ignored. A bare object schema with no declared fields ({"type":"object"}) stays shape-only and accepts any fields.
• A JSON null value satisfies any declared type (a present-but-null field is allowed); only the absence of a required field is an error.

Script Compilation & Hot-Reload

API method scripts are compiled at central startup — all method definitions are loaded from the configuration database and compiled into in-memory delegates.

Update Workflow

• Updating a method via the CLI (api-method update --id <N> --code '...') or Management API triggers immediate recompilation (CompileAndRegister). The updated script takes effect on the next API call — no node restart is required.
• Creating a new method after startup: if the method is created but not yet compiled, the first invocation triggers lazy (on-demand) compilation.

Compile failures on save are non-fatal but surfaced, and the DB row is authoritative. The persisted ApiMethod.Script row is the single source of truth: on every request ExecuteAsync revision-checks the cached delegate against the freshly-fetched script and recompiles in place when it differs, so a saved-but-broken script does not leave a previously registered handler silently serving. The change still persists, so the Create/Update Management result carries a top-level compileWarning (null when the script compiled) listing the Roslyn/trust-model errors. On both create and update, once a non-compiling script is saved the method returns a compilation error (HTTP 500) consistently on every central node until a compiling version is saved — there is no node-divergent "old delegate still works here" state. This means a saved script that does not compile is not silently a no-op — the caller (CLI / Central UI) sees the warning, and the real diagnostics are also logged centrally.

Direct SQL Warning

Direct SQL edits to API method scripts are now picked up automatically. Because the DB row is authoritative and ExecuteAsync revision-checks the cached delegate against the freshly-fetched ApiMethod.Script on every request, a script changed directly in the database takes effect on the next call — no node restart and no Management round-trip is required. A broken direct edit surfaces the same way as a broken save: the method returns a HTTP 500 compilation error consistently on all central nodes until a compiling version is stored. Editing via the CLI, Management API, or Central UI is still preferred because those paths validate the script up front and return the compileWarning, whereas a direct SQL edit is only diagnosed on the next request (and in the central log).

---

API Call Logging

• Every request — success or failure — emits one ApiInbound.Completed row to ICentralAuditWriter from request middleware before the HTTP response is flushed. The row captures the API key name (never the key material), remote IP, user-agent, response status, duration, and the request/response bodies. Bodies are captured in full up to AuditLog:InboundMaxBytes (default 1 MiB); PayloadTruncated = 1 only when that ceiling is hit. Header redaction and per-target body redactors still apply (see Component-AuditLog.md, Payload Capture Policy). This supersedes the earlier failures-only stance: operational API traffic is now part of the centralized audit log, so configuration changes and call activity share a single retention/query surface.
• Script execution errors (500 responses) remain captured on the same ApiInbound.Completed row (response status + error fields) rather than emitting a separate failure-only event.
• Fail-soft semantics (fire-and-forget). The audit write is fire-and-forget (InboundAPI-018): WriteAsync is called without await so the user-facing HTTP response is never blocked or delayed by the audit path. Asynchronous faults are observed via a ContinueWith continuation that logs at Warning and increments CentralAuditWriteFailures (see Health Monitoring #11) rather than being silently dropped. A failed audit append never turns a successful API call into an error returned to the caller.
• No rate limiting — this is a private API in a controlled industrial environment with a known set of callers. Misbehaving callers are handled operationally (disable the API key).

Request Flow

%%{init: {'theme':'base', 'themeVariables': {'textColor':'#111111','lineColor':'#555555','edgeLabelBackground':'#ffffff','fontSize':'15px'}}}%%
flowchart TD
    ext(["External System"])
    api["Inbound API (Central)"]
    s1["1. Extract API key from request"]
    s2["2. Validate key exists and is enabled"]
    s3["3. Resolve method by name"]
    s4["4. Check API key is in method's approved list"]
    s5["5. Validate and deserialize parameters"]
    s6["6. Execute implementation script<br/>(subject to method timeout)"]
    s7["7. Serialize return value"]
    s8["8. Return response"]

    ext --> api
    api --> s1
    s1 --> s2
    s2 --> s3
    s3 --> s4
    s4 --> s5
    s5 --> s6
    s6 --> s7
    s7 --> s8

    classDef start fill:#d5e8d4,stroke:#82b366,color:#111111;
    classDef proc fill:#dae8fc,stroke:#6c8ebf,color:#111111;
    classDef dec fill:#fff2cc,stroke:#d6b656,color:#111111;
    classDef warn fill:#ffe6cc,stroke:#d79b00,color:#111111;
    classDef alt fill:#e1d5e7,stroke:#9673a6,color:#111111;
    class ext start
    class api proc
    class s1,s2,s3,s4,s5,s7 dec
    class s6 alt
    class s8 warn

Implementation Script Capabilities

The C# script that implements an API method executes on the central cluster. Unlike instance scripts at sites, inbound API scripts run on central and can interact with any instance at any site through a routing API.

Inbound API scripts cannot call shared scripts directly — shared scripts are deployed to sites only and execute inline in Script Actors. To execute logic on a site, use Route.To().Call().

Script Runtime API

Instance Routing

• Route.To("instanceUniqueCode").Call("scriptName", parameters) — Invoke a script on a specific instance at any site. Central routes the call to the appropriate site via the Communication Layer. The call reaches the target Instance Actor's Script Actor, which spawns a Script Execution Actor to execute the script. The return value flows back to the calling API script.
• Route.To("instanceUniqueCode").GetAttribute("attributeName") — Read a single attribute value from a specific instance at any site.
• Route.To("instanceUniqueCode").GetAttributes("attr1", "attr2", ...) — Read multiple attribute values in a single call, returned as a dictionary of name-value pairs.
• Route.To("instanceUniqueCode").SetAttribute("attributeName", value) — Write a single attribute value on a specific instance at any site.
• Route.To("instanceUniqueCode").SetAttributes(dictionary) — Write multiple attribute values in a single call, accepting a dictionary of name-value pairs.
• Route.To("instanceUniqueCode").WaitForAttribute("attributeName", targetValue, timeout) — Wait, event-driven, until an attribute on a specific instance at any site reaches targetValue (value-equality only across the wire), bounded by timeout. Returns true if matched within the timeout, false if it timed out. The wait is bounded by its own timeout, not the generic method-level timeout — this is the one routed call that may legitimately outlive the method timeout (the site enforces timeout and returns false when it elapses). A client disconnect still cancels the wait. This is the deliberate exception to the rule below that routed calls inherit the method-level timeout (see "Routing Behavior"): a long event-driven wait is the explicit reason timeout governs here.

Reading alarm state via a routed script. There is deliberately no Route.To(...).GetAlarms(...) verb. An inbound method that needs a machine's current alarms resolves the instance and calls a site script that uses the Site Runtime Alarms.CurrentAsync() accessor — Route.To(code).Call("SimpleAlarmStatus", new { ... }). Alarm state is per-instance and lives on the site's Instance Actor; keeping the read inside a site script means the filtering/shaping happens where the data is, and central stays a thin router. This is the shape the MES alarm-status endpoints use (docs/plans/2026-06-30-mes-alarm-status-api.md §5.1); see Component-SiteRuntime.md → Alarm State (read-only) for the accessor.

Input/Output

• Input parameters are available as defined in the method definition.
• Return value construction matching the defined return structure.

Parameter Access

• Parameters["key"] — Raw dictionary access.
• Parameters.Get<T>("key") — Typed access (same API as site runtime scripts). See Site Runtime component for full type support.

Database Access

Inbound API scripts may read from and write to the configuration / machine-data databases through the curated, scoped Database helper — InboundDatabaseHelper, exposed as InboundScriptContext.Database. This is the "dedicated, scoped helper added as an explicit design change" that the script trust model requires: scripts are never handed a raw SqlConnection, and never reference System.Data (the ForbiddenApiChecker, delegating to the shared ScriptAnalysis ScriptTrustValidator (#25), still statically bans System.IO, Process, Threading, Reflection, and raw network access — that is defence-in-depth static enforcement, not a true runtime sandbox).

The helper API (all asynchronous — scripts await them; bounded by the method timeout):

• await Database.QuerySingleAsync<T>("connectionName", sql, parameters) — first column of the first row as T (default if no rows).
• await Database.QueryAsync("connectionName", sql, parameters) — all rows as case-insensitive column→value dictionaries.
• await Database.ExecuteAsync("connectionName", sql, parameters) — run a write (INSERT/UPDATE/DELETE/DDL); returns rows affected. Writes are permitted — the move-in integration records results, not just reads them.

Containment rules (enforced, not advisory):

• Named connections only. connectionName selects one of the connections configured on the central database gateway (IDatabaseGateway); a script cannot supply an arbitrary connection string or reach a database the gateway is not configured for.
• SQL-injection protection. Statement text is authored by the (design-time) method script, but every request-derived value is passed via parameters and bound as a named @-prefixed SQL parameter — never string-concatenated into the command text. Request input therefore reaches the database only through parameter binding.
• Deadline-bound. Calls use the async ADO.NET path end-to-end (no pool-thread blocking) and honour the method deadline token, with a CommandTimeout backstop derived from the method timeout, so a slow query is bounded by the method timeout.

For everything else, scripts interact with the system through the curated Route and Parameters surfaces above.

Routing Behavior

• The Route.To() helper resolves the instance's site assignment from the configuration database and routes the request to the correct site cluster via the Communication Layer.
• The call is synchronous from the API caller's perspective — the API method blocks until the site responds or the method-level timeout is reached.
• If the target site is unreachable or the call times out, the call fails and the API returns an error to the caller. No store-and-forward buffering is used for inbound API calls.
• Every Route.To(...).Call(...) writes one always-on ScriptTrace.RouteCall audit row (central direct write, actor inbound-api, target {instance}.{script}, ExecutionId = the inbound request's, Extra.childExecutionId = the site run's id) — this row is what joins the central and site halves of a Flow Debugger Replay, so it is not gated. The routed request carries a Trace flag beside ParentExecutionId (RouteToCallRequest.Trace, RouteToSetAttributesRequest.Trace); it is set when the method has TraceAllCalls or the call is a Flow Debugger Live run, and only then does the site emit its own trace rows and return them inline on RouteToCallResponse.TraceEvents (with ChildExecutionId). Untraced calls are byte-identical to before.

Flow Debugger runs

The Flow Debugger (Central UI, design docs/plans/2026-09-05-flow-debugger-design.md) drives this component through three management commands handled by the per-node ManagementActor (Deployer role): FlowDebugRunMethodCommand executes a method exactly as an external caller would — ParameterValidator + SchemaRefResolver, then InboundScriptExecutor.ExecuteAsync(..., trace: true) with a RouteHelper trace sink collecting the RouteCall row and the site's inline events — and synthesizes the ApiInbound.InboundRequest row itself with the operator as actor (flow-debugger source tag) because no HTTP request passed through AuditWriteMiddleware; it refuses on a standby node with STANDBY_NODE (IActiveNodeGate), so the script runs only where a real call would. FlowDebugRunValueChangeCommand writes an attribute through the traced routed-write path and polls the site (PullTraceEvents) for the run the change spawned; FlowDebugReplayCommand reads the chain back from the central Audit Log. The result (FlowDebugRunResult) carries the audit events inline — the graph is a central-side projection, never a wire type.

Authentication Details

• The API key token is accepted from either the Authorization: Bearer <token> header (preferred) or the X-API-Key: <token> header (alternate). Authorization takes precedence when both are present.
• The token format is sbk_<keyId>_<secret>. Both header forms are verified by the same IApiKeyVerifier from ZB.MOM.WW.Auth.ApiKeys.
• The system validates:
  1. The token is well-formed and the key ID exists in the key store.
  2. The key is enabled (not revoked).
  3. The key's provisioned scopes include the requested method name (case-sensitive).
• Failed authentication returns an appropriate HTTP error (401 Unauthorized for invalid/missing credentials; 403 Forbidden for a valid key not approved for the method). For enumeration-safety, both "method not found" and "key not in scope" return 403 with an identical body.

Standby forwarding

Both central nodes expose POST /api/{methodName}, but only the active node (the oldest Up member — ActiveNodeEvaluator.SelfIsOldestUp, the same verdict behind /health/active) executes method scripts and Route.To() calls; running them on a standby could race the active node or read stale singleton state. Before 2026-08-28 a standby simply answered 503 STANDBY_NODE, and "both nodes work" relied entirely on the load balancer probing /health/active — a caller addressing a node directly, or hitting the balancer inside its probe interval after a failover, got the 503.

A standby now relays the request to the active peer instead (InboundApiEndpointFilter → IStandbyRequestForwarder / HttpStandbyRequestForwarder):

• Configuration. ScadaBridge:InboundApi:StandbyForwardPeerEndpoints — the HTTP base URL(s) of the other central node(s), e.g. ["http://scadabridge-central-b:5000"] on central-a. Empty (default) keeps the old 503 behaviour. StandbyForwardTimeout (default 60 s) is the per-peer attempt deadline and must exceed the longest method timeout in use. Validated at startup (absolute http/https URLs, positive timeout). Note the standby discovers the active peer from appsettings, not from cluster state — the Akka member address carries the remoting port, not the HTTP port — so this is a static, restart-to-change list, like CentralGrpcEndpoints on sites.
• Pure relay. Method, path, query string, body and headers travel verbatim (hop-by-hop headers and Host excepted), so the active node authenticates the caller's own Authorization/X-API-Key credential, applies its own body cap and method timeout, and writes the audit row. The peer's status, Content-Type and body are replayed unchanged. The standby's AuditWriteMiddleware skips its own ApiInbound row for a relayed request (HttpContext.Items[RelayedToActiveItemKey]) so a call is audited once, where it ran.
• Order of checks. The body-size cap (413 BODY_TOO_LARGE) runs before the active-node check, so a standby never buffers or relays a body the active node would refuse. The body is buffered once on the standby (bounded by MaxRequestBodyBytes) so it can be re-sent to a second peer.
• Peer selection. The standby knows only that it is not active. It tries the peers in order and accepts the first response that is not a standby refusal; a peer answering 503 with X-ScadaBridge-Standby: 1, a connection failure, or a StandbyForwardTimeout expiry each move to the next peer. A 503 without the marker is the active node's own answer and is replayed, not retried. All peers failing → 503 ACTIVE_NODE_UNREACHABLE.
• One-hop loop guard. Every relayed request carries X-ScadaBridge-Forwarded: <relaying host>. A standby never relays a request that already carries it — it answers 503 STANDBY_NODE with the marker header — so a pair in which neither node is active (a transient during failover) bounces at most one hop and never ping-pongs. The active node ignores the header and serves the request normally.
• Still not a retry layer. The relay makes one HTTP attempt per peer and does not retry a peer that answered; at-least-once callers keep their own retry loop and the idempotent-method authoring note above still applies.

Error Handling

• Invalid API key → 401 Unauthorized.
• Valid key but not approved for method → 403 Forbidden.
• Invalid parameters → 400 Bad Request.
• Script execution failure → 500 Internal Server Error (with safe error message, no internal details exposed).
• Script errors are logged in the central audit/event system.

Dependencies

• Configuration Database (MS SQL): Stores method definitions, and — under Provider=SqlServer — API keys in the zbauth schema (see Key store provider above). Under the Sqlite default, API keys live in a per-node SQLite file instead.
• Communication Layer: Routes requests to sites when method implementations need site data.
• Security & Auth: API key validation (separate from LDAP/AD — API uses key-based auth).
• Configuration Database (via IAuditService): All API key and method definition changes are audit logged.
• Script Analysis (#25): ForbiddenApiChecker delegates to ScriptTrustValidator.FindViolations for the authoritative forbidden-API verdict during script compilation and validation.
• Audit Log (#23): Every inbound API request emits an ApiInbound.Completed row via ICentralAuditWriter from request middleware (non-blocking for the HTTP response). Request and response bodies are captured in full up to AuditLog:InboundMaxBytes (default 1 MiB) per the Audit Log Payload Capture Policy; redaction (headers + per-target body redactors) still applies before persistence.
• Cluster Infrastructure: method execution is hosted on the active central node and fails over with it; IActiveNodeGate (Host) tells the endpoint filter whether to serve locally or relay.

Interactions

• External Systems: Call the API with API keys.
• Communication Layer: API method scripts use this to reach sites.
• Site Runtime (Instance Actors, Script Actors): Routed calls execute on site Instance Actors via their Script Actors.
• Central UI: Administrator manages API keys; Designer manages method definitions.
• Configuration Database (via IAuditService): Configuration changes are audited.