# Component: Inbound API ## Purpose The Inbound API exposes a web API on the central cluster that external systems can call into. This is the reverse of the External System Gateway — where that component handles the SCADA system calling out to external systems, this component handles external systems calling in. It provides API key authentication, method-level authorization, and script-based method implementations. ## Location Central cluster only. Method scripts execute on the **active** central node; a **standby** central node accepts the same endpoint and relays the request to the active peer (see [Standby forwarding](#standby-forwarding)). Not available at site clusters. ## Responsibilities - Host a web API endpoint on the central cluster. - Authenticate inbound requests via API keys. - Route requests to the appropriate API method definition. - Enforce per-method API key authorization (only approved keys can call a given method). - Execute the C# script implementation for the called method. - Return structured responses to the caller. - Failover: API becomes available on the new active node after central failover; during the handover a standby relays to whichever peer is active, so a caller addressing either central node directly is served without waiting for the load balancer's health probe to catch up. ## API Key Management ### Storage - API keys are stored in the shared `ZB.MOM.WW.Auth.ApiKeys` library's store — **SQLite per node by default**, or the shared `zbauth` schema in the **configuration database (MS SQL)** on a central pair when `Provider=SqlServer` (see *Key store provider* below). Revoke/rotate/scope changes are cluster-wide immediately only under `SqlServer`; under `Sqlite` they apply to whichever node served the management call. ### Key store provider Config section `ScadaBridge:InboundApi:ApiKeyStore` (design 2026-09-02, register row 108): | Key | Default | Notes | |---|---|---| | `Provider` | `Sqlite` | `Sqlite` (per node) or `SqlServer` (shared across a central pair). Case-insensitive; an unrecognized value fails startup via `StartupValidator`. | | `SqlServer:ConnectionString` | `ScadaBridge:Database:ConfigurationDb` | Must be a literal or env override, **never** a `${secret:}` reference — the secrets store is itself SQL-backed, so resolving one through the other would be circular. | | `SqlServer:Schema` | `zbauth` | Owns `schema_version`, `api_keys`, `api_key_audit`. | | `SqlServer:CommandTimeout` | 30 s | | | `ImportFromSqlitePath` | the library's `SqlitePath` default | SQLite file imported **once**, at boot, when `Provider=SqlServer`; insert-where-absent, so both nodes of a pair may import their own file and the result is the union. On success the file is renamed `*.imported` (plus `-wal`/`-shm`); on failure the import logs Error and the node keeps booting on the keys already in SQL. | A Central node with ≥2 seed nodes still running `Provider=Sqlite` logs a boot-time Warning (not a hard fail — availability-first, matching the auto-down decision) naming this design, because a failover will otherwise 401 every inbound client until keys are re-issued. Both docker rigs (`docker/`, `docker-env2/`) run `Provider=SqlServer`; QA/prod are a separate deployment record. See `docs/operations/inbound-api-key-reissue.md` → *Central pairs* for the reissue/migration runbook and `docs/operations/sql/zbauth-schema.sql` for the DDL a DBA runs by hand when the app login lacks CREATE SCHEMA/TABLE rights. ### Key Properties - **Name/Label**: Human-readable identifier for the key (e.g., "MES-Production", "RecipeManager-Dev"). - **Key Value**: The secret key string used for authentication. - **Enabled/Disabled Flag**: Keys can be disabled without deletion. ### Management - Managed by users with the **Administrator** role via the Central UI. - All key changes (create, enable/disable, delete) are audit logged. ## API Method Definition ### Properties Each API method definition includes: - **Method Name**: Unique identifier and URL path segment for the endpoint. - **Approved API Keys**: List of API keys authorized to invoke this method. Requests from non-approved keys are rejected. - **Parameter Definitions**: Ordered list of input parameters, each with: - Parameter name. - Data type — the **extended type system** (Boolean, Integer, Float, String, plus the nestable Object and List; see [Extended Type System](#extended-type-system)). - Whether the parameter is required. - **Return Value Definition**: Structure of the response, with: - Field names and (extended-system) data types. Supports returning **lists of objects** and arbitrarily nested structures. - **Implementation Script**: C# script that executes when the method is called. Stored **inline** in the method definition. Follows standard C# authoring patterns but has no template inheritance — it is a standalone script tied to this method. - **Timeout**: Configurable per method. Defines the maximum time the method is allowed to execute (including any routed calls to sites) before returning a timeout error to the caller. - **TraceAllCalls** (Flow Debugger, 2026-09-05): when set, every call to this method runs its routed site scripts with the trace flag on, so their `ScriptRun` / `AttributeWrite` / `AttributeWait` audit rows (`AuditChannel.ScriptTrace`) are recorded for Replay. Default off — adds 7–10 audit rows per call (3 of them trace-only; a cached DB write and a notification account for the upper end), so it is meant for low-volume integration methods. Set from the method form's *Trace all calls* checkbox or `api-method create|update --trace-all-calls`. > **Authoring note — idempotent methods for at-least-once callers**: some external callers retry > on their own connect-failure/timeout failover loop and cannot tell "the first attempt actually > succeeded" from "the first attempt was lost". The updated `DelmiaNotifier` returns an uncertain > result instead of automatically replaying such attempts. Scripts exposed to other retrying > clients should still be idempotent on their > natural key so a duplicate delivery returns the same result without double-applying side effects. ### Management - Managed by users with the **Designer** role via the Central UI. - All method definition changes are audit logged. ## HTTP Contract ### URL Structure - All API calls use `POST /api/{methodName}`. - Method names map directly to URL path segments (e.g., method "GetProductionReport" → `POST /api/GetProductionReport`). - All calls are POST — these are RPC-style script invocations, not RESTful resource operations. ### Authentication Header - The API key token (`sbk__`) is accepted from **either** of two headers: - `Authorization: Bearer sbk__` — the preferred form. - `X-API-Key: sbk__` — alternate form for callers that cannot set the Authorization header. - When both headers are present, `Authorization` takes precedence. - Both forms are processed by the same `IApiKeyVerifier` from the shared `ZB.MOM.WW.Auth.ApiKeys` library; the verifier strips the optional `Bearer ` prefix and performs the same peppered-HMAC constant-time verification regardless of which header was used. ### Key rotation The `sbk__` design already supports zero-downtime rotation — a caller can hold two valid keys at once, so there is no moment where the old key must be cut off before the new one works. The procedure: 1. **Create a new key** with the same scope set as the key being rotated — `security api-key create --name --methods ` (CLI `security api-key` group). Scopes are **copied at creation only**: the operator must pass the same `--methods` list the old key was approved for; the CLI does not read the old key's scopes for you, so scope drift between the pair is an operator error, not something the system detects. 2. **Deliver the new token to the caller and confirm cutover.** The response from `create` surfaces the one-time bearer token (`PrintCreatedKey` — it is shown once and cannot be retrieved again, so capture it immediately). Both the old and new keys verify successfully during this window — that overlap is intended, not a bug, and gives the caller time to redeploy with the new token without a synchronized flag-day cutover. Confirm the caller has actually switched by checking the audit log's Actor field (recorded per call), which shows which key ID is serving live traffic. 3. **Disable — don't delete — the old key** once cutover is confirmed: `security api-key update --key-id --enabled false`. Disabling is reversible (`--enabled true` re-enables it) if the cutover needs to be rolled back, whereas deletion is not. 4. **Delete the old key after a quarantine period** (an operator-judgment window to be sure nothing is still presenting the old token): `security api-key delete --key-id `. ### Request Format - Content-Type: `application/json`. - Parameters are top-level JSON fields in the request body matching the method's parameter definitions: ```json { "siteId": "SiteA", "startDate": "2026-03-01", "endDate": "2026-03-16" } ``` ### Response Format - **Success (200)**: The response body is the method's return value as JSON, with fields matching the return value definition: ```json { "siteName": "Site Alpha", "totalUnits": 14250, "lines": [ { "lineName": "Line-1", "units": 8200, "efficiency": 92.5 }, { "lineName": "Line-2", "units": 6050, "efficiency": 88.1 } ] } ``` - **Failure (4xx/5xx)**: The response body is an error object carrying a human-readable `error` and a stable, machine-readable `code`: ```json { "error": "Site unreachable", "code": "SITE_UNREACHABLE" } ``` - HTTP status codes distinguish success from failure — no envelope wrapper. - **Error codes.** Every failure body carries a `code` an external integrator can branch on (the `error` message is for humans and may change). The two authorization negatives — unknown method and key-not-in-scope — deliberately share one code (`NOT_APPROVED`) and one body so a caller cannot enumerate which method names exist: | `code` | HTTP status | Meaning | |--------|-------------|---------| | `UNAUTHORIZED` | 401 | Missing/invalid API key (every auth-stage failure maps here — no stage leak). | | `NOT_APPROVED` | 403 | Method not found **or** key not in scope — indistinguishable by design. | | `UNSUPPORTED_MEDIA_TYPE` | 415 | Request carried a body — declared by `Content-Length` or `Transfer-Encoding` (chunked) — with a non-JSON `Content-Type`. A body with **no** `Content-Type` is still parsed leniently as JSON, and a chunked (no `Content-Length`) body is treated identically to a fixed-length one. | | `INVALID_JSON` | 400 | Request body was not valid JSON. | | `VALIDATION_FAILED` | 400 | Parameter validation failed (missing required field, wrong type, undeclared field). | | `BODY_TOO_LARGE` | 413 | Request body exceeded `MaxRequestBodyBytes`. | | `STANDBY_NODE` | 503 | Request hit a standby central node that could not relay it: no `StandbyForwardPeerEndpoints` configured, or the request was already relayed once (loop guard — neither node is active, e.g. mid-failover). Carries the `X-ScadaBridge-Standby: 1` response header. | | `ACTIVE_NODE_UNREACHABLE` | 503 | Every configured peer refused before execution or failed with proven name-resolution/connection-refusal errors. | | `OUTCOME_UNKNOWN` | 502 | A relay timed out or lost its response after possible execution; it does not replay the request. Verify the operation before retrying. | | `TIMEOUT` | 500 | The method script exceeded its execution timeout. | | `SITE_UNREACHABLE` | 500 | The target site did not respond to the routed request within the routing timeout (no contact / down / partitioned) — typed from the routed Ask expiring (`AskTimeoutException`), never inferred from an error-message substring. A `Success=false` response means the site answered and is classified `SCRIPT_ERROR`. | | `SCRIPT_COMPILE_FAILED` | 500 | The method's script failed to compile (returned consistently on every node until a compiling version is saved). | | `SCRIPT_ERROR` | 500 | Catch-all for any other script-execution failure (details are logged centrally, never leaked to the caller — the exception's `ToString()` is captured on the audit row as `ErrorDetail`, never in the response body). | ### Extended Type System - API method parameter and return type definitions support an **extended type system** beyond the four template attribute types (Boolean, Integer, Float, String): - **Object**: A named structure with typed fields. Supports nesting. - **List**: An ordered collection of objects or primitive types. - This allows complex request/response structures (e.g., an object containing properties and a list of nested objects). - Template attributes retain the simpler four-type system. The extended types apply only to Inbound API method definitions and External System Gateway method definitions. #### Type Definition Format & Nested Validation - Parameter and return type definitions are persisted as **JSON Schema** (the canonical format produced by the Central UI schema builder; see the `MigrateParametersToJsonSchema` migration). An object declares its fields via `properties` (+ a `required` array); a list declares its element type via `items`. The legacy flat-array form (`[{name,type,required,itemType?}]`) is still accepted on read for transition safety. - Validation is **recursive and type-aware** for the extended types (request parameters and script return values alike, via a single shared engine so the two cannot drift): - **Object**: each declared field's value is validated against its declared (possibly nested) type; a missing required field and a present-but-wrong type are both reported. - **List**: every element is validated against the declared element type (recursing into nested objects/lists). A list whose element type is left undeclared (`array` without `items`) is shape-checked only. - **Scalars at any depth** are checked against the extended type. - Errors are **path-qualified** (e.g. `order.items[2].quantity`) so the caller can locate the offending field. - **Undeclared fields are rejected** at every level (consistent with the top-level "unexpected parameter" rejection): an object that declares its fields rejects any field not in its `properties`, so a typo'd field name surfaces as a `400`/error rather than being silently ignored. A bare object schema with no declared fields (`{"type":"object"}`) stays shape-only and accepts any fields. - A JSON `null` value satisfies any declared type (a present-but-null field is allowed); only the **absence** of a required field is an error. ## Script Compilation & Hot-Reload API method scripts are compiled at central startup — all method definitions are loaded from the configuration database and compiled into in-memory delegates. ### Update Workflow - Updating a method via the CLI (`api-method update --id --code '...'`) or Management API triggers immediate recompilation (`CompileAndRegister`). The updated script takes effect on the next API call — no node restart is required. - Creating a new method after startup: if the method is created but not yet compiled, the first invocation triggers lazy (on-demand) compilation. > **Compile failures on save are non-fatal but surfaced, and the DB row is authoritative.** The persisted `ApiMethod.Script` row is the single source of truth: on every request `ExecuteAsync` revision-checks the cached delegate against the freshly-fetched script and recompiles in place when it differs, so a saved-but-broken script does **not** leave a previously registered handler silently serving. The change still persists, so the Create/Update Management result carries a top-level **`compileWarning`** (null when the script compiled) listing the Roslyn/trust-model errors. On both **create** and **update**, once a non-compiling script is saved the method returns a compilation error (`HTTP 500`) **consistently on every central node** until a compiling version is saved — there is no node-divergent "old delegate still works here" state. This means a saved script that does not compile is **not silently a no-op** — the caller (CLI / Central UI) sees the warning, and the real diagnostics are also logged centrally. ### Direct SQL Warning > **Direct SQL edits to API method scripts are now picked up automatically.** Because the DB row is authoritative and `ExecuteAsync` revision-checks the cached delegate against the freshly-fetched `ApiMethod.Script` on every request, a script changed directly in the database takes effect on the **next call** — no node restart and no Management round-trip is required. A broken direct edit surfaces the same way as a broken save: the method returns a `HTTP 500` compilation error consistently on all central nodes until a compiling version is stored. Editing via the CLI, Management API, or Central UI is still preferred because those paths validate the script up front and return the `compileWarning`, whereas a direct SQL edit is only diagnosed on the next request (and in the central log). --- ## API Call Logging - **Every request — success or failure — emits one `ApiInbound.Completed` row** to `ICentralAuditWriter` from request middleware before the HTTP response is flushed. The row captures the API key **name** (never the key material), remote IP, user-agent, response status, duration, and the request/response bodies. Bodies are captured in full up to `AuditLog:InboundMaxBytes` (default 1 MiB); `PayloadTruncated = 1` only when that ceiling is hit. Header redaction and per-target body redactors still apply (see Component-AuditLog.md, Payload Capture Policy). This supersedes the earlier failures-only stance: operational API traffic is now part of the centralized audit log, so configuration changes and call activity share a single retention/query surface. - Script execution errors (500 responses) remain captured on the same `ApiInbound.Completed` row (response status + error fields) rather than emitting a separate failure-only event. - **Fail-soft semantics (fire-and-forget).** The audit write is **fire-and-forget** (`InboundAPI-018`): `WriteAsync` is called without `await` so the user-facing HTTP response is never blocked or delayed by the audit path. Asynchronous faults are observed via a `ContinueWith` continuation that logs at Warning and increments `CentralAuditWriteFailures` (see Health Monitoring #11) rather than being silently dropped. A failed audit append never turns a successful API call into an error returned to the caller. - No rate limiting — this is a private API in a controlled industrial environment with a known set of callers. Misbehaving callers are handled operationally (disable the API key). ## Request Flow ```mermaid %%{init: {'theme':'base', 'themeVariables': {'textColor':'#111111','lineColor':'#555555','edgeLabelBackground':'#ffffff','fontSize':'15px'}}}%% flowchart TD ext(["External System"]) api["Inbound API (Central)"] s1["1. Extract API key from request"] s2["2. Validate key exists and is enabled"] s3["3. Resolve method by name"] s4["4. Check API key is in method's approved list"] s5["5. Validate and deserialize parameters"] s6["6. Execute implementation script
(subject to method timeout)"] s7["7. Serialize return value"] s8["8. Return response"] ext --> api api --> s1 s1 --> s2 s2 --> s3 s3 --> s4 s4 --> s5 s5 --> s6 s6 --> s7 s7 --> s8 classDef start fill:#d5e8d4,stroke:#82b366,color:#111111; classDef proc fill:#dae8fc,stroke:#6c8ebf,color:#111111; classDef dec fill:#fff2cc,stroke:#d6b656,color:#111111; classDef warn fill:#ffe6cc,stroke:#d79b00,color:#111111; classDef alt fill:#e1d5e7,stroke:#9673a6,color:#111111; class ext start class api proc class s1,s2,s3,s4,s5,s7 dec class s6 alt class s8 warn ``` ## Implementation Script Capabilities The C# script that implements an API method executes on the central cluster. Unlike instance scripts at sites, inbound API scripts run on central and can interact with **any instance at any site** through a routing API. Inbound API scripts **cannot** call shared scripts directly — shared scripts are deployed to sites only and execute inline in Script Actors. To execute logic on a site, use `Route.To().Call()`. ### Script Runtime API #### Instance Routing - `Route.To("instanceUniqueCode").Call("scriptName", parameters)` — Invoke a script on a specific instance at any site. Central routes the call to the appropriate site via the Communication Layer. The call reaches the target Instance Actor's Script Actor, which spawns a Script Execution Actor to execute the script. The return value flows back to the calling API script. - `Route.To("instanceUniqueCode").GetAttribute("attributeName")` — Read a single attribute value from a specific instance at any site. - `Route.To("instanceUniqueCode").GetAttributes("attr1", "attr2", ...)` — Read multiple attribute values in a **single call**, returned as a dictionary of name-value pairs. - `Route.To("instanceUniqueCode").SetAttribute("attributeName", value)` — Write a single attribute value on a specific instance at any site. - `Route.To("instanceUniqueCode").SetAttributes(dictionary)` — Write multiple attribute values in a **single call**, accepting a dictionary of name-value pairs. - `Route.To("instanceUniqueCode").WaitForAttribute("attributeName", targetValue, timeout)` — Wait, event-driven, until an attribute on a specific instance at any site reaches `targetValue` (value-equality only across the wire), bounded by `timeout`. Returns `true` if matched within the timeout, `false` if it timed out. **The wait is bounded by its own `timeout`, not the generic method-level timeout** — this is the one routed call that may legitimately outlive the method timeout (the site enforces `timeout` and returns `false` when it elapses). A client disconnect still cancels the wait. This is the deliberate exception to the rule below that routed calls inherit the method-level timeout (see "Routing Behavior"): a long event-driven wait is the explicit reason `timeout` governs here. > **Reading alarm state via a routed script.** There is deliberately **no** `Route.To(...).GetAlarms(...)` verb. An inbound method that needs a machine's current alarms resolves the instance and calls a **site script** that uses the Site Runtime `Alarms.CurrentAsync()` accessor — `Route.To(code).Call("SimpleAlarmStatus", new { ... })`. Alarm state is per-instance and lives on the site's Instance Actor; keeping the read inside a site script means the filtering/shaping happens where the data is, and central stays a thin router. This is the shape the MES alarm-status endpoints use (`docs/plans/2026-06-30-mes-alarm-status-api.md` §5.1); see [Component-SiteRuntime.md](Component-SiteRuntime.md) → *Alarm State (read-only)* for the accessor. #### Input/Output - **Input parameters** are available as defined in the method definition. - **Return value** construction matching the defined return structure. #### Parameter Access - `Parameters["key"]` — Raw dictionary access. - `Parameters.Get("key")` — Typed access (same API as site runtime scripts). See Site Runtime component for full type support. #### Database Access Inbound API scripts may read from and write to the configuration / machine-data databases through the **curated, scoped `Database` helper** — `InboundDatabaseHelper`, exposed as `InboundScriptContext.Database`. This is the "dedicated, scoped helper added as an explicit design change" that the script trust model requires: scripts are **never** handed a raw `SqlConnection`, and never reference `System.Data` (the `ForbiddenApiChecker`, delegating to the shared `ScriptAnalysis` `ScriptTrustValidator` (#25), still statically bans `System.IO`, `Process`, `Threading`, `Reflection`, and raw network access — that is defence-in-depth static enforcement, not a true runtime sandbox). The helper API (all asynchronous — scripts `await` them; bounded by the method timeout): - `await Database.QuerySingleAsync("connectionName", sql, parameters)` — first column of the first row as `T` (default if no rows). - `await Database.QueryAsync("connectionName", sql, parameters)` — all rows as case-insensitive column→value dictionaries. - `await Database.ExecuteAsync("connectionName", sql, parameters)` — run a write (INSERT/UPDATE/DELETE/DDL); returns rows affected. **Writes are permitted** — the move-in integration records results, not just reads them. **Every helper call is audited (2026-09-06).** Each `QuerySingle` / `Query` / `Execute` writes one `AuditChannel.DbOutbound` / `AuditKind.DbWrite` row through `ICentralAuditWriter` — `Status` `Delivered` or `Failed`, `Target` = the connection name, `Actor` = `api:{methodName}`, the request's `ExecutionId`, `DurationMs`, the SQL + bound parameter names/values as `RequestSummary` (capped and redacted at write time by the shared audit redactor exactly like a site `AuditingDbCommand` row), `Extra` = `{"op":"read","rowsReturned":n}` / `{"op":"write","rowsAffected":n}`, `SourceNode` from `INodeIdentityProvider`. Best-effort: an audit fault never reaches the script, and the original exception still rethrows. Under a Flow Debugger Live run the same instance is also handed to the route's trace sink so the call appears as a `DbCall` node on the External lane (Component-CentralUI.md → Flow Debugger). This closed the outbound-DB gap Component-AuditLog.md's scope table used to list for central method scripts. Containment rules (enforced, not advisory): - **Named connections only.** `connectionName` selects one of the connections configured on the central database gateway (`IDatabaseGateway`); a script cannot supply an arbitrary connection string or reach a database the gateway is not configured for. - **SQL-injection protection.** Statement text is authored by the (design-time) method script, but every request-derived **value** is passed via `parameters` and bound as a named `@`-prefixed SQL parameter — never string-concatenated into the command text. Request input therefore reaches the database only through parameter binding. - **Deadline-bound.** Calls use the async ADO.NET path end-to-end (no pool-thread blocking) and honour the method deadline token, with a `CommandTimeout` backstop derived from the method timeout, so a slow query is bounded by the method timeout. For everything else, scripts interact with the system through the curated `Route` and `Parameters` surfaces above. #### Logging (`Log`) Method scripts get the same leveled `Log` global site scripts do (design `docs/plans/2026-09-09-flow-debugger-step-logs-design.md` D4/D5) — `InboundScriptContext.Log`, backed by `InboundScriptLogger`: - `Log.Debug(message, exception?)` / `Log.Info` / `Log.Warn` / `Log.Error` — synchronous, `void`, never throws into the script. - **Always** forwards to the central node's `ILogger`. A `ScriptTrace` / `ScriptLog` audit row is written **only when the request is traced** (a Flow Debugger Live run, or a real call to a method with `TraceAllCalls`), through the executor's inline trace sink and `ICentralAuditWriter`; an ordinary call pays a log entry and nothing more. - Lines with no explicit step id are attributed to the request's anchor row (the `InboundRequest` row, or the debugger's synthesized `RouteCall` root), so they render on the inbound node. - **Bounded:** `ScadaBridge:InboundApi:MaxLogLinesPerRun` (default 200, validated `> 0`) caps the rows one request may emit, messages are clipped at 2 000 characters, and one closing `warn` row names the dropped count. #### Script exception detail `InboundScriptResult` carries an internal `ErrorDetail` (`ex.ToString()` of the fault the method script threw). It is written to the request's audit row — by `FlowDebugRunner` for a Live run, and on the ordinary HTTP path via `HttpContext.Items[AuditWriteMiddleware.ErrorDetailItemKey]` — and is projected onto `FlowStep.ErrorDetail` for the Flow Debugger's exception block. **It is never returned to the HTTP caller:** the response body stays the generic `SCRIPT_ERROR` envelope, unchanged. **Note the asymmetry with the site half, which is easy to get wrong:** the inbound row's `errorDetail` is written on **every failing method request**, traced or not — the middleware stamps it whenever the script faulted, so it is not gated on `TraceAllCalls` or on a Live run (those only decide whether the row is later *readable as a Flow Debugger step*). Its size is bounded by the redactor's truncation stage, which for an `ApiInbound` row selects `AuditLog:InboundMaxBytes` (1 MiB default — `SelectCap` gives this channel its own ceiling instead of the 64 KB `ErrorCapBytes` other channels' error rows get), so a deep stack is truncated there rather than dropped. The **site** `ScriptRun` row's `errorDetail`, by contrast, is **traced-only** — an untraced site script run emits no `ScriptTrace` row at all, so there is nothing to carry it; that row, being a `ScriptTrace` error row, is the one bounded at the 64 KB `ErrorCapBytes`. ### Routing Behavior - The `Route.To()` helper resolves the instance's site assignment from the configuration database and routes the request to the correct site cluster via the Communication Layer. - The call is **synchronous from the API caller's perspective** — the API method blocks until the site responds or the **method-level timeout** is reached. - If the target site is unreachable or the call times out, the call fails and the API returns an error to the caller. No store-and-forward buffering is used for inbound API calls. - **Every `Route.To(...).Call(...)` writes one always-on `ScriptTrace.RouteCall` audit row** (central direct write, actor `inbound-api`, target `{instance}.{script}`, `ExecutionId` = the inbound request's, `Extra.childExecutionId` = the site run's id) — this row is what joins the central and site halves of a Flow Debugger Replay, so it is not gated. The routed request carries a `Trace` flag beside `ParentExecutionId` (`RouteToCallRequest.Trace`, `RouteToSetAttributesRequest.Trace`); it is set when the method has `TraceAllCalls` or the call is a Flow Debugger Live run, and only then does the site emit its own trace rows and return them inline on `RouteToCallResponse.TraceEvents` (with `ChildExecutionId`). Untraced calls are byte-identical to before. **The `RouteCall` row honours the method's `SkipBodyCapture`** — the executor builds the helper with `RouteHelper.WithBodyCapture(!skipBodyCapture)`, so a method that opts out of body capture no longer leaks its arguments and reply through the always-on trace row; the row itself is still written, with metadata only (arch-review F11). ## Flow Debugger runs The Flow Debugger (Central UI, design `docs/plans/2026-09-05-flow-debugger-design.md`; script entry point `docs/plans/2026-09-08-flow-debugger-script-entry-design.md`) drives this component through four management commands handled by the per-node `ManagementActor` (Deployer role): `FlowDebugRunMethodCommand` executes a method exactly as an external caller would — `ParameterValidator` + `SchemaRefResolver`, then `InboundScriptExecutor.ExecuteAsync(..., trace: true)` with a `RouteHelper` trace sink collecting the `RouteCall` row and the site's inline events — and **synthesizes the `ApiInbound.InboundRequest` row itself** with the operator as actor (`flow-debugger` source tag) because no HTTP request passed through `AuditWriteMiddleware`; it refuses on a standby node with `STANDBY_NODE` (`IActiveNodeGate`), so the script runs only where a real call would. `FlowDebugRunValueChangeCommand` writes an attribute through the traced routed-write path and polls the site (`PullTraceEvents`) for the run the change spawned. `FlowDebugRunScriptCommand` (2026-09-08) resolves the named script on the instance through the flattening pipeline and branches on its trigger kind: a write kind (ValueChange/Conditional) reuses the same traced routed-write + `PullTraceEvents` poll as `FlowDebugRunValueChangeCommand`, using the attribute derived from the script's own trigger configuration; a call kind (Call/Interval/Expression/no trigger) routes a `RouteToCallRequest(Trace: true)` through `IInstanceRouter` **directly** rather than through `RouteHelper` — this keeps exactly one operator-attributed `RouteCall` root row instead of `RouteHelper`'s own always-on, actor-less one — and returns the site's inline trace rows through `FlowTraceCollector`; its error codes are `NOT_FOUND` (unknown script or instance), `VALIDATION_FAILED` (missing write-kind value, or call-kind parameters that fail `ParameterValidator`), `CALL_FAILED` (the routed call completed but the site reported failure), and `TIMEOUT` (no response within the script's execution timeout). `FlowDebugReplayCommand` reads the chain back from the central Audit Log and therefore accepts the wider `AuthorizationPolicies.FlowDebuggerAccessRoles` set (Deployer, Administrator, Viewer) while the three run commands stay Deployer-only (arch-review F3). The result (`FlowDebugRunResult`) carries the audit events inline — the graph is a central-side projection, never a wire type. - **`SchemaRefResolver` is `public` for the runner's benefit.** It was internal to the HTTP path; the Flow Debugger's `FlowDebugRunMethodCommand` validates its operator-supplied parameters through exactly the same `ParameterValidator` + `SchemaRefResolver` pair a real HTTP call uses, so a Live run cannot accept an input the endpoint would reject. Keep the two paths on the one resolver — a second, debugger-local validator would silently fork the contract. - **Site scope is enforced at route time on a Live run** (arch-review F2). `RouteHelper.WithSiteScope(IReadOnlySet? permittedSiteIdentifiers)` carries the operator's permitted **site identifiers**; `RouteTarget.ResolveSiteAsync` throws `RouteSiteScopeException` before any site traffic when the resolved target site is outside that set, and `InboundScriptExecutor` maps it to error code **`SITE_SCOPE_DENIED`**. `FlowDebugRunner` translates that code into a `SiteScopeViolationException` (`ManagementUnauthorized`) *after* writing the inbound audit row, so the denied attempt is recorded under the operator's name. Administrators and unscoped users get a null scope (unrestricted), and the HTTP endpoint passes no scope at all — production routing is unchanged. ## Authentication Details - The API key token is accepted from **either** the `Authorization: Bearer ` header (preferred) **or** the `X-API-Key: ` header (alternate). `Authorization` takes precedence when both are present. - The token format is `sbk__`. Both header forms are verified by the same `IApiKeyVerifier` from `ZB.MOM.WW.Auth.ApiKeys`. - The system validates: 1. The token is well-formed and the key ID exists in the key store. 2. The key is enabled (not revoked). 3. The key's provisioned scopes include the requested method name (case-sensitive). - Failed authentication returns an appropriate HTTP error (401 Unauthorized for invalid/missing credentials; 403 Forbidden for a valid key not approved for the method). For enumeration-safety, both "method not found" and "key not in scope" return 403 with an identical body. ## Standby forwarding Both central nodes expose `POST /api/{methodName}`, but only the active node (the oldest Up member — `ActiveNodeEvaluator.SelfIsOldestUp`, the same verdict behind `/health/active`) executes method scripts and `Route.To()` calls; running them on a standby could race the active node or read stale singleton state. Before 2026-08-28 a standby simply answered 503 `STANDBY_NODE`, and "both nodes work" relied entirely on the load balancer probing `/health/active` — a caller addressing a node directly, or hitting the balancer inside its probe interval after a failover, got the 503. A standby now **relays** the request to the active peer instead (`InboundApiEndpointFilter` → `IStandbyRequestForwarder` / `HttpStandbyRequestForwarder`): - **Configuration.** `ScadaBridge:InboundApi:StandbyForwardPeerEndpoints` — the HTTP base URL(s) of the *other* central node(s), e.g. `["http://scadabridge-central-b:5000"]` on central-a. Empty (default) keeps the old 503 behaviour. `StandbyForwardTimeout` (default 60 s) is the total relay budget and should exceed the longest method timeout plus network overhead. Validated at startup (absolute http/https URLs, positive timeout). Note the standby discovers the active peer from **appsettings**, not from cluster state — the Akka member address carries the remoting port, not the HTTP port — the options snapshot is read per relay. - **Pure relay.** Method, path, query string, body and credentials travel unchanged; hop-by-hop/Host/forwarding-provenance headers are excluded and authenticated provenance is rebuilt, so the **active node** authenticates the caller's own `Authorization`/`X-API-Key` credential, applies its own body cap and method timeout, and writes the audit row. The peer's status, `Content-Type` and body are replayed unchanged. The serving node owns its `ApiInbound` row; the originating standby retains a failure row when no peer serves or the outcome is uncertain. Verified internal standby refusals are not duplicated. - **Order of checks.** The body-size cap (HTTP 413) runs before audit reads and the active-node check, so a standby never buffers or relays a body the active node would refuse. The body is buffered once on the standby (bounded by `MaxRequestBodyBytes`) so it can be re-sent to a second peer. - **Peer selection.** The standby knows only that it is *not* active. It tries the peers in order and accepts the first response that is not a standby refusal; a peer answering 503 with `X-ScadaBridge-Standby: 1`, proven name-resolution failure, or connection refusal permits trying the next peer. Timeout or ambiguous response loss returns 502 `OUTCOME_UNKNOWN` without replay. A 503 *without* the marker is the active node's own answer and is replayed, not retried. All peers failing → 503 `ACTIVE_NODE_UNREACHABLE`. - **One-hop loop guard.** Every relayed request carries `X-ScadaBridge-Forwarded: `. A standby never relays a request that already carries it — it answers 503 `STANDBY_NODE` with the marker header — so a pair in which neither node is active (a transient during failover) bounces at most one hop and never ping-pongs. The active node ignores the header and serves the request normally. - **Still not a retry layer.** The relay makes one HTTP attempt per peer and does not retry a peer that answered; at-least-once callers keep their own retry loop and the idempotent-method authoring note above still applies. ## Error Handling - Invalid API key → 401 Unauthorized. - Valid key but not approved for method → 403 Forbidden. - Invalid parameters → 400 Bad Request. - Script execution failure → 500 Internal Server Error (with safe error message, no internal details exposed). - Script errors are logged in the central audit/event system. ## Dependencies - **Configuration Database (MS SQL)**: Stores method definitions, and — under `Provider=SqlServer` — API keys in the `zbauth` schema (see *Key store provider* above). Under the `Sqlite` default, API keys live in a per-node SQLite file instead. - **Communication Layer**: Routes requests to sites when method implementations need site data. - **Security & Auth**: API key validation (separate from LDAP/AD — API uses key-based auth). - **Configuration Database (via IAuditService)**: All API key and method definition changes are audit logged. - **Script Analysis (#25)**: `ForbiddenApiChecker` delegates to `ScriptTrustValidator.FindViolations` for the authoritative forbidden-API verdict during script compilation and validation. - **Audit Log (#23)**: Every inbound API request emits an `ApiInbound.Completed` row via `ICentralAuditWriter` from request middleware (non-blocking for the HTTP response). Request and response bodies are captured in full up to `AuditLog:InboundMaxBytes` (default 1 MiB) per the Audit Log Payload Capture Policy; redaction (headers + per-target body redactors) still applies before persistence. - **Cluster Infrastructure**: method execution is hosted on the active central node and fails over with it; `IActiveNodeGate` (Host) tells the endpoint filter whether to serve locally or relay. ## Interactions - **External Systems**: Call the API with API keys. - **Communication Layer**: API method scripts use this to reach sites. - **Site Runtime (Instance Actors, Script Actors)**: Routed calls execute on site Instance Actors via their Script Actors. - **Central UI**: Administrator manages API keys; Designer manages method definitions. - **Configuration Database (via IAuditService)**: Configuration changes are audited. ### Relay audit ownership, privacy, and request limits The inbound body limit is installed before audit buffering. Declared oversized bodies and unknown-length HTTP/1.1 or HTTP/2 bodies are rejected with HTTP 413; enabling audit capture cannot relax the route limit. The relay also bounds its buffered copy explicitly. A serving peer owns the inbound audit for its success or method failure. The originating standby writes a correlated failure if no peer serves the request, including an uncertain lost response. Authenticated internal standby refusals do not create duplicate inbound rows. Audit writes remain best effort; ownership is not a durable audit receipt. Relays overwrite caller-supplied provenance with an HMAC signed envelope using a domain-separated key derived from `ApiKeyPepper`. Both centrals must share that server secret. The envelope binds the original verified remote address, execution ID, timestamp, method, path/query, content type, caller credentials, and bounded body digest. The receiver accepts it for at most five minutes and only when the request binding matches. Unsigned loop markers and forwarded-address headers confer no identity trust; missing or invalid proof uses the receiver's verified remote address. Forwarding remains one hop. Use TLS between peers when network confidentiality is required. `StandbyForwardTimeout` is one budget across permitted attempts and response consumption. Only explicit pre-execution standby rejection, name-resolution failure, or connection refusal permits another peer. Timeout or response loss returns HTTP 502 `OUTCOME_UNKNOWN`: verify the operation before manually retrying because it may already have applied. This avoids blindly replaying non-idempotent methods. `SkipBodyCapture` suppresses request/response summaries and all exception message/detail text, including stashed script exceptions; retain status/category and request metadata for diagnosis. Error text can quote payload values and is subject to the same privacy boundary as the body.