ScadaBridge/docs/requirements/Component-InboundAPI.md

# Component: Inbound API

## Purpose

The Inbound API exposes a web API on the central cluster that external systems can call into. This is the reverse of the External System Gateway — where that component handles the SCADA system calling out to external systems, this component handles external systems calling in. It provides API key authentication, method-level authorization, and script-based method implementations.

## Location

Central cluster only. Method scripts execute on the **active** central node; a **standby** central node accepts the same endpoint and relays the request to the active peer (see [Standby forwarding](#standby-forwarding)). Not available at site clusters.

## Responsibilities

- Host a web API endpoint on the central cluster.
- Authenticate inbound requests via API keys.
- Route requests to the appropriate API method definition.
- Enforce per-method API key authorization (only approved keys can call a given method).
- Execute the C# script implementation for the called method.
- Return structured responses to the caller.
- Failover: API becomes available on the new active node after central failover; during the handover a standby relays to whichever peer is active, so a caller addressing either central node directly is served without waiting for the load balancer's health probe to catch up.

## API Key Management

### Storage
- API keys are stored in the shared `ZB.MOM.WW.Auth.ApiKeys` library's store — **SQLite per node by
  default**, or the shared `zbauth` schema in the **configuration database (MS SQL)** on a central
  pair when `Provider=SqlServer` (see *Key store provider* below). Revoke/rotate/scope changes are
  cluster-wide immediately only under `SqlServer`; under `Sqlite` they apply to whichever node served
  the management call.

### Key store provider

Config section `ScadaBridge:InboundApi:ApiKeyStore` (design 2026-09-02, register row 108):

| Key | Default | Notes |
|---|---|---|
| `Provider` | `Sqlite` | `Sqlite` (per node) or `SqlServer` (shared across a central pair). Case-insensitive; an unrecognized value fails startup via `StartupValidator`. |
| `SqlServer:ConnectionString` | `ScadaBridge:Database:ConfigurationDb` | Must be a literal or env override, **never** a `${secret:}` reference — the secrets store is itself SQL-backed, so resolving one through the other would be circular. |
| `SqlServer:Schema` | `zbauth` | Owns `schema_version`, `api_keys`, `api_key_audit`. |
| `SqlServer:CommandTimeout` | 30 s | |
| `ImportFromSqlitePath` | the library's `SqlitePath` default | SQLite file imported **once**, at boot, when `Provider=SqlServer`; insert-where-absent, so both nodes of a pair may import their own file and the result is the union. On success the file is renamed `*.imported` (plus `-wal`/`-shm`); on failure the import logs Error and the node keeps booting on the keys already in SQL. |

A Central node with ≥2 seed nodes still running `Provider=Sqlite` logs a boot-time Warning (not a hard
fail — availability-first, matching the auto-down decision) naming this design, because a failover
will otherwise 401 every inbound client until keys are re-issued. Both docker rigs (`docker/`,
`docker-env2/`) run `Provider=SqlServer`; QA/prod are a separate deployment record. See
`docs/operations/inbound-api-key-reissue.md` → *Central pairs* for the reissue/migration runbook and
`docs/operations/sql/zbauth-schema.sql` for the DDL a DBA runs by hand when the app login lacks
CREATE SCHEMA/TABLE rights.

### Key Properties
- **Name/Label**: Human-readable identifier for the key (e.g., "MES-Production", "RecipeManager-Dev").
- **Key Value**: The secret key string used for authentication.
- **Enabled/Disabled Flag**: Keys can be disabled without deletion.

### Management
- Managed by users with the **Administrator** role via the Central UI.
- All key changes (create, enable/disable, delete) are audit logged.

## API Method Definition

### Properties
Each API method definition includes:
- **Method Name**: Unique identifier and URL path segment for the endpoint.
- **Approved API Keys**: List of API keys authorized to invoke this method. Requests from non-approved keys are rejected.
- **Parameter Definitions**: Ordered list of input parameters, each with:
  - Parameter name.
  - Data type — the **extended type system** (Boolean, Integer, Float, String, plus the nestable Object and List; see [Extended Type System](#extended-type-system)).
  - Whether the parameter is required.
- **Return Value Definition**: Structure of the response, with:
  - Field names and (extended-system) data types. Supports returning **lists of objects** and arbitrarily nested structures.
- **Implementation Script**: C# script that executes when the method is called. Stored **inline** in the method definition. Follows standard C# authoring patterns but has no template inheritance — it is a standalone script tied to this method.
- **Timeout**: Configurable per method. Defines the maximum time the method is allowed to execute (including any routed calls to sites) before returning a timeout error to the caller.
- **TraceAllCalls** (Flow Debugger, 2026-09-05): when set, every call to this method runs its routed site scripts with the trace flag on, so their `ScriptRun` / `AttributeWrite` / `AttributeWait` audit rows (`AuditChannel.ScriptTrace`) are recorded for Replay. Default off — adds 7–10 audit rows per call (3 of them trace-only; a cached DB write and a notification account for the upper end), so it is meant for low-volume integration methods. Set from the method form's *Trace all calls* checkbox or `api-method create|update --trace-all-calls`.

> **Authoring note — idempotent methods for at-least-once callers**: some external callers retry
> on their own connect-failure/timeout failover loop and cannot tell "the first attempt actually
> succeeded" from "the first attempt was lost". The updated `DelmiaNotifier` returns an uncertain
> result instead of automatically replaying such attempts. Scripts exposed to other retrying
> clients should still be idempotent on their
> natural key so a duplicate delivery returns the same result without double-applying side effects.

### Management
- Managed by users with the **Designer** role via the Central UI.
- All method definition changes are audit logged.

## HTTP Contract

### URL Structure
- All API calls use `POST /api/{methodName}`.
- Method names map directly to URL path segments (e.g., method "GetProductionReport" → `POST /api/GetProductionReport`).
- All calls are POST — these are RPC-style script invocations, not RESTful resource operations.

### Authentication Header
- The API key token (`sbk_<keyId>_<secret>`) is accepted from **either** of two headers:
  - `Authorization: Bearer sbk_<keyId>_<secret>` — the preferred form.
  - `X-API-Key: sbk_<keyId>_<secret>` — alternate form for callers that cannot set the Authorization header.
- When both headers are present, `Authorization` takes precedence.
- Both forms are processed by the same `IApiKeyVerifier` from the shared `ZB.MOM.WW.Auth.ApiKeys` library; the verifier strips the optional `Bearer ` prefix and performs the same peppered-HMAC constant-time verification regardless of which header was used.

### Key rotation

The `sbk_<keyId>_<secret>` design already supports zero-downtime rotation — a caller can hold two valid keys at once, so there is no moment where the old key must be cut off before the new one works. The procedure:

1. **Create a new key** with the same scope set as the key being rotated — `security api-key create --name <name> --methods <methods>` (CLI `security api-key` group). Scopes are **copied at creation only**: the operator must pass the same `--methods` list the old key was approved for; the CLI does not read the old key's scopes for you, so scope drift between the pair is an operator error, not something the system detects.
2. **Deliver the new token to the caller and confirm cutover.** The response from `create` surfaces the one-time bearer token (`PrintCreatedKey` — it is shown once and cannot be retrieved again, so capture it immediately). Both the old and new keys verify successfully during this window — that overlap is intended, not a bug, and gives the caller time to redeploy with the new token without a synchronized flag-day cutover. Confirm the caller has actually switched by checking the audit log's Actor field (recorded per call), which shows which key ID is serving live traffic.
3. **Disable — don't delete — the old key** once cutover is confirmed: `security api-key update --key-id <old-key-id> --enabled false`. Disabling is reversible (`--enabled true` re-enables it) if the cutover needs to be rolled back, whereas deletion is not.
4. **Delete the old key after a quarantine period** (an operator-judgment window to be sure nothing is still presenting the old token): `security api-key delete --key-id <old-key-id>`.

### Request Format
- Content-Type: `application/json`.
- Parameters are top-level JSON fields in the request body matching the method's parameter definitions:
```json
{
  "siteId": "SiteA",
  "startDate": "2026-03-01",
  "endDate": "2026-03-16"
}
```

### Response Format
- **Success (200)**: The response body is the method's return value as JSON, with fields matching the return value definition:
```json
{
  "siteName": "Site Alpha",
  "totalUnits": 14250,
  "lines": [
    { "lineName": "Line-1", "units": 8200, "efficiency": 92.5 },
    { "lineName": "Line-2", "units": 6050, "efficiency": 88.1 }
  ]
}
```
- **Failure (4xx/5xx)**: The response body is an error object carrying a human-readable `error` and a stable, machine-readable `code`:
```json
{
  "error": "Site unreachable",
  "code": "SITE_UNREACHABLE"
}
```
- HTTP status codes distinguish success from failure — no envelope wrapper.
- **Error codes.** Every failure body carries a `code` an external integrator can branch on (the `error` message is for humans and may change). The two authorization negatives — unknown method and key-not-in-scope — deliberately share one code (`NOT_APPROVED`) and one body so a caller cannot enumerate which method names exist:

| `code` | HTTP status | Meaning |
|--------|-------------|---------|
| `UNAUTHORIZED` | 401 | Missing/invalid API key (every auth-stage failure maps here — no stage leak). |
| `NOT_APPROVED` | 403 | Method not found **or** key not in scope — indistinguishable by design. |
| `UNSUPPORTED_MEDIA_TYPE` | 415 | Request carried a body — declared by `Content-Length` or `Transfer-Encoding` (chunked) — with a non-JSON `Content-Type`. A body with **no** `Content-Type` is still parsed leniently as JSON, and a chunked (no `Content-Length`) body is treated identically to a fixed-length one. |
| `INVALID_JSON` | 400 | Request body was not valid JSON. |
| `VALIDATION_FAILED` | 400 | Parameter validation failed (missing required field, wrong type, undeclared field). |
| `BODY_TOO_LARGE` | 413 | Request body exceeded `MaxRequestBodyBytes`. |
| `STANDBY_NODE` | 503 | Request hit a standby central node that could not relay it: no `StandbyForwardPeerEndpoints` configured, or the request was already relayed once (loop guard — neither node is active, e.g. mid-failover). Carries the `X-ScadaBridge-Standby: 1` response header. |
| `ACTIVE_NODE_UNREACHABLE` | 503 | Every configured peer refused before execution or failed with proven name-resolution/connection-refusal errors. |
| `OUTCOME_UNKNOWN` | 502 | A relay timed out or lost its response after possible execution; it does not replay the request. Verify the operation before retrying. |
| `TIMEOUT` | 500 | The method script exceeded its execution timeout. |
| `SITE_UNREACHABLE` | 500 | The target site did not respond to the routed request within the routing timeout (no contact / down / partitioned) — typed from the routed Ask expiring (`AskTimeoutException`), never inferred from an error-message substring. A `Success=false` response means the site answered and is classified `SCRIPT_ERROR`. |
| `SCRIPT_COMPILE_FAILED` | 500 | The method's script failed to compile (returned consistently on every node until a compiling version is saved). |
| `SCRIPT_ERROR` | 500 | Catch-all for any other script-execution failure (details are logged centrally, never leaked to the caller — the exception's `ToString()` is captured on the audit row as `ErrorDetail`, never in the response body). |

### Extended Type System
- API method parameter and return type definitions support an **extended type system** beyond the four template attribute types (Boolean, Integer, Float, String):
  - **Object**: A named structure with typed fields. Supports nesting.
  - **List**: An ordered collection of objects or primitive types.
- This allows complex request/response structures (e.g., an object containing properties and a list of nested objects).
- Template attributes retain the simpler four-type system. The extended types apply only to Inbound API method definitions and External System Gateway method definitions.

#### Type Definition Format & Nested Validation

- Parameter and return type definitions are persisted as **JSON Schema** (the canonical format produced by the Central UI schema builder; see the `MigrateParametersToJsonSchema` migration). An object declares its fields via `properties` (+ a `required` array); a list declares its element type via `items`. The legacy flat-array form (`[{name,type,required,itemType?}]`) is still accepted on read for transition safety.
- Validation is **recursive and type-aware** for the extended types (request parameters and script return values alike, via a single shared engine so the two cannot drift):
  - **Object**: each declared field's value is validated against its declared (possibly nested) type; a missing required field and a present-but-wrong type are both reported.
  - **List**: every element is validated against the declared element type (recursing into nested objects/lists). A list whose element type is left undeclared (`array` without `items`) is shape-checked only.
  - **Scalars at any depth** are checked against the extended type.
  - Errors are **path-qualified** (e.g. `order.items[2].quantity`) so the caller can locate the offending field.
- **Undeclared fields are rejected** at every level (consistent with the top-level "unexpected parameter" rejection): an object that declares its fields rejects any field not in its `properties`, so a typo'd field name surfaces as a `400`/error rather than being silently ignored. A bare object schema with no declared fields (`{"type":"object"}`) stays shape-only and accepts any fields.
- A JSON `null` value satisfies any declared type (a present-but-null field is allowed); only the **absence** of a required field is an error.

## Script Compilation & Hot-Reload

API method scripts are compiled at central startup — all method definitions are loaded from the configuration database and compiled into in-memory delegates.

### Update Workflow

- Updating a method via the CLI (`api-method update --id <N> --code '...'`) or Management API triggers immediate recompilation (`CompileAndRegister`). The updated script takes effect on the next API call — no node restart is required.
- Creating a new method after startup: if the method is created but not yet compiled, the first invocation triggers lazy (on-demand) compilation.

> **Compile failures on save are non-fatal but surfaced, and the DB row is authoritative.** The persisted `ApiMethod.Script` row is the single source of truth: on every request `ExecuteAsync` revision-checks the cached delegate against the freshly-fetched script and recompiles in place when it differs, so a saved-but-broken script does **not** leave a previously registered handler silently serving. The change still persists, so the Create/Update Management result carries a top-level **`compileWarning`** (null when the script compiled) listing the Roslyn/trust-model errors. On both **create** and **update**, once a non-compiling script is saved the method returns a compilation error (`HTTP 500`) **consistently on every central node** until a compiling version is saved — there is no node-divergent "old delegate still works here" state. This means a saved script that does not compile is **not silently a no-op** — the caller (CLI / Central UI) sees the warning, and the real diagnostics are also logged centrally.

### Direct SQL Warning

> **Direct SQL edits to API method scripts are now picked up automatically.** Because the DB row is authoritative and `ExecuteAsync` revision-checks the cached delegate against the freshly-fetched `ApiMethod.Script` on every request, a script changed directly in the database takes effect on the **next call** — no node restart and no Management round-trip is required. A broken direct edit surfaces the same way as a broken save: the method returns a `HTTP 500` compilation error consistently on all central nodes until a compiling version is stored. Editing via the CLI, Management API, or Central UI is still preferred because those paths validate the script up front and return the `compileWarning`, whereas a direct SQL edit is only diagnosed on the next request (and in the central log).

---

## API Call Logging

- **Every request — success or failure — emits one `ApiInbound.Completed` row** to `ICentralAuditWriter` from request middleware before the HTTP response is flushed. The row captures the API key **name** (never the key material), remote IP, user-agent, response status, duration, and the request/response bodies. Bodies are captured in full up to `AuditLog:InboundMaxBytes` (default 1 MiB); `PayloadTruncated = 1` only when that ceiling is hit. Header redaction and per-target body redactors still apply (see Component-AuditLog.md, Payload Capture Policy). This supersedes the earlier failures-only stance: operational API traffic is now part of the centralized audit log, so configuration changes and call activity share a single retention/query surface.
- Script execution errors (500 responses) remain captured on the same `ApiInbound.Completed` row (response status + error fields) rather than emitting a separate failure-only event.
- **Fail-soft semantics (fire-and-forget).** The audit write is **fire-and-forget** (`InboundAPI-018`): `WriteAsync` is called without `await` so the user-facing HTTP response is never blocked or delayed by the audit path. Asynchronous faults are observed via a `ContinueWith` continuation that logs at Warning and increments `CentralAuditWriteFailures` (see Health Monitoring #11) rather than being silently dropped. A failed audit append never turns a successful API call into an error returned to the caller.
- No rate limiting — this is a private API in a controlled industrial environment with a known set of callers. Misbehaving callers are handled operationally (disable the API key).

## Request Flow

```mermaid
%%{init: {'theme':'base', 'themeVariables': {'textColor':'#111111','lineColor':'#555555','edgeLabelBackground':'#ffffff','fontSize':'15px'}}}%%
flowchart TD
    ext(["External System"])
    api["Inbound API (Central)"]
    s1["1. Extract API key from request"]
    s2["2. Validate key exists and is enabled"]
    s3["3. Resolve method by name"]
    s4["4. Check API key is in method's approved list"]
    s5["5. Validate and deserialize parameters"]
    s6["6. Execute implementation script<br/>(subject to method timeout)"]
    s7["7. Serialize return value"]
    s8["8. Return response"]

    ext --> api
    api --> s1
    s1 --> s2
    s2 --> s3
    s3 --> s4
    s4 --> s5
    s5 --> s6
    s6 --> s7
    s7 --> s8

    classDef start fill:#d5e8d4,stroke:#82b366,color:#111111;
    classDef proc fill:#dae8fc,stroke:#6c8ebf,color:#111111;
    classDef dec fill:#fff2cc,stroke:#d6b656,color:#111111;
    classDef warn fill:#ffe6cc,stroke:#d79b00,color:#111111;
    classDef alt fill:#e1d5e7,stroke:#9673a6,color:#111111;
    class ext start
    class api proc
    class s1,s2,s3,s4,s5,s7 dec
    class s6 alt
    class s8 warn
```

## Implementation Script Capabilities

The C# script that implements an API method executes on the central cluster. Unlike instance scripts at sites, inbound API scripts run on central and can interact with **any instance at any site** through a routing API.

Inbound API scripts **cannot** call shared scripts directly — shared scripts are deployed to sites only and execute inline in Script Actors. To execute logic on a site, use `Route.To().Call()`.

### Script Runtime API

#### Instance Routing
- `Route.To("instanceUniqueCode").Call("scriptName", parameters)` — Invoke a script on a specific instance at any site. Central routes the call to the appropriate site via the Communication Layer. The call reaches the target Instance Actor's Script Actor, which spawns a Script Execution Actor to execute the script. The return value flows back to the calling API script.
- `Route.To("instanceUniqueCode").GetAttribute("attributeName")` — Read a single attribute value from a specific instance at any site.
- `Route.To("instanceUniqueCode").GetAttributes("attr1", "attr2", ...)` — Read multiple attribute values in a **single call**, returned as a dictionary of name-value pairs.
- `Route.To("instanceUniqueCode").SetAttribute("attributeName", value)` — Write a single attribute value on a specific instance at any site.
- `Route.To("instanceUniqueCode").SetAttributes(dictionary)` — Write multiple attribute values in a **single call**, accepting a dictionary of name-value pairs.
- `Route.To("instanceUniqueCode").WaitForAttribute("attributeName", targetValue, timeout)` — Wait, event-driven, until an attribute on a specific instance at any site reaches `targetValue` (value-equality only across the wire), bounded by `timeout`. Returns `true` if matched within the timeout, `false` if it timed out. **The wait is bounded by its own `timeout`, not the generic method-level timeout** — this is the one routed call that may legitimately outlive the method timeout (the site enforces `timeout` and returns `false` when it elapses). A client disconnect still cancels the wait. This is the deliberate exception to the rule below that routed calls inherit the method-level timeout (see "Routing Behavior"): a long event-driven wait is the explicit reason `timeout` governs here.

> **Reading alarm state via a routed script.** There is deliberately **no** `Route.To(...).GetAlarms(...)` verb. An inbound method that needs a machine's current alarms resolves the instance and calls a **site script** that uses the Site Runtime `Alarms.CurrentAsync()` accessor — `Route.To(code).Call("SimpleAlarmStatus", new { ... })`. Alarm state is per-instance and lives on the site's Instance Actor; keeping the read inside a site script means the filtering/shaping happens where the data is, and central stays a thin router. This is the shape the MES alarm-status endpoints use (`docs/plans/2026-06-30-mes-alarm-status-api.md` §5.1); see [Component-SiteRuntime.md](Component-SiteRuntime.md) → *Alarm State (read-only)* for the accessor.

#### Input/Output
- **Input parameters** are available as defined in the method definition.
- **Return value** construction matching the defined return structure.

#### Parameter Access
- `Parameters["key"]` — Raw dictionary access.
- `Parameters.Get<T>("key")` — Typed access (same API as site runtime scripts). See Site Runtime component for full type support.

#### Database Access

Inbound API scripts may read from and write to the configuration / machine-data
databases through the **curated, scoped `Database` helper** — `InboundDatabaseHelper`,
exposed as `InboundScriptContext.Database`. This is the "dedicated, scoped helper added
as an explicit design change" that the script trust model requires: scripts are **never**
handed a raw `SqlConnection`, and never reference `System.Data` (the `ForbiddenApiChecker`,
delegating to the shared `ScriptAnalysis` `ScriptTrustValidator` (#25), still statically
bans `System.IO`, `Process`, `Threading`, `Reflection`, and raw network access — that is
defence-in-depth static enforcement, not a true runtime sandbox).

The helper API (all asynchronous — scripts `await` them; bounded by the method timeout):

- `await Database.QuerySingleAsync<T>("connectionName", sql, parameters)` — first column of the first row as `T` (default if no rows).
- `await Database.QueryAsync("connectionName", sql, parameters)` — all rows as case-insensitive column→value dictionaries.
- `await Database.ExecuteAsync("connectionName", sql, parameters)` — run a write (INSERT/UPDATE/DELETE/DDL); returns rows affected. **Writes are permitted** — the move-in integration records results, not just reads them.

**Every helper call is audited (2026-09-06).** Each `QuerySingle` / `Query` / `Execute` writes one `AuditChannel.DbOutbound` / `AuditKind.DbWrite` row through `ICentralAuditWriter` — `Status` `Delivered` or `Failed`, `Target` = the connection name, `Actor` = `api:{methodName}`, the request's `ExecutionId`, `DurationMs`, the SQL + bound parameter names/values as `RequestSummary` (capped and redacted at write time by the shared audit redactor exactly like a site `AuditingDbCommand` row), `Extra` = `{"op":"read","rowsReturned":n}` / `{"op":"write","rowsAffected":n}`, `SourceNode` from `INodeIdentityProvider`. Best-effort: an audit fault never reaches the script, and the original exception still rethrows. Under a Flow Debugger Live run the same instance is also handed to the route's trace sink so the call appears as a `DbCall` node on the External lane (Component-CentralUI.md → Flow Debugger). This closed the outbound-DB gap Component-AuditLog.md's scope table used to list for central method scripts.

Containment rules (enforced, not advisory):

- **Named connections only.** `connectionName` selects one of the connections configured
  on the central database gateway (`IDatabaseGateway`); a script cannot supply an arbitrary
  connection string or reach a database the gateway is not configured for.
- **SQL-injection protection.** Statement text is authored by the (design-time) method
  script, but every request-derived **value** is passed via `parameters` and bound as a
  named `@`-prefixed SQL parameter — never string-concatenated into the command text.
  Request input therefore reaches the database only through parameter binding.
- **Deadline-bound.** Calls use the async ADO.NET path end-to-end (no pool-thread blocking)
  and honour the method deadline token, with a `CommandTimeout` backstop derived from the
  method timeout, so a slow query is bounded by the method timeout.

For everything else, scripts interact with the system through the curated `Route` and
`Parameters` surfaces above.

#### Logging (`Log`)

Method scripts get the same leveled `Log` global site scripts do (design
`docs/plans/2026-09-09-flow-debugger-step-logs-design.md` D4/D5) — `InboundScriptContext.Log`,
backed by `InboundScriptLogger`:

- `Log.Debug(message, exception?)` / `Log.Info` / `Log.Warn` / `Log.Error` — synchronous, `void`,
  never throws into the script.
- **Always** forwards to the central node's `ILogger`. A `ScriptTrace` / `ScriptLog` audit row is
  written **only when the request is traced** (a Flow Debugger Live run, or a real call to a method
  with `TraceAllCalls`), through the executor's inline trace sink and `ICentralAuditWriter`; an
  ordinary call pays a log entry and nothing more.
- Lines with no explicit step id are attributed to the request's anchor row (the `InboundRequest`
  row, or the debugger's synthesized `RouteCall` root), so they render on the inbound node.
- **Bounded:** `ScadaBridge:InboundApi:MaxLogLinesPerRun` (default 200, validated `> 0`) caps the
  rows one request may emit, messages are clipped at 2 000 characters, and one closing `warn` row
  names the dropped count.

#### Script exception detail

`InboundScriptResult` carries an internal `ErrorDetail` (`ex.ToString()` of the fault the method
script threw). It is written to the request's audit row — by `FlowDebugRunner` for a Live run, and
on the ordinary HTTP path via `HttpContext.Items[AuditWriteMiddleware.ErrorDetailItemKey]` — and is
projected onto `FlowStep.ErrorDetail` for the Flow Debugger's exception block. **It is never
returned to the HTTP caller:** the response body stays the generic `SCRIPT_ERROR` envelope,
unchanged.

**Note the asymmetry with the site half, which is easy to get wrong:** the inbound row's
`errorDetail` is written on **every failing method request**, traced or not — the middleware stamps
it whenever the script faulted, so it is not gated on `TraceAllCalls` or on a Live run (those only
decide whether the row is later *readable as a Flow Debugger step*). Its size is bounded by the
redactor's truncation stage, which for an `ApiInbound` row selects `AuditLog:InboundMaxBytes`
(1 MiB default — `SelectCap` gives this channel its own ceiling instead of the 64 KB
`ErrorCapBytes` other channels' error rows get), so a deep stack is truncated there rather than
dropped. The **site** `ScriptRun` row's `errorDetail`, by contrast, is **traced-only** — an
untraced site script run emits no `ScriptTrace` row at all, so there is nothing to carry it; that
row, being a `ScriptTrace` error row, is the one bounded at the 64 KB `ErrorCapBytes`.

### Routing Behavior
- The `Route.To()` helper resolves the instance's site assignment from the configuration database and routes the request to the correct site cluster via the Communication Layer.
- The call is **synchronous from the API caller's perspective** — the API method blocks until the site responds or the **method-level timeout** is reached.
- If the target site is unreachable or the call times out, the call fails and the API returns an error to the caller. No store-and-forward buffering is used for inbound API calls.
- **Every `Route.To(...).Call(...)` writes one always-on `ScriptTrace.RouteCall` audit row** (central direct write, actor `inbound-api`, target `{instance}.{script}`, `ExecutionId` = the inbound request's, `Extra.childExecutionId` = the site run's id) — this row is what joins the central and site halves of a Flow Debugger Replay, so it is not gated. The routed request carries a `Trace` flag beside `ParentExecutionId` (`RouteToCallRequest.Trace`, `RouteToSetAttributesRequest.Trace`); it is set when the method has `TraceAllCalls` or the call is a Flow Debugger Live run, and only then does the site emit its own trace rows and return them inline on `RouteToCallResponse.TraceEvents` (with `ChildExecutionId`). Untraced calls are byte-identical to before. **The `RouteCall` row honours the method's `SkipBodyCapture`** — the executor builds the helper with `RouteHelper.WithBodyCapture(!skipBodyCapture)`, so a method that opts out of body capture no longer leaks its arguments and reply through the always-on trace row; the row itself is still written, with metadata only (arch-review F11).

## Flow Debugger runs

The Flow Debugger (Central UI, design `docs/plans/2026-09-05-flow-debugger-design.md`; script entry point `docs/plans/2026-09-08-flow-debugger-script-entry-design.md`) drives this component through four management commands handled by the per-node `ManagementActor` (Deployer role): `FlowDebugRunMethodCommand` executes a method exactly as an external caller would — `ParameterValidator` + `SchemaRefResolver`, then `InboundScriptExecutor.ExecuteAsync(..., trace: true)` with a `RouteHelper` trace sink collecting the `RouteCall` row and the site's inline events — and **synthesizes the `ApiInbound.InboundRequest` row itself** with the operator as actor (`flow-debugger` source tag) because no HTTP request passed through `AuditWriteMiddleware`; it refuses on a standby node with `STANDBY_NODE` (`IActiveNodeGate`), so the script runs only where a real call would. `FlowDebugRunValueChangeCommand` writes an attribute through the traced routed-write path and polls the site (`PullTraceEvents`) for the run the change spawned. `FlowDebugRunScriptCommand` (2026-09-08) resolves the named script on the instance through the flattening pipeline and branches on its trigger kind: a write kind (ValueChange/Conditional) reuses the same traced routed-write + `PullTraceEvents` poll as `FlowDebugRunValueChangeCommand`, using the attribute derived from the script's own trigger configuration; a call kind (Call/Interval/Expression/no trigger) routes a `RouteToCallRequest(Trace: true)` through `IInstanceRouter` **directly** rather than through `RouteHelper` — this keeps exactly one operator-attributed `RouteCall` root row instead of `RouteHelper`'s own always-on, actor-less one — and returns the site's inline trace rows through `FlowTraceCollector`; its error codes are `NOT_FOUND` (unknown script or instance), `VALIDATION_FAILED` (missing write-kind value, or call-kind parameters that fail `ParameterValidator`), `CALL_FAILED` (the routed call completed but the site reported failure), and `TIMEOUT` (no response within the script's execution timeout). `FlowDebugReplayCommand` reads the chain back from the central Audit Log and therefore accepts the wider `AuthorizationPolicies.FlowDebuggerAccessRoles` set (Deployer, Administrator, Viewer) while the three run commands stay Deployer-only (arch-review F3). The result (`FlowDebugRunResult`) carries the audit events inline — the graph is a central-side projection, never a wire type.

- **`SchemaRefResolver` is `public` for the runner's benefit.** It was internal to the HTTP path; the Flow Debugger's `FlowDebugRunMethodCommand` validates its operator-supplied parameters through exactly the same `ParameterValidator` + `SchemaRefResolver` pair a real HTTP call uses, so a Live run cannot accept an input the endpoint would reject. Keep the two paths on the one resolver — a second, debugger-local validator would silently fork the contract.
- **Site scope is enforced at route time on a Live run** (arch-review F2). `RouteHelper.WithSiteScope(IReadOnlySet<string>? permittedSiteIdentifiers)` carries the operator's permitted **site identifiers**; `RouteTarget.ResolveSiteAsync` throws `RouteSiteScopeException` before any site traffic when the resolved target site is outside that set, and `InboundScriptExecutor` maps it to error code **`SITE_SCOPE_DENIED`**. `FlowDebugRunner` translates that code into a `SiteScopeViolationException` (`ManagementUnauthorized`) *after* writing the inbound audit row, so the denied attempt is recorded under the operator's name. Administrators and unscoped users get a null scope (unrestricted), and the HTTP endpoint passes no scope at all — production routing is unchanged.

## Authentication Details

- The API key token is accepted from **either** the `Authorization: Bearer <token>` header (preferred) **or** the `X-API-Key: <token>` header (alternate). `Authorization` takes precedence when both are present.
- The token format is `sbk_<keyId>_<secret>`. Both header forms are verified by the same `IApiKeyVerifier` from `ZB.MOM.WW.Auth.ApiKeys`.
- The system validates:
  1. The token is well-formed and the key ID exists in the key store.
  2. The key is enabled (not revoked).
  3. The key's provisioned scopes include the requested method name (case-sensitive).
- Failed authentication returns an appropriate HTTP error (401 Unauthorized for invalid/missing credentials; 403 Forbidden for a valid key not approved for the method). For enumeration-safety, both "method not found" and "key not in scope" return 403 with an identical body.

## Standby forwarding

Both central nodes expose `POST /api/{methodName}`, but only the active node (the oldest Up member — `ActiveNodeEvaluator.SelfIsOldestUp`, the same verdict behind `/health/active`) executes method scripts and `Route.To()` calls; running them on a standby could race the active node or read stale singleton state. Before 2026-08-28 a standby simply answered 503 `STANDBY_NODE`, and "both nodes work" relied entirely on the load balancer probing `/health/active` — a caller addressing a node directly, or hitting the balancer inside its probe interval after a failover, got the 503.

A standby now **relays** the request to the active peer instead (`InboundApiEndpointFilter` → `IStandbyRequestForwarder` / `HttpStandbyRequestForwarder`):

- **Configuration.** `ScadaBridge:InboundApi:StandbyForwardPeerEndpoints` — the HTTP base URL(s) of the *other* central node(s), e.g. `["http://scadabridge-central-b:5000"]` on central-a. Empty (default) keeps the old 503 behaviour. `StandbyForwardTimeout` (default 60 s) is the total relay budget and should exceed the longest method timeout plus network overhead. Validated at startup (absolute http/https URLs, positive timeout). Note the standby discovers the active peer from **appsettings**, not from cluster state — the Akka member address carries the remoting port, not the HTTP port — the options snapshot is read per relay.
- **Pure relay.** Method, path, query string, body and credentials travel unchanged; hop-by-hop/Host/forwarding-provenance headers are excluded and authenticated provenance is rebuilt, so the **active node** authenticates the caller's own `Authorization`/`X-API-Key` credential, applies its own body cap and method timeout, and writes the audit row. The peer's status, `Content-Type` and body are replayed unchanged. The serving node owns its `ApiInbound` row; the originating standby retains a failure row when no peer serves or the outcome is uncertain. Verified internal standby refusals are not duplicated.
- **Order of checks.** The body-size cap (HTTP 413) runs before audit reads and the active-node check, so a standby never buffers or relays a body the active node would refuse. The body is buffered once on the standby (bounded by `MaxRequestBodyBytes`) so it can be re-sent to a second peer.
- **Peer selection.** The standby knows only that it is *not* active. It tries the peers in order and accepts the first response that is not a standby refusal; a peer answering 503 with `X-ScadaBridge-Standby: 1`, proven name-resolution failure, or connection refusal permits trying the next peer. Timeout or ambiguous response loss returns 502 `OUTCOME_UNKNOWN` without replay. A 503 *without* the marker is the active node's own answer and is replayed, not retried. All peers failing → 503 `ACTIVE_NODE_UNREACHABLE`.
- **One-hop loop guard.** Every relayed request carries `X-ScadaBridge-Forwarded: <relaying host>`. A standby never relays a request that already carries it — it answers 503 `STANDBY_NODE` with the marker header — so a pair in which neither node is active (a transient during failover) bounces at most one hop and never ping-pongs. The active node ignores the header and serves the request normally.
- **Still not a retry layer.** The relay makes one HTTP attempt per peer and does not retry a peer that answered; at-least-once callers keep their own retry loop and the idempotent-method authoring note above still applies.

## Error Handling

- Invalid API key → 401 Unauthorized.
- Valid key but not approved for method → 403 Forbidden.
- Invalid parameters → 400 Bad Request.
- Script execution failure → 500 Internal Server Error (with safe error message, no internal details exposed).
- Script errors are logged in the central audit/event system.

## Dependencies

- **Configuration Database (MS SQL)**: Stores method definitions, and — under `Provider=SqlServer` — API keys in the `zbauth` schema (see *Key store provider* above). Under the `Sqlite` default, API keys live in a per-node SQLite file instead.
- **Communication Layer**: Routes requests to sites when method implementations need site data.
- **Security & Auth**: API key validation (separate from LDAP/AD — API uses key-based auth).
- **Configuration Database (via IAuditService)**: All API key and method definition changes are audit logged.
- **Script Analysis (#25)**: `ForbiddenApiChecker` delegates to `ScriptTrustValidator.FindViolations` for the authoritative forbidden-API verdict during script compilation and validation.
- **Audit Log (#23)**: Every inbound API request emits an `ApiInbound.Completed` row via `ICentralAuditWriter` from request middleware (non-blocking for the HTTP response). Request and response bodies are captured in full up to `AuditLog:InboundMaxBytes` (default 1 MiB) per the Audit Log Payload Capture Policy; redaction (headers + per-target body redactors) still applies before persistence.
- **Cluster Infrastructure**: method execution is hosted on the active central node and fails over with it; `IActiveNodeGate` (Host) tells the endpoint filter whether to serve locally or relay.

## Interactions

- **External Systems**: Call the API with API keys.
- **Communication Layer**: API method scripts use this to reach sites.
- **Site Runtime (Instance Actors, Script Actors)**: Routed calls execute on site Instance Actors via their Script Actors.
- **Central UI**: Administrator manages API keys; Designer manages method definitions.
- **Configuration Database (via IAuditService)**: Configuration changes are audited.

### Relay audit ownership, privacy, and request limits

The inbound body limit is installed before audit buffering. Declared oversized bodies and unknown-length HTTP/1.1 or HTTP/2 bodies are rejected with HTTP 413; enabling audit capture cannot relax the route limit. The relay also bounds its buffered copy explicitly.

A serving peer owns the inbound audit for its success or method failure. The originating standby writes a correlated failure if no peer serves the request, including an uncertain lost response. Authenticated internal standby refusals do not create duplicate inbound rows. Audit writes remain best effort; ownership is not a durable audit receipt.

Relays overwrite caller-supplied provenance with an HMAC signed envelope using a domain-separated key derived from `ApiKeyPepper`. Both centrals must share that server secret. The envelope binds the original verified remote address, execution ID, timestamp, method, path/query, content type, caller credentials, and bounded body digest. The receiver accepts it for at most five minutes and only when the request binding matches. Unsigned loop markers and forwarded-address headers confer no identity trust; missing or invalid proof uses the receiver's verified remote address. Forwarding remains one hop. Use TLS between peers when network confidentiality is required.

`StandbyForwardTimeout` is one budget across permitted attempts and response consumption. Only explicit pre-execution standby rejection, name-resolution failure, or connection refusal permits another peer. Timeout or response loss returns HTTP 502 `OUTCOME_UNKNOWN`: verify the operation before manually retrying because it may already have applied. This avoids blindly replaying non-idempotent methods.

`SkipBodyCapture` suppresses request/response summaries and all exception message/detail text, including stashed script exceptions; retain status/category and request metadata for diagnosis. Error text can quote payload values and is subject to the same privacy boundary as the body.