Add dual call modes for external systems: synchronous Call() and cached CachedCall()

Scripts now choose per invocation whether an external system call is synchronous
(all failures return to script) or cached (transient failures go to store-and-forward).
Mirrors the existing Database.Connection/CachedWrite pattern. Updated ESG, Site
Runtime script API, high-level requirements, and design doc.

Component-ExternalSystemGateway.md

@@ -31,10 +31,16 @@ Site clusters (executes calls directly to external systems). Central cluster (st
 
 Each external system definition includes:
 - **Name**: Unique identifier (e.g., "MES", "RecipeManager").
-- **Connection Details**: Endpoint URL, authentication, protocol.
+- **Base URL**: The root endpoint URL for the external system (e.g., `https://mes.example.com/api`).
-- **Retry Settings**: Max retry count, fixed time between retries (used by Store-and-Forward Engine).
+- **Authentication**: One of:
+  - **API Key**: Header name (e.g., `X-API-Key`) and key value.
+  - **Basic Auth**: Username and password.
+- **Timeout**: Per-system timeout for all method calls (e.g., 30 seconds). Applies to the HTTP request round-trip.
+- **Retry Settings**: Max retry count, fixed time between retries (used by Store-and-Forward Engine for transient failures only).
 - **Method Definitions**: List of available API methods, each with:
   - Method name.
+  - **HTTP method**: GET, POST, PUT, or DELETE.
+  - **Path**: Relative path appended to the base URL (e.g., `/recipes/{id}`).
   - Parameter definitions (name, type).
   - Return type definition.
 
@@ -58,6 +64,46 @@ Each database connection definition includes:
 - Payload includes: connection name, SQL statement, serialized parameter values.
 - If the database is unavailable, the write is buffered and retried per the connection's retry settings.
 
+## Invocation Protocol
+
+All external system calls are **HTTP/REST** with **JSON** serialization:
+
+- The ESG acts as an HTTP client. The external system definition provides the base URL; each method definition specifies the HTTP method and relative path.
+- Request parameters are serialized as JSON in the request body (POST/PUT) or as query parameters (GET/DELETE).
+- Response bodies are deserialized from JSON into the method's defined return type.
+- Credentials (API key header or Basic Auth header) are attached to every request per the system's authentication configuration.
+
+## External System Call Modes
+
+Scripts choose between two call modes per invocation, mirroring the dual-mode database access pattern:
+
+### Synchronous (Real-time)
+- Script calls `ExternalSystem.Call("systemName", "methodName", params)`.
+- The HTTP request is executed immediately. The script blocks until the response is received or the timeout elapses.
+- **All failures** (transient and permanent) return an error to the calling script. No store-and-forward buffering.
+- Use for request/response interactions where the script needs the result (e.g., fetching a recipe, querying inventory).
+
+### Cached (Store-and-Forward)
+- Script calls `ExternalSystem.CachedCall("systemName", "methodName", params)`.
+- The call is attempted immediately. If it succeeds, the response is discarded (fire-and-forget).
+- On **transient failure** (connection refused, timeout, HTTP 5xx), the call is routed to the Store-and-Forward Engine for retry per the system's retry settings. The script does **not** block — the call is buffered and the script continues.
+- On **permanent failure** (HTTP 4xx), the error is returned **synchronously** to the calling script. No retry — the request itself is wrong.
+- Use for outbound data pushes where deferred delivery is acceptable (e.g., posting production data, sending quality reports).
+
+## Call Timeout & Error Handling
+
+- Each external system definition specifies a **timeout** that applies to all method calls on that system.
+- Error classification by HTTP response:
+  - **Transient failures** (connection refused, timeout, HTTP 5xx): Behavior depends on call mode — `CachedCall` buffers for retry; `Call` returns error to script.
+  - **Permanent failures** (HTTP 4xx): Always returned to the calling script regardless of call mode. Logged to Site Event Logging.
+- This classification ensures the S&F buffer is not polluted with requests that will never succeed.
+
+## Database Connection Management
+
+- Database connections use **standard ADO.NET connection pooling** per named connection. No custom pool management.
+- Pool behavior (max pool size, connection lifetime, etc.) can be tuned via connection string parameters in the database connection definition if needed.
+- Synchronous failures on `Database.Connection()` (e.g., unreachable server) return an error to the calling script, consistent with external system permanent failure handling.
+
 ## Dependencies
 
 - **Configuration Database (MS SQL)**: Stores external system and database connection definitions.

Component-SiteRuntime.md

@@ -205,7 +205,8 @@ Available to all Script Execution Actors and Alarm Execution Actors:
 - `Scripts.CallShared("scriptName", parameters)` — Execute shared script code inline (direct method invocation). The call includes the current recursion depth.
 
 ### External Systems
-- Access to predefined external system API methods (see External System Gateway component).
+- `ExternalSystem.Call("systemName", "methodName", params)` — Synchronous HTTP call. Blocks until response or timeout. All failures return to script. Use when the script needs the result.
+- `ExternalSystem.CachedCall("systemName", "methodName", params)` — Fire-and-forget with store-and-forward on transient failure. Use for outbound data pushes where deferred delivery is acceptable.
 
 ### Notifications
 - `Notify.To("listName").Send("subject", "message")` — Send an email notification via a named notification list.

Component-StoreAndForward.md

@@ -51,6 +51,8 @@ Retry settings are defined on the **source entity** (not per-message):
 
 The retry interval is **fixed** (not exponential backoff). Fixed interval is sufficient for the expected use cases.
 
+**Note**: Only **transient failures** are eligible for store-and-forward buffering. For external system calls, transient failures are connection errors, timeouts, and HTTP 5xx responses. Permanent failures (HTTP 4xx) are returned directly to the calling script and are **not** queued for retry. This prevents the buffer from accumulating requests that will never succeed.
+
 ## Buffer Size
 
 There is **no maximum buffer size**. Messages accumulate in the buffer until delivery succeeds or retries are exhausted and the message is parked. Storage is bounded only by available disk space on the site node.

HighLevelReqs.md

@@ -221,7 +221,7 @@ Scripts executing on a site for a given instance can:
 - **Write** attribute values on that instance. For attributes with a data source reference, the write goes to the Data Connection Layer which writes to the physical device; the in-memory value updates when the device confirms the new value via the existing subscription. For static attributes, the write updates the in-memory value directly.
 - **Call other scripts** on that instance via `Instance.CallScript("scriptName", params)`. Calls use the Akka ask pattern and return the called script's return value. Script-to-script calls support concurrent execution.
 - **Call shared scripts** via `Scripts.CallShared("scriptName", params)`. Shared scripts execute **inline** in the calling Script Actor's context — they are compiled code libraries, not separate actors.
-- **Call external system API methods** (see Section 5).
+- **Call external system API methods** in two modes: `ExternalSystem.Call()` for synchronous request/response, or `ExternalSystem.CachedCall()` for fire-and-forget with store-and-forward on transient failure (see Section 5).
 - **Send notifications** (see Section 6).
 - **Access databases** by requesting an MS SQL client connection by name (see Section 5.5).
 

docs/plans/2026-03-16-external-system-gateway-refinement-design.md

@@ -0,0 +1,61 @@
+# External System Gateway Refinement — Design
+
+**Date**: 2026-03-16
+**Component**: External System Gateway (`Component-ExternalSystemGateway.md`)
+**Status**: Approved
+
+## Problem
+
+The External System Gateway doc lacked specification for the invocation protocol, authentication methods, call timeouts, error classification for store-and-forward decisions, and database connection management.
+
+## Decisions
+
+### Invocation Protocol
+- **HTTP/REST only** with **JSON** serialization. The ESG is an HTTP client with predefined endpoints.
+- Method definitions include HTTP method (GET/POST/PUT/DELETE) and relative path.
+- Parameters serialized as JSON body (POST/PUT) or query parameters (GET/DELETE).
+
+### Outbound Authentication
+- Two modes per external system definition:
+  - **API Key**: Configurable header name and key value.
+  - **Basic Auth**: Username and password, sent as standard HTTP Authorization header.
+
+### Call Timeouts
+- **Per-system timeout** — one timeout value applies to all methods on a given external system.
+- Defined in the external system definition.
+
+### Dual Call Modes
+- **`ExternalSystem.Call()`**: Synchronous request/response. All failures (transient and permanent) return to the script. No S&F buffering. Use when the script needs the result.
+- **`ExternalSystem.CachedCall()`**: Fire-and-forget with S&F on transient failure. Use for outbound data pushes where deferred delivery is acceptable.
+- Mirrors the existing `Database.Connection()` / `Database.CachedWrite()` pattern.
+
+### Error Classification
+- **Transient failures** (connection errors, timeouts, HTTP 5xx): `CachedCall` buffers for retry; `Call` returns error to script.
+- **Permanent failures** (HTTP 4xx): Always returned to the calling script regardless of call mode. Logged to Site Event Logging.
+- S&F buffer only accepts transient failures to avoid accumulating unrecoverable requests.
+
+### Database Connection Pooling
+- Standard ADO.NET connection pooling per named connection. No custom pool logic.
+- Pool tuning via connection string parameters if needed.
+
+### Serialization
+- JSON only, consistent with REST-only decision.
+
+## Affected Documents
+
+| Document | Change |
+|----------|--------|
+| `Component-ExternalSystemGateway.md` | Updated External System Definition fields. Added sections: External System Call Modes (dual-mode API), Invocation Protocol, Call Timeout & Error Handling, Database Connection Management. |
+| `Component-StoreAndForward.md` | Clarified that only transient failures are buffered; 4xx errors are not queued. |
+| `Component-SiteRuntime.md` | Updated Script Runtime API with `ExternalSystem.Call()` and `ExternalSystem.CachedCall()`. |
+| `HighLevelReqs.md` | Updated script capabilities (Section 4.4) to reflect dual call modes. |
+
+## Alternatives Considered
+
+- **SOAP support**: Rejected — REST covers modern integrations; SOAP systems can be fronted by a thin REST wrapper.
+- **OAuth2 client credentials**: Rejected — adds token lifecycle complexity at every site for marginal benefit; can be handled by a gateway/proxy.
+- **Per-method timeouts**: Rejected — external systems tend to have consistent latency; per-system is the right granularity.
+- **All failures retryable**: Rejected — retrying 4xx errors pollutes the S&F buffer with requests that will never succeed.
+- **Custom connection pooling**: Rejected — ADO.NET pooling is battle-tested and handles this scenario natively.
+- **XML serialization option**: Rejected — JSON-only is consistent with REST-only; XML systems can use a wrapper.
+- **Per-method sync/cached flag**: Rejected — the same method may need synchronous calls in one script and cached in another. Script author chooses per invocation.