scadalink-design/Component-ExternalSystemGateway.md

Component: External System Gateway

Purpose

The External System Gateway manages predefined integrations with external systems (e.g., MES, recipe managers) and database connections. It provides the runtime for invoking external API methods and executing database operations from scripts at site clusters.

Location

Site clusters (executes calls directly to external systems). Central cluster (stores definitions, brokers inbound requests from external systems to sites).

Responsibilities

Definitions (Central)

• Store external system definitions in the configuration database: connection details, API method signatures (parameters and return types).
• Store database connection definitions: server, database, credentials.
• Deploy definitions uniformly to all sites (no per-site overrides). Deployment requires explicit action by a user with the Deployment role.
• Managed by users with the Design role.

Execution (Site)

• Invoke external system API methods as requested by scripts (via Script Execution Actors and Alarm Execution Actors).
• Provide raw MS SQL client connections (ADO.NET) by name for synchronous database access.
• Submit cached database writes to the Store-and-Forward Engine for reliable delivery.
• Sites communicate with external systems directly (not routed through central).

Integration Brokering (Central)

• Receive inbound requests from external systems (e.g., MES querying machine values).
• Route requests to the appropriate site via the Communication Layer.
• Return responses to the external system.

External System Definition

Each external system definition includes:

• Name: Unique identifier (e.g., "MES", "RecipeManager").
• Base URL: The root endpoint URL for the external system (e.g., https://mes.example.com/api).
• Authentication: One of:
  • API Key: Header name (e.g., X-API-Key) and key value.
  • Basic Auth: Username and password.
• Timeout: Per-system timeout for all method calls (e.g., 30 seconds). Applies to the HTTP request round-trip.
• Retry Settings: Max retry count, fixed time between retries (used by Store-and-Forward Engine for transient failures only).
• Method Definitions: List of available API methods, each with:
  • Method name.
  • HTTP method: GET, POST, PUT, or DELETE.
  • Path: Relative path appended to the base URL (e.g., /recipes/{id}).
  • Parameter definitions (name, type).
  • Return type definition.

Database Connection Definition

Each database connection definition includes:

• Name: Unique identifier (e.g., "MES_DB", "HistorianDB").
• Connection Details: Server address, database name, credentials.
• Retry Settings: Max retry count, fixed time between retries (for cached writes).

Database Access Modes

Synchronous (Real-time)

• Script calls Database.Connection("name") and receives a raw ADO.NET SqlConnection.
• Full control: queries, updates, transactions, stored procedures.
• Failures are immediate — no buffering.

Cached Write (Store-and-Forward)

• Script calls Database.CachedWrite("name", "sql", parameters).
• The write is submitted to the Store-and-Forward Engine.
• Payload includes: connection name, SQL statement, serialized parameter values.
• If the database is unavailable, the write is buffered and retried per the connection's retry settings.

Invocation Protocol

All external system calls are HTTP/REST with JSON serialization:

• The ESG acts as an HTTP client. The external system definition provides the base URL; each method definition specifies the HTTP method and relative path.
• Request parameters are serialized as JSON in the request body (POST/PUT) or as query parameters (GET/DELETE).
• Response bodies are deserialized from JSON into the method's defined return type.
• Credentials (API key header or Basic Auth header) are attached to every request per the system's authentication configuration.

Call Timeout & Error Handling

• Each external system definition specifies a timeout that applies to all method calls on that system.
• Error classification determines whether the Store-and-Forward Engine retries the call:
  • Transient failures (connection refused, timeout, HTTP 5xx): The call is routed to the Store-and-Forward Engine for retry per the system's retry settings. The script does not block waiting for eventual delivery — the call is buffered and the script continues.
  • Permanent failures (HTTP 4xx): No retry. The error is returned synchronously to the calling script for handling (log, notify, try different parameters, etc.). The failure is logged to Site Event Logging.
• This classification ensures the S&F buffer is not polluted with requests that will never succeed.

Database Connection Management

• Database connections use standard ADO.NET connection pooling per named connection. No custom pool management.
• Pool behavior (max pool size, connection lifetime, etc.) can be tuned via connection string parameters in the database connection definition if needed.
• Synchronous failures on Database.Connection() (e.g., unreachable server) return an error to the calling script, consistent with external system permanent failure handling.

Dependencies

• Configuration Database (MS SQL): Stores external system and database connection definitions.
• Store-and-Forward Engine: Handles buffering for failed external system calls and cached database writes.
• Communication Layer: Routes inbound external system requests from central to sites.
• Security & Auth: Design role manages definitions.
• Configuration Database (via IAuditService): Definition changes are audit logged.

Interactions

• Site Runtime (Script/Alarm Execution Actors): Scripts invoke external system methods and database operations through this component.
• Store-and-Forward Engine: Failed calls and cached writes are routed here for reliable delivery.
• Deployment Manager: Receives updated definitions as part of system-wide artifact deployment (triggered explicitly by Deployment role).