merge: external-system retry policy end-to-end (register row 27) — MaxRetries/RetryDelay now travel artifact→gRPC→site schema→S&F enqueue, CLI --max-retries/--retry-delay-seconds, ArtifactDiff compares them

docs/components/SiteRuntime.md

@@ -205,7 +205,7 @@ Both `AlarmActor` and `NativeAlarmActor` tell the `InstanceActor` an `AlarmState
 | `static_attribute_overrides` | Runtime attribute writes (`SetAttribute` on static attrs) | Yes — cleared by `ClearStaticOverridesAsync` |
 | `native_alarm_state` | Mirrored native alarm conditions (survives failover) | Yes — cleared by `ClearNativeAlarmsForInstanceAsync` |
 | `shared_scripts` | Shared script code from artifact deployments | No |
-| `external_systems` | External system definitions | No |
+| `external_systems` | External system definitions (incl. `timeout_seconds`, and `max_retries`/`retry_delay_ms` — the deployed store-and-forward retry policy, `0` = use the engine default) | No |
 | `database_connections` | Database connection strings | No |
 | `data_connection_definitions` | OPC UA / MxGateway endpoint definitions | No |
 | `notification_lists` | Notification list definitions | No |

docs/components/StoreAndForward.md

@@ -51,6 +51,8 @@ The retry interval is fixed — not exponential. There is no maximum buffer size
 
 `StoreAndForwardMessage.MaxRetries` controls how many background-sweep attempts the engine makes before parking. `MaxRetries = 0` means **no limit** — the message retries on every sweep until delivered and is never parked for retry exhaustion. It is not a "never retry" value; callers that want unbounded retry pass `maxRetries: 0` explicitly. The `EnqueueAsync` `maxRetries` parameter defaults to `StoreAndForwardOptions.DefaultMaxRetries` (50).
 
+**The source-entity `0` means the opposite, and the gateways convert.** On an `ExternalSystemDefinition` / `DatabaseConnectionDefinition`, `MaxRetries = 0` (and `RetryDelay = TimeSpan.Zero`) mean *unset*, not *unbounded* — so `ExternalSystemClient.CachedCallAsync` and `DatabaseGateway` pass `system.MaxRetries > 0 ? system.MaxRetries : null`, letting the bounded engine default apply. Forwarding the entity's `0` verbatim would make every unconfigured cached call retry forever. The external system's values only actually reach a site as of deferred-work row 27; before that the artifact did not carry them.
+
 ### Messages not cleared on instance deletion
 
 When an instance is deleted, its buffered S&F messages are not removed. `StoreAndForwardMessage.OriginInstanceName` records the originating instance at enqueue time so the buffer can continue to deliver and so the central UI can attribute parked messages even after the instance is gone.

docs/deployment/topology-guide.md

@@ -291,6 +291,18 @@ will run, but they will not converge, and the divergence is silent.
 
 Stop both nodes of a site pair, upgrade both, then start both.
 
+**A release that adds a column to a replicated table is the same constraint, enforced louder.**
+`ReplicatedTable.ComputeDigest` hashes every column name and declared type, and the sync handshake
+compares those digests fail-closed — so a mixed-version pair whose builds disagree about a table's
+columns does not diverge silently, it refuses to sync with a `LocalDbSchemaMismatchException`.
+Stopping and starting both nodes together, which is already the rule above, is the whole fix; no
+manual trigger or oplog migration is needed, because `RegisterReplicated` re-reads
+`PRAGMA table_info` on every boot and rebuilds the capture triggers (DROP + CREATE) over the
+current column set. The site schema's own `AddColumnIfMissing` runs before registration on that
+same boot, so a node picks up the new columns and the matching triggers in one start.
+(Most recent instance: `external_systems` gained `max_retries` / `retry_delay_ms` for the
+per-system store-and-forward retry policy.)
+
 **Related bound — do not leave one node of a pair offline for long.** A node absent for longer than
 `LocalDb:Replication:TombstoneRetention` (default **7 days**) can **resurrect deleted rows** when
 it rejoins: deletes replicate as HLC-ordered tombstones, and once a tombstone is pruned there is

docs/requirements/Component-CLI.md

@@ -194,8 +194,8 @@ scadabridge data-connection certs remove --site <identifier> --thumbprint <hex>
 ```
 scadabridge external-system list
 scadabridge external-system get --id <id>
-scadabridge external-system create --name <name> --endpoint-url <url> --auth-type <type> [--auth-config <json>]
-scadabridge external-system update --id <id> [--name <name>] [--endpoint-url <url>] [--auth-type <type>] [--auth-config <json>]
+scadabridge external-system create --name <name> --endpoint-url <url> --auth-type <type> [--auth-config <json>] [--timeout-seconds <n>] [--max-retries <n>] [--retry-delay-seconds <n>]
+scadabridge external-system update --id <id> [--name <name>] [--endpoint-url <url>] [--auth-type <type>] [--auth-config <json>] [--timeout-seconds <n>] [--max-retries <n>] [--retry-delay-seconds <n>]
 scadabridge external-system delete --id <id>
 scadabridge external-system method list --external-system-id <id>
 scadabridge external-system method get --id <id>

docs/requirements/Component-ExternalSystemGateway.md

@@ -42,7 +42,7 @@ Each external system definition includes:
 
   **Legacy colon form (deprecated):** a value that does not start with `{` is parsed with the historical colon-split rules for backward compatibility — API Key as `HeaderName:KeyValue` (or a bare `KeyValue` using the default `X-API-Key` header), Basic Auth as `username:password`. This form cannot represent keys or passwords that contain `:` and is retained only so existing rows keep working; new definitions should use the JSON form. Malformed JSON (a value that starts with `{` but does not parse, or yields no usable `key`/`username`) is **not** an exception — the request is sent without the auth header and a warning is logged (the configuration value itself is never logged).
 - **Timeout**: Per-system timeout for all method calls (e.g., 30 seconds). Applies to the HTTP request round-trip. `0` (the default) means "unset" — the gateway's configured `DefaultHttpTimeout` applies instead. Set via the Central UI External System form's "Timeout (seconds)" field; ships to sites in the deployment artifact like the other definition fields.
-- **Retry Settings**: Max retry count, fixed time between retries (used by Store-and-Forward Engine for transient failures only).
+- **Retry Settings**: Max retry count, fixed time between retries (used by Store-and-Forward Engine for transient failures only). Both are **`0` = unset**, in which case the engine's `DefaultMaxRetries` (50) and `DefaultRetryInterval` (30 s) apply — note this is the inverse of the S&F engine's own `MaxRetries` contract, where a stored `0` means *unbounded* retry; the gateway translates an unset `0` to "no per-system override" rather than passing it through. Authored via the Central UI External System form ("Max Retries" / "Retry Delay (seconds)") or `external-system create|update --max-retries --retry-delay-seconds`, and **deployed to sites in the external-system artifact** alongside the timeout, so the site's cached-call buffering honours the configured policy. (Before the row-27 fix the artifact carried no retry fields and the site table had no columns for them, so a centrally-configured policy was silently ignored and every cached call used the engine default.)
 - **Method Definitions**: List of available API methods, each with:
   - Method name.
   - **HTTP method**: GET, POST, PUT, PATCH, or DELETE.

docs/requirements/Component-StoreAndForward.md

@@ -63,6 +63,19 @@ For the external-system-call and cached-database-write categories, retry setting
 - **External systems**: Each external system definition includes max retry count and time between retries.
 - **Cached database writes**: Each database connection definition includes max retry count and time between retries.
 
+Both are carried to the site in the deployment artifact and read back out of the site's own
+tables by the gateway, which passes them to `EnqueueAsync`. An **unset** source-entity value
+(`MaxRetries` `0` / `RetryDelay` `TimeSpan.Zero`) is passed as `null`, so the engine's
+`DefaultMaxRetries` (50) and `DefaultRetryInterval` (30 s) apply — roughly 25 minutes to park.
+The translation is deliberate and must not be removed: the engine reads a stored `MaxRetries`
+of `0` as **no limit**, so forwarding an unconfigured entity's `0` verbatim would turn every
+cached call into an unbounded retry loop.
+
+> Until deferred-work row 27 was fixed, this was true of database connections only.
+> `ExternalSystemArtifact` carried no retry fields and the site `external_systems` table had no
+> columns for them, so an external system's centrally-configured policy never reached the site
+> and every external-system cached call silently used the 50 x 30 s default.
+
 The **notification** category retries differently: it has no source-entity setting. The site→central forward uses a single fixed retry interval configured in the host `appsettings.json`. This interval is infrastructure config for reaching the central cluster, not a per-notification-list setting. It applies uniformly to every buffered notification regardless of its target list. A buffered notification is retried at that interval until central acks it; the engine's `DefaultMaxRetries` cap still applies (matching the cached-call categories) and a notification whose retries are exhausted under a sustained central outage parks like any other buffered message. The cap is sized so the normal central-recovery window stays well inside it — central, once reachable, owns delivery, retry, and parking from the ack point on.
 
 The retry interval is **fixed** (not exponential backoff). Fixed interval is sufficient for the expected use cases.