merge: external-system retry policy end-to-end (register row 27) — MaxRetries/RetryDelay now travel artifact→gRPC→site schema→S&F enqueue, CLI --max-retries/--retry-delay-seconds, ArtifactDiff compares them

This commit is contained in:
Joseph Doherty
2026-08-15 17:56:21 -04:00
28 changed files with 1036 additions and 453 deletions
+2 -2
View File
@@ -194,8 +194,8 @@ scadabridge data-connection certs remove --site <identifier> --thumbprint <hex>
```
scadabridge external-system list
scadabridge external-system get --id <id>
scadabridge external-system create --name <name> --endpoint-url <url> --auth-type <type> [--auth-config <json>]
scadabridge external-system update --id <id> [--name <name>] [--endpoint-url <url>] [--auth-type <type>] [--auth-config <json>]
scadabridge external-system create --name <name> --endpoint-url <url> --auth-type <type> [--auth-config <json>] [--timeout-seconds <n>] [--max-retries <n>] [--retry-delay-seconds <n>]
scadabridge external-system update --id <id> [--name <name>] [--endpoint-url <url>] [--auth-type <type>] [--auth-config <json>] [--timeout-seconds <n>] [--max-retries <n>] [--retry-delay-seconds <n>]
scadabridge external-system delete --id <id>
scadabridge external-system method list --external-system-id <id>
scadabridge external-system method get --id <id>
@@ -42,7 +42,7 @@ Each external system definition includes:
**Legacy colon form (deprecated):** a value that does not start with `{` is parsed with the historical colon-split rules for backward compatibility — API Key as `HeaderName:KeyValue` (or a bare `KeyValue` using the default `X-API-Key` header), Basic Auth as `username:password`. This form cannot represent keys or passwords that contain `:` and is retained only so existing rows keep working; new definitions should use the JSON form. Malformed JSON (a value that starts with `{` but does not parse, or yields no usable `key`/`username`) is **not** an exception — the request is sent without the auth header and a warning is logged (the configuration value itself is never logged).
- **Timeout**: Per-system timeout for all method calls (e.g., 30 seconds). Applies to the HTTP request round-trip. `0` (the default) means "unset" — the gateway's configured `DefaultHttpTimeout` applies instead. Set via the Central UI External System form's "Timeout (seconds)" field; ships to sites in the deployment artifact like the other definition fields.
- **Retry Settings**: Max retry count, fixed time between retries (used by Store-and-Forward Engine for transient failures only).
- **Retry Settings**: Max retry count, fixed time between retries (used by Store-and-Forward Engine for transient failures only). Both are **`0` = unset**, in which case the engine's `DefaultMaxRetries` (50) and `DefaultRetryInterval` (30 s) apply — note this is the inverse of the S&F engine's own `MaxRetries` contract, where a stored `0` means *unbounded* retry; the gateway translates an unset `0` to "no per-system override" rather than passing it through. Authored via the Central UI External System form ("Max Retries" / "Retry Delay (seconds)") or `external-system create|update --max-retries --retry-delay-seconds`, and **deployed to sites in the external-system artifact** alongside the timeout, so the site's cached-call buffering honours the configured policy. (Before the row-27 fix the artifact carried no retry fields and the site table had no columns for them, so a centrally-configured policy was silently ignored and every cached call used the engine default.)
- **Method Definitions**: List of available API methods, each with:
- Method name.
- **HTTP method**: GET, POST, PUT, PATCH, or DELETE.
@@ -63,6 +63,19 @@ For the external-system-call and cached-database-write categories, retry setting
- **External systems**: Each external system definition includes max retry count and time between retries.
- **Cached database writes**: Each database connection definition includes max retry count and time between retries.
Both are carried to the site in the deployment artifact and read back out of the site's own
tables by the gateway, which passes them to `EnqueueAsync`. An **unset** source-entity value
(`MaxRetries` `0` / `RetryDelay` `TimeSpan.Zero`) is passed as `null`, so the engine's
`DefaultMaxRetries` (50) and `DefaultRetryInterval` (30 s) apply — roughly 25 minutes to park.
The translation is deliberate and must not be removed: the engine reads a stored `MaxRetries`
of `0` as **no limit**, so forwarding an unconfigured entity's `0` verbatim would turn every
cached call into an unbounded retry loop.
> Until deferred-work row 27 was fixed, this was true of database connections only.
> `ExternalSystemArtifact` carried no retry fields and the site `external_systems` table had no
> columns for them, so an external system's centrally-configured policy never reached the site
> and every external-system cached call silently used the 50 x 30 s default.
The **notification** category retries differently: it has no source-entity setting. The site→central forward uses a single fixed retry interval configured in the host `appsettings.json`. This interval is infrastructure config for reaching the central cluster, not a per-notification-list setting. It applies uniformly to every buffered notification regardless of its target list. A buffered notification is retried at that interval until central acks it; the engine's `DefaultMaxRetries` cap still applies (matching the cached-call categories) and a notification whose retries are exhausted under a sustained central outage parks like any other buffered message. The cap is sized so the normal central-recovery window stays well inside it — central, once reachable, owns delivery, retry, and parking from the ack point on.
The retry interval is **fixed** (not exponential backoff). Fixed interval is sufficient for the expected use cases.