feat(external-systems): deploy per-system retry policy to sites + CLI/management surface (register row 27)

An external system's MaxRetries/RetryDelay existed on the central entity but never
reached a site and had no management surface, so every ExternalSystem.CachedCall
buffered with the S&F engine default (DefaultMaxRetries 50 x DefaultRetryInterval
30s ~= 25 min to park) whatever central had configured. The sibling
DatabaseConnectionArtifact had always carried its equivalents.

The consumption end was already wired: ExternalSystemClient.CachedCallAsync already
read system.MaxRetries/RetryDelay and passed them to EnqueueAsync with the 0-as-unset
convention. Only the transport and persistence were missing.

Commons
- ExternalSystemArtifact gains MaxRetries/RetryDelay as optional positional params
  (0 / TimeSpan.Zero = unset), matching DatabaseConnectionArtifact rather than going
  nullable, so it maps 1:1 onto the entity and the proto3 scalars.
- Create/UpdateExternalSystemCommand gain nullable MaxRetries/RetryDelay, following
  the existing TimeoutSeconds "omit to preserve" convention. Both additive.

Central
- ManagementActor create maps both; update preserves-if-null.
- ArtifactDeploymentService projects them onto the artifact.

Wire
- site_command.proto ExternalSystemArtifactDto gains max_retries=7 /
  retry_delay=8 (google.protobuf.Duration), mirroring DatabaseConnectionArtifactDto.
  Additive: a pre-existing site decodes them as 0/absent, which IS the unset value,
  so old behaviour is preserved. Vendored C# regenerated via docker/regen-proto.sh
  sitecommand; csproj Protobuf item re-commented, no message types added or removed.

Site
- external_systems gains max_retries / retry_delay_ms via AddColumnIfMissing,
  DEFAULT 0 = unset. Storage upsert, repository SELECT/mapping (new ordinals 6/7
  appended so existing ones are untouched), apply path, and the legacy migrator's
  explicit column list all carry them.

Replicated-table decision: external_systems IS in SiteLocalDbSetup.ReplicatedTables,
so this was checked against the AckTime/native_alarm_state precedent. Real columns
are correct here, not a JSON bag: the table has no general-purpose config column
(auth_configuration and method_definitions are both semantically specific),
timeout_seconds already set the additive-column precedent on this exact table, and
sibling database_connections stores the same two values as plain columns. Verified in
LocalDb source that RegisterReplicated re-reads PRAGMA table_info every boot and its
install script is DROP-then-CREATE, so capture triggers rebuild over the new column
set automatically, and SiteStorageSchema.Apply runs before registration on the same
boot. The only cost is the ReplicatedTable.ComputeDigest change, which the sync
handshake compares fail-closed - a mixed-version pair refuses to sync loudly instead
of diverging silently. Both-nodes-together is already the standing site-pair rule, so
no new operational constraint; recorded in topology-guide.md.

Also fixed: ArtifactDiff.CompareExternalSystem never compared MaxRetries/RetryDelay,
so a bundle import that changed a system's retry budget previewed as "Identical" -
harmless while the values were inert, a live behaviour change now.

No EF migration: the central ExternalSystemDefinitions columns already exist
(InitialSchema). Central UI already had both form fields.

Tests: site storage round-trip + unset-default, site apply path, central artifact
projection, proto round-trip (via the shared sample), management create/preserve,
CLI option parse + alias, and diff reporting. Fail-before verified on the two core
tests (expected 7 got 0 on the read path; expected 9 got 0 on the apply path).

Docs: Component-ExternalSystemGateway, Component-StoreAndForward, Component-CLI,
components/StoreAndForward, components/SiteRuntime, deployment/topology-guide,
CLI README (which was also missing --timeout-seconds entirely).
This commit is contained in:
Joseph Doherty
2026-08-15 12:56:12 -04:00
parent ef876b38c5
commit 41a5ecf14a
28 changed files with 1036 additions and 453 deletions
+1 -1
View File
@@ -205,7 +205,7 @@ Both `AlarmActor` and `NativeAlarmActor` tell the `InstanceActor` an `AlarmState
| `static_attribute_overrides` | Runtime attribute writes (`SetAttribute` on static attrs) | Yes — cleared by `ClearStaticOverridesAsync` |
| `native_alarm_state` | Mirrored native alarm conditions (survives failover) | Yes — cleared by `ClearNativeAlarmsForInstanceAsync` |
| `shared_scripts` | Shared script code from artifact deployments | No |
| `external_systems` | External system definitions | No |
| `external_systems` | External system definitions (incl. `timeout_seconds`, and `max_retries`/`retry_delay_ms` — the deployed store-and-forward retry policy, `0` = use the engine default) | No |
| `database_connections` | Database connection strings | No |
| `data_connection_definitions` | OPC UA / MxGateway endpoint definitions | No |
| `notification_lists` | Notification list definitions | No |
+2
View File
@@ -51,6 +51,8 @@ The retry interval is fixed — not exponential. There is no maximum buffer size
`StoreAndForwardMessage.MaxRetries` controls how many background-sweep attempts the engine makes before parking. `MaxRetries = 0` means **no limit** — the message retries on every sweep until delivered and is never parked for retry exhaustion. It is not a "never retry" value; callers that want unbounded retry pass `maxRetries: 0` explicitly. The `EnqueueAsync` `maxRetries` parameter defaults to `StoreAndForwardOptions.DefaultMaxRetries` (50).
**The source-entity `0` means the opposite, and the gateways convert.** On an `ExternalSystemDefinition` / `DatabaseConnectionDefinition`, `MaxRetries = 0` (and `RetryDelay = TimeSpan.Zero`) mean *unset*, not *unbounded* — so `ExternalSystemClient.CachedCallAsync` and `DatabaseGateway` pass `system.MaxRetries > 0 ? system.MaxRetries : null`, letting the bounded engine default apply. Forwarding the entity's `0` verbatim would make every unconfigured cached call retry forever. The external system's values only actually reach a site as of deferred-work row 27; before that the artifact did not carry them.
### Messages not cleared on instance deletion
When an instance is deleted, its buffered S&F messages are not removed. `StoreAndForwardMessage.OriginInstanceName` records the originating instance at enqueue time so the buffer can continue to deliver and so the central UI can attribute parked messages even after the instance is gone.
+12
View File
@@ -265,6 +265,18 @@ will run, but they will not converge, and the divergence is silent.
Stop both nodes of a site pair, upgrade both, then start both.
**A release that adds a column to a replicated table is the same constraint, enforced louder.**
`ReplicatedTable.ComputeDigest` hashes every column name and declared type, and the sync handshake
compares those digests fail-closed — so a mixed-version pair whose builds disagree about a table's
columns does not diverge silently, it refuses to sync with a `LocalDbSchemaMismatchException`.
Stopping and starting both nodes together, which is already the rule above, is the whole fix; no
manual trigger or oplog migration is needed, because `RegisterReplicated` re-reads
`PRAGMA table_info` on every boot and rebuilds the capture triggers (DROP + CREATE) over the
current column set. The site schema's own `AddColumnIfMissing` runs before registration on that
same boot, so a node picks up the new columns and the matching triggers in one start.
(Most recent instance: `external_systems` gained `max_retries` / `retry_delay_ms` for the
per-system store-and-forward retry policy.)
**Related bound — do not leave one node of a pair offline for long.** A node absent for longer than
`LocalDb:Replication:TombstoneRetention` (default **7 days**) can **resurrect deleted rows** when
it rejoins: deletes replicate as HLC-ordered tombstones, and once a tombstone is pruned there is
+2 -2
View File
@@ -194,8 +194,8 @@ scadabridge data-connection certs remove --site <identifier> --thumbprint <hex>
```
scadabridge external-system list
scadabridge external-system get --id <id>
scadabridge external-system create --name <name> --endpoint-url <url> --auth-type <type> [--auth-config <json>]
scadabridge external-system update --id <id> [--name <name>] [--endpoint-url <url>] [--auth-type <type>] [--auth-config <json>]
scadabridge external-system create --name <name> --endpoint-url <url> --auth-type <type> [--auth-config <json>] [--timeout-seconds <n>] [--max-retries <n>] [--retry-delay-seconds <n>]
scadabridge external-system update --id <id> [--name <name>] [--endpoint-url <url>] [--auth-type <type>] [--auth-config <json>] [--timeout-seconds <n>] [--max-retries <n>] [--retry-delay-seconds <n>]
scadabridge external-system delete --id <id>
scadabridge external-system method list --external-system-id <id>
scadabridge external-system method get --id <id>
@@ -42,7 +42,7 @@ Each external system definition includes:
**Legacy colon form (deprecated):** a value that does not start with `{` is parsed with the historical colon-split rules for backward compatibility — API Key as `HeaderName:KeyValue` (or a bare `KeyValue` using the default `X-API-Key` header), Basic Auth as `username:password`. This form cannot represent keys or passwords that contain `:` and is retained only so existing rows keep working; new definitions should use the JSON form. Malformed JSON (a value that starts with `{` but does not parse, or yields no usable `key`/`username`) is **not** an exception — the request is sent without the auth header and a warning is logged (the configuration value itself is never logged).
- **Timeout**: Per-system timeout for all method calls (e.g., 30 seconds). Applies to the HTTP request round-trip. `0` (the default) means "unset" — the gateway's configured `DefaultHttpTimeout` applies instead. Set via the Central UI External System form's "Timeout (seconds)" field; ships to sites in the deployment artifact like the other definition fields.
- **Retry Settings**: Max retry count, fixed time between retries (used by Store-and-Forward Engine for transient failures only).
- **Retry Settings**: Max retry count, fixed time between retries (used by Store-and-Forward Engine for transient failures only). Both are **`0` = unset**, in which case the engine's `DefaultMaxRetries` (50) and `DefaultRetryInterval` (30 s) apply — note this is the inverse of the S&F engine's own `MaxRetries` contract, where a stored `0` means *unbounded* retry; the gateway translates an unset `0` to "no per-system override" rather than passing it through. Authored via the Central UI External System form ("Max Retries" / "Retry Delay (seconds)") or `external-system create|update --max-retries --retry-delay-seconds`, and **deployed to sites in the external-system artifact** alongside the timeout, so the site's cached-call buffering honours the configured policy. (Before the row-27 fix the artifact carried no retry fields and the site table had no columns for them, so a centrally-configured policy was silently ignored and every cached call used the engine default.)
- **Method Definitions**: List of available API methods, each with:
- Method name.
- **HTTP method**: GET, POST, PUT, PATCH, or DELETE.
@@ -63,6 +63,19 @@ For the external-system-call and cached-database-write categories, retry setting
- **External systems**: Each external system definition includes max retry count and time between retries.
- **Cached database writes**: Each database connection definition includes max retry count and time between retries.
Both are carried to the site in the deployment artifact and read back out of the site's own
tables by the gateway, which passes them to `EnqueueAsync`. An **unset** source-entity value
(`MaxRetries` `0` / `RetryDelay` `TimeSpan.Zero`) is passed as `null`, so the engine's
`DefaultMaxRetries` (50) and `DefaultRetryInterval` (30 s) apply — roughly 25 minutes to park.
The translation is deliberate and must not be removed: the engine reads a stored `MaxRetries`
of `0` as **no limit**, so forwarding an unconfigured entity's `0` verbatim would turn every
cached call into an unbounded retry loop.
> Until deferred-work row 27 was fixed, this was true of database connections only.
> `ExternalSystemArtifact` carried no retry fields and the site `external_systems` table had no
> columns for them, so an external system's centrally-configured policy never reached the site
> and every external-system cached call silently used the 50 x 30 s default.
The **notification** category retries differently: it has no source-entity setting. The site→central forward uses a single fixed retry interval configured in the host `appsettings.json`. This interval is infrastructure config for reaching the central cluster, not a per-notification-list setting. It applies uniformly to every buffered notification regardless of its target list. A buffered notification is retried at that interval until central acks it; the engine's `DefaultMaxRetries` cap still applies (matching the cached-call categories) and a notification whose retries are exhausted under a sustained central outage parks like any other buffered message. The cap is sized so the normal central-recovery window stays well inside it — central, once reachable, owns delivery, retry, and parking from the ack point on.
The retry interval is **fixed** (not exponential backoff). Fixed interval is sufficient for the expected use cases.