feat(external-systems): deploy per-system retry policy to sites + CLI/management surface (register row 27)

An external system's MaxRetries/RetryDelay existed on the central entity but never
reached a site and had no management surface, so every ExternalSystem.CachedCall
buffered with the S&F engine default (DefaultMaxRetries 50 x DefaultRetryInterval
30s ~= 25 min to park) whatever central had configured. The sibling
DatabaseConnectionArtifact had always carried its equivalents.

The consumption end was already wired: ExternalSystemClient.CachedCallAsync already
read system.MaxRetries/RetryDelay and passed them to EnqueueAsync with the 0-as-unset
convention. Only the transport and persistence were missing.

Commons
- ExternalSystemArtifact gains MaxRetries/RetryDelay as optional positional params
  (0 / TimeSpan.Zero = unset), matching DatabaseConnectionArtifact rather than going
  nullable, so it maps 1:1 onto the entity and the proto3 scalars.
- Create/UpdateExternalSystemCommand gain nullable MaxRetries/RetryDelay, following
  the existing TimeoutSeconds "omit to preserve" convention. Both additive.

Central
- ManagementActor create maps both; update preserves-if-null.
- ArtifactDeploymentService projects them onto the artifact.

Wire
- site_command.proto ExternalSystemArtifactDto gains max_retries=7 /
  retry_delay=8 (google.protobuf.Duration), mirroring DatabaseConnectionArtifactDto.
  Additive: a pre-existing site decodes them as 0/absent, which IS the unset value,
  so old behaviour is preserved. Vendored C# regenerated via docker/regen-proto.sh
  sitecommand; csproj Protobuf item re-commented, no message types added or removed.

Site
- external_systems gains max_retries / retry_delay_ms via AddColumnIfMissing,
  DEFAULT 0 = unset. Storage upsert, repository SELECT/mapping (new ordinals 6/7
  appended so existing ones are untouched), apply path, and the legacy migrator's
  explicit column list all carry them.

Replicated-table decision: external_systems IS in SiteLocalDbSetup.ReplicatedTables,
so this was checked against the AckTime/native_alarm_state precedent. Real columns
are correct here, not a JSON bag: the table has no general-purpose config column
(auth_configuration and method_definitions are both semantically specific),
timeout_seconds already set the additive-column precedent on this exact table, and
sibling database_connections stores the same two values as plain columns. Verified in
LocalDb source that RegisterReplicated re-reads PRAGMA table_info every boot and its
install script is DROP-then-CREATE, so capture triggers rebuild over the new column
set automatically, and SiteStorageSchema.Apply runs before registration on the same
boot. The only cost is the ReplicatedTable.ComputeDigest change, which the sync
handshake compares fail-closed - a mixed-version pair refuses to sync loudly instead
of diverging silently. Both-nodes-together is already the standing site-pair rule, so
no new operational constraint; recorded in topology-guide.md.

Also fixed: ArtifactDiff.CompareExternalSystem never compared MaxRetries/RetryDelay,
so a bundle import that changed a system's retry budget previewed as "Identical" -
harmless while the values were inert, a live behaviour change now.

No EF migration: the central ExternalSystemDefinitions columns already exist
(InitialSchema). Central UI already had both form fields.

Tests: site storage round-trip + unset-default, site apply path, central artifact
projection, proto round-trip (via the shared sample), management create/preserve,
CLI option parse + alias, and diff reporting. Fail-before verified on the two core
tests (expected 7 got 0 on the read path; expected 9 got 0 on the apply path).

Docs: Component-ExternalSystemGateway, Component-StoreAndForward, Component-CLI,
components/StoreAndForward, components/SiteRuntime, deployment/topology-guide,
CLI README (which was also missing --timeout-seconds entirely).
This commit is contained in:
Joseph Doherty
2026-08-15 12:56:12 -04:00
parent ef876b38c5
commit 41a5ecf14a
28 changed files with 1036 additions and 453 deletions
@@ -133,6 +133,28 @@ public static class SiteStorageSchema
// Per-external-system call timeout (ExternalSystemGateway Timeout) — carried through the
// artifact pipeline so site-side calls honor it; 0 = use the site default.
AddColumnIfMissing(connection, "external_systems", "timeout_seconds", "INTEGER NOT NULL DEFAULT 0");
// Store-and-Forward retry policy, deployed per external system (deferred-work row 27).
//
// 0 = unset on BOTH columns, which is what makes this additive change behaviourally
// invisible until an operator actually configures a policy: an existing row backfilled
// with the DEFAULT reads back as 0, and ExternalSystemClient.CachedCallAsync maps a 0
// to null so the engine's DefaultMaxRetries/DefaultRetryInterval still apply. (Contrast
// database_connections, whose equivalent columns default to 3/1000 — that table has
// never had an "unset means engine default" contract.)
//
// These land on a table in SiteLocalDbSetup.ReplicatedTables. That is safe and needs no
// trigger migration: RegisterReplicated re-reads PRAGMA table_info on EVERY boot and its
// install script is DROP-then-CREATE, so the capture triggers are rebuilt over the new
// column set automatically — and Apply() runs before RegisterReplicated, so the rebuild
// sees these columns on the same boot that adds them. What DOES change is the table's
// schema digest (ReplicatedTable.ComputeDigest hashes every column name + type), which
// the sync handshake compares fail-closed; a pair running mismatched builds refuses to
// sync loudly (LocalDbSchemaMismatchException) rather than corrupting anything. Both
// nodes must therefore be upgraded together — already the standing rule for a site pair
// (docs/deployment/topology-guide.md), so this introduces no new operational constraint.
AddColumnIfMissing(connection, "external_systems", "max_retries", "INTEGER NOT NULL DEFAULT 0");
AddColumnIfMissing(connection, "external_systems", "retry_delay_ms", "INTEGER NOT NULL DEFAULT 0");
}
/// <summary>