merge: external-system retry policy end-to-end (register row 27) — MaxRetries/RetryDelay now travel artifact→gRPC→site schema→S&F enqueue, CLI --max-retries/--retry-delay-seconds, ArtifactDiff compares them

This commit is contained in:
Joseph Doherty
2026-08-15 17:56:21 -04:00
28 changed files with 1036 additions and 453 deletions
+1 -1
View File
@@ -205,7 +205,7 @@ Both `AlarmActor` and `NativeAlarmActor` tell the `InstanceActor` an `AlarmState
| `static_attribute_overrides` | Runtime attribute writes (`SetAttribute` on static attrs) | Yes — cleared by `ClearStaticOverridesAsync` | | `static_attribute_overrides` | Runtime attribute writes (`SetAttribute` on static attrs) | Yes — cleared by `ClearStaticOverridesAsync` |
| `native_alarm_state` | Mirrored native alarm conditions (survives failover) | Yes — cleared by `ClearNativeAlarmsForInstanceAsync` | | `native_alarm_state` | Mirrored native alarm conditions (survives failover) | Yes — cleared by `ClearNativeAlarmsForInstanceAsync` |
| `shared_scripts` | Shared script code from artifact deployments | No | | `shared_scripts` | Shared script code from artifact deployments | No |
| `external_systems` | External system definitions | No | | `external_systems` | External system definitions (incl. `timeout_seconds`, and `max_retries`/`retry_delay_ms` — the deployed store-and-forward retry policy, `0` = use the engine default) | No |
| `database_connections` | Database connection strings | No | | `database_connections` | Database connection strings | No |
| `data_connection_definitions` | OPC UA / MxGateway endpoint definitions | No | | `data_connection_definitions` | OPC UA / MxGateway endpoint definitions | No |
| `notification_lists` | Notification list definitions | No | | `notification_lists` | Notification list definitions | No |
+2
View File
@@ -51,6 +51,8 @@ The retry interval is fixed — not exponential. There is no maximum buffer size
`StoreAndForwardMessage.MaxRetries` controls how many background-sweep attempts the engine makes before parking. `MaxRetries = 0` means **no limit** — the message retries on every sweep until delivered and is never parked for retry exhaustion. It is not a "never retry" value; callers that want unbounded retry pass `maxRetries: 0` explicitly. The `EnqueueAsync` `maxRetries` parameter defaults to `StoreAndForwardOptions.DefaultMaxRetries` (50). `StoreAndForwardMessage.MaxRetries` controls how many background-sweep attempts the engine makes before parking. `MaxRetries = 0` means **no limit** — the message retries on every sweep until delivered and is never parked for retry exhaustion. It is not a "never retry" value; callers that want unbounded retry pass `maxRetries: 0` explicitly. The `EnqueueAsync` `maxRetries` parameter defaults to `StoreAndForwardOptions.DefaultMaxRetries` (50).
**The source-entity `0` means the opposite, and the gateways convert.** On an `ExternalSystemDefinition` / `DatabaseConnectionDefinition`, `MaxRetries = 0` (and `RetryDelay = TimeSpan.Zero`) mean *unset*, not *unbounded* — so `ExternalSystemClient.CachedCallAsync` and `DatabaseGateway` pass `system.MaxRetries > 0 ? system.MaxRetries : null`, letting the bounded engine default apply. Forwarding the entity's `0` verbatim would make every unconfigured cached call retry forever. The external system's values only actually reach a site as of deferred-work row 27; before that the artifact did not carry them.
### Messages not cleared on instance deletion ### Messages not cleared on instance deletion
When an instance is deleted, its buffered S&F messages are not removed. `StoreAndForwardMessage.OriginInstanceName` records the originating instance at enqueue time so the buffer can continue to deliver and so the central UI can attribute parked messages even after the instance is gone. When an instance is deleted, its buffered S&F messages are not removed. `StoreAndForwardMessage.OriginInstanceName` records the originating instance at enqueue time so the buffer can continue to deliver and so the central UI can attribute parked messages even after the instance is gone.
+12
View File
@@ -291,6 +291,18 @@ will run, but they will not converge, and the divergence is silent.
Stop both nodes of a site pair, upgrade both, then start both. Stop both nodes of a site pair, upgrade both, then start both.
**A release that adds a column to a replicated table is the same constraint, enforced louder.**
`ReplicatedTable.ComputeDigest` hashes every column name and declared type, and the sync handshake
compares those digests fail-closed — so a mixed-version pair whose builds disagree about a table's
columns does not diverge silently, it refuses to sync with a `LocalDbSchemaMismatchException`.
Stopping and starting both nodes together, which is already the rule above, is the whole fix; no
manual trigger or oplog migration is needed, because `RegisterReplicated` re-reads
`PRAGMA table_info` on every boot and rebuilds the capture triggers (DROP + CREATE) over the
current column set. The site schema's own `AddColumnIfMissing` runs before registration on that
same boot, so a node picks up the new columns and the matching triggers in one start.
(Most recent instance: `external_systems` gained `max_retries` / `retry_delay_ms` for the
per-system store-and-forward retry policy.)
**Related bound — do not leave one node of a pair offline for long.** A node absent for longer than **Related bound — do not leave one node of a pair offline for long.** A node absent for longer than
`LocalDb:Replication:TombstoneRetention` (default **7 days**) can **resurrect deleted rows** when `LocalDb:Replication:TombstoneRetention` (default **7 days**) can **resurrect deleted rows** when
it rejoins: deletes replicate as HLC-ordered tombstones, and once a tombstone is pruned there is it rejoins: deletes replicate as HLC-ordered tombstones, and once a tombstone is pruned there is
+2 -2
View File
@@ -194,8 +194,8 @@ scadabridge data-connection certs remove --site <identifier> --thumbprint <hex>
``` ```
scadabridge external-system list scadabridge external-system list
scadabridge external-system get --id <id> scadabridge external-system get --id <id>
scadabridge external-system create --name <name> --endpoint-url <url> --auth-type <type> [--auth-config <json>] scadabridge external-system create --name <name> --endpoint-url <url> --auth-type <type> [--auth-config <json>] [--timeout-seconds <n>] [--max-retries <n>] [--retry-delay-seconds <n>]
scadabridge external-system update --id <id> [--name <name>] [--endpoint-url <url>] [--auth-type <type>] [--auth-config <json>] scadabridge external-system update --id <id> [--name <name>] [--endpoint-url <url>] [--auth-type <type>] [--auth-config <json>] [--timeout-seconds <n>] [--max-retries <n>] [--retry-delay-seconds <n>]
scadabridge external-system delete --id <id> scadabridge external-system delete --id <id>
scadabridge external-system method list --external-system-id <id> scadabridge external-system method list --external-system-id <id>
scadabridge external-system method get --id <id> scadabridge external-system method get --id <id>
@@ -42,7 +42,7 @@ Each external system definition includes:
**Legacy colon form (deprecated):** a value that does not start with `{` is parsed with the historical colon-split rules for backward compatibility — API Key as `HeaderName:KeyValue` (or a bare `KeyValue` using the default `X-API-Key` header), Basic Auth as `username:password`. This form cannot represent keys or passwords that contain `:` and is retained only so existing rows keep working; new definitions should use the JSON form. Malformed JSON (a value that starts with `{` but does not parse, or yields no usable `key`/`username`) is **not** an exception — the request is sent without the auth header and a warning is logged (the configuration value itself is never logged). **Legacy colon form (deprecated):** a value that does not start with `{` is parsed with the historical colon-split rules for backward compatibility — API Key as `HeaderName:KeyValue` (or a bare `KeyValue` using the default `X-API-Key` header), Basic Auth as `username:password`. This form cannot represent keys or passwords that contain `:` and is retained only so existing rows keep working; new definitions should use the JSON form. Malformed JSON (a value that starts with `{` but does not parse, or yields no usable `key`/`username`) is **not** an exception — the request is sent without the auth header and a warning is logged (the configuration value itself is never logged).
- **Timeout**: Per-system timeout for all method calls (e.g., 30 seconds). Applies to the HTTP request round-trip. `0` (the default) means "unset" — the gateway's configured `DefaultHttpTimeout` applies instead. Set via the Central UI External System form's "Timeout (seconds)" field; ships to sites in the deployment artifact like the other definition fields. - **Timeout**: Per-system timeout for all method calls (e.g., 30 seconds). Applies to the HTTP request round-trip. `0` (the default) means "unset" — the gateway's configured `DefaultHttpTimeout` applies instead. Set via the Central UI External System form's "Timeout (seconds)" field; ships to sites in the deployment artifact like the other definition fields.
- **Retry Settings**: Max retry count, fixed time between retries (used by Store-and-Forward Engine for transient failures only). - **Retry Settings**: Max retry count, fixed time between retries (used by Store-and-Forward Engine for transient failures only). Both are **`0` = unset**, in which case the engine's `DefaultMaxRetries` (50) and `DefaultRetryInterval` (30 s) apply — note this is the inverse of the S&F engine's own `MaxRetries` contract, where a stored `0` means *unbounded* retry; the gateway translates an unset `0` to "no per-system override" rather than passing it through. Authored via the Central UI External System form ("Max Retries" / "Retry Delay (seconds)") or `external-system create|update --max-retries --retry-delay-seconds`, and **deployed to sites in the external-system artifact** alongside the timeout, so the site's cached-call buffering honours the configured policy. (Before the row-27 fix the artifact carried no retry fields and the site table had no columns for them, so a centrally-configured policy was silently ignored and every cached call used the engine default.)
- **Method Definitions**: List of available API methods, each with: - **Method Definitions**: List of available API methods, each with:
- Method name. - Method name.
- **HTTP method**: GET, POST, PUT, PATCH, or DELETE. - **HTTP method**: GET, POST, PUT, PATCH, or DELETE.
@@ -63,6 +63,19 @@ For the external-system-call and cached-database-write categories, retry setting
- **External systems**: Each external system definition includes max retry count and time between retries. - **External systems**: Each external system definition includes max retry count and time between retries.
- **Cached database writes**: Each database connection definition includes max retry count and time between retries. - **Cached database writes**: Each database connection definition includes max retry count and time between retries.
Both are carried to the site in the deployment artifact and read back out of the site's own
tables by the gateway, which passes them to `EnqueueAsync`. An **unset** source-entity value
(`MaxRetries` `0` / `RetryDelay` `TimeSpan.Zero`) is passed as `null`, so the engine's
`DefaultMaxRetries` (50) and `DefaultRetryInterval` (30 s) apply — roughly 25 minutes to park.
The translation is deliberate and must not be removed: the engine reads a stored `MaxRetries`
of `0` as **no limit**, so forwarding an unconfigured entity's `0` verbatim would turn every
cached call into an unbounded retry loop.
> Until deferred-work row 27 was fixed, this was true of database connections only.
> `ExternalSystemArtifact` carried no retry fields and the site `external_systems` table had no
> columns for them, so an external system's centrally-configured policy never reached the site
> and every external-system cached call silently used the 50 x 30 s default.
The **notification** category retries differently: it has no source-entity setting. The site→central forward uses a single fixed retry interval configured in the host `appsettings.json`. This interval is infrastructure config for reaching the central cluster, not a per-notification-list setting. It applies uniformly to every buffered notification regardless of its target list. A buffered notification is retried at that interval until central acks it; the engine's `DefaultMaxRetries` cap still applies (matching the cached-call categories) and a notification whose retries are exhausted under a sustained central outage parks like any other buffered message. The cap is sized so the normal central-recovery window stays well inside it — central, once reachable, owns delivery, retry, and parking from the ack point on. The **notification** category retries differently: it has no source-entity setting. The site→central forward uses a single fixed retry interval configured in the host `appsettings.json`. This interval is infrastructure config for reaching the central cluster, not a per-notification-list setting. It applies uniformly to every buffered notification regardless of its target list. A buffered notification is retried at that interval until central acks it; the engine's `DefaultMaxRetries` cap still applies (matching the cached-call categories) and a notification whose retries are exhausted under a sustained central outage parks like any other buffered message. The cap is sized so the normal central-recovery window stays well inside it — central, once reachable, owns delivery, retry, and parking from the ack point on.
The retry interval is **fixed** (not exponential backoff). Fixed interval is sufficient for the expected use cases. The retry interval is **fixed** (not exponential backoff). Fixed interval is sufficient for the expected use cases.
@@ -50,6 +50,9 @@ public static class ExternalSystemCommands
var authTypeOption = new Option<string>("--auth-type") { Description = "Auth type", Required = true }; var authTypeOption = new Option<string>("--auth-type") { Description = "Auth type", Required = true };
var authConfigOption = new Option<string?>("--auth-config") { Description = "Auth configuration JSON" }; var authConfigOption = new Option<string?>("--auth-config") { Description = "Auth configuration JSON" };
var timeoutOption = new Option<int?>("--timeout-seconds") { Description = "Per-system HTTP timeout in seconds (0 = use gateway default); omit to preserve" }; var timeoutOption = new Option<int?>("--timeout-seconds") { Description = "Per-system HTTP timeout in seconds (0 = use gateway default); omit to preserve" };
var maxRetriesOption = new Option<int?>("--max-retries") { Description = "Store-and-forward retry budget for cached calls (0 = use engine default); omit to preserve" };
var retryDelayOption = new Option<int?>("--retry-delay-seconds") { Description = "Fixed interval between store-and-forward retries, in seconds (0 = use engine default); omit to preserve" };
retryDelayOption.Aliases.Add("--retry-delay");
var cmd = new Command("update") { Description = "Update an external system" }; var cmd = new Command("update") { Description = "Update an external system" };
cmd.Add(idOption); cmd.Add(idOption);
@@ -58,6 +61,8 @@ public static class ExternalSystemCommands
cmd.Add(authTypeOption); cmd.Add(authTypeOption);
cmd.Add(authConfigOption); cmd.Add(authConfigOption);
cmd.Add(timeoutOption); cmd.Add(timeoutOption);
cmd.Add(maxRetriesOption);
cmd.Add(retryDelayOption);
cmd.SetAction(async (ParseResult result) => cmd.SetAction(async (ParseResult result) =>
{ {
var id = result.GetValue(idOption); var id = result.GetValue(idOption);
@@ -66,9 +71,13 @@ public static class ExternalSystemCommands
var authType = result.GetValue(authTypeOption)!; var authType = result.GetValue(authTypeOption)!;
var authConfig = result.GetValue(authConfigOption); var authConfig = result.GetValue(authConfigOption);
var timeoutSeconds = result.GetValue(timeoutOption); var timeoutSeconds = result.GetValue(timeoutOption);
var maxRetries = result.GetValue(maxRetriesOption);
var retryDelaySeconds = result.GetValue(retryDelayOption);
return await CommandHelpers.ExecuteCommandAsync( return await CommandHelpers.ExecuteCommandAsync(
result, urlOption, formatOption, usernameOption, passwordOption, result, urlOption, formatOption, usernameOption, passwordOption,
new UpdateExternalSystemCommand(id, name, endpointUrl, authType, authConfig, timeoutSeconds)); new UpdateExternalSystemCommand(id, name, endpointUrl, authType, authConfig, timeoutSeconds,
maxRetries,
retryDelaySeconds is { } rd ? TimeSpan.FromSeconds(rd) : null));
}); });
return cmd; return cmd;
} }
@@ -91,6 +100,9 @@ public static class ExternalSystemCommands
var authTypeOption = new Option<string>("--auth-type") { Description = "Auth type (ApiKey, BasicAuth)", Required = true }; var authTypeOption = new Option<string>("--auth-type") { Description = "Auth type (ApiKey, BasicAuth)", Required = true };
var authConfigOption = new Option<string?>("--auth-config") { Description = "Auth configuration JSON" }; var authConfigOption = new Option<string?>("--auth-config") { Description = "Auth configuration JSON" };
var timeoutOption = new Option<int?>("--timeout-seconds") { Description = "Per-system HTTP timeout in seconds (0 = use gateway default)" }; var timeoutOption = new Option<int?>("--timeout-seconds") { Description = "Per-system HTTP timeout in seconds (0 = use gateway default)" };
var maxRetriesOption = new Option<int?>("--max-retries") { Description = "Store-and-forward retry budget for cached calls (0 = use engine default)" };
var retryDelayOption = new Option<int?>("--retry-delay-seconds") { Description = "Fixed interval between store-and-forward retries, in seconds (0 = use engine default)" };
retryDelayOption.Aliases.Add("--retry-delay");
var cmd = new Command("create") { Description = "Create an external system" }; var cmd = new Command("create") { Description = "Create an external system" };
cmd.Add(nameOption); cmd.Add(nameOption);
@@ -98,6 +110,8 @@ public static class ExternalSystemCommands
cmd.Add(authTypeOption); cmd.Add(authTypeOption);
cmd.Add(authConfigOption); cmd.Add(authConfigOption);
cmd.Add(timeoutOption); cmd.Add(timeoutOption);
cmd.Add(maxRetriesOption);
cmd.Add(retryDelayOption);
cmd.SetAction(async (ParseResult result) => cmd.SetAction(async (ParseResult result) =>
{ {
var name = result.GetValue(nameOption)!; var name = result.GetValue(nameOption)!;
@@ -105,9 +119,13 @@ public static class ExternalSystemCommands
var authType = result.GetValue(authTypeOption)!; var authType = result.GetValue(authTypeOption)!;
var authConfig = result.GetValue(authConfigOption); var authConfig = result.GetValue(authConfigOption);
var timeoutSeconds = result.GetValue(timeoutOption); var timeoutSeconds = result.GetValue(timeoutOption);
var maxRetries = result.GetValue(maxRetriesOption);
var retryDelaySeconds = result.GetValue(retryDelayOption);
return await CommandHelpers.ExecuteCommandAsync( return await CommandHelpers.ExecuteCommandAsync(
result, urlOption, formatOption, usernameOption, passwordOption, result, urlOption, formatOption, usernameOption, passwordOption,
new CreateExternalSystemCommand(name, endpointUrl, authType, authConfig, timeoutSeconds)); new CreateExternalSystemCommand(name, endpointUrl, authType, authConfig, timeoutSeconds,
maxRetries,
retryDelaySeconds is { } rd ? TimeSpan.FromSeconds(rd) : null));
}); });
return cmd; return cmd;
} }
+17 -3
View File
@@ -1206,7 +1206,7 @@ scadabridge --url <url> external-system list
Register an external HTTP system that scripts can call. Register an external HTTP system that scripts can call.
```sh ```sh
scadabridge --url <url> external-system create --name <string> --endpoint-url <url> --auth-type <string> [--auth-config <json>] scadabridge --url <url> external-system create --name <string> --endpoint-url <url> --auth-type <string> [--auth-config <json>] [--timeout-seconds <int>] [--max-retries <int>] [--retry-delay-seconds <int>]
``` ```
| Option | Required | Description | | Option | Required | Description |
@@ -1215,6 +1215,13 @@ scadabridge --url <url> external-system create --name <string> --endpoint-url <u
| `--endpoint-url` | yes | Base URL of the external system | | `--endpoint-url` | yes | Base URL of the external system |
| `--auth-type` | yes | Authentication type: `ApiKey` or `BasicAuth` | | `--auth-type` | yes | Authentication type: `ApiKey` or `BasicAuth` |
| `--auth-config` | no | Auth credentials as a JSON string | | `--auth-config` | no | Auth credentials as a JSON string |
| `--timeout-seconds` | no | Per-system HTTP round-trip timeout. `0` (default) = use the gateway's `DefaultHttpTimeout` |
| `--max-retries` | no | Store-and-forward retry budget for this system's `CachedCall`s. `0` (default) = use the engine's `DefaultMaxRetries` (50) |
| `--retry-delay-seconds` | no | Fixed interval between store-and-forward retries. `0` (default) = use the engine's `DefaultRetryInterval` (30 s). Alias: `--retry-delay` |
The retry options are deployed to sites with the rest of the definition, so they take
effect on the next deployment — they govern how long a transient outage of this system is
buffered before a cached call parks (`--max-retries` x `--retry-delay-seconds`).
#### `external-system update` #### `external-system update`
@@ -1222,7 +1229,7 @@ Update an external system definition. An update **replaces** the whole entity
required field below must be supplied, even if unchanged. required field below must be supplied, even if unchanged.
```sh ```sh
scadabridge --url <url> external-system update --id <int> --name <string> --endpoint-url <url> --auth-type <string> [--auth-config <json>] scadabridge --url <url> external-system update --id <int> --name <string> --endpoint-url <url> --auth-type <string> [--auth-config <json>] [--timeout-seconds <int>] [--max-retries <int>] [--retry-delay-seconds <int>]
``` ```
| Option | Required | Description | | Option | Required | Description |
@@ -1231,7 +1238,14 @@ scadabridge --url <url> external-system update --id <int> --name <string> --endp
| `--name` | yes | Display name | | `--name` | yes | Display name |
| `--endpoint-url` | yes | Base URL | | `--endpoint-url` | yes | Base URL |
| `--auth-type` | yes | Authentication type | | `--auth-type` | yes | Authentication type |
| `--auth-config` | no | Updated auth credentials as a JSON string | | `--auth-config` | no | Updated auth credentials as a JSON string; omit to preserve the stored value |
| `--timeout-seconds` | no | Per-system HTTP round-trip timeout; omit to preserve |
| `--max-retries` | no | Store-and-forward retry budget for this system's `CachedCall`s; omit to preserve |
| `--retry-delay-seconds` | no | Fixed interval between store-and-forward retries; omit to preserve. Alias: `--retry-delay` |
Note the asymmetry with the required fields above: the four optional settings follow an
**omit-to-preserve** rule, so a rename does not silently reset a system's auth config,
timeout, or retry policy.
#### `external-system delete` #### `external-system delete`
@@ -1,9 +1,41 @@
namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts; namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
/// <summary>
/// A single external HTTP system, flattened for deployment to a site.
/// </summary>
/// <param name="Name">Unique system name — the key scripts call it by.</param>
/// <param name="EndpointUrl">Base URL of the system's HTTP endpoint.</param>
/// <param name="AuthType">Authentication type identifier (e.g. <c>ApiKey</c>, <c>BasicAuth</c>).</param>
/// <param name="AuthConfiguration">JSON-serialized authentication configuration for <paramref name="AuthType"/>.</param>
/// <param name="MethodDefinitionsJson">JSON-serialized method list, or <c>null</c> when the system defines none.</param>
/// <param name="TimeoutSeconds">
/// Per-system HTTP round-trip timeout in seconds. <c>0</c> = unset — the gateway's configured
/// <c>DefaultHttpTimeout</c> applies.
/// </param>
/// <param name="MaxRetries">
/// Store-and-Forward retry budget for this system's cached calls. <c>0</c> = unset — the engine's
/// <c>DefaultMaxRetries</c> applies.
/// <para>
/// <b>0 means "unset", not "no limit", at THIS layer</b> — the inverse of
/// <c>StoreAndForwardMessage.MaxRetries</c>, where a stored 0 means unbounded retry. The two
/// meanings are reconciled in <c>ExternalSystemClient.CachedCallAsync</c>, which passes a 0
/// through as <c>null</c> so the bounded engine default applies rather than an unbounded retry
/// loop. Keeping "unset" as 0 here (rather than a nullable) matches
/// <see cref="DatabaseConnectionArtifact"/>, the non-nullable
/// <c>ExternalSystemDefinition.MaxRetries</c> entity property this is projected from, and the
/// proto3 scalar it is carried on.
/// </para>
/// </param>
/// <param name="RetryDelay">
/// Fixed interval between Store-and-Forward retry sweeps for this system's cached calls.
/// <see cref="TimeSpan.Zero"/> = unset — the engine's <c>DefaultRetryInterval</c> applies.
/// </param>
public record ExternalSystemArtifact( public record ExternalSystemArtifact(
string Name, string Name,
string EndpointUrl, string EndpointUrl,
string AuthType, string AuthType,
string? AuthConfiguration, string? AuthConfiguration,
string? MethodDefinitionsJson, string? MethodDefinitionsJson,
int TimeoutSeconds = 0); int TimeoutSeconds = 0,
int MaxRetries = 0,
TimeSpan RetryDelay = default);
@@ -2,8 +2,23 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
public record ListExternalSystemsCommand; public record ListExternalSystemsCommand;
public record GetExternalSystemCommand(int ExternalSystemId); public record GetExternalSystemCommand(int ExternalSystemId);
public record CreateExternalSystemCommand(string Name, string EndpointUrl, string AuthType, string? AuthConfiguration, int? TimeoutSeconds = null); /// <summary>
public record UpdateExternalSystemCommand(int ExternalSystemId, string Name, string EndpointUrl, string AuthType, string? AuthConfiguration, int? TimeoutSeconds = null); /// Creates an external HTTP system.
/// </summary>
/// <remarks>
/// <paramref name="MaxRetries"/> and <paramref name="RetryDelay"/> are the Store-and-Forward
/// retry policy applied to this system's <c>ExternalSystem.CachedCall</c> buffering. Both are
/// nullable on the wire so "omit" is distinguishable from "set to 0"; a null (or an explicit 0)
/// stores the entity's unset value, which the site gateway translates into the engine's
/// <c>DefaultMaxRetries</c>/<c>DefaultRetryInterval</c>.
/// </remarks>
public record CreateExternalSystemCommand(string Name, string EndpointUrl, string AuthType, string? AuthConfiguration, int? TimeoutSeconds = null, int? MaxRetries = null, TimeSpan? RetryDelay = null);
/// <summary>
/// Updates an external HTTP system. Nullable members follow the "omit to preserve" convention —
/// a null leaves the stored value untouched.
/// </summary>
public record UpdateExternalSystemCommand(int ExternalSystemId, string Name, string EndpointUrl, string AuthType, string? AuthConfiguration, int? TimeoutSeconds = null, int? MaxRetries = null, TimeSpan? RetryDelay = null);
public record DeleteExternalSystemCommand(int ExternalSystemId); public record DeleteExternalSystemCommand(int ExternalSystemId);
// External System Methods // External System Methods
@@ -506,7 +506,9 @@ public static class SiteCommandDtoMapper
AuthType = artifact.AuthType, AuthType = artifact.AuthType,
AuthConfiguration = OrEmpty(artifact.AuthConfiguration), AuthConfiguration = OrEmpty(artifact.AuthConfiguration),
MethodDefinitionsJson = OrEmpty(artifact.MethodDefinitionsJson), MethodDefinitionsJson = OrEmpty(artifact.MethodDefinitionsJson),
TimeoutSeconds = artifact.TimeoutSeconds TimeoutSeconds = artifact.TimeoutSeconds,
MaxRetries = artifact.MaxRetries,
RetryDelay = Duration.FromTimeSpan(artifact.RetryDelay)
}; };
} }
@@ -523,7 +525,9 @@ public static class SiteCommandDtoMapper
dto.AuthType, dto.AuthType,
NullIfEmpty(dto.AuthConfiguration), NullIfEmpty(dto.AuthConfiguration),
NullIfEmpty(dto.MethodDefinitionsJson), NullIfEmpty(dto.MethodDefinitionsJson),
dto.TimeoutSeconds); dto.TimeoutSeconds,
dto.MaxRetries,
dto.RetryDelay?.ToTimeSpan() ?? TimeSpan.Zero);
} }
/// <summary>Projects a <see cref="DatabaseConnectionArtifact"/> onto the wire.</summary> /// <summary>Projects a <see cref="DatabaseConnectionArtifact"/> onto the wire.</summary>
@@ -224,6 +224,11 @@ message ExternalSystemArtifactDto {
string auth_configuration = 4; // empty string represents null string auth_configuration = 4; // empty string represents null
string method_definitions_json = 5; // empty string represents null string method_definitions_json = 5; // empty string represents null
int32 timeout_seconds = 6; int32 timeout_seconds = 6;
// Store-and-Forward retry policy for this system's cached calls. Additive (fields 7-8);
// a pre-existing site decodes them as 0/absent, which is exactly the "unset" value that
// makes the engine defaults apply — i.e. the old behaviour.
int32 max_retries = 7;
google.protobuf.Duration retry_delay = 8;
} }
message DatabaseConnectionArtifactDto { message DatabaseConnectionArtifactDto {
File diff suppressed because it is too large Load Diff
@@ -192,8 +192,13 @@ public class ArtifactDeploymentService
m.ReturnDefinition m.ReturnDefinition
})) }))
: null; : null;
// MaxRetries/RetryDelay ride along so the site's Store-and-Forward buffering honours
// the centrally-configured per-system retry policy. Before they were carried, a site
// silently fell back to the engine defaults (50 x 30 s) for every cached call, whatever
// central had configured — the DatabaseConnectionArtifact below always carried them.
externalSystemArtifacts.Add(new ExternalSystemArtifact( externalSystemArtifacts.Add(new ExternalSystemArtifact(
es.Name, es.EndpointUrl, es.AuthType, es.AuthConfiguration, methodsJson, es.TimeoutSeconds)); es.Name, es.EndpointUrl, es.AuthType, es.AuthConfiguration, methodsJson, es.TimeoutSeconds,
es.MaxRetries, es.RetryDelay));
} }
// Map database connections // Map database connections
@@ -113,7 +113,7 @@ public static class SiteLocalDbLegacyMigrator
new("external_systems", "name", new("external_systems", "name",
[ [
"name", "endpoint_url", "auth_type", "auth_configuration", "method_definitions", "name", "endpoint_url", "auth_type", "auth_configuration", "method_definitions",
"updated_at", "timeout_seconds", "updated_at", "timeout_seconds", "max_retries", "retry_delay_ms",
]), ]),
new("database_connections", "name", new("database_connections", "name",
[ [
@@ -2021,7 +2021,11 @@ public class ManagementActor : ReceiveActor
var def = new ExternalSystemDefinition(cmd.Name, cmd.EndpointUrl, cmd.AuthType) var def = new ExternalSystemDefinition(cmd.Name, cmd.EndpointUrl, cmd.AuthType)
{ {
AuthConfiguration = cmd.AuthConfiguration, AuthConfiguration = cmd.AuthConfiguration,
TimeoutSeconds = cmd.TimeoutSeconds ?? 0 TimeoutSeconds = cmd.TimeoutSeconds ?? 0,
// 0 / TimeSpan.Zero is the entity's "unset" value — the site gateway maps it to the
// Store-and-Forward engine defaults rather than to a literal zero budget.
MaxRetries = cmd.MaxRetries ?? 0,
RetryDelay = cmd.RetryDelay ?? TimeSpan.Zero
}; };
await repo.AddExternalSystemAsync(def); await repo.AddExternalSystemAsync(def);
await repo.SaveChangesAsync(); await repo.SaveChangesAsync();
@@ -2042,6 +2046,9 @@ public class ManagementActor : ReceiveActor
// secret intact (mirrors the SMTP/SMS credential preserve-if-null rule). // secret intact (mirrors the SMTP/SMS credential preserve-if-null rule).
if (cmd.AuthConfiguration is not null) def.AuthConfiguration = cmd.AuthConfiguration; if (cmd.AuthConfiguration is not null) def.AuthConfiguration = cmd.AuthConfiguration;
if (cmd.TimeoutSeconds is { } ts) def.TimeoutSeconds = ts; if (cmd.TimeoutSeconds is { } ts) def.TimeoutSeconds = ts;
// Same preserve-if-null rule for the Store-and-Forward retry policy.
if (cmd.MaxRetries is { } mr) def.MaxRetries = mr;
if (cmd.RetryDelay is { } rd) def.RetryDelay = rd;
await repo.UpdateExternalSystemAsync(def); await repo.UpdateExternalSystemAsync(def);
await repo.SaveChangesAsync(); await repo.SaveChangesAsync();
var publicShape = ExternalSystemPublicShape(def); var publicShape = ExternalSystemPublicShape(def);
@@ -2155,7 +2155,8 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
foreach (var es in command.ExternalSystems) foreach (var es in command.ExternalSystems)
{ {
await _storage.StoreExternalSystemAsync(es.Name, es.EndpointUrl, await _storage.StoreExternalSystemAsync(es.Name, es.EndpointUrl,
es.AuthType, es.AuthConfiguration, es.MethodDefinitionsJson, es.TimeoutSeconds); es.AuthType, es.AuthConfiguration, es.MethodDefinitionsJson, es.TimeoutSeconds,
es.MaxRetries, es.RetryDelay);
} }
await _storage.DeleteExternalSystemsExceptAsync( await _storage.DeleteExternalSystemsExceptAsync(
@@ -133,6 +133,28 @@ public static class SiteStorageSchema
// Per-external-system call timeout (ExternalSystemGateway Timeout) — carried through the // Per-external-system call timeout (ExternalSystemGateway Timeout) — carried through the
// artifact pipeline so site-side calls honor it; 0 = use the site default. // artifact pipeline so site-side calls honor it; 0 = use the site default.
AddColumnIfMissing(connection, "external_systems", "timeout_seconds", "INTEGER NOT NULL DEFAULT 0"); AddColumnIfMissing(connection, "external_systems", "timeout_seconds", "INTEGER NOT NULL DEFAULT 0");
// Store-and-Forward retry policy, deployed per external system (deferred-work row 27).
//
// 0 = unset on BOTH columns, which is what makes this additive change behaviourally
// invisible until an operator actually configures a policy: an existing row backfilled
// with the DEFAULT reads back as 0, and ExternalSystemClient.CachedCallAsync maps a 0
// to null so the engine's DefaultMaxRetries/DefaultRetryInterval still apply. (Contrast
// database_connections, whose equivalent columns default to 3/1000 — that table has
// never had an "unset means engine default" contract.)
//
// These land on a table in SiteLocalDbSetup.ReplicatedTables. That is safe and needs no
// trigger migration: RegisterReplicated re-reads PRAGMA table_info on EVERY boot and its
// install script is DROP-then-CREATE, so the capture triggers are rebuilt over the new
// column set automatically — and Apply() runs before RegisterReplicated, so the rebuild
// sees these columns on the same boot that adds them. What DOES change is the table's
// schema digest (ReplicatedTable.ComputeDigest hashes every column name + type), which
// the sync handshake compares fail-closed; a pair running mismatched builds refuses to
// sync loudly (LocalDbSchemaMismatchException) rather than corrupting anything. Both
// nodes must therefore be upgraded together — already the standing rule for a site pair
// (docs/deployment/topology-guide.md), so this introduces no new operational constraint.
AddColumnIfMissing(connection, "external_systems", "max_retries", "INTEGER NOT NULL DEFAULT 0");
AddColumnIfMissing(connection, "external_systems", "retry_delay_ms", "INTEGER NOT NULL DEFAULT 0");
} }
/// <summary> /// <summary>
@@ -647,23 +647,33 @@ public class SiteStorageService
/// <param name="authConfig">Authentication configuration JSON, if applicable.</param> /// <param name="authConfig">Authentication configuration JSON, if applicable.</param>
/// <param name="methodDefs">JSON representation of available method definitions, if any.</param> /// <param name="methodDefs">JSON representation of available method definitions, if any.</param>
/// <param name="timeoutSeconds">Per-system call timeout in seconds (0 = use the site default).</param> /// <param name="timeoutSeconds">Per-system call timeout in seconds (0 = use the site default).</param>
/// <param name="maxRetries">
/// Store-and-Forward retry budget for this system's cached calls (0 = unset — the engine's
/// <c>DefaultMaxRetries</c> applies).
/// </param>
/// <param name="retryDelay">
/// Fixed interval between Store-and-Forward retry sweeps for this system's cached calls
/// (<see cref="TimeSpan.Zero"/> = unset — the engine's <c>DefaultRetryInterval</c> applies).
/// </param>
/// <returns>A task that completes when the external system definition has been stored or updated.</returns> /// <returns>A task that completes when the external system definition has been stored or updated.</returns>
public async Task StoreExternalSystemAsync( public async Task StoreExternalSystemAsync(
string name, string endpointUrl, string authType, string? authConfig, string? methodDefs, string name, string endpointUrl, string authType, string? authConfig, string? methodDefs,
int timeoutSeconds = 0) int timeoutSeconds = 0, int maxRetries = 0, TimeSpan retryDelay = default)
{ {
await using var connection = OpenConnection(); await using var connection = OpenConnection();
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = @" command.CommandText = @"
INSERT INTO external_systems (name, endpoint_url, auth_type, auth_configuration, method_definitions, timeout_seconds, updated_at) INSERT INTO external_systems (name, endpoint_url, auth_type, auth_configuration, method_definitions, timeout_seconds, max_retries, retry_delay_ms, updated_at)
VALUES (@name, @url, @authType, @authConfig, @methodDefs, @timeoutSeconds, @updatedAt) VALUES (@name, @url, @authType, @authConfig, @methodDefs, @timeoutSeconds, @maxRetries, @retryDelayMs, @updatedAt)
ON CONFLICT(name) DO UPDATE SET ON CONFLICT(name) DO UPDATE SET
endpoint_url = excluded.endpoint_url, endpoint_url = excluded.endpoint_url,
auth_type = excluded.auth_type, auth_type = excluded.auth_type,
auth_configuration = excluded.auth_configuration, auth_configuration = excluded.auth_configuration,
method_definitions = excluded.method_definitions, method_definitions = excluded.method_definitions,
timeout_seconds = excluded.timeout_seconds, timeout_seconds = excluded.timeout_seconds,
max_retries = excluded.max_retries,
retry_delay_ms = excluded.retry_delay_ms,
updated_at = excluded.updated_at"; updated_at = excluded.updated_at";
command.Parameters.AddWithValue("@name", name); command.Parameters.AddWithValue("@name", name);
@@ -672,6 +682,8 @@ public class SiteStorageService
command.Parameters.AddWithValue("@authConfig", (object?)authConfig ?? DBNull.Value); command.Parameters.AddWithValue("@authConfig", (object?)authConfig ?? DBNull.Value);
command.Parameters.AddWithValue("@methodDefs", (object?)methodDefs ?? DBNull.Value); command.Parameters.AddWithValue("@methodDefs", (object?)methodDefs ?? DBNull.Value);
command.Parameters.AddWithValue("@timeoutSeconds", timeoutSeconds); command.Parameters.AddWithValue("@timeoutSeconds", timeoutSeconds);
command.Parameters.AddWithValue("@maxRetries", maxRetries);
command.Parameters.AddWithValue("@retryDelayMs", (long)retryDelay.TotalMilliseconds);
command.Parameters.AddWithValue("@updatedAt", DateTimeOffset.UtcNow.ToString("O")); command.Parameters.AddWithValue("@updatedAt", DateTimeOffset.UtcNow.ToString("O"));
await command.ExecuteNonQueryAsync(); await command.ExecuteNonQueryAsync();
@@ -250,7 +250,13 @@ public class SiteExternalSystemRepository : IExternalSystemRepository
{ {
Id = GenerateSyntheticId(name), Id = GenerateSyntheticId(name),
AuthConfiguration = reader.IsDBNull(3) ? null : reader.GetString(3), AuthConfiguration = reader.IsDBNull(3) ? null : reader.GetString(3),
TimeoutSeconds = reader.IsDBNull(4) ? 0 : reader.GetInt32(4) TimeoutSeconds = reader.IsDBNull(4) ? 0 : reader.GetInt32(4),
// Deployed Store-and-Forward retry policy (deferred-work row 27). Ordinals 6/7 are
// deliberately AFTER method_definitions (5) so the pre-existing indexes are untouched.
// 0 stays 0: it is the "unset" value ExternalSystemClient.CachedCallAsync converts to
// null so the engine defaults apply.
MaxRetries = reader.IsDBNull(6) ? 0 : reader.GetInt32(6),
RetryDelay = reader.IsDBNull(7) ? TimeSpan.Zero : TimeSpan.FromMilliseconds(reader.GetInt64(7))
}; };
} }
@@ -271,7 +277,8 @@ public class SiteExternalSystemRepository : IExternalSystemRepository
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = @" command.CommandText = @"
SELECT name, endpoint_url, auth_type, auth_configuration, timeout_seconds, method_definitions SELECT name, endpoint_url, auth_type, auth_configuration, timeout_seconds, method_definitions,
max_retries, retry_delay_ms
FROM external_systems"; FROM external_systems";
var byName = new Dictionary<string, ExternalSystemDefinition>(StringComparer.Ordinal); var byName = new Dictionary<string, ExternalSystemDefinition>(StringComparer.Ordinal);
@@ -165,6 +165,11 @@ public sealed class ArtifactDiff
var changes = new List<FieldChange>(); var changes = new List<FieldChange>();
AddIfDifferent(changes, "BaseUrl", existing.EndpointUrl, incoming.BaseUrl); AddIfDifferent(changes, "BaseUrl", existing.EndpointUrl, incoming.BaseUrl);
AddIfDifferent(changes, "AuthType", existing.AuthType, incoming.AuthType); AddIfDifferent(changes, "AuthType", existing.AuthType, incoming.AuthType);
// The bundle has always CARRIED the retry policy and the importer has always APPLIED it,
// but the diff omitted it — so an import that silently changed a system's retry budget
// previewed as "Identical". Mirrors CompareDatabaseConnection below.
AddIfDifferent(changes, "MaxRetries", existing.MaxRetries, incoming.MaxRetries);
AddIfDifferent(changes, "RetryDelay", existing.RetryDelay.ToString(), incoming.RetryDelay.ToString());
// Secrets: presence-only comparison (we never echo the value in the diff). // Secrets: presence-only comparison (we never echo the value in the diff).
var existingHasSecret = !string.IsNullOrEmpty(existing.AuthConfiguration); var existingHasSecret = !string.IsNullOrEmpty(existing.AuthConfiguration);
@@ -0,0 +1,102 @@
using System.CommandLine;
using ZB.MOM.WW.ScadaBridge.CLI.Commands;
namespace ZB.MOM.WW.ScadaBridge.CLI.Tests.Commands;
/// <summary>
/// Deferred-work row 27(b): <c>external-system create</c> and <c>external-system update</c>
/// must expose the per-system Store-and-Forward retry policy. Before this the fields existed
/// on the central entity but had NO management surface at all — the only way to set them was
/// a direct DB edit of <c>ExternalSystemDefinitions</c>.
/// <para>
/// The delay flag is named <c>--retry-delay-seconds</c> to match the sibling
/// <c>--timeout-seconds</c> on these same commands (the repo-wide CLI convention is an explicit
/// unit suffix), with <c>--retry-delay</c> kept as an alias.
/// </para>
/// </summary>
public class ExternalSystemRetryOptionTests
{
private static readonly Option<string> Url = new("--url") { Recursive = true };
private static readonly Option<string> Format = new("--format") { Recursive = true };
private static readonly Option<string> Username = new("--username") { Recursive = true };
private static readonly Option<string> Password = new("--password") { Recursive = true };
private static Command Group() => ExternalSystemCommands.Build(Url, Format, Username, Password);
private static Option? FindOption(Command command, string name)
=> command.Options.SingleOrDefault(o => o.Name == name);
[Theory]
[InlineData("create")]
[InlineData("update")]
public void HasOptionalRetryPolicyOptions(string subcommand)
{
var cmd = Group().Subcommands.Single(c => c.Name == subcommand);
var maxRetries = FindOption(cmd, "--max-retries");
Assert.NotNull(maxRetries);
Assert.False(maxRetries!.Required);
var retryDelay = FindOption(cmd, "--retry-delay-seconds");
Assert.NotNull(retryDelay);
Assert.False(retryDelay!.Required);
}
[Fact]
public void Create_ParsesRetryPolicyValues()
{
var parse = Group().Parse(new[]
{
"create", "--name", "Sys", "--endpoint-url", "http://x", "--auth-type", "none",
"--max-retries", "8", "--retry-delay-seconds", "60",
});
Assert.Empty(parse.Errors);
Assert.Equal(8, parse.GetValue<int?>("--max-retries"));
Assert.Equal(60, parse.GetValue<int?>("--retry-delay-seconds"));
}
[Fact]
public void Update_ParsesRetryPolicyValues()
{
var parse = Group().Parse(new[]
{
"update", "--id", "7", "--name", "Sys", "--endpoint-url", "http://x",
"--auth-type", "none", "--max-retries", "3", "--retry-delay-seconds", "10",
});
Assert.Empty(parse.Errors);
Assert.Equal(3, parse.GetValue<int?>("--max-retries"));
Assert.Equal(10, parse.GetValue<int?>("--retry-delay-seconds"));
}
[Fact]
public void RetryDelayAliasIsAccepted()
{
var parse = Group().Parse(new[]
{
"create", "--name", "Sys", "--endpoint-url", "http://x", "--auth-type", "none",
"--retry-delay", "45",
});
Assert.Empty(parse.Errors);
Assert.Equal(45, parse.GetValue<int?>("--retry-delay-seconds"));
}
/// <summary>
/// Omission must stay distinguishable from an explicit 0 — that is what makes update's
/// "omit to preserve" rule and create's "fall back to the engine default" rule work.
/// </summary>
[Fact]
public void OmittedRetryPolicy_ParsesAsNull()
{
var parse = Group().Parse(new[]
{
"create", "--name", "Sys", "--endpoint-url", "http://x", "--auth-type", "none",
});
Assert.Empty(parse.Errors);
Assert.Null(parse.GetValue<int?>("--max-retries"));
Assert.Null(parse.GetValue<int?>("--retry-delay-seconds"));
}
}
@@ -217,8 +217,12 @@ internal static class SiteCommandSamples
private static SharedScriptArtifact SharedScript() => private static SharedScriptArtifact SharedScript() =>
new("Calc", "return 1;", "{\"p\":\"int\"}", "{\"r\":\"int\"}"); new("Calc", "return 1;", "{\"p\":\"int\"}", "{\"r\":\"int\"}");
// MaxRetries/RetryDelay are non-default on purpose: they are the deferred-work-row-27
// additions (proto fields 7-8), and the generic round-trip assertion is what pins them
// to the wire. A sample left at the defaults would round-trip vacuously.
private static ExternalSystemArtifact ExternalSystem() => private static ExternalSystemArtifact ExternalSystem() =>
new("Mes", "https://mes/api", "ApiKey", "{\"key\":\"x\"}", "[{\"name\":\"Post\"}]", 45); new("Mes", "https://mes/api", "ApiKey", "{\"key\":\"x\"}", "[{\"name\":\"Post\"}]", 45,
MaxRetries: 12, RetryDelay: TimeSpan.FromSeconds(20));
private static DataConnectionArtifact DataConnection() => private static DataConnectionArtifact DataConnection() =>
new("Plc1", "OpcUa", "{\"url\":\"opc.tcp://a\"}", "{\"url\":\"opc.tcp://b\"}", 5); new("Plc1", "OpcUa", "{\"url\":\"opc.tcp://a\"}", "{\"url\":\"opc.tcp://b\"}", 5);
@@ -11,6 +11,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.Communication; using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems;
namespace ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests; namespace ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests;
@@ -189,6 +190,38 @@ public class ArtifactDeploymentServiceTests : TestKit
await _siteRepo.Received(1).GetDataConnectionsBySiteIdAsync(3, Arg.Any<CancellationToken>()); await _siteRepo.Received(1).GetDataConnectionsBySiteIdAsync(3, Arg.Any<CancellationToken>());
} }
/// <summary>
/// Deferred-work row 27(a): the per-system retry policy must be projected onto the
/// artifact that ships to sites. Before the fix <c>ExternalSystemArtifact</c> had no such
/// fields, so a centrally-configured policy never left central and every site cached call
/// silently used the engine default (50 x 30 s) — while the sibling
/// <c>DatabaseConnectionArtifact</c> had always carried its equivalents.
/// </summary>
[Fact]
public async Task DeployedExternalSystemArtifact_CarriesRetryPolicy()
{
_externalSystemRepo.GetAllExternalSystemsAsync(Arg.Any<CancellationToken>()).Returns(
[
new ExternalSystemDefinition("Mes", "https://mes", "ApiKey")
{
Id = 1,
MaxRetries = 6,
RetryDelay = TimeSpan.FromSeconds(120),
},
]);
var recorder = new ArtifactProbeRecorder();
var probe = Sys.ActorOf(Props.Create(() => new ArtifactProbeActor(recorder)));
var service = CreateServiceWithCommActor(probe);
var result = await service.RetryForSiteAsync(1, "retry-site", "admin");
Assert.True(result.IsSuccess);
var artifact = Assert.Single(Assert.Single(recorder.Received).ExternalSystems!);
Assert.Equal(6, artifact.MaxRetries);
Assert.Equal(TimeSpan.FromSeconds(120), artifact.RetryDelay);
}
[Fact] [Fact]
public async Task RetryForSiteAsync_SingleSitePath_StillRunsTheGlobalQueriesOnce() public async Task RetryForSiteAsync_SingleSitePath_StillRunsTheGlobalQueriesOnce()
{ {
@@ -3968,6 +3968,61 @@ public class ManagementActorTests : TestKit, IDisposable
Arg.Any<CancellationToken>()); Arg.Any<CancellationToken>());
} }
/// <summary>
/// Deferred-work row 27(b): the retry policy had no management surface at all — the only
/// way to set it was a direct DB edit of <c>ExternalSystemDefinitions</c>. Create must now
/// map both fields onto the entity.
/// </summary>
[Fact]
public void CreateExternalSystem_MapsRetryPolicy()
{
var extRepo = Substitute.For<IExternalSystemRepository>();
_services.AddScoped(_ => extRepo);
var actor = CreateActor();
var envelope = Envelope(
new CreateExternalSystemCommand("Sys", "http://x", "none", null,
TimeoutSeconds: 15, MaxRetries: 8, RetryDelay: TimeSpan.FromSeconds(60)),
"Designer");
actor.Tell(envelope);
var response = ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
Assert.Equal(envelope.CorrelationId, response.CorrelationId);
extRepo.Received(1).AddExternalSystemAsync(
Arg.Is<Commons.Entities.ExternalSystems.ExternalSystemDefinition>(
d => d.MaxRetries == 8 && d.RetryDelay == TimeSpan.FromSeconds(60)),
Arg.Any<CancellationToken>());
}
/// <summary>
/// The retry policy follows the same "omit to preserve" rule as TimeoutSeconds, so a
/// partial update (e.g. renaming a system) cannot silently reset it to 0.
/// </summary>
[Fact]
public void UpdateExternalSystem_NullRetryPolicy_PreservesExisting()
{
var existing = new Commons.Entities.ExternalSystems.ExternalSystemDefinition("Sys", "http://x", "none")
{
MaxRetries = 11,
RetryDelay = TimeSpan.FromSeconds(33)
};
var extRepo = Substitute.For<IExternalSystemRepository>();
extRepo.GetExternalSystemByIdAsync(7, Arg.Any<CancellationToken>()).Returns(existing);
_services.AddScoped(_ => extRepo);
var actor = CreateActor();
var envelope = Envelope(
new UpdateExternalSystemCommand(7, "Renamed", "http://x", "none", null),
"Designer");
actor.Tell(envelope);
ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
Assert.Equal(11, existing.MaxRetries);
Assert.Equal(TimeSpan.FromSeconds(33), existing.RetryDelay);
}
// ======================================================================== // ========================================================================
// Area management role gate (arch-review C6): reconciled to any-of // Area management role gate (arch-review C6): reconciled to any-of
// [Designer, Deployer]. The Central UI creates areas inside the Deployment // [Designer, Deployer]. The Central UI creates areas inside the Deployment
@@ -997,6 +997,35 @@ public class DeploymentManagerActorTests : TestKit, IDisposable
SmtpConfigurations: null, SmtpConfigurations: null,
Timestamp: DateTimeOffset.UtcNow); Timestamp: DateTimeOffset.UtcNow);
/// <summary>
/// Deferred-work row 27: the per-system Store-and-Forward retry policy must survive the
/// site APPLY path, not just the storage round-trip. Before the fix
/// <c>ExternalSystemArtifact</c> carried no retry fields at all, so a centrally-configured
/// policy was dropped on the floor here and every cached call fell back to the engine's
/// 50 x 30 s default.
/// </summary>
[Fact]
public async Task ApplyingArtifacts_PersistsExternalSystemRetryPolicy()
{
var manager = CreateDeploymentManager();
manager.Tell(ArtifactsCommand("dep-es-retry", externalSystems:
[
new ExternalSystemArtifact("MES", "https://mes", "ApiKey", null, null, 0,
MaxRetries: 9, RetryDelay: TimeSpan.FromSeconds(45))
]));
Assert.True(ExpectMsg<ArtifactDeploymentResponse>(TimeSpan.FromSeconds(10)).Success);
await using var connection = _storage.CreateConnection();
await using var command = connection.CreateCommand();
command.CommandText = "SELECT max_retries, retry_delay_ms FROM external_systems WHERE name = 'MES'";
await using var reader = await command.ExecuteReaderAsync();
Assert.True(await reader.ReadAsync());
Assert.Equal(9, reader.GetInt32(0));
Assert.Equal(45_000, reader.GetInt64(1));
}
[Fact] [Fact]
public async Task ApplyingArtifacts_DeletesExternalSystemsAbsentFromTheSet() public async Task ApplyingArtifacts_DeletesExternalSystemsAbsentFromTheSet()
{ {
@@ -86,6 +86,52 @@ public class SiteRepositoryTests : IDisposable
Assert.Equal(7, found!.TimeoutSeconds); Assert.Equal(7, found!.TimeoutSeconds);
} }
/// <summary>
/// Deferred-work row 27: the per-system Store-and-Forward retry policy
/// (<c>MaxRetries</c>/<c>RetryDelay</c>) must round-trip through the site SQLite store,
/// so <c>ExternalSystemClient.CachedCallAsync</c> can hand it to the engine instead of
/// silently falling back to <c>DefaultMaxRetries</c> (50) x <c>DefaultRetryInterval</c> (30 s).
/// Before the fix the site table had no such columns and this read back 0 / TimeSpan.Zero.
/// </summary>
[Fact]
public async Task SiteStorage_ExternalSystem_RetryPolicy_RoundTrips()
{
var storage = NewStorage();
await storage.InitializeAsync();
await storage.StoreExternalSystemAsync(
"MesApi", "https://mes.example.com", "ApiKey", null, null,
timeoutSeconds: 0, maxRetries: 7, retryDelay: TimeSpan.FromSeconds(90));
var repo = new SiteExternalSystemRepository(storage);
var found = await repo.GetExternalSystemByNameAsync("MesApi");
Assert.NotNull(found);
Assert.Equal(7, found!.MaxRetries);
Assert.Equal(TimeSpan.FromSeconds(90), found.RetryDelay);
}
/// <summary>
/// The unset case stays unset: a system deployed without a retry policy must read back as
/// 0 / <see cref="TimeSpan.Zero"/>, which is the value the gateway translates to null so the
/// engine defaults apply. This is what makes the new columns behaviourally invisible until
/// an operator actually configures a policy.
/// </summary>
[Fact]
public async Task SiteStorage_ExternalSystem_RetryPolicy_DefaultsToUnset()
{
var storage = NewStorage();
await storage.InitializeAsync();
await storage.StoreExternalSystemAsync(
"PlainApi", "https://plain.example.com", "ApiKey", null, null);
var repo = new SiteExternalSystemRepository(storage);
var found = await repo.GetExternalSystemByNameAsync("PlainApi");
Assert.NotNull(found);
Assert.Equal(0, found!.MaxRetries);
Assert.Equal(TimeSpan.Zero, found.RetryDelay);
}
/// <summary> /// <summary>
/// SiteRuntime-007: the synthetic ID for an external system must be identical when /// SiteRuntime-007: the synthetic ID for an external system must be identical when
/// the storage service and repository are re-created (simulating a process restart). /// the storage service and repository are re-created (simulating a process restart).
@@ -9,6 +9,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Transport; using ZB.MOM.WW.ScadaBridge.Commons.Types.Transport;
using ZB.MOM.WW.ScadaBridge.Transport.Import; using ZB.MOM.WW.ScadaBridge.Transport.Import;
using ZB.MOM.WW.ScadaBridge.Transport.Serialization; using ZB.MOM.WW.ScadaBridge.Transport.Serialization;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems;
namespace ZB.MOM.WW.ScadaBridge.Transport.Tests.Import; namespace ZB.MOM.WW.ScadaBridge.Transport.Tests.Import;
@@ -105,6 +106,56 @@ public sealed class ArtifactDiffTests
Assert.Null(item.FieldDiffJson); Assert.Null(item.FieldDiffJson);
} }
// ============ Deferred-work row 27: ES retry policy in the diff ============
/// <summary>
/// The bundle has always CARRIED the external-system retry policy and
/// <c>BundleImporter</c> has always APPLIED it, but <c>CompareExternalSystem</c> never
/// compared it — so an import that silently changed a system's retry budget previewed as
/// "Identical". Now that the policy actually reaches sites, an unreviewed change to it is a
/// live behaviour change, so it must show up in the preview.
/// </summary>
[Fact]
public void ExternalSystem_ChangedRetryPolicy_IsReported()
{
var existing = new ExternalSystemDefinition("Mes", "https://mes", "ApiKey")
{
MaxRetries = 3,
RetryDelay = TimeSpan.FromSeconds(5),
};
var incoming = new ExternalSystemDto("Mes", "https://mes", "ApiKey",
MaxRetries: 9, RetryDelay: TimeSpan.FromSeconds(45), Methods: [], Secrets: null);
var item = _diff.CompareExternalSystem(incoming, existing, []);
Assert.Equal(ConflictKind.Modified, item.Kind);
var maxRetries = ChangeFor(item, "MaxRetries");
Assert.Equal("3", maxRetries.GetProperty("oldValue").GetString());
Assert.Equal("9", maxRetries.GetProperty("newValue").GetString());
var retryDelay = ChangeFor(item, "RetryDelay");
Assert.Equal(TimeSpan.FromSeconds(5).ToString(), retryDelay.GetProperty("oldValue").GetString());
Assert.Equal(TimeSpan.FromSeconds(45).ToString(), retryDelay.GetProperty("newValue").GetString());
}
/// <summary>An unchanged retry policy must not manufacture a spurious diff row.</summary>
[Fact]
public void ExternalSystem_UnchangedRetryPolicy_IsIdentical()
{
var existing = new ExternalSystemDefinition("Mes", "https://mes", "ApiKey")
{
MaxRetries = 3,
RetryDelay = TimeSpan.FromSeconds(5),
};
var incoming = new ExternalSystemDto("Mes", "https://mes", "ApiKey",
MaxRetries: 3, RetryDelay: TimeSpan.FromSeconds(5), Methods: [], Secrets: null);
var item = _diff.CompareExternalSystem(incoming, existing, []);
Assert.Equal(ConflictKind.Identical, item.Kind);
}
[Fact] [Fact]
public void ApiMethod_ModifiedScript_EmitsStructuredLineDiff() public void ApiMethod_ModifiedScript_EmitsStructuredLineDiff()
{ {