test+docs(resilience): prove authored ResilienceConfig reaches the runtime pipeline (#456) #461

Merged
dohertj2 merged 2 commits from fix/456-resilience-config-artifact-roundtrip into master 2026-07-15 08:49:53 -04:00
2 changed files with 115 additions and 0 deletions
@@ -0,0 +1,62 @@
# Follow-up #10 / issue #456 — live proof that authored ResilienceConfig reaches the runtime pipeline
**Date:** 2026-07-15 · **Rig:** docker-dev (central-1 + central-2, MAIN cluster, Warm/2-node) built from source.
## What this proves
The #456 "sub-gap" worried that per-instance `DriverInstance.ResilienceConfig` never reaches the
runtime resilience pipeline (so it runs tier-defaults only). This is **disproven** — both by a new
regression test and by this live capture.
## Setup
Seeded on MAIN (SQL): an Equipment namespace `ns-res` + an `OpcUaClient` driver `drv-res` carrying a
**non-default** override:
```json
"ResilienceConfig": {"capabilityPolicies":{"Subscribe":{"retryCount":999}}}
```
`retryCount:999` is deliberately over the cap (100) so the parser emits an unmistakable clamp
diagnostic that echoes the authored value. Deployed headless: `POST /api/deployments` → Accepted
(deploymentId 80b4be3c…).
## Evidence
**1. The persisted deployment artifact bytes carry it** (`dbo.Deployment.ArtifactBlob`, 4055 bytes):
`CHARINDEX('ResilienceConfig') > 0` = YES, `CHARINDEX('999') > 0` = YES; the `drv-res` object slice
shows `…"DriverType":"OpcUaClient"…"DriverConfig":"…"…"Resil…"`.
**2. The runtime parsed + applied it — on BOTH redundancy nodes:**
```
central-1 [WRN] Driver resilience config for instance=drv-res type=OpcUaClient: Subscribe.retryCount 999 exceeds the cap (100); clamped.
central-1 [INF] DriverHost central-1:4053: spawned OpcUaClient driver drv-res (stub=False)
central-2 [WRN] Driver resilience config for instance=drv-res type=OpcUaClient: Subscribe.retryCount 999 exceeds the cap (100); clamped.
central-2 [INF] DriverHost central-2:4053: spawned OpcUaClient driver drv-res (stub=False)
```
The authored value `999` appears verbatim in the runtime log. If the artifact did not carry
`ResilienceConfig` (the feared sub-gap), `spec.ResilienceConfig` would be null, the parser would take
pure tier-defaults, and **no clamp diagnostic would fire**. The diagnostic firing = the authored
policy rode DB → ConfigComposer → artifact → DriverHostActor → DriverCapabilityInvokerFactory.Create →
DriverResilienceOptionsParser. This is arch-review follow-up #10 acceptance item 2 ("re-verify the
behavioral gate with a non-default policy"), verified live on both nodes.
## Regression guard
`ConfigComposerTests.ResilienceConfig_survives_ConfigComposer_to_ParseDriverInstances_round_trip`
(master 809e7886) proves the same composer→artifact→parse leg deterministically, guarding against a
future projection / `[JsonIgnore]` silently reverting authored policy to tier defaults.
## Still open (acceptance item 1)
The raw retry/breaker log line **on an actual driver fault** was NOT captured. It requires a
connect-then-fault-mid-flight trigger: the wrapped capability calls (`Subscribe`/`Discover`/`Write`)
only fire after a driver reaches **Connected** (`DriverInstanceActor` `ResubscribeDesired` runs on the
`Become(Connected)` transition), so a driver at a dead endpoint stays in Reconnecting and never
exercises the wrapped path. The reliable trigger is an **idempotent Write to a Connected-then-killed
driver** (LDAP-gated Client.CLI write) or an S7 dead-socket reconnect. The retry/breaker **behaviour**
is already proven deterministically (`CapabilityInvokerTests` retry, the pipeline-builder logging test,
`DriverCapabilityInvokerFactoryTests` override-applies-to-behaviour). This remaining line is
operator-confidence only.
@@ -138,6 +138,59 @@ public sealed class ConfigComposerTests : ControlPlaneActorTestBase
node.DeviceHost.ShouldBe("10.9.9.9:8193");
}
/// <summary>
/// Verifies that a <see cref="DriverInstance"/>'s per-instance <c>ResilienceConfig</c> override
/// survives the real <see cref="ConfigComposer.SnapshotAndFlattenAsync"/> →
/// <see cref="DeploymentArtifact.ParseDriverInstances(ReadOnlySpan{byte})"/> seam (arch-review
/// follow-up #10 sub-gap). The composer serialises the whole entity, so the override rides the
/// artifact into <c>DriverInstanceSpec.ResilienceConfig</c> — which <c>DriverHostActor</c> layers
/// onto the driver's Polly pipeline. Guards the one leg the hand-rolled
/// <c>DeploymentArtifactTests.ParseDriverInstances_carries_ResilienceConfig_onto_the_spec</c>
/// cannot: if the composer ever stopped emitting the column (a projection, a <c>[JsonIgnore]</c>),
/// authored resilience policy would silently revert to tier defaults in production while the
/// hand-built artifact tests stayed green.
/// </summary>
[Fact]
public async Task ResilienceConfig_survives_ConfigComposer_to_ParseDriverInstances_round_trip()
{
const string resilienceJson =
"{\"capabilityPolicies\":{\"Read\":{\"timeoutSeconds\":5,\"retryCount\":7,\"breakerFailureThreshold\":3}}}";
var f = NewInMemoryDbFactory();
await using (var db = f.CreateDbContext())
{
db.ServerClusters.Add(NewCluster("c1"));
db.Namespaces.Add(new Namespace
{
NamespaceId = "ns-eq", ClusterId = "c1",
Kind = NamespaceKind.Equipment, NamespaceUri = "urn:eq",
});
db.DriverInstances.Add(new DriverInstance
{
DriverInstanceId = "drv-1", ClusterId = "c1", NamespaceId = "ns-eq",
Name = "Modbus", DriverType = "Modbus", DriverConfig = "{}",
ResilienceConfig = resilienceJson,
});
// A second instance with no override proves the null case rides the same real composer path.
db.DriverInstances.Add(new DriverInstance
{
DriverInstanceId = "drv-2", ClusterId = "c1", NamespaceId = "ns-eq",
Name = "Modbus2", DriverType = "Modbus", DriverConfig = "{}",
});
await db.SaveChangesAsync();
}
await using var readDb = f.CreateDbContext();
var artifact = await ConfigComposer.SnapshotAndFlattenAsync(readDb);
var specs = DeploymentArtifact.ParseDriverInstances(artifact.Blob);
var withOverride = specs.Single(s => s.DriverInstanceId == "drv-1");
withOverride.ResilienceConfig.ShouldBe(resilienceJson, "the authored override must ride the artifact byte-for-byte");
var withoutOverride = specs.Single(s => s.DriverInstanceId == "drv-2");
withoutOverride.ResilienceConfig.ShouldBeNull("an unset override stays null ⇒ tier defaults at spawn");
}
private static readonly DateTime FixedTimestamp = new(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc);
private static ServerCluster NewCluster(string id) => new()