fix(flow-debugger): review follow-ups — redact runner-synthesized rows, spawner rule, budget doc, CLI ceiling note (arch-review remediation)

Code review of the merged arch-review remediation (Tasks 1-13) found four items.

1. FlowDebugRunner returned its OWN synthesized rows un-redacted. RunMethodAsync's
   inbound row and RunValueChangeAsync's root row were handed to WriteCentralAsync
   (whose writer redacts a private copy) and then put RAW into the returned Events —
   so a Live run's caller saw the unfiltered row while Replay of the same execution
   showed the redacted one, defeating F1's "Live shows what Replay reads back". Both
   methods now resolve IAuditRedactor once per run and place the filtered copy into
   the result (including the early-return path on a failed attribute write); the raw
   row still reaches the writer, which redacts itself.

2. FlowGraphBuilder's spawner search degenerated to "the last step of the parent".
   Since executions are ordered as contiguous blocks, position[p] < childPosition
   holds for EVERY parent step, so a run shaped ScriptRun -> Write(A) -> Wait ->
   Write(B) whose Write(A) fired the cascade drew the Spawn edge from Write(B). New
   rule, in order: (a) the last non-async, non-Response parent step occurring at or
   before the child's first step — parent and child of a trigger cascade share the
   site clock, so this picks the write that actually fired it; (b) the position rule
   as fallback, which is the mixed-clock case F5 was written for (a child clock
   behind its parent's, where no parent step satisfies (a)); (c) parentChain[0].
   The class remark's stale "Replay order is unchanged" claim is replaced with what
   actually happens to a central Database.* call made after Route.To().Call.

3. FlowTraceCollector's byte budget counts UTF-16 chars of DetailsJson only, so
   "well under the 4 MB gRPC default" overstated it. Documented honestly: ~1 MB of
   UTF-8 for ASCII payloads, ~3 MB worst case for non-ASCII, both under 4 MB before
   framing — and raise it only against a real serialized-size measurement.

4. CLI --timeout-seconds help + README row now say the management endpoint caps
   flow-debug run-method at ManagementService:LongRunningCommandTimeout (default
   5 min), so a larger client value still 504s at the server's ceiling.

Deferred-work register rows 112 (site-side inline trace truncation is invisible to
the operator — needs a proto field) and 113 (the F2 method-entry site scope gates
routed traffic only; ApiMethods have no site affinity to scope against) record what
this review deliberately did not fix.

Tests: FlowGraphBuilderTests gains the same-clock Write(A)/Wait/Write(B) cascade
(red before the rule change, and the clock-behind test still passes on rule (b));
FlowDebugRunnerTests extends the redactor test to the inbound row and adds two
value-change cases (settled poll, failed write). Commons 793, ManagementService 615,
CentralUI 1264, CLI 462 — all green.

Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
This commit is contained in:
Joseph Doherty
2026-09-06 14:44:34 -04:00
parent 6c9ea39f12
commit f17bae093a
8 changed files with 120 additions and 14 deletions
@@ -104,6 +104,8 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
| 109 | **`ZB.MOM.WW.Secrets.Replicator.SqlServer`'s store migrator retries its DDL with a flat `100 * attempt` backoff and no jitter — a latent concurrent-boot flake, open in Secrets** *(Low)*. Porting that migrator's retry loop verbatim for the Auth 0.3.0 SQL Server API-key store (register row 108) reproduced an intermittent exhaustion — about once in sixty 8-way concurrent-boot trials the five flat-backoff retries ran out before a DDL/lock conflict cleared; adding jitter to the Auth port's backoff took it to 0 failures in 180 trials. Secrets' own migrator has no concurrent-migrate test, so the same defect is presumed still open there — it just hasn't been exercised at 8-way concurrency, and a ScadaBridge central pair is only ever 2-way, so this is unlikely to bite in practice. | Found 2026-09-02 while implementing `docs/plans/2026-09-02-inbound-api-key-shared-store.md` (Auth 0.3.0 `SqlServerApiKeyStoreMigrator`) | Add jitter to `SqlServerSecretsStoreMigrator`'s backoff in `~/Desktop/scadaproj/ZB.MOM.WW.Secrets` (mirror the Auth fix) and add a `Migrate_Concurrently_FromEightTasks_AllSucceed`-shaped live test to catch a regression. Lives entirely in scadaproj, not this repo. | Next time Secrets' SQL Server migrator is touched, or if a wider-than-2-way concurrent boot of a Secrets-backed cluster is ever planned. |
| 110 | **Flow Debugger follow-ups: the *Vertical trace* list toggle (mockup Direction B) and console capture on `ScriptRun`** *(Low)*. v1 ships the Direction A swimlane only; the mockup's Direction B — the same steps as a vertical, filterable list with the lane as a column — was kept as an alternate for long flows and screen readers, and the detail panel's *Console* tab is absent because script `Console`/log output is not captured per run (`ScriptRun` rows carry parameters, return value, error and timing, not stdout). Both are additive: a list view over the existing `FlowGraph`, and an extra `Extra.console` field on the `ScriptRun` row's `ScriptTraceExtra` (payload cap applies). | Design `docs/plans/2026-09-05-flow-debugger-design.md` §11; plan `2026-09-05-flow-debugger.md` Task 23 | Neither was asked for by an operator; the swimlane + Timing/Writes tabs cover the acceptance flow (Delmia recipe download, rig step H), and console capture needs a per-run writer seam in `ScriptRuntimeContext` plus a redaction decision. | First operator request for a list view or for script log lines in the step detail. |
| 111 | ~~**Flow Debugger outbound nodes: no Retry/Discard on a parked node, and no `?correlationId=` deep link from the node to the Site Calls / Notification report pages** *(Low)*~~ **RESOLVED 2026-09-06 (plan `docs/plans/2026-09-06-flow-debugger-followups.md`).** `FlowStep` now carries the row's `CorrelationId`; the detail panel deep-links to `/site-calls/report?correlationId=` / `/notifications/report?correlationId=` (both pages accept it through additive per-id query filters — `SiteCallQueryRequest.TrackedOperationIdFilter`, `NotificationOutboxQueryRequest.NotificationIdFilter` — pre-filter to the row and auto-open its detail), and a `Parked` node offers Retry / Discard in place through `IFlowDebugService.RetryOutboundAsync` / `DiscardOutboundAsync`, which reuse the report pages' relay, site-scope re-check and confirm wording, after which the page re-polls Replay until the node leaves Parked. Per-attempt request/response diff is still not shown (the lifecycle rows carry the error summary only). Found on the way: the prediction circuit deadlock (`4769e192`). | Plan 2026-09-06 "Out of scope"; `FlowStepDetail.razor` parked hint | The Site Calls page owns the retry semantics and its audit rows; duplicating the action on the debugger node would need the same Deployer gate + confirm flow. | First operator who reaches a parked node from the debugger and asks for the button. |
| 112 | **Flow Debugger: a SITE-side inline trace truncation is invisible to the operator** *(Low)*. `ScriptCallResult` / `RouteToCallResponse` carry `TraceEvents` but no `Truncated` flag, so the page's *event list truncated* badge reflects central's `FlowTraceCollector` only — a site run whose own collector dropped rows returns a silently short list. The new 1 MB byte budget (arch-review F1) makes site truncation reachable sooner than the old 500-event count cap did. The fix is additive: a trailing `bool Truncated = false` on both records, a new field on `RouteToCallResponseDto` (regen via `docker/regen-proto.sh sitecommand`), and `FlowDebugRunner` ORing it into `FlowDebugRunResult.Truncated`. | Arch-review remediation code review 2026-09-06 | Needs a proto regeneration plus a rig rebuild to prove end-to-end, and nothing is lost meanwhile: the site's audit rows are written in full, so a Replay of the same execution is never truncated. | The first Live run that shows fewer site steps than its Replay of the same execution. |
| 113 | **Flow Debugger: the method-entry site scope (F2) gates ROUTED site traffic only** *(Low)*. A site-scoped Deployer who Live-runs a method belonging to another plant is still refused at `RouteTarget.ResolveSiteAsync`, but the method's central-side side effects — its `Database.*` calls, `ExternalSystem` calls, notifications — execute before or regardless of that refusal. Only routing is scoped. | Arch-review remediation code review 2026-09-06 | F2's stated property ("no site traffic outside the operator's scope") holds as written. Going further needs a per-method site affinity, which does not exist in the model — ApiMethods are central objects with no site column — so there is nothing to scope a method against without inventing that concept. | A decision to give `ApiMethod` a site affinity, or an operator report of a scoped user reaching another plant's central-side effects. |
| 68 | ~~**`deploy.sh` run from a git worktree silently rehomes every node's bind-mount data dir**~~ **RESOLVED 2026-08-15 (guard added in the same close-out).** The compose files' bind mounts are relative (`./site-a-node-a/data`), so a deploy from a worktree checkout mounts the worktree's copies — the rig keeps running and looks healthy, but its SQLite stores split from the main checkout's and everything written there is orphaned by the next deploy from main. Bit the rows-46–53 close-out live: the seeding agent's `rig-probe-1` deploy persisted into worktree-homed dirs and "vanished" after the orchestrator's redeploy (no tombstone, no product bug — LWW/tombstone forensics ruled out deletion; the row sat in `.claude/worktrees/…/docker/site-a-node-a/data/site-localdb.db` the whole time). Both `docker/deploy.sh` and `docker-env2/deploy.sh` now refuse to run from a `.claude/worktrees/` path (override: `SCADABRIDGE_ALLOW_WORKTREE_DEPLOY=1`). | Found during the rows-46–53 close-out | Closed 2026-08-15. | — |
## Resolved (verified against the code 2026-07-10)
@@ -44,7 +44,9 @@ public static class FlowDebugCommands
var paramsOption = new Option<string?>("--params") { Description = "Flat JSON object of method parameters (validated against the method's schema)" };
var timeoutOption = new Option<int>("--timeout-seconds")
{
Description = "Client-side wait for the run; set to the method's TimeoutSeconds + 25 for long methods",
Description = "Client-side wait for the run; set to the method's TimeoutSeconds + 25 for long methods. "
+ "The server caps long-running commands at its own ManagementService:LongRunningCommandTimeout "
+ "(default 5 min), so a larger value still 504s at that ceiling",
DefaultValueFactory = _ => 120,
};
var cmd = new Command("run-method") { Description = "Run an inbound API method on the active central node with full tracing and print the flow's steps" };
+1 -1
View File
@@ -2312,7 +2312,7 @@ scadabridge --url <url> flow-debug run-method --method-id <int> [--params <json>
|--------|----------|-------------|
| `--method-id` | yes | Inbound API method ID (`api-method list`) |
| `--params` | no | Flat JSON object of parameters, validated against the method's parameter schema. Omit for a parameterless method. |
| `--timeout-seconds` | no | Client-side wait for the run (default 120). Raise this above the method's own `TimeoutSeconds + 25` for a long-running method, or the client gives up before the server's own run grace does. |
| `--timeout-seconds` | no | Client-side wait for the run (default 120). Raise this above the method's own `TimeoutSeconds + 25` for a long-running method, or the client gives up before the server's own run grace does. **Server ceiling:** the management endpoint applies `ScadaBridge:ManagementService:LongRunningCommandTimeout` (default 5 min) to `flow-debug run-method`, so a value above that still returns `504` at the server's ceiling — raise the server setting too if a method legitimately runs longer. |
Parameters that fail schema validation are reported as `VALIDATION_FAILED` without running
the script.
@@ -19,15 +19,22 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Debugging;
/// <item><b>Order</b> = emission order between executions, timestamp within one (arch-review F5: central and
/// site clocks differ). Executions are ordered by the input index of their earliest-given row; inside one
/// execution — one clock — steps are ordered by <c>OccurredAtUtc</c>, then a kind rank (inbound before route
/// call before script run before writes/waits before outbound), then the input index. Replay input arrives
/// timestamp-sorted from the repository, so its order is unchanged; the Live path's collector order wins.</item>
/// call before script run before writes/waits before outbound), then the input index. Replay input is
/// timestamp-sorted; executions are still grouped into contiguous blocks ordered by their first row, so a
/// root-execution step that ran after a routed call (e.g. a central <c>Database.*</c> call after
/// <c>Route.To().Call</c>) displays before the routed site steps — display order only, edges are id-based.</item>
/// <item>Within one <c>ExecutionId</c> consecutive steps are chained in that order, <b>skipping async steps</b> — a
/// <c>Notify</c>, or a cached operation whose first row is <c>CachedSubmit</c>. Each skipped step instead gets an
/// <c>Async</c> (dashed) edge from the chain step immediately before it in time.</item>
/// <item>The first step of an execution is linked from its spawner: a <c>RouteCall</c> whose
/// <c>childExecutionId</c> matches, else the last step of the parent execution that precedes it in
/// the order above — that second case is a <c>Spawn</c> edge (trigger cascade). The edge
/// <c>CrossesBoundary</c> when exactly one end is in the Central lane.</item>
/// <c>childExecutionId</c> matches, else — a <c>Spawn</c> edge (trigger cascade) — the last non-async,
/// non-<c>Response</c> parent step that <b>occurred at or before</b> the child's first step. Parent and
/// child of a cascade share the site clock, so that timestamp rule picks the write that actually fired
/// the trigger; only when it finds nothing (a child clock BEHIND its parent's — a mixed-clock Replay)
/// does the search fall back to display position, and then to the parent's first step. Position alone
/// is not enough: executions are contiguous blocks, so every parent step precedes the child block and
/// the search would always land on the parent's LAST step. The edge <c>CrossesBoundary</c> when
/// exactly one end is in the Central lane.</item>
/// <item>An <c>AttributeWait</c> that was satisfied gets an <c>External</c> (dashed) edge from the
/// preceding <c>AttributeWrite</c> in the same execution — the device echo the system inferred.</item>
/// <item>When an <c>InboundRequest</c> row carries a response summary, a <c>Response</c> step is
@@ -132,9 +139,17 @@ public static class FlowGraphBuilder
spawner = routeCall;
else if (byExecution.TryGetValue(parentId, out var parentChain))
{
// The parent step that precedes the child in DISPLAY order, not by raw timestamp (F5).
// A trigger cascade's parent and child are both stamped by the SITE clock, so the step
// that fired it is the last eligible parent step at or before the child's first step.
// Display position cannot answer this: executions are contiguous blocks, so every parent
// step precedes the child and the position rule degenerates to "the parent's last step"
// (a run shaped ScriptRun → Write(A) → Wait → Write(B) would spawn from Write(B)).
// Position stays as the fallback for the mixed-clock case F5 was written for — a child
// clock behind its parent's, where no parent step satisfies the time rule.
var childTime = first.OccurredAtUtc;
var childPosition = position[first.EventId];
spawner = parentChain.LastOrDefault(p => position[p.EventId] < childPosition && p.Kind != FlowStepKind.Response && !IsAsync(p))
spawner = parentChain.LastOrDefault(p => Spawnable(p) && p.OccurredAtUtc <= childTime)
?? parentChain.LastOrDefault(p => Spawnable(p) && position[p.EventId] < childPosition)
?? parentChain[0];
}
@@ -205,6 +220,14 @@ public static class FlowGraphBuilder
return new FlowGraph(steps, edges, Predicted: false, Truncated: truncated);
}
/// <summary>
/// Can a step be the spawner of a cascaded child execution? The synthesized <c>Response</c> node is not
/// a real step, and an async step (a fire-and-forget notification / cached call) never blocks long
/// enough to be the thing that fired a trigger.
/// </summary>
private static bool Spawnable(FlowStep step)
=> step.Kind != FlowStepKind.Response && !IsAsync(step);
/// <summary>
/// A step the script did not wait for: a notification, or a cached operation — any API / DB step whose
/// first lifecycle row is not the sync one-shot kind (<c>ApiCall</c> / <c>DbWrite</c>), the same rule the
@@ -22,6 +22,11 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Debugging;
/// The copy is also capped by BYTES as well as by count, because it rides a gRPC reply whose
/// default 4 MB receive limit a few hundred large error rows would otherwise breach — and that
/// same reply carries the script's return value, so an oversized trace would fail a real call.
/// The budget is an approximation, not a wire measurement: it sums the UTF-16 <c>char</c> length
/// of <see cref="AuditEvent.DetailsJson"/> only, so it ignores the event's other fields and the
/// protobuf framing, and 1 048 576 chars is 1 MB of UTF-8 for ASCII payloads but up to ~3 MB for
/// non-ASCII ones. That worst case still fits the 4 MB limit with the rest of the reply, which is
/// what the budget exists to guarantee — raise it only with a real serialized-size measurement.
/// </summary>
public sealed class FlowTraceCollector
{
@@ -30,7 +35,8 @@ public sealed class FlowTraceCollector
/// <summary>
/// Default inline byte budget per run — the summed redacted <see cref="AuditEvent.DetailsJson"/>
/// length, well under the 4 MB gRPC default receive limit the reply must fit in.
/// length in UTF-16 chars (see the type remarks: ~1 MB of UTF-8 for ASCII payloads, ~3 MB worst
/// case for non-ASCII ones, both under the 4 MB gRPC default receive limit before framing).
/// </summary>
public const int DefaultMaxBytes = 1_048_576;
@@ -113,7 +113,11 @@ public sealed class FlowDebugRunner
}
var root = Guid.NewGuid();
var collector = new FlowTraceCollector(redactor: sp.GetService<IAuditRedactor>());
// One redactor for the whole run: the collector filters the emitted rows, and the runner
// filters the rows it synthesizes ITSELF (review follow-up) — the central writer redacts a
// private copy, so without this the caller's copy of the inbound row would be the raw one.
var redactor = sp.GetService<IAuditRedactor>();
var collector = new FlowTraceCollector(redactor: redactor);
var startedAt = DateTimeOffset.UtcNow;
var sw = Stopwatch.StartNew();
var timeout = TimeSpan.FromSeconds(Math.Max(1, method.TimeoutSeconds));
@@ -177,7 +181,7 @@ public sealed class FlowDebugRunner
// clock and site rows the site's, so re-sorting by OccurredAtUtc lets a skewed site clock float
// its steps ahead of the RouteCall that started them. (The value-change path below keeps its
// OrderBy: every row it merges was stamped by the same site.)
var ordered = new List<AuditEvent>(events.Count + 1) { inboundRow };
var ordered = new List<AuditEvent>(events.Count + 1) { redactor?.Apply(inboundRow) ?? inboundRow };
ordered.AddRange(events);
return new FlowDebugRunResult(
@@ -210,6 +214,9 @@ public sealed class FlowDebugRunner
var root = Guid.NewGuid();
var startedAt = DateTimeOffset.UtcNow;
var sw = Stopwatch.StartNew();
// Review follow-up: the root row is synthesized here, so the copy handed back must be
// redacted by this method — WriteCentralAsync's writer only redacts its own private copy.
var redactor = sp.GetService<IAuditRedactor>();
// IInstanceRouter is the same seam RouteHelper uses (central binds it to the comms layer).
var router = sp.GetRequiredService<IInstanceRouter>();
var write = await router.RouteToSetAttributesAsync(
@@ -239,11 +246,12 @@ public sealed class FlowDebugRunner
Writes = [new AttributeWriteDetail(cmd.AttributeName, cmd.Value, write.Success, write.ErrorMessage)],
}.ToJson());
await WriteCentralAsync(sp, rootRow, ct).ConfigureAwait(false);
var redactedRoot = redactor?.Apply(rootRow) ?? rootRow;
if (!write.Success)
{
return new FlowDebugRunResult(root, false, write.ErrorMessage ?? "Attribute write failed", "WRITE_FAILED",
[rootRow], false, startedAt, DateTimeOffset.UtcNow);
[redactedRoot], false, startedAt, DateTimeOffset.UtcNow);
}
// Poll the site for the traced run(s) the write spawned — matched by ParentExecutionId
@@ -281,7 +289,7 @@ public sealed class FlowDebugRunner
await _delay(PollInterval, ct).ConfigureAwait(false);
}
var ordered = new List<AuditEvent> { rootRow };
var ordered = new List<AuditEvent> { redactedRoot };
ordered.AddRange(known.Values.OrderBy(e => e.OccurredAtUtc).ThenBy(e => e.EventId));
var completed = known.Values.Any(e =>
{
@@ -447,6 +447,31 @@ public class FlowGraphBuilderTests
Assert.Equal(run.EventId, spawn.ToEventId);
}
[Fact]
public void SameClock_TriggerCascade_SpawnsFromTheWriteThatFiredIt_NotTheParentsLastStep()
{
// Review follow-up: executions are contiguous blocks, so "the last parent step BEFORE the child
// in display order" is always the parent's last step. Parent and child of a cascade share the
// SITE clock, so the write that fired the trigger is the last parent step at or before the child.
var run = SiteRunRow(0);
var writeA = WriteRow(10, "DelmiaReceiver.Trigger");
var wait = ScadaBridgeAuditEventFactory.Create(AuditChannel.ScriptTrace, AuditKind.AttributeWait, AuditStatus.Delivered,
occurredAtUtc: T0.AddMilliseconds(20), actor: "S", target: "DelmiaReceiver.Processed", sourceNode: "node-a",
executionId: SiteRun, sourceSiteId: "site-a", sourceInstanceId: "M1", durationMs: 5,
extra: new ScriptTraceExtra { WaitAttribute = "DelmiaReceiver.Processed", WaitOutcome = "satisfied" }.ToJson());
var writeB = WriteRow(30, "DelmiaReceiver.Ack");
var cascade = ScadaBridgeAuditEventFactory.Create(AuditChannel.ScriptTrace, AuditKind.ScriptRun, AuditStatus.Delivered,
occurredAtUtc: T0.AddMilliseconds(15), actor: "OnTrigger", target: "OnTrigger", sourceNode: "node-a",
executionId: Cascade, parentExecutionId: SiteRun, sourceSiteId: "site-a", sourceInstanceId: "M1", durationMs: 4,
extra: new ScriptTraceExtra { TriggerKind = "valuechange", TriggerAttribute = "DelmiaReceiver.Trigger" }.ToJson());
var graph = FlowGraphBuilder.Build([run, writeA, wait, writeB, cascade]);
var spawn = Assert.Single(graph.Edges, e => e.Spawn);
Assert.Equal(writeA.EventId, spawn.FromEventId);
Assert.Equal(cascade.EventId, spawn.ToEventId);
}
[Fact]
public void FoldedNode_TakesTheEmissionIndexOfItsEarliestEmittedRow()
{
@@ -394,6 +394,46 @@ public class FlowDebugRunnerTests
Assert.Equal(2, result.Events.Count); // inbound row + the one sink event
Assert.Equal("{\"SECRET\":\"ABC\"}", result.Events[1].DetailsJson);
Assert.False(result.Truncated);
// Review follow-up: the row the RUNNER synthesizes is redacted too — the central writer
// only redacts its own private copy, so the caller's copy would otherwise be the raw one.
var written = Assert.Single(_central, e => e.AsRow().Kind == AuditKind.InboundRequest);
Assert.Equal(written.DetailsJson!.ToUpperInvariant(), result.Events[0].DetailsJson);
Assert.NotEqual(written.DetailsJson, result.Events[0].DetailsJson); // the raw row still reaches the writer
}
[Fact]
public async Task RunValueChange_SynthesizedRootRow_IsRedacted_WhenARedactorIsRegistered()
{
var (_, _, puller) = WireValueChange();
puller.PullAsync(default!, default!, default, default, default).ReturnsForAnyArgs(Array.Empty<AuditEvent>());
_services.AddSingleton<IAuditRedactor>(new UpperRedactor());
var result = await Runner().RunValueChangeAsync(_services.BuildServiceProvider(),
new FlowDebugRunValueChangeCommand(5, "Status", "Running", WaitSeconds: 1), Deployer, CancellationToken.None);
var root = Assert.Single(result.Events);
var written = Assert.Single(_central);
Assert.Equal(written.DetailsJson!.ToUpperInvariant(), root.DetailsJson);
Assert.NotEqual(written.DetailsJson, root.DetailsJson);
}
[Fact]
public async Task RunValueChange_FailedWrite_ReturnsTheRedactedRootRow()
{
WireValueChange();
_router.RouteToSetAttributesAsync("site-a", Arg.Any<RouteToSetAttributesRequest>(), Arg.Any<CancellationToken>())
.Returns(ci => new RouteToSetAttributesResponse(((RouteToSetAttributesRequest)ci[1]).CorrelationId, false, "no such attribute", DateTimeOffset.UtcNow));
_services.AddSingleton<IAuditRedactor>(new UpperRedactor());
var result = await Runner().RunValueChangeAsync(_services.BuildServiceProvider(),
new FlowDebugRunValueChangeCommand(5, "Status", "Running", WaitSeconds: 1), Deployer, CancellationToken.None);
Assert.False(result.Completed);
Assert.Equal("WRITE_FAILED", result.ErrorCode);
var root = Assert.Single(result.Events);
var written = Assert.Single(_central);
Assert.Equal(written.DetailsJson!.ToUpperInvariant(), root.DetailsJson);
Assert.NotEqual(written.DetailsJson, root.DetailsJson);
}
[Fact]