diff --git a/docs/plans/2026-07-08-deferred-work-register.md b/docs/plans/2026-07-08-deferred-work-register.md index 85ba97f7..78552192 100644 --- a/docs/plans/2026-07-08-deferred-work-register.md +++ b/docs/plans/2026-07-08-deferred-work-register.md @@ -104,6 +104,8 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round | 109 | **`ZB.MOM.WW.Secrets.Replicator.SqlServer`'s store migrator retries its DDL with a flat `100 * attempt` backoff and no jitter — a latent concurrent-boot flake, open in Secrets** *(Low)*. Porting that migrator's retry loop verbatim for the Auth 0.3.0 SQL Server API-key store (register row 108) reproduced an intermittent exhaustion — about once in sixty 8-way concurrent-boot trials the five flat-backoff retries ran out before a DDL/lock conflict cleared; adding jitter to the Auth port's backoff took it to 0 failures in 180 trials. Secrets' own migrator has no concurrent-migrate test, so the same defect is presumed still open there — it just hasn't been exercised at 8-way concurrency, and a ScadaBridge central pair is only ever 2-way, so this is unlikely to bite in practice. | Found 2026-09-02 while implementing `docs/plans/2026-09-02-inbound-api-key-shared-store.md` (Auth 0.3.0 `SqlServerApiKeyStoreMigrator`) | Add jitter to `SqlServerSecretsStoreMigrator`'s backoff in `~/Desktop/scadaproj/ZB.MOM.WW.Secrets` (mirror the Auth fix) and add a `Migrate_Concurrently_FromEightTasks_AllSucceed`-shaped live test to catch a regression. Lives entirely in scadaproj, not this repo. | Next time Secrets' SQL Server migrator is touched, or if a wider-than-2-way concurrent boot of a Secrets-backed cluster is ever planned. | | 110 | **Flow Debugger follow-ups: the *Vertical trace* list toggle (mockup Direction B) and console capture on `ScriptRun`** *(Low)*. v1 ships the Direction A swimlane only; the mockup's Direction B — the same steps as a vertical, filterable list with the lane as a column — was kept as an alternate for long flows and screen readers, and the detail panel's *Console* tab is absent because script `Console`/log output is not captured per run (`ScriptRun` rows carry parameters, return value, error and timing, not stdout). Both are additive: a list view over the existing `FlowGraph`, and an extra `Extra.console` field on the `ScriptRun` row's `ScriptTraceExtra` (payload cap applies). | Design `docs/plans/2026-09-05-flow-debugger-design.md` §11; plan `2026-09-05-flow-debugger.md` Task 23 | Neither was asked for by an operator; the swimlane + Timing/Writes tabs cover the acceptance flow (Delmia recipe download, rig step H), and console capture needs a per-run writer seam in `ScriptRuntimeContext` plus a redaction decision. | First operator request for a list view or for script log lines in the step detail. | | 111 | ~~**Flow Debugger outbound nodes: no Retry/Discard on a parked node, and no `?correlationId=` deep link from the node to the Site Calls / Notification report pages** *(Low)*~~ **RESOLVED 2026-09-06 (plan `docs/plans/2026-09-06-flow-debugger-followups.md`).** `FlowStep` now carries the row's `CorrelationId`; the detail panel deep-links to `/site-calls/report?correlationId=` / `/notifications/report?correlationId=` (both pages accept it through additive per-id query filters — `SiteCallQueryRequest.TrackedOperationIdFilter`, `NotificationOutboxQueryRequest.NotificationIdFilter` — pre-filter to the row and auto-open its detail), and a `Parked` node offers Retry / Discard in place through `IFlowDebugService.RetryOutboundAsync` / `DiscardOutboundAsync`, which reuse the report pages' relay, site-scope re-check and confirm wording, after which the page re-polls Replay until the node leaves Parked. Per-attempt request/response diff is still not shown (the lifecycle rows carry the error summary only). Found on the way: the prediction circuit deadlock (`4769e192`). | Plan 2026-09-06 "Out of scope"; `FlowStepDetail.razor` parked hint | The Site Calls page owns the retry semantics and its audit rows; duplicating the action on the debugger node would need the same Deployer gate + confirm flow. | First operator who reaches a parked node from the debugger and asks for the button. | +| 112 | **Flow Debugger: a SITE-side inline trace truncation is invisible to the operator** *(Low)*. `ScriptCallResult` / `RouteToCallResponse` carry `TraceEvents` but no `Truncated` flag, so the page's *event list truncated* badge reflects central's `FlowTraceCollector` only — a site run whose own collector dropped rows returns a silently short list. The new 1 MB byte budget (arch-review F1) makes site truncation reachable sooner than the old 500-event count cap did. The fix is additive: a trailing `bool Truncated = false` on both records, a new field on `RouteToCallResponseDto` (regen via `docker/regen-proto.sh sitecommand`), and `FlowDebugRunner` ORing it into `FlowDebugRunResult.Truncated`. | Arch-review remediation code review 2026-09-06 | Needs a proto regeneration plus a rig rebuild to prove end-to-end, and nothing is lost meanwhile: the site's audit rows are written in full, so a Replay of the same execution is never truncated. | The first Live run that shows fewer site steps than its Replay of the same execution. | +| 113 | **Flow Debugger: the method-entry site scope (F2) gates ROUTED site traffic only** *(Low)*. A site-scoped Deployer who Live-runs a method belonging to another plant is still refused at `RouteTarget.ResolveSiteAsync`, but the method's central-side side effects — its `Database.*` calls, `ExternalSystem` calls, notifications — execute before or regardless of that refusal. Only routing is scoped. | Arch-review remediation code review 2026-09-06 | F2's stated property ("no site traffic outside the operator's scope") holds as written. Going further needs a per-method site affinity, which does not exist in the model — ApiMethods are central objects with no site column — so there is nothing to scope a method against without inventing that concept. | A decision to give `ApiMethod` a site affinity, or an operator report of a scoped user reaching another plant's central-side effects. | | 68 | ~~**`deploy.sh` run from a git worktree silently rehomes every node's bind-mount data dir**~~ **RESOLVED 2026-08-15 (guard added in the same close-out).** The compose files' bind mounts are relative (`./site-a-node-a/data`), so a deploy from a worktree checkout mounts the worktree's copies — the rig keeps running and looks healthy, but its SQLite stores split from the main checkout's and everything written there is orphaned by the next deploy from main. Bit the rows-46–53 close-out live: the seeding agent's `rig-probe-1` deploy persisted into worktree-homed dirs and "vanished" after the orchestrator's redeploy (no tombstone, no product bug — LWW/tombstone forensics ruled out deletion; the row sat in `.claude/worktrees/…/docker/site-a-node-a/data/site-localdb.db` the whole time). Both `docker/deploy.sh` and `docker-env2/deploy.sh` now refuse to run from a `.claude/worktrees/` path (override: `SCADABRIDGE_ALLOW_WORKTREE_DEPLOY=1`). | Found during the rows-46–53 close-out | Closed 2026-08-15. | — | ## Resolved (verified against the code 2026-07-10) diff --git a/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/FlowDebugCommands.cs b/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/FlowDebugCommands.cs index 944ee576..c054676a 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/FlowDebugCommands.cs +++ b/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/FlowDebugCommands.cs @@ -44,7 +44,9 @@ public static class FlowDebugCommands var paramsOption = new Option("--params") { Description = "Flat JSON object of method parameters (validated against the method's schema)" }; var timeoutOption = new Option("--timeout-seconds") { - Description = "Client-side wait for the run; set to the method's TimeoutSeconds + 25 for long methods", + Description = "Client-side wait for the run; set to the method's TimeoutSeconds + 25 for long methods. " + + "The server caps long-running commands at its own ManagementService:LongRunningCommandTimeout " + + "(default 5 min), so a larger value still 504s at that ceiling", DefaultValueFactory = _ => 120, }; var cmd = new Command("run-method") { Description = "Run an inbound API method on the active central node with full tracing and print the flow's steps" }; diff --git a/src/ZB.MOM.WW.ScadaBridge.CLI/README.md b/src/ZB.MOM.WW.ScadaBridge.CLI/README.md index d4bfde8a..c69e28a7 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CLI/README.md +++ b/src/ZB.MOM.WW.ScadaBridge.CLI/README.md @@ -2312,7 +2312,7 @@ scadabridge --url flow-debug run-method --method-id [--params |--------|----------|-------------| | `--method-id` | yes | Inbound API method ID (`api-method list`) | | `--params` | no | Flat JSON object of parameters, validated against the method's parameter schema. Omit for a parameterless method. | -| `--timeout-seconds` | no | Client-side wait for the run (default 120). Raise this above the method's own `TimeoutSeconds + 25` for a long-running method, or the client gives up before the server's own run grace does. | +| `--timeout-seconds` | no | Client-side wait for the run (default 120). Raise this above the method's own `TimeoutSeconds + 25` for a long-running method, or the client gives up before the server's own run grace does. **Server ceiling:** the management endpoint applies `ScadaBridge:ManagementService:LongRunningCommandTimeout` (default 5 min) to `flow-debug run-method`, so a value above that still returns `504` at the server's ceiling — raise the server setting too if a method legitimately runs longer. | Parameters that fail schema validation are reported as `VALIDATION_FAILED` without running the script. diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Debugging/FlowGraphBuilder.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Debugging/FlowGraphBuilder.cs index 96138463..643db394 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Debugging/FlowGraphBuilder.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Debugging/FlowGraphBuilder.cs @@ -19,15 +19,22 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Debugging; /// Order = emission order between executions, timestamp within one (arch-review F5: central and /// site clocks differ). Executions are ordered by the input index of their earliest-given row; inside one /// execution — one clock — steps are ordered by OccurredAtUtc, then a kind rank (inbound before route -/// call before script run before writes/waits before outbound), then the input index. Replay input arrives -/// timestamp-sorted from the repository, so its order is unchanged; the Live path's collector order wins. +/// call before script run before writes/waits before outbound), then the input index. Replay input is +/// timestamp-sorted; executions are still grouped into contiguous blocks ordered by their first row, so a +/// root-execution step that ran after a routed call (e.g. a central Database.* call after +/// Route.To().Call) displays before the routed site steps — display order only, edges are id-based. /// Within one ExecutionId consecutive steps are chained in that order, skipping async steps — a /// Notify, or a cached operation whose first row is CachedSubmit. Each skipped step instead gets an /// Async (dashed) edge from the chain step immediately before it in time. /// The first step of an execution is linked from its spawner: a RouteCall whose -/// childExecutionId matches, else the last step of the parent execution that precedes it in -/// the order above — that second case is a Spawn edge (trigger cascade). The edge -/// CrossesBoundary when exactly one end is in the Central lane. +/// childExecutionId matches, else — a Spawn edge (trigger cascade) — the last non-async, +/// non-Response parent step that occurred at or before the child's first step. Parent and +/// child of a cascade share the site clock, so that timestamp rule picks the write that actually fired +/// the trigger; only when it finds nothing (a child clock BEHIND its parent's — a mixed-clock Replay) +/// does the search fall back to display position, and then to the parent's first step. Position alone +/// is not enough: executions are contiguous blocks, so every parent step precedes the child block and +/// the search would always land on the parent's LAST step. The edge CrossesBoundary when +/// exactly one end is in the Central lane. /// An AttributeWait that was satisfied gets an External (dashed) edge from the /// preceding AttributeWrite in the same execution — the device echo the system inferred. /// When an InboundRequest row carries a response summary, a Response step is @@ -132,9 +139,17 @@ public static class FlowGraphBuilder spawner = routeCall; else if (byExecution.TryGetValue(parentId, out var parentChain)) { - // The parent step that precedes the child in DISPLAY order, not by raw timestamp (F5). + // A trigger cascade's parent and child are both stamped by the SITE clock, so the step + // that fired it is the last eligible parent step at or before the child's first step. + // Display position cannot answer this: executions are contiguous blocks, so every parent + // step precedes the child and the position rule degenerates to "the parent's last step" + // (a run shaped ScriptRun → Write(A) → Wait → Write(B) would spawn from Write(B)). + // Position stays as the fallback for the mixed-clock case F5 was written for — a child + // clock behind its parent's, where no parent step satisfies the time rule. + var childTime = first.OccurredAtUtc; var childPosition = position[first.EventId]; - spawner = parentChain.LastOrDefault(p => position[p.EventId] < childPosition && p.Kind != FlowStepKind.Response && !IsAsync(p)) + spawner = parentChain.LastOrDefault(p => Spawnable(p) && p.OccurredAtUtc <= childTime) + ?? parentChain.LastOrDefault(p => Spawnable(p) && position[p.EventId] < childPosition) ?? parentChain[0]; } @@ -205,6 +220,14 @@ public static class FlowGraphBuilder return new FlowGraph(steps, edges, Predicted: false, Truncated: truncated); } + /// + /// Can a step be the spawner of a cascaded child execution? The synthesized Response node is not + /// a real step, and an async step (a fire-and-forget notification / cached call) never blocks long + /// enough to be the thing that fired a trigger. + /// + private static bool Spawnable(FlowStep step) + => step.Kind != FlowStepKind.Response && !IsAsync(step); + /// /// A step the script did not wait for: a notification, or a cached operation — any API / DB step whose /// first lifecycle row is not the sync one-shot kind (ApiCall / DbWrite), the same rule the diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Debugging/FlowTraceCollector.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Debugging/FlowTraceCollector.cs index aa92efc1..37980852 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Debugging/FlowTraceCollector.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Debugging/FlowTraceCollector.cs @@ -22,6 +22,11 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Debugging; /// The copy is also capped by BYTES as well as by count, because it rides a gRPC reply whose /// default 4 MB receive limit a few hundred large error rows would otherwise breach — and that /// same reply carries the script's return value, so an oversized trace would fail a real call. +/// The budget is an approximation, not a wire measurement: it sums the UTF-16 char length +/// of only, so it ignores the event's other fields and the +/// protobuf framing, and 1 048 576 chars is 1 MB of UTF-8 for ASCII payloads but up to ~3 MB for +/// non-ASCII ones. That worst case still fits the 4 MB limit with the rest of the reply, which is +/// what the budget exists to guarantee — raise it only with a real serialized-size measurement. /// public sealed class FlowTraceCollector { @@ -30,7 +35,8 @@ public sealed class FlowTraceCollector /// /// Default inline byte budget per run — the summed redacted - /// length, well under the 4 MB gRPC default receive limit the reply must fit in. + /// length in UTF-16 chars (see the type remarks: ~1 MB of UTF-8 for ASCII payloads, ~3 MB worst + /// case for non-ASCII ones, both under the 4 MB gRPC default receive limit before framing). /// public const int DefaultMaxBytes = 1_048_576; diff --git a/src/ZB.MOM.WW.ScadaBridge.ManagementService/FlowDebug/FlowDebugRunner.cs b/src/ZB.MOM.WW.ScadaBridge.ManagementService/FlowDebug/FlowDebugRunner.cs index 00470a04..26eca87a 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ManagementService/FlowDebug/FlowDebugRunner.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ManagementService/FlowDebug/FlowDebugRunner.cs @@ -113,7 +113,11 @@ public sealed class FlowDebugRunner } var root = Guid.NewGuid(); - var collector = new FlowTraceCollector(redactor: sp.GetService()); + // One redactor for the whole run: the collector filters the emitted rows, and the runner + // filters the rows it synthesizes ITSELF (review follow-up) — the central writer redacts a + // private copy, so without this the caller's copy of the inbound row would be the raw one. + var redactor = sp.GetService(); + var collector = new FlowTraceCollector(redactor: redactor); var startedAt = DateTimeOffset.UtcNow; var sw = Stopwatch.StartNew(); var timeout = TimeSpan.FromSeconds(Math.Max(1, method.TimeoutSeconds)); @@ -177,7 +181,7 @@ public sealed class FlowDebugRunner // clock and site rows the site's, so re-sorting by OccurredAtUtc lets a skewed site clock float // its steps ahead of the RouteCall that started them. (The value-change path below keeps its // OrderBy: every row it merges was stamped by the same site.) - var ordered = new List(events.Count + 1) { inboundRow }; + var ordered = new List(events.Count + 1) { redactor?.Apply(inboundRow) ?? inboundRow }; ordered.AddRange(events); return new FlowDebugRunResult( @@ -210,6 +214,9 @@ public sealed class FlowDebugRunner var root = Guid.NewGuid(); var startedAt = DateTimeOffset.UtcNow; var sw = Stopwatch.StartNew(); + // Review follow-up: the root row is synthesized here, so the copy handed back must be + // redacted by this method — WriteCentralAsync's writer only redacts its own private copy. + var redactor = sp.GetService(); // IInstanceRouter is the same seam RouteHelper uses (central binds it to the comms layer). var router = sp.GetRequiredService(); var write = await router.RouteToSetAttributesAsync( @@ -239,11 +246,12 @@ public sealed class FlowDebugRunner Writes = [new AttributeWriteDetail(cmd.AttributeName, cmd.Value, write.Success, write.ErrorMessage)], }.ToJson()); await WriteCentralAsync(sp, rootRow, ct).ConfigureAwait(false); + var redactedRoot = redactor?.Apply(rootRow) ?? rootRow; if (!write.Success) { return new FlowDebugRunResult(root, false, write.ErrorMessage ?? "Attribute write failed", "WRITE_FAILED", - [rootRow], false, startedAt, DateTimeOffset.UtcNow); + [redactedRoot], false, startedAt, DateTimeOffset.UtcNow); } // Poll the site for the traced run(s) the write spawned — matched by ParentExecutionId @@ -281,7 +289,7 @@ public sealed class FlowDebugRunner await _delay(PollInterval, ct).ConfigureAwait(false); } - var ordered = new List { rootRow }; + var ordered = new List { redactedRoot }; ordered.AddRange(known.Values.OrderBy(e => e.OccurredAtUtc).ThenBy(e => e.EventId)); var completed = known.Values.Any(e => { diff --git a/tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Debugging/FlowGraphBuilderTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Debugging/FlowGraphBuilderTests.cs index d2b4e15c..bfd3f478 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Debugging/FlowGraphBuilderTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Debugging/FlowGraphBuilderTests.cs @@ -447,6 +447,31 @@ public class FlowGraphBuilderTests Assert.Equal(run.EventId, spawn.ToEventId); } + [Fact] + public void SameClock_TriggerCascade_SpawnsFromTheWriteThatFiredIt_NotTheParentsLastStep() + { + // Review follow-up: executions are contiguous blocks, so "the last parent step BEFORE the child + // in display order" is always the parent's last step. Parent and child of a cascade share the + // SITE clock, so the write that fired the trigger is the last parent step at or before the child. + var run = SiteRunRow(0); + var writeA = WriteRow(10, "DelmiaReceiver.Trigger"); + var wait = ScadaBridgeAuditEventFactory.Create(AuditChannel.ScriptTrace, AuditKind.AttributeWait, AuditStatus.Delivered, + occurredAtUtc: T0.AddMilliseconds(20), actor: "S", target: "DelmiaReceiver.Processed", sourceNode: "node-a", + executionId: SiteRun, sourceSiteId: "site-a", sourceInstanceId: "M1", durationMs: 5, + extra: new ScriptTraceExtra { WaitAttribute = "DelmiaReceiver.Processed", WaitOutcome = "satisfied" }.ToJson()); + var writeB = WriteRow(30, "DelmiaReceiver.Ack"); + var cascade = ScadaBridgeAuditEventFactory.Create(AuditChannel.ScriptTrace, AuditKind.ScriptRun, AuditStatus.Delivered, + occurredAtUtc: T0.AddMilliseconds(15), actor: "OnTrigger", target: "OnTrigger", sourceNode: "node-a", + executionId: Cascade, parentExecutionId: SiteRun, sourceSiteId: "site-a", sourceInstanceId: "M1", durationMs: 4, + extra: new ScriptTraceExtra { TriggerKind = "valuechange", TriggerAttribute = "DelmiaReceiver.Trigger" }.ToJson()); + + var graph = FlowGraphBuilder.Build([run, writeA, wait, writeB, cascade]); + + var spawn = Assert.Single(graph.Edges, e => e.Spawn); + Assert.Equal(writeA.EventId, spawn.FromEventId); + Assert.Equal(cascade.EventId, spawn.ToEventId); + } + [Fact] public void FoldedNode_TakesTheEmissionIndexOfItsEarliestEmittedRow() { diff --git a/tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/FlowDebugRunnerTests.cs b/tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/FlowDebugRunnerTests.cs index 4595a45e..c7ce6c99 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/FlowDebugRunnerTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/FlowDebugRunnerTests.cs @@ -394,6 +394,46 @@ public class FlowDebugRunnerTests Assert.Equal(2, result.Events.Count); // inbound row + the one sink event Assert.Equal("{\"SECRET\":\"ABC\"}", result.Events[1].DetailsJson); Assert.False(result.Truncated); + // Review follow-up: the row the RUNNER synthesizes is redacted too — the central writer + // only redacts its own private copy, so the caller's copy would otherwise be the raw one. + var written = Assert.Single(_central, e => e.AsRow().Kind == AuditKind.InboundRequest); + Assert.Equal(written.DetailsJson!.ToUpperInvariant(), result.Events[0].DetailsJson); + Assert.NotEqual(written.DetailsJson, result.Events[0].DetailsJson); // the raw row still reaches the writer + } + + [Fact] + public async Task RunValueChange_SynthesizedRootRow_IsRedacted_WhenARedactorIsRegistered() + { + var (_, _, puller) = WireValueChange(); + puller.PullAsync(default!, default!, default, default, default).ReturnsForAnyArgs(Array.Empty()); + _services.AddSingleton(new UpperRedactor()); + + var result = await Runner().RunValueChangeAsync(_services.BuildServiceProvider(), + new FlowDebugRunValueChangeCommand(5, "Status", "Running", WaitSeconds: 1), Deployer, CancellationToken.None); + + var root = Assert.Single(result.Events); + var written = Assert.Single(_central); + Assert.Equal(written.DetailsJson!.ToUpperInvariant(), root.DetailsJson); + Assert.NotEqual(written.DetailsJson, root.DetailsJson); + } + + [Fact] + public async Task RunValueChange_FailedWrite_ReturnsTheRedactedRootRow() + { + WireValueChange(); + _router.RouteToSetAttributesAsync("site-a", Arg.Any(), Arg.Any()) + .Returns(ci => new RouteToSetAttributesResponse(((RouteToSetAttributesRequest)ci[1]).CorrelationId, false, "no such attribute", DateTimeOffset.UtcNow)); + _services.AddSingleton(new UpperRedactor()); + + var result = await Runner().RunValueChangeAsync(_services.BuildServiceProvider(), + new FlowDebugRunValueChangeCommand(5, "Status", "Running", WaitSeconds: 1), Deployer, CancellationToken.None); + + Assert.False(result.Completed); + Assert.Equal("WRITE_FAILED", result.ErrorCode); + var root = Assert.Single(result.Events); + var written = Assert.Single(_central); + Assert.Equal(written.DetailsJson!.ToUpperInvariant(), root.DetailsJson); + Assert.NotEqual(written.DetailsJson, root.DetailsJson); } [Fact]