The MES/Delmia handshake fast-paths on the previous run's cached CompleteFlag
because the reset write has not echoed back before the wait registers. An
edge-triggered wait is rejected (a subscription can swallow the reset when the
device flips inside one sampling interval); the fix encodes the reset leg inside
a new WriteBatchAndWaitAsync overload that observes the reset echo before writing
the trigger and returns a phase-aware HandshakeResult.
Claude-Session: https://claude.ai/code/session_01XPsuy2ZZ5Gbyfpt8PrK5BV
The Run-inbound-API-method pane rendered one text box per top-level schema
field and coerced the text on the page; an array/object field went through
JsonNode.Parse and, on failure, was SENT AS A STRING, while an empty box was
silently dropped. A required array-of-object parameter (MesMoveOut WorkOrders)
could not be entered at all — QA runs reached the site with "WorkOrders":[].
- The method pane now renders the same shared ParameterValueForm the script
pane uses (typed inputs, nested groups, add/remove list item editors,
$ref:lib: resolution, inline validation), keyed on the method id so a change
of method remounts it. The page keeps ONE value tree and BuildParameters()
serializes it as-is; the flat _inputs / Coerce() / SetInput path is deleted.
- Form | Raw JSON toggle: raw mode is a Monaco json editor for the whole
request body, given the method's root schema for completion, parsed and
schema-validated (InboundApiSchema.Validate) on every edit with the error
inline. Entering raw mode seeds the editor from the tree; leaving it
re-mounts the form over the last valid tree.
- Run is disabled while the form or the raw document is invalid (a required
field left empty disables Run before the first keystroke).
- localStorage per-method entry stores the value tree; on restore it is
validated against the schema and dropped if invalid, which also retires the
pre-#37 flat string-per-field entries.
- ParameterValueForm.ToJsonSchema / JsonElementToObject made internal static
for page reuse.
- Tests: 7 new bUnit tests (WorkOrders array-of-object round-trip to the
RunMethodAsync payload, required list-item gating, invalid raw JSON blocks
Run + shows parse/schema/non-object errors, valid raw JSON sent verbatim and
survives the switch back to the form, method switch resets, persisted tree
restored, legacy flat entry dropped); existing method-pane tests moved to
the form's #param-<field> ids and fill the required field before Run.
Playwright smoke selector moved to #param-Text.
- Docs: design doc, Component-CentralUI.md entry-points bullet, CLAUDE.md.
CentralUI.Tests 1425/1425 green; Playwright project builds.
Claude-Session: https://claude.ai/code/session_01XPsuy2ZZ5Gbyfpt8PrK5BV
Both driven from the Mac over the winrm CLI (WinRM 5985 + SMB 445, no helper
on the boxes). Migration 53->55; the first Central boot encrypts
DataConnections.PrimaryConfiguration (dp:protocol:v1:), so rollback is the
COPY_ONLY DB restore, not a binary swap. QA: all four nodes stopped together,
one shared-DB migration, 2 min 35 s window, preflight against a restored clone.
Claude-Session: https://claude.ai/code/session_01RgthVLVU5CJ8SUuEwruTkt
Second slice of the 2026-09-09 whole-system review remediation
(docs/plans/2026-09-09-arch-review.md §10), on top of the High slice 146a1323.
Runtime/devices: SR-2 alarm on-trigger integration scope; SR-3 correlated NativeAlarmActor
subscribe retry; SR-4 immutable audited DB target; SR-5 ordered InstanceActor stop/flush;
SR-12 rehydration fence; SR-13 S&F gate recheck per delivery; DC-4 no OPC UA security
downgrade; DC-5 alarm snapshot for every new subscriber; ORCH-1 sync-dispose joins the pump.
Management/UI: MG-2 site scope on audit tree + Flow Debugger Replay; MG-3 stale-probe memo
removed; MG-4 server-owned bulk-deploy deadline; UI-5 sandbox runs in a killable child
process (new CentralUI.SandboxWorker project); UI-14 bounded sandbox output/IPC.
Edge/audit: EI-1 relay audit ownership; EI-2 SkipBodyCapture suppresses error text; EI-3/EI-11
error-field redaction + caps; EI-4 single relay deadline, pre-execution-only retry; EI-5
authenticated relay provenance; EI-10 body cap before audit read.
Persistence/contracts: DL-1 global Instance.UniqueName unique index + conflict translation;
DL-2 DataConnection protocol config encrypted at rest (migration
20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration — coordinated
central upgrade, old binaries cannot read protected rows); DL-3 bounded Replay paging;
DL-6 cross-site Overwrite of a deployed instance rejected; XC-4 proto descriptor contract test.
Config/hygiene: XC-1 central listener isolation by local port; XC-2/XC-3 inert SMTP + cron
keys removed from rig settings; CM-2 env2 bootstrap guard enabled; DC-6 deploy site exit code;
DC-7 notifier retries only explicit standby rejection; XC-5 ClusterClient purged from docs.
Gates recorded in the report: build 0/0, 9 397 passed / 0 failed, rig redeployed on all
eight nodes with the migration applied. 46 Low findings remain open.
Note the shared InstancePicker combobox (2026-09-08 design) replacing
the flat instance <select> on both pages, in Component-CentralUI.md's
Flow Debugger and Debug View sections and the CLAUDE.md Flow Debugger
bullet.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
The trigger-cascade (Spawn) edge used a lightning-bolt marker IN PLACE of the arrowhead. On the upward
Tags→Site hop the bolt sat on top of the riser and read as a broken arrow (rig screenshot 2026-09-08).
Every connector now ends in an ordinary arrowhead; the bolt is a badge (circle + bolt) centred on the
connector's vertical run (FlowGraphView.SpawnBadgePoint, kept in step with OrthogonalPath/AsyncPath).
Same screenshot, second defect: the cascade spawned by "RecipeDownloadFlag = false" was drawn from the
LAST write of the burst (RecipeProcessedFlag) because all four writes share a millisecond and the
spawner rule was time-only. FlowGraphBuilder now prefers the parent's last AttributeWrite of the attribute
the child's triggerAttribute names (at or before the child, else the last such write); the time/position
rules remain the fallback when the child names no attribute or the parent never wrote it.
Verified live on the rebuilt rig (DelmiaSim01 / DelmiaReceiverSimulator): both spawn edges show an
arrowhead + badge, the second one originates from the RecipeDownloadFlag write. bUnit + builder tests
updated/added; Playwright FlowDebuggerTests 4/4.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Replaces the value-change (site/instance/attribute) entry with a script picker.
ValueChange/Conditional scripts keep the write-and-poll path with the attribute
derived from the trigger config; every other kind is invoked directly through the
traced RouteToCall path with a parameter form from the script's ParameterDefinitions.
New additive FlowDebugRunScriptCommand; run-value-change stays as the CLI escape hatch.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Code review of the merged arch-review remediation (Tasks 1-13) found four items.
1. FlowDebugRunner returned its OWN synthesized rows un-redacted. RunMethodAsync's
inbound row and RunValueChangeAsync's root row were handed to WriteCentralAsync
(whose writer redacts a private copy) and then put RAW into the returned Events —
so a Live run's caller saw the unfiltered row while Replay of the same execution
showed the redacted one, defeating F1's "Live shows what Replay reads back". Both
methods now resolve IAuditRedactor once per run and place the filtered copy into
the result (including the early-return path on a failed attribute write); the raw
row still reaches the writer, which redacts itself.
2. FlowGraphBuilder's spawner search degenerated to "the last step of the parent".
Since executions are ordered as contiguous blocks, position[p] < childPosition
holds for EVERY parent step, so a run shaped ScriptRun -> Write(A) -> Wait ->
Write(B) whose Write(A) fired the cascade drew the Spawn edge from Write(B). New
rule, in order: (a) the last non-async, non-Response parent step occurring at or
before the child's first step — parent and child of a trigger cascade share the
site clock, so this picks the write that actually fired it; (b) the position rule
as fallback, which is the mixed-clock case F5 was written for (a child clock
behind its parent's, where no parent step satisfies (a)); (c) parentChain[0].
The class remark's stale "Replay order is unchanged" claim is replaced with what
actually happens to a central Database.* call made after Route.To().Call.
3. FlowTraceCollector's byte budget counts UTF-16 chars of DetailsJson only, so
"well under the 4 MB gRPC default" overstated it. Documented honestly: ~1 MB of
UTF-8 for ASCII payloads, ~3 MB worst case for non-ASCII, both under 4 MB before
framing — and raise it only against a real serialized-size measurement.
4. CLI --timeout-seconds help + README row now say the management endpoint caps
flow-debug run-method at ManagementService:LongRunningCommandTimeout (default
5 min), so a larger client value still 504s at the server's ceiling.
Deferred-work register rows 112 (site-side inline trace truncation is invisible to
the operator — needs a proto field) and 113 (the F2 method-entry site scope gates
routed traffic only; ApiMethods have no site affinity to scope against) record what
this review deliberately did not fix.
Tests: FlowGraphBuilderTests gains the same-clock Write(A)/Wait/Write(B) cascade
(red before the rule change, and the clock-behind test still passes on rule (b));
FlowDebugRunnerTests extends the redactor test to the inbound row and adds two
value-change cases (settled poll, failed write). Commons 793, ManagementService 615,
CentralUI 1264, CLI 462 — all green.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
The Flow Debugger design doc still described the pre-remediation shape: a
debug-stream subscription for the value-change entry, Deployer everywhere, an
unredacted inline trace copy and "~7 audit rows per call". Reconciled with what
shipped on this branch (Tasks 1-12):
- design D6 / §5.4 / §9 / §10: RequireFlowDebugger page policy with Deployer-only
execution (F3), the PullTraceEvents polling path with AttributeValueUpdate
fields 7/8 recorded as RESERVED and unconsumed (F12), the redacting + byte-
capped shared FlowTraceCollector (F1), route-time site scope (F2), the derived
client waits (F4), the 7-10 row estimate, the doubled central audit write per
routed call, and the inbound-row-after-the-run ordering.
- sitestream.proto: fields 7/8 marked reserved-but-populated, numbers not to be
reused (comment only; no regeneration needed).
- Component-InboundAPI: SkipBodyCapture on the RouteCall row (F11),
SchemaRefResolver's public visibility, WithSiteScope / SITE_SCOPE_DENIED (F2),
Replay's wider role set (F3).
- Component-CentralUI: page policy vs Deployer-only controls (F3), route-time
site scope (F2), client wait derivation (F4).
- "7-10 audit rows per call (3 of them trace-only)" everywhere the estimate is
quoted: the method form, both CLI --trace-all-calls help strings, the CLI
README, the ApiMethod entity doc, the manual migration script and the two
Flow Debugger plans.
- arch-review doc: new §7 Resolution listing F1-F13 with the fixing commit, plus
the tracking-store test race fixed alongside them (86451382).
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Full review of the Flow Debugger commit range (v1.0, v1.1 outbound calls, v1.2 follow-ups,
outcome-merge fix, two perf commits). Also records the exact 11 role-negative Playwright
tests that cannot pass on the login-disabled rig in the cluster-ops skill.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Measuring the tracking-store queue on the rig moved nothing: the DB-done -> Notify.Send
gap stayed at 108 ms. The cost was the audit half of CachedCallTelemetryForwarder:
SqliteAuditWriter.WriteAsync enqueues synchronously but returns the BATCH completion,
and the writer loop coalesces for FlushIntervalMs (50 ms) before it commits, so each
awaited audit row cost ~50 ms + fsync and a cached call awaited three (submit,
attempted, resolve) on the script thread. Every other script-path emitter already
fired-and-observed; only this forwarder awaited.
TryEmitAuditAsync now hands the row to the writer and observes the completion on a
fault-logging continuation (the FallbackAuditWriter ring-buffer fallback still runs
inside that task). Best-effort contract unchanged; per-store FIFO unchanged; nothing
joins the audit and tracking stores by commit order.
Rig, FlowOutboundDemo, 5-run medians: inbound 194 -> 25 ms, site script run 176 -> 17 ms,
CachedWrite submit->done 61 -> 11 ms, DB done -> Notify.Send 108 -> 0.1 ms. The tracking
queue stays (three fewer fsyncs per call, batched). Docs corrected to attribute the cost
to the flush wait, with the trap recorded in CLAUDE.md and Component-SiteRuntime.md.
Tests: ForwardAsync returns while the audit commit is still pending (never-completing
writer task) and still records tracking; a late commit fault is swallowed.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
A Flow Debugger live run (execution 1130c01f) spent 108 of a 176 ms site script run
between the cached DB write completing and Notify.Send starting: the immediate-
completion telemetry awaited two autocommit SQLite transactions on the consolidated
LocalDb file (RecordAttemptAsync + RecordTerminalAsync), on the script thread, each a
WAL fsync on a Docker volume; the submit row was a third. The audit half of the same
forwarder was already a channel write, so the tracking half was the whole cost.
OperationTrackingStore is now a single-writer queue: Record*Async enqueue an ordered
work item and return; one background pump drains the bounded channel (8192, wait
backpressure - tracking never drops) and applies each batch (<=256) in ONE transaction,
falling back to per-row apply when a batch fails so a poison row cannot take its
neighbours. Preserved invariants: FIFO across enqueue/attempt/terminal; read-your-
writes (GetStatusAsync and ReadChangedSinceAsync barrier on every write queued before
them, skipped when idle) so Tracking.Status, the telemetry actor's status join and
central's PullSiteCalls never see a stale row; timestamps stamped when recorded;
PurgeTerminalAsync returns after the purge lands; DisposeAsync drains the queue before
closing; sync Dispose keeps its documented in-flight-writes-fail semantics. New public
FlushAsync() is the explicit barrier. Accepted: a queued-but-unapplied row is lost on a
hard crash (ms window); S&F buffer and audit rows are unaffected.
No interface, forwarder or ScriptRuntimeContext change. Tests: write returns while
another connection holds the DB write lock and lands once released (fails on the old
store: 31 s); reads see 100 ordered attempts + terminal; DisposeAsync drains 200
queued rows; write after dispose throws ObjectDisposed. Existing tests that poke the
SQLite file right after a store call now FlushAsync first.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Found in a browser Live run on the rig: the run drew API, DB and Notify nodes from
its inline rows; the 5 s pending-refresh re-read the tree through Replay before the
site's cached-call telemetry (12 s) had reached central, swapped the graph wholesale,
and the resolved DB node vanished (7 -> 6 steps) until a manual refresh. Polling had
already stopped because nothing was Pending.
MergeRefreshedGraph now unions the re-read rows into the run's known rows by EventId
(fresh copy wins), stores the union back on _raw and rebuilds through
FlowGraphBuilder, so a refresh only ever adds nodes. The union is exact: the inline
rows are the same AuditEvent objects the site writer persists (FlowTraceCollector).
Tests: new PendingRefresh_KeepsStepsTheLiveRunAlreadyHad_WhenReplayLagsBehindTelemetry
(fails on the old code). Two existing fixtures were internally inconsistent and only
passed under the old replace semantics: ParkedThenResolved built its Resolved outcome
with the Parked raw rows, and OutboundOutcome minted fresh event/correlation ids per
call, so no row could ever match across live and replay. Both now share rows per run.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
An ephemeral inbound method (created and deleted through the CLI, with a never-taken
Route.To branch so prediction resolves a route target) is run live in a real browser through
the fixture's /auth/login; the reload test drives the OnAfterRenderAsync localStorage restore
and then proves the page is still interactive. Both fail on the pre-fix image at the
'Predicted' wait (the circuit deadlock) and pass on the fixed one.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
The page wires FlowStepDetail's OnRetry/OnDiscard to IFlowDebugService.RetryOutboundAsync /
DiscardOutboundAsync behind the same site-scope re-check and confirm wording the Site Calls /
Notification report pages use, toasts the relay outcome, and on success re-polls Replay (same
5 s / 2 min loop) until the acted node leaves Parked. Also drops a duplicated ?correlationId=
test block that a concurrent index-only commit left in QueryStringDrillInTests.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
FlowDebugService.PredictMethodAsync resolved route targets with a sync-over-async
GetAwaiter().GetResult() inside the predictor's synchronous resolver — on the circuit's
single-threaded renderer sync context that blocked the dispatcher forever the moment a
method whose script routes to a site script was chosen (every later click and JS-interop
reply queued behind it; the page looked inert with a healthy websocket and no error).
Misdiagnosed on 2026-09-06 as a login-disabled localStorage stall. Now a resolve-and-rewalk
loop: each pass records the targets it could not answer, they are awaited between passes,
and the walk converges once a pass asks for nothing new (bounded by MaxPredictionPasses).
Regression test drives the service under a single-thread pump sync context (fails in 10 s
on the old code, passes now) plus a nested Scripts.Call resolve-across-passes test.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Rig run 2026-09-06 (image built from this branch, containers recreated from
main's compose file): full drill A-H YES (active-central kill takeover 27 s,
rejoin as standby in 3 s), then with FLOW_ONLY=1:
H live+replay -> YES
live: YES root=8e628d0a-428b-48df-b937-f4c2ac631fc3 events=8 api=Delivered db=Delivered notify=Submitted
replay: YES root=8e628d0a-428b-48df-b937-f4c2ac631fc3 events=10 api=Delivered db=Delivered notify=Delivered
I outbound live+replay -> YES
Two rig facts the drill established: (1) with the DB reachable the site's
immediate-completion path resolves a CachedWrite INSIDE the run, so the DbCall
node is already Delivered inline (Submit/Attempted/Resolve share a tick — the
check now ranks statuses instead of taking list order); (2) cached-call
telemetry reaches central on its own ~20-30 s cadence, later than the 5 s audit
push, so replay needs the bounded retry. New FLOW_ONLY=1 switch runs H + I
without the notifier build or the docker kill.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
After a successful Live run whose graph has Pending outbound steps the page
re-reads the tree through the existing Replay command every 5 s for up to
2 min (page constants), merging by replacing the graph and keeping the
selection by event id; never in Replay mode; cancelled by a new run, a mode
switch or Dispose. Status strip gains a Pending count, a 'Refresh outcomes'
button (both modes) and an 'outcomes still arriving' note while polling. The
run's own timing and the Completed chip are left alone — pending outbound
work is not a failed run. Plan 2026-09-06 D5.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
A folded outbound step shows a Lifecycle tab (time · kind · status · node ·
error, oldest first; a Parked step adds the 'retry or discard from Site Calls /
Notification report' hint), replaces the never-populated Writes tab for those
steps, labels the Input tab Request / SQL + parameters / Subject + body by kind,
and 'Where this ran' names the System / Connection / List plus 'Delivered by'
(the terminal row's node). Pending/Parked outcome badges.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Four lanes (LaneCount 4; External sublabel = distinct targets, three then +n);
ApiCall/DbCall/Notify nodes carry a status pill (Ok/Pending/Parked/Failed —
pending pulses, reduced-motion aware) plus a sync/cached/notify mode; folded
lifecycles show '+4.1 s' submit→terminal wall time. Async edges are dashed
and routed through the inter-lane strip with a gutter tag; spawn edges use a
bolt marker. Predicted async edges (Task 6's PredictedEdge.Async) render the
same way. de-DE height pinned at 319.2 = (4×110+16)×0.7.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
FlowShapePredictor emits ExternalSystem.Call/CachedCall → ApiCall ('{s}.{m}'),
Database.Execute/Query/QuerySingle/CachedWrite → DbCall ('{conn} · {member}'),
Notify.Send(list) and Notify.To(list).Send() → Notify ('Notify {list}'), all on
FlowLane.External; Async-suffixed spellings map identically. PredictedEdge gains
an additive Async flag; an async edge (cached call, notify) does not advance the
chain, mirroring FlowGraphBuilder.IsAsync (D4/D8). Database.Connection() raw
handles no longer predict a node (command count is not statically knowable).
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Joseph Doherty