The Flow Debugger design doc still described the pre-remediation shape: a
debug-stream subscription for the value-change entry, Deployer everywhere, an
unredacted inline trace copy and "~7 audit rows per call". Reconciled with what
shipped on this branch (Tasks 1-12):
- design D6 / §5.4 / §9 / §10: RequireFlowDebugger page policy with Deployer-only
execution (F3), the PullTraceEvents polling path with AttributeValueUpdate
fields 7/8 recorded as RESERVED and unconsumed (F12), the redacting + byte-
capped shared FlowTraceCollector (F1), route-time site scope (F2), the derived
client waits (F4), the 7-10 row estimate, the doubled central audit write per
routed call, and the inbound-row-after-the-run ordering.
- sitestream.proto: fields 7/8 marked reserved-but-populated, numbers not to be
reused (comment only; no regeneration needed).
- Component-InboundAPI: SkipBodyCapture on the RouteCall row (F11),
SchemaRefResolver's public visibility, WithSiteScope / SITE_SCOPE_DENIED (F2),
Replay's wider role set (F3).
- Component-CentralUI: page policy vs Deployer-only controls (F3), route-time
site scope (F2), client wait derivation (F4).
- "7-10 audit rows per call (3 of them trace-only)" everywhere the estimate is
quoted: the method form, both CLI --trace-all-calls help strings, the CLI
README, the ApiMethod entity doc, the manual migration script and the two
Flow Debugger plans.
- arch-review doc: new §7 Resolution listing F1-F13 with the fixing commit, plus
the tracking-store test race fixed alongside them (86451382).
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Full review of the Flow Debugger commit range (v1.0, v1.1 outbound calls, v1.2 follow-ups,
outcome-merge fix, two perf commits). Also records the exact 11 role-negative Playwright
tests that cannot pass on the login-disabled rig in the cluster-ops skill.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Measuring the tracking-store queue on the rig moved nothing: the DB-done -> Notify.Send
gap stayed at 108 ms. The cost was the audit half of CachedCallTelemetryForwarder:
SqliteAuditWriter.WriteAsync enqueues synchronously but returns the BATCH completion,
and the writer loop coalesces for FlushIntervalMs (50 ms) before it commits, so each
awaited audit row cost ~50 ms + fsync and a cached call awaited three (submit,
attempted, resolve) on the script thread. Every other script-path emitter already
fired-and-observed; only this forwarder awaited.
TryEmitAuditAsync now hands the row to the writer and observes the completion on a
fault-logging continuation (the FallbackAuditWriter ring-buffer fallback still runs
inside that task). Best-effort contract unchanged; per-store FIFO unchanged; nothing
joins the audit and tracking stores by commit order.
Rig, FlowOutboundDemo, 5-run medians: inbound 194 -> 25 ms, site script run 176 -> 17 ms,
CachedWrite submit->done 61 -> 11 ms, DB done -> Notify.Send 108 -> 0.1 ms. The tracking
queue stays (three fewer fsyncs per call, batched). Docs corrected to attribute the cost
to the flush wait, with the trap recorded in CLAUDE.md and Component-SiteRuntime.md.
Tests: ForwardAsync returns while the audit commit is still pending (never-completing
writer task) and still records tracking; a late commit fault is swallowed.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
A Flow Debugger live run (execution 1130c01f) spent 108 of a 176 ms site script run
between the cached DB write completing and Notify.Send starting: the immediate-
completion telemetry awaited two autocommit SQLite transactions on the consolidated
LocalDb file (RecordAttemptAsync + RecordTerminalAsync), on the script thread, each a
WAL fsync on a Docker volume; the submit row was a third. The audit half of the same
forwarder was already a channel write, so the tracking half was the whole cost.
OperationTrackingStore is now a single-writer queue: Record*Async enqueue an ordered
work item and return; one background pump drains the bounded channel (8192, wait
backpressure - tracking never drops) and applies each batch (<=256) in ONE transaction,
falling back to per-row apply when a batch fails so a poison row cannot take its
neighbours. Preserved invariants: FIFO across enqueue/attempt/terminal; read-your-
writes (GetStatusAsync and ReadChangedSinceAsync barrier on every write queued before
them, skipped when idle) so Tracking.Status, the telemetry actor's status join and
central's PullSiteCalls never see a stale row; timestamps stamped when recorded;
PurgeTerminalAsync returns after the purge lands; DisposeAsync drains the queue before
closing; sync Dispose keeps its documented in-flight-writes-fail semantics. New public
FlushAsync() is the explicit barrier. Accepted: a queued-but-unapplied row is lost on a
hard crash (ms window); S&F buffer and audit rows are unaffected.
No interface, forwarder or ScriptRuntimeContext change. Tests: write returns while
another connection holds the DB write lock and lands once released (fails on the old
store: 31 s); reads see 100 ordered attempts + terminal; DisposeAsync drains 200
queued rows; write after dispose throws ObjectDisposed. Existing tests that poke the
SQLite file right after a store call now FlushAsync first.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
An ephemeral inbound method (created and deleted through the CLI, with a never-taken
Route.To branch so prediction resolves a route target) is run live in a real browser through
the fixture's /auth/login; the reload test drives the OnAfterRenderAsync localStorage restore
and then proves the page is still interactive. Both fail on the pre-fix image at the
'Predicted' wait (the circuit deadlock) and pass on the fixed one.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
The page wires FlowStepDetail's OnRetry/OnDiscard to IFlowDebugService.RetryOutboundAsync /
DiscardOutboundAsync behind the same site-scope re-check and confirm wording the Site Calls /
Notification report pages use, toasts the relay outcome, and on success re-polls Replay (same
5 s / 2 min loop) until the acted node leaves Parked. Also drops a duplicated ?correlationId=
test block that a concurrent index-only commit left in QueryStringDrillInTests.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
FlowDebugService.PredictMethodAsync resolved route targets with a sync-over-async
GetAwaiter().GetResult() inside the predictor's synchronous resolver — on the circuit's
single-threaded renderer sync context that blocked the dispatcher forever the moment a
method whose script routes to a site script was chosen (every later click and JS-interop
reply queued behind it; the page looked inert with a healthy websocket and no error).
Misdiagnosed on 2026-09-06 as a login-disabled localStorage stall. Now a resolve-and-rewalk
loop: each pass records the targets it could not answer, they are awaited between passes,
and the walk converges once a pass asks for nothing new (bounded by MaxPredictionPasses).
Regression test drives the service under a single-thread pump sync context (fails in 10 s
on the old code, passes now) plus a nested Scripts.Call resolve-across-passes test.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Rig run 2026-09-06 (image built from this branch, containers recreated from
main's compose file): full drill A-H YES (active-central kill takeover 27 s,
rejoin as standby in 3 s), then with FLOW_ONLY=1:
H live+replay -> YES
live: YES root=8e628d0a-428b-48df-b937-f4c2ac631fc3 events=8 api=Delivered db=Delivered notify=Submitted
replay: YES root=8e628d0a-428b-48df-b937-f4c2ac631fc3 events=10 api=Delivered db=Delivered notify=Delivered
I outbound live+replay -> YES
Two rig facts the drill established: (1) with the DB reachable the site's
immediate-completion path resolves a CachedWrite INSIDE the run, so the DbCall
node is already Delivered inline (Submit/Attempted/Resolve share a tick — the
check now ranks statuses instead of taking list order); (2) cached-call
telemetry reaches central on its own ~20-30 s cadence, later than the 5 s audit
push, so replay needs the bounded retry. New FLOW_ONLY=1 switch runs H + I
without the notifier build or the docker kill.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
After a successful Live run whose graph has Pending outbound steps the page
re-reads the tree through the existing Replay command every 5 s for up to
2 min (page constants), merging by replacing the graph and keeping the
selection by event id; never in Replay mode; cancelled by a new run, a mode
switch or Dispose. Status strip gains a Pending count, a 'Refresh outcomes'
button (both modes) and an 'outcomes still arriving' note while polling. The
run's own timing and the Completed chip are left alone — pending outbound
work is not a failed run. Plan 2026-09-06 D5.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
A folded outbound step shows a Lifecycle tab (time · kind · status · node ·
error, oldest first; a Parked step adds the 'retry or discard from Site Calls /
Notification report' hint), replaces the never-populated Writes tab for those
steps, labels the Input tab Request / SQL + parameters / Subject + body by kind,
and 'Where this ran' names the System / Connection / List plus 'Delivered by'
(the terminal row's node). Pending/Parked outcome badges.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Four lanes (LaneCount 4; External sublabel = distinct targets, three then +n);
ApiCall/DbCall/Notify nodes carry a status pill (Ok/Pending/Parked/Failed —
pending pulses, reduced-motion aware) plus a sync/cached/notify mode; folded
lifecycles show '+4.1 s' submit→terminal wall time. Async edges are dashed
and routed through the inter-lane strip with a gutter tag; spawn edges use a
bolt marker. Predicted async edges (Task 6's PredictedEdge.Async) render the
same way. de-DE height pinned at 319.2 = (4×110+16)×0.7.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
FlowShapePredictor emits ExternalSystem.Call/CachedCall → ApiCall ('{s}.{m}'),
Database.Execute/Query/QuerySingle/CachedWrite → DbCall ('{conn} · {member}'),
Notify.Send(list) and Notify.To(list).Send() → Notify ('Notify {list}'), all on
FlowLane.External; Async-suffixed spellings map identically. PredictedEdge gains
an additive Async flag; an async edge (cached call, notify) does not advance the
chain, mirroring FlowGraphBuilder.IsAsync (D4/D8). Database.Connection() raw
handles no longer predict a node (command count is not statically knowable).
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
GetTraceEventsAsync widens its channel filter to ScriptTrace ∪ ApiOutbound ∪
DbOutbound ∪ Notification ∪ (ApiInbound & InboundRequest) (plan D6); SecuredWrite,
Cluster and InboundAuthFailure rows in the same tree stay excluded. The site
SqliteAuditWriter.ReadByExecutionIdsAsync already had no channel predicate — pinned
by a test. Interface doc updated to match.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Fold rule: outbound rows sharing (kind, CorrelationId) become one FlowStep with
a Lifecycle list — first row's identity/time/input, latest row's outcome/
subtitle/output/error, DurationMs = sync row's own or submit→terminal wall time
(null while pending). Chain skips async steps (Notify, or first row
CachedSubmit) and links them with Async edges from the preceding chain step.
Spawner edges from a non-RouteCall are Spawn: true. The Response links from the
last non-async step of the inbound execution (the RouteCall), not the globally
last cascaded step (D9). FlowGraphView places a Response at max column + 1 so
the D9 origin does not pull it left (pulled forward from Task 7 to keep the
layout test green).
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
ResolveKind maps ApiOutbound/DbOutbound/Notification to ApiCall/DbCall/Notify;
ResolveLane puts the three on FlowLane.External; outbound outcomes map
Delivered→Ok, Failed→Failed, Parked/Discarded→Parked, else Pending; subtitles
'{status} · sync|cached', '{op} · n rows' from a DbWrite Extra, '{status}' for
Notify. New FlowStepProjector.IsOutbound helper.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Gap verified 2026-09-06: the projector and predictor already know the three outbound
channels, but the live FlowTraceCollector is fed only by EmitTrace (writes/waits) and
the replay query filters to ScriptTrace + the inbound row, so a run leaves every
predicted outbound node as a ghost. Central method-script Database.* calls are not
audited at all.
Plan (11 tasks): three step kinds ApiCall/DbCall/Notify on a fourth External lane;
one node per operation with the cached/notify lifecycle folded in (Pending/Parked
outcomes, Lifecycle tab); sync calls in the chain, async ones on dashed branch edges;
site rows ride the existing collector, replay filter widens, central DB helper gets
DbOutbound rows; the page refreshes pending outcomes via the existing Replay command;
spawn edges styled and Response linked from the route call (the "two parallel paths"
question). Mockup: docs/plans/…-mockup.html, published as a Claude artifact.
Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
Rig recreated from the branch-built image through the main checkout's compose file
(deploy.sh refuses a worktree, register row 68). Fresh key delmia-flowdbg-0906 scoped to
DelmiaRecipeDownload. Full drill: A/B/E/F YES, C/G NO, takeover 29 s, rejoin 3 s, then
H live+replay -> YES
live: 11 events (InboundRequest, RouteCall, ScriptRun, 7x AttributeWrite, AttributeWait)
replay: 17 events on the first attempt — the six extra rows are the simulator's two
cascaded ValueChange runs (2x ScriptRun + 4x AttributeWrite): inline TraceEvents
carry the routed run's own steps, replay walks the whole execution tree.
Recorded as row H in Delmia-Recipe-Download-API.md §8; ledger marks Task 22 completed
(all 24 tasks done). Note: the rig still runs main's appsettings, so the ScriptTrace
30-day retention override lands only after merge + a deploy from main.
Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
Component-CentralUI (new Flow Debugger page section), Component-InboundAPI (TraceAllCalls
property, always-on RouteCall row + Trace flag under Routing Behavior, new 'Flow Debugger
runs' section), Component-SiteRuntime (Instance Actor trace arm; Script Actor traced runs;
CallScript flag inheritance), Component-AuditLog (ScriptTrace channel in Scope, the
PullTraceEvents point-read note + mixed-version caveat under Reconciliation, per-channel
retention example), Component-Communication (§4 trace flag + inline events with proto field
numbers; §8 PullTraceEvents; RPC list), Component-ScriptAnalysis (REQ-SA-7 FlowShapePredictor),
CLAUDE.md UI bullet, deferred-work register row 110 (vertical-trace list toggle + console
capture), design doc status → implemented. CLI README sections landed in Tasks 3/16.
Umbrella index (../scadaproj/CLAUDE.md) checked: its ScadaBridge row records stack/wire
relationships only, unchanged by a new page — no edit.
Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
Flow Debugger (design 2026-09-05 §7.1): FlowDebugRunMethodCommand / FlowDebugRunValueChangeCommand /
FlowDebugReplayCommand on the per-node ManagementActor (Deployer role; the two run commands
are long-running), all delegating to FlowDebugRunner. RunMethod refuses on a standby with
STANDBY_NODE (ManagementCommandException now carries an optional Code that replaces
COMMAND_FAILED), validates parameters through the real ParameterValidator/SchemaRefResolver
(made public), synthesizes the inbound row (actor = operator, Extra.source = flow-debugger)
and runs the method through InboundScriptExecutor with a trace-sink RouteHelper so the reply
carries the RouteCall row + the site's inline events. RunValueChange does a traced routed
write via IInstanceRouter, then polls IPullTraceEventsClient (interface moved to Commons so
ManagementService can consume it) until a ScriptRun parented to the root settles, else
NO_TRIGGER_WITHIN_WINDOW. Replay reads IAuditLogRepository.GetTraceEventsAsync.
Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
Flow Debugger (design 2026-09-05 §5.2): bit NOT NULL DEFAULT 0 column + EF migration
20260906034113 with its docs/plans/sql production twin. CreateApiMethodCommand gains
TraceAllCalls=false; UpdateApiMethodCommand gains bool? TraceAllCalls (null = unchanged so an
older CLI never resets it). Management result echoes the flag; CLI create/update expose
--trace-all-calls; README carries the ~7-rows-per-call warning.
Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
Live + Replay modes (no dry run); entry points = inbound API method and
value-change trigger; data-sourced value-change does a real tag write with a
30 s trace arm on the instance actor for echo correlation; gated trace rows in
a new ScriptTrace audit channel (RouteCall always-on); inline step return on
RouteToCallResponse for Live; Deployer gates the page and Live run.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
The 2026-06-26 design kept the key env-only. The file is now an explicit fallback: SCADABRIDGE_API_KEY (non-blank) -> ScadaBridge:ApiKey (non-blank, trimmed) -> not configured. Env-only stays the recommended production posture; the failure message names both sources.
Claude-Session: https://claude.ai/code/session_013L3XdmVii9i5sY4FChU8pB
Joseph Doherty