Replaces the value-change (site/instance/attribute) entry with a script picker.
ValueChange/Conditional scripts keep the write-and-poll path with the attribute
derived from the trigger config; every other kind is invoked directly through the
traced RouteToCall path with a parameter form from the script's ParameterDefinitions.
New additive FlowDebugRunScriptCommand; run-value-change stays as the CLI escape hatch.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Code review of the merged arch-review remediation (Tasks 1-13) found four items.
1. FlowDebugRunner returned its OWN synthesized rows un-redacted. RunMethodAsync's
inbound row and RunValueChangeAsync's root row were handed to WriteCentralAsync
(whose writer redacts a private copy) and then put RAW into the returned Events —
so a Live run's caller saw the unfiltered row while Replay of the same execution
showed the redacted one, defeating F1's "Live shows what Replay reads back". Both
methods now resolve IAuditRedactor once per run and place the filtered copy into
the result (including the early-return path on a failed attribute write); the raw
row still reaches the writer, which redacts itself.
2. FlowGraphBuilder's spawner search degenerated to "the last step of the parent".
Since executions are ordered as contiguous blocks, position[p] < childPosition
holds for EVERY parent step, so a run shaped ScriptRun -> Write(A) -> Wait ->
Write(B) whose Write(A) fired the cascade drew the Spawn edge from Write(B). New
rule, in order: (a) the last non-async, non-Response parent step occurring at or
before the child's first step — parent and child of a trigger cascade share the
site clock, so this picks the write that actually fired it; (b) the position rule
as fallback, which is the mixed-clock case F5 was written for (a child clock
behind its parent's, where no parent step satisfies (a)); (c) parentChain[0].
The class remark's stale "Replay order is unchanged" claim is replaced with what
actually happens to a central Database.* call made after Route.To().Call.
3. FlowTraceCollector's byte budget counts UTF-16 chars of DetailsJson only, so
"well under the 4 MB gRPC default" overstated it. Documented honestly: ~1 MB of
UTF-8 for ASCII payloads, ~3 MB worst case for non-ASCII, both under 4 MB before
framing — and raise it only against a real serialized-size measurement.
4. CLI --timeout-seconds help + README row now say the management endpoint caps
flow-debug run-method at ManagementService:LongRunningCommandTimeout (default
5 min), so a larger client value still 504s at the server's ceiling.
Deferred-work register rows 112 (site-side inline trace truncation is invisible to
the operator — needs a proto field) and 113 (the F2 method-entry site scope gates
routed traffic only; ApiMethods have no site affinity to scope against) record what
this review deliberately did not fix.
Tests: FlowGraphBuilderTests gains the same-clock Write(A)/Wait/Write(B) cascade
(red before the rule change, and the clock-behind test still passes on rule (b));
FlowDebugRunnerTests extends the redactor test to the inbound row and adds two
value-change cases (settled poll, failed write). Commons 793, ManagementService 615,
CentralUI 1264, CLI 462 — all green.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
The Flow Debugger design doc still described the pre-remediation shape: a
debug-stream subscription for the value-change entry, Deployer everywhere, an
unredacted inline trace copy and "~7 audit rows per call". Reconciled with what
shipped on this branch (Tasks 1-12):
- design D6 / §5.4 / §9 / §10: RequireFlowDebugger page policy with Deployer-only
execution (F3), the PullTraceEvents polling path with AttributeValueUpdate
fields 7/8 recorded as RESERVED and unconsumed (F12), the redacting + byte-
capped shared FlowTraceCollector (F1), route-time site scope (F2), the derived
client waits (F4), the 7-10 row estimate, the doubled central audit write per
routed call, and the inbound-row-after-the-run ordering.
- sitestream.proto: fields 7/8 marked reserved-but-populated, numbers not to be
reused (comment only; no regeneration needed).
- Component-InboundAPI: SkipBodyCapture on the RouteCall row (F11),
SchemaRefResolver's public visibility, WithSiteScope / SITE_SCOPE_DENIED (F2),
Replay's wider role set (F3).
- Component-CentralUI: page policy vs Deployer-only controls (F3), route-time
site scope (F2), client wait derivation (F4).
- "7-10 audit rows per call (3 of them trace-only)" everywhere the estimate is
quoted: the method form, both CLI --trace-all-calls help strings, the CLI
README, the ApiMethod entity doc, the manual migration script and the two
Flow Debugger plans.
- arch-review doc: new §7 Resolution listing F1-F13 with the fixing commit, plus
the tracking-store test race fixed alongside them (86451382).
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Central rows carry central's clock, site rows the site's, so the global
OccurredAtUtc sort let a 1-2 s skew float a whole site execution ahead of the
inbound row / RouteCall that started it, and pushed the trigger-cascade spawner
search onto its parentChain[0] fallback.
FlowGraphBuilder.Build now orders executions by the input index of their
earliest-given row and orders steps by timestamp only INSIDE one execution
(one clock), breaking ties by kind rank then input index. A folded outbound
node keeps its earliest-by-time row's identity, which need not be the row given
first, so its index is the minimum over the rows folded into it. The spawner
search uses position in the ordered list rather than a raw timestamp compare.
FlowDebugRunner.RunMethodAsync keeps the collector's emission order instead of
re-sorting; the value-change path keeps its OrderBy (all rows share one site
clock). Replay input arrives timestamp-sorted from the repository, so its order
is unchanged.
The DelmiaShaped test fixture is still shuffled within each execution (the
inside-one-execution timestamp rule is unchanged) but now lists the executions
in emission order, since that is what fixes their relative order.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
The single-writer queue (d91135f9) made RecordEnqueueAsync return on enqueue, so the
test's verifier-connection UPDATE raced the pump and touched zero rows whenever the pump
lost. Every other verifier-connection test already flushed; this one was missed.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
A traced run refused before launch — by the per-script in-flight cap
(MaxConcurrentRunsPerScript) or by MinTimeBetweenRuns — emitted no ScriptTrace row at
all, so the Flow Debugger reported NO_TRIGGER_WITHIN_WINDOW ("no value-change trigger on
this attribute, or no device echo"), which is wrong in both cases and sends the operator
to check the wrong things.
Both guards now leave a Failed ScriptTrace/ScriptRun row naming the guard that refused
the run, parented to the writing execution and carrying the trigger detail. The new
ScriptRunLauncher.EmitSuppressedRun reuses EmitScriptRun with elapsedMs 0 and no
collector (a shed run has no reply to ride on — the row reaches the debugger via
PullTraceEvents / Replay), resolving the writer and identity providers from a fresh
scope because it runs on the actor thread. Best-effort, never throws, no-op untraced.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
RouteHelper/RouteTarget carry a new captureBodies flag (default true) threaded
through every With* copy, same shape as the site-scope field Task 3 added.
InboundScriptExecutor resolves IOptionsMonitor<AuditLogOptions> from the
per-execution DI scope and looks up PerTargetOverrides[method.Name]?.SkipBodyCapture
exactly as AuditWriteMiddleware.cs:183-187 does for the inbound row, then calls
.WithBodyCapture(!skip) when building the script's Route helper. When set, the
always-on ScriptTrace.RouteCall row omits its request/response summary, so an
operator who suppressed a method's body from the audit log doesn't get it back
via the routed call's arguments (derived from that same body).
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
RunMethodAsync collected sink events into a plain ConcurrentQueue and always
reported Truncated: false, so the Live result carried RAW audit events — the
per-target SQL-parameter overrides, header scrubbing and 8 KB / 64 KB payload
caps that Replay reads back never applied to what the operator saw, and the
reply had no upper bound at all.
It now collects through the same FlowTraceCollector the site launcher uses,
constructed with the host's IAuditRedactor (AddAuditLog's singleton; a host
without one collects raw but is still capped by count and bytes), and reports
Truncated from the collector. Complete() is called as soon as the run returns,
so a late un-awaited call cannot mutate the snapshot the reply is built from.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Every emit site hands FlowTraceCollector the same AuditEvent it hands the
writer, but redaction and payload capping lived inside the writers, which
return a NEW filtered record — so the inline copy that rides
ScriptCallResult.TraceEvents / RouteToCallResponse.TraceEvents was the RAW
event: no per-target SQL-parameter redaction, no header scrubbing, no 8 KB /
64 KB truncation. Live and Replay of the same execution could therefore show
different payloads, and the 500-event-only cap left the reply free to breach
the 4 MB gRPC receive limit — which would fail a real call, since the same
reply carries the script's return value.
The collector now applies the host IAuditRedactor itself (outside the lock;
the writers keep their own idempotent Apply) and enforces a byte budget
(DefaultMaxBytes = 1 MB over the redacted DetailsJson) alongside the event
cap, flagging Truncated when either trips. It moves to Commons
(Types/Debugging) so the central FlowDebugRunner can share it.
The redactor is a singleton, so ScriptRunLauncher resolves it from the ROOT
provider and keeps constructing the collector before the try — the traced
ScriptRun row is emitted from the catch blocks too, including the
shed-at-dequeue path that throws before the DI scope exists.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
A method-entry Live run had no site-scope check at all. RunValueChangeAsync
calls EnforceSiteScope on its named instance, but a method run has no single
target site up front — the operator picks a METHOD, and where its script routes
is only known once it runs — so a site-scoped Deployer could reach any site the
method happened to route to.
The scope is therefore pushed into RouteHelper and enforced at route time:
- RouteHelper.WithSiteScope(IReadOnlySet<string>?) carries a set of site
IDENTIFIERS (null = unscoped, the state of every ordinary inbound HTTP
request; an EMPTY set permits nothing, the fail-closed outcome for a scoped
user whose sites no longer exist). Threaded through every With* copy and To().
- The gate lives in RouteTarget.ResolveSiteAsync — the single choke point that
Call, GetAttributes, WaitForAttribute and SetAttributes all pass through — so
it fires BEFORE any site traffic and throws the new RouteSiteScopeException.
- InboundScriptExecutor maps it to error code SITE_SCOPE_DENIED. The type is
InboundAPI's own (that assembly cannot reference ManagementService), and it
derives from InvalidOperationException so base-type callers are unaffected.
- FlowDebugRunner.RunMethodAsync resolves the user's PermittedSiteIds (numeric)
to identifiers via ISiteRepository and converts SITE_SCOPE_DENIED back to
SiteScopeViolationException (-> ManagementUnauthorized), AFTER writing the
inbound audit row so a denied attempt is still recorded against the operator.
Administrators and unscoped users get a null scope and are unaffected, exactly
as EnforceSiteScope treats them. Every contract change is a trailing optional
parameter. Mutation-checked: disabling the gate fails 3 of the 5 new RouteHelper
tests and 1 of the runner tests, while the in-scope/unscoped positives still pass.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Design D6 / §9 says a Live run is a Deployer act while Replay needs only
OperationalAudit, but the page and all three FlowDebug* commands were gated on
Deployer. The Audit Log and Execution Chain pages -- themselves OperationalAudit
-- show an "Open in Flow Debugger" drill-in that an auditor could not follow.
Implements the split:
- New `RequireFlowDebugger` policy over `FlowDebuggerAccessRoles`
{Deployer, Administrator, Viewer}; the page now carries it instead of
`RequireDeployment`.
- `FlowDebugReplayCommand` takes the same role set in the ManagementActor
matrix; `FlowDebugRunMethodCommand` / `FlowDebugRunValueChangeCommand` stay
DeployerOnly.
- The page resolves `_canLive` (Deployer) in OnInitializedAsync and, without it,
pins the mode to Replay, hides the entry-point and mode toggles, the Run
button and the Live pickers, honours only the `?executionId=` drill-in, and
leaves the parked-node Retry/Discard callbacks unset so those buttons never
render. `CanRun`, `SetMode(Live)` and `ActOnStepAsync` repeat the check so the
split does not rest on markup alone.
The NavMenu entry stays inside the Deployer-gated Deployment section -- auditors
arrive through the audit drill-ins, which is the point of the finding.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
GetTraceEventsAsync re-implemented GetExecutionTreeAsync's execution-chain walk
as a second copy: the upward ParentExecutionId climb, the root MIN(OccurredAtUtc)
window, and the recursive Edges/Chain CTE with its DISTINCT-edges trick and
MAXRECURSION cap. Two walkers that had to stay in step on window slack, depth and
the edge distillation.
Extracted into three shared members:
- ChainAnchor (root execution id + root's first event timestamp, with the
Edges window predicate derived from it)
- ResolveChainAnchorAsync — phase 1 up-walk plus the root window read
- ChainCtePrefix / AddChainParameters — the Edges/Chain CTE prefix and its
@root/@winStart/@winEnd binding
Each walker appends its own projection to the prefix: GetExecutionTreeAsync the
ChainIds CTE plus the per-execution aggregate, GetTraceEventsAsync the bare id
union. Both keep OPTION (MAXRECURSION 32).
Pure refactor. The tree walker's generated SQL is byte-identical (verified by
diffing the composed command text against the pre-change literal); the trace
walker's differs only by the SQL comment lines it now inherits from the shared
prefix. Behaviour is pinned by the existing MSSQL-backed tests
(GetExecutionTree_MultiLevelChain / _StubParentNode / _RespectsMaxRecursion,
ExecutionTree_Bounds_EdgeScan_To_Root_Window, and both
AuditLogRepositoryTraceEventsTests) — 410 passed / 0 skipped before and after.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
The Flow Debugger trace arm was recorded when a traced write was ISSUED, not
when the DCL acknowledged it. A write the device rejected (or that timed out)
left the arm standing for the whole TraceArmWindowSeconds (30 s), so the next
unrelated subscription update on that attribute was stamped with the debugger's
execution id and spawned a traced, parented run — the debugger drew a "device
echo" of a write that never landed.
The write's continuation runs off the actor's dispatcher, so it cannot touch
_traceArms directly; it Tells self a new DisarmTrace message instead, handled in
a Receive<>. The disarm is keyed on the writer's execution id, so a late failure
cannot steal an arm a newer traced write has since taken.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Full review of the Flow Debugger commit range (v1.0, v1.1 outbound calls, v1.2 follow-ups,
outcome-merge fix, two perf commits). Also records the exact 11 role-negative Playwright
tests that cannot pass on the login-disabled rig in the cluster-ops skill.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Measuring the tracking-store queue on the rig moved nothing: the DB-done -> Notify.Send
gap stayed at 108 ms. The cost was the audit half of CachedCallTelemetryForwarder:
SqliteAuditWriter.WriteAsync enqueues synchronously but returns the BATCH completion,
and the writer loop coalesces for FlushIntervalMs (50 ms) before it commits, so each
awaited audit row cost ~50 ms + fsync and a cached call awaited three (submit,
attempted, resolve) on the script thread. Every other script-path emitter already
fired-and-observed; only this forwarder awaited.
TryEmitAuditAsync now hands the row to the writer and observes the completion on a
fault-logging continuation (the FallbackAuditWriter ring-buffer fallback still runs
inside that task). Best-effort contract unchanged; per-store FIFO unchanged; nothing
joins the audit and tracking stores by commit order.
Rig, FlowOutboundDemo, 5-run medians: inbound 194 -> 25 ms, site script run 176 -> 17 ms,
CachedWrite submit->done 61 -> 11 ms, DB done -> Notify.Send 108 -> 0.1 ms. The tracking
queue stays (three fewer fsyncs per call, batched). Docs corrected to attribute the cost
to the flush wait, with the trap recorded in CLAUDE.md and Component-SiteRuntime.md.
Tests: ForwardAsync returns while the audit commit is still pending (never-completing
writer task) and still records tracking; a late commit fault is swallowed.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
A Flow Debugger live run (execution 1130c01f) spent 108 of a 176 ms site script run
between the cached DB write completing and Notify.Send starting: the immediate-
completion telemetry awaited two autocommit SQLite transactions on the consolidated
LocalDb file (RecordAttemptAsync + RecordTerminalAsync), on the script thread, each a
WAL fsync on a Docker volume; the submit row was a third. The audit half of the same
forwarder was already a channel write, so the tracking half was the whole cost.
OperationTrackingStore is now a single-writer queue: Record*Async enqueue an ordered
work item and return; one background pump drains the bounded channel (8192, wait
backpressure - tracking never drops) and applies each batch (<=256) in ONE transaction,
falling back to per-row apply when a batch fails so a poison row cannot take its
neighbours. Preserved invariants: FIFO across enqueue/attempt/terminal; read-your-
writes (GetStatusAsync and ReadChangedSinceAsync barrier on every write queued before
them, skipped when idle) so Tracking.Status, the telemetry actor's status join and
central's PullSiteCalls never see a stale row; timestamps stamped when recorded;
PurgeTerminalAsync returns after the purge lands; DisposeAsync drains the queue before
closing; sync Dispose keeps its documented in-flight-writes-fail semantics. New public
FlushAsync() is the explicit barrier. Accepted: a queued-but-unapplied row is lost on a
hard crash (ms window); S&F buffer and audit rows are unaffected.
No interface, forwarder or ScriptRuntimeContext change. Tests: write returns while
another connection holds the DB write lock and lands once released (fails on the old
store: 31 s); reads see 100 ordered attempts + terminal; DisposeAsync drains 200
queued rows; write after dispose throws ObjectDisposed. Existing tests that poke the
SQLite file right after a store call now FlushAsync first.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Found in a browser Live run on the rig: the run drew API, DB and Notify nodes from
its inline rows; the 5 s pending-refresh re-read the tree through Replay before the
site's cached-call telemetry (12 s) had reached central, swapped the graph wholesale,
and the resolved DB node vanished (7 -> 6 steps) until a manual refresh. Polling had
already stopped because nothing was Pending.
MergeRefreshedGraph now unions the re-read rows into the run's known rows by EventId
(fresh copy wins), stores the union back on _raw and rebuilds through
FlowGraphBuilder, so a refresh only ever adds nodes. The union is exact: the inline
rows are the same AuditEvent objects the site writer persists (FlowTraceCollector).
Tests: new PendingRefresh_KeepsStepsTheLiveRunAlreadyHad_WhenReplayLagsBehindTelemetry
(fails on the old code). Two existing fixtures were internally inconsistent and only
passed under the old replace semantics: ParkedThenResolved built its Resolved outcome
with the Parked raw rows, and OutboundOutcome minted fresh event/correlation ids per
call, so no row could ever match across live and replay. Both now share rows per run.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Rig-only posture, set in the tracked docker/central-node-{a,b}/appsettings.Central.json
so it survives deploy.sh / container recreates (it used to be a throwaway compose
override on central-a alone). Both nodes so Traefik on :9000 is login-free
regardless of which node is active. POST /auth/login still answers 302, so the
Playwright fixture login is unaffected; LoginTests will fail on this rig by design.
Documented in the scadabridge-cluster-ops skill.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Notify.Send mints NotificationId as Guid "N" (no dashes) and the outbox parses it back into
the audit row's CorrelationId, so the Flow Debugger's Notify node linked and relayed a "D"
string the Notifications table never matches (found live: the report opened empty). The
link and RetryNotificationRequest now use "N", and the Notification report normalises any
Guid-shaped ?correlationId= to "N" so Audit Log and Flow Debugger callers both land.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
An ephemeral inbound method (created and deleted through the CLI, with a never-taken
Route.To branch so prediction resolves a route target) is run live in a real browser through
the fixture's /auth/login; the reload test drives the OnAfterRenderAsync localStorage restore
and then proves the page is still interactive. Both fail on the pre-fix image at the
'Predicted' wait (the circuit deadlock) and pass on the fixed one.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
The page wires FlowStepDetail's OnRetry/OnDiscard to IFlowDebugService.RetryOutboundAsync /
DiscardOutboundAsync behind the same site-scope re-check and confirm wording the Site Calls /
Notification report pages use, toasts the relay outcome, and on success re-polls Replay (same
5 s / 2 min loop) until the acted node leaves Parked. Also drops a duplicated ?correlationId=
test block that a concurrent index-only commit left in QueryStringDrillInTests.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Joseph Doherty