ScadaBridge/docs/plans/2026-09-05-flow-debugger-design.md

Flow Debugger — design

Status: implemented 2026-09-06 (branch feat/flow-debugger); plan 2026-09-05-flow-debugger.md (24 tasks, all complete). Residual follow-ups: deferred-work register row 110. v1.1 outbound calls (a fourth External lane for API / DB / notification nodes, lifecycle folding, async/spawn edges, pending refresh) — see the addendum plan 2026-09-06-flow-debugger-outbound-calls.md; §4's "three lanes fixed" is amended by its D2. Mockup: Claude Design canvas Flow Debugger Mockup — https://claude.ai/code/artifact/3f3e81e5-8e5e-4333-bdf6-59256108494d (Direction A swimlane boards + two low-fi alternates + decision notes). Owner components: Central UI (#page), Inbound API, Site Runtime, Audit Log, Communication, Script Analysis, Configuration Database.

1. What it is

A Central UI page, Deployment → Flow Debugger, that runs one entry point with operator-supplied inputs and renders every step the run passes through as a swimlane flowchart: lanes are where code ran (Central, the Site the route landed on, the tag source), nodes are steps (request, script run, route call, attribute write, wait, response) each carrying its input, output, duration and status. Clicking a node opens a tabbed detail panel (Input · Output · Attribute writes · Console · Timing).

Worked example, and the acceptance flow for the whole feature: the Delmia recipe download (docs/api-specs/Delmia-Recipe-Download-API.md §4) — inbound POST /api/DelmiaRecipeDownload → central method script → Route.To(MachineCode).Call("ProcessRecipeDownload") → site Call script on the DelmiaReceiver child → seven attribute writes → wait for Galaxy to raise RecipeProcessedFlag → read result → response.

Decisions taken (2026-09-05)

#	Decision	Rejected
D1	Two modes: Live and Replay. Live is a real run with real values, real writes, audited like any inbound call. Replay executes nothing — it rebuilds the flow from an existing ExecutionId's trace rows.	Dry run (execute scripts but capture writes/ES/DB/Notify instead of sending). Needs an interception seam in the site runtime that does not exist and would fork the semantics of every helper.
D2	Two entry points in v1: an Inbound API method, and a value-change-triggered site script on a named instance. Both may land on a site script on instance X.	Alarm/timer trigger simulation, direct "call script X" — see §11.
D3	Value-change on a data-sourced attribute does a REAL tag write through the instance actor's normal write path; the device echo fires the trigger. Correlation is by a short-lived trace arm on the instance actor (§5.3).	A debug-only "inject value" command that synthesises the change without touching the device — leaves the instance's cached value diverged from the device until the next real update.
D4	Live gets its trace inline; Replay reads audit rows. Site audit telemetry forwards on a 5 s busy / 30 s idle cadence (AuditLog:SiteTelemetry), too slow to drive an interactive run, so the routed reply carries the site steps back and the same steps are also written as audit rows for Replay.	Live waiting on the audit pipeline.
D5	Tracing is gated, not always-on. Sites run scripts at high rates from timers and value changes; a trace flag rides beside ParentExecutionId and only flagged runs emit trace rows. One exception is always-on: a central RouteCall row per Route.To().Call (inbound-driven volume, joins the two halves of a Replay).	Tracing every run (volume), or a purely time-window join.
D6	Live run is a Deployer act. RequireDeployment gates the page and the Run button; Replay needs OperationalAudit.	A separate RequireOperator gate for Live.

2. What exists today (reused, not rebuilt)

• Correlation ids already thread the chain. AuditWriteMiddleware mints an ExecutionId per inbound request; RouteHelper.Call stamps it as RouteToCallRequest.ParentExecutionId; InstanceActor forwards it on ScriptCallRequest.ParentExecutionId; the site run mints its own id (ScriptRuntimeContext._executionId) and records the parent; a static write carries the writer's id (SetStaticAttributeCommand.SourceExecutionId → AttributeValueChanged.SourceExecutionId) into the value-change / alarm cascade. IAuditLogRepository.GetExecutionTreeAsync walks it.
• Audit captures the ends. AuditKind.InboundRequest stores the full request and response (up to InboundMaxBytes); ApiCall/DbWrite/NotifySend rows are per-execution.
• Central already drives live instances. SiteCommandService routing group: RouteToCall, RouteToGetAttributes, RouteToSetAttributes, RouteToWaitForAttribute; the Test Run sandbox's SandboxInstanceGateway runs a central script against a deployed instance over exactly this transport.
• Roslyn compiles every script centrally (ScriptAnalysisService, ScriptTrustValidator's syntax walk), so the predicted graph is a sibling walker, not new infrastructure.
• Attribute writes already have one path. WriteAttributeBatchRequest → InstanceActor.HandleWriteAttributeBatch → WriteTagBatchRequest to the DCL for data-sourced attributes; SetStaticAttributeCommand for static ones.

3. What is missing (the build)

Gap	Why it blocks	Closed by
G1 No record that a site script ran — no AuditKind for a run; per-run site events are opt-in, aggregated, id-less.	Replay would show the inbound row and any outbound leaf rows with the site script itself absent.	§5.1 ScriptRun trace row.
G2 Attribute writes and waits are invisible — WriteAttributeBatchRequest/WaitForAttributeRequest carry no execution id; nothing records outcomes.	The Delmia flow's steps 5–7 cannot be drawn.	§5.1 AttributeWrite/AttributeWait rows + §6 additive fields.
G3 Route calls are not audited centrally and the routed reply does not return the child run's id.	Central and site halves of a Replay cannot be joined without guessing.	§5.1 RouteCall row (always-on) + RouteToCallResponse.ChildExecutionId.
G4 No trace gate.	Always-on would flood AuditLog from timer/value-change scripts.	§5.2 trace flag + per-method TraceAllCalls.
G5 A value-change entry needs a way to fire the trigger and correlate the echo.	Data-sourced echoes arrive as roots.	§5.3 trace arm.
G6 No page, no graph builder, no predictor.	—	§7, §8.
G7 Live runs have no API key, so no actor on the inbound row.	Audit would show an anonymous inbound call.	§9.

4. User experience

Reference the mockup's two full-fidelity boards; this section only records what the mockup fixes.

Toolbar. Entry point (Inbound API method | Value-change trigger) + a picker for the method or the instance + script; Mode (Live | Replay); Edit inputs, Load from audit…, Run.

Status strip. State chip (Not run / Running / Completed / Failed / Timed out), run number, wall-clock and node, total duration, counts (steps, boundaries crossed, attribute writes), root ExecutionId linking to the existing Execution Chain page.

Flow panel. Three lanes fixed in v1: Central, Site ({siteId} · {instance}), Tags ({connection} → {source}). Before a run the graph is predicted (§8) — nodes whose target depends on an input value or on live site state render dashed. After a run every node is real; predicted nodes that did not execute (an early-return branch not taken) render greyed. Node = kind eyebrow + duration, title, one-line in/out summary; the selected node gets the accent border. Connectors: solid = executed edge, accent = crossed a gRPC boundary, dashed = external actor (the tag source responding). Zoom −/+/Fit only; no pan-by-drag in v1.

Detail panel (below the flow, tabbed): Input, Output, Attribute writes (attribute · before · written · quality · ack ms), Console, Timing; a right column Where this ran (site/node, instance, composed child, connection, attribute kind, source line) with Open in Debug View and Audit rows (n) drill-ins.

Inputs form (the Step 1 detail before a run): generated from the method's ParameterDefinitions (type, required) or, for a value-change entry, one field for the trigger attribute's new value plus read-only current value/quality. Fill from last real call pre-fills from the most recent InboundRequest row for that method.

Replay. Load from audit… takes an ExecutionId (or arrives via a new Open in Flow Debugger action on the Audit Log drawer and the Execution Chain page). The graph is built from trace rows (§7.3); a run that was not traced degrades honestly — inbound row + RouteCall row + outbound leaf rows, with the site interior rendered as one collapsed not traced node.

5. Trace model

5.1 Trace rows are audit rows

A new AuditChannel.ScriptTrace with kinds:

Kind	Emitted by	Payload (DetailsJson)
RouteCall	Central RouteHelper.Call (always-on, not gated)	target instance, script, args (redacted per existing policy), return value, success/error, duration ms, ChildExecutionId
ScriptRun	Site ScriptRunLauncher on run completion (gated)	script, instance, trigger {kind, attribute, value, source} (a value-change run's trigger rides here — there is no separate TriggerFired kind), parameters, return value, success/error, duration ms, source node
AttributeWrite	Site ScriptRuntimeContext.SetAttribute / WriteBatchAndWait (gated)	one row per write or batch: writes[{attribute, written, success, error}], trigger attribute
AttributeWait	Site ScriptRuntimeContext.WaitInternal (gated)	attribute, predicate/target, timeout, outcome (satisfied/timeout/cancelled/error), elapsed ms, observed value

Who emits AttributeWrite/AttributeWait (planning correction 2026-09-05): every accessor path funnels into three ScriptRuntimeContext methods — SetAttribute, WriteBatchAndWait and WaitInternal — and the context already holds IAuditWriter, _executionId and _parentExecutionId, so the InstanceActor needs no new ids and no new request fields. Batch-write echoes are not armed in v1 — only SetStaticAttributeCommand writes arm (§5.3).

Every row carries the run's ExecutionId and ParentExecutionId exactly as the existing channels do, so the Execution Chain walk, IX_AuditLog_ParentExecution, telemetry forwarding, reconciliation pulls, site 7-day retention and the central purge all apply unchanged. RouteCall is a central-direct-write row (ICentralAuditWriter, SourceNode = central-x); the site kinds go through the site IAuditWriter (SQLite hot path → telemetry → central).

Retention: ScriptTrace gets a per-channel override in AuditLog:PerChannelRetentionDays (rig default 30 days; must stay under the 365-day global window per the existing rule).

5.2 The trace gate

A boolean Trace rides beside ParentExecutionId everywhere the parent id already travels: RouteToCallRequest, RouteToCallResponse (echoed), ScriptCallRequest, SetStaticAttributeCommand, and the new SourceExecutionId carriers in §6. Site-side the flag lands in ScriptRuntimeContext as IsTraced and every helper consults it before emitting a ScriptTrace row. Nested runs inherit it (Scripts.Call → ScriptCallRequest.Trace; static-write cascades → SetStaticAttributeCommand.Trace → AttributeValueChanged.Trace → the spawned run), so a traced root traces its whole tree and nothing else.

Two ways a run becomes traced:

1. The debugger's Live run sets it on the root.
2. ApiMethod.TraceAllCalls (new bit NOT NULL DEFAULT 0 column, EF migration, CLI api-method update --trace-all-calls, Central UI checkbox on the method form): every real production call to that method is traced, so Replay of a real Delmia download is complete. Meant for low-volume integration methods; the method form shows the warning that it adds ~5 audit rows per call.

5.3 Value-change entry and the trace arm

The debugger writes the chosen attribute through the instance actor's normal path:

• Static attribute: RouteToSetAttributes → SetStaticAttributeCommand with SourceExecutionId = <debugger root id>, Trace = true. The existing cascade already stamps the resulting AttributeValueChanged and the spawned run's ParentExecutionId. Nothing new.
• Data-sourced attribute: the same RouteToSetAttributes → SetStaticAttributeCommand path — InstanceActor already dispatches data-sourced attributes to the DCL (HandleSetDataAttribute → WriteTagRequest); no WriteAttributeBatchRequest change is needed. The device echo comes back through the DCL as an ordinary AttributeValueChanged with no source. Trace arm: when HandleSetDataAttribute sees command.Trace && command.SourceExecutionId != null it records (attribute → sourceExecutionId, expiresAt = now + TraceArmWindow) (default 30 s, SiteRuntimeOptions, one entry per attribute, newest wins). The next DCL value change for that attribute — whatever the value, so a device that clamps or rejects the write still shows what it actually did — is published with SourceExecutionId and Trace set (consumed in HandleTagValueUpdate), and the arm is consumed. An arm that expires unconsumed emits nothing; the debugger renders the trigger node as no echo within 30 s.

The arm exists only for traced writes, so production behaviour is unchanged. Alarm cascades that the echo also fires inherit the parent id exactly as static cascades do today.

5.4 Live-run inline transport

A trace step IS an AuditEvent in channel ScriptTrace — there is no separate wire record. RouteToCallResponse gains additive ChildExecutionId (Guid?) and TraceEvents (IReadOnlyList<AuditEvent>?, the very events that became the site's audit rows, in emission order); on the wire it is repeated AuditEventDto trace_events (the DTO sitestream.proto already defines). The site collects events in a per-run FlowTraceCollector held by ScriptRuntimeContext and attaches the list to the ScriptCallResult → RouteToCallResponse when Trace is set. Capped at MaxTraceEventsPerRun (500); beyond it the collector keeps the first 500, sets Truncated, and the audit rows remain complete. A site on an older build ignores Trace, returns no TraceEvents, and the UI labels its lane site not traced (older build). FlowStep is a central-side projection (Commons/Types/Debugging/FlowStepProjector) computed from AuditEvent + AuditDetails; nothing serializes it.

For the value-change entry there is no routed reply to ride on. The debugger instead subscribes to the instance's debug stream (SubscribeDebugView, the Debug View's existing path) for the run's duration and takes the spawned run's ExecutionId from the stamped AttributeValueChanged, then pulls the site's trace rows for that id via a new PullTraceEvents RPC on SiteStreamService (same PSK interceptor as PullAuditEvents; reads the site audit SQLite by ExecutionId / ParentExecutionId). One additive RPC, no streaming change.

6. Message and wire changes (all additive)

Contract	Change
RouteToCallRequest	+ bool Trace = false
RouteToCallResponse	+ Guid? ChildExecutionId, + IReadOnlyList<AuditEvent>? TraceEvents
ScriptCallRequest / ScriptCallResult	+ bool Trace; + Guid? ExecutionId, + IReadOnlyList<AuditEvent>? TraceEvents on the result
RouteToSetAttributesRequest	+ Guid? ParentExecutionId, + bool Trace
SetStaticAttributeCommand, AttributeValueChanged	+ Guid? SourceExecutionId (where missing), + bool Trace
site_command.proto	trace on RouteToCallRequestDto/RouteToSetAttributesRequestDto; child_execution_id + repeated AuditEventDto trace_events on RouteToCallResponseDto (no FlowStepDto); regenerate with docker/regen-proto.sh sitecommand
sitestream.proto	+ PullTraceEvents RPC (PullTraceEventsRequest { repeated execution_ids, since_utc, max_events } → PullTraceEventsResponse { repeated AuditEventDto events }); AttributeValueUpdate + string source_execution_id = 7, + bool trace = 8 so the debug stream can hand the debugger the spawned id
ApiMethod	+ TraceAllCalls bit NOT NULL DEFAULT 0 (EF migration + manual SQL script)
AuditChannel/AuditKind	ScriptTrace; RouteCall, ScriptRun, AttributeWrite, AttributeWait

FlowStep (Commons Types/Debugging/FlowStep.cs, projection only — never on the wire): EventId, ExecutionId, ParentExecutionId, Kind, Lane (Central/Site/Tags), Title, Subtitle, OccurredAtUtc, DurationMs, Outcome, InputJson, OutputJson, ErrorMessage, SiteId/Node/Instance/Target, and a kind-specific ScriptTraceExtra parsed from AuditDetails.Extra. Field numbers never reused; proto3 defaults are the negotiation, as for stream batching.

7. Central components

7.1 FlowDebugService (Central UI, scoped)

• PredictAsync(entry) → §8.
• RunMethodAsync / RunValueChangeAsync / ReplayAsync → Live runs and Replay are FlowDebugRunMethodCommand / FlowDebugRunValueChangeCommand / FlowDebugReplayCommand on the ManagementActor (Deployer role, long-running timeout). The ManagementActor is a per-node actor (AkkaHostedService, /user/management), not a singleton. The run-method handler consults IActiveNodeGate and refuses on a standby with error code STANDBY_NODE naming the design rule (Component-InboundAPI: method scripts and Route.To never run on a standby); Traefik routes UI traffic to the active node so this is rare. The Central UI calls it through ManagementActorHolder (the DataConnectionMoveService pattern) and the CLI through /management — one path, one audit story. The inbound row for a Live run is written by the handler via ICentralAuditWriter (kind InboundRequest, actor = operator username, Extra.source = "flow-debugger"). Inbound entry: InboundScriptExecutor.ExecuteAsync(method, inputs, route, timeout, parentExecutionId: root, trace: true) with a RouteHelper whose Trace flag is set and whose inline trace sink collects the RouteCall row plus the site's TraceEvents. Value-change entry: §5.3 + §5.4.
• ReplayAsync(executionId) → IAuditLogRepository.GetExecutionTreeAsync + a new GetTraceEventsAsync(rootExecutionId) (all ScriptTrace rows in the tree, plus the InboundRequest row) → FlowGraphBuilder.

7.2 FlowGraphBuilder (pure, Commons or Central UI)

Steps → FlowGraph { Lanes, Nodes, Edges }. Lane assignment by Where (central node → Central; site node → Site; AttributeWrite → Tags). Edges follow emission order within an execution and parent→child across executions; an edge whose endpoints sit on different lanes is marked CrossesBoundary. A predicted graph and a real graph share the model; the page overlays them (predicted-but-not-executed → greyed).

7.3 Page and renderer

Components/Pages/Deployment/FlowDebugger.razor (+ .razor.cs), nav item under Deployment after Debug View. Renderer is a custom SVG component FlowGraphView (no third-party charting or diagram libraries, matching KpiTrendChart): fixed 110 px lanes, nodes placed by topological column, connectors as orthogonal paths with arrow markers; zoom via a viewBox scale. Detail panel is a plain Bootstrap tab strip. Persisted per-browser: last entry point and inputs (same localStorage pattern Debug View uses for its selection).

8. Predicted graph (static analysis)

FlowShapePredictor in Script Analysis (#25), a CSharpSyntaxWalker beside ScriptTrustValidator.HardeningWalker, over the entry script and every script it can reach:

• Recognised sites: Route.To(x).Call(name, …) / .SetAttributes / .WaitForAttribute / .GetAttributes; Scripts.Call(name) / CallShared; Attributes["a"] = v, Children["c"].Attributes["a"] = v; WriteBatchAndWaitAsync(…), WaitAsync(…), WaitForAsync(…); ExternalSystem.Call/CachedCall; Database.*; Notify.Send; return.
• Constant string arguments resolve; anything else yields a dashed node (target = input value). Route.To(constant) resolves the instance → template → the named script's body, which is then walked too (one hop; depth cap 4, the same as the runtime call-depth cap).
• Branches: an if … return before a recognised site becomes an early return annotation on the edge, as the mockup shows; the predictor does not attempt path-sensitivity beyond that.
• Output is a FlowGraph with Predicted = true; the page renders it before the run and overlays the real graph after.

Compile is the existing cached compile (CachingScriptMetadataResolver attached like every other surface); the walker is syntax-only and never executes anything.

9. Security, authorisation, audit

• Page visibility and Live run: RequireDeployment (same as Debug View) — decided 2026-09-05: a Live run writes real tags, but the Deployer already deploys and redeploys the scripts that do exactly that, so no separate Operator gate. Replay: OperationalAudit (it reads audit rows). No Verifier step — the debugger is a diagnostic, not a production write control, and the audit row is the safeguard.
• Inbound audit row for a Live run: the shared AuditEvent.Actor carries the operator's username (today it carries the API-key name); Extra.source = "flow-debugger", Extra.runNumber; Target is the method as usual. API-key authentication is bypassed (there is no key), parameter validation is not — ParameterValidator runs exactly as for HTTP so a bad input fails the same way. The method's TimeoutSeconds applies. Site-scope (SiteScopeService) restricts which instances a value-change entry may target.
• Trace payloads pass through the existing redaction policy (auth headers redacted, 8 KB cap, 64 KB on error rows). Console output is capped the same way.
• Nothing here touches the PSK boundary; PullTraceEvents sits behind ControlPlaneAuthInterceptor.

10. Volume and failure behaviour

• Untraced production is unchanged except one RouteCall row per routed call (inbound-driven).
• A traced Delmia call emits ≈ 1 InboundRequest + 1 RouteCall + 1 ScriptRun + 2 AttributeWrite + 1 AttributeWait + 1 value-change ScriptRun ≈ 7 rows. TraceAllCalls on a 100-call/day method is ~700 rows/day.
• Trace emission is best-effort like all audit (a failed trace write never fails the run).
• Live run timeout = method timeout + FlowDebugRunGrace (10 s); a timeout renders whatever steps arrived and marks the run Timed out. The site's trace rows still land later, so Replay of the same ExecutionId is complete.
• Older site build: no TraceEvents, lane labelled; older central build: ignores new fields; proto3 defaults negotiate as for stream batching.
• Mixed-version caveat: an OLD central ingesting ScriptTrace rows from a NEW site parses unknown channel/kind names to the AuditRowProjection fallbacks (ApiInbound/InboundRequest); upgrade central first.

11. Out of scope (recorded so they are not rediscovered)

• Dry run (D1). Alarm-trigger and timer-trigger entry points. Step-into for CallShared inline shared scripts (they execute inline; only Scripts.Call spawns a traceable child). Live step-by-step / breakpoints. Editing a script from the debugger. Cross-run comparison. A Vertical trace list toggle (mockup Direction B) — cheap follow-up on the same FlowGraph. Line-level pinning (Direction C) — needs per-statement instrumentation. Console output capture on ScriptRun rows (no script console exists today).

12. Testing

• Pure: FlowGraphBuilderTests (lane/edge/boundary rules, truncation), FlowShapePredictorTests (each recognised site, constant vs input-dependent target, one-hop route resolution, depth cap, early-return annotation), FlowTraceCollectorTests (cap + Truncated).
• Site runtime: ScriptRuntimeContext traced write emits AttributeWrite; InstanceActor arms; echo consumes the arm and stamps SourceExecutionId/Trace; expiry emits nothing; untraced write emits nothing and arms nothing (the production-unchanged pin). ScriptRunLauncher emits ScriptRun only when traced. ScriptRuntimeContext.WaitInternal emits AttributeWait for satisfied/timeout.
• Inbound API: RouteHelper.Call always writes RouteCall; ChildExecutionId round-trips; Trace propagates to RouteToCallRequest.
• Communication: DTO mapper round-trips for every new field; an old-shape response (no TraceEvents) maps to null.
• Audit: new kinds accepted by ingest; per-channel retention deletes ScriptTrace rows early; GetTraceEventsAsync returns the tree's rows in order.
• Central UI (bUnit): page renders predicted graph, overlays real graph, Run button hidden without Deployer, Replay degrades to the collapsed not traced node.
• Rig: extend docker/verify-delmia-notifier.sh with step H — a Live run of DelmiaRecipeDownload against DelmiaSim01 from the page's service, asserting 9 steps, and a Replay of the same ExecutionId producing the same graph.

13. Documents to update with the implementation

Component-CentralUI.md (new page), Component-InboundAPI.md (RouteCall row, TraceAllCalls, Live-run actor), Component-SiteRuntime.md (trace collector, trace arm, gated emission), Component-AuditLog.md (ScriptTrace channel, retention override, PullTraceEvents), Component-Communication.md (proto changes), Component-ScriptAnalysis.md (predictor), Component-ConfigurationDatabase.md (migration), README.md component table if a component is added (none planned — the debugger is a Central UI page plus seams in existing components), CLAUDE.md key-decisions bullet, CLI README (--trace-all-calls), deferred-work register row for the Direction B list toggle.