docs(plans): Flow Debugger design — visual swimlane debugger for inbound API + value-change flows

Live + Replay modes (no dry run); entry points = inbound API method and
value-change trigger; data-sourced value-change does a real tag write with a
30 s trace arm on the instance actor for echo correlation; gated trace rows in
a new ScriptTrace audit channel (RouteCall always-on); inline step return on
RouteToCallResponse for Live; Deployer gates the page and Live run.

Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
This commit is contained in:
Joseph Doherty
2026-09-05 22:51:52 -04:00
parent d2bfd8340c
commit 7fa916e2c7
@@ -0,0 +1,306 @@
# Flow Debugger — design
**Status:** approved 2026-09-05; implementation plan follows.
**Mockup:** Claude Design canvas *Flow Debugger Mockup* — <https://claude.ai/code/artifact/3f3e81e5-8e5e-4333-bdf6-59256108494d> (Direction A swimlane boards + two low-fi alternates + decision notes).
**Owner components:** Central UI (#page), Inbound API, Site Runtime, Audit Log, Communication, Script Analysis, Configuration Database.
## 1. What it is
A Central UI page, **Deployment → Flow Debugger**, that runs one entry point with operator-supplied
inputs and renders every step the run passes through as a swimlane flowchart: lanes are *where* code
ran (Central, the Site the route landed on, the tag source), nodes are steps (request, script run,
route call, attribute write, wait, response) each carrying its input, output, duration and status.
Clicking a node opens a tabbed detail panel (Input · Output · Attribute writes · Console · Timing).
Worked example, and the acceptance flow for the whole feature: the Delmia recipe download
(`docs/api-specs/Delmia-Recipe-Download-API.md` §4) — inbound `POST /api/DelmiaRecipeDownload` →
central method script → `Route.To(MachineCode).Call("ProcessRecipeDownload")` → site Call script
on the `DelmiaReceiver` child → seven attribute writes → wait for Galaxy to raise
`RecipeProcessedFlag` → read result → response.
### Decisions taken (2026-09-05)
| # | Decision | Rejected |
|---|----------|----------|
| D1 | **Two modes: Live and Replay.** Live is a real run with real values, real writes, audited like any inbound call. Replay executes nothing — it rebuilds the flow from an existing ExecutionId's trace rows. | *Dry run* (execute scripts but capture writes/ES/DB/Notify instead of sending). Needs an interception seam in the site runtime that does not exist and would fork the semantics of every helper. |
| D2 | **Two entry points in v1:** an Inbound API method, and a value-change-triggered site script on a named instance. Both may land on a site script on instance X. | Alarm/timer trigger simulation, direct "call script X" — see §11. |
| D3 | **Value-change on a data-sourced attribute does a REAL tag write** through the instance actor's normal write path; the device echo fires the trigger. Correlation is by a short-lived *trace arm* on the instance actor (§5.3). | A debug-only "inject value" command that synthesises the change without touching the device — leaves the instance's cached value diverged from the device until the next real update. |
| D4 | **Live gets its trace inline; Replay reads audit rows.** Site audit telemetry forwards on a 5 s busy / 30 s idle cadence (`AuditLog:SiteTelemetry`), too slow to drive an interactive run, so the routed reply carries the site steps back and the same steps are *also* written as audit rows for Replay. | Live waiting on the audit pipeline. |
| D5 | **Tracing is gated, not always-on.** Sites run scripts at high rates from timers and value changes; a trace flag rides beside `ParentExecutionId` and only flagged runs emit trace rows. One exception is always-on: a central `RouteCall` row per `Route.To().Call` (inbound-driven volume, joins the two halves of a Replay). | Tracing every run (volume), or a purely time-window join. |
| D6 | **Live run is a Deployer act.** `RequireDeployment` gates the page and the Run button; Replay needs `OperationalAudit`. | A separate `RequireOperator` gate for Live. |
## 2. What exists today (reused, not rebuilt)
- **Correlation ids already thread the chain.** `AuditWriteMiddleware` mints an `ExecutionId` per inbound request; `RouteHelper.Call` stamps it as `RouteToCallRequest.ParentExecutionId`; `InstanceActor` forwards it on `ScriptCallRequest.ParentExecutionId`; the site run mints its own id (`ScriptRuntimeContext._executionId`) and records the parent; a static write carries the writer's id (`SetStaticAttributeCommand.SourceExecutionId` → `AttributeValueChanged.SourceExecutionId`) into the value-change / alarm cascade. `IAuditLogRepository.GetExecutionTreeAsync` walks it.
- **Audit captures the ends.** `AuditKind.InboundRequest` stores the full request and response (up to `InboundMaxBytes`); `ApiCall`/`DbWrite`/`NotifySend` rows are per-execution.
- **Central already drives live instances.** `SiteCommandService` routing group: `RouteToCall`, `RouteToGetAttributes`, `RouteToSetAttributes`, `RouteToWaitForAttribute`; the Test Run sandbox's `SandboxInstanceGateway` runs a central script against a deployed instance over exactly this transport.
- **Roslyn compiles every script centrally** (`ScriptAnalysisService`, `ScriptTrustValidator`'s syntax walk), so the predicted graph is a sibling walker, not new infrastructure.
- **Attribute writes already have one path.** `WriteAttributeBatchRequest` → `InstanceActor.HandleWriteAttributeBatch` → `WriteTagBatchRequest` to the DCL for data-sourced attributes; `SetStaticAttributeCommand` for static ones.
## 3. What is missing (the build)
| Gap | Why it blocks | Closed by |
|-----|---------------|-----------|
| G1 No record that a site script ran — no `AuditKind` for a run; per-run site events are opt-in, aggregated, id-less. | Replay would show the inbound row and any outbound leaf rows with the site script itself absent. | §5.1 `ScriptRun` trace row. |
| G2 Attribute writes and waits are invisible — `WriteAttributeBatchRequest`/`WaitForAttributeRequest` carry no execution id; nothing records outcomes. | The Delmia flow's steps 5–7 cannot be drawn. | §5.1 `AttributeWrite`/`AttributeWait` rows + §6 additive fields. |
| G3 Route calls are not audited centrally and the routed reply does not return the child run's id. | Central and site halves of a Replay cannot be joined without guessing. | §5.1 `RouteCall` row (always-on) + `RouteToCallResponse.ChildExecutionId`. |
| G4 No trace gate. | Always-on would flood `AuditLog` from timer/value-change scripts. | §5.2 trace flag + per-method `TraceAllCalls`. |
| G5 A value-change entry needs a way to fire the trigger and correlate the echo. | Data-sourced echoes arrive as roots. | §5.3 trace arm. |
| G6 No page, no graph builder, no predictor. | — | §7, §8. |
| G7 Live runs have no API key, so no actor on the inbound row. | Audit would show an anonymous inbound call. | §9. |
## 4. User experience
Reference the mockup's two full-fidelity boards; this section only records what the mockup fixes.
**Toolbar.** Entry point (`Inbound API method` | `Value-change trigger`) + a picker for the method or
the instance + script; Mode (`Live` | `Replay`); `Edit inputs`, `Load from audit…`, `Run`.
**Status strip.** State chip (`Not run` / `Running` / `Completed` / `Failed` / `Timed out`), run
number, wall-clock and node, total duration, counts (steps, boundaries crossed, attribute writes),
root `ExecutionId` linking to the existing Execution Chain page.
**Flow panel.** Three lanes fixed in v1: Central, Site (`{siteId} · {instance}`), Tags
(`{connection} → {source}`). Before a run the graph is *predicted* (§8) — nodes whose target
depends on an input value or on live site state render dashed. After a run every node is real;
predicted nodes that did not execute (an early-return branch not taken) render greyed. Node =
kind eyebrow + duration, title, one-line in/out summary; the selected node gets the accent border.
Connectors: solid = executed edge, accent = crossed a gRPC boundary, dashed = external actor
(the tag source responding). Zoom −/+/Fit only; no pan-by-drag in v1.
**Detail panel** (below the flow, tabbed): Input, Output, Attribute writes (attribute · before ·
written · quality · ack ms), Console, Timing; a right column *Where this ran* (site/node,
instance, composed child, connection, attribute kind, source line) with `Open in Debug View` and
`Audit rows (n)` drill-ins.
**Inputs form** (the `Step 1` detail before a run): generated from the method's
`ParameterDefinitions` (type, required) or, for a value-change entry, one field for the trigger
attribute's new value plus read-only current value/quality. `Fill from last real call` pre-fills
from the most recent `InboundRequest` row for that method.
**Replay.** `Load from audit…` takes an ExecutionId (or arrives via a new `Open in Flow Debugger`
action on the Audit Log drawer and the Execution Chain page). The graph is built from trace rows
(§7.3); a run that was not traced degrades honestly — inbound row + `RouteCall` row + outbound
leaf rows, with the site interior rendered as one collapsed `not traced` node.
## 5. Trace model
### 5.1 Trace rows are audit rows
A new `AuditChannel.ScriptTrace` with kinds:
| Kind | Emitted by | Payload (`DetailsJson`) |
|------|-----------|-------------------------|
| `RouteCall` | Central `RouteHelper.Call` (**always-on**, not gated) | target instance, script, args (redacted per existing policy), return value, success/error, duration ms, `ChildExecutionId` |
| `ScriptRun` | Site `ScriptRunLauncher` on run completion (gated) | script, instance, trigger kind, parameters, return value, success/error, duration ms, console output (capped 8 KB like other payloads), source node |
| `AttributeWrite` | Site `InstanceActor` write handlers (gated) | one row per batch: attributes[{name, before, written, quality, ack ms}], trigger attribute, static/data-sourced |
| `AttributeWait` | Site `InstanceActor.HandleWaitForAttribute` (gated) | attribute, predicate/target, timeout, outcome (satisfied/timeout/cancelled), elapsed ms, observed value |
| `TriggerFired` | Site `ScriptActor` when a value-change run is spawned with a stamped `SourceExecutionId` (gated) | attribute, old/new value, source (static write / device echo) |
Every row carries the run's `ExecutionId` and `ParentExecutionId` exactly as the existing channels
do, so the Execution Chain walk, `IX_AuditLog_ParentExecution`, telemetry forwarding, reconciliation
pulls, site 7-day retention and the central purge all apply unchanged. `RouteCall` is a
central-direct-write row (`ICentralAuditWriter`, `SourceNode = central-x`); the site kinds go
through the site `IAuditWriter` (SQLite hot path → telemetry → central).
Retention: `ScriptTrace` gets a per-channel override in `AuditLog:PerChannelRetentionDays`
(rig default 30 days; must stay under the 365-day global window per the existing rule).
### 5.2 The trace gate
A boolean `Trace` rides beside `ParentExecutionId` everywhere the parent id already travels:
`RouteToCallRequest`, `RouteToCallResponse` (echoed), `ScriptCallRequest`,
`SetStaticAttributeCommand`, and the new `SourceExecutionId` carriers in §6. Site-side the flag
lands in `ScriptRuntimeContext` as `IsTraced` and every helper consults it before emitting a
`ScriptTrace` row. Nested runs inherit it (`Scripts.Call` → `ScriptCallRequest.Trace`;
static-write cascades → `SetStaticAttributeCommand.Trace` → `AttributeValueChanged.Trace` →
the spawned run), so a traced root traces its whole tree and nothing else.
Two ways a run becomes traced:
1. **The debugger's Live run** sets it on the root.
2. **`ApiMethod.TraceAllCalls`** (new `bit NOT NULL DEFAULT 0` column, EF migration, CLI
`api-method update --trace-all-calls`, Central UI checkbox on the method form): every real
production call to that method is traced, so Replay of a real Delmia download is complete.
Meant for low-volume integration methods; the method form shows the warning that it adds
~5 audit rows per call.
### 5.3 Value-change entry and the trace arm
The debugger writes the chosen attribute through the instance actor's normal path:
- **Static attribute:** `RouteToSetAttributes` → `SetStaticAttributeCommand` with
`SourceExecutionId = <debugger root id>`, `Trace = true`. The existing cascade already stamps
the resulting `AttributeValueChanged` and the spawned run's `ParentExecutionId`. Nothing new.
- **Data-sourced attribute:** a single-attribute `WriteAttributeBatchRequest` (new additive
`SourceExecutionId` + `Trace`) → DCL `WriteTagBatchRequest`. The device echo comes back through
the DCL as an ordinary `AttributeValueChanged` with no source. **Trace arm:** when
`InstanceActor` performs a traced write it records `(attribute → sourceExecutionId, expiresAt =
now + TraceArmWindow)` (default 30 s, `SiteRuntimeOptions`, one entry per attribute, newest
wins). The next DCL value change for that attribute — whatever the value, so a device that
clamps or rejects the write still shows what it actually did — is published with
`SourceExecutionId` and `Trace` set, and the arm is consumed. An arm that expires unconsumed
emits nothing; the debugger renders the trigger node as `no echo within 30 s`.
The arm exists only for traced writes, so production behaviour is unchanged. Alarm cascades that
the echo also fires inherit the parent id exactly as static cascades do today.
### 5.4 Live-run inline transport
`RouteToCallResponse` gains additive `ChildExecutionId` (`Guid?`) and `Steps`
(`IReadOnlyList<FlowStep>?`, the same records that became the site's audit rows, in emission
order). The site collects steps in a per-run `FlowTraceCollector` held by `ScriptRuntimeContext`
and attaches the list to the `ScriptCallResult` → `RouteToCallResponse` when `Trace` is set.
Capped at `MaxTraceStepsPerRun` (500); beyond it the collector keeps the first 500, sets
`Truncated`, and the audit rows remain complete. A site on an older build ignores `Trace`, returns
no `Steps`, and the UI labels its lane `site not traced (older build)`.
For the value-change entry there is no routed reply to ride on. The debugger instead subscribes
to the instance's debug stream (`SubscribeDebugView`, the Debug View's existing path) for the run's
duration and takes the spawned run's `ExecutionId` from the stamped `AttributeValueChanged`, then
pulls the site's trace rows for that id via a new `PullTraceSteps` RPC on `SiteStreamService`
(same PSK interceptor as `PullAuditEvents`; reads the site audit SQLite by `ExecutionId`). One
additive RPC, no streaming change.
## 6. Message and wire changes (all additive)
| Contract | Change |
|----------|--------|
| `RouteToCallRequest` | `+ bool Trace = false` |
| `RouteToCallResponse` | `+ Guid? ChildExecutionId`, `+ IReadOnlyList<FlowStep>? Steps` |
| `ScriptCallRequest` / `ScriptCallResult` | `+ bool Trace`; `+ Guid ExecutionId`, `+ IReadOnlyList<FlowStep>? Steps` on the result |
| `WriteAttributeBatchRequest`, `WaitForAttributeRequest` | `+ Guid? SourceExecutionId`, `+ bool Trace` |
| `SetStaticAttributeCommand`, `AttributeValueChanged` | `+ bool Trace` (id already present) |
| `site_command.proto` | new fields on `RouteToCallRequestDto`/`ResponseDto`; new `FlowStepDto`; regenerate with `docker/regen-proto.sh sitecommand` |
| `sitestream.proto` | `+ PullTraceSteps` RPC (`PullTraceStepsRequest { execution_id }` → `PullTraceStepsResponse { repeated FlowStepDto }`); `AttributeValueUpdate` `+ string source_execution_id = 7` so the debug stream can hand the debugger the spawned id |
| `ApiMethod` | `+ TraceAllCalls bit NOT NULL DEFAULT 0` (EF migration + manual SQL script) |
| `AuditChannel`/`AuditKind` | `ScriptTrace`; `RouteCall`, `ScriptRun`, `AttributeWrite`, `AttributeWait`, `TriggerFired` |
`FlowStep` (Commons `Types/Debugging/FlowStep.cs`): `StepId`, `ExecutionId`, `ParentExecutionId`,
`Kind`, `Lane` (`Central`/`Site`/`Tags`), `Title`, `StartedAtUtc`, `DurationMs`, `Outcome`,
`InputJson`, `OutputJson`, `Where` (site, node, instance, child path, connection), `SourceLine?`.
Field numbers never reused; proto3 defaults are the negotiation, as for stream batching.
## 7. Central components
### 7.1 `FlowDebugService` (Central UI, scoped)
- `PredictAsync(entry)` → §8.
- `RunLiveAsync(entry, inputs, operator)` → dispatches a `MgmtFlowDebugRunCommand` through the
`ManagementActor` singleton so the method script and its `Route.To` calls execute on the
**active** central node, honouring `IActiveNodeGate`'s rule that a standby never runs method
scripts. The actor hands the run to a detached task (`PipeTo`) so a 45 s method does not block
its mailbox. Inbound entry: `InboundScriptExecutor.ExecuteAsync(method, inputs, route, timeout,
parentExecutionId: root, trace: true)` with a `RouteHelper` whose `Trace` flag is set; the
central steps (request, method run, route call, response) are recorded by the service itself and
the site steps arrive on `RouteToCallResponse.Steps`. Value-change entry: §5.3 + §5.4.
- `ReplayAsync(executionId)` → `IAuditLogRepository.GetExecutionTreeAsync` + a new
`GetTraceStepsAsync(rootExecutionId)` (all `ScriptTrace` rows in the tree, plus the
`InboundRequest` row) → `FlowGraphBuilder`.
### 7.2 `FlowGraphBuilder` (pure, Commons or Central UI)
Steps → `FlowGraph { Lanes, Nodes, Edges }`. Lane assignment by `Where` (central node → Central;
site node → Site; `AttributeWrite`/`TriggerFired` → Tags). Edges follow emission order within an
execution and parent→child across executions; an edge whose endpoints sit on different lanes is
marked `CrossesBoundary`. A predicted graph and a real graph share the model; the page overlays
them (predicted-but-not-executed → greyed).
### 7.3 Page and renderer
`Components/Pages/Deployment/FlowDebugger.razor` (+ `.razor.cs`), nav item under Deployment after
Debug View. Renderer is a custom SVG component `FlowGraphView` (no third-party charting or diagram
libraries, matching `KpiTrendChart`): fixed 110 px lanes, nodes placed by topological column,
connectors as orthogonal paths with arrow markers; zoom via a `viewBox` scale. Detail panel is a
plain Bootstrap tab strip. Persisted per-browser: last entry point and inputs (same
`localStorage` pattern Debug View uses for its selection).
## 8. Predicted graph (static analysis)
`FlowShapePredictor` in Script Analysis (#25), a `CSharpSyntaxWalker` beside
`ScriptTrustValidator.HardeningWalker`, over the entry script and every script it can reach:
- Recognised sites: `Route.To(x).Call(name, …)` / `.SetAttributes` / `.WaitForAttribute` /
`.GetAttributes`; `Scripts.Call(name)` / `CallShared`; `Attributes["a"] = v`,
`Children["c"].Attributes["a"] = v`; `WriteBatchAndWaitAsync(…)`, `WaitAsync(…)`,
`WaitForAsync(…)`; `ExternalSystem.Call/CachedCall`; `Database.*`; `Notify.Send`; `return`.
- Constant string arguments resolve; anything else yields a dashed node (`target = input value`).
`Route.To(constant)` resolves the instance → template → the named script's body, which is then
walked too (one hop; depth cap 4, the same as the runtime call-depth cap).
- Branches: an `if … return` before a recognised site becomes an `early return` annotation on the
edge, as the mockup shows; the predictor does not attempt path-sensitivity beyond that.
- Output is a `FlowGraph` with `Predicted = true`; the page renders it before the run and overlays
the real graph after.
Compile is the existing cached compile (`CachingScriptMetadataResolver` attached like every other
surface); the walker is syntax-only and never executes anything.
## 9. Security, authorisation, audit
- **Page visibility and Live run:** `RequireDeployment` (same as Debug View) — decided
2026-09-05: a Live run writes real tags, but the Deployer already deploys and redeploys the
scripts that do exactly that, so no separate Operator gate. **Replay:** `OperationalAudit` (it
reads audit rows). No Verifier step — the debugger is a diagnostic, not a production write
control, and the audit row is the safeguard.
- **Inbound audit row for a Live run:** the shared `AuditEvent.Actor` carries the operator's
username (today it carries the API-key name); `Extra.source = "flow-debugger"`,
`Extra.runNumber`; `Target` is the method as usual. API-key authentication is bypassed (there is
no key), **parameter validation is not** — `ParameterValidator` runs exactly as for HTTP so a
bad input fails the same way. The method's `TimeoutSeconds` applies. Site-scope (`SiteScopeService`)
restricts which instances a value-change entry may target.
- Trace payloads pass through the existing redaction policy (auth headers redacted, 8 KB cap,
64 KB on error rows). Console output is capped the same way.
- Nothing here touches the PSK boundary; `PullTraceSteps` sits behind `ControlPlaneAuthInterceptor`.
## 10. Volume and failure behaviour
- Untraced production is unchanged except one `RouteCall` row per routed call (inbound-driven).
- A traced Delmia call emits ≈ 1 `InboundRequest` + 1 `RouteCall` + 1 `ScriptRun` + 2
`AttributeWrite` + 1 `AttributeWait` + 1 `TriggerFired` ≈ 7 rows. `TraceAllCalls` on a
100-call/day method is ~700 rows/day.
- Trace emission is best-effort like all audit (a failed trace write never fails the run).
- Live run timeout = method timeout + `FlowDebugRunGrace` (10 s); a timeout renders whatever steps
arrived and marks the run `Timed out`. The site's trace rows still land later, so Replay of the
same ExecutionId is complete.
- Older site build: no `Steps`, lane labelled; older central build: ignores new fields; proto3
defaults negotiate as for stream batching.
## 11. Out of scope (recorded so they are not rediscovered)
- Dry run (D1). Alarm-trigger and timer-trigger entry points. Step-into for `CallShared` inline
shared scripts (they execute inline; only `Scripts.Call` spawns a traceable child). Live
step-by-step / breakpoints. Editing a script from the debugger. Cross-run comparison. A
`Vertical trace` list toggle (mockup Direction B) — cheap follow-up on the same `FlowGraph`.
Line-level pinning (Direction C) — needs per-statement instrumentation.
## 12. Testing
- **Pure:** `FlowGraphBuilderTests` (lane/edge/boundary rules, truncation), `FlowShapePredictorTests`
(each recognised site, constant vs input-dependent target, one-hop route resolution, depth cap,
early-return annotation), `FlowTraceCollectorTests` (cap + `Truncated`).
- **Site runtime:** `InstanceActor` traced write emits `AttributeWrite` and arms; echo consumes the
arm and stamps `SourceExecutionId`/`Trace`; expiry emits nothing; untraced write emits nothing
and arms nothing (the production-unchanged pin). `ScriptRunLauncher` emits `ScriptRun` only when
traced. `HandleWaitForAttribute` emits `AttributeWait` for satisfied/timeout.
- **Inbound API:** `RouteHelper.Call` always writes `RouteCall`; `ChildExecutionId` round-trips;
`Trace` propagates to `RouteToCallRequest`.
- **Communication:** DTO mapper round-trips for every new field; an old-shape response (no
`Steps`) maps to null.
- **Audit:** new kinds accepted by ingest; per-channel retention deletes `ScriptTrace` rows early;
`GetTraceStepsAsync` returns the tree's rows in order.
- **Central UI (bUnit):** page renders predicted graph, overlays real graph, Run button hidden
without Deployer, Replay degrades to the collapsed `not traced` node.
- **Rig:** extend `docker/verify-delmia-notifier.sh` with step H — a Live run of
`DelmiaRecipeDownload` against `DelmiaSim01` from the page's service, asserting 9 steps, and a
Replay of the same ExecutionId producing the same graph.
## 13. Documents to update with the implementation
`Component-CentralUI.md` (new page), `Component-InboundAPI.md` (RouteCall row, TraceAllCalls,
Live-run actor), `Component-SiteRuntime.md` (trace collector, trace arm, gated emission),
`Component-AuditLog.md` (ScriptTrace channel, retention override, PullTraceSteps),
`Component-Communication.md` (proto changes), `Component-ScriptAnalysis.md` (predictor),
`Component-ConfigurationDatabase.md` (migration), `README.md` component table if a component is
added (none planned — the debugger is a Central UI page plus seams in existing components),
`CLAUDE.md` key-decisions bullet, CLI README (`--trace-all-calls`), deferred-work register row for
the Direction B list toggle.