docs(plans): flow debugger design — trace steps are audit events; mgmt-command execution path; context-emitted write/wait rows

Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
This commit is contained in:
Joseph Doherty
2026-09-05 23:26:28 -04:00
parent 2e3eb94d38
commit 61d1785fcb
+72 -48
View File
@@ -92,10 +92,15 @@ A new `AuditChannel.ScriptTrace` with kinds:
| Kind | Emitted by | Payload (`DetailsJson`) | | Kind | Emitted by | Payload (`DetailsJson`) |
|------|-----------|-------------------------| |------|-----------|-------------------------|
| `RouteCall` | Central `RouteHelper.Call` (**always-on**, not gated) | target instance, script, args (redacted per existing policy), return value, success/error, duration ms, `ChildExecutionId` | | `RouteCall` | Central `RouteHelper.Call` (**always-on**, not gated) | target instance, script, args (redacted per existing policy), return value, success/error, duration ms, `ChildExecutionId` |
| `ScriptRun` | Site `ScriptRunLauncher` on run completion (gated) | script, instance, trigger kind, parameters, return value, success/error, duration ms, console output (capped 8 KB like other payloads), source node | | `ScriptRun` | Site `ScriptRunLauncher` on run completion (gated) | script, instance, trigger `{kind, attribute, value, source}` (a value-change run's trigger rides here — there is no separate `TriggerFired` kind), parameters, return value, success/error, duration ms, source node |
| `AttributeWrite` | Site `InstanceActor` write handlers (gated) | one row per batch: attributes[{name, before, written, quality, ack ms}], trigger attribute, static/data-sourced | | `AttributeWrite` | Site `ScriptRuntimeContext.SetAttribute` / `WriteBatchAndWait` (gated) | one row per write or batch: writes[{attribute, written, success, error}], trigger attribute |
| `AttributeWait` | Site `InstanceActor.HandleWaitForAttribute` (gated) | attribute, predicate/target, timeout, outcome (satisfied/timeout/cancelled), elapsed ms, observed value | | `AttributeWait` | Site `ScriptRuntimeContext.WaitInternal` (gated) | attribute, predicate/target, timeout, outcome (satisfied/timeout/cancelled/error), elapsed ms, observed value |
| `TriggerFired` | Site `ScriptActor` when a value-change run is spawned with a stamped `SourceExecutionId` (gated) | attribute, old/new value, source (static write / device echo) |
**Who emits `AttributeWrite`/`AttributeWait` (planning correction 2026-09-05):** every accessor path
funnels into three `ScriptRuntimeContext` methods — `SetAttribute`, `WriteBatchAndWait` and
`WaitInternal` — and the context already holds `IAuditWriter`, `_executionId` and
`_parentExecutionId`, so the `InstanceActor` needs no new ids and no new request fields. Batch-write
echoes are **not** armed in v1 — only `SetStaticAttributeCommand` writes arm (§5.3).
Every row carries the run's `ExecutionId` and `ParentExecutionId` exactly as the existing channels Every row carries the run's `ExecutionId` and `ParentExecutionId` exactly as the existing channels
do, so the Execution Chain walk, `IX_AuditLog_ParentExecution`, telemetry forwarding, reconciliation do, so the Execution Chain walk, `IX_AuditLog_ParentExecution`, telemetry forwarding, reconciliation
@@ -132,14 +137,17 @@ The debugger writes the chosen attribute through the instance actor's normal pat
- **Static attribute:** `RouteToSetAttributes` → `SetStaticAttributeCommand` with - **Static attribute:** `RouteToSetAttributes` → `SetStaticAttributeCommand` with
`SourceExecutionId = <debugger root id>`, `Trace = true`. The existing cascade already stamps `SourceExecutionId = <debugger root id>`, `Trace = true`. The existing cascade already stamps
the resulting `AttributeValueChanged` and the spawned run's `ParentExecutionId`. Nothing new. the resulting `AttributeValueChanged` and the spawned run's `ParentExecutionId`. Nothing new.
- **Data-sourced attribute:** a single-attribute `WriteAttributeBatchRequest` (new additive - **Data-sourced attribute:** the **same** `RouteToSetAttributes` → `SetStaticAttributeCommand`
`SourceExecutionId` + `Trace`) → DCL `WriteTagBatchRequest`. The device echo comes back through path — `InstanceActor` already dispatches data-sourced attributes to the DCL
the DCL as an ordinary `AttributeValueChanged` with no source. **Trace arm:** when (`HandleSetDataAttribute` → `WriteTagRequest`); no `WriteAttributeBatchRequest` change is needed.
`InstanceActor` performs a traced write it records `(attribute → sourceExecutionId, expiresAt = The device echo comes back through the DCL as an ordinary `AttributeValueChanged` with no
source. **Trace arm:** when `HandleSetDataAttribute` sees `command.Trace &&
command.SourceExecutionId != null` it records `(attribute → sourceExecutionId, expiresAt =
now + TraceArmWindow)` (default 30 s, `SiteRuntimeOptions`, one entry per attribute, newest now + TraceArmWindow)` (default 30 s, `SiteRuntimeOptions`, one entry per attribute, newest
wins). The next DCL value change for that attribute — whatever the value, so a device that wins). The next DCL value change for that attribute — whatever the value, so a device that
clamps or rejects the write still shows what it actually did — is published with clamps or rejects the write still shows what it actually did — is published with
`SourceExecutionId` and `Trace` set, and the arm is consumed. An arm that expires unconsumed `SourceExecutionId` and `Trace` set (consumed in `HandleTagValueUpdate`), and the arm is
consumed. An arm that expires unconsumed
emits nothing; the debugger renders the trigger node as `no echo within 30 s`. emits nothing; the debugger renders the trigger node as `no echo within 30 s`.
The arm exists only for traced writes, so production behaviour is unchanged. Alarm cascades that The arm exists only for traced writes, so production behaviour is unchanged. Alarm cascades that
@@ -147,38 +155,44 @@ the echo also fires inherit the parent id exactly as static cascades do today.
### 5.4 Live-run inline transport ### 5.4 Live-run inline transport
`RouteToCallResponse` gains additive `ChildExecutionId` (`Guid?`) and `Steps` **A trace step IS an `AuditEvent` in channel `ScriptTrace`** — there is no separate wire record.
(`IReadOnlyList<FlowStep>?`, the same records that became the site's audit rows, in emission `RouteToCallResponse` gains additive `ChildExecutionId` (`Guid?`) and `TraceEvents`
order). The site collects steps in a per-run `FlowTraceCollector` held by `ScriptRuntimeContext` (`IReadOnlyList<AuditEvent>?`, the very events that became the site's audit rows, in emission
and attaches the list to the `ScriptCallResult` → `RouteToCallResponse` when `Trace` is set. order); on the wire it is `repeated AuditEventDto trace_events` (the DTO `sitestream.proto`
Capped at `MaxTraceStepsPerRun` (500); beyond it the collector keeps the first 500, sets already defines). The site collects events in a per-run `FlowTraceCollector` held by
`Truncated`, and the audit rows remain complete. A site on an older build ignores `Trace`, returns `ScriptRuntimeContext` and attaches the list to the `ScriptCallResult` → `RouteToCallResponse`
no `Steps`, and the UI labels its lane `site not traced (older build)`. when `Trace` is set. Capped at `MaxTraceEventsPerRun` (500); beyond it the collector keeps the
first 500, sets `Truncated`, and the audit rows remain complete. A site on an older build ignores
`Trace`, returns no `TraceEvents`, and the UI labels its lane `site not traced (older build)`.
`FlowStep` is a **central-side projection** (`Commons/Types/Debugging/FlowStepProjector`)
computed from `AuditEvent` + `AuditDetails`; nothing serializes it.
For the value-change entry there is no routed reply to ride on. The debugger instead subscribes For the value-change entry there is no routed reply to ride on. The debugger instead subscribes
to the instance's debug stream (`SubscribeDebugView`, the Debug View's existing path) for the run's to the instance's debug stream (`SubscribeDebugView`, the Debug View's existing path) for the run's
duration and takes the spawned run's `ExecutionId` from the stamped `AttributeValueChanged`, then duration and takes the spawned run's `ExecutionId` from the stamped `AttributeValueChanged`, then
pulls the site's trace rows for that id via a new `PullTraceSteps` RPC on `SiteStreamService` pulls the site's trace rows for that id via a new `PullTraceEvents` RPC on `SiteStreamService`
(same PSK interceptor as `PullAuditEvents`; reads the site audit SQLite by `ExecutionId`). One (same PSK interceptor as `PullAuditEvents`; reads the site audit SQLite by `ExecutionId` /
additive RPC, no streaming change. `ParentExecutionId`). One additive RPC, no streaming change.
## 6. Message and wire changes (all additive) ## 6. Message and wire changes (all additive)
| Contract | Change | | Contract | Change |
|----------|--------| |----------|--------|
| `RouteToCallRequest` | `+ bool Trace = false` | | `RouteToCallRequest` | `+ bool Trace = false` |
| `RouteToCallResponse` | `+ Guid? ChildExecutionId`, `+ IReadOnlyList<FlowStep>? Steps` | | `RouteToCallResponse` | `+ Guid? ChildExecutionId`, `+ IReadOnlyList<AuditEvent>? TraceEvents` |
| `ScriptCallRequest` / `ScriptCallResult` | `+ bool Trace`; `+ Guid ExecutionId`, `+ IReadOnlyList<FlowStep>? Steps` on the result | | `ScriptCallRequest` / `ScriptCallResult` | `+ bool Trace`; `+ Guid? ExecutionId`, `+ IReadOnlyList<AuditEvent>? TraceEvents` on the result |
| `WriteAttributeBatchRequest`, `WaitForAttributeRequest` | `+ Guid? SourceExecutionId`, `+ bool Trace` | | `RouteToSetAttributesRequest` | `+ Guid? ParentExecutionId`, `+ bool Trace` |
| `SetStaticAttributeCommand`, `AttributeValueChanged` | `+ bool Trace` (id already present) | | `SetStaticAttributeCommand`, `AttributeValueChanged` | `+ Guid? SourceExecutionId` (where missing), `+ bool Trace` |
| `site_command.proto` | new fields on `RouteToCallRequestDto`/`ResponseDto`; new `FlowStepDto`; regenerate with `docker/regen-proto.sh sitecommand` | | `site_command.proto` | `trace` on `RouteToCallRequestDto`/`RouteToSetAttributesRequestDto`; `child_execution_id` + `repeated AuditEventDto trace_events` on `RouteToCallResponseDto` (no `FlowStepDto`); regenerate with `docker/regen-proto.sh sitecommand` |
| `sitestream.proto` | `+ PullTraceSteps` RPC (`PullTraceStepsRequest { execution_id }` → `PullTraceStepsResponse { repeated FlowStepDto }`); `AttributeValueUpdate` `+ string source_execution_id = 7` so the debug stream can hand the debugger the spawned id | | `sitestream.proto` | `+ PullTraceEvents` RPC (`PullTraceEventsRequest { repeated execution_ids, since_utc, max_events }` → `PullTraceEventsResponse { repeated AuditEventDto events }`); `AttributeValueUpdate` `+ string source_execution_id = 7`, `+ bool trace = 8` so the debug stream can hand the debugger the spawned id |
| `ApiMethod` | `+ TraceAllCalls bit NOT NULL DEFAULT 0` (EF migration + manual SQL script) | | `ApiMethod` | `+ TraceAllCalls bit NOT NULL DEFAULT 0` (EF migration + manual SQL script) |
| `AuditChannel`/`AuditKind` | `ScriptTrace`; `RouteCall`, `ScriptRun`, `AttributeWrite`, `AttributeWait`, `TriggerFired` | | `AuditChannel`/`AuditKind` | `ScriptTrace`; `RouteCall`, `ScriptRun`, `AttributeWrite`, `AttributeWait` |
`FlowStep` (Commons `Types/Debugging/FlowStep.cs`): `StepId`, `ExecutionId`, `ParentExecutionId`, `FlowStep` (Commons `Types/Debugging/FlowStep.cs`, **projection only — never on the wire**):
`Kind`, `Lane` (`Central`/`Site`/`Tags`), `Title`, `StartedAtUtc`, `DurationMs`, `Outcome`, `EventId`, `ExecutionId`, `ParentExecutionId`, `Kind`, `Lane` (`Central`/`Site`/`Tags`), `Title`,
`InputJson`, `OutputJson`, `Where` (site, node, instance, child path, connection), `SourceLine?`. `Subtitle`, `OccurredAtUtc`, `DurationMs`, `Outcome`, `InputJson`, `OutputJson`, `ErrorMessage`,
`SiteId`/`Node`/`Instance`/`Target`, and a kind-specific `ScriptTraceExtra` parsed from
`AuditDetails.Extra`.
Field numbers never reused; proto3 defaults are the negotiation, as for stream batching. Field numbers never reused; proto3 defaults are the negotiation, as for stream batching.
## 7. Central components ## 7. Central components
@@ -186,22 +200,28 @@ Field numbers never reused; proto3 defaults are the negotiation, as for stream b
### 7.1 `FlowDebugService` (Central UI, scoped) ### 7.1 `FlowDebugService` (Central UI, scoped)
- `PredictAsync(entry)` → §8. - `PredictAsync(entry)` → §8.
- `RunLiveAsync(entry, inputs, operator)` → dispatches a `MgmtFlowDebugRunCommand` through the - `RunMethodAsync` / `RunValueChangeAsync` / `ReplayAsync` → Live runs and Replay are
`ManagementActor` singleton so the method script and its `Route.To` calls execute on the `FlowDebugRunMethodCommand` / `FlowDebugRunValueChangeCommand` / `FlowDebugReplayCommand` on the
**active** central node, honouring `IActiveNodeGate`'s rule that a standby never runs method `ManagementActor` (Deployer role, long-running timeout). The `ManagementActor` is a **per-node**
scripts. The actor hands the run to a detached task (`PipeTo`) so a 45 s method does not block actor (`AkkaHostedService`, `/user/management`), not a singleton. The run-method handler
its mailbox. Inbound entry: `InboundScriptExecutor.ExecuteAsync(method, inputs, route, timeout, consults `IActiveNodeGate` and refuses on a standby with error code `STANDBY_NODE` naming the
parentExecutionId: root, trace: true)` with a `RouteHelper` whose `Trace` flag is set; the design rule (Component-InboundAPI: method scripts and `Route.To` never run on a standby);
central steps (request, method run, route call, response) are recorded by the service itself and Traefik routes UI traffic to the active node so this is rare. The Central UI calls it through
the site steps arrive on `RouteToCallResponse.Steps`. Value-change entry: §5.3 + §5.4. `ManagementActorHolder` (the `DataConnectionMoveService` pattern) and the CLI through
`/management` — one path, one audit story. The inbound row for a Live run is written by the
handler via `ICentralAuditWriter` (kind `InboundRequest`, actor = operator username,
`Extra.source = "flow-debugger"`). Inbound entry: `InboundScriptExecutor.ExecuteAsync(method,
inputs, route, timeout, parentExecutionId: root, trace: true)` with a `RouteHelper` whose
`Trace` flag is set and whose inline trace sink collects the `RouteCall` row plus the site's
`TraceEvents`. Value-change entry: §5.3 + §5.4.
- `ReplayAsync(executionId)` → `IAuditLogRepository.GetExecutionTreeAsync` + a new - `ReplayAsync(executionId)` → `IAuditLogRepository.GetExecutionTreeAsync` + a new
`GetTraceStepsAsync(rootExecutionId)` (all `ScriptTrace` rows in the tree, plus the `GetTraceEventsAsync(rootExecutionId)` (all `ScriptTrace` rows in the tree, plus the
`InboundRequest` row) → `FlowGraphBuilder`. `InboundRequest` row) → `FlowGraphBuilder`.
### 7.2 `FlowGraphBuilder` (pure, Commons or Central UI) ### 7.2 `FlowGraphBuilder` (pure, Commons or Central UI)
Steps → `FlowGraph { Lanes, Nodes, Edges }`. Lane assignment by `Where` (central node → Central; Steps → `FlowGraph { Lanes, Nodes, Edges }`. Lane assignment by `Where` (central node → Central;
site node → Site; `AttributeWrite`/`TriggerFired` → Tags). Edges follow emission order within an site node → Site; `AttributeWrite` → Tags). Edges follow emission order within an
execution and parent→child across executions; an edge whose endpoints sit on different lanes is execution and parent→child across executions; an edge whose endpoints sit on different lanes is
marked `CrossesBoundary`. A predicted graph and a real graph share the model; the page overlays marked `CrossesBoundary`. A predicted graph and a real graph share the model; the page overlays
them (predicted-but-not-executed → greyed). them (predicted-but-not-executed → greyed).
@@ -250,20 +270,23 @@ surface); the walker is syntax-only and never executes anything.
restricts which instances a value-change entry may target. restricts which instances a value-change entry may target.
- Trace payloads pass through the existing redaction policy (auth headers redacted, 8 KB cap, - Trace payloads pass through the existing redaction policy (auth headers redacted, 8 KB cap,
64 KB on error rows). Console output is capped the same way. 64 KB on error rows). Console output is capped the same way.
- Nothing here touches the PSK boundary; `PullTraceSteps` sits behind `ControlPlaneAuthInterceptor`. - Nothing here touches the PSK boundary; `PullTraceEvents` sits behind `ControlPlaneAuthInterceptor`.
## 10. Volume and failure behaviour ## 10. Volume and failure behaviour
- Untraced production is unchanged except one `RouteCall` row per routed call (inbound-driven). - Untraced production is unchanged except one `RouteCall` row per routed call (inbound-driven).
- A traced Delmia call emits ≈ 1 `InboundRequest` + 1 `RouteCall` + 1 `ScriptRun` + 2 - A traced Delmia call emits ≈ 1 `InboundRequest` + 1 `RouteCall` + 1 `ScriptRun` + 2
`AttributeWrite` + 1 `AttributeWait` + 1 `TriggerFired` ≈ 7 rows. `TraceAllCalls` on a `AttributeWrite` + 1 `AttributeWait` + 1 value-change `ScriptRun` ≈ 7 rows. `TraceAllCalls` on a
100-call/day method is ~700 rows/day. 100-call/day method is ~700 rows/day.
- Trace emission is best-effort like all audit (a failed trace write never fails the run). - Trace emission is best-effort like all audit (a failed trace write never fails the run).
- Live run timeout = method timeout + `FlowDebugRunGrace` (10 s); a timeout renders whatever steps - Live run timeout = method timeout + `FlowDebugRunGrace` (10 s); a timeout renders whatever steps
arrived and marks the run `Timed out`. The site's trace rows still land later, so Replay of the arrived and marks the run `Timed out`. The site's trace rows still land later, so Replay of the
same ExecutionId is complete. same ExecutionId is complete.
- Older site build: no `Steps`, lane labelled; older central build: ignores new fields; proto3 - Older site build: no `TraceEvents`, lane labelled; older central build: ignores new fields; proto3
defaults negotiate as for stream batching. defaults negotiate as for stream batching.
- **Mixed-version caveat:** an OLD central ingesting `ScriptTrace` rows from a NEW site parses
unknown channel/kind names to the `AuditRowProjection` fallbacks (`ApiInbound`/`InboundRequest`);
upgrade central first.
## 11. Out of scope (recorded so they are not rediscovered) ## 11. Out of scope (recorded so they are not rediscovered)
@@ -271,23 +294,24 @@ surface); the walker is syntax-only and never executes anything.
shared scripts (they execute inline; only `Scripts.Call` spawns a traceable child). Live shared scripts (they execute inline; only `Scripts.Call` spawns a traceable child). Live
step-by-step / breakpoints. Editing a script from the debugger. Cross-run comparison. A step-by-step / breakpoints. Editing a script from the debugger. Cross-run comparison. A
`Vertical trace` list toggle (mockup Direction B) — cheap follow-up on the same `FlowGraph`. `Vertical trace` list toggle (mockup Direction B) — cheap follow-up on the same `FlowGraph`.
Line-level pinning (Direction C) — needs per-statement instrumentation. Line-level pinning (Direction C) — needs per-statement instrumentation. Console output capture
on `ScriptRun` rows (no script console exists today).
## 12. Testing ## 12. Testing
- **Pure:** `FlowGraphBuilderTests` (lane/edge/boundary rules, truncation), `FlowShapePredictorTests` - **Pure:** `FlowGraphBuilderTests` (lane/edge/boundary rules, truncation), `FlowShapePredictorTests`
(each recognised site, constant vs input-dependent target, one-hop route resolution, depth cap, (each recognised site, constant vs input-dependent target, one-hop route resolution, depth cap,
early-return annotation), `FlowTraceCollectorTests` (cap + `Truncated`). early-return annotation), `FlowTraceCollectorTests` (cap + `Truncated`).
- **Site runtime:** `InstanceActor` traced write emits `AttributeWrite` and arms; echo consumes the - **Site runtime:** `ScriptRuntimeContext` traced write emits `AttributeWrite`; `InstanceActor` arms; echo consumes the
arm and stamps `SourceExecutionId`/`Trace`; expiry emits nothing; untraced write emits nothing arm and stamps `SourceExecutionId`/`Trace`; expiry emits nothing; untraced write emits nothing
and arms nothing (the production-unchanged pin). `ScriptRunLauncher` emits `ScriptRun` only when and arms nothing (the production-unchanged pin). `ScriptRunLauncher` emits `ScriptRun` only when
traced. `HandleWaitForAttribute` emits `AttributeWait` for satisfied/timeout. traced. `ScriptRuntimeContext.WaitInternal` emits `AttributeWait` for satisfied/timeout.
- **Inbound API:** `RouteHelper.Call` always writes `RouteCall`; `ChildExecutionId` round-trips; - **Inbound API:** `RouteHelper.Call` always writes `RouteCall`; `ChildExecutionId` round-trips;
`Trace` propagates to `RouteToCallRequest`. `Trace` propagates to `RouteToCallRequest`.
- **Communication:** DTO mapper round-trips for every new field; an old-shape response (no - **Communication:** DTO mapper round-trips for every new field; an old-shape response (no
`Steps`) maps to null. `TraceEvents`) maps to null.
- **Audit:** new kinds accepted by ingest; per-channel retention deletes `ScriptTrace` rows early; - **Audit:** new kinds accepted by ingest; per-channel retention deletes `ScriptTrace` rows early;
`GetTraceStepsAsync` returns the tree's rows in order. `GetTraceEventsAsync` returns the tree's rows in order.
- **Central UI (bUnit):** page renders predicted graph, overlays real graph, Run button hidden - **Central UI (bUnit):** page renders predicted graph, overlays real graph, Run button hidden
without Deployer, Replay degrades to the collapsed `not traced` node. without Deployer, Replay degrades to the collapsed `not traced` node.
- **Rig:** extend `docker/verify-delmia-notifier.sh` with step H — a Live run of - **Rig:** extend `docker/verify-delmia-notifier.sh` with step H — a Live run of
@@ -298,7 +322,7 @@ surface); the walker is syntax-only and never executes anything.
`Component-CentralUI.md` (new page), `Component-InboundAPI.md` (RouteCall row, TraceAllCalls, `Component-CentralUI.md` (new page), `Component-InboundAPI.md` (RouteCall row, TraceAllCalls,
Live-run actor), `Component-SiteRuntime.md` (trace collector, trace arm, gated emission), Live-run actor), `Component-SiteRuntime.md` (trace collector, trace arm, gated emission),
`Component-AuditLog.md` (ScriptTrace channel, retention override, PullTraceSteps), `Component-AuditLog.md` (ScriptTrace channel, retention override, PullTraceEvents),
`Component-Communication.md` (proto changes), `Component-ScriptAnalysis.md` (predictor), `Component-Communication.md` (proto changes), `Component-ScriptAnalysis.md` (predictor),
`Component-ConfigurationDatabase.md` (migration), `README.md` component table if a component is `Component-ConfigurationDatabase.md` (migration), `README.md` component table if a component is
added (none planned — the debugger is a Central UI page plus seams in existing components), added (none planned — the debugger is a Central UI page plus seams in existing components),