From 61d1785fcb51621d1520ae503e1473a61bb61253 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sat, 5 Sep 2026 23:26:28 -0400 Subject: [PATCH] =?UTF-8?q?docs(plans):=20flow=20debugger=20design=20?= =?UTF-8?q?=E2=80=94=20trace=20steps=20are=20audit=20events;=20mgmt-comman?= =?UTF-8?q?d=20execution=20path;=20context-emitted=20write/wait=20rows?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP --- docs/plans/2026-09-05-flow-debugger-design.md | 120 +++++++++++------- 1 file changed, 72 insertions(+), 48 deletions(-) diff --git a/docs/plans/2026-09-05-flow-debugger-design.md b/docs/plans/2026-09-05-flow-debugger-design.md index 7b932cc0..d655217d 100644 --- a/docs/plans/2026-09-05-flow-debugger-design.md +++ b/docs/plans/2026-09-05-flow-debugger-design.md @@ -92,10 +92,15 @@ A new `AuditChannel.ScriptTrace` with kinds: | Kind | Emitted by | Payload (`DetailsJson`) | |------|-----------|-------------------------| | `RouteCall` | Central `RouteHelper.Call` (**always-on**, not gated) | target instance, script, args (redacted per existing policy), return value, success/error, duration ms, `ChildExecutionId` | -| `ScriptRun` | Site `ScriptRunLauncher` on run completion (gated) | script, instance, trigger kind, parameters, return value, success/error, duration ms, console output (capped 8 KB like other payloads), source node | -| `AttributeWrite` | Site `InstanceActor` write handlers (gated) | one row per batch: attributes[{name, before, written, quality, ack ms}], trigger attribute, static/data-sourced | -| `AttributeWait` | Site `InstanceActor.HandleWaitForAttribute` (gated) | attribute, predicate/target, timeout, outcome (satisfied/timeout/cancelled), elapsed ms, observed value | -| `TriggerFired` | Site `ScriptActor` when a value-change run is spawned with a stamped `SourceExecutionId` (gated) | attribute, old/new value, source (static write / device echo) | +| `ScriptRun` | Site `ScriptRunLauncher` on run completion (gated) | script, instance, trigger `{kind, attribute, value, source}` (a value-change run's trigger rides here — there is no separate `TriggerFired` kind), parameters, return value, success/error, duration ms, source node | +| `AttributeWrite` | Site `ScriptRuntimeContext.SetAttribute` / `WriteBatchAndWait` (gated) | one row per write or batch: writes[{attribute, written, success, error}], trigger attribute | +| `AttributeWait` | Site `ScriptRuntimeContext.WaitInternal` (gated) | attribute, predicate/target, timeout, outcome (satisfied/timeout/cancelled/error), elapsed ms, observed value | + +**Who emits `AttributeWrite`/`AttributeWait` (planning correction 2026-09-05):** every accessor path +funnels into three `ScriptRuntimeContext` methods — `SetAttribute`, `WriteBatchAndWait` and +`WaitInternal` — and the context already holds `IAuditWriter`, `_executionId` and +`_parentExecutionId`, so the `InstanceActor` needs no new ids and no new request fields. Batch-write +echoes are **not** armed in v1 — only `SetStaticAttributeCommand` writes arm (§5.3). Every row carries the run's `ExecutionId` and `ParentExecutionId` exactly as the existing channels do, so the Execution Chain walk, `IX_AuditLog_ParentExecution`, telemetry forwarding, reconciliation @@ -132,14 +137,17 @@ The debugger writes the chosen attribute through the instance actor's normal pat - **Static attribute:** `RouteToSetAttributes` → `SetStaticAttributeCommand` with `SourceExecutionId = `, `Trace = true`. The existing cascade already stamps the resulting `AttributeValueChanged` and the spawned run's `ParentExecutionId`. Nothing new. -- **Data-sourced attribute:** a single-attribute `WriteAttributeBatchRequest` (new additive - `SourceExecutionId` + `Trace`) → DCL `WriteTagBatchRequest`. The device echo comes back through - the DCL as an ordinary `AttributeValueChanged` with no source. **Trace arm:** when - `InstanceActor` performs a traced write it records `(attribute → sourceExecutionId, expiresAt = +- **Data-sourced attribute:** the **same** `RouteToSetAttributes` → `SetStaticAttributeCommand` + path — `InstanceActor` already dispatches data-sourced attributes to the DCL + (`HandleSetDataAttribute` → `WriteTagRequest`); no `WriteAttributeBatchRequest` change is needed. + The device echo comes back through the DCL as an ordinary `AttributeValueChanged` with no + source. **Trace arm:** when `HandleSetDataAttribute` sees `command.Trace && + command.SourceExecutionId != null` it records `(attribute → sourceExecutionId, expiresAt = now + TraceArmWindow)` (default 30 s, `SiteRuntimeOptions`, one entry per attribute, newest wins). The next DCL value change for that attribute — whatever the value, so a device that clamps or rejects the write still shows what it actually did — is published with - `SourceExecutionId` and `Trace` set, and the arm is consumed. An arm that expires unconsumed + `SourceExecutionId` and `Trace` set (consumed in `HandleTagValueUpdate`), and the arm is + consumed. An arm that expires unconsumed emits nothing; the debugger renders the trigger node as `no echo within 30 s`. The arm exists only for traced writes, so production behaviour is unchanged. Alarm cascades that @@ -147,38 +155,44 @@ the echo also fires inherit the parent id exactly as static cascades do today. ### 5.4 Live-run inline transport -`RouteToCallResponse` gains additive `ChildExecutionId` (`Guid?`) and `Steps` -(`IReadOnlyList?`, the same records that became the site's audit rows, in emission -order). The site collects steps in a per-run `FlowTraceCollector` held by `ScriptRuntimeContext` -and attaches the list to the `ScriptCallResult` → `RouteToCallResponse` when `Trace` is set. -Capped at `MaxTraceStepsPerRun` (500); beyond it the collector keeps the first 500, sets -`Truncated`, and the audit rows remain complete. A site on an older build ignores `Trace`, returns -no `Steps`, and the UI labels its lane `site not traced (older build)`. +**A trace step IS an `AuditEvent` in channel `ScriptTrace`** — there is no separate wire record. +`RouteToCallResponse` gains additive `ChildExecutionId` (`Guid?`) and `TraceEvents` +(`IReadOnlyList?`, the very events that became the site's audit rows, in emission +order); on the wire it is `repeated AuditEventDto trace_events` (the DTO `sitestream.proto` +already defines). The site collects events in a per-run `FlowTraceCollector` held by +`ScriptRuntimeContext` and attaches the list to the `ScriptCallResult` → `RouteToCallResponse` +when `Trace` is set. Capped at `MaxTraceEventsPerRun` (500); beyond it the collector keeps the +first 500, sets `Truncated`, and the audit rows remain complete. A site on an older build ignores +`Trace`, returns no `TraceEvents`, and the UI labels its lane `site not traced (older build)`. +`FlowStep` is a **central-side projection** (`Commons/Types/Debugging/FlowStepProjector`) +computed from `AuditEvent` + `AuditDetails`; nothing serializes it. For the value-change entry there is no routed reply to ride on. The debugger instead subscribes to the instance's debug stream (`SubscribeDebugView`, the Debug View's existing path) for the run's duration and takes the spawned run's `ExecutionId` from the stamped `AttributeValueChanged`, then -pulls the site's trace rows for that id via a new `PullTraceSteps` RPC on `SiteStreamService` -(same PSK interceptor as `PullAuditEvents`; reads the site audit SQLite by `ExecutionId`). One -additive RPC, no streaming change. +pulls the site's trace rows for that id via a new `PullTraceEvents` RPC on `SiteStreamService` +(same PSK interceptor as `PullAuditEvents`; reads the site audit SQLite by `ExecutionId` / +`ParentExecutionId`). One additive RPC, no streaming change. ## 6. Message and wire changes (all additive) | Contract | Change | |----------|--------| | `RouteToCallRequest` | `+ bool Trace = false` | -| `RouteToCallResponse` | `+ Guid? ChildExecutionId`, `+ IReadOnlyList? Steps` | -| `ScriptCallRequest` / `ScriptCallResult` | `+ bool Trace`; `+ Guid ExecutionId`, `+ IReadOnlyList? Steps` on the result | -| `WriteAttributeBatchRequest`, `WaitForAttributeRequest` | `+ Guid? SourceExecutionId`, `+ bool Trace` | -| `SetStaticAttributeCommand`, `AttributeValueChanged` | `+ bool Trace` (id already present) | -| `site_command.proto` | new fields on `RouteToCallRequestDto`/`ResponseDto`; new `FlowStepDto`; regenerate with `docker/regen-proto.sh sitecommand` | -| `sitestream.proto` | `+ PullTraceSteps` RPC (`PullTraceStepsRequest { execution_id }` → `PullTraceStepsResponse { repeated FlowStepDto }`); `AttributeValueUpdate` `+ string source_execution_id = 7` so the debug stream can hand the debugger the spawned id | +| `RouteToCallResponse` | `+ Guid? ChildExecutionId`, `+ IReadOnlyList? TraceEvents` | +| `ScriptCallRequest` / `ScriptCallResult` | `+ bool Trace`; `+ Guid? ExecutionId`, `+ IReadOnlyList? TraceEvents` on the result | +| `RouteToSetAttributesRequest` | `+ Guid? ParentExecutionId`, `+ bool Trace` | +| `SetStaticAttributeCommand`, `AttributeValueChanged` | `+ Guid? SourceExecutionId` (where missing), `+ bool Trace` | +| `site_command.proto` | `trace` on `RouteToCallRequestDto`/`RouteToSetAttributesRequestDto`; `child_execution_id` + `repeated AuditEventDto trace_events` on `RouteToCallResponseDto` (no `FlowStepDto`); regenerate with `docker/regen-proto.sh sitecommand` | +| `sitestream.proto` | `+ PullTraceEvents` RPC (`PullTraceEventsRequest { repeated execution_ids, since_utc, max_events }` → `PullTraceEventsResponse { repeated AuditEventDto events }`); `AttributeValueUpdate` `+ string source_execution_id = 7`, `+ bool trace = 8` so the debug stream can hand the debugger the spawned id | | `ApiMethod` | `+ TraceAllCalls bit NOT NULL DEFAULT 0` (EF migration + manual SQL script) | -| `AuditChannel`/`AuditKind` | `ScriptTrace`; `RouteCall`, `ScriptRun`, `AttributeWrite`, `AttributeWait`, `TriggerFired` | +| `AuditChannel`/`AuditKind` | `ScriptTrace`; `RouteCall`, `ScriptRun`, `AttributeWrite`, `AttributeWait` | -`FlowStep` (Commons `Types/Debugging/FlowStep.cs`): `StepId`, `ExecutionId`, `ParentExecutionId`, -`Kind`, `Lane` (`Central`/`Site`/`Tags`), `Title`, `StartedAtUtc`, `DurationMs`, `Outcome`, -`InputJson`, `OutputJson`, `Where` (site, node, instance, child path, connection), `SourceLine?`. +`FlowStep` (Commons `Types/Debugging/FlowStep.cs`, **projection only — never on the wire**): +`EventId`, `ExecutionId`, `ParentExecutionId`, `Kind`, `Lane` (`Central`/`Site`/`Tags`), `Title`, +`Subtitle`, `OccurredAtUtc`, `DurationMs`, `Outcome`, `InputJson`, `OutputJson`, `ErrorMessage`, +`SiteId`/`Node`/`Instance`/`Target`, and a kind-specific `ScriptTraceExtra` parsed from +`AuditDetails.Extra`. Field numbers never reused; proto3 defaults are the negotiation, as for stream batching. ## 7. Central components @@ -186,22 +200,28 @@ Field numbers never reused; proto3 defaults are the negotiation, as for stream b ### 7.1 `FlowDebugService` (Central UI, scoped) - `PredictAsync(entry)` → §8. -- `RunLiveAsync(entry, inputs, operator)` → dispatches a `MgmtFlowDebugRunCommand` through the - `ManagementActor` singleton so the method script and its `Route.To` calls execute on the - **active** central node, honouring `IActiveNodeGate`'s rule that a standby never runs method - scripts. The actor hands the run to a detached task (`PipeTo`) so a 45 s method does not block - its mailbox. Inbound entry: `InboundScriptExecutor.ExecuteAsync(method, inputs, route, timeout, - parentExecutionId: root, trace: true)` with a `RouteHelper` whose `Trace` flag is set; the - central steps (request, method run, route call, response) are recorded by the service itself and - the site steps arrive on `RouteToCallResponse.Steps`. Value-change entry: §5.3 + §5.4. +- `RunMethodAsync` / `RunValueChangeAsync` / `ReplayAsync` → Live runs and Replay are + `FlowDebugRunMethodCommand` / `FlowDebugRunValueChangeCommand` / `FlowDebugReplayCommand` on the + `ManagementActor` (Deployer role, long-running timeout). The `ManagementActor` is a **per-node** + actor (`AkkaHostedService`, `/user/management`), not a singleton. The run-method handler + consults `IActiveNodeGate` and refuses on a standby with error code `STANDBY_NODE` naming the + design rule (Component-InboundAPI: method scripts and `Route.To` never run on a standby); + Traefik routes UI traffic to the active node so this is rare. The Central UI calls it through + `ManagementActorHolder` (the `DataConnectionMoveService` pattern) and the CLI through + `/management` — one path, one audit story. The inbound row for a Live run is written by the + handler via `ICentralAuditWriter` (kind `InboundRequest`, actor = operator username, + `Extra.source = "flow-debugger"`). Inbound entry: `InboundScriptExecutor.ExecuteAsync(method, + inputs, route, timeout, parentExecutionId: root, trace: true)` with a `RouteHelper` whose + `Trace` flag is set and whose inline trace sink collects the `RouteCall` row plus the site's + `TraceEvents`. Value-change entry: §5.3 + §5.4. - `ReplayAsync(executionId)` → `IAuditLogRepository.GetExecutionTreeAsync` + a new - `GetTraceStepsAsync(rootExecutionId)` (all `ScriptTrace` rows in the tree, plus the + `GetTraceEventsAsync(rootExecutionId)` (all `ScriptTrace` rows in the tree, plus the `InboundRequest` row) → `FlowGraphBuilder`. ### 7.2 `FlowGraphBuilder` (pure, Commons or Central UI) Steps → `FlowGraph { Lanes, Nodes, Edges }`. Lane assignment by `Where` (central node → Central; -site node → Site; `AttributeWrite`/`TriggerFired` → Tags). Edges follow emission order within an +site node → Site; `AttributeWrite` → Tags). Edges follow emission order within an execution and parent→child across executions; an edge whose endpoints sit on different lanes is marked `CrossesBoundary`. A predicted graph and a real graph share the model; the page overlays them (predicted-but-not-executed → greyed). @@ -250,20 +270,23 @@ surface); the walker is syntax-only and never executes anything. restricts which instances a value-change entry may target. - Trace payloads pass through the existing redaction policy (auth headers redacted, 8 KB cap, 64 KB on error rows). Console output is capped the same way. -- Nothing here touches the PSK boundary; `PullTraceSteps` sits behind `ControlPlaneAuthInterceptor`. +- Nothing here touches the PSK boundary; `PullTraceEvents` sits behind `ControlPlaneAuthInterceptor`. ## 10. Volume and failure behaviour - Untraced production is unchanged except one `RouteCall` row per routed call (inbound-driven). - A traced Delmia call emits ≈ 1 `InboundRequest` + 1 `RouteCall` + 1 `ScriptRun` + 2 - `AttributeWrite` + 1 `AttributeWait` + 1 `TriggerFired` ≈ 7 rows. `TraceAllCalls` on a + `AttributeWrite` + 1 `AttributeWait` + 1 value-change `ScriptRun` ≈ 7 rows. `TraceAllCalls` on a 100-call/day method is ~700 rows/day. - Trace emission is best-effort like all audit (a failed trace write never fails the run). - Live run timeout = method timeout + `FlowDebugRunGrace` (10 s); a timeout renders whatever steps arrived and marks the run `Timed out`. The site's trace rows still land later, so Replay of the same ExecutionId is complete. -- Older site build: no `Steps`, lane labelled; older central build: ignores new fields; proto3 +- Older site build: no `TraceEvents`, lane labelled; older central build: ignores new fields; proto3 defaults negotiate as for stream batching. +- **Mixed-version caveat:** an OLD central ingesting `ScriptTrace` rows from a NEW site parses + unknown channel/kind names to the `AuditRowProjection` fallbacks (`ApiInbound`/`InboundRequest`); + upgrade central first. ## 11. Out of scope (recorded so they are not rediscovered) @@ -271,23 +294,24 @@ surface); the walker is syntax-only and never executes anything. shared scripts (they execute inline; only `Scripts.Call` spawns a traceable child). Live step-by-step / breakpoints. Editing a script from the debugger. Cross-run comparison. A `Vertical trace` list toggle (mockup Direction B) — cheap follow-up on the same `FlowGraph`. - Line-level pinning (Direction C) — needs per-statement instrumentation. + Line-level pinning (Direction C) — needs per-statement instrumentation. Console output capture + on `ScriptRun` rows (no script console exists today). ## 12. Testing - **Pure:** `FlowGraphBuilderTests` (lane/edge/boundary rules, truncation), `FlowShapePredictorTests` (each recognised site, constant vs input-dependent target, one-hop route resolution, depth cap, early-return annotation), `FlowTraceCollectorTests` (cap + `Truncated`). -- **Site runtime:** `InstanceActor` traced write emits `AttributeWrite` and arms; echo consumes the +- **Site runtime:** `ScriptRuntimeContext` traced write emits `AttributeWrite`; `InstanceActor` arms; echo consumes the arm and stamps `SourceExecutionId`/`Trace`; expiry emits nothing; untraced write emits nothing and arms nothing (the production-unchanged pin). `ScriptRunLauncher` emits `ScriptRun` only when - traced. `HandleWaitForAttribute` emits `AttributeWait` for satisfied/timeout. + traced. `ScriptRuntimeContext.WaitInternal` emits `AttributeWait` for satisfied/timeout. - **Inbound API:** `RouteHelper.Call` always writes `RouteCall`; `ChildExecutionId` round-trips; `Trace` propagates to `RouteToCallRequest`. - **Communication:** DTO mapper round-trips for every new field; an old-shape response (no - `Steps`) maps to null. + `TraceEvents`) maps to null. - **Audit:** new kinds accepted by ingest; per-channel retention deletes `ScriptTrace` rows early; - `GetTraceStepsAsync` returns the tree's rows in order. + `GetTraceEventsAsync` returns the tree's rows in order. - **Central UI (bUnit):** page renders predicted graph, overlays real graph, Run button hidden without Deployer, Replay degrades to the collapsed `not traced` node. - **Rig:** extend `docker/verify-delmia-notifier.sh` with step H — a Live run of @@ -298,7 +322,7 @@ surface); the walker is syntax-only and never executes anything. `Component-CentralUI.md` (new page), `Component-InboundAPI.md` (RouteCall row, TraceAllCalls, Live-run actor), `Component-SiteRuntime.md` (trace collector, trace arm, gated emission), -`Component-AuditLog.md` (ScriptTrace channel, retention override, PullTraceSteps), +`Component-AuditLog.md` (ScriptTrace channel, retention override, PullTraceEvents), `Component-Communication.md` (proto changes), `Component-ScriptAnalysis.md` (predictor), `Component-ConfigurationDatabase.md` (migration), `README.md` component table if a component is added (none planned — the debugger is a Central UI page plus seams in existing components),