ScadaBridge/docs/plans/2026-09-09-flow-debugger-step-logs-design.md

Flow Debugger — Step Logs and Exceptions (design)

Date: 2026-09-09 Status: implemented 2026-09-09 (plan 2026-09-09-flow-debugger-step-logs.md, 15 tasks). Deltas between this design and what shipped are marked as built in place. Extends: 2026-09-05-flow-debugger-design.md (§5 trace model, §7 central components), 2026-09-06-flow-debugger-outbound-calls.md (folded nodes), register row 110 (console capture on ScriptRun) Mockup: docs/plans/2026-09-09-flow-debugger-step-logs-mockup.html (published as a Claude artifact — link in the session recap)

1. What changes

A traced Flow Debugger run shows, on the node that produced them:

1. The exception a failing step threw — type, message, stack, inner exception — not just the one-line Script 'X' on instance 'Y' failed: … summary the ScriptRun row carries today.
2. Log lines — a new Log global every script surface exposes (Log.Debug/Info/Warn/Error(message, exception?)), plus the warnings the runtime logs on the script's behalf while a helper step runs.

Rendering: a per-level count badge in the node footer; a failed node's subtitle becomes ExceptionType: message; a Log tab in the detail panel (exception block first, then the lines, All | Warn + filter, copy); run-wide n errors · n warnings · n info in the status strip. Live and Replay read the same rows.

2. What exists (verified 2026-09-09)

• Scripts cannot log. ScriptGlobals (site) and InboundScriptContext (central) expose no Log, Logger or ILogger; the four script imports have no Console. ScriptRuntimeContext._logger and _siteEventLogger are private. So "logger messages" needs a new API surface, not a rewiring.
• The failure row exists but lacks the detail. ScriptRunLauncher.RunScriptAsync emits a Failed ScriptRun row on both catch paths (ScriptRunLauncher.cs:458-506) with errorMessage only; errorDetail is never passed (:588) although ScadaBridgeAuditEventFactory.Create accepts it and the DB helpers already fill it with ex.ToString(). FlowStepProjector.Project drops AuditDetails.ErrorDetail. Centrally, InboundScriptExecutor.ExecuteAsync swallows the exception into "Internal script error" / SCRIPT_ERROR (InboundScriptExecutor.cs:548-552) and the runner's inbound row gets that text.
• No execution-id enrichment on node loggers (as of 2026-09-09; since closed in part by Gitea #56, see §9). Zero LogContext.PushProperty calls; Enrich.FromLogContext is not configured; the TraceContextEnricher stamps W3C ids only. site_events has no execution-id column. Capturing what other actors log for a run is therefore out of reach without a separate enrichment initiative (§9).
• The plumbing to reuse: ScriptRuntimeContext.EmitTrace (collector first, writer second, same AuditEvent both ways); FlowTraceCollector (500 events / 1 MB, redacted, Complete() at reply time); RouteHelper.TraceSink and the runner's central collector; GetTraceEventsAsync already returns every ScriptTrace row of the tree; FlowGraphBuilder.Fold already folds multi-row operations into one step; the Test Run sandbox already has a ConsoleOutput pane (SandboxRunResult.ConsoleOutput).

3. Decisions

#	Decision	Rejected / why
D1	A log line is an audit row: new AuditKind.ScriptLog in channel ScriptTrace (appended last in the enum), one row per line, never a node — FlowGraphBuilder folds it into its owning step's new Logs list.	A side channel on the reply (RouteToCallResponse.LogLines) — Replay would have nothing; a second store to redact/retain/purge. Folding into the ScriptRun row's Extra (register row 110's sketch) — the 8 KB payload cap on that row would silently drop lines, and helper-step attribution would need every helper to buffer.
D2	Attribution is explicit. Extra.stepEventId names the step; absent, the line belongs to the execution's anchor (site: the ScriptRun row of that ExecutionId; central: the InboundRequest row, or the debugger's synthesized RouteCall root for a script entry). No anchor and no matching step id → the first step of that execution; no step at all → dropped and counted (FlowGraph.UnattachedLogLines, shown in the strip).	Attribution by time window (OccurredAtUtc inside a step's [start, start+duration]) — step rows record different instants per kind and a 1 ms burst of writes makes it a coin toss.
D3	Exception detail rides errorDetail. Site: EmitScriptRun passes ex.ToString() on the exception path (timeouts stay detail-less — an OperationCanceledException stack is noise). Central: InboundScriptResult gains an internal ErrorDetail (never returned to the HTTP caller); FlowDebugRunner puts it on the inbound row, and the HTTP path stashes it in HttpContext.Items for AuditWriteMiddleware so a TraceAllCalls Replay has it too. FlowStep gains ErrorDetail.	A separate ScriptLog row for the throw — duplicates the failure row and needs an ordering rule against it.
D4	Log global on every surface: ScriptGlobals.Log and InboundScriptContext.Log at runtime; ScriptCompileSurface.Log (CompileLog) and SandboxScriptHost.Log / SandboxInboundScriptHost.Log for parity (the sandbox mirror appends to the Test Run ConsoleOutput). Signature void Debug/Info/Warn/Error(string message, Exception? exception = null); synchronous; never throws into the script. Log is not on ScriptTrustPolicy's deny list (it is a deny-list over API roots), pinned by a test that Log.Info(...) passes the gate and binds on all surfaces.	A Console.WriteLine shim — Console is not imported today and a stdout metaphor has no level.
D5	Untraced runs pay nothing. Log.* always forwards to the node's ILogger (category ScadaBridge.ScriptLog, structured Instance/Script/ExecutionId/Level) and emits a ScriptLog row only when _isTraced. No site event. TraceAllCalls methods gain lines in Replay at ~1 row per line.	Always auditing lines (volume: timer/value-change scripts run at high rates). A no-op when untraced (authors would lose the messages they wrote).
D6	Runtime lines through one seam. When traced, ScriptRuntimeContext wraps its _logger in a TraceCapturingLogger that forwards everything and copies Information and above into ScriptLog rows with logSource = "runtime". Attribution uses an ambient TraceStepScope (AsyncLocal<Guid?>): the three attribute helpers (SetAttribute, WriteBatchAndWait, WaitInternal) and the sync ExternalSystem.Call pre-mint their step's EventId, open the scope for their body, and pass the id to EmitTrace (new optional eventId parameter). Everything else falls back to the anchor (D2). The launcher's own LogError for the run's exception is not captured as a line — it is already the exception block (D3).	Editing every _logger.LogWarning call site to also emit a row. Capturing Debug (internal chatter).
D7	Bounded. MaxLogLinesPerRun (default 200; SiteRuntimeOptions + InboundApiOptions, validated > 0) and a 2 000-char message cap at the API; past the line cap, lines are dropped and one closing runtime warn row names the dropped count at run end. Inline, ScriptLog rows ride the collector's low-priority lane — as built that is an optional parameter on the existing method, FlowTraceCollector.Add(evt, lowPriority: true), not a fourth public AddLowPriority method (Commons keeps its public method count tight, and every caller already had the event in hand): they stop at max − 50 events / 75 % of the byte budget so the step rows emitted at run end (the ScriptRun anchor is written last) are never crowded out; a dropped low-priority row sets LogsTruncated, surfaced separately from Truncated (see §7 for the shipped wording).	Raising the collector cap (the reply also carries the return value under the 4 MB gRPC limit).
D8	UI rules. Badge: highest severity first, ✕ n danger, ! n warn, ≡ n faint; hidden when no lines. Failed subtitle: Type: message (type without namespace). Detail: Log (n) tab after Output; a step with an exception opens on Log, every other step on Input; the Output tab's existing alert stays and links to Log. Strip: Log: n errors · n warnings · n info counting lines, not steps. Ordering inside a step: (OccurredAtUtc, Extra.seq) — one clock per execution, seq breaks same-ms ties.	Drawing lines as nodes (a 40-line script would be unreadable).
D9	Replay and CLI. GetTraceEventsAsync needs no change (channel filter already includes all of ScriptTrace). CLI `flow-debug run-*	replaytable gains alogscolumn (count) and--show-logs` prints lines indented under their step.

4. Trace model (additive)

ScriptLog row: OccurredAtUtc = line time; Status = Delivered; Target = script name (as ScriptRun); RequestSummary = message; ErrorMessage / ErrorDetail = the attached exception's message / ToString(); ExecutionId / ParentExecutionId / SourceSiteId / SourceInstanceId / SourceNode as the run's other rows. ScriptTraceExtra gains LogLevel (debug|info|warn|error), LogSource (script|runtime), StepEventId (Guid?), Seq (int?). Redaction (payload caps, header/body regexes) applies unchanged; AuditLog:PerChannelRetentionDays:ScriptTrace covers the new kind.

FlowStep gains ErrorDetail (string?) and Logs (IReadOnlyList<FlowLogLine>?); FlowLogLine(EventId, ExecutionId, OccurredAtUtc, Level, Source, Message, ExceptionMessage, ExceptionDetail, StepEventId, Seq) — as built, the line keeps its own ExecutionId (so a folded step can say which execution a line came from, and the cross-step hint can name it) plus the attribution inputs StepEventId and Seq that FlowGraphBuilder sorted by, rather than discarding them at projection time. FlowGraph gains LogsTruncated (bool) and UnattachedLogLines (int). None of this is on the wire.

Mixed versions. An OLD central ingesting ScriptLog rows from a NEW site projects them to FlowStepKind.Unknown nodes (the projector renders unknown kinds rather than dropping them) — the existing rule upgrade central first covers it. An old site sends no lines; the page is unchanged.

5. Site runtime

• ScriptLogger (SiteRuntime/Scripts/), created by ScriptRuntimeContext and exposed as Log; holds the context's EmitTrace delegate, the ILogger, the caps and a per-run counter (CallShared shares the context, so inline shared scripts share the counter and the anchor).
• EmitTrace(kind, …, eventId: Guid? = null); TraceStepScope.Current (AsyncLocal<Guid?>) read when building a ScriptLog row and no explicit step id is given.
• TraceCapturingLogger : ILogger — decorator installed only when _isTraced; IsEnabled delegates to the inner logger OR level >= Information, so a node whose Serilog minimum is Warning still captures Information lines into the trace (they are trace data, not node log output).
• ScriptRunLauncher: pass errorDetail: ex.ToString() on the exception path; before collector?.Complete() ask the context to flush its "dropped lines" closing row.

6. Central

• InboundScriptContext.Log → InboundScriptLogger writing through the executor's existing inline sink (route.TraceSink) and ICentralAuditWriter when traced (InboundScriptExecutor.ExecuteAsync(..., trace) already knows); ILogger always.
• InboundScriptResult + string? ErrorDetail (positional-last, default null). FlowDebugRunner inbound row errorDetail: result.ErrorDetail. EndpointExtensions (the executor.ExecuteAsync call site, :276) → HttpContext.Items[AuditErrorDetailKey] → AuditWriteMiddleware errorDetail: (the HTTP body stays SCRIPT_ERROR).
• FlowGraphBuilder: partition ScriptLog rows out before Fold, attach per D2 after the steps list is final (a folded outbound step's EventId is its first row's, so stepEventId must be the pre-minted first-row id — only the sync helpers pre-mint in v1, which is exactly the set D6 scopes).

7. Central UI

FlowGraphView: footer becomes a flex row (subtitle + badge); NodeClass unchanged; data-test="flow-node-logs". FlowStepDetail: Log tab (data-test="flow-step-tab-log"), exception block (flow-step-exception), line table (flow-step-log-lines), level filter, copy (JS interop navigator.clipboard, loose in bUnit), the cross-step hint ("n more on " selects that step via a new OnSelectStep callback). FlowDebugger: strip counts (flow-log-counts, Log: n errors · n warnings · n info, counting LINES not steps), the flow-logs-truncated badge and the flow-unattached-logs note; ApplyOutcome unchanged (a run with only warning lines is still Completed).

Truncation badge wording and its clearing rule. The badge reads "inline log lines capped — refresh outcomes pulls the rest": it is a this-copy-is-short notice with the remedy in it, not a data-loss warning — the rows exist in full in the audit store, and Replay reads them all. It is driven by FlowGraph.LogsTruncated, which the page recomputes from the rows it currently holds, so a Refresh outcomes that merges in more log rows clears the badge on its own; it must never be latched on the run. (Contrast Truncated, the event-list cap, which is a property of the Live reply itself.)

8. Security

Log messages are script-authored text and pass the same IAuditRedactor as every other summary (secrets in a Log.Info($"token={t}") are scrubbed by the body regexes where they match, and the 8 KB cap applies). Exception detail can carry connection strings in ex.Message for DB helpers — the DB helpers already store ex.ToString() in errorDetail today, so no new exposure class; the redactor's error-row cap (64 KB) bounds it. Nothing new crosses the PSK boundary: ScriptLog rows ride the existing AuditEventDto.

9. Out of scope (recorded)

Live streaming of lines during a long run (rows still arrive with the reply — D4 of the base design); a run-wide chronological log list (belongs to the Direction B list, register row 110); capturing what other actors log for a run (instance actor, DCL) — enrichment done, capture open (Gitea #56): the site script/alarm launcher pushes ExecutionId/InstanceName/ScriptName onto the Serilog LogContext for the run, the inbound executor pushes the request's ExecutionId for the method execution, and the host enables Enrich.FromLogContext(). Because LogContext is an AsyncLocal that does not cross an Akka message hop, the id rides the message instead: Commons.Observability.IExecutionCorrelated (explicit interface member, no wire change) is implemented by SetStaticAttributeCommand, AttributeValueChanged (not when the writer is inferred by the trace arm), ScriptCallRequest, the four RouteTo*Requests, and — via a new additive SourceExecutionId — GetAttributeRequest, WaitForAttributeRequest, WriteAttributeBatchRequest, WriteTagRequest, WriteTagBatchRequest; the Instance, Script, Alarm, Deployment Manager and both DCL actors re-push it per message in AroundReceive (ExecutionLogContext.PushFor; one type test for an uncorrelated message). So a node-log line the Instance Actor or a protocol adapter writes while serving a run's write carries the run's id. Capture done too — see §12 (no site_events schema change: a Serilog sink copies the tagged lines of a traced execution into its ScriptLog rows, and the Flow Debugger lists them in a Node log section); the DCL actors' own Akka ILoggingAdapter lines now reach Serilog with the id (Gitea #120, §12). Still out of scope: a Log sink to the site event log; capturing Console.Out on a site run — System is imported so Console.WriteLine compiles today, and the Test Run sandbox captures it (SandboxConsoleCapture), but on a site node Console.Out is process-global and shared by every concurrent run, so per-run capture is unsound; Log.* is the leveled, per-run replacement and the sandbox mirror writes to the same console pane.

10. Testing

Pure: FlowGraphBuilderTests (attachment by step id, anchor fallback, first-step fallback, unattached count, ordering by seq, folded-step attachment), FlowStepProjectorTests (ErrorDetail, ScriptLog never a node), FlowTraceCollectorTests (low-priority reserve, LogsTruncated). Site: ScriptLoggerTests (levels, caps, closing row, untraced = ILogger only), TraceCapturingLoggerTests, ScriptRuntimeContext scope attribution, ScriptRunLauncher errorDetail. Central: executor ErrorDetail, runner inbound row, middleware item key, InboundScriptContext.Log traced/untraced. Trust: Log.* passes the gate and binds on all four surfaces. UI (bUnit): badge, failed subtitle, Log tab default, filter, cross-step hint, strip counts. Rig: verify-delmia-notifier.sh step J — a seeded FlowLogDemo script logs three levels and throws; Live shows the exception on the script node and the lines on their steps; Replay matches.

11. Documents to update

Component-CentralUI.md (Flow Debugger bullet), Component-SiteRuntime.md (Log global, caps, capturing logger), Component-InboundAPI.md (Log, ErrorDetail), Component-AuditLog.md (ScriptLog kind), Component-ScriptAnalysis.md (compile surface), CLI README (--show-logs), CLAUDE.md Flow Debugger bullet (v1.4), deferred-work register row 110 (close the console half; keep Direction B), plus a new row for the node-logger enrichment follow-up (§9).

12. Other actors' lines per execution (Gitea #120 + #56, 2026-11-10)

Akka logging is bridged to Serilog (#120). AkkaHostedService.BuildHocon sets akka.loggers = [SerilogAkkaLogger] (Host) and keeps akka.loglevel = INFO, replacing Akka's stdout-only DefaultLogger: every ILoggingAdapter line now reaches the file sinks with the node enrichers, as SourceContext = the logging class's full name plus AkkaSource = the actor path, at the timestamp Akka recorded, with the adapter's format arguments as structured properties. Akka's Debug chatter is still filtered at the adapter, so the bridge adds no volume: the INFO+ lines that used to go to stdout only now go to the Serilog sinks instead. Akka writes each line later on its logger actor's thread, where the logging actor's LogContext no longer applies. The two DCL actors therefore log through DataConnectionLayer.Logging.ExecutionAwareLogging.GetLogger(Context), which records ExecutionLogContext.Current (a new AsyncLocal mirror of the pushed id) by wrapping the message in an ExecutionTaggedLogMessage (ToString() unchanged, so stdout and TestKit filters see the same text). The logger actor re-pushes that id while it writes the line.

Per-execution capture without a schema change (#56). The decision was to add no replicated column. Instead, the host pipeline gains Commons.Observability.ExecutionLogCaptureSink. While a run is traced, its step logger registers its ExecutionId in ExecutionLogCapture: the site ScriptLogger registers from the launcher, plus each inline CallShared child's id, and the central InboundScriptLogger registers from the executor. The sink hands each line tagged with a registered id to that logger, which emits it as a ScriptLog row with LogSource = "node" and ScriptTraceExtra.LogCategory = the logger category. Consequences:

• Untraced runs cost ~nothing. The sink reads one volatile counter and returns. An untraced step logger never registers.
• Same bounds as every other line. Node lines count against MaxLogLinesPerRun together with the script's own lines and are clipped at 2 000 chars. They ride the collector's low-priority lane and pass the same redactor inline and in the store.
• No double capture. The step loggers' own node forwards (ScriptLogger, TraceCapturingLogger, InboundScriptLogger) run inside ExecutionLogCapture.Suppress(), and the sink suppresses itself while it calls a target. The registration is declared inside the launcher's and executor's try, so it ends before any catch block logs the run's outcome, which is already the ScriptRun / inbound row. FlushDropped also closes it, so no node line lands after the closing row.
• What it sees is what the node logs: its minimum level and per-source overrides apply first, so a node at Information never yields Debug lines. A line logged after the run ends (for example a late Akka logger-actor write) is not captured; it still reaches the node log.
• Graph and UI. FlowGraphBuilder lists node lines in FlowGraph.NodeLogs in time order and never folds them onto a step, because no step can honestly claim another actor's line. The Flow Debugger page renders them in a Node log card under the detail panel (FlowNodeLogPanel, data-test="flow-nodelog-panel"), grouped per execution under that execution's run step (the title selects it), with an All | Warn + filter and the short logger name. Live and Replay build the graph from the same rows, so site scope and the RequireFlowDebugger policy govern them as they govern every other row. CLI --show-logs prints them as a trailing node log (n lines): block.
• Mixed versions. An old central projects node rows like any ScriptLog row and folds them onto the anchor step. An old site sends none.