feat(logging): bridge Akka logging to Serilog and copy traced runs' node-log lines into the Flow Debugger (ScadaBridge#120, #56)

#120: Akka ILoggingAdapter lines used to bypass Serilog and go to stdout only.
- BuildHocon sets akka.loggers = [Host.Actors.SerilogAkkaLogger] and keeps akka.loglevel = INFO,
  so Akka Debug chatter is still filtered at the adapter. The INFO+ lines that used to reach
  stdout now go through the Serilog pipeline instead, with no extra volume.
- Lines reach the file sinks with the node enrichers. SourceContext is the logging class's full
  name, AkkaSource is the actor path, Akka's own timestamp is kept, and format arguments become
  structured properties.
- The Akka logger actor writes on its own thread, after the logging actor's LogContext is gone.
  To keep the ExecutionId, the two DCL actors log through ExecutionAwareLogging.GetLogger(Context),
  which records ExecutionLogContext.Current (a new AsyncLocal mirror of the pushed id) in an
  ExecutionTaggedLogMessage. The logger actor pushes that id again while it writes the line.

#56 remainder (coordinator decision: no replicated schema change):
- The host pipeline gains ExecutionLogCaptureSink.
- While a run is traced, its step logger registers its ExecutionId: the site ScriptLogger from the
  launcher (plus each inline CallShared child id), and the central InboundScriptLogger from the
  executor.
- Each tagged line becomes a ScriptLog row with LogSource "node" and the new
  ScriptTraceExtra.LogCategory.
- The rows count against the existing MaxLogLinesPerRun cap, are clipped at 2 000 chars, ride the
  low-priority collector lane and pass the same redactor. Untraced runs cost one volatile read.
- Lines are never copied twice. The step loggers' own forwards run inside
  ExecutionLogCapture.Suppress(), and the sink suppresses itself while it calls a target. The
  registration is declared inside the launcher's and executor's try, so it ends before the catch
  blocks log the outcome, and FlushDropped closes it.
- FlowGraphBuilder lists node lines in FlowGraph.NodeLogs and never folds them onto a step.
- The Flow Debugger shows them in a Node log card (FlowNodeLogPanel): grouped per execution under
  its run step, with an All | Warn + filter. Live and Replay both read the same rows, so site
  scope and role rules apply unchanged.
- CLI --show-logs prints them as a trailing block.

Tests: Commons ExecutionLogCaptureTests, ExecutionLogContext.Current, FlowGraphBuilder node lines.
SiteRuntime NodeLogCaptureTests covers cross-actor capture via the launcher, untraced runs
capturing nothing, no double capture of script and runtime lines or of the launcher failure line,
the shared cap, collector redaction, shared-script child ids and FlushDropped closing capture.
InboundNodeLogCaptureTests covers the central path. DCL ExecutionAwareLoggingTests covers the
DataConnectionActor write-path Debug line being tagged and an uncorrelated line not being tagged.
Host SerilogAkkaLoggerTests covers mapping, the ExecutionId crossing the logger hop, HOCON,
ActorSystem boot and the Akka line reaching a traced run's step log end to end. Host
LoggerConfigurationTests checks that the sink is wired. bUnit FlowNodeLogPanelTests and a CLI
projection test cover the UI and CLI. Every new assertion was mutation-proven.

Docs: step-logs design section 12, Component-SiteRuntime, CentralUI, InboundAPI, Host, CLI README,
CLAUDE.md.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-10 06:12:34 -05:00
parent 463aced7a1
commit c810bd0f7c
43 changed files with 2261 additions and 46 deletions
@@ -0,0 +1,216 @@
using System.Collections.Concurrent;
using Serilog.Core;
using Serilog.Events;
namespace ZB.MOM.WW.ScadaBridge.Commons.Observability;
/// <summary>One node-log line copied for a traced execution (ScadaBridge#56).</summary>
/// <param name="OccurredAtUtc">When the line was logged.</param>
/// <param name="Level"><c>debug</c> | <c>info</c> | <c>warn</c> | <c>error</c>.</param>
/// <param name="Category">The logger category (<c>SourceContext</c>), when the line had one.</param>
/// <param name="Message">The rendered message.</param>
/// <param name="Exception">The exception attached to the line, if any.</param>
public sealed record CapturedLogLine(
DateTime OccurredAtUtc, string Level, string? Category, string Message, Exception? Exception);
/// <summary>Receives the node-log lines of one traced execution (ScadaBridge#56).</summary>
public interface IExecutionLogTarget
{
/// <summary>Records one line. Called on the logging thread; must not throw and must not block.</summary>
/// <param name="line">The captured line.</param>
void Capture(CapturedLogLine line);
}
/// <summary>
/// The process-wide set of executions whose node-log lines are being copied into their Flow
/// Debugger trace (ScadaBridge#56).
/// </summary>
/// <remarks>
/// <para><b>What it is for.</b> Every node-log line written while a correlated message is served
/// carries the run's <c>ExecutionId</c> (<see cref="ExecutionLogContext"/>). While a run is traced,
/// its step logger registers here under that id, and <see cref="ExecutionLogCaptureSink"/> hands it
/// each such line, wherever it was logged: the Instance Actor, a DCL actor (through the Akka bridge,
/// #120), a protocol adapter, an HTTP client. The step logger turns each line into a
/// <c>ScriptLog</c> row with source <c>node</c>, under the run's existing caps and redaction.</para>
/// <para><b>Cost when nothing is traced.</b> The sink reads one volatile counter and returns. No
/// property lookup, no rendering, no allocation.</para>
/// <para><b>No double capture.</b> The step loggers already turn their own lines into rows (the
/// script's <c>Log.*</c> calls and the runtime lines a traced context captures). They write those
/// lines to the node logger inside <see cref="Suppress"/>, so the sink skips them. The sink also
/// suppresses itself while it calls a target, so a line the target logs while emitting its row
/// cannot loop back.</para>
/// <para><b>Lifetime.</b> A registration is disposed when the run's step log closes (the site
/// launcher's and inbound executor's run end). A line logged after that is not captured: it still
/// reaches the node log, just not the trace.</para>
/// </remarks>
public static class ExecutionLogCapture
{
private static readonly ConcurrentDictionary<Guid, IExecutionLogTarget> Targets = new();
private static int _count;
[ThreadStatic]
private static int _suppressDepth;
/// <summary>True when at least one execution is registered. One volatile read.</summary>
public static bool HasTargets => Volatile.Read(ref _count) > 0;
/// <summary>True while the current thread is inside <see cref="Suppress"/>.</summary>
public static bool IsSuppressed => _suppressDepth > 0;
/// <summary>Registers <paramref name="target"/> for the lines tagged with <paramref name="executionId"/>.</summary>
/// <param name="executionId">The traced execution.</param>
/// <param name="target">Where its lines go.</param>
/// <returns>
/// A disposable that removes the registration (idempotent). When the id is already registered
/// the first registration is kept and the returned disposable does nothing.
/// </returns>
public static IDisposable Register(Guid executionId, IExecutionLogTarget target)
{
ArgumentNullException.ThrowIfNull(target);
if (!Targets.TryAdd(executionId, target))
{
return NoOp.Instance;
}
Interlocked.Increment(ref _count);
return new Registration(executionId, target);
}
/// <summary>Finds the target registered for an execution.</summary>
/// <param name="executionId">The execution id.</param>
/// <param name="target">The registered target, when found.</param>
/// <returns><see langword="true"/> when the execution is registered.</returns>
public static bool TryGetTarget(Guid executionId, out IExecutionLogTarget target)
=> Targets.TryGetValue(executionId, out target!);
/// <summary>
/// Stops the sink from capturing lines logged on this thread until the scope is disposed. Used
/// around writes whose line is already a trace row, so it is not copied a second time.
/// </summary>
/// <returns>A disposable that ends the suppression; dispose it on the same thread.</returns>
public static SuppressScope Suppress()
{
_suppressDepth++;
return new SuppressScope(true);
}
/// <summary>Ends one <see cref="Suppress"/> on the thread that started it.</summary>
public readonly struct SuppressScope : IDisposable
{
private readonly bool _active;
internal SuppressScope(bool active) => _active = active;
/// <summary>Ends the suppression.</summary>
public void Dispose()
{
if (_active && _suppressDepth > 0)
{
_suppressDepth--;
}
}
}
/// <summary>Maps a Serilog level to the step-log level names.</summary>
/// <param name="level">The Serilog level.</param>
/// <returns><c>debug</c>, <c>info</c>, <c>warn</c> or <c>error</c>.</returns>
public static string LevelName(LogEventLevel level) => level switch
{
LogEventLevel.Verbose or LogEventLevel.Debug => "debug",
LogEventLevel.Information => "info",
LogEventLevel.Warning => "warn",
_ => "error",
};
private sealed class Registration(Guid executionId, IExecutionLogTarget target) : IDisposable
{
private int _disposed;
public void Dispose()
{
if (Interlocked.Exchange(ref _disposed, 1) != 0)
{
return;
}
if (Targets.TryRemove(new KeyValuePair<Guid, IExecutionLogTarget>(executionId, target)))
{
Interlocked.Decrement(ref _count);
}
}
}
private sealed class NoOp : IDisposable
{
public static readonly NoOp Instance = new();
public void Dispose()
{
}
}
}
/// <summary>
/// The Serilog sink behind <see cref="ExecutionLogCapture"/> (ScadaBridge#56): copies each line
/// tagged with a registered execution's <c>ExecutionId</c> to that execution's target.
/// </summary>
/// <remarks>
/// <para>It sees what the node logs: the configured minimum level and every per-source override
/// apply before it, so a node at Information never hands it a Debug line. It must be attached
/// directly, never behind an async wrapper, because <see cref="ExecutionLogCapture.Suppress"/> is
/// per thread.</para>
/// </remarks>
public sealed class ExecutionLogCaptureSink : ILogEventSink
{
/// <summary>Copies one line to its execution's target, when that execution is being traced.</summary>
/// <param name="logEvent">The Serilog event.</param>
public void Emit(LogEvent logEvent)
{
if (!ExecutionLogCapture.HasTargets || ExecutionLogCapture.IsSuppressed || logEvent is null)
{
return;
}
if (!logEvent.Properties.TryGetValue(ExecutionLogContext.ExecutionIdProperty, out var value)
|| !TryReadGuid(value, out var executionId)
|| !ExecutionLogCapture.TryGetTarget(executionId, out var target))
{
return;
}
using (ExecutionLogCapture.Suppress())
{
try
{
var category = logEvent.Properties.TryGetValue(Serilog.Core.Constants.SourceContextPropertyName, out var sc)
&& sc is ScalarValue { Value: string s } ? s : null;
target.Capture(new CapturedLogLine(
logEvent.Timestamp.UtcDateTime,
ExecutionLogCapture.LevelName(logEvent.Level),
category,
logEvent.RenderMessage(),
logEvent.Exception));
}
catch
{
// Deliberate swallow: a debug observer must never fail the write that logged the line.
}
}
}
private static bool TryReadGuid(LogEventPropertyValue value, out Guid id)
{
switch (value)
{
case ScalarValue { Value: Guid g }:
id = g;
return true;
case ScalarValue { Value: string s } when Guid.TryParse(s, out var parsed):
id = parsed;
return true;
default:
id = Guid.Empty;
return false;
}
}
}
@@ -32,15 +32,55 @@ public static class ExecutionLogContext
/// <summary>The log-event property carrying the execution id.</summary>
public const string ExecutionIdProperty = "ExecutionId";
/// <summary>
/// The innermost execution id pushed on this async flow, or null. Mirrors the
/// <see cref="ExecutionIdProperty"/> on the Serilog <c>LogContext</c>, which cannot be read back.
/// </summary>
/// <remarks>
/// ScadaBridge#120: Akka's <c>ILoggingAdapter</c> hands each line to a logger actor on another
/// thread, where this flow's <c>LogContext</c> is gone. An adapter that captures this value when
/// the line is logged lets the logger actor re-push it when it writes the line to Serilog.
/// </remarks>
public static Guid? Current => CurrentId.Value;
private static readonly AsyncLocal<Guid?> CurrentId = new();
/// <summary>Pushes <paramref name="executionId"/> as <see cref="ExecutionIdProperty"/>.</summary>
/// <param name="executionId">The execution id; null pushes nothing.</param>
/// <returns>A disposable that pops the property, or null when nothing was pushed.</returns>
public static IDisposable? Push(Guid? executionId)
=> executionId is { } id ? LogContext.PushProperty(ExecutionIdProperty, id) : null;
{
if (executionId is not { } id)
{
return null;
}
var previous = CurrentId.Value;
var property = LogContext.PushProperty(ExecutionIdProperty, id);
CurrentId.Value = id;
return new Scope(property, previous);
}
/// <summary>Pushes the execution a message serves, when it names one.</summary>
/// <param name="message">Any actor message.</param>
/// <returns>A disposable that pops the property, or null when the message names no execution.</returns>
public static IDisposable? PushFor(object? message)
=> message is IExecutionCorrelated correlated ? Push(correlated.CorrelatedExecutionId) : null;
/// <summary>Pops the pushed property and restores the previous <see cref="Current"/>.</summary>
private sealed class Scope(IDisposable property, Guid? previous) : IDisposable
{
private int _disposed;
public void Dispose()
{
if (Interlocked.Exchange(ref _disposed, 1) != 0)
{
return;
}
CurrentId.Value = previous;
property.Dispose();
}
}
}
@@ -0,0 +1,27 @@
namespace ZB.MOM.WW.ScadaBridge.Commons.Observability;
/// <summary>
/// An Akka log message together with the execution id that was current when it was logged
/// (ScadaBridge#120).
/// </summary>
/// <remarks>
/// <para>Akka's logging adapter publishes each line to a logger actor, which writes it on its own
/// thread. By then the logging actor's <c>LogContext</c>, including the <c>ExecutionId</c> that
/// <see cref="ExecutionLogContext.PushFor"/> pushed for the message being served, is no longer
/// available. An execution-aware adapter wraps the message in this type when
/// <see cref="ExecutionLogContext.Current"/> is set. The host's Serilog logger actor unwraps it and
/// pushes the id again while it writes the line.</para>
/// <para><see cref="ToString"/> returns the inner message's text. A logger that does not know this
/// type, such as Akka's stdout logger during startup or the TestKit's event filters, still sees the
/// same text it would have seen without the wrapper.</para>
/// <para>This type has no Akka dependency. It travels only inside one process, on the local event
/// stream, and is never serialized.</para>
/// </remarks>
/// <param name="Message">The original log message (an Akka <c>LogMessage</c>, or any object).</param>
/// <param name="ExecutionId">The execution the line was logged for.</param>
public sealed record ExecutionTaggedLogMessage(object? Message, Guid ExecutionId)
{
/// <summary>The inner message's text, unchanged by the wrapper.</summary>
/// <returns>The formatted inner message, or an empty string when it is null.</returns>
public override string ToString() => Message?.ToString() ?? string.Empty;
}
@@ -16,13 +16,19 @@ public sealed record FlowEdge(Guid FromEventId, Guid ToEventId, bool CrossesBoun
/// <param name="Truncated">True when the inline event list was capped (the audit rows remain complete — Replay shows everything).</param>
/// <param name="LogsTruncated">True when the inline trace collector dropped low-priority log rows to keep room for the step rows — the audit rows remain complete, so Replay shows every line (design 2026-09-09 D7).</param>
/// <param name="UnattachedLogLines">Count of log lines the builder could not place on any step (design 2026-09-09 D2).</param>
/// <param name="NodeLogs">
/// Node-log lines copied for the traced executions (source <c>node</c>, ScadaBridge#56), in time
/// order. They are listed per execution, never folded onto a step: they come from other actors and
/// threads, so no step can honestly claim them. Null or empty when there are none.
/// </param>
public sealed record FlowGraph(
IReadOnlyList<FlowStep> Steps,
IReadOnlyList<FlowEdge> Edges,
bool Predicted,
bool Truncated,
bool LogsTruncated = false,
int UnattachedLogLines = 0)
int UnattachedLogLines = 0,
IReadOnlyList<FlowLogLine>? NodeLogs = null)
{
/// <summary>An empty, non-predicted graph.</summary>
public static FlowGraph Empty { get; } = new(Array.Empty<FlowStep>(), Array.Empty<FlowEdge>(), false, false);
@@ -51,6 +51,9 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Debugging;
/// dropped and counted in <see cref="FlowGraph.UnattachedLogLines"/>. Within a step, lines are ordered
/// by <c>OccurredAtUtc</c> then <c>Seq</c>, with a null <c>Seq</c> sorting last, so same-millisecond
/// lines keep their emission order.</item>
/// <item><b>Node-log lines are listed, not folded</b> (ScadaBridge#56). A <c>ScriptLog</c> row with source
/// <c>node</c> — another actor's line the per-execution sink copied — goes to
/// <see cref="FlowGraph.NodeLogs"/> in time order and never onto a step or into the unattached count.</item>
/// </list>
/// </remarks>
public static class FlowGraphBuilder
@@ -82,9 +85,13 @@ public static class FlowGraphBuilder
events = stepEvents.Select(e => e.Event).ToList();
// ScadaBridge#56: node-log lines (other actors' lines copied for a traced execution) are
// listed per execution, not folded onto a step, so they never count as unattached either.
var nodeLogs = TakeNodeLines(lines);
if (events.Count == 0)
return new FlowGraph(Array.Empty<FlowStep>(), Array.Empty<FlowEdge>(), Predicted: false, Truncated: truncated,
LogsTruncated: logsTruncated, UnattachedLogLines: lines.Count);
LogsTruncated: logsTruncated, UnattachedLogLines: lines.Count, NodeLogs: nodeLogs);
// Arch-review F5: a step's position in the input is the only cross-node ordering we can trust —
// central rows carry central's clock, site rows the site's, and a 1-2 s skew used to sort a whole
@@ -268,7 +275,19 @@ public static class FlowGraphBuilder
AttachLogLines(steps, lines, out var unattached);
return new FlowGraph(steps, edges, Predicted: false, Truncated: truncated,
LogsTruncated: logsTruncated, UnattachedLogLines: unattached);
LogsTruncated: logsTruncated, UnattachedLogLines: unattached, NodeLogs: nodeLogs);
}
/// <summary>
/// Removes the node-log lines (source <c>node</c>, ScadaBridge#56) from <paramref name="lines"/>
/// and returns them ordered by time, then per-run <c>Seq</c> (null last). Null when there are none.
/// </summary>
private static IReadOnlyList<FlowLogLine>? TakeNodeLines(List<FlowLogLine> lines)
{
var node = lines.Where(l => l.IsNodeLine).ToList();
if (node.Count == 0) return null;
lines.RemoveAll(l => l.IsNodeLine);
return node.OrderBy(l => l.OccurredAtUtc).ThenBy(l => l.Seq ?? int.MaxValue).ToList();
}
/// <summary>
@@ -84,12 +84,13 @@ public sealed record FlowLifecycleRow(
/// <param name="ExecutionId">Execution the line was written in.</param>
/// <param name="OccurredAtUtc">When the line was written.</param>
/// <param name="Level"><c>debug</c> | <c>info</c> | <c>warn</c> | <c>error</c>.</param>
/// <param name="Source"><c>script</c> (a <c>Log.*</c> call) | <c>runtime</c> (the context's own logger while traced).</param>
/// <param name="Source"><c>script</c> (a <c>Log.*</c> call) | <c>runtime</c> (the context's own logger while traced) | <c>node</c> (another node-log line tagged with the execution, ScadaBridge#56).</param>
/// <param name="Message">The logged message.</param>
/// <param name="ExceptionMessage">Message of an attached exception, when the line carried one.</param>
/// <param name="ExceptionDetail">Full <c>ToString()</c> of that exception.</param>
/// <param name="StepEventId">Step the line belongs to; null = the execution's anchor step (design D2).</param>
/// <param name="Seq">Per-run emission counter, breaking same-millisecond ties in display order.</param>
/// <param name="Category">On a <c>node</c> line (ScadaBridge#56): the logger category that wrote it.</param>
public sealed record FlowLogLine(
Guid EventId,
Guid? ExecutionId,
@@ -100,7 +101,15 @@ public sealed record FlowLogLine(
string? ExceptionMessage,
string? ExceptionDetail,
Guid? StepEventId,
int? Seq);
int? Seq,
string? Category = null)
{
/// <summary>The <see cref="Source"/> of a node-log line copied by the per-execution sink (ScadaBridge#56).</summary>
public const string NodeSource = "node";
/// <summary>True for a node-log line, which the graph lists per execution rather than on a step.</summary>
public bool IsNodeLine => string.Equals(Source, NodeSource, StringComparison.Ordinal);
}
/// <summary>
/// One node of the flow graph — a central-side projection of a <c>ScriptTrace</c> /
@@ -102,7 +102,8 @@ public static class FlowStepProjector
ExceptionMessage: d.ErrorMessage,
ExceptionDetail: d.ErrorDetail,
StepEventId: x?.StepEventId,
Seq: x?.Seq);
Seq: x?.Seq,
Category: x?.LogCategory);
}
/// <summary>Projects one outbound audit row onto its lifecycle entry (<see cref="FlowStep.Lifecycle"/>).</summary>
@@ -52,9 +52,12 @@ public sealed record ScriptTraceExtra
/// <summary>On a <c>ScriptLog</c> row: <c>debug</c> | <c>info</c> | <c>warn</c> | <c>error</c>.</summary>
public string? LogLevel { get; init; }
/// <summary>On a <c>ScriptLog</c> row: <c>script</c> (a <c>Log.*</c> call) | <c>runtime</c> (the context's own logger while traced).</summary>
/// <summary>On a <c>ScriptLog</c> row: <c>script</c> (a <c>Log.*</c> call) | <c>runtime</c> (the context's own logger while traced) | <c>node</c> (any other node-log line tagged with the execution's id, ScadaBridge#56).</summary>
public string? LogSource { get; init; }
/// <summary>On a <c>node</c> <c>ScriptLog</c> row: the logger category (<c>SourceContext</c>) that wrote the line (ScadaBridge#56).</summary>
public string? LogCategory { get; init; }
/// <summary>On a <c>ScriptLog</c> row: the pre-minted <c>EventId</c> of the step the line belongs to; null = the execution's anchor step (design D2).</summary>
public Guid? StepEventId { get; init; }