Rig run 2026-09-06 (image built from this branch, containers recreated from
main's compose file): full drill A-H YES (active-central kill takeover 27 s,
rejoin as standby in 3 s), then with FLOW_ONLY=1:
H live+replay -> YES
live: YES root=8e628d0a-428b-48df-b937-f4c2ac631fc3 events=8 api=Delivered db=Delivered notify=Submitted
replay: YES root=8e628d0a-428b-48df-b937-f4c2ac631fc3 events=10 api=Delivered db=Delivered notify=Delivered
I outbound live+replay -> YES
Two rig facts the drill established: (1) with the DB reachable the site's
immediate-completion path resolves a CachedWrite INSIDE the run, so the DbCall
node is already Delivered inline (Submit/Attempted/Resolve share a tick — the
check now ranks statuses instead of taking list order); (2) cached-call
telemetry reaches central on its own ~20-30 s cadence, later than the 5 s audit
push, so replay needs the bounded retry. New FLOW_ONLY=1 switch runs H + I
without the notifier build or the docker kill.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
After a successful Live run whose graph has Pending outbound steps the page
re-reads the tree through the existing Replay command every 5 s for up to
2 min (page constants), merging by replacing the graph and keeping the
selection by event id; never in Replay mode; cancelled by a new run, a mode
switch or Dispose. Status strip gains a Pending count, a 'Refresh outcomes'
button (both modes) and an 'outcomes still arriving' note while polling. The
run's own timing and the Completed chip are left alone — pending outbound
work is not a failed run. Plan 2026-09-06 D5.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
A folded outbound step shows a Lifecycle tab (time · kind · status · node ·
error, oldest first; a Parked step adds the 'retry or discard from Site Calls /
Notification report' hint), replaces the never-populated Writes tab for those
steps, labels the Input tab Request / SQL + parameters / Subject + body by kind,
and 'Where this ran' names the System / Connection / List plus 'Delivered by'
(the terminal row's node). Pending/Parked outcome badges.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Four lanes (LaneCount 4; External sublabel = distinct targets, three then +n);
ApiCall/DbCall/Notify nodes carry a status pill (Ok/Pending/Parked/Failed —
pending pulses, reduced-motion aware) plus a sync/cached/notify mode; folded
lifecycles show '+4.1 s' submit→terminal wall time. Async edges are dashed
and routed through the inter-lane strip with a gutter tag; spawn edges use a
bolt marker. Predicted async edges (Task 6's PredictedEdge.Async) render the
same way. de-DE height pinned at 319.2 = (4×110+16)×0.7.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
FlowShapePredictor emits ExternalSystem.Call/CachedCall → ApiCall ('{s}.{m}'),
Database.Execute/Query/QuerySingle/CachedWrite → DbCall ('{conn} · {member}'),
Notify.Send(list) and Notify.To(list).Send() → Notify ('Notify {list}'), all on
FlowLane.External; Async-suffixed spellings map identically. PredictedEdge gains
an additive Async flag; an async edge (cached call, notify) does not advance the
chain, mirroring FlowGraphBuilder.IsAsync (D4/D8). Database.Connection() raw
handles no longer predict a node (command count is not statically knowable).
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
GetTraceEventsAsync widens its channel filter to ScriptTrace ∪ ApiOutbound ∪
DbOutbound ∪ Notification ∪ (ApiInbound & InboundRequest) (plan D6); SecuredWrite,
Cluster and InboundAuthFailure rows in the same tree stay excluded. The site
SqliteAuditWriter.ReadByExecutionIdsAsync already had no channel predicate — pinned
by a test. Interface doc updated to match.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Fold rule: outbound rows sharing (kind, CorrelationId) become one FlowStep with
a Lifecycle list — first row's identity/time/input, latest row's outcome/
subtitle/output/error, DurationMs = sync row's own or submit→terminal wall time
(null while pending). Chain skips async steps (Notify, or first row
CachedSubmit) and links them with Async edges from the preceding chain step.
Spawner edges from a non-RouteCall are Spawn: true. The Response links from the
last non-async step of the inbound execution (the RouteCall), not the globally
last cascaded step (D9). FlowGraphView places a Response at max column + 1 so
the D9 origin does not pull it left (pulled forward from Task 7 to keep the
layout test green).
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
ResolveKind maps ApiOutbound/DbOutbound/Notification to ApiCall/DbCall/Notify;
ResolveLane puts the three on FlowLane.External; outbound outcomes map
Delivered→Ok, Failed→Failed, Parked/Discarded→Parked, else Pending; subtitles
'{status} · sync|cached', '{op} · n rows' from a DbWrite Extra, '{status}' for
Notify. New FlowStepProjector.IsOutbound helper.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Gap verified 2026-09-06: the projector and predictor already know the three outbound
channels, but the live FlowTraceCollector is fed only by EmitTrace (writes/waits) and
the replay query filters to ScriptTrace + the inbound row, so a run leaves every
predicted outbound node as a ghost. Central method-script Database.* calls are not
audited at all.
Plan (11 tasks): three step kinds ApiCall/DbCall/Notify on a fourth External lane;
one node per operation with the cached/notify lifecycle folded in (Pending/Parked
outcomes, Lifecycle tab); sync calls in the chain, async ones on dashed branch edges;
site rows ride the existing collector, replay filter widens, central DB helper gets
DbOutbound rows; the page refreshes pending outcomes via the existing Replay command;
spawn edges styled and Response linked from the route call (the "two parallel paths"
question). Mockup: docs/plans/…-mockup.html, published as a Claude artifact.
Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
Rig recreated from the branch-built image through the main checkout's compose file
(deploy.sh refuses a worktree, register row 68). Fresh key delmia-flowdbg-0906 scoped to
DelmiaRecipeDownload. Full drill: A/B/E/F YES, C/G NO, takeover 29 s, rejoin 3 s, then
H live+replay -> YES
live: 11 events (InboundRequest, RouteCall, ScriptRun, 7x AttributeWrite, AttributeWait)
replay: 17 events on the first attempt — the six extra rows are the simulator's two
cascaded ValueChange runs (2x ScriptRun + 4x AttributeWrite): inline TraceEvents
carry the routed run's own steps, replay walks the whole execution tree.
Recorded as row H in Delmia-Recipe-Download-API.md §8; ledger marks Task 22 completed
(all 24 tasks done). Note: the rig still runs main's appsettings, so the ScriptTrace
30-day retention override lands only after merge + a deploy from main.
Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
Component-CentralUI (new Flow Debugger page section), Component-InboundAPI (TraceAllCalls
property, always-on RouteCall row + Trace flag under Routing Behavior, new 'Flow Debugger
runs' section), Component-SiteRuntime (Instance Actor trace arm; Script Actor traced runs;
CallScript flag inheritance), Component-AuditLog (ScriptTrace channel in Scope, the
PullTraceEvents point-read note + mixed-version caveat under Reconciliation, per-channel
retention example), Component-Communication (§4 trace flag + inline events with proto field
numbers; §8 PullTraceEvents; RPC list), Component-ScriptAnalysis (REQ-SA-7 FlowShapePredictor),
CLAUDE.md UI bullet, deferred-work register row 110 (vertical-trace list toggle + console
capture), design doc status → implemented. CLI README sections landed in Tasks 3/16.
Umbrella index (../scadaproj/CLAUDE.md) checked: its ScadaBridge row records stack/wire
relationships only, unchanged by a new page — no edit.
Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
Flow Debugger (design 2026-09-05 §7.1): FlowDebugRunMethodCommand / FlowDebugRunValueChangeCommand /
FlowDebugReplayCommand on the per-node ManagementActor (Deployer role; the two run commands
are long-running), all delegating to FlowDebugRunner. RunMethod refuses on a standby with
STANDBY_NODE (ManagementCommandException now carries an optional Code that replaces
COMMAND_FAILED), validates parameters through the real ParameterValidator/SchemaRefResolver
(made public), synthesizes the inbound row (actor = operator, Extra.source = flow-debugger)
and runs the method through InboundScriptExecutor with a trace-sink RouteHelper so the reply
carries the RouteCall row + the site's inline events. RunValueChange does a traced routed
write via IInstanceRouter, then polls IPullTraceEventsClient (interface moved to Commons so
ManagementService can consume it) until a ScriptRun parented to the root settles, else
NO_TRIGGER_WITHIN_WINDOW. Replay reads IAuditLogRepository.GetTraceEventsAsync.
Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
Flow Debugger (design 2026-09-05 §5.2): bit NOT NULL DEFAULT 0 column + EF migration
20260906034113 with its docs/plans/sql production twin. CreateApiMethodCommand gains
TraceAllCalls=false; UpdateApiMethodCommand gains bool? TraceAllCalls (null = unchanged so an
older CLI never resets it). Management result echoes the flag; CLI create/update expose
--trace-all-calls; README carries the ~7-rows-per-call warning.
Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
Live + Replay modes (no dry run); entry points = inbound API method and
value-change trigger; data-sourced value-change does a real tag write with a
30 s trace arm on the instance actor for echo correlation; gated trace rows in
a new ScriptTrace audit channel (RouteCall always-on); inline step return on
RouteToCallResponse for Live; Deployer gates the page and Live run.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
The 2026-06-26 design kept the key env-only. The file is now an explicit fallback: SCADABRIDGE_API_KEY (non-blank) -> ScadaBridge:ApiKey (non-blank, trimmed) -> not configured. Env-only stays the recommended production posture; the failure message names both sources.
Claude-Session: https://claude.ai/code/session_013L3XdmVii9i5sY4FChU8pB
Row 104: docker/vacuum-site-localdb.sh stops both nodes of a pair together, sets
auto_vacuum=INCREMENTAL + VACUUM on each node volume with integrity_check before/after,
then starts both (DRY_RUN=1 reports). Run on all three rig pairs: six files 0 -> 2, ok,
~11% smaller; site-a replication reconnected with backlog 0 and no watermark change —
VACUUM is replication-invisible (ledger keys by declared PK, oplog seq is INTEGER PRIMARY
KEY AUTOINCREMENT). Topology guide gains the recipe.
Row 105: SiteLocalDbReplicatedTableFkGuardTests builds the real site schema via OnReady
(unreplicated config so 0.3.3's own guard cannot mask the culprit) and asserts every
SiteLocalDbSetup.ReplicatedTables entry (now internal) exists and has no inbound
CASCADE/SET NULL/SET DEFAULT foreign key, with a negative control. The constraint is
enforced at build instead of at the first replicated boot.
Row 106 (found during 104's live verification, NOT caused by it — the identical 127k-row
full snapshot also landed at the earlier deploy restart): SnapshotStreamer.SendAsync stamps
as_of_seq = MAX(seq), which is 0 once a converged sender's oplog is ack-pruned to empty;
a zero-watermark receiver (rebuilt node) is never lifted, so the empty-oplog horizon rule
re-streams a full snapshot on every reconnect. Sender-side one-line fix for LocalDb 0.3.4:
as_of_seq = max(MAX(seq), last_acked_seq).
P5-3: add IX_Notifications_Created_NotificationId (CreatedAt DESC,
NotificationId DESC) — the Central UI Notification Outbox list page sorts on
exactly that pair, but no existing index leads unconditionally on CreatedAt,
so an unfiltered/lightly filtered page fell back to a full clustered-index
scan + sort on every COUNT + page request. Mirrors the Medium wave's
IX_SiteCalls_Created_TrackedOperationId migration + SQL twin pattern
faithfully, including the handwritten docs/plans/sql script.
P5-4: operator Retry/Discard was read-check-write keyed on NotificationId
only — a detached PipeTo task interleaving with the dispatcher's own write
(or a second concurrent operator action) could land between the read and
the write and get silently clobbered by the unconditional UpdateAsync,
stamping a Delivered row Discarded with stale fields. Adds
INotificationOutboxRepository.TryUpdateFromParkedAsync, a single-statement
compare-and-swap (WHERE Status == Parked) mirroring
SecuredWriteRepository.TryMarkApprovedAsync's CAS shape. NotificationOutboxActor's
RetryAsync/DiscardAsync now use the CAS write and, on a lost race, re-read
to report "notification already progressed to <Status>" instead of silently
overwriting whatever is there now.
Proof of failure: reverting only the actor's CAS usage while keeping the
repository addition reproduces the silent-overwrite bug — 7 tests fail,
including two new deterministic (non-sleep) race-proof tests that mock the
exact GetByIdAsync -> lost-CAS -> re-read sequence the finding describes.
P5-3 is perf-only; verified via migration/SQL-twin parity instead (no
runtime proof-of-failure is meaningful for an index).
P5-1: AuditLogIngestActor.OnCachedTelemetryAsync's per-entry fallback loop
(used when the cached-telemetry batch fast path fails) ran with no
budget/cancellation, unlike OnIngestAsync's fallback. A slow/wedged DB could
freeze the central ingest singleton's mailbox for minutes. The loop now runs
under its own fresh IngestFallbackBudget CancellationTokenSource (mirroring
OnIngestAsync), passes the token into BeginTransactionAsync/
InsertIfNotExistsAsync/UpsertAsync, and stops early on budget exhaustion,
logging how many entries were left for the periodic reconciliation pull to
pick up.
P5-2: SiteCallAuditRepository.QueryAsync ordered by
(CreatedAtUtc DESC, TrackedOperationId DESC) with no index leading
unconditionally on CreatedAtUtc, forcing a full scan+sort on an unfiltered
page. Added IX_SiteCalls_Created_TrackedOperationId (migration
AddSiteCallsListSortIndex + matching hand-written production-parity script
under docs/plans/sql/). Also switched the query from FromSqlInterpolated
(which binds string filters as nvarchar(4000)) to FromSqlRaw with explicitly
SqlDbType.VarChar-typed parameters, matching the SiteCalls columns'
IsUnicode(false) declaration and the write path's existing Param(...)
pattern, so seeks — including the keyset predicate — are no longer blocked
by an implicit conversion.
Tests: new AuditLogIngestActorCombinedTelemetryTests case proves the
fallback budget bounds the wait (hanging repo stub, ~5s not indefinite),
that remaining entries are skipped, and that a non-default/cancellable
token reaches the repository call; verified fail-unfixed by reverting the
actor change (test times out at its 20s bound). Existing
SiteCallAuditRepositoryTests QueryAsync/keyset/stuck-cutoff cases (real
MSSQL) confirm the raw-SQL rewrite still returns correct pages.
Joseph Doherty