docs: Flow Debugger design reconciled with the implementation (arch-review F12, F13)

The Flow Debugger design doc still described the pre-remediation shape: a
debug-stream subscription for the value-change entry, Deployer everywhere, an
unredacted inline trace copy and "~7 audit rows per call". Reconciled with what
shipped on this branch (Tasks 1-12):

- design D6 / §5.4 / §9 / §10: RequireFlowDebugger page policy with Deployer-only
  execution (F3), the PullTraceEvents polling path with AttributeValueUpdate
  fields 7/8 recorded as RESERVED and unconsumed (F12), the redacting + byte-
  capped shared FlowTraceCollector (F1), route-time site scope (F2), the derived
  client waits (F4), the 7-10 row estimate, the doubled central audit write per
  routed call, and the inbound-row-after-the-run ordering.
- sitestream.proto: fields 7/8 marked reserved-but-populated, numbers not to be
  reused (comment only; no regeneration needed).
- Component-InboundAPI: SkipBodyCapture on the RouteCall row (F11),
  SchemaRefResolver's public visibility, WithSiteScope / SITE_SCOPE_DENIED (F2),
  Replay's wider role set (F3).
- Component-CentralUI: page policy vs Deployer-only controls (F3), route-time
  site scope (F2), client wait derivation (F4).
- "7-10 audit rows per call (3 of them trace-only)" everywhere the estimate is
  quoted: the method form, both CLI --trace-all-calls help strings, the CLI
  README, the ApiMethod entity doc, the manual migration script and the two
  Flow Debugger plans.
- arch-review doc: new §7 Resolution listing F1-F13 with the fixing commit, plus
  the tracking-store test race fixed alongside them (86451382).

Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29

docs/plans/2026-09-05-flow-debugger-design.md

@@ -27,7 +27,7 @@ on the `DelmiaReceiver` child → seven attribute writes → wait for Galaxy to
 | D3 | **Value-change on a data-sourced attribute does a REAL tag write** through the instance actor's normal write path; the device echo fires the trigger. Correlation is by a short-lived *trace arm* on the instance actor (§5.3). | A debug-only "inject value" command that synthesises the change without touching the device — leaves the instance's cached value diverged from the device until the next real update. |
 | D4 | **Live gets its trace inline; Replay reads audit rows.** Site audit telemetry forwards on a 5 s busy / 30 s idle cadence (`AuditLog:SiteTelemetry`), too slow to drive an interactive run, so the routed reply carries the site steps back and the same steps are *also* written as audit rows for Replay. | Live waiting on the audit pipeline. |
 | D5 | **Tracing is gated, not always-on.** Sites run scripts at high rates from timers and value changes; a trace flag rides beside `ParentExecutionId` and only flagged runs emit trace rows. One exception is always-on: a central `RouteCall` row per `Route.To().Call` (inbound-driven volume, joins the two halves of a Replay). | Tracing every run (volume), or a purely time-window join. |
-| D6 | **Live run is a Deployer act.** `RequireDeployment` gates the page and the Run button; Replay needs `OperationalAudit`. | A separate `RequireOperator` gate for Live. |
+| D6 | **Live run is a Deployer act.** `RequireDeployment` gates the Run controls and the two run commands; the page and Replay are open to `OperationalAudit` roles + Deployer (`RequireFlowDebugger`, arch-review F3). | A separate `RequireOperator` gate for Live. |
 
 ## 2. What exists today (reused, not rebuilt)
 
@@ -159,20 +159,24 @@ the echo also fires inherit the parent id exactly as static cascades do today.
 `RouteToCallResponse` gains additive `ChildExecutionId` (`Guid?`) and `TraceEvents`
 (`IReadOnlyList<AuditEvent>?`, the very events that became the site's audit rows, in emission
 order); on the wire it is `repeated AuditEventDto trace_events` (the DTO `sitestream.proto`
-already defines). The site collects events in a per-run `FlowTraceCollector` held by
+already defines). The site collects events in a per-run `FlowTraceCollector` (Commons
-`ScriptRuntimeContext` and attaches the list to the `ScriptCallResult` → `RouteToCallResponse`
+`Types/Debugging/`, shared with the central `FlowDebugRunner`) held by `ScriptRuntimeContext` and
-when `Trace` is set. Capped at `MaxTraceEventsPerRun` (500); beyond it the collector keeps the
+attaches the list to the `ScriptCallResult` → `RouteToCallResponse` when `Trace` is set. The
-first 500, sets `Truncated`, and the audit rows remain complete. A site on an older build ignores
+collector applies `IAuditRedactor` inside `Add`, so the inline record is the same filtered event
+the writer persists, and it caps at `MaxTraceEventsPerRun` (500) **or** 1 MB of summed
+`DetailsJson` (`DefaultMaxBytes`), whichever comes first; beyond either bound it keeps the earlier
+events, sets `Truncated`, and the audit rows remain complete (arch-review F1). A site on an older build ignores
 `Trace`, returns no `TraceEvents`, and the UI labels its lane `site not traced (older build)`.
 `FlowStep` is a **central-side projection** (`Commons/Types/Debugging/FlowStepProjector`)
 computed from `AuditEvent` + `AuditDetails`; nothing serializes it.
 
-For the value-change entry there is no routed reply to ride on. The debugger instead subscribes
+For the value-change entry there is no routed reply to ride on. The runner polls the site's
-to the instance's debug stream (`SubscribeDebugView`, the Debug View's existing path) for the run's
+`PullTraceEvents` RPC by `ParentExecutionId` (1 s cadence, 2 s settle window, `WaitSeconds` bound) —
-duration and takes the spawned run's `ExecutionId` from the stamped `AttributeValueChanged`, then
+simpler than the debug-stream subscription originally planned. `AttributeValueUpdate.source_execution_id`/`trace`
-pulls the site's trace rows for that id via a new `PullTraceEvents` RPC on `SiteStreamService`
+(fields 7/8) are populated by the site and **consumed by nothing**; they are reserved for a future
-(same PSK interceptor as `PullAuditEvents`; reads the site audit SQLite by `ExecutionId` /
+debug-stream path and must not be reused. The RPC sits behind the same PSK interceptor as
-`ParentExecutionId`). One additive RPC, no streaming change.
+`PullAuditEvents` and reads the site audit SQLite by `ExecutionId` / `ParentExecutionId` — one
+additive RPC, no streaming change.
 
 ## 6. Message and wire changes (all additive)
 
@@ -257,9 +261,14 @@ surface); the walker is syntax-only and never executes anything.
 
 ## 9. Security, authorisation, audit
 
-- **Page visibility and Live run:** `RequireDeployment` (same as Debug View) — decided
+- **Page visibility:** `RequireFlowDebugger` — the `OperationalAudit` roles plus Deployer
-  2026-09-05: a Live run writes real tags, but the Deployer already deploys and redeploys the
+  (`AuthorizationPolicies.FlowDebuggerAccessRoles` = Deployer, Administrator, Viewer), so an
-  scripts that do exactly that, so no separate Operator gate. **Replay:** `OperationalAudit` (it
+  auditor can open the page and Replay a past execution (arch-review F3). **Live run:**
+  `RequireDeployment` (same as Debug View) — decided 2026-09-05: a Live run writes real tags, but
+  the Deployer already deploys and redeploys the scripts that do exactly that, so no separate
+  Operator gate. The two run commands, the Live controls, the parked-node Retry/Discard actions and
+  the `?methodId=`/`?instanceId=` drill-ins stay Deployer-only, and the page pins Replay mode for a
+  non-Deployer. **Replay:** `FlowDebugReplayCommand` accepts the same `OperationalAudit` roles (it
   reads audit rows). No Verifier step — the debugger is a diagnostic, not a production write
   control, and the audit row is the safeguard.
 - **Inbound audit row for a Live run:** the shared `AuditEvent.Actor` carries the operator's
@@ -267,21 +276,38 @@ surface); the walker is syntax-only and never executes anything.
   `Extra.runNumber`; `Target` is the method as usual. API-key authentication is bypassed (there is
   no key), **parameter validation is not** — `ParameterValidator` runs exactly as for HTTP so a
   bad input fails the same way. The method's `TimeoutSeconds` applies. Site-scope (`SiteScopeService`)
-  restricts which instances a value-change entry may target.
+  restricts which instances a value-change entry may target. The method entry enforces the
-- Trace payloads pass through the existing redaction policy (auth headers redacted, 8 KB cap,
+  operator's site scope at route time (`RouteHelper.WithSiteScope`, `SITE_SCOPE_DENIED` →
-  64 KB on error rows). Console output is capped the same way.
+  `ManagementUnauthorized`, arch-review F2): the runner resolves the operator's permitted numeric
+  site ids to site identifiers and hands them to `RouteHelper`, which refuses an out-of-scope target
+  **before any site traffic**; the denial is rethrown as `SiteScopeViolationException` *after* the
+  inbound audit row is written, so the attempt is recorded under the operator's name.
+  Administrators and unscoped users are exempt, exactly as `EnforceSiteScope` treats them.
+- Trace payloads pass through the existing redaction policy on BOTH copies: the stored row (writer)
+  and the inline Live copy (`FlowTraceCollector` applies `IAuditRedactor` and a 1 MB byte budget
+  beside the 500-event cap — arch-review F1). The row-level caps are unchanged (auth headers
+  redacted, 8 KB cap, 64 KB on error rows). Console output is capped the same way.
 - Nothing here touches the PSK boundary; `PullTraceEvents` sits behind `ControlPlaneAuthInterceptor`.
 
 ## 10. Volume and failure behaviour
 
 - Untraced production is unchanged except one `RouteCall` row per routed call (inbound-driven).
+  Every routed call adds one central `RouteCall` insert — at inbound-driven rates that doubles the
+  per-call central audit write (accepted, D5).
 - A traced Delmia call emits ≈ 1 `InboundRequest` + 1 `RouteCall` + 1 `ScriptRun` + 2
-  `AttributeWrite` + 1 `AttributeWait` + 1 value-change `ScriptRun` ≈ 7 rows. `TraceAllCalls` on a
+  `AttributeWrite` + 1 `AttributeWait` + 1 value-change `ScriptRun` ≈ **7–10 rows** (a cached DB
-  100-call/day method is ~700 rows/day.
+  write and a notification add 3 trace-only rows). `TraceAllCalls` on a 100-call/day method is
+  ~700–1 000 rows/day.
 - Trace emission is best-effort like all audit (a failed trace write never fails the run).
 - Live run timeout = method timeout + `FlowDebugRunGrace` (10 s); a timeout renders whatever steps
   arrived and marks the run `Timed out`. The site's trace rows still land later, so Replay of the
-  same ExecutionId is complete.
+  same ExecutionId is complete. The client waits longer than the server on purpose: the UI derives
+  its Ask wait from the same bound (`RunAskTimeoutFor` = method timeout + grace + 15 s slack,
+  floored at 30 s; `ValueChangeAskTimeoutFor` = the wait window + 15 s) and the CLI's
+  `run-method --timeout-seconds` defaults to 120 while `run-value-change` waits
+  `--wait-seconds + 25` (arch-review F4).
+- **The Live run's inbound row is written AFTER the run completes**, so a Replay taken during a
+  long Live run shows only the `RouteCall`.
 - Older site build: no `TraceEvents`, lane labelled; older central build: ignores new fields; proto3
   defaults negotiate as for stream batching.
 - **Mixed-version caveat:** an OLD central ingesting `ScriptTrace` rows from a NEW site parses

docs/plans/2026-09-05-flow-debugger.md

@@ -226,7 +226,7 @@ Add `using ZB.MOM.WW.Audit;` where `AuditEvent` is used (Commons already referen
 **Step 1: Failing test** (follow the existing ManagementService test style for invoking `HandleUpdateApiMethod` — find a sibling test that exercises `UpdateApiMethodCommand` and copy its fixture; assert `method.TraceAllCalls` round-trips and defaults to `false` for the 5-arg constructor).
 
 **Step 2: Implement**
-- `ApiMethod`: `public bool TraceAllCalls { get; set; }` with summary "When true every production call to this method is fully traced (Flow Debugger design §5.2); intended for low-volume integration methods — ~7 audit rows per call."
+- `ApiMethod`: `public bool TraceAllCalls { get; set; }` with summary "When true every production call to this method is fully traced (Flow Debugger design §5.2); intended for low-volume integration methods — 7-10 audit rows per call."
 - EF: `builder.Property(m => m.TraceAllCalls).IsRequired().HasDefaultValue(false);`
 - Migration: build first (CLAUDE.md gotcha), then `dotnet ef migrations add AddApiMethodTraceAllCalls --project src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase --startup-project src/ZB.MOM.WW.ScadaBridge.Host` (use the same `--startup-project` the previous migration used — check `git log -1 --stat -- src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260819201210_AddNotificationsListSortIndex.cs` for its commit message). Verify the generated `Up` contains exactly one `AddColumn<bool>("TraceAllCalls", "ApiMethods", nullable: false, defaultValue: false)`; delete and redo if empty.
 - Commands: `CreateApiMethodCommand(..., string? ReturnDefinition, bool TraceAllCalls = false)`, `UpdateApiMethodCommand(..., string? ReturnDefinition, bool? TraceAllCalls = null)` (null = leave unchanged, so old CLIs don't reset it). Handlers copy the value.
@@ -754,7 +754,7 @@ public sealed record FlowDebugOutcome(FlowGraph? Graph, FlowDebugRunResult? Raw,
 **Files:**
 - Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Audit/AuditEventDetail.razor` (+ `.razor.cs`): button `Open in Flow Debugger` (`data-test="open-flow-debugger"`) beside *View execution chain* → `/deployment/flow-debugger?executionId={exec}`; rendered when `ExecutionId` is non-null.
 - Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Audit/ExecutionTreePage.razor`: same button next to *View this execution in the Audit Log*.
-- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Design/ApiMethodForm.razor`: under the Timeout field a `form-check` "Trace all calls (Flow Debugger)" bound to `_traceAllCalls`, help text "Adds ~7 audit rows per call; for low-volume integration methods", saved onto `_existing.TraceAllCalls` / the create path.
+- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Design/ApiMethodForm.razor`: under the Timeout field a `form-check` "Trace all calls (Flow Debugger)" bound to `_traceAllCalls`, help text "Adds 7–10 audit rows per call; for low-volume integration methods", saved onto `_existing.TraceAllCalls` / the create path.
 - Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Audit/FlowDebuggerDrillinTests.cs`, extend the existing `ApiMethodForm` test file if one exists (grep `ApiMethodForm` in tests).
 
 Commit — `git commit -am "feat(central-ui): Flow Debugger drill-ins from Audit Log + Execution Chain; TraceAllCalls checkbox on the method form"`

docs/plans/2026-09-06-flow-debugger-arch-review-remediation.md

@@ -591,7 +591,7 @@ var m = unmatched.FirstOrDefault(p => p.Kind == s.Kind && string.Equals(p.Title,
 - Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Design/ApiMethodForm.razor:50`, `src/ZB.MOM.WW.ScadaBridge.CLI/Commands/ApiMethodCommands.cs:65,101`, `src/ZB.MOM.WW.ScadaBridge.CLI/README.md:2122,2140`, `docs/requirements/Component-InboundAPI.md:73` ("~7" → "7–10 audit rows per call (3 of them trace-only)")
 - Modify: `docs/requirements/Component-InboundAPI.md` (new sentences: `SchemaRefResolver` is public for the Flow Debugger runner; `RouteHelper.WithSiteScope` + `SITE_SCOPE_DENIED`; `RouteCall` honours `SkipBodyCapture`)
 - Modify: `docs/requirements/Component-CentralUI.md` (Flow Debugger: page policy `RequireFlowDebugger`, Live controls Deployer-only, client wait derivation)
-- Modify: `CLAUDE.md` (Flow Debugger bullet: "~7 rows/call" → "7–10", one sentence on F1/F2/F3 outcomes with the review doc path)
+- Modify: `CLAUDE.md` (Flow Debugger bullet: the per-call trace-row estimate → "7–10", one sentence on F1/F2/F3 outcomes with the review doc path)
 - Modify: `docs/plans/2026-09-06-flow-debugger-arch-review.md` (append `## 7. Resolution` listing each finding → commit)
 
 Text to write (verbatim where possible):
@@ -600,7 +600,9 @@ Text to write (verbatim where possible):
 - §9 redaction bullet: "Trace payloads pass through the existing redaction policy on BOTH copies: the stored row (writer) and the inline Live copy (`FlowTraceCollector` applies `IAuditRedactor` and a 1 MB byte budget beside the 500-event cap — arch-review F1)." Add: "The method entry enforces the operator's site scope at route time (`RouteHelper.WithSiteScope`, `SITE_SCOPE_DENIED` → `ManagementUnauthorized`, arch-review F2)."
 - §10: "≈ 7–10 rows (a cached DB write and a notification add 3 trace-only rows)"; "Every routed call adds one central `RouteCall` insert — at inbound-driven rates that doubles the per-call central audit write (accepted, D5)"; "The Live run's inbound row is written AFTER the run completes, so a Replay taken during a long Live run shows only the `RouteCall`."
 
-**Step 4:** `grep -rn "~7 audit\|~7 rows\|roughly seven" docs src CLAUDE.md` → no hits.
+**Step 4:** grep `docs src CLAUDE.md` for the pre-remediation estimate wordings (the tilde-7 forms
+and the spelled-out "seven audit rows" one) → no hits. The patterns are deliberately not written out
+in this file so the check can return zero hits.
 
 **Step 5: Commit** `docs: Flow Debugger design reconciled with the implementation (arch-review F12, F13)`
 

docs/plans/2026-09-06-flow-debugger-arch-review.md

@@ -246,8 +246,9 @@ for the debug-stream path" or dropped from the client mapping.
 - Design §5.4 (debug-stream subscription) vs the polling implementation — see F12.
 - Design D6 / §9 (Replay = OperationalAudit) vs Deployer everywhere — see F3.
 - Design §9 (redaction applies to trace payloads) vs F1.
-- `ApiMethodForm` / CLI help say "~7 audit rows per call"; the outbound-calls addendum makes a
+- `ApiMethodForm` / CLI help understate the row count (they were written before the outbound-calls
-  call with one cached DB write and one notification **10 rows** (3 trace-only). Say "7–10".
+  addendum, which makes a call with one cached DB write and one notification **10 rows**, 3 of them
+  trace-only). Say "7–10 audit rows per call".
 - `FlowDebugRunner.RunMethodAsync` writes the inbound row *after* the run, so a Replay taken during
   a long Live run shows only the `RouteCall`. Fine, but not stated anywhere.
 - `SchemaRefResolver` became `public` for the runner; note it in Component-InboundAPI or move the
@@ -328,3 +329,30 @@ touches the Flow Debugger; `Deployment/FlowDebuggerTests` (Live run + localStora
 Consequence worth recording alongside the login-disabled decision: the rig can no longer prove
 role gating in the browser, so the Playwright suite's role-negative tests need either a second,
 login-enabled central to target or a fixture that overrides the identity per test.
+
+## 7. Resolution
+
+All thirteen findings were remediated on branch `fix/flow-debugger-arch-review` (plan
+`docs/plans/2026-09-06-flow-debugger-arch-review-remediation.md`, 2026-09-06). Nothing was
+deferred to the register.
+
+| # | Outcome | Commit |
+|---|---------|--------|
+| F1 | `FlowTraceCollector` moved to Commons `Types/Debugging/`, applies `IAuditRedactor` inside `Add` and caps at 500 events **and** 1 MB of summed `DetailsJson`; the site launcher resolves the redactor from the run's scope, and the central `FlowDebugRunner` collects through the same instance and reports its `Truncated`. | `e79452d2` fix(site-runtime): Flow Debugger inline trace copy is redacted and byte-capped at the collector (arch-review F1, site half) · `def6bab3` fix(management): Flow Debugger Live result is the redacted, capped trace copy (arch-review F1, central half) |
+| F2 | `RouteHelper.WithSiteScope(IReadOnlySet<string>?)` refuses an out-of-scope target site before any site traffic (`RouteSiteScopeException` → executor code `SITE_SCOPE_DENIED`); the runner rethrows it as `SiteScopeViolationException` (`ManagementUnauthorized`) **after** writing the inbound audit row. Administrators and unscoped users are exempt; the HTTP endpoint is unchanged. | `20a05941` fix(inbound-api): Flow Debugger method-entry Live run enforces the operator's site scope at route time (arch-review F2) |
+| F3 | New `AuthorizationPolicies.RequireFlowDebugger` (`FlowDebuggerAccessRoles` = Deployer, Administrator, Viewer) gates the page, and `FlowDebugReplayCommand` accepts the same set; the two run commands, the Live controls, Retry/Discard and the `?methodId=`/`?instanceId=` drill-ins stay Deployer-only, with the page pinning Replay mode for a non-Deployer. | `f86eb61d` fix(central-ui): Flow Debugger Replay opens to OperationalAudit roles; Live run and Retry/Discard stay Deployer-only (arch-review F3) |
+| F4 | Client waits derive from the server's bound: `RunAskTimeoutFor(timeoutSeconds)` = method timeout + `RunGrace` + 15 s slack (floor 30 s), `ValueChangeAskTimeoutFor(wait)` = wait + 15 s; the CLI gains `run-method --timeout-seconds` (default 120) and waits `--wait-seconds + 25` on `run-value-change`. | `97b0a4ba` fix(central-ui,cli): Flow Debugger client waits derive from the method timeout / wait window instead of a fixed 2 minutes (arch-review F4) |
+| F5 | The graph orders executions by emission sequence and uses timestamps only **within** one execution, so a site clock offset can no longer reorder the swimlane; the Live runner keeps the collector's emission order. | `1dfb9cee` fix(commons): Flow graph orders executions by emission sequence, timestamps only within one clock (arch-review F5) |
+| F6 | A failed traced DCL write disarms the trace arm (`DisarmTrace` self-message), so an unrelated device update can no longer consume it. | `50bf9ac7` fix(site-runtime): trace arm is cleared when the traced DCL write fails (arch-review F6) |
+| F7 | A traced run shed by `MaxConcurrentRunsPerScript` or suppressed by `MinTimeBetweenRuns` emits a Failed `ScriptRun` row (`shed: MaxConcurrentRunsPerScript …` / `suppressed: MinTimeBetweenRuns …`) instead of vanishing from the graph. | `105541b6` fix(site-runtime): shed or rate-limited traced runs emit a Failed ScriptRun row (arch-review F7) |
+| F8 | All four trace/outbound fault continuations run on `TaskScheduler.Default`, and the central emitters log a Warning on a failed audit write. | `262de38c` fix(audit): trace/outbound fault continuations run on the default scheduler and central emitters log (arch-review F8) |
+| F9 | `GetTraceEventsAsync` and `GetExecutionTreeAsync` share one chain walker in `AuditLogRepository`. | `297ae56b` refactor(config-db): GetTraceEventsAsync and GetExecutionTreeAsync share one chain walker (arch-review F9) |
+| F10 | The prediction overlay matches a real step by `(Kind, Title)` first, falling back to kind only, so two predicted nodes of the same kind no longer collapse onto one another. | `688136de` fix(central-ui): Flow Debugger prediction overlay matches by kind and title (arch-review F10) |
+| F11 | The always-on `RouteCall` row honours the method's `SkipBodyCapture` — the executor builds the helper with `RouteHelper.WithBodyCapture(false)`; the row is still written, metadata only. | `453c3dec` fix(inbound-api): RouteCall audit row honours the method's SkipBodyCapture override (arch-review F11) |
+| F12 | `AttributeValueUpdate.source_execution_id` / `trace` (fields 7/8) are **kept, populated and unconsumed** — documented as reserved for a future debug-stream path in `sitestream.proto` and design §5.4, with the field numbers pinned against reuse. Not removed: dropping them would be a wire change for no gain. | this commit |
+| F13 | Design D6 / §5.4 / §9 / §10, `Component-CentralUI.md`, `Component-InboundAPI.md`, `CLAUDE.md`, the proto comment and the CLI/UI help strings reconciled with the shipped code; the trace-row estimate is "7–10 audit rows per call" everywhere; `SchemaRefResolver`'s `public` visibility and the inbound-row-after-the-run ordering are now stated. | this commit — docs: Flow Debugger design reconciled with the implementation (arch-review F12, F13) |
+
+One test defect surfaced by §3's performance commits was fixed on the same branch:
+`OperationTrackingStoreTests.ReadChangedSince_BoundaryTimestamp…` raced the new single-writer
+queue (a missing `FlushAsync` before the test's direct `UPDATE`) — `86451382` test(site-runtime):
+flush the tracking-store queue before the boundary-timestamp test's direct UPDATE.

docs/plans/sql/AddApiMethodTraceAllCalls.sql

@@ -6,7 +6,7 @@ SET ANSI_NULLS ON;
 GO
 
 -- Flow Debugger (design 2026-09-05 §5.2): ApiMethod.TraceAllCalls — when 1, every
--- production call to the method is fully traced (~7 audit rows per call). Production
+-- production call to the method is fully traced (7-10 audit rows per call). Production
 -- twin of EF migration 20260906034113_AddApiMethodTraceAllCalls.
 BEGIN TRANSACTION;
 IF NOT EXISTS (

docs/requirements/Component-CentralUI.md

@@ -215,13 +215,14 @@ Per-leaf alarm rendering (leaf nodes are individual conditions for native alarms
 - **Row tooltip** — surfaces native metadata not warranting its own column: `AlarmTypeName`, category, operator user and comment, original raise time, current/limit value.
 - **Computed alarms render unchanged** from the prior flat-table style; the enrichment is purely additive for native rows.
 
-### Flow Debugger (Deployment Role)
+### Flow Debugger (Flow Debugger policy; Live run = Deployment Role)
 - **`/deployment/flow-debugger`** (nav: Deployment → Flow Debugger; design `docs/plans/2026-09-05-flow-debugger-design.md`, plan `docs/plans/2026-09-05-flow-debugger.md`). Runs one entry point with operator-supplied inputs and renders every step the run passes through as a **four-lane swimlane** (Central / Site / Tags / **External**) — request, route call, site script run, attribute writes, attribute waits, **outbound API / database / notification calls**, response — each node carrying its input, output, duration and outcome; a tabbed detail panel (Input · Output · Writes *or* Lifecycle · Timing, plus a *Where this ran* column) opens on the selected node with `Open in Debug View` (value-change entries, where the instance is known) and `Audit rows (n)` (→ `/audit/log?executionId=`) drill-ins.
 - **Two modes.** *Live* is a real run — the method script executes on the active central node with tracing on, or a real attribute write goes through the routed-write path and the site's triggered run is polled back — audited like any other call. *Replay* executes nothing: it rebuilds the flow from an existing execution's `ScriptTrace` audit rows plus its `ApiOutbound` / `DbOutbound` / `Notification` rows (`?executionId=` drill-in from the Audit Log drawer's and the Execution Chain page's **Open in Flow Debugger** actions).
 - **External lane (v1.1, plan `docs/plans/2026-09-06-flow-debugger-outbound-calls.md`).** Every `ExternalSystem.Call/CachedCall`, `Database.*` and `Notify.Send` a traced run makes is a node on the fourth lane — kinds `ApiCall` / `DbCall` / `Notify` (eyebrows `API CALL` / `DATABASE` / `NOTIFY`), title = the system.method / connection / list, a status pill (`Ok` / `Pending` / `Parked` / `Failed`) plus `sync` | `cached` | `notify`. **One node per operation:** a cached call's or notification's later lifecycle rows (Forwarded → Attempted → Delivered / Parked, written minutes later by the S&amp;F forwarder or the Notification Outbox) fold into the node (`FlowStep.Lifecycle`, a Lifecycle tab in the detail panel with *Delivered by*) instead of drawing four boxes. A sync call sits in the execution chain; a cached call or notification hangs off it on a dashed **async** connector, and a trigger cascade's spawn edge carries a bolt marker. A Live run returns only the rows the run itself emitted, so a graph left with `Pending` nodes **refreshes itself** through the Replay command every 5 s for up to 2 min (page constants), and a *Refresh outcomes* button re-reads on demand in either mode; the state chip stays *Completed* — pending outbound work is not a failed run. A refresh **merges** the re-read rows into the run's known rows by `EventId` (fresh copy wins) and rebuilds the graph from the union, so it only ever adds nodes: the site's outbound rows reach central's audit store by telemetry seconds after the run returned them inline, and a Replay read taken in between would otherwise erase an already-drawn node (observed on the rig 2026-09-06 with a resolved cached DB write). The union is exact because the inline rows are the very `AuditEvent` objects the site writer persists. **v1.2 (follow-ups plan `docs/plans/2026-09-06-flow-debugger-followups.md`, register row 111 closed):** a folded node carries its `CorrelationId` (`FlowStep.CorrelationId` — the `TrackedOperationId` / `NotificationId`), so the detail panel deep-links to **`/site-calls/report?correlationId=<id>`** / **`/notifications/report?correlationId=<id>`** (*Open in Site Calls* / *Open in Notification report*; both pages accept that query, pre-filter to the one row through an additive per-id query filter — `SiteCallQueryRequest.TrackedOperationIdFilter` / `NotificationOutboxQueryRequest.NotificationIdFilter` — show a removable *Operation* / *Notification* pill and auto-open the row's detail modal; note `NotificationId` is minted as a Guid in **`N` form** — no dashes — so the Notify link passes `N` and the report page normalises any Guid-shaped value to `N`, while `TrackedOperationId` is a `varchar(36)` `D` string and the Site Calls filter binds a `varchar` parameter for the PK seek), and a **`Parked` node offers Retry / Discard in place**: the same site-scope re-check, confirm dialog (Discard = danger) and central→site / Notification Outbox relay the report pages use (`IFlowDebugService.RetryOutboundAsync` / `DiscardOutboundAsync` → `CommunicationService`), then the page re-polls Replay (5 s / 2 min) until the node leaves `Parked`. **Prediction is async-safe:** `FlowDebugService.PredictMethodAsync` resolves route targets by a resolve-and-rewalk loop — its original blocking resolver deadlocked the whole Blazor circuit on the renderer sync context the moment a method routing to a site script was chosen (found 2026-09-06; pinned by a single-thread-pump unit test and the Playwright `FlowDebuggerTests`, which log in through the fixture and run a Live run of an ephemeral method in a real browser).
-- **Two entry points.** An inbound API method (inputs form generated from the method's `ParameterDefinitions`, values coerced by declared type) or a value-change trigger on a named instance attribute (Site → Instance → Attribute pickers, site-scope filtered).
+- **Authorization (arch-review F3).** The page is `RequireFlowDebugger` — `AuthorizationPolicies.FlowDebuggerAccessRoles` = Deployer, Administrator, Viewer — so an auditor can open it and Replay a past execution without the ability to run anything, and `FlowDebugReplayCommand` accepts the same set. **Everything that executes stays Deployer-only:** the two run commands (`FlowDebugRunMethodCommand` / `FlowDebugRunValueChangeCommand`), the Live entry-point controls, the parked-node Retry/Discard actions and the `?methodId=` / `?instanceId=` drill-ins; for a non-Deployer the page pins itself to Replay mode.
+- **Two entry points.** An inbound API method (inputs form generated from the method's `ParameterDefinitions`, values coerced by declared type) or a value-change trigger on a named instance attribute (Site → Instance → Attribute pickers, site-scope filtered). A method-entry Live run is additionally site-scoped **at route time** on the server (`RouteHelper.WithSiteScope` → `SITE_SCOPE_DENIED`, arch-review F2), so an operator cannot reach a site outside their scope by choosing a method that routes there.
 - **Predicted graph first.** Choosing an entry point renders the statically predicted shape (`FlowShapePredictor`, Script Analysis REQ-SA-7) before anything runs — nodes whose target depends on inputs render dashed; after a run, predicted nodes the run never produced render greyed.
-- **Execution path.** `IFlowDebugService` dispatches `FlowDebugRunMethodCommand` / `FlowDebugRunValueChangeCommand` / `FlowDebugReplayCommand` to the **per-node `ManagementActor`** through `ManagementActorHolder` (the same seam the HTTP `/management` endpoint uses), so the server enforces the Deployer gate and the active-node refusal. A standby node answers the two run commands with `STANDBY_NODE`, which the page renders as an inline alert with the server message verbatim; Replay is served from any node. The same three commands back the CLI's `flow-debug run-method | run-value-change | replay`.
+- **Execution path.** `IFlowDebugService` dispatches `FlowDebugRunMethodCommand` / `FlowDebugRunValueChangeCommand` / `FlowDebugReplayCommand` to the **per-node `ManagementActor`** through `ManagementActorHolder` (the same seam the HTTP `/management` endpoint uses), so the server enforces the Deployer gate and the active-node refusal. A standby node answers the two run commands with `STANDBY_NODE`, which the page renders as an inline alert with the server message verbatim; Replay is served from any node. The same three commands back the CLI's `flow-debug run-method | run-value-change | replay`. **The client wait is derived from the server's own bound, not a fixed duration (arch-review F4):** `FlowDebugService.RunAskTimeoutFor(timeoutSeconds)` = the method's `TimeoutSeconds` + the server's `FlowDebugRunner.RunGrace` (10 s) + 15 s client slack, floored at 30 s, and `ValueChangeAskTimeoutFor(waitSeconds)` = the trigger-wait window + 15 s (Replay stays a flat 30 s); the CLI mirrors it with `run-method --timeout-seconds` (default 120, set it to the method's timeout + 25 for long methods) and a `run-value-change` wait of `--wait-seconds + 25`. A long-running method used to be cut off client-side before the server's own grace expired.
 - State machine `Idle → Predicted → Running → Completed | Failed | TimedOut`; the status strip shows run number, start, total duration, step/boundary/write counts and the root `ExecutionId` (linking to the Execution Chain page). A run whose route call came back with no child execution and no Site-lane steps shows a *site not traced* note (older site build, or an untraced call). Last entry point and per-method inputs persist in `localStorage`.
 - The renderer is the custom-SVG `FlowGraphView` (no third-party charting) — pure, unit-tested layout; culture-invariant coordinates; each node a keyboard-activatable `role="button"`. Node text is HTML inside an SVG `foreignObject` so the browser measures it: the title wraps to two lines and ellipsizes by width (never by character count), the subtitle by width on one line, and the hover tooltip carries the full text. A lane whose real titles share a dotted head (Tags: `DelmiaReceiver.RecipeDownloadFlag`, `DelmiaReceiver.RecipeProcessedFlag`, …) shows that head once in the lane gutter (`DelmiaReceiver.*`) and each node shows only the tail that tells it apart (`FlowGraphView.CommonDottedPrefix`).
 

docs/requirements/Component-InboundAPI.md

@@ -70,7 +70,7 @@ Each API method definition includes:
   - Field names and (extended-system) data types. Supports returning **lists of objects** and arbitrarily nested structures.
 - **Implementation Script**: C# script that executes when the method is called. Stored **inline** in the method definition. Follows standard C# authoring patterns but has no template inheritance — it is a standalone script tied to this method.
 - **Timeout**: Configurable per method. Defines the maximum time the method is allowed to execute (including any routed calls to sites) before returning a timeout error to the caller.
-- **TraceAllCalls** (Flow Debugger, 2026-09-05): when set, every call to this method runs its routed site scripts with the trace flag on, so their `ScriptRun` / `AttributeWrite` / `AttributeWait` audit rows (`AuditChannel.ScriptTrace`) are recorded for Replay. Default off — adds roughly seven audit rows per call, so it is meant for low-volume integration methods. Set from the method form's *Trace all calls* checkbox or `api-method create|update --trace-all-calls`.
+- **TraceAllCalls** (Flow Debugger, 2026-09-05): when set, every call to this method runs its routed site scripts with the trace flag on, so their `ScriptRun` / `AttributeWrite` / `AttributeWait` audit rows (`AuditChannel.ScriptTrace`) are recorded for Replay. Default off — adds 7–10 audit rows per call (3 of them trace-only; a cached DB write and a notification account for the upper end), so it is meant for low-volume integration methods. Set from the method form's *Trace all calls* checkbox or `api-method create|update --trace-all-calls`.
 
 > **Authoring note — idempotent methods for at-least-once callers**: some external callers retry
 > on their own connect-failure/timeout failover loop and cannot tell "the first attempt actually
@@ -299,11 +299,14 @@ For everything else, scripts interact with the system through the curated `Route
 - The `Route.To()` helper resolves the instance's site assignment from the configuration database and routes the request to the correct site cluster via the Communication Layer.
 - The call is **synchronous from the API caller's perspective** — the API method blocks until the site responds or the **method-level timeout** is reached.
 - If the target site is unreachable or the call times out, the call fails and the API returns an error to the caller. No store-and-forward buffering is used for inbound API calls.
-- **Every `Route.To(...).Call(...)` writes one always-on `ScriptTrace.RouteCall` audit row** (central direct write, actor `inbound-api`, target `{instance}.{script}`, `ExecutionId` = the inbound request's, `Extra.childExecutionId` = the site run's id) — this row is what joins the central and site halves of a Flow Debugger Replay, so it is not gated. The routed request carries a `Trace` flag beside `ParentExecutionId` (`RouteToCallRequest.Trace`, `RouteToSetAttributesRequest.Trace`); it is set when the method has `TraceAllCalls` or the call is a Flow Debugger Live run, and only then does the site emit its own trace rows and return them inline on `RouteToCallResponse.TraceEvents` (with `ChildExecutionId`). Untraced calls are byte-identical to before.
+- **Every `Route.To(...).Call(...)` writes one always-on `ScriptTrace.RouteCall` audit row** (central direct write, actor `inbound-api`, target `{instance}.{script}`, `ExecutionId` = the inbound request's, `Extra.childExecutionId` = the site run's id) — this row is what joins the central and site halves of a Flow Debugger Replay, so it is not gated. The routed request carries a `Trace` flag beside `ParentExecutionId` (`RouteToCallRequest.Trace`, `RouteToSetAttributesRequest.Trace`); it is set when the method has `TraceAllCalls` or the call is a Flow Debugger Live run, and only then does the site emit its own trace rows and return them inline on `RouteToCallResponse.TraceEvents` (with `ChildExecutionId`). Untraced calls are byte-identical to before. **The `RouteCall` row honours the method's `SkipBodyCapture`** — the executor builds the helper with `RouteHelper.WithBodyCapture(!skipBodyCapture)`, so a method that opts out of body capture no longer leaks its arguments and reply through the always-on trace row; the row itself is still written, with metadata only (arch-review F11).
 
 ## Flow Debugger runs
 
-The Flow Debugger (Central UI, design `docs/plans/2026-09-05-flow-debugger-design.md`) drives this component through three management commands handled by the per-node `ManagementActor` (Deployer role): `FlowDebugRunMethodCommand` executes a method exactly as an external caller would — `ParameterValidator` + `SchemaRefResolver`, then `InboundScriptExecutor.ExecuteAsync(..., trace: true)` with a `RouteHelper` trace sink collecting the `RouteCall` row and the site's inline events — and **synthesizes the `ApiInbound.InboundRequest` row itself** with the operator as actor (`flow-debugger` source tag) because no HTTP request passed through `AuditWriteMiddleware`; it refuses on a standby node with `STANDBY_NODE` (`IActiveNodeGate`), so the script runs only where a real call would. `FlowDebugRunValueChangeCommand` writes an attribute through the traced routed-write path and polls the site (`PullTraceEvents`) for the run the change spawned; `FlowDebugReplayCommand` reads the chain back from the central Audit Log. The result (`FlowDebugRunResult`) carries the audit events inline — the graph is a central-side projection, never a wire type.
+The Flow Debugger (Central UI, design `docs/plans/2026-09-05-flow-debugger-design.md`) drives this component through three management commands handled by the per-node `ManagementActor` (Deployer role): `FlowDebugRunMethodCommand` executes a method exactly as an external caller would — `ParameterValidator` + `SchemaRefResolver`, then `InboundScriptExecutor.ExecuteAsync(..., trace: true)` with a `RouteHelper` trace sink collecting the `RouteCall` row and the site's inline events — and **synthesizes the `ApiInbound.InboundRequest` row itself** with the operator as actor (`flow-debugger` source tag) because no HTTP request passed through `AuditWriteMiddleware`; it refuses on a standby node with `STANDBY_NODE` (`IActiveNodeGate`), so the script runs only where a real call would. `FlowDebugRunValueChangeCommand` writes an attribute through the traced routed-write path and polls the site (`PullTraceEvents`) for the run the change spawned; `FlowDebugReplayCommand` reads the chain back from the central Audit Log and therefore accepts the wider `AuthorizationPolicies.FlowDebuggerAccessRoles` set (Deployer, Administrator, Viewer) while the two run commands stay Deployer-only (arch-review F3). The result (`FlowDebugRunResult`) carries the audit events inline — the graph is a central-side projection, never a wire type.
+
+- **`SchemaRefResolver` is `public` for the runner's benefit.** It was internal to the HTTP path; the Flow Debugger's `FlowDebugRunMethodCommand` validates its operator-supplied parameters through exactly the same `ParameterValidator` + `SchemaRefResolver` pair a real HTTP call uses, so a Live run cannot accept an input the endpoint would reject. Keep the two paths on the one resolver — a second, debugger-local validator would silently fork the contract.
+- **Site scope is enforced at route time on a Live run** (arch-review F2). `RouteHelper.WithSiteScope(IReadOnlySet<string>? permittedSiteIdentifiers)` carries the operator's permitted **site identifiers**; `RouteTarget.ResolveSiteAsync` throws `RouteSiteScopeException` before any site traffic when the resolved target site is outside that set, and `InboundScriptExecutor` maps it to error code **`SITE_SCOPE_DENIED`**. `FlowDebugRunner` translates that code into a `SiteScopeViolationException` (`ManagementUnauthorized`) *after* writing the inbound audit row, so the denied attempt is recorded under the operator's name. Administrators and unscoped users get a null scope (unrestricted), and the HTTP endpoint passes no scope at all — production routing is unchanged.
 
 ## Authentication Details
 

src/ZB.MOM.WW.ScadaBridge.CLI/Commands/ApiMethodCommands.cs

@@ -62,7 +62,7 @@ public static class ApiMethodCommands
         var returnDefOption = new Option<string?>("--return-def") { Description = "Return type definition" };
         var traceAllCallsOption = new Option<bool>("--trace-all-calls")
         {
-            Description = "Trace every production call to this method for the Flow Debugger (adds ~7 audit rows per call; low-volume integration methods only)"
+            Description = "Trace every production call to this method for the Flow Debugger (adds 7-10 audit rows per call; low-volume integration methods only)"
         };
 
         var cmd = new Command("create") { Description = "Create an API method" };
@@ -98,7 +98,7 @@ public static class ApiMethodCommands
         // bool? so an update that omits the flag leaves the stored value unchanged.
         var traceAllCallsOption = new Option<bool?>("--trace-all-calls")
         {
-            Description = "Trace every production call to this method for the Flow Debugger (true/false; omitted = unchanged; adds ~7 audit rows per call)"
+            Description = "Trace every production call to this method for the Flow Debugger (true/false; omitted = unchanged; adds 7-10 audit rows per call)"
         };
 
         var cmd = new Command("update") { Description = "Update an API method" };

src/ZB.MOM.WW.ScadaBridge.CLI/README.md

@@ -2119,7 +2119,7 @@ scadabridge --url <url> api-method create --name <string> --script <string> [--t
 | `--timeout` | no | `30` | Timeout in seconds |
 | `--parameters` | no | — | Parameter definitions JSON |
 | `--return-def` | no | — | Return type definition JSON |
-| `--trace-all-calls` | no | off | Trace every production call for the Flow Debugger. **Adds ~7 audit rows per call** — low-volume integration methods only |
+| `--trace-all-calls` | no | off | Trace every production call for the Flow Debugger. **Adds 7–10 audit rows per call** (3 of them trace-only) — low-volume integration methods only |
 
 #### `api-method update`
 
@@ -2137,7 +2137,7 @@ scadabridge --url <url> api-method update --id <int> --script <string> [--timeou
 | `--timeout` | no | `30` | Timeout in seconds |
 | `--parameters` | no | — | Parameter definitions JSON |
 | `--return-def` | no | — | Return type definition JSON |
-| `--trace-all-calls` | no | unchanged | `true`/`false` — trace every production call for the Flow Debugger. **Adds ~7 audit rows per call.** Omitted = the stored value is left as-is (the one field the replace-all rule does not apply to) |
+| `--trace-all-calls` | no | unchanged | `true`/`false` — trace every production call for the Flow Debugger. **Adds 7–10 audit rows per call** (3 of them trace-only). Omitted = the stored value is left as-is (the one field the replace-all rule does not apply to) |
 
 Script changes take effect immediately — the updated code is recompiled in-memory on the active central node. No restart is required.
 

src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Design/ApiMethodForm.razor

@@ -47,7 +47,7 @@
                         </label>
                         <div class="form-text">
                             Every call to this method records its routed site steps as audit rows the
-                            Flow Debugger can replay. Adds ~7 audit rows per call; for low-volume
+                            Flow Debugger can replay. Adds 7–10 audit rows per call; for low-volume
                             integration methods.
                         </div>
                     </div>

src/ZB.MOM.WW.ScadaBridge.Commons/Entities/InboundApi/ApiMethod.cs

@@ -17,7 +17,7 @@ public class ApiMethod
 
     /// <summary>
     /// When true every production call to this method is fully traced (Flow Debugger design
-    /// 2026-09-05 §5.2); intended for low-volume integration methods — ~7 audit rows per call.
+    /// 2026-09-05 §5.2); intended for low-volume integration methods — 7-10 audit rows per call.
     /// </summary>
     public bool TraceAllCalls { get; set; }
 

src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto

@@ -100,8 +100,10 @@ message AttributeValueUpdate {
   Quality quality = 5;
   google.protobuf.Timestamp timestamp = 6;
   // Flow Debugger (design 2026-09-05 §5.3/§5.4): the writing execution's id ("D" GUID
-  // form; empty represents null) and the trace flag, so the debug stream can hand the
+  // form; empty represents null) and the trace flag. Additive; an older site never sets them.
-  // debugger the spawned run's parent. Additive; an older site never sets them.
+  // RESERVED — populated by the site, consumed by nothing (arch-review F12): the value-change
+  // entry polls the site's PullTraceEvents RPC instead of subscribing to the debug stream, so
+  // these two fields are kept for a future debug-stream path. Do not reuse the field numbers.
   string source_execution_id = 7;
   bool trace = 8;
 }