diff --git a/CLAUDE.md b/CLAUDE.md index 892ebc83..987bb300 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -196,7 +196,7 @@ spec for each is `docs/requirements/Component-.md`, and `README.md` carrie - Central UI: Blazor Server (ASP.NET Core + SignalR) with Bootstrap CSS. No third-party component frameworks (no Blazorise, MudBlazor, Radzen, etc.). Build custom Blazor components for tables, grids, forms, etc. - UI design: Clean, corporate, internal-use aesthetic. Not flashy. Use the `frontend-design` skill when designing UI pages/components. - Debug view: real-time streaming via DebugStreamBridgeActor + gRPC (events via `SiteStreamGrpcClient`, snapshot via the `SiteCommandService` query surface — `QueryReply.DebugViewSnapshot`). Health dashboard: 10s polling timer. Deployment status: real-time push via SignalR. -- **Flow Debugger (2026-09-05, Deployment → `/deployment/flow-debugger` + CLI `flow-debug run-method|run-value-change|replay`).** Runs one entry point (an inbound API method, or a value change on an instance attribute) and draws every step as a Central / Site / Tags / **External** swimlane; *Live* really runs it, *Replay* rebuilds a past execution. **v1.1 (2026-09-06, `docs/plans/2026-09-06-flow-debugger-outbound-calls.md`):** every `ExternalSystem.*` / `Database.*` / `Notify.Send` a traced run makes is an `ApiCall` / `DbCall` / `Notify` node on the External lane — the site routes those already-existing `ApiOutbound` / `DbOutbound` / `Notification` rows through the same inline trace collector, the replay query widens to those channels, and `FlowGraphBuilder` folds a cached operation's rows by `(kind, CorrelationId)` into ONE node with a `Lifecycle` (sync calls in-chain, cached/notify on dashed `Async` edges, trigger cascades on `Spawn` edges, the Response linked from the inbound execution's last step). Later lifecycle rows are not inline, so after a Live run the page polls the Replay command (5 s, ≤ 2 min) while any node is `Pending`, plus a *Refresh outcomes* button. **A refresh MERGES, never replaces (2026-09-06, found live in the browser):** the page unions the re-read rows into the run's own rows by `EventId` and rebuilds the graph, because a Live run carries its site rows inline while the same rows reach central's audit store by telemetry ~5–30 s later — a Replay read taken in between lacks them, and the old wholesale swap made an already-drawn node (a resolved cached DB write) vanish until the next manual refresh. The inline rows ARE the audited rows (`FlowTraceCollector` sinks the same `AuditEvent` objects the site writer persists), so the union is exact. Central method-script `Database.*` calls (`InboundDatabaseHelper`) are now audited (`DbOutbound`/`DbWrite`) — they were not. Rig demo: `docker/seed-flow-outbound-demo.sh` + `verify-delmia-notifier.sh` step I. **Trace steps ARE audit events** — `AuditChannel.ScriptTrace` with kinds `RouteCall` (central, **always-on** per `Route.To().Call`, joins the two halves of a Replay) and `ScriptRun` / `AttributeWrite` / `AttributeWait` (site, **gated** by a `bool Trace` riding beside `ParentExecutionId` on `RouteToCallRequest` / `ScriptCallRequest` / `SetStaticAttributeCommand` / `AttributeValueChanged`; proto fields additive, `site_command.proto` imports `sitestream.proto`). A Live run gets the site rows **inline** on `RouteToCallResponse.TraceEvents` (+ `ChildExecutionId`) because the 5 s telemetry cadence is too slow; Replay reads them from `AuditLog` (`GetTraceEventsAsync`). A value-change on a data-sourced attribute is a REAL tag write; the Instance Actor's **trace arm** (`SiteRuntime:TraceArmWindowSeconds`, 30) stamps the device echo with the writer's execution id so the triggered run is parented and traced, and central polls the site's new `PullTraceEvents` RPC (point read, not reconciliation) for it. Execution path is three management commands on the **per-node `ManagementActor`** (`FlowDebug*`, Deployer role) — a standby refuses the two run commands with **`STANDBY_NODE`** (shown verbatim); `ApiMethod.TraceAllCalls` (migration `AddApiMethodTraceAllCalls`, CLI `--trace-all-calls`) traces every real call to a method for later Replay (~7 rows/call). Prediction before a run is a Roslyn syntax walk (`FlowShapePredictor`, REQ-SA-7, DB-free); the graph (`FlowStep`/`FlowGraphBuilder`, Commons) is a central projection, never a wire type; the renderer is custom SVG (`FlowGraphView`). Rig: `AuditLog:PerChannelRetentionDays:ScriptTrace=30` on both centrals; `verify-delmia-notifier.sh` step H proves live + replay. **Mixed-version caveat:** an older site ignores the flag → central `RouteCall` only, page says *site not traced*. **v1.2 (2026-09-06, `docs/plans/2026-09-06-flow-debugger-followups.md`):** a `Parked` External node offers **Retry / Discard in place** (same site-scope check, confirm and relay as the report pages, via `IFlowDebugService.RetryOutboundAsync`/`DiscardOutboundAsync`, then the page re-polls Replay until the node leaves Parked) and every folded node deep-links to `/site-calls/report?correlationId=` / `/notifications/report?correlationId=` (both pages accept it through additive `TrackedOperationIdFilter` / `NotificationIdFilter` query params and auto-open the row); register row 111 closed. **Trap closed the same day:** `FlowDebugService.PredictMethodAsync` used to resolve route targets with `GetAwaiter().GetResult()` inside the synchronous predictor walk — on the Blazor renderer sync context that DEADLOCKED the whole circuit (page inert, healthy websocket, no error) whenever a method routing to a site script was chosen; now a resolve-and-rewalk loop, pinned by a single-thread-pump test. Never block on a task from a Blazor event/lifecycle path. Browser checks of the page go through Playwright (`FlowDebuggerTests`, fixture login + an ephemeral method), not the login-disabled override. Design `docs/plans/2026-09-05-flow-debugger-design.md`; plan `docs/plans/2026-09-05-flow-debugger.md`. +- **Flow Debugger (2026-09-05, Deployment → `/deployment/flow-debugger` + CLI `flow-debug run-method|run-value-change|replay`).** Runs one entry point (an inbound API method, or a value change on an instance attribute) and draws every step as a Central / Site / Tags / **External** swimlane; *Live* really runs it, *Replay* rebuilds a past execution. **v1.1 (2026-09-06, `docs/plans/2026-09-06-flow-debugger-outbound-calls.md`):** every `ExternalSystem.*` / `Database.*` / `Notify.Send` a traced run makes is an `ApiCall` / `DbCall` / `Notify` node on the External lane — the site routes those already-existing `ApiOutbound` / `DbOutbound` / `Notification` rows through the same inline trace collector, the replay query widens to those channels, and `FlowGraphBuilder` folds a cached operation's rows by `(kind, CorrelationId)` into ONE node with a `Lifecycle` (sync calls in-chain, cached/notify on dashed `Async` edges, trigger cascades on `Spawn` edges, the Response linked from the inbound execution's last step). Later lifecycle rows are not inline, so after a Live run the page polls the Replay command (5 s, ≤ 2 min) while any node is `Pending`, plus a *Refresh outcomes* button. **A refresh MERGES, never replaces (2026-09-06, found live in the browser):** the page unions the re-read rows into the run's own rows by `EventId` and rebuilds the graph, because a Live run carries its site rows inline while the same rows reach central's audit store by telemetry ~5–30 s later — a Replay read taken in between lacks them, and the old wholesale swap made an already-drawn node (a resolved cached DB write) vanish until the next manual refresh. The inline rows ARE the audited rows (`FlowTraceCollector` sinks the same `AuditEvent` objects the site writer persists), so the union is exact. Central method-script `Database.*` calls (`InboundDatabaseHelper`) are now audited (`DbOutbound`/`DbWrite`) — they were not. Rig demo: `docker/seed-flow-outbound-demo.sh` + `verify-delmia-notifier.sh` step I. **Trace steps ARE audit events** — `AuditChannel.ScriptTrace` with kinds `RouteCall` (central, **always-on** per `Route.To().Call`, joins the two halves of a Replay) and `ScriptRun` / `AttributeWrite` / `AttributeWait` (site, **gated** by a `bool Trace` riding beside `ParentExecutionId` on `RouteToCallRequest` / `ScriptCallRequest` / `SetStaticAttributeCommand` / `AttributeValueChanged`; proto fields additive, `site_command.proto` imports `sitestream.proto`). A Live run gets the site rows **inline** on `RouteToCallResponse.TraceEvents` (+ `ChildExecutionId`) because the 5 s telemetry cadence is too slow; Replay reads them from `AuditLog` (`GetTraceEventsAsync`). A value-change on a data-sourced attribute is a REAL tag write; the Instance Actor's **trace arm** (`SiteRuntime:TraceArmWindowSeconds`, 30) stamps the device echo with the writer's execution id so the triggered run is parented and traced, and central polls the site's new `PullTraceEvents` RPC (point read, not reconciliation) for it. Execution path is three management commands on the **per-node `ManagementActor`** (`FlowDebug*`, Deployer role) — a standby refuses the two run commands with **`STANDBY_NODE`** (shown verbatim); `ApiMethod.TraceAllCalls` (migration `AddApiMethodTraceAllCalls`, CLI `--trace-all-calls`) traces every real call to a method for later Replay (7–10 rows/call). Prediction before a run is a Roslyn syntax walk (`FlowShapePredictor`, REQ-SA-7, DB-free); the graph (`FlowStep`/`FlowGraphBuilder`, Commons) is a central projection, never a wire type; the renderer is custom SVG (`FlowGraphView`). Rig: `AuditLog:PerChannelRetentionDays:ScriptTrace=30` on both centrals; `verify-delmia-notifier.sh` step H proves live + replay. **Mixed-version caveat:** an older site ignores the flag → central `RouteCall` only, page says *site not traced*. **v1.2 (2026-09-06, `docs/plans/2026-09-06-flow-debugger-followups.md`):** a `Parked` External node offers **Retry / Discard in place** (same site-scope check, confirm and relay as the report pages, via `IFlowDebugService.RetryOutboundAsync`/`DiscardOutboundAsync`, then the page re-polls Replay until the node leaves Parked) and every folded node deep-links to `/site-calls/report?correlationId=` / `/notifications/report?correlationId=` (both pages accept it through additive `TrackedOperationIdFilter` / `NotificationIdFilter` query params and auto-open the row); register row 111 closed. **Trap closed the same day:** `FlowDebugService.PredictMethodAsync` used to resolve route targets with `GetAwaiter().GetResult()` inside the synchronous predictor walk — on the Blazor renderer sync context that DEADLOCKED the whole circuit (page inert, healthy websocket, no error) whenever a method routing to a site script was chosen; now a resolve-and-rewalk loop, pinned by a single-thread-pump test. Never block on a task from a Blazor event/lifecycle path. Browser checks of the page go through Playwright (`FlowDebuggerTests`, fixture login + an ephemeral method), not the login-disabled override. **Arch-reviewed 2026-09-06** (`docs/plans/2026-09-06-flow-debugger-arch-review.md`, remediation `docs/plans/2026-09-06-flow-debugger-arch-review-remediation.md`): the inline Live trace copy is now redacted and capped by the shared `FlowTraceCollector` (500 events **and** 1 MB of `DetailsJson`, F1), a method-entry Live run enforces the operator's site scope at route time (`RouteHelper.WithSiteScope` → `SITE_SCOPE_DENIED` → `ManagementUnauthorized`, F2), and the page/Replay opened to the `OperationalAudit` roles under the new `RequireFlowDebugger` policy while everything that executes stays Deployer-only (F3). Design `docs/plans/2026-09-05-flow-debugger-design.md`; plan `docs/plans/2026-09-05-flow-debugger.md`. - Health reports: 30s interval, 60s offline threshold, monotonic sequence numbers, raw error counts per interval. - Dead letter monitoring as a health metric. - Site Event Logging: 30-day retention, 1GB storage cap, daily purge, paginated queries with keyword search. **The cap measures `site_events` + its four indexes, NOT the consolidated LocalDb file** (register row 81, 2026-08-17): whole-file measurement made the trim unwinnable on a replicated node — each batch freed event pages while adding an equal count of oplog rows + tombstones to the same file, so it stopped 2.5x over a 64 MB cap and a node with a down peer could not enforce the cap at all. `dbstat` (the exact per-table measurement) is **NOT compiled into the bundled `SQLitePCLRaw.lib.e_sqlite3`** — probed and pinned by `SqliteCapabilityTests`, so don't reach for it — the footprint is modelled from sampled row width instead (−0.8%…+12.8%, biased high). Consequence: `MaxStorageMb` no longer bounds the data volume alone; LocalDb bookkeeping is separate overhead bounded by `MaxOplogRows`/`TombstoneRetention` and is deliberately not this component's job. Decision: `docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md`. diff --git a/docs/plans/2026-09-05-flow-debugger-design.md b/docs/plans/2026-09-05-flow-debugger-design.md index f626e0ff..be5fb658 100644 --- a/docs/plans/2026-09-05-flow-debugger-design.md +++ b/docs/plans/2026-09-05-flow-debugger-design.md @@ -27,7 +27,7 @@ on the `DelmiaReceiver` child → seven attribute writes → wait for Galaxy to | D3 | **Value-change on a data-sourced attribute does a REAL tag write** through the instance actor's normal write path; the device echo fires the trigger. Correlation is by a short-lived *trace arm* on the instance actor (§5.3). | A debug-only "inject value" command that synthesises the change without touching the device — leaves the instance's cached value diverged from the device until the next real update. | | D4 | **Live gets its trace inline; Replay reads audit rows.** Site audit telemetry forwards on a 5 s busy / 30 s idle cadence (`AuditLog:SiteTelemetry`), too slow to drive an interactive run, so the routed reply carries the site steps back and the same steps are *also* written as audit rows for Replay. | Live waiting on the audit pipeline. | | D5 | **Tracing is gated, not always-on.** Sites run scripts at high rates from timers and value changes; a trace flag rides beside `ParentExecutionId` and only flagged runs emit trace rows. One exception is always-on: a central `RouteCall` row per `Route.To().Call` (inbound-driven volume, joins the two halves of a Replay). | Tracing every run (volume), or a purely time-window join. | -| D6 | **Live run is a Deployer act.** `RequireDeployment` gates the page and the Run button; Replay needs `OperationalAudit`. | A separate `RequireOperator` gate for Live. | +| D6 | **Live run is a Deployer act.** `RequireDeployment` gates the Run controls and the two run commands; the page and Replay are open to `OperationalAudit` roles + Deployer (`RequireFlowDebugger`, arch-review F3). | A separate `RequireOperator` gate for Live. | ## 2. What exists today (reused, not rebuilt) @@ -159,20 +159,24 @@ the echo also fires inherit the parent id exactly as static cascades do today. `RouteToCallResponse` gains additive `ChildExecutionId` (`Guid?`) and `TraceEvents` (`IReadOnlyList?`, the very events that became the site's audit rows, in emission order); on the wire it is `repeated AuditEventDto trace_events` (the DTO `sitestream.proto` -already defines). The site collects events in a per-run `FlowTraceCollector` held by -`ScriptRuntimeContext` and attaches the list to the `ScriptCallResult` → `RouteToCallResponse` -when `Trace` is set. Capped at `MaxTraceEventsPerRun` (500); beyond it the collector keeps the -first 500, sets `Truncated`, and the audit rows remain complete. A site on an older build ignores +already defines). The site collects events in a per-run `FlowTraceCollector` (Commons +`Types/Debugging/`, shared with the central `FlowDebugRunner`) held by `ScriptRuntimeContext` and +attaches the list to the `ScriptCallResult` → `RouteToCallResponse` when `Trace` is set. The +collector applies `IAuditRedactor` inside `Add`, so the inline record is the same filtered event +the writer persists, and it caps at `MaxTraceEventsPerRun` (500) **or** 1 MB of summed +`DetailsJson` (`DefaultMaxBytes`), whichever comes first; beyond either bound it keeps the earlier +events, sets `Truncated`, and the audit rows remain complete (arch-review F1). A site on an older build ignores `Trace`, returns no `TraceEvents`, and the UI labels its lane `site not traced (older build)`. `FlowStep` is a **central-side projection** (`Commons/Types/Debugging/FlowStepProjector`) computed from `AuditEvent` + `AuditDetails`; nothing serializes it. -For the value-change entry there is no routed reply to ride on. The debugger instead subscribes -to the instance's debug stream (`SubscribeDebugView`, the Debug View's existing path) for the run's -duration and takes the spawned run's `ExecutionId` from the stamped `AttributeValueChanged`, then -pulls the site's trace rows for that id via a new `PullTraceEvents` RPC on `SiteStreamService` -(same PSK interceptor as `PullAuditEvents`; reads the site audit SQLite by `ExecutionId` / -`ParentExecutionId`). One additive RPC, no streaming change. +For the value-change entry there is no routed reply to ride on. The runner polls the site's +`PullTraceEvents` RPC by `ParentExecutionId` (1 s cadence, 2 s settle window, `WaitSeconds` bound) — +simpler than the debug-stream subscription originally planned. `AttributeValueUpdate.source_execution_id`/`trace` +(fields 7/8) are populated by the site and **consumed by nothing**; they are reserved for a future +debug-stream path and must not be reused. The RPC sits behind the same PSK interceptor as +`PullAuditEvents` and reads the site audit SQLite by `ExecutionId` / `ParentExecutionId` — one +additive RPC, no streaming change. ## 6. Message and wire changes (all additive) @@ -257,9 +261,14 @@ surface); the walker is syntax-only and never executes anything. ## 9. Security, authorisation, audit -- **Page visibility and Live run:** `RequireDeployment` (same as Debug View) — decided - 2026-09-05: a Live run writes real tags, but the Deployer already deploys and redeploys the - scripts that do exactly that, so no separate Operator gate. **Replay:** `OperationalAudit` (it +- **Page visibility:** `RequireFlowDebugger` — the `OperationalAudit` roles plus Deployer + (`AuthorizationPolicies.FlowDebuggerAccessRoles` = Deployer, Administrator, Viewer), so an + auditor can open the page and Replay a past execution (arch-review F3). **Live run:** + `RequireDeployment` (same as Debug View) — decided 2026-09-05: a Live run writes real tags, but + the Deployer already deploys and redeploys the scripts that do exactly that, so no separate + Operator gate. The two run commands, the Live controls, the parked-node Retry/Discard actions and + the `?methodId=`/`?instanceId=` drill-ins stay Deployer-only, and the page pins Replay mode for a + non-Deployer. **Replay:** `FlowDebugReplayCommand` accepts the same `OperationalAudit` roles (it reads audit rows). No Verifier step — the debugger is a diagnostic, not a production write control, and the audit row is the safeguard. - **Inbound audit row for a Live run:** the shared `AuditEvent.Actor` carries the operator's @@ -267,21 +276,38 @@ surface); the walker is syntax-only and never executes anything. `Extra.runNumber`; `Target` is the method as usual. API-key authentication is bypassed (there is no key), **parameter validation is not** — `ParameterValidator` runs exactly as for HTTP so a bad input fails the same way. The method's `TimeoutSeconds` applies. Site-scope (`SiteScopeService`) - restricts which instances a value-change entry may target. -- Trace payloads pass through the existing redaction policy (auth headers redacted, 8 KB cap, - 64 KB on error rows). Console output is capped the same way. + restricts which instances a value-change entry may target. The method entry enforces the + operator's site scope at route time (`RouteHelper.WithSiteScope`, `SITE_SCOPE_DENIED` → + `ManagementUnauthorized`, arch-review F2): the runner resolves the operator's permitted numeric + site ids to site identifiers and hands them to `RouteHelper`, which refuses an out-of-scope target + **before any site traffic**; the denial is rethrown as `SiteScopeViolationException` *after* the + inbound audit row is written, so the attempt is recorded under the operator's name. + Administrators and unscoped users are exempt, exactly as `EnforceSiteScope` treats them. +- Trace payloads pass through the existing redaction policy on BOTH copies: the stored row (writer) + and the inline Live copy (`FlowTraceCollector` applies `IAuditRedactor` and a 1 MB byte budget + beside the 500-event cap — arch-review F1). The row-level caps are unchanged (auth headers + redacted, 8 KB cap, 64 KB on error rows). Console output is capped the same way. - Nothing here touches the PSK boundary; `PullTraceEvents` sits behind `ControlPlaneAuthInterceptor`. ## 10. Volume and failure behaviour - Untraced production is unchanged except one `RouteCall` row per routed call (inbound-driven). + Every routed call adds one central `RouteCall` insert — at inbound-driven rates that doubles the + per-call central audit write (accepted, D5). - A traced Delmia call emits ≈ 1 `InboundRequest` + 1 `RouteCall` + 1 `ScriptRun` + 2 - `AttributeWrite` + 1 `AttributeWait` + 1 value-change `ScriptRun` ≈ 7 rows. `TraceAllCalls` on a - 100-call/day method is ~700 rows/day. + `AttributeWrite` + 1 `AttributeWait` + 1 value-change `ScriptRun` ≈ **7–10 rows** (a cached DB + write and a notification add 3 trace-only rows). `TraceAllCalls` on a 100-call/day method is + ~700–1 000 rows/day. - Trace emission is best-effort like all audit (a failed trace write never fails the run). - Live run timeout = method timeout + `FlowDebugRunGrace` (10 s); a timeout renders whatever steps arrived and marks the run `Timed out`. The site's trace rows still land later, so Replay of the - same ExecutionId is complete. + same ExecutionId is complete. The client waits longer than the server on purpose: the UI derives + its Ask wait from the same bound (`RunAskTimeoutFor` = method timeout + grace + 15 s slack, + floored at 30 s; `ValueChangeAskTimeoutFor` = the wait window + 15 s) and the CLI's + `run-method --timeout-seconds` defaults to 120 while `run-value-change` waits + `--wait-seconds + 25` (arch-review F4). +- **The Live run's inbound row is written AFTER the run completes**, so a Replay taken during a + long Live run shows only the `RouteCall`. - Older site build: no `TraceEvents`, lane labelled; older central build: ignores new fields; proto3 defaults negotiate as for stream batching. - **Mixed-version caveat:** an OLD central ingesting `ScriptTrace` rows from a NEW site parses diff --git a/docs/plans/2026-09-05-flow-debugger.md b/docs/plans/2026-09-05-flow-debugger.md index 33a3343f..1eadb5b3 100644 --- a/docs/plans/2026-09-05-flow-debugger.md +++ b/docs/plans/2026-09-05-flow-debugger.md @@ -226,7 +226,7 @@ Add `using ZB.MOM.WW.Audit;` where `AuditEvent` is used (Commons already referen **Step 1: Failing test** (follow the existing ManagementService test style for invoking `HandleUpdateApiMethod` — find a sibling test that exercises `UpdateApiMethodCommand` and copy its fixture; assert `method.TraceAllCalls` round-trips and defaults to `false` for the 5-arg constructor). **Step 2: Implement** -- `ApiMethod`: `public bool TraceAllCalls { get; set; }` with summary "When true every production call to this method is fully traced (Flow Debugger design §5.2); intended for low-volume integration methods — ~7 audit rows per call." +- `ApiMethod`: `public bool TraceAllCalls { get; set; }` with summary "When true every production call to this method is fully traced (Flow Debugger design §5.2); intended for low-volume integration methods — 7-10 audit rows per call." - EF: `builder.Property(m => m.TraceAllCalls).IsRequired().HasDefaultValue(false);` - Migration: build first (CLAUDE.md gotcha), then `dotnet ef migrations add AddApiMethodTraceAllCalls --project src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase --startup-project src/ZB.MOM.WW.ScadaBridge.Host` (use the same `--startup-project` the previous migration used — check `git log -1 --stat -- src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260819201210_AddNotificationsListSortIndex.cs` for its commit message). Verify the generated `Up` contains exactly one `AddColumn("TraceAllCalls", "ApiMethods", nullable: false, defaultValue: false)`; delete and redo if empty. - Commands: `CreateApiMethodCommand(..., string? ReturnDefinition, bool TraceAllCalls = false)`, `UpdateApiMethodCommand(..., string? ReturnDefinition, bool? TraceAllCalls = null)` (null = leave unchanged, so old CLIs don't reset it). Handlers copy the value. @@ -754,7 +754,7 @@ public sealed record FlowDebugOutcome(FlowGraph? Graph, FlowDebugRunResult? Raw, **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Audit/AuditEventDetail.razor` (+ `.razor.cs`): button `Open in Flow Debugger` (`data-test="open-flow-debugger"`) beside *View execution chain* → `/deployment/flow-debugger?executionId={exec}`; rendered when `ExecutionId` is non-null. - Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Audit/ExecutionTreePage.razor`: same button next to *View this execution in the Audit Log*. -- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Design/ApiMethodForm.razor`: under the Timeout field a `form-check` "Trace all calls (Flow Debugger)" bound to `_traceAllCalls`, help text "Adds ~7 audit rows per call; for low-volume integration methods", saved onto `_existing.TraceAllCalls` / the create path. +- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Design/ApiMethodForm.razor`: under the Timeout field a `form-check` "Trace all calls (Flow Debugger)" bound to `_traceAllCalls`, help text "Adds 7–10 audit rows per call; for low-volume integration methods", saved onto `_existing.TraceAllCalls` / the create path. - Test: `tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Audit/FlowDebuggerDrillinTests.cs`, extend the existing `ApiMethodForm` test file if one exists (grep `ApiMethodForm` in tests). Commit — `git commit -am "feat(central-ui): Flow Debugger drill-ins from Audit Log + Execution Chain; TraceAllCalls checkbox on the method form"` diff --git a/docs/plans/2026-09-06-flow-debugger-arch-review-remediation.md b/docs/plans/2026-09-06-flow-debugger-arch-review-remediation.md index bef47108..d266ca59 100644 --- a/docs/plans/2026-09-06-flow-debugger-arch-review-remediation.md +++ b/docs/plans/2026-09-06-flow-debugger-arch-review-remediation.md @@ -591,7 +591,7 @@ var m = unmatched.FirstOrDefault(p => p.Kind == s.Kind && string.Equals(p.Title, - Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Design/ApiMethodForm.razor:50`, `src/ZB.MOM.WW.ScadaBridge.CLI/Commands/ApiMethodCommands.cs:65,101`, `src/ZB.MOM.WW.ScadaBridge.CLI/README.md:2122,2140`, `docs/requirements/Component-InboundAPI.md:73` ("~7" → "7–10 audit rows per call (3 of them trace-only)") - Modify: `docs/requirements/Component-InboundAPI.md` (new sentences: `SchemaRefResolver` is public for the Flow Debugger runner; `RouteHelper.WithSiteScope` + `SITE_SCOPE_DENIED`; `RouteCall` honours `SkipBodyCapture`) - Modify: `docs/requirements/Component-CentralUI.md` (Flow Debugger: page policy `RequireFlowDebugger`, Live controls Deployer-only, client wait derivation) -- Modify: `CLAUDE.md` (Flow Debugger bullet: "~7 rows/call" → "7–10", one sentence on F1/F2/F3 outcomes with the review doc path) +- Modify: `CLAUDE.md` (Flow Debugger bullet: the per-call trace-row estimate → "7–10", one sentence on F1/F2/F3 outcomes with the review doc path) - Modify: `docs/plans/2026-09-06-flow-debugger-arch-review.md` (append `## 7. Resolution` listing each finding → commit) Text to write (verbatim where possible): @@ -600,7 +600,9 @@ Text to write (verbatim where possible): - §9 redaction bullet: "Trace payloads pass through the existing redaction policy on BOTH copies: the stored row (writer) and the inline Live copy (`FlowTraceCollector` applies `IAuditRedactor` and a 1 MB byte budget beside the 500-event cap — arch-review F1)." Add: "The method entry enforces the operator's site scope at route time (`RouteHelper.WithSiteScope`, `SITE_SCOPE_DENIED` → `ManagementUnauthorized`, arch-review F2)." - §10: "≈ 7–10 rows (a cached DB write and a notification add 3 trace-only rows)"; "Every routed call adds one central `RouteCall` insert — at inbound-driven rates that doubles the per-call central audit write (accepted, D5)"; "The Live run's inbound row is written AFTER the run completes, so a Replay taken during a long Live run shows only the `RouteCall`." -**Step 4:** `grep -rn "~7 audit\|~7 rows\|roughly seven" docs src CLAUDE.md` → no hits. +**Step 4:** grep `docs src CLAUDE.md` for the pre-remediation estimate wordings (the tilde-7 forms +and the spelled-out "seven audit rows" one) → no hits. The patterns are deliberately not written out +in this file so the check can return zero hits. **Step 5: Commit** `docs: Flow Debugger design reconciled with the implementation (arch-review F12, F13)` diff --git a/docs/plans/2026-09-06-flow-debugger-arch-review.md b/docs/plans/2026-09-06-flow-debugger-arch-review.md index 8cf46ee7..8a69e47b 100644 --- a/docs/plans/2026-09-06-flow-debugger-arch-review.md +++ b/docs/plans/2026-09-06-flow-debugger-arch-review.md @@ -246,8 +246,9 @@ for the debug-stream path" or dropped from the client mapping. - Design §5.4 (debug-stream subscription) vs the polling implementation — see F12. - Design D6 / §9 (Replay = OperationalAudit) vs Deployer everywhere — see F3. - Design §9 (redaction applies to trace payloads) vs F1. -- `ApiMethodForm` / CLI help say "~7 audit rows per call"; the outbound-calls addendum makes a - call with one cached DB write and one notification **10 rows** (3 trace-only). Say "7–10". +- `ApiMethodForm` / CLI help understate the row count (they were written before the outbound-calls + addendum, which makes a call with one cached DB write and one notification **10 rows**, 3 of them + trace-only). Say "7–10 audit rows per call". - `FlowDebugRunner.RunMethodAsync` writes the inbound row *after* the run, so a Replay taken during a long Live run shows only the `RouteCall`. Fine, but not stated anywhere. - `SchemaRefResolver` became `public` for the runner; note it in Component-InboundAPI or move the @@ -328,3 +329,30 @@ touches the Flow Debugger; `Deployment/FlowDebuggerTests` (Live run + localStora Consequence worth recording alongside the login-disabled decision: the rig can no longer prove role gating in the browser, so the Playwright suite's role-negative tests need either a second, login-enabled central to target or a fixture that overrides the identity per test. + +## 7. Resolution + +All thirteen findings were remediated on branch `fix/flow-debugger-arch-review` (plan +`docs/plans/2026-09-06-flow-debugger-arch-review-remediation.md`, 2026-09-06). Nothing was +deferred to the register. + +| # | Outcome | Commit | +|---|---------|--------| +| F1 | `FlowTraceCollector` moved to Commons `Types/Debugging/`, applies `IAuditRedactor` inside `Add` and caps at 500 events **and** 1 MB of summed `DetailsJson`; the site launcher resolves the redactor from the run's scope, and the central `FlowDebugRunner` collects through the same instance and reports its `Truncated`. | `e79452d2` fix(site-runtime): Flow Debugger inline trace copy is redacted and byte-capped at the collector (arch-review F1, site half) · `def6bab3` fix(management): Flow Debugger Live result is the redacted, capped trace copy (arch-review F1, central half) | +| F2 | `RouteHelper.WithSiteScope(IReadOnlySet?)` refuses an out-of-scope target site before any site traffic (`RouteSiteScopeException` → executor code `SITE_SCOPE_DENIED`); the runner rethrows it as `SiteScopeViolationException` (`ManagementUnauthorized`) **after** writing the inbound audit row. Administrators and unscoped users are exempt; the HTTP endpoint is unchanged. | `20a05941` fix(inbound-api): Flow Debugger method-entry Live run enforces the operator's site scope at route time (arch-review F2) | +| F3 | New `AuthorizationPolicies.RequireFlowDebugger` (`FlowDebuggerAccessRoles` = Deployer, Administrator, Viewer) gates the page, and `FlowDebugReplayCommand` accepts the same set; the two run commands, the Live controls, Retry/Discard and the `?methodId=`/`?instanceId=` drill-ins stay Deployer-only, with the page pinning Replay mode for a non-Deployer. | `f86eb61d` fix(central-ui): Flow Debugger Replay opens to OperationalAudit roles; Live run and Retry/Discard stay Deployer-only (arch-review F3) | +| F4 | Client waits derive from the server's bound: `RunAskTimeoutFor(timeoutSeconds)` = method timeout + `RunGrace` + 15 s slack (floor 30 s), `ValueChangeAskTimeoutFor(wait)` = wait + 15 s; the CLI gains `run-method --timeout-seconds` (default 120) and waits `--wait-seconds + 25` on `run-value-change`. | `97b0a4ba` fix(central-ui,cli): Flow Debugger client waits derive from the method timeout / wait window instead of a fixed 2 minutes (arch-review F4) | +| F5 | The graph orders executions by emission sequence and uses timestamps only **within** one execution, so a site clock offset can no longer reorder the swimlane; the Live runner keeps the collector's emission order. | `1dfb9cee` fix(commons): Flow graph orders executions by emission sequence, timestamps only within one clock (arch-review F5) | +| F6 | A failed traced DCL write disarms the trace arm (`DisarmTrace` self-message), so an unrelated device update can no longer consume it. | `50bf9ac7` fix(site-runtime): trace arm is cleared when the traced DCL write fails (arch-review F6) | +| F7 | A traced run shed by `MaxConcurrentRunsPerScript` or suppressed by `MinTimeBetweenRuns` emits a Failed `ScriptRun` row (`shed: MaxConcurrentRunsPerScript …` / `suppressed: MinTimeBetweenRuns …`) instead of vanishing from the graph. | `105541b6` fix(site-runtime): shed or rate-limited traced runs emit a Failed ScriptRun row (arch-review F7) | +| F8 | All four trace/outbound fault continuations run on `TaskScheduler.Default`, and the central emitters log a Warning on a failed audit write. | `262de38c` fix(audit): trace/outbound fault continuations run on the default scheduler and central emitters log (arch-review F8) | +| F9 | `GetTraceEventsAsync` and `GetExecutionTreeAsync` share one chain walker in `AuditLogRepository`. | `297ae56b` refactor(config-db): GetTraceEventsAsync and GetExecutionTreeAsync share one chain walker (arch-review F9) | +| F10 | The prediction overlay matches a real step by `(Kind, Title)` first, falling back to kind only, so two predicted nodes of the same kind no longer collapse onto one another. | `688136de` fix(central-ui): Flow Debugger prediction overlay matches by kind and title (arch-review F10) | +| F11 | The always-on `RouteCall` row honours the method's `SkipBodyCapture` — the executor builds the helper with `RouteHelper.WithBodyCapture(false)`; the row is still written, metadata only. | `453c3dec` fix(inbound-api): RouteCall audit row honours the method's SkipBodyCapture override (arch-review F11) | +| F12 | `AttributeValueUpdate.source_execution_id` / `trace` (fields 7/8) are **kept, populated and unconsumed** — documented as reserved for a future debug-stream path in `sitestream.proto` and design §5.4, with the field numbers pinned against reuse. Not removed: dropping them would be a wire change for no gain. | this commit | +| F13 | Design D6 / §5.4 / §9 / §10, `Component-CentralUI.md`, `Component-InboundAPI.md`, `CLAUDE.md`, the proto comment and the CLI/UI help strings reconciled with the shipped code; the trace-row estimate is "7–10 audit rows per call" everywhere; `SchemaRefResolver`'s `public` visibility and the inbound-row-after-the-run ordering are now stated. | this commit — docs: Flow Debugger design reconciled with the implementation (arch-review F12, F13) | + +One test defect surfaced by §3's performance commits was fixed on the same branch: +`OperationTrackingStoreTests.ReadChangedSince_BoundaryTimestamp…` raced the new single-writer +queue (a missing `FlushAsync` before the test's direct `UPDATE`) — `86451382` test(site-runtime): +flush the tracking-store queue before the boundary-timestamp test's direct UPDATE. diff --git a/docs/plans/sql/AddApiMethodTraceAllCalls.sql b/docs/plans/sql/AddApiMethodTraceAllCalls.sql index 7dbce4a6..e837e767 100644 --- a/docs/plans/sql/AddApiMethodTraceAllCalls.sql +++ b/docs/plans/sql/AddApiMethodTraceAllCalls.sql @@ -6,7 +6,7 @@ SET ANSI_NULLS ON; GO -- Flow Debugger (design 2026-09-05 §5.2): ApiMethod.TraceAllCalls — when 1, every --- production call to the method is fully traced (~7 audit rows per call). Production +-- production call to the method is fully traced (7-10 audit rows per call). Production -- twin of EF migration 20260906034113_AddApiMethodTraceAllCalls. BEGIN TRANSACTION; IF NOT EXISTS ( diff --git a/docs/requirements/Component-CentralUI.md b/docs/requirements/Component-CentralUI.md index bdfc2602..b0349b58 100644 --- a/docs/requirements/Component-CentralUI.md +++ b/docs/requirements/Component-CentralUI.md @@ -215,13 +215,14 @@ Per-leaf alarm rendering (leaf nodes are individual conditions for native alarms - **Row tooltip** — surfaces native metadata not warranting its own column: `AlarmTypeName`, category, operator user and comment, original raise time, current/limit value. - **Computed alarms render unchanged** from the prior flat-table style; the enrichment is purely additive for native rows. -### Flow Debugger (Deployment Role) +### Flow Debugger (Flow Debugger policy; Live run = Deployment Role) - **`/deployment/flow-debugger`** (nav: Deployment → Flow Debugger; design `docs/plans/2026-09-05-flow-debugger-design.md`, plan `docs/plans/2026-09-05-flow-debugger.md`). Runs one entry point with operator-supplied inputs and renders every step the run passes through as a **four-lane swimlane** (Central / Site / Tags / **External**) — request, route call, site script run, attribute writes, attribute waits, **outbound API / database / notification calls**, response — each node carrying its input, output, duration and outcome; a tabbed detail panel (Input · Output · Writes *or* Lifecycle · Timing, plus a *Where this ran* column) opens on the selected node with `Open in Debug View` (value-change entries, where the instance is known) and `Audit rows (n)` (→ `/audit/log?executionId=`) drill-ins. - **Two modes.** *Live* is a real run — the method script executes on the active central node with tracing on, or a real attribute write goes through the routed-write path and the site's triggered run is polled back — audited like any other call. *Replay* executes nothing: it rebuilds the flow from an existing execution's `ScriptTrace` audit rows plus its `ApiOutbound` / `DbOutbound` / `Notification` rows (`?executionId=` drill-in from the Audit Log drawer's and the Execution Chain page's **Open in Flow Debugger** actions). - **External lane (v1.1, plan `docs/plans/2026-09-06-flow-debugger-outbound-calls.md`).** Every `ExternalSystem.Call/CachedCall`, `Database.*` and `Notify.Send` a traced run makes is a node on the fourth lane — kinds `ApiCall` / `DbCall` / `Notify` (eyebrows `API CALL` / `DATABASE` / `NOTIFY`), title = the system.method / connection / list, a status pill (`Ok` / `Pending` / `Parked` / `Failed`) plus `sync` | `cached` | `notify`. **One node per operation:** a cached call's or notification's later lifecycle rows (Forwarded → Attempted → Delivered / Parked, written minutes later by the S&F forwarder or the Notification Outbox) fold into the node (`FlowStep.Lifecycle`, a Lifecycle tab in the detail panel with *Delivered by*) instead of drawing four boxes. A sync call sits in the execution chain; a cached call or notification hangs off it on a dashed **async** connector, and a trigger cascade's spawn edge carries a bolt marker. A Live run returns only the rows the run itself emitted, so a graph left with `Pending` nodes **refreshes itself** through the Replay command every 5 s for up to 2 min (page constants), and a *Refresh outcomes* button re-reads on demand in either mode; the state chip stays *Completed* — pending outbound work is not a failed run. A refresh **merges** the re-read rows into the run's known rows by `EventId` (fresh copy wins) and rebuilds the graph from the union, so it only ever adds nodes: the site's outbound rows reach central's audit store by telemetry seconds after the run returned them inline, and a Replay read taken in between would otherwise erase an already-drawn node (observed on the rig 2026-09-06 with a resolved cached DB write). The union is exact because the inline rows are the very `AuditEvent` objects the site writer persists. **v1.2 (follow-ups plan `docs/plans/2026-09-06-flow-debugger-followups.md`, register row 111 closed):** a folded node carries its `CorrelationId` (`FlowStep.CorrelationId` — the `TrackedOperationId` / `NotificationId`), so the detail panel deep-links to **`/site-calls/report?correlationId=`** / **`/notifications/report?correlationId=`** (*Open in Site Calls* / *Open in Notification report*; both pages accept that query, pre-filter to the one row through an additive per-id query filter — `SiteCallQueryRequest.TrackedOperationIdFilter` / `NotificationOutboxQueryRequest.NotificationIdFilter` — show a removable *Operation* / *Notification* pill and auto-open the row's detail modal; note `NotificationId` is minted as a Guid in **`N` form** — no dashes — so the Notify link passes `N` and the report page normalises any Guid-shaped value to `N`, while `TrackedOperationId` is a `varchar(36)` `D` string and the Site Calls filter binds a `varchar` parameter for the PK seek), and a **`Parked` node offers Retry / Discard in place**: the same site-scope re-check, confirm dialog (Discard = danger) and central→site / Notification Outbox relay the report pages use (`IFlowDebugService.RetryOutboundAsync` / `DiscardOutboundAsync` → `CommunicationService`), then the page re-polls Replay (5 s / 2 min) until the node leaves `Parked`. **Prediction is async-safe:** `FlowDebugService.PredictMethodAsync` resolves route targets by a resolve-and-rewalk loop — its original blocking resolver deadlocked the whole Blazor circuit on the renderer sync context the moment a method routing to a site script was chosen (found 2026-09-06; pinned by a single-thread-pump unit test and the Playwright `FlowDebuggerTests`, which log in through the fixture and run a Live run of an ephemeral method in a real browser). -- **Two entry points.** An inbound API method (inputs form generated from the method's `ParameterDefinitions`, values coerced by declared type) or a value-change trigger on a named instance attribute (Site → Instance → Attribute pickers, site-scope filtered). +- **Authorization (arch-review F3).** The page is `RequireFlowDebugger` — `AuthorizationPolicies.FlowDebuggerAccessRoles` = Deployer, Administrator, Viewer — so an auditor can open it and Replay a past execution without the ability to run anything, and `FlowDebugReplayCommand` accepts the same set. **Everything that executes stays Deployer-only:** the two run commands (`FlowDebugRunMethodCommand` / `FlowDebugRunValueChangeCommand`), the Live entry-point controls, the parked-node Retry/Discard actions and the `?methodId=` / `?instanceId=` drill-ins; for a non-Deployer the page pins itself to Replay mode. +- **Two entry points.** An inbound API method (inputs form generated from the method's `ParameterDefinitions`, values coerced by declared type) or a value-change trigger on a named instance attribute (Site → Instance → Attribute pickers, site-scope filtered). A method-entry Live run is additionally site-scoped **at route time** on the server (`RouteHelper.WithSiteScope` → `SITE_SCOPE_DENIED`, arch-review F2), so an operator cannot reach a site outside their scope by choosing a method that routes there. - **Predicted graph first.** Choosing an entry point renders the statically predicted shape (`FlowShapePredictor`, Script Analysis REQ-SA-7) before anything runs — nodes whose target depends on inputs render dashed; after a run, predicted nodes the run never produced render greyed. -- **Execution path.** `IFlowDebugService` dispatches `FlowDebugRunMethodCommand` / `FlowDebugRunValueChangeCommand` / `FlowDebugReplayCommand` to the **per-node `ManagementActor`** through `ManagementActorHolder` (the same seam the HTTP `/management` endpoint uses), so the server enforces the Deployer gate and the active-node refusal. A standby node answers the two run commands with `STANDBY_NODE`, which the page renders as an inline alert with the server message verbatim; Replay is served from any node. The same three commands back the CLI's `flow-debug run-method | run-value-change | replay`. +- **Execution path.** `IFlowDebugService` dispatches `FlowDebugRunMethodCommand` / `FlowDebugRunValueChangeCommand` / `FlowDebugReplayCommand` to the **per-node `ManagementActor`** through `ManagementActorHolder` (the same seam the HTTP `/management` endpoint uses), so the server enforces the Deployer gate and the active-node refusal. A standby node answers the two run commands with `STANDBY_NODE`, which the page renders as an inline alert with the server message verbatim; Replay is served from any node. The same three commands back the CLI's `flow-debug run-method | run-value-change | replay`. **The client wait is derived from the server's own bound, not a fixed duration (arch-review F4):** `FlowDebugService.RunAskTimeoutFor(timeoutSeconds)` = the method's `TimeoutSeconds` + the server's `FlowDebugRunner.RunGrace` (10 s) + 15 s client slack, floored at 30 s, and `ValueChangeAskTimeoutFor(waitSeconds)` = the trigger-wait window + 15 s (Replay stays a flat 30 s); the CLI mirrors it with `run-method --timeout-seconds` (default 120, set it to the method's timeout + 25 for long methods) and a `run-value-change` wait of `--wait-seconds + 25`. A long-running method used to be cut off client-side before the server's own grace expired. - State machine `Idle → Predicted → Running → Completed | Failed | TimedOut`; the status strip shows run number, start, total duration, step/boundary/write counts and the root `ExecutionId` (linking to the Execution Chain page). A run whose route call came back with no child execution and no Site-lane steps shows a *site not traced* note (older site build, or an untraced call). Last entry point and per-method inputs persist in `localStorage`. - The renderer is the custom-SVG `FlowGraphView` (no third-party charting) — pure, unit-tested layout; culture-invariant coordinates; each node a keyboard-activatable `role="button"`. Node text is HTML inside an SVG `foreignObject` so the browser measures it: the title wraps to two lines and ellipsizes by width (never by character count), the subtitle by width on one line, and the hover tooltip carries the full text. A lane whose real titles share a dotted head (Tags: `DelmiaReceiver.RecipeDownloadFlag`, `DelmiaReceiver.RecipeProcessedFlag`, …) shows that head once in the lane gutter (`DelmiaReceiver.*`) and each node shows only the tail that tells it apart (`FlowGraphView.CommonDottedPrefix`). diff --git a/docs/requirements/Component-InboundAPI.md b/docs/requirements/Component-InboundAPI.md index 13c9c235..dcc67a7a 100644 --- a/docs/requirements/Component-InboundAPI.md +++ b/docs/requirements/Component-InboundAPI.md @@ -70,7 +70,7 @@ Each API method definition includes: - Field names and (extended-system) data types. Supports returning **lists of objects** and arbitrarily nested structures. - **Implementation Script**: C# script that executes when the method is called. Stored **inline** in the method definition. Follows standard C# authoring patterns but has no template inheritance — it is a standalone script tied to this method. - **Timeout**: Configurable per method. Defines the maximum time the method is allowed to execute (including any routed calls to sites) before returning a timeout error to the caller. -- **TraceAllCalls** (Flow Debugger, 2026-09-05): when set, every call to this method runs its routed site scripts with the trace flag on, so their `ScriptRun` / `AttributeWrite` / `AttributeWait` audit rows (`AuditChannel.ScriptTrace`) are recorded for Replay. Default off — adds roughly seven audit rows per call, so it is meant for low-volume integration methods. Set from the method form's *Trace all calls* checkbox or `api-method create|update --trace-all-calls`. +- **TraceAllCalls** (Flow Debugger, 2026-09-05): when set, every call to this method runs its routed site scripts with the trace flag on, so their `ScriptRun` / `AttributeWrite` / `AttributeWait` audit rows (`AuditChannel.ScriptTrace`) are recorded for Replay. Default off — adds 7–10 audit rows per call (3 of them trace-only; a cached DB write and a notification account for the upper end), so it is meant for low-volume integration methods. Set from the method form's *Trace all calls* checkbox or `api-method create|update --trace-all-calls`. > **Authoring note — idempotent methods for at-least-once callers**: some external callers retry > on their own connect-failure/timeout failover loop and cannot tell "the first attempt actually @@ -299,11 +299,14 @@ For everything else, scripts interact with the system through the curated `Route - The `Route.To()` helper resolves the instance's site assignment from the configuration database and routes the request to the correct site cluster via the Communication Layer. - The call is **synchronous from the API caller's perspective** — the API method blocks until the site responds or the **method-level timeout** is reached. - If the target site is unreachable or the call times out, the call fails and the API returns an error to the caller. No store-and-forward buffering is used for inbound API calls. -- **Every `Route.To(...).Call(...)` writes one always-on `ScriptTrace.RouteCall` audit row** (central direct write, actor `inbound-api`, target `{instance}.{script}`, `ExecutionId` = the inbound request's, `Extra.childExecutionId` = the site run's id) — this row is what joins the central and site halves of a Flow Debugger Replay, so it is not gated. The routed request carries a `Trace` flag beside `ParentExecutionId` (`RouteToCallRequest.Trace`, `RouteToSetAttributesRequest.Trace`); it is set when the method has `TraceAllCalls` or the call is a Flow Debugger Live run, and only then does the site emit its own trace rows and return them inline on `RouteToCallResponse.TraceEvents` (with `ChildExecutionId`). Untraced calls are byte-identical to before. +- **Every `Route.To(...).Call(...)` writes one always-on `ScriptTrace.RouteCall` audit row** (central direct write, actor `inbound-api`, target `{instance}.{script}`, `ExecutionId` = the inbound request's, `Extra.childExecutionId` = the site run's id) — this row is what joins the central and site halves of a Flow Debugger Replay, so it is not gated. The routed request carries a `Trace` flag beside `ParentExecutionId` (`RouteToCallRequest.Trace`, `RouteToSetAttributesRequest.Trace`); it is set when the method has `TraceAllCalls` or the call is a Flow Debugger Live run, and only then does the site emit its own trace rows and return them inline on `RouteToCallResponse.TraceEvents` (with `ChildExecutionId`). Untraced calls are byte-identical to before. **The `RouteCall` row honours the method's `SkipBodyCapture`** — the executor builds the helper with `RouteHelper.WithBodyCapture(!skipBodyCapture)`, so a method that opts out of body capture no longer leaks its arguments and reply through the always-on trace row; the row itself is still written, with metadata only (arch-review F11). ## Flow Debugger runs -The Flow Debugger (Central UI, design `docs/plans/2026-09-05-flow-debugger-design.md`) drives this component through three management commands handled by the per-node `ManagementActor` (Deployer role): `FlowDebugRunMethodCommand` executes a method exactly as an external caller would — `ParameterValidator` + `SchemaRefResolver`, then `InboundScriptExecutor.ExecuteAsync(..., trace: true)` with a `RouteHelper` trace sink collecting the `RouteCall` row and the site's inline events — and **synthesizes the `ApiInbound.InboundRequest` row itself** with the operator as actor (`flow-debugger` source tag) because no HTTP request passed through `AuditWriteMiddleware`; it refuses on a standby node with `STANDBY_NODE` (`IActiveNodeGate`), so the script runs only where a real call would. `FlowDebugRunValueChangeCommand` writes an attribute through the traced routed-write path and polls the site (`PullTraceEvents`) for the run the change spawned; `FlowDebugReplayCommand` reads the chain back from the central Audit Log. The result (`FlowDebugRunResult`) carries the audit events inline — the graph is a central-side projection, never a wire type. +The Flow Debugger (Central UI, design `docs/plans/2026-09-05-flow-debugger-design.md`) drives this component through three management commands handled by the per-node `ManagementActor` (Deployer role): `FlowDebugRunMethodCommand` executes a method exactly as an external caller would — `ParameterValidator` + `SchemaRefResolver`, then `InboundScriptExecutor.ExecuteAsync(..., trace: true)` with a `RouteHelper` trace sink collecting the `RouteCall` row and the site's inline events — and **synthesizes the `ApiInbound.InboundRequest` row itself** with the operator as actor (`flow-debugger` source tag) because no HTTP request passed through `AuditWriteMiddleware`; it refuses on a standby node with `STANDBY_NODE` (`IActiveNodeGate`), so the script runs only where a real call would. `FlowDebugRunValueChangeCommand` writes an attribute through the traced routed-write path and polls the site (`PullTraceEvents`) for the run the change spawned; `FlowDebugReplayCommand` reads the chain back from the central Audit Log and therefore accepts the wider `AuthorizationPolicies.FlowDebuggerAccessRoles` set (Deployer, Administrator, Viewer) while the two run commands stay Deployer-only (arch-review F3). The result (`FlowDebugRunResult`) carries the audit events inline — the graph is a central-side projection, never a wire type. + +- **`SchemaRefResolver` is `public` for the runner's benefit.** It was internal to the HTTP path; the Flow Debugger's `FlowDebugRunMethodCommand` validates its operator-supplied parameters through exactly the same `ParameterValidator` + `SchemaRefResolver` pair a real HTTP call uses, so a Live run cannot accept an input the endpoint would reject. Keep the two paths on the one resolver — a second, debugger-local validator would silently fork the contract. +- **Site scope is enforced at route time on a Live run** (arch-review F2). `RouteHelper.WithSiteScope(IReadOnlySet? permittedSiteIdentifiers)` carries the operator's permitted **site identifiers**; `RouteTarget.ResolveSiteAsync` throws `RouteSiteScopeException` before any site traffic when the resolved target site is outside that set, and `InboundScriptExecutor` maps it to error code **`SITE_SCOPE_DENIED`**. `FlowDebugRunner` translates that code into a `SiteScopeViolationException` (`ManagementUnauthorized`) *after* writing the inbound audit row, so the denied attempt is recorded under the operator's name. Administrators and unscoped users get a null scope (unrestricted), and the HTTP endpoint passes no scope at all — production routing is unchanged. ## Authentication Details diff --git a/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/ApiMethodCommands.cs b/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/ApiMethodCommands.cs index fbb022e2..60302da0 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/ApiMethodCommands.cs +++ b/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/ApiMethodCommands.cs @@ -62,7 +62,7 @@ public static class ApiMethodCommands var returnDefOption = new Option("--return-def") { Description = "Return type definition" }; var traceAllCallsOption = new Option("--trace-all-calls") { - Description = "Trace every production call to this method for the Flow Debugger (adds ~7 audit rows per call; low-volume integration methods only)" + Description = "Trace every production call to this method for the Flow Debugger (adds 7-10 audit rows per call; low-volume integration methods only)" }; var cmd = new Command("create") { Description = "Create an API method" }; @@ -98,7 +98,7 @@ public static class ApiMethodCommands // bool? so an update that omits the flag leaves the stored value unchanged. var traceAllCallsOption = new Option("--trace-all-calls") { - Description = "Trace every production call to this method for the Flow Debugger (true/false; omitted = unchanged; adds ~7 audit rows per call)" + Description = "Trace every production call to this method for the Flow Debugger (true/false; omitted = unchanged; adds 7-10 audit rows per call)" }; var cmd = new Command("update") { Description = "Update an API method" }; diff --git a/src/ZB.MOM.WW.ScadaBridge.CLI/README.md b/src/ZB.MOM.WW.ScadaBridge.CLI/README.md index 7c680d27..d4bfde8a 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CLI/README.md +++ b/src/ZB.MOM.WW.ScadaBridge.CLI/README.md @@ -2119,7 +2119,7 @@ scadabridge --url api-method create --name --script [--t | `--timeout` | no | `30` | Timeout in seconds | | `--parameters` | no | — | Parameter definitions JSON | | `--return-def` | no | — | Return type definition JSON | -| `--trace-all-calls` | no | off | Trace every production call for the Flow Debugger. **Adds ~7 audit rows per call** — low-volume integration methods only | +| `--trace-all-calls` | no | off | Trace every production call for the Flow Debugger. **Adds 7–10 audit rows per call** (3 of them trace-only) — low-volume integration methods only | #### `api-method update` @@ -2137,7 +2137,7 @@ scadabridge --url api-method update --id --script [--timeou | `--timeout` | no | `30` | Timeout in seconds | | `--parameters` | no | — | Parameter definitions JSON | | `--return-def` | no | — | Return type definition JSON | -| `--trace-all-calls` | no | unchanged | `true`/`false` — trace every production call for the Flow Debugger. **Adds ~7 audit rows per call.** Omitted = the stored value is left as-is (the one field the replace-all rule does not apply to) | +| `--trace-all-calls` | no | unchanged | `true`/`false` — trace every production call for the Flow Debugger. **Adds 7–10 audit rows per call** (3 of them trace-only). Omitted = the stored value is left as-is (the one field the replace-all rule does not apply to) | Script changes take effect immediately — the updated code is recompiled in-memory on the active central node. No restart is required. diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Design/ApiMethodForm.razor b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Design/ApiMethodForm.razor index 35b1bda8..d1e3b893 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Design/ApiMethodForm.razor +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Design/ApiMethodForm.razor @@ -47,7 +47,7 @@
Every call to this method records its routed site steps as audit rows the - Flow Debugger can replay. Adds ~7 audit rows per call; for low-volume + Flow Debugger can replay. Adds 7–10 audit rows per call; for low-volume integration methods.
diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Entities/InboundApi/ApiMethod.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Entities/InboundApi/ApiMethod.cs index 7ebb5875..0bf313f3 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Entities/InboundApi/ApiMethod.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Entities/InboundApi/ApiMethod.cs @@ -17,7 +17,7 @@ public class ApiMethod /// /// When true every production call to this method is fully traced (Flow Debugger design - /// 2026-09-05 §5.2); intended for low-volume integration methods — ~7 audit rows per call. + /// 2026-09-05 §5.2); intended for low-volume integration methods — 7-10 audit rows per call. /// public bool TraceAllCalls { get; set; } diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto b/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto index e8829bf9..0b741012 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto @@ -100,8 +100,10 @@ message AttributeValueUpdate { Quality quality = 5; google.protobuf.Timestamp timestamp = 6; // Flow Debugger (design 2026-09-05 §5.3/§5.4): the writing execution's id ("D" GUID - // form; empty represents null) and the trace flag, so the debug stream can hand the - // debugger the spawned run's parent. Additive; an older site never sets them. + // form; empty represents null) and the trace flag. Additive; an older site never sets them. + // RESERVED — populated by the site, consumed by nothing (arch-review F12): the value-change + // entry polls the site's PullTraceEvents RPC instead of subscribing to the debug stream, so + // these two fields are kept for a future debug-stream path. Do not reuse the field numbers. string source_execution_id = 7; bool trace = 8; }