docs: Flow Debugger — component specs, CLAUDE.md, CLI README, register row 110
Component-CentralUI (new Flow Debugger page section), Component-InboundAPI (TraceAllCalls property, always-on RouteCall row + Trace flag under Routing Behavior, new 'Flow Debugger runs' section), Component-SiteRuntime (Instance Actor trace arm; Script Actor traced runs; CallScript flag inheritance), Component-AuditLog (ScriptTrace channel in Scope, the PullTraceEvents point-read note + mixed-version caveat under Reconciliation, per-channel retention example), Component-Communication (§4 trace flag + inline events with proto field numbers; §8 PullTraceEvents; RPC list), Component-ScriptAnalysis (REQ-SA-7 FlowShapePredictor), CLAUDE.md UI bullet, deferred-work register row 110 (vertical-trace list toggle + console capture), design doc status → implemented. CLI README sections landed in Tasks 3/16. Umbrella index (../scadaproj/CLAUDE.md) checked: its ScadaBridge row records stack/wire relationships only, unchanged by a new page — no edit. Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
This commit is contained in:
Joseph Doherty
@@ -196,6 +196,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- Central UI: Blazor Server (ASP.NET Core + SignalR) with Bootstrap CSS. No third-party component frameworks (no Blazorise, MudBlazor, Radzen, etc.). Build custom Blazor components for tables, grids, forms, etc.
|
||||
- UI design: Clean, corporate, internal-use aesthetic. Not flashy. Use the `frontend-design` skill when designing UI pages/components.
|
||||
- Debug view: real-time streaming via DebugStreamBridgeActor + gRPC (events via `SiteStreamGrpcClient`, snapshot via the `SiteCommandService` query surface — `QueryReply.DebugViewSnapshot`). Health dashboard: 10s polling timer. Deployment status: real-time push via SignalR.
|
||||
- **Flow Debugger (2026-09-05, Deployment → `/deployment/flow-debugger` + CLI `flow-debug run-method|run-value-change|replay`).** Runs one entry point (an inbound API method, or a value change on an instance attribute) and draws every step as a Central / Site / Tags swimlane; *Live* really runs it, *Replay* rebuilds a past execution. **Trace steps ARE audit events** — `AuditChannel.ScriptTrace` with kinds `RouteCall` (central, **always-on** per `Route.To().Call`, joins the two halves of a Replay) and `ScriptRun` / `AttributeWrite` / `AttributeWait` (site, **gated** by a `bool Trace` riding beside `ParentExecutionId` on `RouteToCallRequest` / `ScriptCallRequest` / `SetStaticAttributeCommand` / `AttributeValueChanged`; proto fields additive, `site_command.proto` imports `sitestream.proto`). A Live run gets the site rows **inline** on `RouteToCallResponse.TraceEvents` (+ `ChildExecutionId`) because the 5 s telemetry cadence is too slow; Replay reads them from `AuditLog` (`GetTraceEventsAsync`). A value-change on a data-sourced attribute is a REAL tag write; the Instance Actor's **trace arm** (`SiteRuntime:TraceArmWindowSeconds`, 30) stamps the device echo with the writer's execution id so the triggered run is parented and traced, and central polls the site's new `PullTraceEvents` RPC (point read, not reconciliation) for it. Execution path is three management commands on the **per-node `ManagementActor`** (`FlowDebug*`, Deployer role) — a standby refuses the two run commands with **`STANDBY_NODE`** (shown verbatim); `ApiMethod.TraceAllCalls` (migration `AddApiMethodTraceAllCalls`, CLI `--trace-all-calls`) traces every real call to a method for later Replay (~7 rows/call). Prediction before a run is a Roslyn syntax walk (`FlowShapePredictor`, REQ-SA-7, DB-free); the graph (`FlowStep`/`FlowGraphBuilder`, Commons) is a central projection, never a wire type; the renderer is custom SVG (`FlowGraphView`). Rig: `AuditLog:PerChannelRetentionDays:ScriptTrace=30` on both centrals; `verify-delmia-notifier.sh` step H proves live + replay. **Mixed-version caveat:** an older site ignores the flag → central `RouteCall` only, page says *site not traced*. Design `docs/plans/2026-09-05-flow-debugger-design.md`; plan `docs/plans/2026-09-05-flow-debugger.md`.
|
||||
- Health reports: 30s interval, 60s offline threshold, monotonic sequence numbers, raw error counts per interval.
|
||||
- Dead letter monitoring as a health metric.
|
||||
- Site Event Logging: 30-day retention, 1GB storage cap, daily purge, paginated queries with keyword search. **The cap measures `site_events` + its four indexes, NOT the consolidated LocalDb file** (register row 81, 2026-08-17): whole-file measurement made the trim unwinnable on a replicated node — each batch freed event pages while adding an equal count of oplog rows + tombstones to the same file, so it stopped 2.5x over a 64 MB cap and a node with a down peer could not enforce the cap at all. `dbstat` (the exact per-table measurement) is **NOT compiled into the bundled `SQLitePCLRaw.lib.e_sqlite3`** — probed and pinned by `SqliteCapabilityTests`, so don't reach for it — the footprint is modelled from sampled row width instead (−0.8%…+12.8%, biased high). Consequence: `MaxStorageMb` no longer bounds the data volume alone; LocalDb bookkeeping is separate overhead bounded by `MaxOplogRows`/`TombstoneRetention` and is deliberately not this component's job. Decision: `docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md`.
|
||||
|
||||
@@ -102,6 +102,7 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
|
||||
| 107 | **Prod `CvdReactor.ProcessRecipeDownload` still addresses `Children["DelmiaRecv"]` after the 2026-06-27 `DelmiaRecv` → `DelmiaReceiver` composition rename** — inferred from the deployed script (`docs/api-specs/Delmia-Recipe-Download-API.md` §7 item 1): `ChildrenAccessor` concatenates the name into a path, an unknown path yields null, so every call answers `Receiver ready flag not set to true` regardless of the real flag. No call has hit prod since the rename, so this is inferred, not observed. The docker rig's stand-in (`docker/seed-delmia-sim.sh`) uses `Children["DelmiaReceiver"]` and the same handshake answers correctly, which is the working reference. | Found 2026-09-02 while capturing the as-deployed spec; QA copy unread (credentials) | Separate change with its own deployment record: `scadabridge template script update` on the `CvdReactor` template + redeploy Z28061/Z28062/Z28061Sim, then a live `WWNotifier` call. | Before the first production Delmia recipe download. |
|
||||
| 108 | ~~**Inbound API keys are a PER-NODE SQLite store — a central failover invalidates every inbound client's key**~~ **RESOLVED 2026-09-02 — `ZB.MOM.WW.Auth` 0.3.0 (new `ZB.MOM.WW.Auth.ApiKeys.SqlServer` package) + ScadaBridge `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer`.** A central pair now shares one row set: both nodes' key stores, admin stores, and audit stores point at the same `zbauth` schema in the configuration database (connection string defaults to `ScadaBridge:Database:ConfigurationDb` — never a `${secret:}` reference, to avoid a circular dependency on the SQL-backed secrets store), so create/revoke/rotate/scope changes are cluster-wide immediately instead of per-node. Each node's existing SQLite file is imported once at boot (insert-where-absent, so both nodes importing is safe and the result is the union) and renamed `*.imported`. `Provider` defaults to `Sqlite` (unchanged behavior for single-node installs, MxAccessGateway, HistorianGateway); a clustered Central left on `Sqlite` now gets a boot-time `StartupValidator` Warning naming this design instead of silently keeping the gap. Both docker rigs (`docker/`, `docker-env2/`) run `Provider=SqlServer` on both centrals; `docker/sync-inbound-api-keys.sh` is deleted, superseded by the shared store. Rig proof 2026-09-03 (`docker/`, both centrals redeployed on `Provider=SqlServer`): both nodes imported their SQLite file at the same second with no PK race — central-a `1 key / 264 audit rows inserted`, central-b `0 / 26 inserted, 1 / 264 already present` (the union), files renamed `*.imported`; a key minted via `:9001` answered `200` via `:9002`, `--enabled false` via `:9001` made both nodes answer `401` and `:9002`'s own list showed `enabled=False`, re-enable via `:9002` restored `200` on `:9001`, delete via `:9002` made `:9001` answer `401`; `verify-delmia-notifier.sh` with a freshly minted key and NO file copy: A/B/E/F `YES`, C/G `NO`, active-central `docker kill` takeover **28 s**, rejoin as standby in 2 s. `docker-env2` (second pair, `ScadaBridgeConfig2`): both centrals booted, `zbauth` v1 created, empty files imported. QA (`vq03`/`vq04`) and prod (`vd03`) still run `Provider=Sqlite` — their move is a planned deployment record (`docs/deployment/deployment-records/2026-09-XX-zb-servers-inbound-key-store.md`), not yet applied. Design: `docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`. Original: | `docs/api-specs/Delmia-Recipe-Download-API.md` §8; `docker/sync-inbound-api-keys.sh` header | Fix is an architecture decision, not a patch: move the store to the shared SQL Server (as Secrets 0.5.0 did for its central store) or replicate the file; either way revocation must become cluster-wide. Needs the `ZB.MOM.WW.Auth.ApiKeys` library (scadaproj), not just this repo. | Before any 2-node central goes live with inbound API clients; before the next arch-review wave. |
|
||||
| 109 | **`ZB.MOM.WW.Secrets.Replicator.SqlServer`'s store migrator retries its DDL with a flat `100 * attempt` backoff and no jitter — a latent concurrent-boot flake, open in Secrets** *(Low)*. Porting that migrator's retry loop verbatim for the Auth 0.3.0 SQL Server API-key store (register row 108) reproduced an intermittent exhaustion — about once in sixty 8-way concurrent-boot trials the five flat-backoff retries ran out before a DDL/lock conflict cleared; adding jitter to the Auth port's backoff took it to 0 failures in 180 trials. Secrets' own migrator has no concurrent-migrate test, so the same defect is presumed still open there — it just hasn't been exercised at 8-way concurrency, and a ScadaBridge central pair is only ever 2-way, so this is unlikely to bite in practice. | Found 2026-09-02 while implementing `docs/plans/2026-09-02-inbound-api-key-shared-store.md` (Auth 0.3.0 `SqlServerApiKeyStoreMigrator`) | Add jitter to `SqlServerSecretsStoreMigrator`'s backoff in `~/Desktop/scadaproj/ZB.MOM.WW.Secrets` (mirror the Auth fix) and add a `Migrate_Concurrently_FromEightTasks_AllSucceed`-shaped live test to catch a regression. Lives entirely in scadaproj, not this repo. | Next time Secrets' SQL Server migrator is touched, or if a wider-than-2-way concurrent boot of a Secrets-backed cluster is ever planned. |
|
||||
| 110 | **Flow Debugger follow-ups: the *Vertical trace* list toggle (mockup Direction B) and console capture on `ScriptRun`** *(Low)*. v1 ships the Direction A swimlane only; the mockup's Direction B — the same steps as a vertical, filterable list with the lane as a column — was kept as an alternate for long flows and screen readers, and the detail panel's *Console* tab is absent because script `Console`/log output is not captured per run (`ScriptRun` rows carry parameters, return value, error and timing, not stdout). Both are additive: a list view over the existing `FlowGraph`, and an extra `Extra.console` field on the `ScriptRun` row's `ScriptTraceExtra` (payload cap applies). | Design `docs/plans/2026-09-05-flow-debugger-design.md` §11; plan `2026-09-05-flow-debugger.md` Task 23 | Neither was asked for by an operator; the swimlane + Timing/Writes tabs cover the acceptance flow (Delmia recipe download, rig step H), and console capture needs a per-run writer seam in `ScriptRuntimeContext` plus a redaction decision. | First operator request for a list view or for script log lines in the step detail. |
|
||||
| 68 | ~~**`deploy.sh` run from a git worktree silently rehomes every node's bind-mount data dir**~~ **RESOLVED 2026-08-15 (guard added in the same close-out).** The compose files' bind mounts are relative (`./site-a-node-a/data`), so a deploy from a worktree checkout mounts the worktree's copies — the rig keeps running and looks healthy, but its SQLite stores split from the main checkout's and everything written there is orphaned by the next deploy from main. Bit the rows-46–53 close-out live: the seeding agent's `rig-probe-1` deploy persisted into worktree-homed dirs and "vanished" after the orchestrator's redeploy (no tombstone, no product bug — LWW/tombstone forensics ruled out deletion; the row sat in `.claude/worktrees/…/docker/site-a-node-a/data/site-localdb.db` the whole time). Both `docker/deploy.sh` and `docker-env2/deploy.sh` now refuse to run from a `.claude/worktrees/` path (override: `SCADABRIDGE_ALLOW_WORKTREE_DEPLOY=1`). | Found during the rows-46–53 close-out | Closed 2026-08-15. | — |
|
||||
|
||||
## Resolved (verified against the code 2026-07-10)
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Flow Debugger — design
|
||||
|
||||
**Status:** approved 2026-09-05; implementation plan follows.
|
||||
**Status:** implemented 2026-09-06 (branch `feat/flow-debugger`); plan `2026-09-05-flow-debugger.md` (24 tasks, all complete). Residual follow-ups: deferred-work register row 110.
|
||||
**Mockup:** Claude Design canvas *Flow Debugger Mockup* — <https://claude.ai/code/artifact/3f3e81e5-8e5e-4333-bdf6-59256108494d> (Direction A swimlane boards + two low-fi alternates + decision notes).
|
||||
**Owner components:** Central UI (#page), Inbound API, Site Runtime, Audit Log, Communication, Script Analysis, Configuration Database.
|
||||
|
||||
|
||||
@@ -55,6 +55,10 @@ boundary:
|
||||
| `Database.CachedWrite(...)` | Script | Outbound | Yes — `SiteCalls` (Site Call Audit) |
|
||||
| `Notify.To(list).Send(...)` | Script | Outbound | Yes — `Notifications` (Notification Outbox) |
|
||||
| `POST /api/{method}` (Inbound API) | External | Inbound (invokes a script) | No (gap) |
|
||||
| `Route.To(x).Call(y)` (Inbound API → site) | Script | Central → site | Yes — `ScriptTrace.RouteCall`, **always-on** (2026-09-05) |
|
||||
| Site script run / attribute write / attribute wait under a **traced** run | Script | Site-local | Yes — `ScriptTrace.ScriptRun` / `AttributeWrite` / `AttributeWait`, **gated** by the `Trace` flag (Flow Debugger Live runs, `ApiMethod.TraceAllCalls`) |
|
||||
|
||||
**`AuditChannel.ScriptTrace` (Flow Debugger, design `docs/plans/2026-09-05-flow-debugger-design.md` §5.1).** Trace steps ARE audit events — no parallel store. One kind is always-on (`RouteCall`, a central direct write per `Route.To().Call`, carrying the child run's id in `Extra.childExecutionId`); the three site kinds are emitted only when the run's `Trace` flag is set, ride the ordinary site `IAuditWriter` → SQLite → telemetry → central path, share the run's `ExecutionId` / `ParentExecutionId`, and are additionally returned inline on the routed reply so a Live run does not wait on the 5 s telemetry cadence. The Flow Debugger's graph (`FlowStep` / `FlowGraphBuilder`, Commons) is a central-side projection of these rows.
|
||||
|
||||
Out of scope — framework traffic is not audited:
|
||||
|
||||
@@ -348,6 +352,17 @@ non-draining (e.g., telemetry actor wedged), central issues a
|
||||
`PullAuditEvents(sinceUtc, batchSize[, afterId])` and inserts-if-not-exists.
|
||||
Same self-healing pattern as Site Call Audit's reconciliation of `SiteCalls`.
|
||||
|
||||
**`PullTraceEvents` is a point read, not a reconciliation path (2026-09-05).** The
|
||||
Flow Debugger's value-change run cannot get its steps inline (the device echo
|
||||
triggers the run after the write reply has returned), so central polls the site
|
||||
with `PullTraceEvents(execution_ids[], since_utc, max_events)` — `ISiteAuditQueue.
|
||||
ReadByExecutionIdsAsync`, capped at 64 ids / 2000 events, served whatever the row's
|
||||
`ForwardState` — until the spawned `ScriptRun` settles. It moves no cursor and
|
||||
retires nothing; the telemetry push and the reconciliation pull are unchanged.
|
||||
**Mixed-version caveat:** a pre-2026-09-05 site ignores the `Trace` flag and has no
|
||||
`PullTraceEvents`, so a run against it shows the central `RouteCall` row with no
|
||||
child id and an empty Site lane — the page labels this *site not traced*.
|
||||
|
||||
**The pull is at-least-once (WP2.3).** Serving a row is not proof that central
|
||||
received it, so rows are **not** flipped when they are served. The site flips
|
||||
`ForwardState = 'Reconciled'` for everything at or before the cursor carried by
|
||||
@@ -612,8 +627,10 @@ MS SQL for direct-write events). Unredacted secrets never persist.
|
||||
index did not exist at all. See migration `AlignAuditLogEventIdUniqueness`.
|
||||
- **Per-channel retention overrides (M5.5 T3):** `AuditLog:PerChannelRetentionDays`
|
||||
is a dictionary keyed by canonical channel name (`ApiOutbound`, `DbOutbound`,
|
||||
`Notification`, `ApiInbound`, `SecuredWrite` — all five `AuditChannel` values are
|
||||
accepted by the validator) whose value is a retention window in days that
|
||||
`Notification`, `ApiInbound`, `SecuredWrite`, `Cluster`, `ScriptTrace` — every
|
||||
`AuditChannel` value is accepted by the validator; the docker rig's central nodes
|
||||
set `"ScriptTrace": 30`, the validator floor, because traced runs are a debugging
|
||||
aid, not a compliance record) whose value is a retention window in days that
|
||||
MUST be strictly shorter than the global `RetentionDays`. After the daily
|
||||
partition switch-out, the purge actor runs a bounded, batched row DELETE
|
||||
(`PurgeChannelOlderThanAsync`) for each channel whose override is shorter than
|
||||
|
||||
@@ -215,6 +215,15 @@ Per-leaf alarm rendering (leaf nodes are individual conditions for native alarms
|
||||
- **Row tooltip** — surfaces native metadata not warranting its own column: `AlarmTypeName`, category, operator user and comment, original raise time, current/limit value.
|
||||
- **Computed alarms render unchanged** from the prior flat-table style; the enrichment is purely additive for native rows.
|
||||
|
||||
### Flow Debugger (Deployment Role)
|
||||
- **`/deployment/flow-debugger`** (nav: Deployment → Flow Debugger; design `docs/plans/2026-09-05-flow-debugger-design.md`, plan `docs/plans/2026-09-05-flow-debugger.md`). Runs one entry point with operator-supplied inputs and renders every step the run passes through as a **three-lane swimlane** (Central / Site / Tags) — request, route call, site script run, attribute writes, attribute waits, response — each node carrying its input, output, duration and outcome; a tabbed detail panel (Input · Output · Writes · Timing, plus a *Where this ran* column) opens on the selected node with `Open in Debug View` (value-change entries, where the instance is known) and `Audit rows (n)` (→ `/audit/log?executionId=`) drill-ins.
|
||||
- **Two modes.** *Live* is a real run — the method script executes on the active central node with tracing on, or a real attribute write goes through the routed-write path and the site's triggered run is polled back — audited like any other call. *Replay* executes nothing: it rebuilds the flow from an existing execution's `ScriptTrace` audit rows (`?executionId=` drill-in from the Audit Log drawer's and the Execution Chain page's **Open in Flow Debugger** actions).
|
||||
- **Two entry points.** An inbound API method (inputs form generated from the method's `ParameterDefinitions`, values coerced by declared type) or a value-change trigger on a named instance attribute (Site → Instance → Attribute pickers, site-scope filtered).
|
||||
- **Predicted graph first.** Choosing an entry point renders the statically predicted shape (`FlowShapePredictor`, Script Analysis REQ-SA-7) before anything runs — nodes whose target depends on inputs render dashed; after a run, predicted nodes the run never produced render greyed.
|
||||
- **Execution path.** `IFlowDebugService` dispatches `FlowDebugRunMethodCommand` / `FlowDebugRunValueChangeCommand` / `FlowDebugReplayCommand` to the **per-node `ManagementActor`** through `ManagementActorHolder` (the same seam the HTTP `/management` endpoint uses), so the server enforces the Deployer gate and the active-node refusal. A standby node answers the two run commands with `STANDBY_NODE`, which the page renders as an inline alert with the server message verbatim; Replay is served from any node. The same three commands back the CLI's `flow-debug run-method | run-value-change | replay`.
|
||||
- State machine `Idle → Predicted → Running → Completed | Failed | TimedOut`; the status strip shows run number, start, total duration, step/boundary/write counts and the root `ExecutionId` (linking to the Execution Chain page). A run whose route call came back with no child execution and no Site-lane steps shows a *site not traced* note (older site build, or an untraced call). Last entry point and per-method inputs persist in `localStorage`.
|
||||
- The renderer is the custom-SVG `FlowGraphView` (no third-party charting) — pure, unit-tested layout; culture-invariant coordinates; each node a keyboard-activatable `role="button"`.
|
||||
|
||||
### Alarm Summary (Deployment Role)
|
||||
- A dedicated operator **Alarm Summary** page (`/monitoring/alarms`, `RequireDeployment`) gives a **cross-instance, read-only** roll-up of live alarm state at a site — the operator-facing complement to the per-instance Debug View.
|
||||
- **Data path** — no new site-side code and no central alarm store. The page selects a site, queries its deployed instances, then fans out the existing per-instance `DebugViewSnapshot` Ask **concurrently** (capped with a `SemaphoreSlim`) and aggregates the returned `AlarmStates` client-side. The fan-out is **partial-results tolerant**: instances that time out are listed as "not reporting" while the rest still render. This snapshot fan-out now doubles as the **seed** for a near-real-time live feed (see **Live updates** below) rather than the sole refresh mechanism.
|
||||
|
||||
@@ -47,6 +47,7 @@ Both central and site clusters. Each side has communication actors that handle m
|
||||
- Site reads values from the Instance Actor and responds.
|
||||
- Central returns the response to the external system.
|
||||
- **Implementation**: this brokered round-trip is served by the **Inbound API's routed-site-script path** — a central-side inbound API method script composes the typed `RouteTo*` verbs on `CommunicationService` (`RouteToCall`, `RouteToGetAttributes`, `RouteToSetAttributes`, `RouteToWaitForAttribute`), driven from `InboundAPI/CommunicationServiceInstanceRouter` and sharing `IntegrationTimeout`. An earlier generic `IntegrationCallRequest` primitive was scaffolded for this pattern but never wired to a producer or a site handler; it was removed as dead code (Gitea #32, 2026-07-23).
|
||||
- **Trace flag + inline trace events (Flow Debugger, 2026-09-05; additive):** `RouteToCallRequestDto.trace` (field 7) and `RouteToSetAttributesRequestDto.trace` (field 6) carry the `Trace` flag beside the existing `parent_execution_id`; `RouteToCallResponseDto.child_execution_id` (field 6) and `repeated sitestream.AuditEventDto trace_events` (field 7) return the site run's id and its `ScriptTrace` rows in the same reply. `site_command.proto` now imports `sitestream.proto` for the shared `AuditEventDto`. Proto3 defaults are the negotiation: an old site ignores the flag and returns no events; an old central never sets it. `SiteCommandDtoMapper` maps an empty `trace_events` to `null`.
|
||||
|
||||
### 5. Recipe/Command Delivery (External System → Central → Site)
|
||||
- **Pattern**: Fire-and-forget with acknowledgment.
|
||||
@@ -113,6 +114,7 @@ The streaming protocol is defined in `sitestream.proto` (`src/ZB.MOM.WW.ScadaBri
|
||||
- `IngestCachedTelemetry(CachedTelemetryBatch) returns (IngestAck)` — ingest receiving surface for the combined cached-call telemetry packet (audit row + `SiteCalls` operational upsert written in one transaction).
|
||||
- `PullAuditEvents(PullAuditEventsRequest) returns (PullAuditEventsResponse)` — central→site **reconciliation pull** for the Audit Log self-heal feed; the site serves `Pending`/`Forwarded` rows from its `ISiteAuditQueue`. **At-least-once (WP2.3):** rows are NOT retired when served. The site retires (`Reconciled`) everything at or before the cursor of the **next** pull, because that cursor is central's only proof of receipt — a fault between the response leaving the site and central committing it re-serves the rows instead of losing them (central dedups on `EventId`). The request carries an additive composite-keyset `after_id` (field 3) mirroring `PullSiteCallsRequest`; with it the read is a strict `(OccurredAtUtc, EventId)` keyset and the retirement is exact, without it the legacy inclusive `>=` read applies and only rows strictly older than the cursor instant are provably received.
|
||||
- `PullSiteCalls(PullSiteCallsRequest) returns (PullSiteCallsResponse)` — central→site reconciliation pull for the Site Call Audit (#22) self-heal feed; the site serves operation-tracking rows changed since a cursor from its `IOperationTrackingStore`. A separate RPC from `PullAuditEvents` because the tracking store is the operational source of truth, distinct from the site audit queue.
|
||||
- `PullTraceEvents(PullTraceEventsRequest) returns (PullTraceEventsResponse)` — central→site **point read** of `ScriptTrace` rows for named execution ids (Flow Debugger value-change runs; see §8). `AttributeValueUpdate` also gained `source_execution_id` (field 7) and `trace` (field 8) so a traced change's provenance reaches central's stream consumers.
|
||||
- **Messages**: `InstanceStreamRequest` (correlation_id, instance_unique_name, `batching_supported`), `SiteStreamRequest` (correlation_id + `batching_supported` — no instance name; drives the site-wide `SubscribeSite` stream), `SiteStreamEvent` (correlation_id, oneof event: `AttributeValueUpdate`, `AlarmStateUpdate`, `SiteStreamEventBatch`); `AuditEventDto`/`AuditEventBatch`/`IngestAck` for ingest; `CachedTelemetryPacket`/`CachedTelemetryBatch` (each packet pairing an `AuditEventDto` with a `SiteCallOperationalDto`); `PullAuditEventsRequest`/`PullAuditEventsResponse` and `PullSiteCallsRequest`/`PullSiteCallsResponse` (each request carries `since_utc` + `batch_size` + an optional composite-keyset `after_id`; each response carries `more_available` to signal a saturated batch).
|
||||
- The `oneof event` pattern is extensible — future event types (health metrics, connection state changes) are added as new fields without breaking existing consumers.
|
||||
- Proto field numbers are never reused; new RPCs and message fields are appended additively. Old clients ignore unknown `oneof` variants — **but "ignore" means the event vanishes silently**, which is why the batch frame below is gated behind an explicit negotiation flag rather than simply emitted.
|
||||
@@ -275,6 +277,7 @@ Keepalive settings are configurable via `CommunicationOptions`:
|
||||
- Parked messages (store-and-forward dead letters).
|
||||
- Site event logs.
|
||||
- Instance debug snapshots (attribute values and alarm states).
|
||||
- **Trace events by execution id** — `PullTraceEvents(PullTraceEventsRequest{execution_ids[], since_utc, max_events}) returns (PullTraceEventsResponse{events[]})` on `SiteStreamService` (Flow Debugger, 2026-09-05): a bounded point read of the site audit queue (64 ids / 2000 events, `InvalidArgument` beyond) that the central `FlowDebugRunner` polls for the run a traced value change spawned. Not a reconciliation path — it moves no cursor and does not flip the NodeA→NodeB channel on error. Client: `IPullTraceEventsClient` (Commons) → `GrpcPullTraceEventsClient` (AuditLog, PSK-authenticated like `PullAuditEvents`).
|
||||
- Central can also send management commands:
|
||||
- Retry or discard parked messages and parked cached calls — central sends `RetryParkedOperation` / `DiscardParkedOperation` (keyed by `TrackedOperationId`) to the owning site, which applies the change to its S&F buffer and tracking table.
|
||||
|
||||
|
||||
@@ -70,6 +70,7 @@ Each API method definition includes:
|
||||
- Field names and (extended-system) data types. Supports returning **lists of objects** and arbitrarily nested structures.
|
||||
- **Implementation Script**: C# script that executes when the method is called. Stored **inline** in the method definition. Follows standard C# authoring patterns but has no template inheritance — it is a standalone script tied to this method.
|
||||
- **Timeout**: Configurable per method. Defines the maximum time the method is allowed to execute (including any routed calls to sites) before returning a timeout error to the caller.
|
||||
- **TraceAllCalls** (Flow Debugger, 2026-09-05): when set, every call to this method runs its routed site scripts with the trace flag on, so their `ScriptRun` / `AttributeWrite` / `AttributeWait` audit rows (`AuditChannel.ScriptTrace`) are recorded for Replay. Default off — adds roughly seven audit rows per call, so it is meant for low-volume integration methods. Set from the method form's *Trace all calls* checkbox or `api-method create|update --trace-all-calls`.
|
||||
|
||||
> **Authoring note — idempotent methods for at-least-once callers**: some external callers retry
|
||||
> on their own connect-failure/timeout failover loop and cannot tell "the first attempt actually
|
||||
@@ -296,6 +297,11 @@ For everything else, scripts interact with the system through the curated `Route
|
||||
- The `Route.To()` helper resolves the instance's site assignment from the configuration database and routes the request to the correct site cluster via the Communication Layer.
|
||||
- The call is **synchronous from the API caller's perspective** — the API method blocks until the site responds or the **method-level timeout** is reached.
|
||||
- If the target site is unreachable or the call times out, the call fails and the API returns an error to the caller. No store-and-forward buffering is used for inbound API calls.
|
||||
- **Every `Route.To(...).Call(...)` writes one always-on `ScriptTrace.RouteCall` audit row** (central direct write, actor `inbound-api`, target `{instance}.{script}`, `ExecutionId` = the inbound request's, `Extra.childExecutionId` = the site run's id) — this row is what joins the central and site halves of a Flow Debugger Replay, so it is not gated. The routed request carries a `Trace` flag beside `ParentExecutionId` (`RouteToCallRequest.Trace`, `RouteToSetAttributesRequest.Trace`); it is set when the method has `TraceAllCalls` or the call is a Flow Debugger Live run, and only then does the site emit its own trace rows and return them inline on `RouteToCallResponse.TraceEvents` (with `ChildExecutionId`). Untraced calls are byte-identical to before.
|
||||
|
||||
## Flow Debugger runs
|
||||
|
||||
The Flow Debugger (Central UI, design `docs/plans/2026-09-05-flow-debugger-design.md`) drives this component through three management commands handled by the per-node `ManagementActor` (Deployer role): `FlowDebugRunMethodCommand` executes a method exactly as an external caller would — `ParameterValidator` + `SchemaRefResolver`, then `InboundScriptExecutor.ExecuteAsync(..., trace: true)` with a `RouteHelper` trace sink collecting the `RouteCall` row and the site's inline events — and **synthesizes the `ApiInbound.InboundRequest` row itself** with the operator as actor (`flow-debugger` source tag) because no HTTP request passed through `AuditWriteMiddleware`; it refuses on a standby node with `STANDBY_NODE` (`IActiveNodeGate`), so the script runs only where a real call would. `FlowDebugRunValueChangeCommand` writes an attribute through the traced routed-write path and polls the site (`PullTraceEvents`) for the run the change spawned; `FlowDebugReplayCommand` reads the chain back from the central Audit Log. The result (`FlowDebugRunResult`) carries the audit events inline — the graph is a central-side projection, never a wire type.
|
||||
|
||||
## Authentication Details
|
||||
|
||||
|
||||
@@ -220,6 +220,16 @@ The decorator changes **no resolution result**, only object identity: it returns
|
||||
- `ResolveReference` keys on the full `(reference, baseFilePath, properties)` tuple, case-**sensitively**, because `baseFilePath` is a filesystem path on the Linux containers these nodes run in.
|
||||
- A `GetOrAdd` factory race can mint one duplicate — bounded and benign. Entries are never disposed; the cache is bounded by the distinct assemblies on disk, the same order as the static `AnalysisReferences`.
|
||||
|
||||
### REQ-SA-7: Flow Shape Predictor (`FlowShapePredictor`)
|
||||
|
||||
`FlowShapePredictor` (Flow Debugger design `docs/plans/2026-09-05-flow-debugger-design.md` §8) predicts, from script **syntax** alone, the swimlane shape a run would produce before anything executes: `PredictInboundMethod(methodName, methodScript, resolveRouteScript, maxDepth = 4)` and `PredictSiteScript(scriptName, scriptBody, maxDepth = 4)` return a `PredictedFlow` of `PredictedNode(Id, Kind, Lane, Title, InputDependent, Note)` + `PredictedEdge(From, To, CrossesBoundary)`.
|
||||
|
||||
- **Syntax walk, not semantic.** A `CSharpSyntaxWalker` over `CSharpSyntaxTree.ParseText(code, SourceCodeKind.Script)` matches the script-API member-access chains by whitespace-stripped text, exactly as `ScriptTrustValidator`'s hardening walker does: `Route.To(x).Call(y)` / `.SetAttributes` / `.WaitForAttribute`, `Scripts.Call(y)`, `…Attributes.WriteBatchAndWaitAsync` / `WaitAsync` / `WaitForAsync` / `SetAsync`, `…Attributes[x] = v`, `ExternalSystem.Call/CachedCall`, `Database.*`, `Notify.Send`. Reads (`GetAttributes`, indexer reads) produce no node because the runtime records no trace row for them.
|
||||
- **Literals resolve; everything else is `InputDependent`** and titled with the expression's source text (`Route.To(code)`). Branching is not modelled: every recognised call is emitted once in source order, and an `if (…) return/throw` guard becomes a `Note = "early return: <condition>"` on the node that **follows** it (the condition is visited first, so a guard whose condition *is* the traced call — Delmia's `if (!await …WriteBatchAndWaitAsync(…)) return …` — attributes correctly).
|
||||
- **DB-free by contract.** Routed and nested script bodies come from the caller-supplied `resolveRouteScript(instanceCode, scriptName)` (Central UI resolves instance → template parent chain → script; called with the literal code when constant, else with the expression text so a resolver may still answer); recursion stops at `maxDepth` with a `depth limit` note.
|
||||
- **Total.** Parse errors or any walker fault degrade to the request/response pair (or the root `ScriptRun`) carrying `Note = "parse error"`; the predictor never throws into the page.
|
||||
- Predicted kinds are exactly the traced kinds (`FlowStepKind`), so `FlowGraphView` can overlay the real graph on the prediction by kind and grey the nodes a run never produced. There is deliberately **no separate central "method script" node** — the real graph records none, so the `InboundRequest` node stands for it.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
@@ -192,6 +192,9 @@ flowchart TD
|
||||
- Holds current alarm states (active/normal), updated by child Alarm Actors.
|
||||
- Publishes attribute value changes and alarm state changes to the site-wide Akka stream.
|
||||
|
||||
### Trace arm (Flow Debugger, 2026-09-05)
|
||||
- A `SetStaticAttributeCommand` / `SetDataAttribute` carrying `Trace = true` and a `SourceExecutionId` **arms** the attribute on the Instance Actor for `TraceArmWindowSeconds` (default 30, validated `> 0`). A static write publishes `AttributeValueChanged` with the writer's `SourceExecutionId` and `Trace` immediately; a **data-sourced** write is a real tag write, and the next device echo of that attribute inside the window consumes the arm and is published with the same ids — so the script run the echo triggers is parented to the writer's execution and runs traced. An echo outside the window, or a second echo, is an ordinary untraced change. Batch-write echoes are not armed (design §5.3); only the single-attribute write path arms.
|
||||
|
||||
### Initialization
|
||||
1. Load all attribute values from the flattened configuration (static defaults).
|
||||
2. Set quality to **uncertain** for all attributes that have a data source reference. Static attributes (no data source reference) have quality **good**. The uncertain quality persists until the first value update arrives from the Data Connection Layer, distinguishing "not yet received" from "known good" or "connection lost."
|
||||
@@ -276,9 +279,16 @@ When the Instance Actor is stopped (due to disable, delete, or redeployment), Ak
|
||||
- **Concurrent runs per script are capped (WP3.1).** `MaxConcurrentRunsPerScript` (default 4) bounds runs in flight — queued or executing — for any one script or alarm on-trigger script. Over the cap the **newest** run is shed: the four already in flight are closest to their own deadlines and already charged against them, so nothing is ever reordered and no extra queue is needed (the scheduler's FIFO already is the queue). A shed increments `ISiteHealthCollector.IncrementScriptRunShed` (surfaced as `ScriptRunShedCount`), emits a `script`/`Warning` site event **rate-limited to one per script per minute** so a hot trigger cannot flood `site_events`, and — for an Ask-based `CallScript` — replies with an explicit error so a nested call or inbound-API route fails fast instead of hanging.
|
||||
- **The cap gates only depth-0 launches (review fix).** A trigger fire or a depth-0 `CallScript`/inbound-API route counts against `MaxConcurrentRunsPerScript`; a *nested* `Instance.CallScript` self-recursion (`callDepth > 0`) is exempt — it is already bounded by `MaxScriptCallDepth` instead, and the calling run is holding one of the four cap slots itself while it awaits the callee, so counting the nested launch too would spuriously shed legitimate self-recursion.
|
||||
|
||||
#### Traced runs (Flow Debugger, 2026-09-05)
|
||||
- A run is **traced** when its `ScriptCallRequest.Trace` flag is set (a routed inbound call with `TraceAllCalls` or a Flow Debugger Live run), or when it was spawned by a value change whose `AttributeValueChanged.Trace` was set (see *Trace arm* under the Instance Actor). `ScriptRunLauncher` passes `isTraced` + a per-run `FlowTraceCollector` (cap 500 events, `Truncated` flag) into `ScriptRuntimeContext`, and on completion — success, timeout or failure — emits one `ScriptTrace.ScriptRun` audit row (occurred-at = run start, `Extra.triggerKind` = `call` or `valuechange` with the trigger attribute/value/source) through the site `IAuditWriter`.
|
||||
- Every trace row is **also** collected in the run's collector and returned inline on `ScriptCallResult.TraceEvents` (with `ScriptCallResult.ExecutionId`), which `DeploymentManagerActor` maps onto `RouteToCallResponse.ChildExecutionId` / `TraceEvents` — Live gets its steps in the same reply; Replay reads the same rows from the Audit Log after the 5 s telemetry push.
|
||||
- A value-change run spawned by a traced write inherits the writer's execution id as `ParentExecutionId` and runs traced itself, so a Flow Debugger value-change entry sees the run the write caused.
|
||||
- Untraced runs emit nothing new — the flag defaults to `false` on every record and the collector is null.
|
||||
|
||||
### Handling `Instance.CallScript`
|
||||
- When an external caller (another script run, an alarm on-trigger run, or a routed call from the Inbound API) sends a `CallScript` message to the Script Actor, it launches a run to handle the call.
|
||||
- The caller uses the **Akka ask pattern** and receives the return value when the execution completes.
|
||||
- **Trace flag inheritance:** `ScriptRuntimeContext.CallScript` forwards the calling run's `IsTraced` on `ScriptCallRequest.Trace`, and `CreateChildContextForSharedScript` hands the same flag + collector to an inline shared-script context, so a nested run's `ScriptRun` / `AttributeWrite` / `AttributeWait` rows land in the same flow.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user