Second slice of the 2026-09-09 whole-system review remediation
(docs/plans/2026-09-09-arch-review.md §10), on top of the High slice 146a1323.
Runtime/devices: SR-2 alarm on-trigger integration scope; SR-3 correlated NativeAlarmActor
subscribe retry; SR-4 immutable audited DB target; SR-5 ordered InstanceActor stop/flush;
SR-12 rehydration fence; SR-13 S&F gate recheck per delivery; DC-4 no OPC UA security
downgrade; DC-5 alarm snapshot for every new subscriber; ORCH-1 sync-dispose joins the pump.
Management/UI: MG-2 site scope on audit tree + Flow Debugger Replay; MG-3 stale-probe memo
removed; MG-4 server-owned bulk-deploy deadline; UI-5 sandbox runs in a killable child
process (new CentralUI.SandboxWorker project); UI-14 bounded sandbox output/IPC.
Edge/audit: EI-1 relay audit ownership; EI-2 SkipBodyCapture suppresses error text; EI-3/EI-11
error-field redaction + caps; EI-4 single relay deadline, pre-execution-only retry; EI-5
authenticated relay provenance; EI-10 body cap before audit read.
Persistence/contracts: DL-1 global Instance.UniqueName unique index + conflict translation;
DL-2 DataConnection protocol config encrypted at rest (migration
20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration — coordinated
central upgrade, old binaries cannot read protected rows); DL-3 bounded Replay paging;
DL-6 cross-site Overwrite of a deployed instance rejected; XC-4 proto descriptor contract test.
Config/hygiene: XC-1 central listener isolation by local port; XC-2/XC-3 inert SMTP + cron
keys removed from rig settings; CM-2 env2 bootstrap guard enabled; DC-6 deploy site exit code;
DC-7 notifier retries only explicit standby rejection; XC-5 ClusterClient purged from docs.
Gates recorded in the report: build 0/0, 9 397 passed / 0 failed, rig redeployed on all
eight nodes with the migration applied. 46 Low findings remain open.
FlowDebugRunner.RunMethodAsync now passes InboundScriptResult.ErrorDetail through to the
synthesized inbound audit row's errorDetail, so a SCRIPT_ERROR run's exception ToString() lands
on AuditDetails.ErrorDetail for the Flow Debugger's inbound node to read (never surfaced to the
HTTP caller). FlowDebugRunResult grows an additive LogsTruncated (default false, last positional
member) reporting the inline FlowTraceCollector's low-priority (ScriptLog) truncation separately
from Truncated (step-row fidelity); RunMethodAsync/RunScriptAsync thread collector.LogsTruncated
through on their two collector-backed results.
Code review of the merged arch-review remediation (Tasks 1-13) found four items.
1. FlowDebugRunner returned its OWN synthesized rows un-redacted. RunMethodAsync's
inbound row and RunValueChangeAsync's root row were handed to WriteCentralAsync
(whose writer redacts a private copy) and then put RAW into the returned Events —
so a Live run's caller saw the unfiltered row while Replay of the same execution
showed the redacted one, defeating F1's "Live shows what Replay reads back". Both
methods now resolve IAuditRedactor once per run and place the filtered copy into
the result (including the early-return path on a failed attribute write); the raw
row still reaches the writer, which redacts itself.
2. FlowGraphBuilder's spawner search degenerated to "the last step of the parent".
Since executions are ordered as contiguous blocks, position[p] < childPosition
holds for EVERY parent step, so a run shaped ScriptRun -> Write(A) -> Wait ->
Write(B) whose Write(A) fired the cascade drew the Spawn edge from Write(B). New
rule, in order: (a) the last non-async, non-Response parent step occurring at or
before the child's first step — parent and child of a trigger cascade share the
site clock, so this picks the write that actually fired it; (b) the position rule
as fallback, which is the mixed-clock case F5 was written for (a child clock
behind its parent's, where no parent step satisfies (a)); (c) parentChain[0].
The class remark's stale "Replay order is unchanged" claim is replaced with what
actually happens to a central Database.* call made after Route.To().Call.
3. FlowTraceCollector's byte budget counts UTF-16 chars of DetailsJson only, so
"well under the 4 MB gRPC default" overstated it. Documented honestly: ~1 MB of
UTF-8 for ASCII payloads, ~3 MB worst case for non-ASCII, both under 4 MB before
framing — and raise it only against a real serialized-size measurement.
4. CLI --timeout-seconds help + README row now say the management endpoint caps
flow-debug run-method at ManagementService:LongRunningCommandTimeout (default
5 min), so a larger client value still 504s at the server's ceiling.
Deferred-work register rows 112 (site-side inline trace truncation is invisible to
the operator — needs a proto field) and 113 (the F2 method-entry site scope gates
routed traffic only; ApiMethods have no site affinity to scope against) record what
this review deliberately did not fix.
Tests: FlowGraphBuilderTests gains the same-clock Write(A)/Wait/Write(B) cascade
(red before the rule change, and the clock-behind test still passes on rule (b));
FlowDebugRunnerTests extends the redactor test to the inbound row and adds two
value-change cases (settled poll, failed write). Commons 793, ManagementService 615,
CentralUI 1264, CLI 462 — all green.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Central rows carry central's clock, site rows the site's, so the global
OccurredAtUtc sort let a 1-2 s skew float a whole site execution ahead of the
inbound row / RouteCall that started it, and pushed the trigger-cascade spawner
search onto its parentChain[0] fallback.
FlowGraphBuilder.Build now orders executions by the input index of their
earliest-given row and orders steps by timestamp only INSIDE one execution
(one clock), breaking ties by kind rank then input index. A folded outbound
node keeps its earliest-by-time row's identity, which need not be the row given
first, so its index is the minimum over the rows folded into it. The spawner
search uses position in the ordered list rather than a raw timestamp compare.
FlowDebugRunner.RunMethodAsync keeps the collector's emission order instead of
re-sorting; the value-change path keeps its OrderBy (all rows share one site
clock). Replay input arrives timestamp-sorted from the repository, so its order
is unchanged.
The DelmiaShaped test fixture is still shuffled within each execution (the
inside-one-execution timestamp rule is unchanged) but now lists the executions
in emission order, since that is what fixes their relative order.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
RunMethodAsync collected sink events into a plain ConcurrentQueue and always
reported Truncated: false, so the Live result carried RAW audit events — the
per-target SQL-parameter overrides, header scrubbing and 8 KB / 64 KB payload
caps that Replay reads back never applied to what the operator saw, and the
reply had no upper bound at all.
It now collects through the same FlowTraceCollector the site launcher uses,
constructed with the host's IAuditRedactor (AddAuditLog's singleton; a host
without one collects raw but is still capped by count and bytes), and reports
Truncated from the collector. Complete() is called as soon as the run returns,
so a late un-awaited call cannot mutate the snapshot the reply is built from.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
A method-entry Live run had no site-scope check at all. RunValueChangeAsync
calls EnforceSiteScope on its named instance, but a method run has no single
target site up front — the operator picks a METHOD, and where its script routes
is only known once it runs — so a site-scoped Deployer could reach any site the
method happened to route to.
The scope is therefore pushed into RouteHelper and enforced at route time:
- RouteHelper.WithSiteScope(IReadOnlySet<string>?) carries a set of site
IDENTIFIERS (null = unscoped, the state of every ordinary inbound HTTP
request; an EMPTY set permits nothing, the fail-closed outcome for a scoped
user whose sites no longer exist). Threaded through every With* copy and To().
- The gate lives in RouteTarget.ResolveSiteAsync — the single choke point that
Call, GetAttributes, WaitForAttribute and SetAttributes all pass through — so
it fires BEFORE any site traffic and throws the new RouteSiteScopeException.
- InboundScriptExecutor maps it to error code SITE_SCOPE_DENIED. The type is
InboundAPI's own (that assembly cannot reference ManagementService), and it
derives from InvalidOperationException so base-type callers are unaffected.
- FlowDebugRunner.RunMethodAsync resolves the user's PermittedSiteIds (numeric)
to identifiers via ISiteRepository and converts SITE_SCOPE_DENIED back to
SiteScopeViolationException (-> ManagementUnauthorized), AFTER writing the
inbound audit row so a denied attempt is still recorded against the operator.
Administrators and unscoped users get a null scope and are unaffected, exactly
as EnforceSiteScope treats them. Every contract change is a trailing optional
parameter. Mutation-checked: disabling the gate fails 3 of the 5 new RouteHelper
tests and 1 of the runner tests, while the in-scope/unscoped positives still pass.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Design D6 / §9 says a Live run is a Deployer act while Replay needs only
OperationalAudit, but the page and all three FlowDebug* commands were gated on
Deployer. The Audit Log and Execution Chain pages -- themselves OperationalAudit
-- show an "Open in Flow Debugger" drill-in that an auditor could not follow.
Implements the split:
- New `RequireFlowDebugger` policy over `FlowDebuggerAccessRoles`
{Deployer, Administrator, Viewer}; the page now carries it instead of
`RequireDeployment`.
- `FlowDebugReplayCommand` takes the same role set in the ManagementActor
matrix; `FlowDebugRunMethodCommand` / `FlowDebugRunValueChangeCommand` stay
DeployerOnly.
- The page resolves `_canLive` (Deployer) in OnInitializedAsync and, without it,
pins the mode to Replay, hides the entry-point and mode toggles, the Run
button and the Live pickers, honours only the `?executionId=` drill-in, and
leaves the parked-node Retry/Discard callbacks unset so those buttons never
render. `CanRun`, `SetMode(Live)` and `ActOnStepAsync` repeat the check so the
split does not rest on markup alone.
The NavMenu entry stays inside the Deployer-gated Deployment section -- auditors
arrive through the audit drill-ins, which is the point of the finding.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Flow Debugger (design 2026-09-05 §7.1): FlowDebugRunMethodCommand / FlowDebugRunValueChangeCommand /
FlowDebugReplayCommand on the per-node ManagementActor (Deployer role; the two run commands
are long-running), all delegating to FlowDebugRunner. RunMethod refuses on a standby with
STANDBY_NODE (ManagementCommandException now carries an optional Code that replaces
COMMAND_FAILED), validates parameters through the real ParameterValidator/SchemaRefResolver
(made public), synthesizes the inbound row (actor = operator, Extra.source = flow-debugger)
and runs the method through InboundScriptExecutor with a trace-sink RouteHelper so the reply
carries the RouteCall row + the site's inline events. RunValueChange does a traced routed
write via IInstanceRouter, then polls IPullTraceEventsClient (interface moved to Commons so
ManagementService can consume it) until a ScriptRun parented to the root settles, else
NO_TRIGGER_WITHIN_WINDOW. Replay reads IAuditLogRepository.GetTraceEventsAsync.
Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
Joseph Doherty