Code review of the merged arch-review remediation (Tasks 1-13) found four items.
1. FlowDebugRunner returned its OWN synthesized rows un-redacted. RunMethodAsync's
inbound row and RunValueChangeAsync's root row were handed to WriteCentralAsync
(whose writer redacts a private copy) and then put RAW into the returned Events —
so a Live run's caller saw the unfiltered row while Replay of the same execution
showed the redacted one, defeating F1's "Live shows what Replay reads back". Both
methods now resolve IAuditRedactor once per run and place the filtered copy into
the result (including the early-return path on a failed attribute write); the raw
row still reaches the writer, which redacts itself.
2. FlowGraphBuilder's spawner search degenerated to "the last step of the parent".
Since executions are ordered as contiguous blocks, position[p] < childPosition
holds for EVERY parent step, so a run shaped ScriptRun -> Write(A) -> Wait ->
Write(B) whose Write(A) fired the cascade drew the Spawn edge from Write(B). New
rule, in order: (a) the last non-async, non-Response parent step occurring at or
before the child's first step — parent and child of a trigger cascade share the
site clock, so this picks the write that actually fired it; (b) the position rule
as fallback, which is the mixed-clock case F5 was written for (a child clock
behind its parent's, where no parent step satisfies (a)); (c) parentChain[0].
The class remark's stale "Replay order is unchanged" claim is replaced with what
actually happens to a central Database.* call made after Route.To().Call.
3. FlowTraceCollector's byte budget counts UTF-16 chars of DetailsJson only, so
"well under the 4 MB gRPC default" overstated it. Documented honestly: ~1 MB of
UTF-8 for ASCII payloads, ~3 MB worst case for non-ASCII, both under 4 MB before
framing — and raise it only against a real serialized-size measurement.
4. CLI --timeout-seconds help + README row now say the management endpoint caps
flow-debug run-method at ManagementService:LongRunningCommandTimeout (default
5 min), so a larger client value still 504s at the server's ceiling.
Deferred-work register rows 112 (site-side inline trace truncation is invisible to
the operator — needs a proto field) and 113 (the F2 method-entry site scope gates
routed traffic only; ApiMethods have no site affinity to scope against) record what
this review deliberately did not fix.
Tests: FlowGraphBuilderTests gains the same-clock Write(A)/Wait/Write(B) cascade
(red before the rule change, and the clock-behind test still passes on rule (b));
FlowDebugRunnerTests extends the redactor test to the inbound row and adds two
value-change cases (settled poll, failed write). Commons 793, ManagementService 615,
CentralUI 1264, CLI 462 — all green.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Component-CentralUI (new Flow Debugger page section), Component-InboundAPI (TraceAllCalls
property, always-on RouteCall row + Trace flag under Routing Behavior, new 'Flow Debugger
runs' section), Component-SiteRuntime (Instance Actor trace arm; Script Actor traced runs;
CallScript flag inheritance), Component-AuditLog (ScriptTrace channel in Scope, the
PullTraceEvents point-read note + mixed-version caveat under Reconciliation, per-channel
retention example), Component-Communication (§4 trace flag + inline events with proto field
numbers; §8 PullTraceEvents; RPC list), Component-ScriptAnalysis (REQ-SA-7 FlowShapePredictor),
CLAUDE.md UI bullet, deferred-work register row 110 (vertical-trace list toggle + console
capture), design doc status → implemented. CLI README sections landed in Tasks 3/16.
Umbrella index (../scadaproj/CLAUDE.md) checked: its ScadaBridge row records stack/wire
relationships only, unchanged by a new page — no edit.
Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
Row 104: docker/vacuum-site-localdb.sh stops both nodes of a pair together, sets
auto_vacuum=INCREMENTAL + VACUUM on each node volume with integrity_check before/after,
then starts both (DRY_RUN=1 reports). Run on all three rig pairs: six files 0 -> 2, ok,
~11% smaller; site-a replication reconnected with backlog 0 and no watermark change —
VACUUM is replication-invisible (ledger keys by declared PK, oplog seq is INTEGER PRIMARY
KEY AUTOINCREMENT). Topology guide gains the recipe.
Row 105: SiteLocalDbReplicatedTableFkGuardTests builds the real site schema via OnReady
(unreplicated config so 0.3.3's own guard cannot mask the culprit) and asserts every
SiteLocalDbSetup.ReplicatedTables entry (now internal) exists and has no inbound
CASCADE/SET NULL/SET DEFAULT foreign key, with a negative control. The constraint is
enforced at build instead of at the first replicated boot.
Row 106 (found during 104's live verification, NOT caused by it — the identical 127k-row
full snapshot also landed at the earlier deploy restart): SnapshotStreamer.SendAsync stamps
as_of_seq = MAX(seq), which is 0 once a converged sender's oplog is ack-pruned to empty;
a zero-watermark receiver (rebuilt node) is never lifted, so the empty-oplog horizon rule
re-streams a full snapshot on every reconnect. Sender-side one-line fix for LocalDb 0.3.4:
as_of_seq = max(MAX(seq), last_acked_seq).
The gauge test polls baseline+1, but the process-global gauge's baseline is
captured once: a parallel class's streams closing after capture lowers the
gauge below baseline+1 forever. Survey confirmed only SiteStreamGrpcServerTests
and CleanupVerificationTests drive the real SubscribeInstance handler that
moves the counter; both now share the SiteStreamSubscriberGauge collection
(row-96 pattern). 4/4 full-suite runs green, durations unchanged.
Row 79 executed its own prescribed remedy without waiting for a second
corruption: all ten rig data dirs (8 docker/ nodes + 2 docker-env2/ site
nodes) move from virtiofs bind mounts to named docker volumes
(scadabridge-<node>-data), seeded in place with both rigs stopped and
live-verified (site-a localDbReplicationConnected=true / backlog 0 over
the migrated files; env2 stable, 0 restarts). The feared inspectability
loss is moot — OrbStack exposes volumes at ~/OrbStack/docker/volumes/.
The rebuild surfaced latent env2 drift (pre-Secrets-0.6.2 image): env2
now carries the x-secrets-store-env posture + central data volumes.
Row 93's silent maintenance trap is now loud: SiteEventsSchemaCalibrationTests
pins the site_events columns/indexes the EventLogPurgeService size model is
calibrated against, failing with a pointer to the memo §4 recalibration on
any schema drift. Rows 92 (accepted by design) and 94 (standing tripwire)
annotated accordingly.
Full-scale run executed on this machine: 10 sites x 500 instances x 75 tags =
375,000 live tag subscriptions, 37,518 updates/s achieved vs 37,500 nominal,
45,021,375 updates over a 20-minute steady-state window (M4 Pro, 14 cores, 48 GB,
with the 8-node docker rig still running so the figures are pessimistic).
11 clean passes, 1 pass with a caveat, 0 failures:
tag latency P50 0.88ms P99 4.57ms max 37.41ms (1.1M samples, end-to-end)
stream 900,675 delivered, 0 dropped at 100 live subscribers
health collect+ingest P99 0.31ms, 10/10 sites tracked
debug view P99 2.19ms, 264 completed, 0 timeouts
deploy 500 instances to a site in 2.6s
cpu 41% of ONE core = 2.9% of the box
memory working-set slope +8.83 MB/min
Three findings, reported rather than tuned away:
F1 (Low) 20 min with ZERO gen-2 collections cannot fully settle the leak
question; the heap demonstrably sawtooths but an uncompacted gen-2 makes a
positive slope ambiguous. The 1-hour run would settle it. Not tuned.
F2 (informational, by design) a deferred S&F backlog sits for one full
DefaultRetryInterval (28.9s measured) before anything drains --
EnqueueAsync(attemptImmediateDelivery:false) stamps LastAttemptAt. Easy to
misread as slow drainage, so drain is reported as two numbers: retry wait,
then 3,533 msg/s of actual capacity.
F3 (positive) slow-subscriber isolation is TOTAL: 4 healthy subscribers at
100.00% with zero drops while a peer lost 197,028/200,000 events entirely
within its own bounded channel. Mechanism recorded link by link.
Also records what the run does NOT prove (not clustered, not real-network, not
real OPC UA, not 1 hour) and the four WP-4 sub-criteria this harness does not
cover, so the evidence is not over-read.
Register rows 25 and 50 -> RESOLVED 2026-08-15; remediation execution-log
residual 7 -> resolved; phase-8-checklist WP-4 section replaced with the
measured numbers.
Adds rows 23-26 to the deferred-work register: live LDAP group-membership
re-query (blocked on ZB.MOM.WW.Auth.Ldap gaining a passwordless group
search), M8 large-bundle perf hardening (logged in the M9 completion design,
never given a plan or perf test), the Phase-8 WP-4 target-scale load test
(claimed complete by a 107-byte checklist stub with no evidence), and the
Ipsen MES MoveIn tail (-LT routing, PLC-output flags, Z28062 data).
Splits the review-08 "Failover-timing + broader perf envelope" row: the
failover-timing half is resolved by FailoverTimingTests, now a live [Fact]
on TwoNodeClusterFixture at production timings (PLAN-R2-01 T4); the S&F
drain-rate + per-subscriber backpressure half stays open as its own row.
Moves the closed folder drag-drop row (18, [PERM]) out of Deferred into the
Resolved table per the register's own rule.
Two-node keep-oldest could NEVER survive a crash of the oldest/active node:
Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a
side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs
ITSELF — proven live on the rig ('SBR took decision ... including myself')
before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll);
keep-majority just re-keys the fatal crash to the lowest address.
SplitBrainResolverStrategy gains 'auto-down' (new default): BuildHocon emits
Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter.
The leader among the REACHABLE members downs the unreachable peer, so the
survivor takes over singletons and /health/active in ~25s regardless of which
node died. Accepted trade (explicit owner decision): a real network partition
runs dual-active until an operator restarts one side. keep-oldest remains
supported; DownIfAlone validation is now scoped to it.
Live drill on the rebuilt rig: active-crash TAKEOVER in 28s (victim still
down; all 7 singletons Younger->Oldest), standby-crash removal 27s with 0
routing blips; victims rejoin as standby in 2s. New real-cluster tests pin
both directions (SbrFailoverTests.AutoDown_*); TwoNodeClusterFixture gains a
strategy knob. All 16 appsettings flipped (src, docker, docker-env2, and the
gitignored wonder-app-vd03 overlay on disk — owner must sync to the host).
Docs: decision record docs/plans/2026-07-21-auto-down-availability-decision.md,
Component-ClusterInfrastructure downing section rewritten, drill + README
reworked (active mode now asserts takeover), deferred-work SBR row resolved.
Joseph Doherty