ScadaBridge/docs/plans/2026-08-15-arch-review-remediation-execution-log.md

Arch-review remediation — execution log

Ran: 2026-08-14/15 · Plan: docs/plans/2026-08-14-arch-review-remediation-plan.md · Branch: arch-review-remediation (off main) · Status: code-complete, docs-propagated, not yet pushed/merged.

Source: the 2026-08-14 eight-pass performance/architecture review (all 27 components + ZB.MOM.WW.LocalDb). This log is the compact record of what shipped, how it was gated, and what's left. Full package specs live in the plan; this is the after-action summary.

Phases and package → commit map

Every package ran as an isolated-worktree subagent, merged --no-ff onto arch-review-remediation; commit hashes below are the substantive commit on the branch (merge commits omitted).

Phase 0 — Preflight. 0b201e41 docs(plans): the remediation plan itself.

Phase 1 — Quick wins (7 packages, parallel).

WP	Commit	What
1.1	34a3f4bb	Reconnect on graceful (OK-status) stream completion — closes the 4h silent stream death
1.2	2e4e41a8	Site audit DB onto the mounted data volume; required path + soft flush
1.3	7ebdcd37	CDC capture installed only when replication is configured
1.4	600659d5	Sweep/KPI covering indexes + sliced notification terminal purge
1.5	125055d9	O(1) attribute resolution, precomputed types, coalesced static writes, shared JSON options
1.6	c5e66ed4	Fail known-dead sends immediately instead of burning Ask timeouts
1.7	2cfcd890	Batched event-log commits + sliced retention purge

Phase 2 — Seam rework (6 packages, parallel after the WP2.1a design doc 1040dc0f).

WP	Commit	What
2.1	d15c5f02	DCL batch subscribe/read/write seam, bounded reconnect, sharded subscriptions
2.2	5db2a810	Central set-based ingest, aligned partition purge, KPI query shapes, EF hygiene
2.3	2ce0ad7e	Alarms-only stream seed, capped buffers, at-least-once audit pull
2.4	8c0b36b2	Shared KPI cache, live-cache-backed alarm summary, coalesced Debug View renders
2.5	48b3c40a	Flatten-session caching, bulk DeploySiteAsync, paged management queries
2.6	a2122831	Cached hot-path lookups, bounded observer queue, alarm-priority stream path

Plus a5882753 closing Phase-2-gate residuals (direct ingest path, monotonic timeouts, synthetic probe, not-reporting set, cursor-exact audit pull) found while gating.

Phase 3 — Structural (design-first).

WP	Design	Commit	What
3.1	312216ff	c4fc1f8e	Script execution pool split — trigger evals off the blocking pool, bounded/deadline-aware execution
3.2	6cfb2dd8	c254d074	site_events volume policy — sampled per-run events, interval summaries, replication pinned
3.3	(scadaproj)	cca7f178 + scadaproj 9377fa1	LocalDb 0.2.0 — dereg cleanup, late-opt-in baselining, byte-budget replication

Phase 4 — Verification, adversarial review, docs. Six parallel code-reviewer passes over the full diff, one per area (site runtime, DCL, comms, site persistence, central SQL, UI/deploy), each instructed to try to refute the fixes. Confirmed findings landed as targeted follow-ups:

Area	Commit	What
Site runtime	950c54c5	Recursion-exempt run cap (nested CallScript no longer double-gated), atomic detach counter, summary edge cases, per-row event-log fallback
DCL	37f13e2e	Discard in-flight subscribe results for tags unsubscribed mid-flight; release the orphaned handle
Central SQL	5d075f13	No client-side audit truncation, insert-first upsert (Site Call Audit), QI-safe filtered-index scripts, honest operator-not-found replies
Comms	fd5e023d	Consumer-based debug-stream orphan net, foreign-cancel triad, honest onConnected, served-row-exact retirement, full-rate reconcile
UI/deploy	e0e4b246	Honest CLI HTTP timeouts, watermark-complete staleness (3 missed bump sites), phase-2 PendingDeployment staging, lock-safe cancellation
Site persistence	56c99c92 + f689f495	Required audit DB path on wonder; explicit MaxBatchSize:64 LocalDb read-page cap; rate-limited observer drop logging; LocalDb 0.2.1 (HLC anchor flush on dereg, bounded 64-message sync inbox)

Gate results

• Baseline: 7587 tests green at Phase 0 entry; test count never reduced across any phase gate.
• Live probes (rig):
  • Stream lifetime forced to 2 minutes — the alarm stream reconnected in 8.5s, within one reconcile tick of the OK completion, IsLive correctly reflected the gap (WP1.1).
  • CDC conditional-registration: site-b booted clean on LocalDb 0.2.0 with 30 stale triggers dropped at startup (WP1.3/WP3.3) — confirms the self-heal path fires, not just the steady state.
  • S&F due-sweep: EXPLAIN QUERY PLAN confirmed index-terminated (WP1.4).
  • Failover drill (docker/failover-drill.sh) unaffected by the actor/timeout changes.
• Adversarial review tally: ~25 confirmed findings across the six areas, 4 High, all fixed in the Phase 4 commits above. Zero findings deferred as won't-fix.
• Test-flake root causes (3, all test-side, not production bugs):
  1. cfa6acbf — an assertion on MarkForwarded ran before the push it depended on was guaranteed to have landed; reordered behind the push.
  2. c4caebe9 — two dispatcher audit-safety tests asserted an attempt count without synchronizing on the async write that produced it; the unsynchronized assertion was removed.
  3. 950c54c5 (embedded) — ScriptDeadlineAtEnqueueTests' "no started event" assertion went vacuous once WP3.2 flipped PerRunScriptEvents to off-by-default (fixed by opting the test back in); ScriptRunLauncherParityTests widened an ExpectMsg window that would have passed for any deadline from 1s to 300s, not just the intended one (sharpened to assert the reported timeout value AND a tight wall-clock range).
  • Follow-up recommended, not done here: a suite-wide sweep for the same AwaitAssert(...)-then-bare-Assert pattern — an AwaitAssert that only proves "eventually true," followed by a plain assertion that silently inherits its timing slack, is the shape behind all three; worth a grep-and-review pass rather than fixing on-demand as flakes surface.

Residuals register

Deliberately not fixed in this program — each has a stated reason, not an oversight:

1. DCL unsubscribe-during-reconnect count staleness. The 37f13e2e fix discards orphaned in-flight results but a per-connection counter can still drift under rapid subscribe/unsubscribe churn during a reconnect; needs a per-tag counted set. Low severity, cosmetic (a health-report number), deferred. RESOLVED 2026-08-15 — DataConnectionActor's _totalSubscribed/_resolvedTags scalars are deleted and both health counts are now DERIVED at report time from the authoritative per-tag state (_instancesByTag.Count, the per-tag counted set the residual called for, and _subscriptionIds.Count), so no accumulated counter exists to drift; this also closes the connection-level-failure case that let resolved climb above total. Regression tests: TagResolutionCounts_* in DataConnectionActorBatchTests.
2. Per-table needs_snapshot in LocalDb. RESOLVED 2026-08-15 (LocalDb 0.3.0, scadaproj cea3930; ScadaBridge pin bump a5ac309a, R1). Bookkeeping schema v3 adds __localdb_snapshot_state (per-table flags); the database-wide flag survives by design for whole-oplog events (cap prune, impossible ack). Wire-compatible by capability negotiation, not a version bump: Handshake.supports_partial_snapshot + SnapshotBegin.tables (empty = all) + SnapshotAck on the unbounded control lane — a peer that doesn't advertise gets a FULL snapshot, which is required (an old receiver advances its watermark to the snapshot's as-of and would silently discard un-included tables' deltas). A partial snapshot moves no watermark on either side; debt clears on receiver ack (crash-safe), clear-on-enqueue kept for old peers. Library suite 199/199 incl. old-peer interop; ScadaBridge Host 490/490 on the bump. Adding an eleventh table to ReplicatedTables on a replicating site now snapshots ONE table instead of re-streaming all, both directions.
3. Event batching per proto message. RESOLVED 2026-08-15 (9b5cb3dd+e6842c10, R2). Additive wire shape, no new RPC: batching_supported request flags + SiteStreamEvent.batch (oneof field 4) carrying SiteStreamEventBatch. Negotiation is the proto3 default itself and is LOAD-BEARING — an un-negotiated batch frame would parse as EventOneofCase.None on an old central and vanish silently, so an old central (which cannot set the flag) never receives one; an old site ignores the flag and keeps per-event frames. Per-subscriber SiteStreamEventBatcher downstream of the relay's bounded channel (framing only — never delays a lone event; lingers only once a backlog is proven); defaults 100 events / 25 ms, options-validated strictly < 250 ms to pin the load-test latency budget. Measured worst case P99 25.4 ms; 600 queued events → 6 frames. Skew proven both directions over real wire bytes. Histogram scadabridge.site.stream.batch_size (allowlist-verified). Does NOT change the burst ceiling of register row 31 (documented). In-flight it also found and fixed a pre-existing SiteStreamGrpcServer bug (register row 47).
4. Deployments page server-side paging + status counts. RESOLVED 2026-08-15 (35ce1413, R3). The Deployments page is database-paged: QueryDeploymentListPageAsync returns one instance-joined page + the filtered total, GetDeploymentStatusCountsAsync returns the tiles from one grouped aggregation, and site scoping runs in SQL. Offset paging (not keyset) because the numbered jump-to-page pager needs a total, which a cursor cannot express, and the retention purge bounds the table — mirrors the Notification Outbox precedent; DeployedAt DESC, Id DESC tie-break keeps offset pages stable. Whole-table GetAllDeploymentRecordsAsync deleted with its last caller. Tiles double as the status filter; 500 ms trailing-edge debounced search. 34 new tests; Component-CentralUI.md updated.
5. OtOpcUa still pins LocalDb 0.1.3. RESOLVED 2026-08-15 (OtOpcUa branch localdb-bump, commit a1a258d8, R4). Bumped 0.1.3 → 0.2.1 (newest on the feed at the time). Pure pin bump — no code change; all 0.2.x additions are additive and unused there. Also corrected OtOpcUa's now-false MaxBatchSize = 16 "row-count vs gRPC 4 MB" rationale in 5 places (kept at 16 as the DB read-page bound per finding F2) and documented MaxBatchBytes. Full 57-project suite green bar one unrelated pre-existing flake (register row 38). Two OtOpcUa-side findings recorded as register rows 38–39.
6. Fragile SandboxTests timing pin. RESOLVED 2026-08-15 (ca30d17f+9fb52153, R5). Root-caused causally, not by loosening timeouts. The pin was a wall-clock CancellationTokenSource (100 ms) raced against a fixed workload measured at 298 ms — fails outright on faster hardware, and passes vacuously in the opposite direction because a pre-cancelled token throws before the script body is entered. Replaced with a deterministic same-thread cancellation edge plus a negative control; 600 ms of test sleeping removed. The deferred sweep ran with it: 3 further instances of the c4caebe9/cfa6acbf flake class fixed (ScriptRunShedTests — where CollectReport DRAINS the shed counter, so the fix accumulates across polls; SyncCallEmissionEndToEndTests; DebugStreamBridgeActorTests), each with fail-before-fix by injected delay + a suppression control; 2 ambiguous candidates cleared with product-code evidence. Test-side only (git diff src/ empty). A different timing-pin class (wall-clock sleeps asserting absences) was found and registered, not fixed (register row 40).
7. Target-scale load test (deferred-work register #25). RESOLVED 2026-08-15. The follow-on validation ran: 10 sites x 500 instances x 75 tags = 375,000 live tag subscriptions, 37,518 updates/s achieved vs 37,500 nominal, 45,021,375 updates over a 20-minute steady-state window. The moved ceilings hold with room to spare — end-to-end tag latency P99 4.57 ms, zero dropped events at 100 live subscribers, health report collect+ingest P99 0.31 ms, debug view P99 2.19 ms with no timeouts, CPU 2.9% of the box. 11 clean passes, 1 pass with a caveat (F1: 20 minutes with zero gen-2 collections cannot fully settle the leak question; the 1-hour run would), 0 failures. Register row 50 (S&F drain rate + per-subscriber stream backpressure) was absorbed into the same run and closed with it — S&F drains at 3,533 msg/s, and a stalled subscriber costs healthy peers nothing (100.00%, zero drops). Results: docs/plans/2026-08-15-target-scale-load-test-results.md; design: docs/plans/2026-08-15-target-scale-load-test-design.md; harness: tests/ZB.MOM.WW.ScadaBridge.LoadHarness/. Residual: WP-4's [xc-3]/[xc-4]/[xc-5]/[xc-8] are central-cluster/browser concerns outside this harness and remain open scope.
8. Playwright 14 pre-existing env failures. RESOLVED 2026-08-15 (014038fb, R6). All 14 triaged to root cause: every one was selector drift from two merged Central UI refactors (9e243493 density sweep moved .btn-sm off buttons onto btn-group-sm groups and re-cast SiteForm node sections as cards; a506b19d moved TemplateEdit modals to the global DialogService host) — no stale rig data, no timing bug, no app defect among the 14. Suite now 0 failed / 172 passed / 1 honestly-skipped (SmtpEdit test self-skips: SMTP config rows cannot be created or deleted from the management surface, UI-only). Runtime halved (10m → 4.5m — each failure had been burning a 30 s locator timeout). All 49 .btn-sm selectors audited individually rather than swept; the 35 still-correct ones left alone with comments. One stray test area deleted from the rig via CLI. Three genuine app bugs found live during triage — recorded as register rows 43–45 (import-session slot leak, dead unlock rate limiter, shared DbContext in outbox dispatch), plus a CLI query oddity as row 46 — found, not fixed, per the program's scope rules.
9. site_events retention purge still oplog-visible. RESOLVED 2026-08-15 (documented by design, 9d2834e3, R7). The CDC capture of retention/cap deletes is correct and unchanged — there is deliberately no purge-exemption path. The operator-facing gap is closed three ways: docs/deployment/topology-guide.md → Reading the replication backlog (burst shape, 24 h cadence anchored to active-node process start, healthy-vs-faulty signature), Component-SiteEventLogging.md → Storage, and Component-HealthMonitoring.md, which gained the two previously-undocumented LocalDbReplicationConnected/LocalDbOplogBacklog metric rows. EventLogPurgeService now emits one Information line correlating the spike to the purge on a replication-enabled node (both the retention and the storage-cap delete paths), via a Host-supplied SiteEventLogReplicationCheck delegate sourced from the single SiteLocalDbSetup.ReplicationIsConfigured predicate. No behavior change; 5 unit tests.

Docs propagated

Component docs (DataConnectionLayer, SiteRuntime, Communication, AuditLog, SiteEventLogging, StoreAndForward, NotificationOutbox, SiteCallAudit, TemplateEngine, DeploymentManager, CentralUI, ConfigurationDatabase), CLAUDE.md Key Design Decisions (stream-completion reconnect, required site audit DB path, CLI HTTP timeout honesty, bulk DeploySiteAsync, LocalDb 0.2.1), and docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md (amended for the MaxBatchSize:64 read-page pin superseding the "retired, left at defaults" note). No components added/removed; README component table unchanged. Cross-reference sweep found no stale live references to ScriptExecutionActor/AlarmExecutionActor or UX_AuditLog_EventId outside historical plan/known-issue records, after correcting four residual mentions in docs/requirements/Component-SiteRuntime.md left over from the WP3.1 doc pass.