ScadaBridge/docs/plans/2026-08-15-arch-review-remediation-execution-log.md

# Arch-review remediation — execution log

**Ran:** 2026-08-14/15 · **Plan:** `docs/plans/2026-08-14-arch-review-remediation-plan.md` ·
**Branch:** `arch-review-remediation` (off `main`) · **Status:** code-complete, docs-propagated,
not yet pushed/merged.

Source: the 2026-08-14 eight-pass performance/architecture review (all 27 components +
`ZB.MOM.WW.LocalDb`). This log is the compact record of what shipped, how it was gated, and
what's left. Full package specs live in the plan; this is the after-action summary.

## Phases and package → commit map

Every package ran as an isolated-worktree subagent, merged `--no-ff` onto
`arch-review-remediation`; commit hashes below are the substantive commit on the branch (merge
commits omitted).

**Phase 0 — Preflight.** `0b201e41` docs(plans): the remediation plan itself.

**Phase 1 — Quick wins (7 packages, parallel).**
| WP | Commit | What |
|---|---|---|
| 1.1 | `34a3f4bb` | Reconnect on graceful (OK-status) stream completion — closes the 4h silent stream death |
| 1.2 | `2e4e41a8` | Site audit DB onto the mounted data volume; required path + soft flush |
| 1.3 | `7ebdcd37` | CDC capture installed only when replication is configured |
| 1.4 | `600659d5` | Sweep/KPI covering indexes + sliced notification terminal purge |
| 1.5 | `125055d9` | O(1) attribute resolution, precomputed types, coalesced static writes, shared JSON options |
| 1.6 | `c5e66ed4` | Fail known-dead sends immediately instead of burning Ask timeouts |
| 1.7 | `2cfcd890` | Batched event-log commits + sliced retention purge |

**Phase 2 — Seam rework (6 packages, parallel after the WP2.1a design doc `1040dc0f`).**
| WP | Commit | What |
|---|---|---|
| 2.1 | `d15c5f02` | DCL batch subscribe/read/write seam, bounded reconnect, sharded subscriptions |
| 2.2 | `5db2a810` | Central set-based ingest, aligned partition purge, KPI query shapes, EF hygiene |
| 2.3 | `2ce0ad7e` | Alarms-only stream seed, capped buffers, at-least-once audit pull |
| 2.4 | `8c0b36b2` | Shared KPI cache, live-cache-backed alarm summary, coalesced Debug View renders |
| 2.5 | `48b3c40a` | Flatten-session caching, bulk `DeploySiteAsync`, paged management queries |
| 2.6 | `a2122831` | Cached hot-path lookups, bounded observer queue, alarm-priority stream path |

Plus `a5882753` closing Phase-2-gate residuals (direct ingest path, monotonic timeouts, synthetic
probe, not-reporting set, cursor-exact audit pull) found while gating.

**Phase 3 — Structural (design-first).**
| WP | Design | Commit | What |
|---|---|---|---|
| 3.1 | `312216ff` | `c4fc1f8e` | Script execution pool split — trigger evals off the blocking pool, bounded/deadline-aware execution |
| 3.2 | `6cfb2dd8` | `c254d074` | `site_events` volume policy — sampled per-run events, interval summaries, replication pinned |
| 3.3 | (scadaproj) | `cca7f178` + scadaproj `9377fa1` | LocalDb 0.2.0 — dereg cleanup, late-opt-in baselining, byte-budget replication |

**Phase 4 — Verification, adversarial review, docs.** Six parallel `code-reviewer` passes over
the full diff, one per area (site runtime, DCL, comms, site persistence, central SQL, UI/deploy),
each instructed to try to refute the fixes. Confirmed findings landed as targeted follow-ups:

| Area | Commit | What |
|---|---|---|
| Site runtime | `950c54c5` | Recursion-exempt run cap (nested `CallScript` no longer double-gated), atomic detach counter, summary edge cases, per-row event-log fallback |
| DCL | `37f13e2e` | Discard in-flight subscribe results for tags unsubscribed mid-flight; release the orphaned handle |
| Central SQL | `5d075f13` | No client-side audit truncation, insert-first upsert (Site Call Audit), QI-safe filtered-index scripts, honest operator-not-found replies |
| Comms | `fd5e023d` | Consumer-based debug-stream orphan net, foreign-cancel triad, honest `onConnected`, served-row-exact retirement, full-rate reconcile |
| UI/deploy | `e0e4b246` | Honest CLI HTTP timeouts, watermark-complete staleness (3 missed bump sites), phase-2 `PendingDeployment` staging, lock-safe cancellation |
| Site persistence | `56c99c92` + `f689f495` | Required audit DB path on wonder; explicit `MaxBatchSize:64` LocalDb read-page cap; rate-limited observer drop logging; LocalDb 0.2.1 (HLC anchor flush on dereg, bounded 64-message sync inbox) |

## Gate results

- **Baseline:** 7587 tests green at Phase 0 entry; test count never reduced across any phase gate.
- **Live probes (rig):**
  - Stream lifetime forced to 2 minutes — the alarm stream reconnected in **8.5s**, within one
    reconcile tick of the OK completion, `IsLive` correctly reflected the gap (WP1.1).
  - CDC conditional-registration: site-b booted clean on LocalDb 0.2.0 with **30 stale triggers
    dropped** at startup (WP1.3/WP3.3) — confirms the self-heal path fires, not just the steady
    state.
  - S&F due-sweep: `EXPLAIN QUERY PLAN` confirmed index-terminated (WP1.4).
  - Failover drill (`docker/failover-drill.sh`) unaffected by the actor/timeout changes.
- **Adversarial review tally:** ~25 confirmed findings across the six areas, **4 High**, all
  fixed in the Phase 4 commits above. Zero findings deferred as won't-fix.
- **Test-flake root causes (3, all test-side, not production bugs):**
  1. `cfa6acbf` — an assertion on `MarkForwarded` ran before the push it depended on was
     guaranteed to have landed; reordered behind the push.
  2. `c4caebe9` — two dispatcher audit-safety tests asserted an attempt count without
     synchronizing on the async write that produced it; the unsynchronized assertion was removed.
  3. `950c54c5` (embedded) — `ScriptDeadlineAtEnqueueTests`' "no started event" assertion went
     vacuous once WP3.2 flipped `PerRunScriptEvents` to off-by-default (fixed by opting the test
     back in); `ScriptRunLauncherParityTests` widened an `ExpectMsg` window that would have passed
     for any deadline from 1s to 300s, not just the intended one (sharpened to assert the reported
     timeout value AND a tight wall-clock range).
  - **Follow-up recommended, not done here:** a suite-wide sweep for the same
    `AwaitAssert(...)`-then-bare-`Assert` pattern — an `AwaitAssert` that only proves "eventually
    true," followed by a plain assertion that silently inherits its timing slack, is the shape
    behind all three; worth a grep-and-review pass rather than fixing on-demand as flakes surface.

## Residuals register

Deliberately not fixed in this program — each has a stated reason, not an oversight:

1. ~~**DCL unsubscribe-during-reconnect count staleness.** The `37f13e2e` fix discards orphaned
   in-flight results but a per-connection counter can still drift under rapid
   subscribe/unsubscribe churn during a reconnect; needs a per-tag counted set. Low severity,
   cosmetic (a health-report number), deferred.~~ **RESOLVED 2026-08-15** — `DataConnectionActor`'s
   `_totalSubscribed`/`_resolvedTags` scalars are deleted and both health counts are now DERIVED at
   report time from the authoritative per-tag state (`_instancesByTag.Count`, the per-tag counted
   set the residual called for, and `_subscriptionIds.Count`), so no accumulated counter exists to
   drift; this also closes the connection-level-failure case that let resolved climb above total.
   Regression tests: `TagResolutionCounts_*` in `DataConnectionActorBatchTests`.
2. ~~**Per-table `needs_snapshot` in LocalDb.**~~ **RESOLVED 2026-08-15 (LocalDb 0.3.0, scadaproj
   `cea3930`; ScadaBridge pin bump `a5ac309a`, R1).** Bookkeeping schema v3 adds
   `__localdb_snapshot_state` (per-table flags); the database-wide flag survives by design for
   whole-oplog events (cap prune, impossible ack). Wire-compatible by capability negotiation, not
   a version bump: `Handshake.supports_partial_snapshot` + `SnapshotBegin.tables` (empty = all) +
   `SnapshotAck` on the unbounded control lane — a peer that doesn't advertise gets a FULL
   snapshot, which is *required* (an old receiver advances its watermark to the snapshot's as-of
   and would silently discard un-included tables' deltas). A partial snapshot moves no watermark
   on either side; debt clears on receiver ack (crash-safe), clear-on-enqueue kept for old peers.
   Library suite 199/199 incl. old-peer interop; ScadaBridge Host 490/490 on the bump. Adding an
   eleventh table to `ReplicatedTables` on a replicating site now snapshots ONE table instead of
   re-streaming all, both directions.
3. ~~**Event batching per proto message.**~~ **RESOLVED 2026-08-15 (`9b5cb3dd`+`e6842c10`,
   R2).** Additive wire shape, no new RPC: `batching_supported` request flags +
   `SiteStreamEvent.batch` (oneof field 4) carrying `SiteStreamEventBatch`. Negotiation is the
   proto3 default itself and is LOAD-BEARING — an un-negotiated batch frame would parse as
   `EventOneofCase.None` on an old central and vanish silently, so an old central (which cannot
   set the flag) never receives one; an old site ignores the flag and keeps per-event frames.
   Per-subscriber `SiteStreamEventBatcher` downstream of the relay's bounded channel (framing
   only — never delays a lone event; lingers only once a backlog is proven); defaults 100
   events / 25 ms, options-validated strictly < 250 ms to pin the load-test latency budget.
   Measured worst case P99 25.4 ms; 600 queued events → 6 frames. Skew proven both directions
   over real wire bytes. Histogram `scadabridge.site.stream.batch_size` (allowlist-verified).
   Does NOT change the burst ceiling of register row 31 (documented). In-flight it also found
   and fixed a pre-existing `SiteStreamGrpcServer` bug (register row 47).
4. ~~**Deployments page server-side paging + status counts.**~~ **RESOLVED 2026-08-15
   (`35ce1413`, R3).** The Deployments page is database-paged: `QueryDeploymentListPageAsync`
   returns one instance-joined page + the filtered total, `GetDeploymentStatusCountsAsync`
   returns the tiles from one grouped aggregation, and site scoping runs in SQL. Offset paging
   (not keyset) because the numbered jump-to-page pager needs a total, which a cursor cannot
   express, and the retention purge bounds the table — mirrors the Notification Outbox
   precedent; `DeployedAt DESC, Id DESC` tie-break keeps offset pages stable. Whole-table
   `GetAllDeploymentRecordsAsync` deleted with its last caller. Tiles double as the status
   filter; 500 ms trailing-edge debounced search. 34 new tests; `Component-CentralUI.md` updated.
5. ~~**OtOpcUa still pins LocalDb 0.1.3.**~~ **RESOLVED 2026-08-15 (OtOpcUa branch
   `localdb-bump`, commit `a1a258d8`, R4).** Bumped 0.1.3 → 0.2.1 (newest on the feed at the
   time). Pure pin bump — no code change; all 0.2.x additions are additive and unused there.
   Also corrected OtOpcUa's now-false `MaxBatchSize = 16` "row-count vs gRPC 4 MB" rationale in
   5 places (kept at 16 as the DB read-page bound per finding F2) and documented
   `MaxBatchBytes`. Full 57-project suite green bar one unrelated pre-existing flake (register
   row 38). Two OtOpcUa-side findings recorded as register rows 38–39.
6. ~~**Fragile `SandboxTests` timing pin.**~~ **RESOLVED 2026-08-15 (`ca30d17f`+`9fb52153`, R5).**
   Root-caused causally, not by loosening timeouts. The pin was a wall-clock
   `CancellationTokenSource` (100 ms) raced against a fixed workload measured at 298 ms — fails
   outright on faster hardware, and passes *vacuously* in the opposite direction because a
   pre-cancelled token throws before the script body is entered. Replaced with a deterministic
   same-thread cancellation edge plus a negative control; 600 ms of test sleeping removed. The
   deferred sweep ran with it: 3 further instances of the `c4caebe9`/`cfa6acbf` flake class fixed
   (`ScriptRunShedTests` — where `CollectReport` DRAINS the shed counter, so the fix accumulates
   across polls; `SyncCallEmissionEndToEndTests`; `DebugStreamBridgeActorTests`), each with
   fail-before-fix by injected delay + a suppression control; 2 ambiguous candidates cleared with
   product-code evidence. Test-side only (`git diff src/` empty). A different timing-pin class
   (wall-clock sleeps asserting absences) was found and registered, not fixed (register row 40).
7. ~~**Target-scale load test (deferred-work register #25).**~~ **RESOLVED 2026-08-15.** The
   follow-on validation ran: 10 sites x 500 instances x 75 tags = **375,000 live tag
   subscriptions**, 37,518 updates/s achieved vs 37,500 nominal, **45,021,375 updates** over a
   20-minute steady-state window. The moved ceilings hold with room to spare — end-to-end tag
   latency **P99 4.57 ms**, zero dropped events at 100 live subscribers, health report
   collect+ingest **P99 0.31 ms**, debug view **P99 2.19 ms** with no timeouts, CPU **2.9% of the
   box**. 11 clean passes, 1 pass with a caveat (F1: 20 minutes with zero gen-2 collections cannot
   fully settle the leak question; the 1-hour run would), 0 failures. Register **row 50** (S&F
   drain rate + per-subscriber stream backpressure) was absorbed into the same run and closed with
   it — S&F drains at **3,533 msg/s**, and a stalled subscriber costs healthy peers **nothing**
   (100.00%, zero drops). Results: `docs/plans/2026-08-15-target-scale-load-test-results.md`;
   design: `docs/plans/2026-08-15-target-scale-load-test-design.md`; harness:
   `tests/ZB.MOM.WW.ScadaBridge.LoadHarness/`. Residual: WP-4's `[xc-3]`/`[xc-4]`/`[xc-5]`/`[xc-8]`
   are central-cluster/browser concerns outside this harness and remain open scope.
8. ~~**Playwright 14 pre-existing env failures.**~~ **RESOLVED 2026-08-15 (`014038fb`, R6).**
   All 14 triaged to root cause: every one was **selector drift** from two merged Central UI
   refactors (`9e243493` density sweep moved `.btn-sm` off buttons onto `btn-group-sm` groups and
   re-cast SiteForm node sections as cards; `a506b19d` moved TemplateEdit modals to the global
   DialogService host) — no stale rig data, no timing bug, no app defect among the 14. Suite now
   **0 failed / 172 passed / 1 honestly-skipped** (SmtpEdit test self-skips: SMTP config rows
   cannot be created or deleted from the management surface, UI-only). Runtime halved (10m → 4.5m
   — each failure had been burning a 30 s locator timeout). All 49 `.btn-sm` selectors audited
   individually rather than swept; the 35 still-correct ones left alone with comments. One stray
   test area deleted from the rig via CLI. Three genuine app bugs found live during triage —
   recorded as register rows 43–45 (import-session slot leak, dead unlock rate limiter, shared
   DbContext in outbox dispatch), plus a CLI query oddity as row 46 — found, not fixed, per the
   program's scope rules.
9. ~~**`site_events` retention purge still oplog-visible.**~~ **RESOLVED 2026-08-15 (documented
   by design, `9d2834e3`, R7).** The CDC capture of retention/cap deletes is correct and unchanged
   — there is deliberately no purge-exemption path. The operator-facing gap is closed three ways:
   `docs/deployment/topology-guide.md` → *Reading the replication backlog* (burst shape, 24 h
   cadence anchored to active-node process start, healthy-vs-faulty signature),
   `Component-SiteEventLogging.md` → Storage, and `Component-HealthMonitoring.md`, which gained
   the two previously-undocumented `LocalDbReplicationConnected`/`LocalDbOplogBacklog` metric
   rows. `EventLogPurgeService` now emits one Information line correlating the spike to the purge
   on a replication-enabled node (both the retention and the storage-cap delete paths), via a
   Host-supplied `SiteEventLogReplicationCheck` delegate sourced from the single
   `SiteLocalDbSetup.ReplicationIsConfigured` predicate. No behavior change; 5 unit tests.

## Docs propagated

Component docs (`DataConnectionLayer`, `SiteRuntime`, `Communication`, `AuditLog`,
`SiteEventLogging`, `StoreAndForward`, `NotificationOutbox`, `SiteCallAudit`, `TemplateEngine`,
`DeploymentManager`, `CentralUI`, `ConfigurationDatabase`), CLAUDE.md Key Design Decisions
(stream-completion reconnect, required site audit DB path, CLI HTTP timeout honesty, bulk
`DeploySiteAsync`, LocalDb 0.2.1), and `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`
(amended for the `MaxBatchSize:64` read-page pin superseding the "retired, left at defaults" note).
No components added/removed; README component table unchanged. Cross-reference sweep found no
stale live references to `ScriptExecutionActor`/`AlarmExecutionActor` or `UX_AuditLog_EventId`
outside historical plan/known-issue records, after correcting four residual mentions in
`docs/requirements/Component-SiteRuntime.md` left over from the WP3.1 doc pass.