This commit is contained in:
Joseph Doherty
@@ -28,6 +28,11 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
|
||||
| 32 | **`QueueDepthGaugeTests` full-suite flake — shared static gauge (load-test finding F4)** | `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/QueueDepthGaugeTests.cs`; surfaced by the row-25 verification runs (`docs/plans/2026-08-15-target-scale-load-test-results.md` F4) | Pre-existing, test-only: fails when the full suite runs concurrently, passes in isolation — the gauge under test is process-global static state shared across test classes. Zero `src/` involvement; needs test isolation (per-test meter/collector instance), not a product fix. | Next test-infra session, or the first time it fails a CI/full-suite run that matters |
|
||||
| 33 | **`LocalDbOplogBacklog` has no threshold, KPI history, or dashboard treatment** | Found by R7 (`docs/plans/2026-08-15-residual-remediation-plan.md`) while documenting the purge burst | `SiteAuditBacklog` drives a configurable threshold warning on the site tile, but the LocalDb backlog is reported and Prometheus-exported with nothing consuming it for alerting or trending — it is not among the metrics `SiteHealthKpiSampleSource` samples into KPI history, so the purge-burst spike R7 documents cannot be reviewed after the fact; an operator must be watching live. Candidate: add it to the KPI sample source and/or a sustained-backlog threshold badge. Pairs naturally with the R1 per-table-snapshot work, which changes snapshot/backlog behavior. | Next health-dashboard session, or first operator question about a backlog spike they didn't see live |
|
||||
| 34 | **`site_events` purge burst is unbounded in the oplog and unmeasured at scale** | Found by R7 | Retention deletes are sliced at 1000 rows/statement, but the whole expired set is deleted in one tick, so a replicated site's oplog gains one row per deleted event with no pacing — a first purge after enabling replication (or after a long low-purge period) queues the entire batch at once. The wire path is bounded (`MaxBatchBytes`), but the oplog write itself is not paced, and no measurement exists for batch size or drain time at target scale. Candidate: measure on the rig or in the load harness; consider a per-tick delete ceiling if drain time is material. | Before enabling replication on a site with a large existing `site_events` backlog, or the next load-harness session |
|
||||
| 35 | **`DeploymentRecords` index does not cover the deployment-list sort key** | Found by R3 (`35ce1413`) | The table has a single-column `IX DeployedAt`; the new list query orders `DeployedAt DESC, Id DESC` and joins `Instance`, so SQL Server sorts per page. A composite `(DeployedAt DESC, Id DESC)` index would serve the ordering directly — deliberately not done in R3 because it needs an EF migration and the terminal-record purge bounds the table. *(Low)* | Deployment list measurably slow at a real fleet's retention depth |
|
||||
| 36 | **Deployments free-text search is unindexable by construction** | Found by R3 | Four `LIKE '%term%'` predicates (instance name, deployment id, revision hash, user) — leading wildcards preclude index seeks, so a search is a scan of the join. Acceptable at the purge-bounded row count. Fix if hot: anchor the id/hash legs as prefix matches, or full-text. *(Low)* | Search latency complaints on the Deployments page |
|
||||
| 37 | **Two paged reads of `DeploymentRecords` with different contracts** | Found by R3 | `QueryDeploymentSummariesAsync` (CLI/ManagementActor: offset, no total, no instance join, instance-id-list scoping) and R3's `QueryDeploymentListPageAsync` (UI: total + instance join + site-id scoping) duplicate query logic. Consolidating on one contract — giving the CLI a total, both scoping by site — removes the duplication. Genuinely different contracts today, so not urgent. *(Low)* | Next time either query's shape changes |
|
||||
| 38 | **OtOpcUa: `RoslynVirtualTagEvaluatorTests` racing-clear test is load-fragile** (OtOpcUa repo, not this one) | Found by R4 during the full-suite bump verification | `Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed` fails under full-solution parallel test load, passes 3/3 isolated: its design assumes at most one `ClearCompiledScripts` lands inside one evaluation (so the product's one-shot retry suffices); under contention two can. Same timing-assumption family as the R5 sweep, but lives in OtOpcUa. Test-side fix: bound clear cadence relative to observed evaluation latency, or assert on retry-exhaustion instead of zero failures. | Next OtOpcUa test session, or when it fails a run that matters |
|
||||
| 39 | **OtOpcUa: late-enabled replication cannot baseline pre-existing rows** (OtOpcUa repo, not this one) | Found by R4; documented in OtOpcUa's CLAUDE.md | `LocalDbSetup.OnReady` registers all four tables unconditionally on every driver node, so it can never pass `baselineExistingRows: true` (only correct when gated on replication being configured). Turning replication ON for a long-running default-OFF node converges on subsequent writes only — existing rows are never baselined. ScadaBridge closed this class with conditional registration + boot-time `DeregisterReplicated` self-heal (WP1.3/WP3.3); porting it is a behavior change beyond a pin bump (registered-set pinned by tests, deregistration must be pair-symmetric). Low current impact: site-a was enabled from fresh. | Before enabling replication on an established OtOpcUa node |
|
||||
|
||||
## Resolved (verified against the code 2026-07-10)
|
||||
Rows removed from the Deferred table above once confirmed shipped. Kept here for traceability.
|
||||
|
||||
@@ -108,11 +108,22 @@ Deliberately not fixed in this program — each has a stated reason, not an over
|
||||
3. **Event batching per proto message.** Individual `AttributeValueChanged`/`AlarmStateChanged`
|
||||
events still ride one gRPC message each; batching them is a new wire shape (proto + both
|
||||
client/server), deferred rather than folded into this program's additive-only changes.
|
||||
4. **Deployments page server-side paging + status counts.** Still client-materializes the full
|
||||
list; out of scope for this pass (WP2.5 touched the deploy pipeline, not this specific UI
|
||||
surface).
|
||||
5. **OtOpcUa still pins LocalDb 0.1.3.** A supported skew — 0.1.x peers sync with 0.2.x under the
|
||||
library's wire-compatibility guarantee — not a blocker for this program.
|
||||
4. ~~**Deployments page server-side paging + status counts.**~~ **RESOLVED 2026-08-15
|
||||
(`35ce1413`, R3).** The Deployments page is database-paged: `QueryDeploymentListPageAsync`
|
||||
returns one instance-joined page + the filtered total, `GetDeploymentStatusCountsAsync`
|
||||
returns the tiles from one grouped aggregation, and site scoping runs in SQL. Offset paging
|
||||
(not keyset) because the numbered jump-to-page pager needs a total, which a cursor cannot
|
||||
express, and the retention purge bounds the table — mirrors the Notification Outbox
|
||||
precedent; `DeployedAt DESC, Id DESC` tie-break keeps offset pages stable. Whole-table
|
||||
`GetAllDeploymentRecordsAsync` deleted with its last caller. Tiles double as the status
|
||||
filter; 500 ms trailing-edge debounced search. 34 new tests; `Component-CentralUI.md` updated.
|
||||
5. ~~**OtOpcUa still pins LocalDb 0.1.3.**~~ **RESOLVED 2026-08-15 (OtOpcUa branch
|
||||
`localdb-bump`, commit `a1a258d8`, R4).** Bumped 0.1.3 → 0.2.1 (newest on the feed at the
|
||||
time). Pure pin bump — no code change; all 0.2.x additions are additive and unused there.
|
||||
Also corrected OtOpcUa's now-false `MaxBatchSize = 16` "row-count vs gRPC 4 MB" rationale in
|
||||
5 places (kept at 16 as the DB read-page bound per finding F2) and documented
|
||||
`MaxBatchBytes`. Full 57-project suite green bar one unrelated pre-existing flake (register
|
||||
row 38). Two OtOpcUa-side findings recorded as register rows 38–39.
|
||||
6. **Fragile `SandboxTests` timing pin.** Pre-existing, unrelated to this remediation's changes;
|
||||
noted so it isn't mistaken for a regression if it flakes later.
|
||||
7. ~~**Target-scale load test (deferred-work register #25).**~~ **RESOLVED 2026-08-15.** The
|
||||
|
||||
Reference in New Issue
Block a user