docs(register): close rows 27, 28, 32, 33, 35, 40, 54-57, 61, 63, 66, 67 with evidence; add rows 71-77; record the row-42 seven-script audit result

This commit is contained in:
Joseph Doherty
2026-08-15 18:03:27 -04:00
parent 744ff36a7b
commit caf9f45165
+22 -15
View File
@@ -20,22 +20,22 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
| 24 | M8 large-bundle performance hardening | `docs/plans/2026-06-15-stillpending-completion-design.md:106` — "Small follow-ups logged (not blocking): … large-bundle/perf hardening" | Logged as a non-blocking follow-up when M8 shipped and never given an artifact: **no plan, no task entry, no perf/load test exists** (`tests/…Transport.Tests/Import/BundleImporterLoadTests.cs` is a `LoadAsync` unit suite despite the name). No measured problem; the only sizing controls in place are the 5-minute CLI transport timeout, `LineDiffer`'s `MaxInputLines`=4000 summary-only cap, and `MaxConcurrentImportSessions`=8. | First real bundle that times out, exhausts memory, or makes the import wizard's diff step unusable |
| 25 | ~~**Phase-8 WP-4 target-scale load test** (10 sites x 500 instances x 75 tags = 37,500 subscriptions/site, 375,000 total)~~ **RESOLVED 2026-08-15 — run, measured, evidenced.** The 107-byte stub is replaced by a real full-scale run: **10 x 500 x 75 = 375,000 live tag subscriptions**, 37,518 tag updates/s achieved against 37,500 nominal (100.0%), **45,021,375 updates offered** over a 20-minute steady-state window on an M4 Pro / 14-core / 48 GB host with the 8-node `docker/` rig still running. Headline numbers: tag update latency **P50 0.88 ms / P99 4.57 ms / max 37.41 ms** end-to-end (DCL boundary to stream subscriber, the emit instant carried verbatim through production code); **0 events dropped** at 100 live subscribers; health report collect+ingest **P99 0.31 ms** with 10/10 sites tracked; debug view snapshot under load **P99 2.19 ms, 0 timeouts**; 500 instances deployed to a site in **2.6 s**; CPU **2.9% of the box**; working-set slope **+8.83 MB/min**. **11 clean passes, 1 pass with a caveat (F1: 20 min with zero gen-2 collections cannot fully settle the leak question — the 1-hour run would), 0 failures.** Harness: `tests/ZB.MOM.WW.ScadaBridge.LoadHarness/` (standalone Exe — the `Category=Performance` trait enables a filter but does NOT exclude by default, so a 20-minute test could not live in `PerformanceTests`) plus a CI-scale smoke `[Fact]` (`PerformanceTests/TargetScale/TargetScaleHarnessSmokeTests.cs`). Real: per-site ActorSystem + LocalDb, the real DCL via the documented `DataConnectionFactory.RegisterAdapter` seam, real `InstanceActor`s, real `SiteStreamManager`, real `StreamRelayActor` + production-capacity bounded channel, real S&F, real health collector/aggregator; only the socket hops are stood in for. **Results: `docs/plans/2026-08-15-target-scale-load-test-results.md`; design + deviations + thresholds: `docs/plans/2026-08-15-target-scale-load-test-design.md`.** | as before | Closed. Residual scope: `[xc-3]` site-event-log retention volume, `[xc-4]` audit-vs-central degradation, `[xc-5]` large-template flattening and `[xc-8]` UI responsiveness are NOT covered by this harness (central-cluster/browser concerns) and remain open WP-4 scope — see results doc SS6. | Closed 2026-08-15. |
| 26 | Ipsen MES MoveIn tail: leak-test (`-LT`) receivers + routing, PLC-output-flag writes, `Z28062` BTDB data completeness | `docs/plans/2026-06-16-ipsen-mes-movein.md:409` ("Out of scope (future)"); design `2026-06-16-ipsen-mes-movein-design.md:58-60, :196-198` | Customer-site scope, not a platform gap. `-LT` routing needs an MES-receiver child + Galaxy reference that do not exist on the reactor template (any `-LT`/unknown suffix returns `WasSuccessful=false` with an "unsupported side/target" message by decision); `MoveInComplete`/`Successful`/`ErrorText` are **PLC-owned** by locked decision, so ScadaBridge deliberately does not write them; `Z28062` completeness is an operational data fix, not code. Note the separate alarm-status path already handles the suffix — `_LT` is stripped before side-scoping (`2026-06-30-mes-alarm-status-api.md:158`). | Ipsen creates the leak-test receiver + Galaxy reference, or asks ScadaBridge to own the PLC-output flags — otherwise a **candidate won't-do** (`[PERM]`) at the next Ipsen scope review |
| 27 | External-system per-system retry config (`MaxRetries`/`RetryDelay`) never reaches sites, and has no CLI/management surface | Found live 2026-08-01 (rig session, #11 gRPC live checks) | Two stacked gaps: (a) `ExternalSystemArtifact` (Commons) carries `TimeoutSeconds` but NOT `MaxRetries`/`RetryDelay`, and the site `external_systems` table has no such columns — so a centrally-configured retry policy is silently ignored on sites; every cached call buffers with the S&F default (`DefaultMaxRetries` 50 × `DefaultRetryInterval` 30s ≈ 25 min to park). (b) `Create/UpdateExternalSystemCommand` don't expose the fields either — the only way to set them today is a direct DB edit of `ExternalSystemDefinitions`. Transport bundles DO carry them (arch-review 05 "ES retry config"), which masks the gap in export/import round-trips. Fix is additive: extend the artifact + site schema + apply path, and add `--max-retries`/`--retry-delay` to the CLI. | First operator who tunes retry policy on an external system and expects site cached calls to honor it |
| 28 | Health-dashboard "Trigger failover" confirm dialog's confirm button is labeled **"Delete"** | Found live 2026-08-01 (rig session, #11 TriggerSiteFailover check) | The DialogService confirmation host's default destructive-action label leaks through — the dialog copy is correct but the red confirm button says "Delete" for a failover. One-line fix: pass an explicit confirm label ("Fail over") at the Health-dashboard call site (and audit other confirm-dialog call sites for the same default). | Next Central UI session |
| 27 | ~~External-system per-system retry config (`MaxRetries`/`RetryDelay`) never reaches sites, and has no CLI/management surface~~ **RESOLVED 2026-08-15 (`41a5ecf1`).** The gap was transport + persistence only — the consumption end already worked (`ExternalSystemClient.CachedCallAsync` already passed `system.MaxRetries`/`RetryDelay` to `EnqueueAsync` with 0-as-unset, and the Central UI form already collected both fields, inert until now with defaults 3×5s that will take effect on next deploy). Shipped: artifact fields, additive `site_command.proto` fields 7/8 (regen via `docker/regen-proto.sh`, csproj left clean), site `external_systems` columns `max_retries`/`retry_delay_ms` as REAL COLUMNS — not a JSON bag; verified safe against the AckTime precedent in the LocalDb source (capture triggers are DROP-then-CREATE from `PRAGMA table_info` each boot, and the fail-closed handshake digest turns a mixed-version pair into a loud sync refusal, no new constraint since both-nodes-together is already the standing pair rule; `timeout_seconds` set the additive-column precedent on this exact table) — apply path, and CLI `--max-retries`/`--retry-delay-seconds` (+`--retry-delay` alias). Bonus fix: `ArtifactDiff.CompareExternalSystem` never compared the retry fields, so a bundle import changing a retry budget previewed as "Identical". Fail-before verified on the site round-trip + apply tests; 12 suites green. | Found live 2026-08-01 (rig session, #11 gRPC live checks) | Closed 2026-08-15. | — |
| 28 | ~~Health-dashboard "Trigger failover" confirm dialog's confirm button is labeled **"Delete"**~~ **RESOLVED 2026-08-15 (`0db253b5`).** Structural, not a missing argument: `IDialogService.ConfirmAsync` had NO label parameter — `DialogHost` derived the verb from `danger` alone (`danger ? "Delete" : "Confirm"`), so no call site could ever have passed one. New optional `confirmLabel`; `danger` now selects styling only. Full audit of all 34 `ConfirmAsync` call sites (the only confirmation path — no `window.confirm`, no per-page modals): NINE were wrongly labeled "Delete" — both failover buttons, secured-write **Approve** and **Reject**, instance Disable, and four Discard actions — all now name their action; 18 genuine deletes + 7 non-danger sites verified correct. The `danger→"Delete"` fallback deliberately kept: forcing an explicit label would churn ~20 correct sites, and a missed site would silently degrade to "Confirm", detectable only by the rig-dependent Playwright suite — contract pinned in the interface XML doc, Component-CentralUI.md, and a fallback test instead. Gotcha recorded in the tests: adding an optional parameter silently un-matches existing 3-arg NSubstitute stubs (returns null Task → NRE). Follow-ups: rows 75 (Approve's red styling), and the bUnit real-dialog deadlock hazard documented in `FailoverConfirmLabelTests`. | Found live 2026-08-01 (rig session, #11 TriggerSiteFailover check) | Closed 2026-08-15. | — |
| 29 | **1-hour sustained run to settle the memory-slope ambiguity (load-test finding F1)** | `docs/plans/2026-08-15-target-scale-load-test-results.md` F1 (Low) | The row-25 20-minute run PASSED the working-set threshold (+8.83 MB/min < 20) but recorded **zero gen-2 collections across 45M events**, so the +19.52 MB/min managed-heap drift on an uncompacted gen-2 is ambiguous — the heap demonstrably sawtooths, but 20 min cannot distinguish "gen-2 not yet collected" from a slow leak. Not a defect finding; an evidence gap. The harness exists (`tests/ZB.MOM.WW.ScadaBridge.LoadHarness/`) and the full 1-hour protocol is one documented command (results doc §7) — deferred only because the measured window was scaled for practicality. | Before production go-live at target scale, or first field report of unexplained site-node memory growth |
| 30 | **WP-4 residual scope: `[xc-3]` site-event-log retention volume, `[xc-4]` audit-load-vs-central degradation, `[xc-5]` large-template flattening time, `[xc-8]` UI responsiveness under load** | `docs/plans/2026-08-15-target-scale-load-test-results.md` §6 + row 25 closure note | The row-25 harness is a site-runtime rig; these four WP-4 sub-criteria are central-cluster/browser concerns it deliberately does not cover. Recorded so the row-25 closure never reads as "WP-4 fully done" — phase-8 WP-4 is closed for the site-runtime data path only. | Before declaring phase-8 WP-4 complete, or production go-live at target scale |
| 31 | **Unpaced-burst absorption is bounded by the shared publish stage, not per-subscriber buffers (load-test finding F3 caveat)** | `docs/plans/2026-08-15-target-scale-load-test-results.md` F3 | Measurement-backed design note, no defect: slow-subscriber isolation is structural and total (a stalled reader loses events only in its own bounded `DropOldest` channel; healthy subscribers delivered 100.00%), but the publish `Source.ActorRef` **upstream** of the `BroadcastHub` is shared, so an unpaced burst degrades every subscriber equally — that stage, not per-subscriber buffering, is the burst ceiling. | First field report of event loss on *healthy* subscribers during a burst, or any redesign of the site stream publish stage |
| 32 | **`QueueDepthGaugeTests` full-suite flake — shared static gauge (load-test finding F4)** | `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/QueueDepthGaugeTests.cs`; surfaced by the row-25 verification runs (`docs/plans/2026-08-15-target-scale-load-test-results.md` F4) | Pre-existing, test-only: fails when the full suite runs concurrently, passes in isolation — the gauge under test is process-global static state shared across test classes. Zero `src/` involvement; needs test isolation (per-test meter/collector instance), not a product fix. | Next test-infra session, or the first time it fails a CI/full-suite run that matters |
| 33 | **`LocalDbOplogBacklog` has no threshold, KPI history, or dashboard treatment** | Found by R7 (`docs/plans/2026-08-15-residual-remediation-plan.md`) while documenting the purge burst | `SiteAuditBacklog` drives a configurable threshold warning on the site tile, but the LocalDb backlog is reported and Prometheus-exported with nothing consuming it for alerting or trending — it is not among the metrics `SiteHealthKpiSampleSource` samples into KPI history, so the purge-burst spike R7 documents cannot be reviewed after the fact; an operator must be watching live. Candidate: add it to the KPI sample source and/or a sustained-backlog threshold badge. Pairs naturally with the R1 per-table-snapshot work, which changes snapshot/backlog behavior. | Next health-dashboard session, or first operator question about a backlog spike they didn't see live |
| 32 | ~~**`QueueDepthGaugeTests` full-suite flake — shared static gauge (load-test finding F4)**~~ **RESOLVED 2026-08-15 (`65b19516`).** Reproduced first (2 failures in 5 parallel runs — the gauge read a sibling class's provider). The register's suggested fix (per-test meter) was rejected on inspection: the slot is process-global by product design and 3 of the 6 facts assert the slot-ownership semantics themselves (compare-and-clear identity, stop/restart re-seed), so per-test instances would change what they prove. Correct isolation is mutual exclusion — the three `StartAsync`-calling classes now share xunit collection `QueueDepthGaugeCollection` (with a `[CallerFilePath]`-anchored source-scan drift guard that names an offending file), the other eight classes stay parallel, and `parallelizeTestCollections` was flipped back ON so the fix is exercised rather than masked. Negative control: dropping the attribute fails the guard AND resurfaces the original flake in the same run. 8/8 green parallel runs. | `tests/…StoreAndForward.Tests/QueueDepthGaugeTests.cs`; surfaced by the row-25 verification runs | Closed 2026-08-15. | — |
| 33 | ~~**`LocalDbOplogBacklog` has no threshold, KPI history, or dashboard treatment**~~ **RESOLVED 2026-08-15 (`8da60f57`).** Both halves shipped: sampled into KPI history via `SiteHealthKpiSampleSource` (conditionally — 12 samples unreplicated / 14 replicated, so a series gap truthfully means "not replicated") plus a backlog trend chart on the site health panel, and a SUSTAINED threshold badge (`LocalDbBacklogThreshold` 1000 held for `LocalDbBacklogSustainedFor` 15 min, anchored by `LocalDbBacklogAboveSince` in the aggregator) — a bare instantaneous threshold would fire on every replicated site every day from the documented purge burst. Null backlog is a deliberate third branch: it leaves the anchor unchanged (neither evidence of a drain nor of a backlog), so unreplicated sites can never anchor and can never be badged. NOTE the row's premise was false — the `SiteAuditBacklog` "configurable threshold badge" it told us to mirror was described twice in `Component-HealthMonitoring.md` but NEVER BUILT; both stale doc claims corrected, and the new badge was modeled on the real Metrics-stale badge instead. | Found by R7 while documenting the purge burst | Closed 2026-08-15. | — |
| 34 | **`site_events` purge burst is unbounded in the oplog and unmeasured at scale** | Found by R7 | Retention deletes are sliced at 1000 rows/statement, but the whole expired set is deleted in one tick, so a replicated site's oplog gains one row per deleted event with no pacing — a first purge after enabling replication (or after a long low-purge period) queues the entire batch at once. The wire path is bounded (`MaxBatchBytes`), but the oplog write itself is not paced, and no measurement exists for batch size or drain time at target scale. Candidate: measure on the rig or in the load harness; consider a per-tick delete ceiling if drain time is material. | Before enabling replication on a site with a large existing `site_events` backlog, or the next load-harness session |
| 35 | **`DeploymentRecords` index does not cover the deployment-list sort key** | Found by R3 (`35ce1413`) | The table has a single-column `IX DeployedAt`; the new list query orders `DeployedAt DESC, Id DESC` and joins `Instance`, so SQL Server sorts per page. A composite `(DeployedAt DESC, Id DESC)` index would serve the ordering directly — deliberately not done in R3 because it needs an EF migration and the terminal-record purge bounds the table. *(Low)* | Deployment list measurably slow at a real fleet's retention depth |
| 35 | ~~**`DeploymentRecords` index does not cover the deployment-list sort key**~~ **RESOLVED 2026-08-15 (`be29bf47`).** Migration `20260815165452_AddDeploymentRecordListSortIndex` + hand-written `docs/plans/sql/AddDeploymentRecordListSortIndex.sql`: composite `IX_DeploymentRecords_DeployedAt_Id (DeployedAt DESC, Id DESC)`; the old single-column index DROPPED as strictly redundant (`Id` is the clustered PK and was already in the old index's leaf as the row locator — same leaf width, now a declared ordered key; keeping it would cost a second index on every insert/status transition for no read benefit). Serves the three reads sharing the key (`QueryDeploymentListPageAsync`, `QueryDeploymentSummariesAsync`, `GetCurrentDeploymentStatusAsync`). `Id` in the key is also a CORRECTNESS matter: a bulk site deploy stamps many rows in one `DeployedAt` instant, so without the tie-break keyset pages can repeat or drop rows. Recorded caveat: the optimizer may already have served the ordering via a backward scan of the old index, so the "sorts per page" premise was possibly optimizer-dependent — the composite makes it deterministic and self-documenting; an actual plan capture against the rig would quantify the before/after. The ConfigurationDatabase suite exercises the descending composite as real DDL via SQLite `EnsureCreated()`, proving provider portability. | Found by R3 (`35ce1413`) | Closed 2026-08-15. | — |
| 36 | **Deployments free-text search is unindexable by construction** | Found by R3 | Four `LIKE '%term%'` predicates (instance name, deployment id, revision hash, user) — leading wildcards preclude index seeks, so a search is a scan of the join. Acceptable at the purge-bounded row count. Fix if hot: anchor the id/hash legs as prefix matches, or full-text. *(Low)* | Search latency complaints on the Deployments page |
| 37 | **Two paged reads of `DeploymentRecords` with different contracts** | Found by R3 | `QueryDeploymentSummariesAsync` (CLI/ManagementActor: offset, no total, no instance join, instance-id-list scoping) and R3's `QueryDeploymentListPageAsync` (UI: total + instance join + site-id scoping) duplicate query logic. Consolidating on one contract — giving the CLI a total, both scoping by site — removes the duplication. Genuinely different contracts today, so not urgent. *(Low)* | Next time either query's shape changes |
| 38 | **OtOpcUa: `RoslynVirtualTagEvaluatorTests` racing-clear test is load-fragile** (OtOpcUa repo, not this one) | Found by R4 during the full-suite bump verification | `Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed` fails under full-solution parallel test load, passes 3/3 isolated: its design assumes at most one `ClearCompiledScripts` lands inside one evaluation (so the product's one-shot retry suffices); under contention two can. Same timing-assumption family as the R5 sweep, but lives in OtOpcUa. Test-side fix: bound clear cadence relative to observed evaluation latency, or assert on retry-exhaustion instead of zero failures. | Next OtOpcUa test session, or when it fails a run that matters |
| 39 | **OtOpcUa: late-enabled replication cannot baseline pre-existing rows** (OtOpcUa repo, not this one) | Found by R4; documented in OtOpcUa's CLAUDE.md | `LocalDbSetup.OnReady` registers all four tables unconditionally on every driver node, so it can never pass `baselineExistingRows: true` (only correct when gated on replication being configured). Turning replication ON for a long-running default-OFF node converges on subsequent writes only — existing rows are never baselined. ScadaBridge closed this class with conditional registration + boot-time `DeregisterReplicated` self-heal (WP1.3/WP3.3); porting it is a behavior change beyond a pin bump (registered-set pinned by tests, deregistration must be pair-symmetric). Low current impact: site-a was enabled from fresh. | Before enabling replication on an established OtOpcUa node |
| 40 | **Wall-clock sleeps used as synchronization in tests (absence-assertion class)** | Found by the R5 sweep (`9fb52153`) — a different class from the one swept, deliberately left unfixed | Tests that sleep, then assert an *absence*: `ScriptDeadlineAtEnqueueTests.cs:105,128` (two `Thread.Sleep(2s)`, with `Assert.Equal(0, scheduler.DetachedThreadCount)` depending on the sleep out-racing the watchdog — worst offender), `DeploymentWarmThenGateTests.cs:117`, `SiteAlarmLiveCacheServiceTests.cs:195,216`, `DataConnectionActorTests.cs:435` (`Task.Delay(200)` to "settle" after an `AwaitCondition`). These fail toward false-pass rather than flake, so they don't trip CI — but each is a timing pin that weakens what the test proves. Fix pattern: replace the sleep with an explicit observable edge (or assert on the positive event with a bounded wait plus a negative control). | Next test-infra session, or the first time one of these areas regresses without its test catching it |
| 40 | ~~**Wall-clock sleeps used as synchronization in tests (absence-assertion class)**~~ **RESOLVED 2026-08-15 (`7a7d345b`).** All six listed sites replaced with observable edges + positive controls, strengthening proven by stash-based negative controls: `ScriptDeadlineAtEnqueueTests:128` now pins the watchdog via a genuinely-wedged positive control (armed strictly later, self-releasing) + per-script absence assertion + detach counter pinned at 1 — **disabling the entire stuck-script watchdog passed the OLD test 2/2 and fails the new one**; `:105` now proves the run was still queued when its budget expired via `QueueDepth` edges (the actual design claim — the old sleep asserted nothing about it); `DeploymentWarmThenGateTests:117` (+2 sibling `Task.Delay(500)`s) — the empty-site zero is now the manager's own observation (phantom-startup control: old passed 2/2, new fails); `SiteAlarmLiveCacheServiceTests:195,216` poll a new subscription-cancellation observable across 3 linger windows, each closing with a positive control (control: linger cancellation disabled + viewer guard dropped — `Resubscribe_Within_Linger` old passed 2/2, new fails; honest caveat: the other case's old later assertion did already catch that break, so its gain is the observable + control, not a false-pass fix); `DataConnectionActorTests:435` waits for the Connected health publication so the subscribe provably runs against Connected (the sleep guarded no assertion — determinism gain only). Bonus: the de-padded schedule exposed a LATENT race in `InstanceActorChildAttributeRaceTests` (~1-in-6 under load, A/B'd 14/14 green on the unmodified tree so pre-existing) — fixed with a bounded-deadline resolution retry; pattern recorded as row 77. | Found by the R5 sweep (`9fb52153`) | Closed 2026-08-15. | — |
| 41 | **LocalDb (scadaproj) sync-session tests flake under parallel load** (scadaproj repo, not this one) | Found by R1; confirmed pre-existing at base `f627cee` (1 failure in 3 full runs) | `SyncSessionTests.Handshake_DigestMismatch_FailsClosed` and `BatchBudgetTests.Pump_RowLargerThanTheBudget_IsSentAloneRatherThanStallingTheStream` intermittently time out. Root cause looks structural, not a timing pin: on a handshake fault, `RunAsync`'s catch cancels the writer loop, which can kill the session's OWN Handshake before it transmits — the healthy peer then waits out its full 15 s cap for a message that never comes. Product-adjacent (the same early-cancel could delay a real peer's fail-closed verdict), so worth a look beyond the tests. | Next LocalDb session |
| 42 | **`build/push.sh`-class publish bug across the family repos** | Found by R1 the hard way (incident: 25 unrelated packages briefly published to the feed, deleted + verified same session; script fixed in scadaproj `0038d46`) | The bug shape: `dotnet pack -c Release -o ./artifacts` with no project argument and no `cd`, so the script packs the CALLER'S CWD. Fixed in LocalDb's script (cd to own repo root, name the solution, absolute output path). Peer repos' publish scripts (MxAccessGateway, HistorianGateway, OtOpcUa, others in scadaproj) should be checked for the same shape — the recovery only worked because the token happened to carry delete rights. | Before anyone runs a family publish script from outside its repo, or next umbrella-repo session |
| 42 | **`build/push.sh`-class publish bug across the family repos** | Found by R1 the hard way (incident: 25 unrelated packages briefly published to the feed, deleted + verified same session; script fixed in scadaproj `0038d46`) | The bug shape: `dotnet pack -c Release -o ./artifacts` with no project argument and no `cd`, so the script packs the CALLER'S CWD. Fixed in LocalDb's script (cd to own repo root, name the solution, absolute output path). **AUDITED 2026-08-15 (read-only, this wave): SEVEN scadaproj scripts still carry the exact shape** — `ZB.MOM.WW.Secrets/build/push.sh`, `ZB.MOM.WW.Theme/build/{pack,push}.sh`, `ZB.MOM.WW.Audit/build/{pack,push}.sh`, `ZB.MOM.WW.Auth/build/{pack,push}.sh` (all `dotnet pack -c Release -o ./artifacts` with no project argument and no `cd`, plus a CWD-relative `nuget push "./artifacts/*.nupkg"` that would publish whatever sits in the caller's artifacts dir). MxAccessGateway's `scripts/pack-clients.ps1` is SAFE (`$PSScriptRoot`-anchored, explicit csproj paths); HistorianGateway and OtOpcUa have no pack scripts. Fix = copy LocalDb's `0038d46` pattern into the seven scripts; lives in the scadaproj repo, outside this wave's edit scope. | Before anyone runs a family publish script from outside its repo, or next umbrella-repo session — **now with the exact seven-script fix list ready** |
| 43 | ~~**Central UI import wizard leaks a `BundleSession` slot on every abandoned import — imports lock out for 30 min**~~ **RESOLVED 2026-08-15 (`70838982`).** New `IBundleImporter.Abandon(sessionId)` does the same zero-then-remove release `ApplyAsync` already did (T-007 discipline), idempotent (double/unknown/expired ids are no-ops) and only ever given the caller's own random-GUID id, so it cannot evict a concurrent circuit's session. Wired into EVERY abandonment path: wizard new-upload, `TryLoadAsync` pre-overwrite, Back-to-Upload/`ResetSessionState`, 3-strike lockout reset, `Dispose` — plus two CLI leaks the original row didn't know about: `transport preview` released nothing at all (every preview burned a slot for 30 min) and `transport import` stranded the session on pre-apply aborts; both now release in a `finally`. `BundleSessionEvictionService` is documented as the TTL backstop, not an abandonment path. Fail-before-fix stash-verified (3 of 4 new bUnit tests fail pre-fix). 10 new tests. | Found live by R6 | Closed 2026-08-15. | — |
| 44 | ~~**`BundleUnlockRateLimiter` is dead code — the documented per-IP unlock throttle is not enforced**~~ **RESOLVED 2026-08-15 (`5ab26a42`).** `BundleImporter.LoadAsync` now registers every unlock attempt against `MaxUnlockAttemptsPerIpPerHour` (10). Ordering is load-bearing: AFTER the per-bundle 3-strike check (hammering a locked bundle spends no hourly budget) and BEFORE the decrypt (a throttle denial burns no per-bundle strike, so the throttle can't be abused to lock out a legitimate import); successful unlocks count too. Denial = `BundleUnlockThrottledException` (placed in Commons so the management surface returns a curated message). Client key: Central UI via a scoped `CircuitClientAddressProvider` snapshotting the IP at circuit establishment; CLI via additive `ManagementEnvelope.ClientIp`. No attributable IP falls into one shared `<unattributed>` bucket — stricter, never a bypass; per-client isolation is only real behind a ForwardedHeaders-trusted proxy (same caveat as the login throttle, documented). Fail-before-fix verified (4 of 6 new tests fail with the limiter neutralised; 2 negative controls). | Found by R6 | Closed 2026-08-15. | — |
| 45 | ~~**Notification Outbox dispatcher shares one `DbContext` across up to 4 parallel deliveries**~~ **RESOLVED 2026-08-15 (`9e769735`).** `_adaptersCache`/`_adaptersScope`/`ResolveAdapters()` deleted; the typed adapter is resolved inside the per-delivery DI scope `DeliverGatedAsync` already opens (static `ResolveAdapter` preserving DI last-wins + the null-adapter park path), so each concurrent delivery gets its own adapter and `DbContext`. Nothing shared was lost: the Email adapter builds a fresh SMTP client per attempt from an injected factory, the SMS adapter draws from the singleton `IHttpClientFactory`, and CentralUI's adapter-gated Type selector (`NotificationChannelCatalog`) is an independent scoped consumer, untouched. Fail-before-fix stash-verified (simulated DbContext throwing EF's exact second-operation exception under a rendezvous-guaranteed overlap: 1 of 2 rows persisted pre-fix; 1-vs-4 distinct adapter instances). Root cause of note: the bug was INTRODUCED by code-review finding NotificationOutbox-006's caching "fix" (2026-05-28), whose own regression test pinned the broken behavior and predates bounded parallel delivery — that finding is marked SUPERSEDED. NotificationOutbox 153, AuditLog 383 green. | Found by R6; observed live on rig central-b | Closed 2026-08-15. | — |
@@ -47,20 +47,27 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
| 51 | ~~**`BundleLockedException` reaches the CLI as "An internal error occurred"**~~ **RESOLVED 2026-08-15 (`b449fe20`).** Relocated to `Commons/Types/Transport/` beside `BundleUnlockThrottledException` (row 44's pattern); both consumers already carried the namespace, so no call-site edits. `ManagementActor.HandlePreviewBundle`/`HandleImportBundle` now rethrow it as `ManagementCommandException`, so CLI and UI print the lockout reason instead of a correlation id. Fault-mapping tests mirrored from row 44 on both handlers. Follow-up in the same diff (`73f518d6`): deleted the dead `BundleUnlockRateLimitedException` (superseded by row 44, zero references — three similarly-named unlock exceptions with one inert was a live trap). | Found while fixing row 44 | Closed 2026-08-15. | — |
| 52 | ~~**The unlock throttle's counters are per-node and per-process**~~ **RESOLVED 2026-08-15 (`677263af`).** The trailing-hour window now lives in the central MS SQL `BundleUnlockAttempt` table — the HA store both central nodes already share, so no new infra, port or consensus. `MaxUnlockAttemptsPerIpPerHour` is a true fleet cap and survives a node restart; config keys, semantics and the rig's 500 override are unchanged, as are all row-44 behaviours (lockout checked first so locked bundles spend no budget; denials burn no strike and are not recorded; `<unattributed>` bucket; attempts-not-failures). Three recorded trades: **fails open** (store error → Warning naming client + cap, attempt permitted — availability of legitimate imports outranks a best-effort cap over an already-authenticated Administrator-gated surface; a process-local fallback was considered and rejected as it would still block an operator's legitimate 11th import); **approximate** (count-then-insert is not atomic across nodes; overshoot bounded by in-flight count, self-corrects within the window — row 64); **opportunistic prune on the unlock path** (CAS-claimed ≤ every 15 min per process, ≤1000 rows, retains 2× the window so a prune can't race a count — row 65) rather than a new background service. EF migration `20260815154803_AddBundleUnlockAttemptTable` + hand-written production script `docs/plans/sql/AddBundleUnlockAttemptTable.sql`. Integration test proves two limiter instances over one shared store enforce one combined window, restart durability, and store-failure fail-open. | Row 44 implementation note; documented in `docs/components/Transport.md` | Closed 2026-08-15. | — |
| 53 | ~~**`MaxConcurrentImportSessions` has no observability**~~ **RESOLVED 2026-08-15 (`372ea4df`).** Gauge `scadabridge.central.transport.import_sessions.open` on the `ZB.MOM.WW.ScadaBridge` meter — already in `SiteServiceRegistration.ObservedMeters` (the single shared allowlist source for both roles; no appsettings pins one), so no allowlist change was needed and the series is not silently dropped. Provider-backed like the S&F queue-depth gauge: registered by `BundleSessionEvictionService.StartAsync`, identity-checked-cleared on `StopAsync`. `IBundleSessionStore.OpenSessionCount` counts only **unexpired** sessions, so the gauge doesn't read "pinned at the cap" for a sweep interval after a burst drains. `BundleSessionStore.Open` logs Warning with open count, cap and TTL on a genuine cap hit only (a store full of expired sessions is reclaimed silently). No alertable rejection counter yet — row 66. | Found while fixing row 43 — R6 could only infer the leak from rotating test failures | Closed 2026-08-15. | — |
| 54 | **`scadabridge_site_connection_up` is misleadingly named** *(Low, observability)* | Row-48 close-out (2026-08-15) — the row's original mis-diagnosis traced directly to this | It counts open central→site gRPC subscriber streams (`SiteStreamGrpcServer`), not DCL data connections, and correctly reads 0 whenever no viewer is attached. Rename (e.g. `scadabridge.site.stream.subscribers.active`) and/or add a real data-connection gauge — the data exists on the health report (`dataConnectionEndpoints`/`dataConnectionTagQuality`) but neither is exported as a metric. | Next observability/metrics-naming session, or the next time someone reads this series as data-connection health |
| 55 | **`tagResolutionCounts` reads `{totalSubscribed:0, successfullyResolved:0}` for a connection with 2 live Good-quality subscribed tags** *(Low)* | Row-48 live probe: site-a's health report shows the zero counts while `dataConnectionTagQuality` on the same report says `good:2` | The "device still booting / tag path unresolved" diagnostic is blind if this stays zero. Code inspection at close-out shows `DataConnectionActor` calls `_healthCollector.UpdateTagResolution` with the derived counts at all the right lifecycle points, so this is likely not a `491df111` regression — suspect a reporting-path issue instead (e.g. a standby-node collector's empty entry shadowing the active node's, or a keying/removal race in `SiteHealthCollector`). Needs live debugging. | Next health-monitoring or DCL session |
| 56 | **The rig's central nodes expose no Prometheus endpoint** *(Low, rig config)* | Row-48 live probe | `docker/site-*/appsettings.Site.json` set `MetricsPort: 8084`, but neither `docker/central-node-a` nor `-b` does, so central-only series (KPI/outbox counters, the new import-session gauge, alarm-cache gauges) cannot be scraped live and any live probe naming "central metrics" is unsatisfiable. | Next rig/observability session |
| 57 | **`audit-config query` has no CLI table projection** *(Low, cosmetic)* | Row-46 fix (2026-08-15) | `--format table` renders the whole `entries` array as raw JSON inside one cell because the command passes no `tableProjector` (unlike e.g. `template list`). JSON output is correct and is the primary format. | Next CLI session |
| 54 | ~~**`scadabridge_site_connection_up` is misleadingly named** *(Low, observability)*~~ **RESOLVED 2026-08-15 (`b5f120c3`).** Renamed to `scadabridge.site.stream.subscribers.active` (a test pins the new name AND asserts the old one is gone; doc/script references updated) and BOTH real data-connection gauges added: `scadabridge.site.data_connections.connected` / `.total` — the denominator ships with the numerator because neither alone separates "device down" from "nothing configured"; `connected < total` is the alert condition. Sourced from the DCL's four state transitions, deliberately NOT from the health report, so the metric survives a dead report pipeline (and stays clear of the concurrent row-55 fix in `SiteHealthCollector`). All on the already-allowlisted `ZB.MOM.WW.ScadaBridge` meter. Expected rig shape: connected==total on the active site node, 0/0 on the standby (healthy — connections open on the active node only). | Row-48 close-out (2026-08-15) — the row's original mis-diagnosis traced directly to this | Closed 2026-08-15. | — |
| 55 | ~~**`tagResolutionCounts` reads `{totalSubscribed:0, successfullyResolved:0}` for a connection with 2 live Good-quality subscribed tags** *(Low)*~~ **RESOLVED 2026-08-15 (`5da27e05`).** All three suspected reporting-path hypotheses were WRONG — live debugging showed the zeros and the `good:2` on the SAME report from the SAME collector on the active node, ruling out standby shadowing, keying races, and the central merge in one stroke. The real bug is a DCL omission, not a regression: `DataConnectionActor.HandleSubscribeCompleted` — the first-time, everything-resolves-cleanly subscribe path — was the missing FOURTH `UpdateTagResolution` push site. `BecomeConnected`'s legitimate 0/0 push fires before the stashed subscribe unstashes (proven in live logs: "Entering Connected state" follows the subscribe line), `ScheduleTagResolutionRetry` cancels outright at zero unresolved, and any reconnect masks the bug via the batch path — so the diagnostic was blindest on a perfectly HEALTHY connection, exactly when it should read clean rather than empty. One mirrored push + regression test (stash-verified fail-before: exactly one `0/0` call received pre-fix while the actor's own `GetHealthReport` already said 2/2, pinning the actor-correct/collector-stale asymmetry). Also corrected `Component-DataConnectionLayer.md`, which claimed the counts were "derived at report time" — they are pushed, and a cached push is only as fresh as its last push site. | Row-48 live probe; root-caused live 2026-08-15 | Closed 2026-08-15. | — |
| 56 | ~~**The rig's central nodes expose no Prometheus endpoint** *(Low, rig config)*~~ **RESOLVED 2026-08-15 (`103f2f19`).** Premise refined on inspection: central already served `/metrics` on its web port (anonymous, no fallback policy) — what was missing was the dedicated 8084 listener the documented docker-exec scrape recipe targets. Both rigs' central appsettings now set `MetricsPort`, and the Host binds it as a PRESENCE check, not a defaulted read — `NodeOptions` defaults to 8084, so a defaulted read would make every existing production central surprise-bind a new port on upgrade, and port-in-use is a hard boot failure. | Row-48 live probe | Closed 2026-08-15. | — |
| 57 | ~~**`audit-config query` has no CLI table projection** *(Low, cosmetic)*~~ **RESOLVED 2026-08-15 (`b2cab59c`).** `AuditConfigTableProjection` mirrors `TemplateTableProjection`: unwraps the `{entries, totalCount}` envelope to rows (`id, timestamp, user, action, entityType, entityName, entityId`), drops the fat `afterStateJson`, adds `--detail` to skip projection (matching `template list`), and surfaces `totalCount` as a trailing note row (`showing N of M — use --page/--page-size`) emitted ONLY when the page is a strict subset, with no `id` cell so it can't be mistaken for an entry. The deprecated `audit-log` alias shares the command tree and gets the projector too. Tests serialize the REAL `AuditLogQueryResult`/`AuditLogEntry` types, so a server-side property rename fails the test instead of silently blanking a column. Stale-README follow-up recorded as row 76. | Row-46 fix (2026-08-15) | Closed 2026-08-15. | — |
| 58 | **`IDeploymentStatusNotifier` singleton retains delegates targeting circuit-scoped Blazor components** *(Low)* | Row-49 sweep (CentralUI-006 re-verification) | Safe today because the single subscriber unsubscribes in `Dispose` (pinned by test) and the notifier isolates each handler. | A second page subscribes, or any subscriber lands without a matching `Dispose` unsubscribe |
| 59 | **`SiteCallAuditActor` per-tick scope spans all sites × ≤50 upsert pages** *(Low)* | Row-49 sweep (SiteCallAudit-009 re-verification) | Safe only because site reconciliation is a serial `foreach … await` (`SiteCallAuditActor.cs:649-653`); one scope across a parallel fan-out would be row 45 again. | `ReconcileSiteAsync` is ever fanned out with `Task.WhenAll` |
| 60 | **`ArtifactDeploymentService.GlobalArtifactSnapshot` holds EF-materialised entities across the parallel per-site send** *(Low)* | Row-49 sweep (DeploymentManager-023 re-verification) | Safe because lazy loading is off and the parallel phase reads only materialised POCOs (per-site commands are built serially first, commented "DbContext is not thread-safe"). | Lazy-loading proxies enabled, or per-site command building moved inside the fan-out |
| 61 | **`_injectedRepository` test-mode constructors are one call site from re-introducing row 45** *(Low)* | Row-50 guard allowlist: `AuditLogIngestActor` / `SiteCallAuditActor` each expose a constructor taking a scoped repository and cache it for the actor's lifetime | Safe only because no production call site picks that overload (production wiring uses the `IServiceProvider` overload; `SiteCallAuditActor` additionally wraps in `SerializedRepository`). The `CaptiveDependencyGuard` allowlists both with a staleness self-test. Preferred fix: fold the test fixtures onto the `IServiceProvider` overload with a stub provider and delete the overloads. | Anyone adding a production `Props` for either actor, or the next time those actors are touched |
| 61 | ~~**`_injectedRepository` test-mode constructors are one call site from re-introducing row 45** *(Low)*~~ **RESOLVED 2026-08-15 (`92fe5acc`).** Preferred fix done: both actors now have exactly ONE constructor (`IServiceProvider`); 12 test call sites folded onto stub providers via two `TestSupport` helpers. Also deleted as newly-dead: `OnCachedTelemetryWithoutDualWriteAsync`, `SiteCallAuditActor`'s `_backgroundTimersEnabled` (a constant once the test ctors went), and the private `SerializedRepository` — moved into the test project, since production resolves a fresh repository per scope and the concurrency hazard existed only for fixtures sharing one instance. `CaptiveDependencyGuard` allowlist is now EMPTY; the staleness self-test replaced by two stronger pins — `Allowlist_IsEmpty` plus `FormerlyAllowlistedActors_NowScanClean`, which scans both actors with the allowlist DISABLED, so emptiness is evidence the fields are gone, not that the scan stopped looking. Control: re-adding a field fails both pins and the real scan. | Row-50 guard allowlist | Closed 2026-08-15. | — |
| 62 | **Five SiteRuntime actors resolve `ISiteEventLogger` off the root provider without a scope** *(Low)* | Row-50 guard sweep: `AlarmActor` (which also caches it in a field), `ScriptActor`, `InstanceActor`, `NativeAlarmActor`, `DeploymentManagerActor` | Correct today (`ISiteEventLogger` is `AddSingleton`), but it is the one pattern that silently becomes row 45 if that registration ever changes lifetime. The guard catches the `AlarmActor` field case automatically; the per-call root resolves it cannot see. | Any change to `ISiteEventLogger`'s lifetime, or a second service pulled this way |
| 63 | **Rig site containers run without `ASPNETCORE_ENVIRONMENT`, so the row-50 Development validation never fires there** *(Low, rig config)* | Row-50 implementation: central containers set it, site containers don't | The site half of the scope validation is exercised only in CI/tests. Setting it on the six site containers would give live coverage now that the site graph validates clean. | Next rig config pass |
| 63 | ~~**Rig site containers run without `ASPNETCORE_ENVIRONMENT`, so the row-50 Development validation never fires there** *(Low, rig config)*~~ **RESOLVED 2026-08-15 (`0d2d6de6`).** Set on all six `docker/` site containers and both `docker-env2` site containers, mirroring the centrals; both compose files re-parsed to prove well-formedness; the now-stale `Program.cs` comment corrected. A captive dependency in the site graph now fails the container at boot on the rig, not just in CI. | Row-50 implementation | Closed 2026-08-15. | — |
| 64 | **The fleet-wide unlock throttle's count-then-insert is not atomic across central nodes** *(Low, accepted by design)* | Row-52 implementation trade, documented in both Transport docs | Simultaneous attempts can overshoot `MaxUnlockAttemptsPerIpPerHour` by the number in flight; bounded and self-correcting within the window. A conditional single-statement insert would close it at the cost of provider-specific raw SQL. | A threat model needing a hard cap rather than a rate damper, or evidence of a real brute-force exploiting the overshoot |
| 65 | **`BundleUnlockAttempt` rows are pruned only by nodes actively serving unlocks** *(Low)* | Row-52 implementation trade | The prune rides the unlock path, so a cluster that stops importing entirely leaves the last rows in place indefinitely (bounded and tiny: one row per permitted attempt, retained 2× the window). | The table appears in a size review, or Transport gains a central scheduler a prune tick could ride |
| 66 | **No counter for import-session cap rejections** *(Low)* | Row-53 implementation note | Row 53 shipped the open-sessions gauge and a Warning log, so a rejection is diagnosable from logs, but there is no series to alert on. | Someone wants an alert rule on import-session exhaustion rather than a log search |
| 67 | **Startup reconcile stores a fetched instance config but does not start the InstanceActor** *(Medium-Low, availability)* | Proven live during the rows-46–53 close-out (2026-08-15): site-a booted without `rig-probe-1`'s config, `SiteReconciliationActor`'s node-STARTUP fetch stored it ("Reconcile: fetched + stored config"), but no `InstanceActor started` followed — `debug snapshot` returned `instanceNotFound` and the enabled, successfully-deployed instance stayed dead until the NEXT restart, whose SQLite load finally started it | The reconcile heal path heals the *file*, not the *runtime*: an instance whose config is missing at boot (lost volume, first boot after restore, rehomed data dir) needs two restarts to run again. Narrow trigger, self-healing, but silent — nothing surfaces "config stored, actor not running". Fix shape: after a startup-reconcile fetch stores a config for an enabled instance with no running actor, tell the DeploymentManager to start it (or re-run the SQLite load path for the fetched set). | Next SiteRuntime/reconcile session |
| 66 | ~~**No counter for import-session cap rejections** *(Low)*~~ **RESOLVED 2026-08-15 (`90ce0999`).** `scadabridge.central.transport.import_sessions.rejected` on the existing allowlisted meter, emitted ONLY at the post-eviction recheck — the cap is tested twice and a store full of expired sessions reclaims and succeeds, so counting the first test would fire routinely on any node whose sweep hadn't run; duplicate-session-id re-`Open` is also excluded. New tests use delta-over-scoped-action with unique per-test keys (the pre-existing gauge tests' absolute asserts are row 74). | Row-53 implementation note | Closed 2026-08-15. | — |
| 67 | ~~**Startup reconcile stores a fetched instance config but does not start the InstanceActor** *(Medium-Low, availability)*~~ **RESOLVED 2026-08-15 (`482165cb`).** Root cause: InstanceActors are created from exactly two paths — the boot-time SQLite load in `DeploymentManagerActor.PreStart` and an explicit deploy/enable command — and the reconcile pass fires ~5s AFTER the boot load with a pure file write (`StoreDeployedConfigIfNewerAsync`) as its only side effect. Fix: reconcile collects the enabled instances it stored and Tells a new `EnsureInstanceStartedCommand` to the DM **singleton proxy**, so the actor is created under the singleton no matter which node's pass found the gap. Handler is advisory and idempotent: no-op if already running; re-reads the durable row (authority on config body AND `is_enabled`, so a disable landing after the fetch wins); defers to `_deployWarms`/`_terminatingActorsByName`/`_inFlightDeployments` so deploy-gate ordering and all-or-nothing-per-instance are untouched (it starts an already-persisted config, never applies one); quiet no-op on an absent row (normal on an unreplicated site whose peer healed its own file). The heal is no longer silent — a Warning site event `Instance <X> started by startup reconcile self-heal` fires. Fail-before by behavioral revert: 3 of 8 new tests fail pre-fix, including the exact live symptom (`debug snapshot` → `instanceNotFound`); the 5 negative guards (disabled / failed-fetch / unknown / idempotence) green in both states. SiteRuntime 613/613, Host 504/504. Sibling gaps found during the fix recorded as rows 72/73. | Proven live during the rows-46–53 close-out (2026-08-15) | Closed 2026-08-15. | — |
| 71 | **Standby nodes log the active-node health check at ERR every 10s** *(Low, log noise)* | Row-55 live session (2026-08-15): site-a-b logs `Health check active-node with status Unhealthy … 'not the site pair primary (standby)'` at Error level every 10s | Correct-by-design standby behavior emitted at an alarming severity — floods the standby's log and trains operators to ignore ERR. Candidate: filter/downgrade the ASP.NET health-check publisher's severity for the expected-standby case. | Next observability/log-hygiene session |
| 72 | **Reconcile refreshes a STALE config but does not restart the running InstanceActor** *(Medium-Low — same silent-divergence class as row 67, arguably wider blast radius)* | Row-67 fix session (2026-08-15): `ReconcileGapItem` covers both *missing* and *stale* (`StaleInstance_IsRefreshed` pins a revision bump); for a stale item the new config is written to SQLite while the running actor keeps serving the OLD compiled config until the next restart or redeploy | Deliberately out of row-67's scope: restarting a LIVE instance from reconcile carries redeploy semantics (stop/start, subscription re-establish, in-flight runs) that need their own design decision. Nothing surfaces "stored revision ≠ running revision". | Next SiteRuntime/reconcile session, or first field report of an instance serving a stale config after a central redeploy raced a site outage |
| 73 | **`SiteReconciliationActor` is silently skipped when `IDeploymentConfigFetcher` is null** *(Low)* | Row-67 fix session: `AkkaHostedService` (~line 1046) logs only a Warning and spawns no reconciler | The entire self-heal path — file heal AND the new row-67 runtime heal — can be absent with no health-report surface; a misconfigured site looks healthy while unable to heal. Candidate: surface reconciler presence on the site health report. | Next health-report or reconcile session |
| 74 | **`BundleSessionObservabilityTests` gauge tests assert absolute values — latent row-32-class race** *(Low, test-only)* | Row-66 implementation (2026-08-15): the pre-existing gauge tests `Assert.Equal(0, baseline)` against a process-global provider slot, safe only because Transport is that slot's sole registrant; the new rejection-counter tests use delta-over-scoped-action with unique keys instead | Same shape as the row-32 flake — becomes flaky the day another suite registers the slot. The fix pattern already exists in the same file. | First flake, or next test-infra session |
| 75 | **SecuredWrites Approve renders as a red destructive button** *(Low, UI)* | Row-28 audit (2026-08-15): the call site passes `danger: true`, so Approve is styled destructive; the label is now correct ("Approve") but the styling still signals delete | Approving a secured write DOES execute a device write, so a caution affordance is arguable — which is why row 28 fixed only the label. A deliberate styling decision belongs to whoever owns the page. | Next Secured Writes / Central UI session |
| 76 | **CLI README's `audit-config query` JSON sample omits `afterStateJson`/`bundleImportId`** *(Low, docs)* | Row-57 fix (2026-08-15): the real payload returns both fields; the sample doesn't show them | Misleading as written; one-line doc fix, out of row-57's table-projection scope. | Next CLI docs touch |
| 77 | **Test pattern: reflecting `ActorCell.Actor` right after `ResolveOne` is unordered vs child construction** *(Low, test-infra)* | Row-40 session (2026-08-15): `InstanceActorChildAttributeRaceTests` failed ~1-in-6 under full-suite load once the sleeps stopped padding the schedule (latent — A/B'd 14/14 green on the unmodified tree); fixed with a bounded-deadline resolution retry | `ResolveOne` throws until the child's cell publishes on the parent, and `cell.Actor` stays null until the child's ctor runs on its own mailbox thread — neither ordered against the test thread. The one instance is fixed; the pattern may exist elsewhere. | Next time a child-resolution test flakes under load |
| 68 | ~~**`deploy.sh` run from a git worktree silently rehomes every node's bind-mount data dir**~~ **RESOLVED 2026-08-15 (guard added in the same close-out).** The compose files' bind mounts are relative (`./site-a-node-a/data`), so a deploy from a worktree checkout mounts the worktree's copies — the rig keeps running and looks healthy, but its SQLite stores split from the main checkout's and everything written there is orphaned by the next deploy from main. Bit the rows-46–53 close-out live: the seeding agent's `rig-probe-1` deploy persisted into worktree-homed dirs and "vanished" after the orchestrator's redeploy (no tombstone, no product bug — LWW/tombstone forensics ruled out deletion; the row sat in `.claude/worktrees/…/docker/site-a-node-a/data/site-localdb.db` the whole time). Both `docker/deploy.sh` and `docker-env2/deploy.sh` now refuse to run from a `.claude/worktrees/` path (override: `SCADABRIDGE_ALLOW_WORKTREE_DEPLOY=1`). | Found during the rows-46–53 close-out | Closed 2026-08-15. | — |
## Resolved (verified against the code 2026-07-10)