Joseph Doherty
2650f79472
Also: DeploymentDeadlineParityTests' stub replies with a real zero-failure summary (the artifact aliases now derive their exit code from failureCount), and register row 127 reflects the merge and the two new replicated site tables.
153 KiB
153 KiB
Deferred-Work Register (established 2026-07-08, from architecture review 08)
Single tracked list of consciously-deferred work. Rules: every deferral gets a row (rationale + revisit trigger); fix-now items reference the archreview plan that owns them and are removed from this table when that plan's task lands.
Fix-now (owned by archreview plans)
All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round 2, 2026-07-12) — rows removed per the rule above; see the round-2 report §1 for the per-item evidence.
Deferred (with rationale + revisit trigger)
| # | Item | Where noted | Rationale for deferral | Revisit trigger |
|---|---|---|---|---|
| 8 | Hash-chain tamper evidence (T1); CLI verify-chain is a no-op stub | audit-log roadmap :12 | v1.x by locked decision; append-only DB roles are the control | Compliance requirement for cryptographic tamper evidence |
| 9 | Parquet audit archival (T2); endpoint returns 501 | AuditEndpoints.cs:204 | v1.x; 501 + CLI messaging are honest | AuditLog partition volume nears retention ceiling |
| 11 | Central-persisted OPC UA cert-trust audit | m7 follow-ups | Broadcast-to-both-nodes covers HA | Governance/audit requirement for trust decisions |
| 17 | Unified notifications+site-calls outbox page | stillpending :118 | Explicit M9 decision to keep two pages | Operator confusion reports |
| 19 | Bundle signing / cluster-to-cluster pull / differential bundles | transport-design :402 | v1 manifest hash + AES-GCM held sufficient | Non-repudiation requirement across orgs |
| 23 | Live LDAP group-membership re-query for an active session | docs/requirements/Component-Security.md :61-69 (+ :78-79) |
Blocked on an external package. The mid-session refresh re-maps the stored groups against the central DB with no LDAP call, so a directory group-membership change lands only at next login. A live re-query needs a passwordless service-account group-search method on the shared ZB.MOM.WW.Auth.Ldap library — an external NuGet PackageReference (src/ZB.MOM.WW.ScadaBridge.Security/…csproj:23) exposing only AuthenticateAsync(username, password, ct). Central role-mapping/scope changes still apply within ~15 min (RoleRefreshThresholdMinutes). |
ZB.MOM.WW.Auth.Ldap gains a standalone group-search API, or a requirement that a directory-side group revocation take effect mid-session rather than at next login |
| 24 | M8 large-bundle performance hardening | docs/plans/2026-06-15-stillpending-completion-design.md:106 — "Small follow-ups logged (not blocking): … large-bundle/perf hardening" |
Logged as a non-blocking follow-up when M8 shipped and never given an artifact: no plan, no task entry, no perf/load test exists (tests/…Transport.Tests/Import/BundleImporterLoadTests.cs is a LoadAsync unit suite despite the name). No measured problem; the only sizing controls in place are the 5-minute CLI transport timeout, LineDiffer's MaxInputLines=4000 summary-only cap, and MaxConcurrentImportSessions=8. |
First real bundle that times out, exhausts memory, or makes the import wizard's diff step unusable |
| 25 | docker/ rig still running. Headline numbers: tag update latency P50 0.88 ms / P99 4.57 ms / max 37.41 ms end-to-end (DCL boundary to stream subscriber, the emit instant carried verbatim through production code); 0 events dropped at 100 live subscribers; health report collect+ingest P99 0.31 ms with 10/10 sites tracked; debug view snapshot under load P99 2.19 ms, 0 timeouts; 500 instances deployed to a site in 2.6 s; CPU 2.9% of the box; working-set slope +8.83 MB/min. 11 clean passes, 1 pass with a caveat (F1: 20 min with zero gen-2 collections cannot fully settle the leak question — the 1-hour run would), 0 failures. Harness: tests/ZB.MOM.WW.ScadaBridge.LoadHarness/ (standalone Exe — the Category=Performance trait enables a filter but does NOT exclude by default, so a 20-minute test could not live in PerformanceTests) plus a CI-scale smoke [Fact] (PerformanceTests/TargetScale/TargetScaleHarnessSmokeTests.cs). Real: per-site ActorSystem + LocalDb, the real DCL via the documented DataConnectionFactory.RegisterAdapter seam, real InstanceActors, real SiteStreamManager, real StreamRelayActor + production-capacity bounded channel, real S&F, real health collector/aggregator; only the socket hops are stood in for. Results: docs/plans/2026-08-15-target-scale-load-test-results.md; design + deviations + thresholds: docs/plans/2026-08-15-target-scale-load-test-design.md. |
as before | Closed. Residual scope: [xc-3] site-event-log retention volume, [xc-4] audit-vs-central degradation, [xc-5] large-template flattening and [xc-8] UI responsiveness are NOT covered by this harness (central-cluster/browser concerns) and remain open WP-4 scope — see results doc SS6. |
Closed 2026-08-15. |
| 26 | Ipsen MES MoveIn tail: leak-test (-LT) receivers + routing, PLC-output-flag writes, Z28062 BTDB data completeness |
docs/plans/2026-06-16-ipsen-mes-movein.md:409 ("Out of scope (future)"); design 2026-06-16-ipsen-mes-movein-design.md:58-60, :196-198 |
Customer-site scope, not a platform gap. -LT routing needs an MES-receiver child + Galaxy reference that do not exist on the reactor template (any -LT/unknown suffix returns WasSuccessful=false with an "unsupported side/target" message by decision); MoveInComplete/Successful/ErrorText are PLC-owned by locked decision, so ScadaBridge deliberately does not write them; Z28062 completeness is an operational data fix, not code. Note the separate alarm-status path already handles the suffix — _LT is stripped before side-scoping (2026-06-30-mes-alarm-status-api.md:158). |
Ipsen creates the leak-test receiver + Galaxy reference, or asks ScadaBridge to own the PLC-output flags — otherwise a candidate won't-do ([PERM]) at the next Ipsen scope review |
| 27 | MaxRetries/RetryDelay) never reaches sites, and has no CLI/management surface41a5ecf1). The gap was transport + persistence only — the consumption end already worked (ExternalSystemClient.CachedCallAsync already passed system.MaxRetries/RetryDelay to EnqueueAsync with 0-as-unset, and the Central UI form already collected both fields, inert until now with defaults 3×5s that will take effect on next deploy). Shipped: artifact fields, additive site_command.proto fields 7/8 (regen via docker/regen-proto.sh, csproj left clean), site external_systems columns max_retries/retry_delay_ms as REAL COLUMNS — not a JSON bag; verified safe against the AckTime precedent in the LocalDb source (capture triggers are DROP-then-CREATE from PRAGMA table_info each boot, and the fail-closed handshake digest turns a mixed-version pair into a loud sync refusal, no new constraint since both-nodes-together is already the standing pair rule; timeout_seconds set the additive-column precedent on this exact table) — apply path, and CLI --max-retries/--retry-delay-seconds (+--retry-delay alias). Bonus fix: ArtifactDiff.CompareExternalSystem never compared the retry fields, so a bundle import changing a retry budget previewed as "Identical". Fail-before verified on the site round-trip + apply tests; 12 suites green. |
Found live 2026-08-01 (rig session, #11 gRPC live checks) | Closed 2026-08-15. | — |
| 28 | 0db253b5). Structural, not a missing argument: IDialogService.ConfirmAsync had NO label parameter — DialogHost derived the verb from danger alone (danger ? "Delete" : "Confirm"), so no call site could ever have passed one. New optional confirmLabel; danger now selects styling only. Full audit of all 34 ConfirmAsync call sites (the only confirmation path — no window.confirm, no per-page modals): NINE were wrongly labeled "Delete" — both failover buttons, secured-write Approve and Reject, instance Disable, and four Discard actions — all now name their action; 18 genuine deletes + 7 non-danger sites verified correct. The danger→"Delete" fallback deliberately kept: forcing an explicit label would churn ~20 correct sites, and a missed site would silently degrade to "Confirm", detectable only by the rig-dependent Playwright suite — contract pinned in the interface XML doc, Component-CentralUI.md, and a fallback test instead. Gotcha recorded in the tests: adding an optional parameter silently un-matches existing 3-arg NSubstitute stubs (returns null Task → NRE). Follow-ups: rows 75 (Approve's red styling), and the bUnit real-dialog deadlock hazard documented in FailoverConfirmLabelTests. |
Found live 2026-08-01 (rig session, #11 TriggerSiteFailover check) | Closed 2026-08-15. | — |
| 29 | --sustain-minutes 60, 135,041,250 updates at 37,511/s = 100.0 % of nominal, concurrent with the live docker rig AND a central failover drill on the same host). Decisive: working-set slope −28.89 MB/min (2,722 → 1,757 MB over the 3,600 s window, peak 3,258 MB) with one gen-2 collection observed and threads flat at 59 — a working set that FALLS ~1 GB across a sustained hour is incompatible with a slow leak; the positive managed-heap least-squares slope was the sawtooth-segment artifact F1 hypothesised. All other metrics held or improved vs the 20-min run: P99 4.01 ms, 0 healthy-subscriber drops, 744/744 debug snapshots, S&F drain to depth 0. Full numbers appended to the results doc F1 section. |
docs/plans/2026-08-15-target-scale-load-test-results.md F1 postscript |
Closed 2026-08-15. | — |
| 30 | WP-4 residual scope: [xc-3] site-event-log retention volume, [xc-4] audit-load-vs-central degradation, [xc-5] large-template flattening time, [xc-8] UI responsiveness under load |
docs/plans/2026-08-15-target-scale-load-test-results.md §6 + row 25 closure note |
The row-25 harness is a site-runtime rig; these four WP-4 sub-criteria are central-cluster/browser concerns it deliberately does not cover. Recorded so the row-25 closure never reads as "WP-4 fully done" — phase-8 WP-4 is closed for the site-runtime data path only. | Before declaring phase-8 WP-4 complete, or production go-live at target scale |
| 31 | Unpaced-burst absorption is bounded by the shared publish stage, not per-subscriber buffers (load-test finding F3 caveat) | docs/plans/2026-08-15-target-scale-load-test-results.md F3 |
Measurement-backed design note, no defect: slow-subscriber isolation is structural and total (a stalled reader loses events only in its own bounded DropOldest channel; healthy subscribers delivered 100.00%), but the publish Source.ActorRef upstream of the BroadcastHub is shared, so an unpaced burst degrades every subscriber equally — that stage, not per-subscriber buffering, is the burst ceiling. |
First field report of event loss on healthy subscribers during a burst, or any redesign of the site stream publish stage |
| 32 | QueueDepthGaugeTests full-suite flake — shared static gauge (load-test finding F4)65b19516). Reproduced first (2 failures in 5 parallel runs — the gauge read a sibling class's provider). The register's suggested fix (per-test meter) was rejected on inspection: the slot is process-global by product design and 3 of the 6 facts assert the slot-ownership semantics themselves (compare-and-clear identity, stop/restart re-seed), so per-test instances would change what they prove. Correct isolation is mutual exclusion — the three StartAsync-calling classes now share xunit collection QueueDepthGaugeCollection (with a [CallerFilePath]-anchored source-scan drift guard that names an offending file), the other eight classes stay parallel, and parallelizeTestCollections was flipped back ON so the fix is exercised rather than masked. Negative control: dropping the attribute fails the guard AND resurfaces the original flake in the same run. 8/8 green parallel runs. |
tests/…StoreAndForward.Tests/QueueDepthGaugeTests.cs; surfaced by the row-25 verification runs |
Closed 2026-08-15. | — |
| 33 | LocalDbOplogBacklog has no threshold, KPI history, or dashboard treatment8da60f57). Both halves shipped: sampled into KPI history via SiteHealthKpiSampleSource (conditionally — 12 samples unreplicated / 14 replicated, so a series gap truthfully means "not replicated") plus a backlog trend chart on the site health panel, and a SUSTAINED threshold badge (LocalDbBacklogThreshold 1000 held for LocalDbBacklogSustainedFor 15 min, anchored by LocalDbBacklogAboveSince in the aggregator) — a bare instantaneous threshold would fire on every replicated site every day from the documented purge burst. Null backlog is a deliberate third branch: it leaves the anchor unchanged (neither evidence of a drain nor of a backlog), so unreplicated sites can never anchor and can never be badged. NOTE the row's premise was false — the SiteAuditBacklog "configurable threshold badge" it told us to mirror was described twice in Component-HealthMonitoring.md but NEVER BUILT; both stale doc claims corrected, and the new badge was modeled on the real Metrics-stale badge instead. |
Found by R7 while documenting the purge burst | Closed 2026-08-15. | — |
| 34 | site_events purge burst is unbounded in the oplog and unmeasured at scaled9a35846) — measured, and paced. Measured with the REAL EventLogPurgeService over a real RegisterReplicated LocalDb file + a loopback sync pair at the rig's MaxBatchSize: 64: 393 B/row incl. indexes ⇒ a 1 GB event log holds ~2.73 M rows, and expiring them in one tick took 285 s and queued 2 731 947 oplog rows = 2.7x LocalDb's MaxOplogRows default (1 M) — enough to make the maintenance tick prune the oplog and flag a FULL ten-table snapshot resync, bought by a routine retention purge. The decisive nuance: the resident backlog is produced−minus−drained, so a healthy peer blunts it (391 k peak on loopback, drained 6.0 min) — the old safety margin was a function of peer throughput and vanished with the peer down. Fix: MaxPurgedRowsPerTick (default 250 000, 0 = unlimited escape hatch), one budget SHARED across retention + cap trim; a ceiling-limited tick reschedules after PurgeCatchUpInterval (5 min, validated > 0 and <= PurgeInterval) so worst-case retention converges in ~55 min instead of 11 days; purge loop is now a delay loop anchored to the END of the previous purge; rate-limited Warning + counter scadabridge.site.event_log.purge_ceiling_hits. No purge-exemption path added. Row-33 interplay verified empirically (500 k drained in 46 s vs the 15-min dwell). Fail-before: PurgeBudget neutralised → 6 of the new tests fail, all 19 pre-existing pass. Measurement + decision record: docs/plans/2026-08-15-site-events-purge-burst-measurement.md. Side-finding recorded as row 81 (storage cap unenforceable on a replicated node). |
Found by R7 | Closed 2026-08-16. | — |
| 35 | DeploymentRecords index does not cover the deployment-list sort keybe29bf47). Migration 20260815165452_AddDeploymentRecordListSortIndex + hand-written docs/plans/sql/AddDeploymentRecordListSortIndex.sql: composite IX_DeploymentRecords_DeployedAt_Id (DeployedAt DESC, Id DESC); the old single-column index DROPPED as strictly redundant (Id is the clustered PK and was already in the old index's leaf as the row locator — same leaf width, now a declared ordered key; keeping it would cost a second index on every insert/status transition for no read benefit). Serves the three reads sharing the key (QueryDeploymentListPageAsync, QueryDeploymentSummariesAsync, GetCurrentDeploymentStatusAsync). Id in the key is also a CORRECTNESS matter: a bulk site deploy stamps many rows in one DeployedAt instant, so without the tie-break keyset pages can repeat or drop rows. Recorded caveat: the optimizer may already have served the ordering via a backward scan of the old index, so the "sorts per page" premise was possibly optimizer-dependent — the composite makes it deterministic and self-documenting; an actual plan capture against the rig would quantify the before/after. The ConfigurationDatabase suite exercises the descending composite as real DDL via SQLite EnsureCreated(), proving provider portability. |
Found by R3 (35ce1413) |
Closed 2026-08-15. | — |
| 36 | Deployments free-text search is unindexable by construction | Found by R3; re-verified 2026-08-17 (Low-tail sweep) | Four LIKE '%term%' predicates (instance name, deployment id, revision hash, user) — leading wildcards preclude index seeks, so a search is a scan of the join (DeploymentManagerRepository.cs:100-104; only searched column with any index is DeploymentId, unseekable under a leading wildcard). Acceptable at the purge-bounded row count. 2026-08-17 refresh: the row undercounted the cost by half — BuildListQuery is shared with GetDeploymentStatusCountsAsync, so one search = TWO scans of the join (list + status tiles), mitigated by the UI's trailing-edge debounce (one pair per pause, not per keystroke). Anchoring the id/hash legs was evaluated and REJECTED as a micro-fix: DeploymentListFilter.Search's doc comment promises substring matching and a 4-case [Theory] deliberately pins mid-string matches — changing it is a contract renegotiation, not an optimization. Fix if hot: renegotiate to prefix on id/hash legs, or full-text. (Low) |
Search latency complaints on the Deployments page |
| 37 | DeploymentRecords with different contractsad0100f2, agent commit 9f67147e). QueryDeploymentSummariesAsync and DeploymentRecordSummary are DELETED; the CLI/HTTP QueryDeployments handler now rides QueryDeploymentListPageAsync + DeploymentListFilter — one row shape (DeploymentListRow), one scoping mechanism (site-id via the query join), total count available to both surfaces (DeploymentListPage now also owns MaxPageSize + DefaultPageSize = 50). Rows 86–88 were resolved with the consolidation, as this row anticipated. Row 36's 4-case substring-search Theory untouched and still passing. Remaining UI-vs-CLI asymmetry (GetDeploymentStatusCountsAsync status tiles are UI-only) is deliberate, not drift. Live-probed on the rig: CLI deploy status --status Success --page-size 5 returns the consolidated page — rows now carry siteId + instanceUniqueName, and the response carries totalCount (11) the CLI never had. |
Found by R3; consolidation shipped with rows 86–88 | Closed 2026-08-17. | — |
| 38 | RoslynVirtualTagEvaluatorTests racing-clear test is load-fragile34829ea0 + 837d8a17, branch perf/followups-r3). Rebuilt on the assert-on-retry-exhaustion option, renamed …never_leaks_a_raw_disposed_failure. Two findings beyond the brief: (1) the DOMINANT load-failure mode was the test's own 45 s WaitAsync budget — every clear forces a recompile, so the fixed 12×400 workload's wall-clock tracked spare CPU; reproduced RED 4/4 at 4-way self-concurrency, fixed by duration-bounding the loops (10 s window OR 400 iters), GREEN 4/4 and 8/8 after. (2) A second flake vector: TimedScriptEvaluator's 5 s clock starts at enqueue, so pool starvation surfaces "script timed out" from healthy evaluations — now its own bounded bucket, not counted as a leak; failure-reason literals promoted to shared consts. Honest caveat recorded: the disposed race itself does not reproduce on the arm64 macOS box (zero ODEs even with the product retry disabled), so the raw-escape assertion is the load-independent guarantee. Commit stats cross-checked from this repo. |
Found by R4; fixed cross-session by the OtOpcUa agent 2026-08-18 | Closed 2026-08-18. | — |
| 39 | 3e62f161 + review round df533916, branch perf/followups-r3). ScadaBridge's WP1.3/WP3.3 pattern ported: registration follows replication config — configured ⇒ RegisterReplicated(..., baselineExistingRows: true) (probed live on 0.2.1: seeds __localdb_row_version at HLC 0), unconfigured ⇒ boot-time DeregisterReplicated self-heal, pair-symmetric; review round added deregister-preserves-data + partial-flip-fails-loudly tests. Green: LocalDbSetupTests 15/15, Host.IntegrationTests 226 pass/0 fail. Stayed pinned to LocalDb 0.2.1 by deliberate choice — the 0.2.1 API already carries both needed methods, and the v2→v3 one-way bookkeeping upgrade makes a 0.3.x bump a both-nodes-together operation; recorded caveat in OtOpcUa's CLAUDE.md: on 0.2.1 the snapshot-resync flag is per-DATABASE, so a late enable snapshots the whole file rather than per-table (0.3.1 available; bump logged as a deliberate future change in OtOpcUa's own follow-up register). Commit stats cross-checked from this repo. |
Found by R4; fixed cross-session by the OtOpcUa agent 2026-08-18 | Closed 2026-08-18. | — |
| 40 | 7a7d345b). All six listed sites replaced with observable edges + positive controls, strengthening proven by stash-based negative controls: ScriptDeadlineAtEnqueueTests:128 now pins the watchdog via a genuinely-wedged positive control (armed strictly later, self-releasing) + per-script absence assertion + detach counter pinned at 1 — disabling the entire stuck-script watchdog passed the OLD test 2/2 and fails the new one; :105 now proves the run was still queued when its budget expired via QueueDepth edges (the actual design claim — the old sleep asserted nothing about it); DeploymentWarmThenGateTests:117 (+2 sibling Task.Delay(500)s) — the empty-site zero is now the manager's own observation (phantom-startup control: old passed 2/2, new fails); SiteAlarmLiveCacheServiceTests:195,216 poll a new subscription-cancellation observable across 3 linger windows, each closing with a positive control (control: linger cancellation disabled + viewer guard dropped — Resubscribe_Within_Linger old passed 2/2, new fails; honest caveat: the other case's old later assertion did already catch that break, so its gain is the observable + control, not a false-pass fix); DataConnectionActorTests:435 waits for the Connected health publication so the subscribe provably runs against Connected (the sleep guarded no assertion — determinism gain only). Bonus: the de-padded schedule exposed a LATENT race in InstanceActorChildAttributeRaceTests (~1-in-6 under load, A/B'd 14/14 green on the unmodified tree so pre-existing) — fixed with a bounded-deadline resolution retry; pattern recorded as row 77. |
Found by the R5 sweep (9fb52153) |
Closed 2026-08-15. | — |
| 41 | 3e5a5fd, LocalDb 0.3.1) — hypothesis CONFIRMED as a product bug, fixed on the product path. The enqueue→receive→validate→fault path in HandshakeAsync has no yields while the writer's wakeup sits in the thread-pool queue, so the catch's linked.Cancel() beat the writer 39/40 times in a new pinning test (Handshake_FaultTeardown_StillTransmitsTheLocalHandshake) — mismatch detection is symmetric (each side computes its verdict FROM the other's Handshake), so the healthy peer learned nothing: 15 s stall in tests; in production a permanent schema mismatch would be misclassified as a generic disconnect and reconnect-loop at fast backoff instead of the FailedPrecondition/max-backoff path. Fix: RunAsync's fault teardown completes both lanes so the writer drains what's enqueued and exits on its own, with a 2 s deferred cancel as the wedged-transport backstop; caller-initiated cancellation unchanged; no wire change (a 0.3.0 peer just finally receives the Handshake it was owed). A/B under doubled load: pre-fix 2/6 full runs failed (exact predicted stack), post-fix 6/6 green (200/200) + 5/5 filtered. The OTHER named test (BatchBudgetTests.Pump_RowLargerThanTheBudget…) was a harness defect, not this root cause (scripted peer, no handshake fault): it awaited the peer channel without observing the session task, so any session fault surfaced as an opaque 15 s timeout — hardened with a run-observing NextAsync that races the session task at every read site. Residual (scadaproj-side, pre-existing class): two non-recurring one-offs in 10 post-fix runs (Handshake_Exchanged_BothProceed @53 ms, Metrics_NullSafe… @208 ms; detail lost, never recurred in 12 further loaded runs either side) — both share the SwallowAsync-catches-only-OCE teardown shape, best guess a cancelled SQLite op surfacing SqliteException at teardown; not attributable to this change, not proven either. 0.3.1 published to the feed 2026-08-18 (all three packages, restore-verified from a scratch consumer with a positive control confirming the restored Replication assembly carries the fix; scadaproj main pushed — 3e5a5fd, b62981f, index dbcbc5b). ScadaBridge's pin bumped 0.3.0 → 0.3.1 the same day: build clean, all six LocalDb-consuming test suites green (SiteRuntime 652, IntegrationTests 115, HealthMonitoring/ExternalSystemGateway/AuditLog/StoreAndForward all pass); a 0.3.0/0.3.1 mixed pair is wire-safe (teardown-ordering change only). |
Found by R1; confirmed pre-existing at base f627cee; fixed cross-session by the scadaproj agent 2026-08-18 |
Closed 2026-08-18. | — |
| 42 | build/push.sh-class publish bug across the family reposb62981f). The bug shape: dotnet pack -c Release -o ./artifacts with no project argument and no cd, so the script packs the CALLER'S CWD. Fixed in LocalDb's script (cd to own repo root, name the solution, absolute output path). AUDITED 2026-08-15 (read-only, this wave): SEVEN scadaproj scripts still carry the exact shape — ZB.MOM.WW.Secrets/build/push.sh, ZB.MOM.WW.Theme/build/{pack,push}.sh, ZB.MOM.WW.Audit/build/{pack,push}.sh, ZB.MOM.WW.Auth/build/{pack,push}.sh (all dotnet pack -c Release -o ./artifacts with no project argument and no cd, plus a CWD-relative nuget push "./artifacts/*.nupkg" that would publish whatever sits in the caller's artifacts dir). MxAccessGateway's scripts/pack-clients.ps1 is SAFE ($PSScriptRoot-anchored, explicit csproj paths); HistorianGateway and OtOpcUa have no pack scripts. Fix = copy LocalDb's 0038d46 pattern into the seven scripts; lives in the scadaproj repo, outside this wave's edit scope. Fixed: all seven scripts now carry the 0038d46 pattern (cd to own repo root, named .slnx, absolute artifacts output, anchored push glob); verified by bash -n on all seven plus a positive control on the pack half only (Audit's pack.sh run from a foreign CWD packed only ZB.MOM.WW.Audit into its own artifacts dir, caller's directory untouched); no push script executed, no credentials touched. Commit stat cross-checked from this repo: exactly the seven listed files. |
Found by R1 the hard way (incident: 25 unrelated packages briefly published to the feed, deleted + verified same session); fixed cross-session by the scadaproj agent 2026-08-18 | Closed 2026-08-18. | — |
| 43 | BundleSession slot on every abandoned import — imports lock out for 30 min70838982). New IBundleImporter.Abandon(sessionId) does the same zero-then-remove release ApplyAsync already did (T-007 discipline), idempotent (double/unknown/expired ids are no-ops) and only ever given the caller's own random-GUID id, so it cannot evict a concurrent circuit's session. Wired into EVERY abandonment path: wizard new-upload, TryLoadAsync pre-overwrite, Back-to-Upload/ResetSessionState, 3-strike lockout reset, Dispose — plus two CLI leaks the original row didn't know about: transport preview released nothing at all (every preview burned a slot for 30 min) and transport import stranded the session on pre-apply aborts; both now release in a finally. BundleSessionEvictionService is documented as the TTL backstop, not an abandonment path. Fail-before-fix stash-verified (3 of 4 new bUnit tests fail pre-fix). 10 new tests. |
Found live by R6 | Closed 2026-08-15. | — |
| 44 | BundleUnlockRateLimiter is dead code — the documented per-IP unlock throttle is not enforced5ab26a42). BundleImporter.LoadAsync now registers every unlock attempt against MaxUnlockAttemptsPerIpPerHour (10). Ordering is load-bearing: AFTER the per-bundle 3-strike check (hammering a locked bundle spends no hourly budget) and BEFORE the decrypt (a throttle denial burns no per-bundle strike, so the throttle can't be abused to lock out a legitimate import); successful unlocks count too. Denial = BundleUnlockThrottledException (placed in Commons so the management surface returns a curated message). Client key: Central UI via a scoped CircuitClientAddressProvider snapshotting the IP at circuit establishment; CLI via additive ManagementEnvelope.ClientIp. No attributable IP falls into one shared <unattributed> bucket — stricter, never a bypass; per-client isolation is only real behind a ForwardedHeaders-trusted proxy (same caveat as the login throttle, documented). Fail-before-fix verified (4 of 6 new tests fail with the limiter neutralised; 2 negative controls). |
Found by R6 | Closed 2026-08-15. | — |
| 45 | DbContext across up to 4 parallel deliveries9e769735). _adaptersCache/_adaptersScope/ResolveAdapters() deleted; the typed adapter is resolved inside the per-delivery DI scope DeliverGatedAsync already opens (static ResolveAdapter preserving DI last-wins + the null-adapter park path), so each concurrent delivery gets its own adapter and DbContext. Nothing shared was lost: the Email adapter builds a fresh SMTP client per attempt from an injected factory, the SMS adapter draws from the singleton IHttpClientFactory, and CentralUI's adapter-gated Type selector (NotificationChannelCatalog) is an independent scoped consumer, untouched. Fail-before-fix stash-verified (simulated DbContext throwing EF's exact second-operation exception under a rendezvous-guaranteed overlap: 1 of 2 rows persisted pre-fix; 1-vs-4 distinct adapter instances). Root cause of note: the bug was INTRODUCED by code-review finding NotificationOutbox-006's caching "fix" (2026-05-28), whose own regression test pinned the broken behavior and predates bounded parallel delivery — that finding is marked SUPERSEDED. NotificationOutbox 153, AuditLog 383 green. |
Found by R6; observed live on rig central-b | Closed 2026-08-15. | — |
| 46 | audit-config query returns {} for every filter combination09108018). Not a CLI bug: ManagementActor.HandleQueryAuditLog returned the repository's ValueTuple (Entries, TotalCount) straight to SerializeResult; System.Text.Json writes properties only and a ValueTuple's Item1/Item2 are fields, so the whole payload serialized to the literal {} — every filter combination looked empty with no error anywhere in the stack. Fixed with a named Commons record AuditLogQueryResult (mirroring SecuredWriteListResult) plus a class-wide guard: SerializeResult now throws on any ITuple result, so the next tuple-returning handler fails loudly instead of silently emitting {}. Fail-before verified (test asserted exactly "{}" pre-fix); live re-verified on the rig (totalCount 2725 unfiltered, 712 for --entity-type Template --action Delete). The only other tuple-returning repo method in Commons (INotificationOutboxRepository.QueryAsync) is never serialized. |
Found by R6 against the rig; underlying AuditLogEntries rows verified present |
Closed 2026-08-15. | — |
| 47 | SiteStreamGrpcServer duplicate-stream replacement disposed another handler's CancellationTokenSourcee6842c10). SubscribeInstance's duplicate-prevention path called Cancel() AND Dispose() on the replaced stream's CTS, which belongs to that still-running handler's using var and is still read — escaping as an unhandled ObjectDisposedException under scheduling pressure (confirmed pre-existing on 2b74851f; load-only, passed 10/10 isolated). Fixed to cancel-only (the owner's using disposes exactly once), Cancel wrapped for the converse race, mirroring CancelAllStreams(). Deterministic regression test via a gate inside the setup window, fail-before/pass-after verified by reinstating the Dispose() as a negative control. Recorded here (though closed) because it is a product-bug instance of the disposal-race class, useful precedent for future stream-lifecycle work. |
Closed 2026-08-15. | ||
| 48 | b5e6821c). Working set re-seeded on site-a entirely via the CLI (soak-* rows left alone): template RigProbe + instance rig-probe-1, bound to the OPC PLC sim's built-in moving tags (SpikeData, RandomSignedInt32) plus a 5s heartbeat script. Deploy Success; debug snapshot returns live Good-quality data; health site shows OPC PLC Simulator: Connected, good:2 bad:0 uncertain:0; a 60s debug stream produced 239 events; and scadabridge.site.stream.batch_size recorded live for the first time (count 70 / sum 238, batches to ~10 events) — the R2 batching histogram is now observable on the rig, which was this row's purpose. Reproducible via docker/seed-live-probe.sh. Two diagnostic corrections worth keeping: the custom tags in infra/opcua/nodes.json are declared but not simulated by opc-plc (a healthy-looking connection with zero traffic — bind to the built-ins: SpikeData, RandomSignedInt32, StepUp, FastUInt1..5, SlowUInt1..5); and scadabridge_site_connection_up counts open central→site gRPC subscriber streams, not data connections, so this row's original "0 on all six site nodes" was healthy steady state, not the fault (row 54). Close-out postscript: the seeding agent's live verification ran against worktree-homed data dirs (row 68), so after the orchestrator's redeploy from main the config was absent at boot and re-fetched by the startup reconcile — which exposed row 67 — and rig-probe-1 was re-verified live on the main checkout's data dirs (snapshot Good ×3, heartbeat ticking, sim values moving). |
Found during the R2 close-out live probe (2026-08-15) | Closed 2026-08-15. | — |
| 49 | 75520bee). All 576 resolved findings across the 25 component findings files were re-read for the row-45 generalization (a resolution that caches, hoists, or widens the lifetime of a DI-resolved service, later invalidated by a concurrency/lifetime feature). Nine matched or nearly matched the shape; NotificationOutbox-006 is the only one that ever became a bug and was already fixed as row 45. The other eight verify STILL-SAFE or SUPERSEDED-BENIGN — the two with real concurrency nearby each anticipated it explicitly (DeploymentService's dbGate serialises the one repository touch inside the bounded-parallel Send phase; ArtifactDeploymentService builds all per-site commands serially before fanning out, commenting "DbContext is not thread-safe"). An independent product-code cross-check confirms the pattern is absent: no IServiceScope is stored on a field anywhere in src/ (every long-lived holder takes IServiceScopeFactory and opens a scope per operation), every Task.WhenAll/Parallel.* site either touches no scoped service or opens a per-operation scope, and no scoped-registered service is captured by a singleton. One stale archive corrected: NotificationOutbox-004 was marked Resolved with an _Unresolved._ body; the real resolution is now recorded. Near-miss shapes recorded as rows 58–60. |
Found while fixing row 45 | Closed 2026-08-15. | — |
| 50 | 00a804e6). Both mitigations shipped, and they are complementary. (1) ValidateScopes/ValidateOnBuild in Development on both role branches via Host/ServiceProviderValidation.cs — written out rather than left to CreateBuilder's silent default; production stays unvalidated deliberately (fail-closed at boot). Switching it on surfaced two real Site-role defects: ReconcileService and AuditLogKpiSampleSource were registered on sites but need central-only repositories, so neither could ever be constructed there — one unconstructible descriptor fails ValidateOnBuild for the entire site graph, which is exactly why SiteHealthEndpointTests had quietly pinned UseEnvironment("Production") to dodge validation. Both moved to central-only registrations (AddCentralCommunication() / AddAuditLogCentralKpi()); the workaround is gone and the site graph now boots validated. (2) CaptiveDependencyGuard reflects over all 28 shipped actor classes and fails on any instance field (backing fields included) typed IServiceScope, DbContext, or Scoped-in-the-real-containers — the scoped set is derived from the built Central and Site graphs, so a lifetime change auto-re-scopes the check; IServiceProvider/IServiceScopeFactory fields are allowed (the correct pattern). Proven non-vacuous via permanent positive-control dummy actors. No live violation; the two test-only _injectedRepository ctor fields allowlisted with written safety arguments plus a staleness self-test (row 61). Doc: REQ-HOST-11 in Component-Host.md. |
Found while fixing row 45 | Closed 2026-08-15. | — |
| 51 | BundleLockedException reaches the CLI as "An internal error occurred"b449fe20). Relocated to Commons/Types/Transport/ beside BundleUnlockThrottledException (row 44's pattern); both consumers already carried the namespace, so no call-site edits. ManagementActor.HandlePreviewBundle/HandleImportBundle now rethrow it as ManagementCommandException, so CLI and UI print the lockout reason instead of a correlation id. Fault-mapping tests mirrored from row 44 on both handlers. Follow-up in the same diff (73f518d6): deleted the dead BundleUnlockRateLimitedException (superseded by row 44, zero references — three similarly-named unlock exceptions with one inert was a live trap). |
Found while fixing row 44 | Closed 2026-08-15. | — |
| 52 | 677263af). The trailing-hour window now lives in the central MS SQL BundleUnlockAttempt table — the HA store both central nodes already share, so no new infra, port or consensus. MaxUnlockAttemptsPerIpPerHour is a true fleet cap and survives a node restart; config keys, semantics and the rig's 500 override are unchanged, as are all row-44 behaviours (lockout checked first so locked bundles spend no budget; denials burn no strike and are not recorded; <unattributed> bucket; attempts-not-failures). Three recorded trades: fails open (store error → Warning naming client + cap, attempt permitted — availability of legitimate imports outranks a best-effort cap over an already-authenticated Administrator-gated surface; a process-local fallback was considered and rejected as it would still block an operator's legitimate 11th import); approximate (count-then-insert is not atomic across nodes; overshoot bounded by in-flight count, self-corrects within the window — row 64); opportunistic prune on the unlock path (CAS-claimed ≤ every 15 min per process, ≤1000 rows, retains 2× the window so a prune can't race a count — row 65) rather than a new background service. EF migration 20260815154803_AddBundleUnlockAttemptTable + hand-written production script docs/plans/sql/AddBundleUnlockAttemptTable.sql. Integration test proves two limiter instances over one shared store enforce one combined window, restart durability, and store-failure fail-open. |
Row 44 implementation note; documented in docs/components/Transport.md |
Closed 2026-08-15. | — |
| 53 | MaxConcurrentImportSessions has no observability372ea4df). Gauge scadabridge.central.transport.import_sessions.open on the ZB.MOM.WW.ScadaBridge meter — already in SiteServiceRegistration.ObservedMeters (the single shared allowlist source for both roles; no appsettings pins one), so no allowlist change was needed and the series is not silently dropped. Provider-backed like the S&F queue-depth gauge: registered by BundleSessionEvictionService.StartAsync, identity-checked-cleared on StopAsync. IBundleSessionStore.OpenSessionCount counts only unexpired sessions, so the gauge doesn't read "pinned at the cap" for a sweep interval after a burst drains. BundleSessionStore.Open logs Warning with open count, cap and TTL on a genuine cap hit only (a store full of expired sessions is reclaimed silently). No alertable rejection counter yet — row 66. |
Found while fixing row 43 — R6 could only infer the leak from rotating test failures | Closed 2026-08-15. | — |
| 54 | scadabridge_site_connection_up is misleadingly named (Low, observability)b5f120c3). Renamed to scadabridge.site.stream.subscribers.active (a test pins the new name AND asserts the old one is gone; doc/script references updated) and BOTH real data-connection gauges added: scadabridge.site.data_connections.connected / .total — the denominator ships with the numerator because neither alone separates "device down" from "nothing configured"; connected < total is the alert condition. Sourced from the DCL's four state transitions, deliberately NOT from the health report, so the metric survives a dead report pipeline (and stays clear of the concurrent row-55 fix in SiteHealthCollector). All on the already-allowlisted ZB.MOM.WW.ScadaBridge meter. Expected rig shape: connected==total on the active site node, 0/0 on the standby (healthy — connections open on the active node only). |
Row-48 close-out (2026-08-15) — the row's original mis-diagnosis traced directly to this | Closed 2026-08-15. | — |
| 55 | tagResolutionCounts reads {totalSubscribed:0, successfullyResolved:0} for a connection with 2 live Good-quality subscribed tags (Low)5da27e05). All three suspected reporting-path hypotheses were WRONG — live debugging showed the zeros and the good:2 on the SAME report from the SAME collector on the active node, ruling out standby shadowing, keying races, and the central merge in one stroke. The real bug is a DCL omission, not a regression: DataConnectionActor.HandleSubscribeCompleted — the first-time, everything-resolves-cleanly subscribe path — was the missing FOURTH UpdateTagResolution push site. BecomeConnected's legitimate 0/0 push fires before the stashed subscribe unstashes (proven in live logs: "Entering Connected state" follows the subscribe line), ScheduleTagResolutionRetry cancels outright at zero unresolved, and any reconnect masks the bug via the batch path — so the diagnostic was blindest on a perfectly HEALTHY connection, exactly when it should read clean rather than empty. One mirrored push + regression test (stash-verified fail-before: exactly one 0/0 call received pre-fix while the actor's own GetHealthReport already said 2/2, pinning the actor-correct/collector-stale asymmetry). Also corrected Component-DataConnectionLayer.md, which claimed the counts were "derived at report time" — they are pushed, and a cached push is only as fresh as its last push site. |
Row-48 live probe; root-caused live 2026-08-15 | Closed 2026-08-15. | — |
| 56 | 103f2f19). Premise refined on inspection: central already served /metrics on its web port (anonymous, no fallback policy) — what was missing was the dedicated 8084 listener the documented docker-exec scrape recipe targets. Both rigs' central appsettings now set MetricsPort, and the Host binds it as a PRESENCE check, not a defaulted read — NodeOptions defaults to 8084, so a defaulted read would make every existing production central surprise-bind a new port on upgrade, and port-in-use is a hard boot failure. |
Row-48 live probe | Closed 2026-08-15. | — |
| 57 | audit-config query has no CLI table projection (Low, cosmetic)b2cab59c). AuditConfigTableProjection mirrors TemplateTableProjection: unwraps the {entries, totalCount} envelope to rows (id, timestamp, user, action, entityType, entityName, entityId), drops the fat afterStateJson, adds --detail to skip projection (matching template list), and surfaces totalCount as a trailing note row (showing N of M — use --page/--page-size) emitted ONLY when the page is a strict subset, with no id cell so it can't be mistaken for an entry. The deprecated audit-log alias shares the command tree and gets the projector too. Tests serialize the REAL AuditLogQueryResult/AuditLogEntry types, so a server-side property rename fails the test instead of silently blanking a column. Stale-README follow-up recorded as row 76. |
Row-46 fix (2026-08-15) | Closed 2026-08-15. | — |
| 58 | IDeploymentStatusNotifier singleton retains delegates targeting circuit-scoped Blazor components (Low)2807b5a3). Still exactly one subscriber (Deployments.razor), but the cited pin was BLIND: Deployments_Dispose_UnsubscribesFromNotifier was satisfied by the _disposed guard alone — mutation-verified (with the -= deleted, the old test stayed green while the leak was wide open). Closed by two real pins: DeploymentsPushUpdateTests.Deployments_Dispose_ActuallyDetachesTheDelegate_NotJustGuardsIt (structural — the invocation list must shrink to 0) and EveryCentralUiSubscriberToTheNotifier_UnsubscribesAndIsDisposable (a source scan pinning this row's own trigger: any new subscriber without a matching Dispose unsubscribe fails the suite). Both fail on the mutation. |
Row-49 sweep (CentralUI-006 re-verification) | Closed 2026-08-17. | — |
| 59 | SiteCallAuditActor per-tick scope spans all sites × ≤50 upsert pages (Low) |
Row-49 sweep (SiteCallAudit-009 re-verification); re-verified 2026-08-17 (Low-tail sweep — loop still serial, line numbers drifted to SiteCallAuditActor.cs:528-545) |
Safe only because site reconciliation is a serial foreach … await; one scope across a parallel fan-out would be row 45 again. 2026-08-17: the missing in-code constraint remark was added at the loop (2807b5a3 — "serial by design … do not fan out"); a source-scan pin was judged too brittle, so the comment + this row remain the tripwire. |
ReconcileSiteAsync is ever fanned out with Task.WhenAll |
| 60 | ArtifactDeploymentService.GlobalArtifactSnapshot holds EF-materialised entities across the parallel per-site send (Low)2807b5a3) — with a premise correction so the row is never reopened on the wrong reasoning. The row's premise was factually wrong: GlobalArtifactSnapshot holds pure Commons DTOs, not EF entities (ArtifactDeploymentService.cs:170-212); the EF state that actually crosses the fan-out is the tracked Site entities the per-site lambda closes over. The safety conclusion still holds — commands are built serially before the fan-out and lazy loading is confirmed absent — and the load-bearing half of the trigger is now permanently pinned by ConfigurationDatabase.Tests.AddConfigurationDatabase_DoesNotEnableLazyLoading. |
Row-49 sweep (DeploymentManager-023 re-verification) | Closed 2026-08-17. | — |
| 61 | _injectedRepository test-mode constructors are one call site from re-introducing row 45 (Low)92fe5acc). Preferred fix done: both actors now have exactly ONE constructor (IServiceProvider); 12 test call sites folded onto stub providers via two TestSupport helpers. Also deleted as newly-dead: OnCachedTelemetryWithoutDualWriteAsync, SiteCallAuditActor's _backgroundTimersEnabled (a constant once the test ctors went), and the private SerializedRepository — moved into the test project, since production resolves a fresh repository per scope and the concurrency hazard existed only for fixtures sharing one instance. CaptiveDependencyGuard allowlist is now EMPTY; the staleness self-test replaced by two stronger pins — Allowlist_IsEmpty plus FormerlyAllowlistedActors_NowScanClean, which scans both actors with the allowlist DISABLED, so emptiness is evidence the fields are gone, not that the scan stopped looking. Control: re-adding a field fails both pins and the real scan. |
Row-50 guard allowlist | Closed 2026-08-15. | — |
| 62 | ISiteEventLogger off the root provider without a scope (Low)9d668861) — pinned, and the row undercounted: there are SIX sites. SiteEventLoggerLifetimePinTests (Host tests, registration-only inspection, no provider built) pins TWO facts — the interface registration is a factory forwarding to concrete SiteEventLogger, so pinning the interface lifetime alone doesn't prove callers share an instance; the concrete registration is pinned too, and flipping only the concrete to Scoped is caught ONLY by the second pin (verified both directions, registration restored). The pin's failure message names all six root-resolve sites: the five in this row plus SiteRuntime/Scripts/ScriptRunLauncher.cs:224 — the run body WP3.1 lifted verbatim out of the deleted execution actors, which inherited the pattern unnoticed. |
Row-50 guard allowlist | Closed 2026-08-16. | — |
| 63 | ASPNETCORE_ENVIRONMENT, so the row-50 Development validation never fires there (Low, rig config)0d2d6de6). Set on all six docker/ site containers and both docker-env2 site containers, mirroring the centrals; both compose files re-parsed to prove well-formedness; the now-stale Program.cs comment corrected. A captive dependency in the site graph now fails the container at boot on the rig, not just in CI. |
Row-50 implementation | Closed 2026-08-15. | — |
| 64 | The fleet-wide unlock throttle's count-then-insert is not atomic across central nodes (Low, accepted by design) | Row-52 implementation trade, documented in both Transport docs (re-verified 2026-08-17: Component-Transport.md:341, docs/components/Transport.md:140, plus the code XML doc) |
Simultaneous attempts can overshoot MaxUnlockAttemptsPerIpPerHour by the number in flight; bounded and self-correcting within the window. A conditional single-statement insert would close it at the cost of provider-specific raw SQL. |
A threat model needing a hard cap rather than a rate damper, or evidence of a real brute-force exploiting the overshoot |
| 65 | BundleUnlockAttempt rows are pruned only by nodes actively serving unlocks (Low)2807b5a3) — the revisit trigger had FIRED. Transport gained exactly the central scheduler the trigger named: BundleSessionEvictionService (central-only BackgroundService; AddTransport is called only from the central branch of Program.cs). The prune tick now rides it, so an idle cluster's rows age out without an unlock ever occurring; the opportunistic unlock-path prune remains. Operational note: the service now has two maintenance duties, deliberately in separate try blocks so neither can mask the other. Tests: FleetWideUnlockThrottleTests additions + BundleSessionObservabilityTests extension, both green. |
Row-52 implementation trade | Closed 2026-08-17. | — |
| 66 | 90ce0999). scadabridge.central.transport.import_sessions.rejected on the existing allowlisted meter, emitted ONLY at the post-eviction recheck — the cap is tested twice and a store full of expired sessions reclaims and succeeds, so counting the first test would fire routinely on any node whose sweep hadn't run; duplicate-session-id re-Open is also excluded. New tests use delta-over-scoped-action with unique per-test keys (the pre-existing gauge tests' absolute asserts are row 74). |
Row-53 implementation note | Closed 2026-08-15. | — |
| 67 | 482165cb). Root cause: InstanceActors are created from exactly two paths — the boot-time SQLite load in DeploymentManagerActor.PreStart and an explicit deploy/enable command — and the reconcile pass fires ~5s AFTER the boot load with a pure file write (StoreDeployedConfigIfNewerAsync) as its only side effect. Fix: reconcile collects the enabled instances it stored and Tells a new EnsureInstanceStartedCommand to the DM singleton proxy, so the actor is created under the singleton no matter which node's pass found the gap. Handler is advisory and idempotent: no-op if already running; re-reads the durable row (authority on config body AND is_enabled, so a disable landing after the fetch wins); defers to _deployWarms/_terminatingActorsByName/_inFlightDeployments so deploy-gate ordering and all-or-nothing-per-instance are untouched (it starts an already-persisted config, never applies one); quiet no-op on an absent row (normal on an unreplicated site whose peer healed its own file). The heal is no longer silent — a Warning site event Instance <X> started by startup reconcile self-heal fires. Fail-before by behavioral revert: 3 of 8 new tests fail pre-fix, including the exact live symptom (debug snapshot → instanceNotFound); the 5 negative guards (disabled / failed-fetch / unknown / idempotence) green in both states. SiteRuntime 613/613, Host 504/504. Sibling gaps found during the fix recorded as rows 72/73. |
Proven live during the rows-46–53 close-out (2026-08-15) | Closed 2026-08-15. | — |
| 69 | SQLite Error 11: 'database disk image is malformed' on every sync call (replication connected: false, backlog 892, faulted session retried ~every 60s) and, after the redeploy, site-a-a crash-looped (SIGABRT ×12) on the same error at SqliteLocalDb construction — the new boot-time re-registration reads pages the old build never touched, so site-a-a's corruption (present but latent) became fatal only on upgrade. Recovery: pair stopped together, both files quarantined (*.corrupt-2026-08-15, kept for forensics; 394 MB / 416 MB), fresh files reseeded — site-a-b via its startup reconcile (which live-proved the row-67 fix, see row 67) and CDC snapshot resync; converged to connected: true, backlog 0, with the row-27 schema change proving the digest handshake green. Suspected cause is host-level, not product: see row 79. |
Found live by the row-55 debug + this wave's close-out redeploy | Closed 2026-08-15. | — |
| 70 | auto-down-unreachable-after is deprecated — Akka warns "keep-majority will be used instead" on every boot (Medium info — verified NOT currently true, but a live removal risk) |
Row-55 live session; investigated + drill-verified at this wave's close-out | The warning is real (fires at Cluster start on every node, Akka.NET 1.5.62 — same version as the 2026-07-21 decision, so nothing changed underneath; it fires on the presence of the auto-down-unreachable-after key). Drill-proven that the substitution does NOT happen when downing-provider-class = "Akka.Cluster.AutoDowning" is EXPLICIT (as BuildHocon sets): DRILL_MODE=active kills central-a (oldest AND lowest address) — under keep-majority a 2-node even split keeps the lowest-address side, so surviving central-b would have downed ITSELF (total outage); instead central-b observed OldestChanged → became Oldest, took over all singletons, and the victim rejoined ready 2s after restart (2026-08-15 drill, post-merge rig). The 2026-07-21 decision therefore still holds exactly as documented. The residual risk is the deprecation itself: a future Akka.NET bump can delete AutoDowning, silently or loudly breaking the availability-first strategy. |
Any Akka.NET version bump — re-run DRILL_MODE=active bash docker/failover-drill.sh and re-read the release notes before accepting it |
| 71 | 8b761841). One narrowly-keyed Serilog Filter.ByExcluding in LoggerConfigurationFactory: Level == Error AND the DefaultHealthCheckService category AND HealthCheckName == "active-node" — check-NAME keying is what preserves the constraint (localdb/akka-cluster/database/required-singletons keep their Error lines, and active-node has no error path of its own on the active side, so the only suppressible event is a standby reporting it's a standby). Rejected on evidence: MinimumLevel.Override can only raise a floor (can't rewrite Error→Information), and reporting Degraded would change the HTTP status /health/active publishes, which is load-bearing for Traefik. Bonus reach: the CENTRAL standby was logging the same Error 2x as often (Traefik probes /health/active every 5 s) — covered by the same filter. Fail-before: filter disabled → StandbyActiveNodeVerdict_IsNotLogged fails; the 9 must-still-log constraint tests pass either way, proving narrowness. Side-finding: the observed ~10 s SITE-side cadence comes from an out-of-repo prober (nothing in this repo probes a site's /health/active), and CLAUDE.md's "site nodes map no /health/*" claim was stale — corrected. |
Row-55 live session (2026-08-15) | Closed 2026-08-16. | — |
| 72 | e606b24a, with row 78 — one fix, the DM instance sweep). Design decision made and recorded (docs/plans/2026-08-15-instance-sweep-self-heal-design.md): converge, don't merely warn — the sweep re-applies the stored config through HandleDeploy, the exact path a central redeploy uses, so the compile gate, warm-then-gate ordering, redeploy-supersede, static-override/native-alarm reset and all-or-nothing-per-instance are the production ones (a stop still does not cancel an in-flight run). Rationale: the stored row IS what central successfully deployed; surface-only leaves the operator to hand-issue the very redeploy the sweep would perform. Ordering hazards checked: the sweep never acts on an instance a deploy pipeline owns; a gate-rejected restart is inert, not partial; the synthetic deploy replies to Self, never NoSender (dead letters are a health metric). "Nothing surfaces it" is now false in BOTH paths — even a failed restart yields two deployment site events naming both revisions. Safety property beyond the brief: one repair attempt per (instance, revision) pair, so the self-heal can never become a restart loop for an init-crashing actor or a config the gate can never accept. Fail-before: sweep unarmed → Sweep_RestartsInstance_ServingAStaleRevision fails showing rev-1's value served while the store holds rev-2. |
Row-67 fix session (2026-08-15) | Closed 2026-08-16. | — |
| 73 | SiteReconciliationActor is silently skipped when IDeploymentConfigFetcher is null (Low)a2f0086f, with row 80). New tri-state health-report field StartupReconcilerActive (bool?, matching the LocalDbReplicationConnected nullable precedent; additive proto field 29, vendored C# regenerated via docker/regen-proto.sh centralcontrol with the csproj verified restored): null = not reported (central self-report or pre-row-73 site build), false = booted with no fetcher. Latched once at boot by the Host in both branches. The skip branch was raised Warning → Error (no configuration legitimately omits the fetcher on a site node) plus a best-effort deployment/Error site event so evidence outlives the boot log. Central treatment: a "No self-heal" Warning badge, deliberately not a failure — config still arrives via replication while the node is up; what's lost is recovery from a deploy landing while it's down. Only explicit false is badged (badging null would light the fleet the moment central upgrades ahead of the sites); offline sites skipped, matching the backlog badge. |
Row-67 fix session | Closed 2026-08-16. | — |
| 74 | BundleSessionObservabilityTests gauge tests assert absolute values — latent row-32-class race (Low, test-only)98c94588). Both gauge facts converted to delta-over-scoped-action via a ReadGaugeDelta helper — and two assertions got STRONGER: the post-clear check (Assert.Equal(0, …) — satisfied by any unregistered slot) now asserts opens on the detached store move the gauge by nothing (the actual claim), and the compare-and-clear fact gained a negative control (the superseded store's opens must NOT reach the instrument), without which it passed by coincidence whenever both stores held equal counts. Fail-before: a temporary foreign-provider probe returning 7 pre-start — old body fails Expected: 0 / Actual: 7, new body passes the identical scenario (probe deleted, not committed). Documented residual: a registrant clobbering the slot BETWEEN the two reads still needs row-32's mutual-exclusion pattern; cannot arise today (sole constructor of BundleSessionEvictionService in the assembly). |
Row-66 implementation (2026-08-15) | Closed 2026-08-16. | — |
| 75 | 40944a0d). The real defect was deeper than "red means delete": Approve and Reject rendered as IDENTICAL red buttons, so at the moment of a two-person-control decision colour carried no discriminating information — because danger was a two-state flag for a three-state concept. Fix: DialogSeverity (Normal→btn-primary, Caution→btn-warning, Destructive→btn-danger); Approve → Caution amber (executes a real write to live plant equipment — consequential, not destroying), Reject keeps Destructive red (terminal for the request; and holding it red keeps the two opposite decisions distinct — dropping it to neutral would just invert the bias). Only Destructive inherits row-28's "Delete" label fallback, or row 28 returns through the new door. Implemented as an ADDITIVE overload deliberately: a 5th optional parameter would silently un-match every 4-arg NSubstitute stub (the row-28 gotcha, null Task → NRE); the overload's only breakage was 6 hand-written fakes failing to COMPILE — the loud mode. DialogState.Danger became Severity (one field can't disagree with itself). Playwright .btn-danger selectors stay valid (Destructive mapping unchanged). Fail-before: call site reverted to danger: true → both Approve pins fail; Caution→btn-danger remap → amber styling pin fails. Decision recorded in Component-CentralUI.md. |
Row-28 audit (2026-08-15) | Closed 2026-08-16. | — |
| 76 | audit-config query JSON sample omits afterStateJson/bundleImportId (Low, docs)4ae57bdb). Sample now shows both fields with their nullability semantics, and was grown to two entries mirroring the table-output sample below it row-for-row — which is what makes the "the table drops these two fields" sentence checkable. |
Row-57 fix (2026-08-15) | Closed 2026-08-16. | — |
| 77 | ActorCell.Actor right after ResolveOne is unordered vs child construction (Low, test-infra)tests/ sweep: ActorCell (1 hit — the already-fixed file's comment), ActorRefWithCell (1 — fixed file :135), .Underlying (1 — same line), BindingFlags near actor/cell (2 — same file), .Cell (0), ResolveOne (3 files: the fixed one; SiteAlarmLiveCacheServiceTests:432 resolves-then-Sys.Stops, benign; ReconcileStartsInstanceTests:363 uses the ref only for IActorRef equality, benign), UnderlyingActor (6 files — all Akka.TestKit TestActorRef<T> via ActorOfAsTestActorRef, an ORDERED mechanism, not this pattern). The unordered idiom exists in exactly one place and already carries the bounded-deadline retry. |
Row-40 session (2026-08-15) | Closed 2026-08-16. | — |
| 78 | e606b24a, with row 72 — the DM instance sweep). A periodic sweep on the Deployment Manager singleton (ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds, default 60, validated >= 0, 0 disables — default is live everywhere with no rig config change, which is intended): each tick re-reads the durable rows off-thread and, on the actor thread, starts any enabled stored config with no running actor. Arrival-path-agnostic by design — covers replication seeding, post-boot snapshot resyncs, and any future path. A LocalDb change hook was considered and rejected on evidence: it covers only the replication path and only the replicated half of the fleet (site-b/c aren't), AND ILocalDb 0.3.0 exposes no change-notification API at all, so a hook means a cross-repo upstream capability to close a site-local gap. Warning site event on a real heal names the sweep as the trigger; silent when there is nothing to do. Fail-before (sweep unarmed): 5 of 10 new tests fail including the exact live symptom ("A replication-seeded config for an enabled instance left no running Instance Actor"); 5 negative guards (disabled / in-flight-deploy deferral / matching-revision no-op / silence / interval-0) green in both states. Design note: docs/plans/2026-08-15-instance-sweep-self-heal-design.md. LIVE-PROVEN at close-out (2026-08-16) in the exact original failure sequence: site-a pair stopped, node-a's LocalDb file quarantined, pair restarted together — node-a's 08:45:55 reconcile reported "0 fetched" (replication had seeded the fresh store; central rightly saw no gap — the precise side door), and at 08:46:50, one sweep interval later, BOTH enabled instances started with the Warning Instance rig-probe-1 started by deployment-manager sweep self-heal — its config is present in the deployed-config store but no Instance Actor was running until now; debug snapshot live with no extra restart. The probe also exposed the DCL sibling gap — row 85. |
This wave's close-out (2026-08-15) | Closed 2026-08-16. | — |
| 79 | docker/ nodes + 2 docker-env2/ site nodes) moved to named docker volumes (scadabridge-<node>-data / scadabridge-env2-<node>-data, explicit name:s in both compose files), seeded from the old bind dirs with both rigs stopped, then redeployed and live-verified: site-a localDbReplicationConnected: true / backlog 0 over the migrated files, deployed instances resubscribed on boot (data continuity proven), env2 stable with 0 restarts. The trade-off that had deferred this — losing host-side inspectability — turned out to be mostly moot: OrbStack exposes every volume on the Mac at ~/OrbStack/docker/volumes/<name>/ (read-only inspection; never write through it). The rebuild also surfaced latent env2 drift: env2 was running a pre-Secrets-0.6.2 image, so on the fresh image all four env2 app nodes failed the new store-path validation at boot — fixed by porting the main rig's x-secrets-store-env posture (Secrets__SqlitePath=/data/… + /data second mounts, incl. NEW env2 central data volumes; env2 centrals previously kept data on the ephemeral overlay). Residual accepted: failure mode 2 (VM idle-sleep, orb restart docker recovery) is availability-only and unaffected — data now survives it on ext4. Hazard history + recovery runbook retained in docs/deployment/topology-guide.md (section updated). Old docker/<node>/data/ dirs left in place as dormant pre-migration backups. |
This wave's close-out (2026-08-15); runbook 2026-08-16; migration executed 2026-08-18 | Closed 2026-08-18. | — |
| 80 | LocalDbReplicationConnected: false / LocalDbOplogBacklog: 0 — contradicting the documented nullable semantics (Low-Medium, observability correctness)a2f0086f, with row 73). Root cause was type-level: ISiteHealthCollector.SetLocalDbReplicationStatus(bool …) made null INEXPRESSIBLE for the connected flag, and the Host wired the bridge unconditionally — the replication engine is registered on every site node and idles answering false/0 with no peer. Fix mirrors the SiteEventLogReplicationCheck precedent exactly: the Host evaluates SiteLocalDbSetup.ReplicationIsConfigured once and passes it as a third provider to AddLocalDbReplicationHealthBridge (the SAME captured value now also feeds SiteEventLogReplicationCheck, so the two surfaces can't disagree); unconfigured short-circuits to (null, null); the collector widens to bool?. The rest of the chain needed NO change — proto already BoolValue/Int64Value wrappers, NextBacklogAboveSince already leaves the anchor unchanged on null, KPI source already conditional: row 33's machinery was correct and had simply never received a null (its null branch was dead code in production until now). false now means exactly one thing: configured, peer session down = real fault. Trap that produced the bug, now pinned in tests: configured-ness is PeerAddress OR ApiKey — "no PeerAddress" is NOT "unreplicated" (a passive-half node carries the key alone); the pre-existing Host fixture asserting false/0 was exactly such a node and its assertion is CORRECT — kept with an explaining comment, plus a new genuinely-unreplicated fixture. Fail-before: gate behaviourally reverted → UnreplicatedSite_HealthReport_ReportsNullForBothLocalDbFields fails Expected: null / Actual: False. |
Found at the previous wave's close-out verifying row 33 | Closed 2026-08-16. | — |
| 81 | ddb3c32e) — but NOT via the row's proposed dbstat fix, which is impossible on this stack. Probed BEFORE implementing: the bundled SQLite 3.53.3 has no ENABLE_DBSTAT_VTAB (pragma_module_list returns 10 modules, none of them dbstat/sqlite_dbpage); pinned by SqliteCapabilityTests, written to FAIL if a future bump enables it (row 94). Switching SQLite builds rejected — the provider is shared with ZB.MOM.WW.LocalDb fleet-wide. The cap instead measures site_events + its four indexes via a sampled page-layout model: row width sampled 500 rows from each end per tick (octet_length), whole-page table cost (linear models under-estimate near-half-page rows by ~40%), clamped by whole-file logical size (a true upper bound). Index pages count against the cap (~34% of footprint — the 1 GB default was sized against a figure that included them). Measured model error −0.8%/+8.4%/+12.8%, biased high on purpose; cross-checks row 34's independent 393 B/row at 397.2. Guard semantics re-based: (1) unsampleable row width ⇒ refuse to trim (checked BEFORE the cap comparison); (2) the old "size did not decrease" guard now runs on the table-scoped estimate, so replication bookkeeping can no longer trip it — it now means exactly one thing: events arriving as fast as the trim removes them. Bytes-per-row held fixed per tick (re-sampling per batch would jitter the estimate through guard 2 — the same self-defeating measurement as the row itself). LocalDb bookkeeping explicitly NOT managed here (MaxOplogRows/tombstone retention own it); topology guide gained a volume-sizing subsection stating the honest budget. Fail-before: reverting to whole-file semantics fails 5 new tests and reproduces the exact original signature (rows destroyed, cap still unmet). SiteEventLogging 104/104, Host 520/520. No config keys added or renamed. Decision memo: docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md. Model limitations recorded as rows 92/93. |
Row-34 measurement harness; §5 of the measurement doc (postscript appended) | Closed 2026-08-17. | — |
| 82 | AuditLog.Tests flakes under parallel full-solution runs — MsSqlMigrationFixture contention (Low, test-infra)da23f2dc) — three mechanisms, and the row's "per-run database name" prescription was already in place (the fixture has minted ScadaBridgeAuditMigTest_<guid> per class since it was written). (1) The headline fingerprint was create+migrate SATURATION: nothing bounded how many fixtures ran CREATE DATABASE + the ~100-migration chain at once (measured peak: 45 concurrent test DBs / 19 blocked requests / 98 sessions on one container), pushing DDL past EF's default 30 s command timeout — every observed failure was ..ctor → MigrateAsync → SqlException 258 Execution Timeout Expired, which xunit reports against every test in the class (hence "29 failures"). Fix: a machine-wide counting semaphore on sp_getapplock in master (ScadaBridgeTestMigrate/0..N-1, 4 slots, SCADABRIDGE_MSSQL_TEST_MIGRATE_SLOTS override) held across create+migrate — the arbiter must live on the shared SERVER because the contention is cross-process; gate connection is deliberately UNPOOLED (a pooled one defers the session-lock release until reuse, stranding a slot); crashed process ⇒ SQL releases the lock; a gate deadline miss proceeds UNGATED so the throttle can never itself be a red. Timeouts raised to 300 s migrate / 120 s create-drop (bounded queue = busy but progressing); once-per-process sweep of orphaned ScadaBridgeAuditMigTest_* >2 h; connectivity probe retries 3× (a busy server misread as absent flips every MSSQL test to a green-looking Skip — worse than a red). (2)+(3) Two more flakes behind it: PartitionPurgeTests and NotifyDispatcherAuditTrailTests LEAKED 100 ms-tick actors (purge / outbox dispatcher) whose only stop was timeout-bounded TestKit.Dispose — a leaked dispatcher even delivered the NEXT test's notification through its own exhausted adapter, stranding it Retrying. Fixed with Watch/Stop/ExpectTerminated in the creating test. Evidence: 10× concurrent ConfigurationDatabase.Tests went from 10/10 processes failing (17–77 failures each) to 3 consecutive 10/10-green runs; the AuditLog 3×+1× mix from failing to 3 clean runs; standalone AuditLog 383/383, ConfigurationDatabase 395/395 (no runtime regression), SiteCallAudit 66/66 (inherits the fixture); zero orphan DBs or stuck app locks left. Host.Tests/PlaywrightTests hit the PERSISTENT DB on an already-serialized axis — untouched. Systemic residue recorded as rows 95/96. Doc: docs/test_infra/test_infra_db.md. |
Found independently by two agents in the 2026-08-16 wave | Closed 2026-08-17. | — |
| 83 | ReconcileStartsInstanceTests still uses a bare Task.Delay(500) (Low, test-infra)524bdbbe). ReconcileStartsInstanceTests converted to the first-SetInstanceCounts-report gate; the pattern was extracted into a shared TestSupport/DeploymentManagerStartupProbe now used by both this suite and InstanceSweepSelfHealTests, so the convention has one implementation to copy instead of two to keep in sync. SiteRuntime 633/633 green. |
Row-78 fix session | Closed 2026-08-17. | — |
| 84 | LogReplicationBacklogNote can fire twice in one purge tick (Low, cosmetic)ddb3c32e). The note is emitted at most once per tick. Fail-before: RunPurge_WhenBothPhasesDeleteRows_LogsTheBacklogNoteOnceForTheWholeTick fails pre-fix with Assert.Single(): collection contained 2 matching items. |
Row-34 session | Closed 2026-08-17. | — |
| 85 | 8f2c2813) — with a premise correction AND a widened blast radius. The row's mechanism was wrong: data_connection_definitions has NO production reader (its getter is test-only) — connection actors are created solely from the Connections map in an instance's own flattened config via EnsureDclConnections, so there was never a store→actor path to repair. The real defect: that ensure was the CALLER'S job and only 2 of 4 instance-start paths did it (boot batch ✅, ApplyDeployment ✅, StartStoredInstance ❌ — both the row-67 ensure-started and the row-78 sweep — and HandleEnableResult ❌; the two misses are exactly the two paths added later). So the bug was NOT replication-specific: an ordinary unreplicated site that boots with an instance disabled and enables it later hit the same dead end, because the boot load reads only enabled configs. Fix is structural: EnsureDclConnections moved INTO CreateInstanceActor (the documented single creation funnel), before the already-exists guard so the two working paths keep exact semantics; reuses the per-name hash guard so steady state is silent; a fifth path cannot forget. Store-level sweep pass explicitly REJECTED (memo §7.4): it wouldn't have fixed this (the table is empty on deploy-only sites) and the unguarded form ping-pongs the hash slot against the deploy path, dropping live subscriptions every tick. Fail-before by behavioral revert: 4 of 7 new tests fail (sweep-started, reconcile-ensure-started, enable-after-disabled-boot, exactly-once-across-sweeps). SiteRuntime 633/633, DCL 320/320. Sibling gaps found during the fix recorded as rows 89/90. |
Proven live 2026-08-16 by the row-78 close-out probe (sweep started rig-probe-1/soak-notify-1 but every tag subscribe failed for 6+ min with Unknown connection: OPC PLC Simulator — will retry; a second pair restart healed it) |
Closed 2026-08-17. Live-proven on the rig same day in the exact original failure sequence (pair stopped, node-a site-localdb.db quarantined as .probe85-quarantine-2026-08-17, pair restarted): reconcile "0 fetched" at 08:29:59, then at the 08:30:54 sweep tick Created DCL connection OPC PLC Simulator fired ALONGSIDE the two self-heal Warnings (pre-fix they appeared alone), zero No connection actor/Unknown connection lines on either node, all three rig-probe-1 tags Good with live values ~2 min after restart — NO second restart; steady-state silence confirmed (0 Created/Updated lines across subsequent ticks); replication reseeded (localDbReplicationConnected: true, backlog 0), site-b/c still correctly omit the fields. |
— |
| 86 | ad0100f2, agent commit 9f67147e) — Administrator = system-wide is the intended semantic; the UI was the defect. The rule was already documented on BOTH sides (ManagementActor.cs ~:522 "Users with the Administrator role, or system-wide Deployer, are not restricted"; SiteScopeService's own XML doc "any Admin/Design user — is system-wide") and enforced at 6+ ManagementActor sites — only SiteScopeService.ResolveAsync ignored it. It now treats an Administrator as system-wide regardless of SiteId claims, read via explicit FindAll(RoleClaimType) + OrdinalIgnoreCase (NOT IsInRole, which compares ordinally and needs a roleType the harness identities don't set); an Administrator's site claims aren't collected at all, preserving system-wide ⇒ empty permitted-id set. All 15 UI consumer call sites funnel through ResolveAsync (surveyed). Fail-before verified: 6/6 new Administrator pins failed with only SiteScopeService.cs reverted; DesignerWithSiteClaims_IsStillScoped pins that the bypass was NOT widened to Designer (the XML doc's "Design user" claim was inaccurate — corrected). |
Found by the 2026-08-17 Low-tail sweep; fixed with the row-37 consolidation | Closed 2026-08-17. | — |
| 87 | ad0100f2, agent commit 9f67147e), with the row-37 consolidation. Site grants now ride as SiteIdScope resolved inside the query's join — the whole-table GetAllInstancesAsync read is gone (pinned by ..._ReadsNoInstancesAtAll, a strictly stronger replacement for the old "bulk load exactly once" pin, consciously migrated). An unparseable permitted-site id is dropped (narrows the scope) rather than discarding the grant (which would widen to fleet-wide). PageSize <= 0 no longer means unlimited: it takes the CLI's own default 50, capped at MaxPageSize 1000. |
Found by the 2026-08-17 Low-tail sweep; fixed with the row-37 consolidation | Closed 2026-08-17. | — |
| 88 | --status silently means "no filter"ad0100f2, agent commit 9f67147e). An unrecognized non-empty status now produces a ManagementError naming the valid values (row-46 principle: loud failure over silent empty/full results) — enforced actor-side so HTTP API callers get it too. Case-insensitive parse retained; numeric strings rejected via Enum.IsDefined (Enum.TryParse accepts "7"). CLI help text documents the valid set; deliberately NOT AcceptOnlyFromAmong, whose case-sensitive matching is a live minor defect on audit query --status (row 99). Live-probed on the rig: deploy status --status Sucess → Unknown deployment status 'Sucess'. Valid values: Pending, InProgress, Success, Failed. Postscript 2026-08-17 (Wave 6, 16056f8e): the "numeric strings rejected via Enum.IsDefined" claim was WRONG — Enum.IsDefined(parsed) is true for any valid ordinal, so --status 2 silently resolved to InProgress. Fixed with the same LooksNumeric first-character guard the row-99 strict parser uses; live-probed: deploy status --status 2 now fails loud naming the valid values. |
Found by the 2026-08-17 Low-tail sweep | Closed 2026-08-17. | — |
| 89 | 6cfb6a79, agent commit c57e74e6) — premise corrected TWICE, then fixed. (1) The standby's DCL is EMPTY, not "serving the old endpoint": DeploymentManagerActor (role-scoped singleton) is the sole sender of CreateConnectionCommand and only populates its own node's dcl-manager. The real defect was create-time and not replication-specific: _createdConnections is per singleton incarnation, and data_connection_definitions had NO production reader — so any restart/failover rebuilt every connection from the flattened-config snapshots and silently discarded newer connection-only artifact deploys (identical loss on an unreplicated single-node restart). (2) Arbitration decided and shipped: the definitions store wins wherever it has an opinion; flattened config is authoritative only for names the store never owned (deploy-only sites unchanged — memo §7.4's objection). A _connectionDefinitions overlay refreshes from ReadStoreSnapshotAsync at boot AND every sweep tick, applied before instance creation; ping-pong killed by construction (every ensure path resolves the winner before hashing, so one config per hash slot). Fail-before: 5/8 new DclConnectionArbitrationTests fail with the overlay inert. Decision memo: docs/plans/2026-08-17-dcl-connection-arbitration-decision.md (records the accepted residuals: re-adds land at next instance start, not instantly — mirrors the removal's own "bad quality until redeploy" semantic, pinned by test). The discovery that a live connection actor is never reconfigured in place on ANY node is a separate pre-existing defect → row 97. Live rig probe at wave close-out (post-redeploy): the singleton node logged exactly ONE boot-time Created DCL connection OPC PLC Simulator with zero update churn across later sweep ticks (no ping-pong from the overlay), the standby stayed silent (topology as corrected), all three rig-probe-1 tags Good, site-a replication connected / backlog 0. |
Found by the 2026-08-17 row-85 fix session | Closed 2026-08-17. | — |
| 90 | 6cfb6a79, agent commit c57e74e6), with row 89. Both removal paths now share RemoveConnectionCentrally + an in-memory tombstone that EnsureDclConnections honors — which also closed a pre-existing bug where an artifact removal was undone by the very next instance start. The sweep detects a previously-definition-sourced row's disappearance via its own overlay bookkeeping. Accepted residuals, documented in the decision memo: tombstones are not durable across a restart (by decision — matches the long-standing restart-resurrection semantic), and a delete is undetectable on a node that never adopted the row (the stronger "non-empty table ⇒ complete set" inference would stop live connections on a guess; deliberately not taken). |
Found by the 2026-08-17 row-85 fix session | Closed 2026-08-17. | — |
| 91 | SiteScriptCompileCacheTests.GetOrAdd_SameCodeAndGlobals_ComputesOnce is parallel-load fragile106e467d, agent commit 26e3a1b9) — row-74 pattern applied (exemplar 98c94588), and wider than the named test. Two habits: a per-test-instance guid key token (no other class can collide with or serve this test's entries, making factoryCalls/Assert.Same exact and owned) + one-sided bounds on the process-global counters (>= baseline + n; safe because all Clear() callers share the serialized collection). Four assertions changed, two got STRONGER (GetOrAdd_SameCode_DifferentGlobals had a second absolute pair; Overflow_StaysWithinTheBound gained an eviction probe over its own keys — a slack bound is not proof a sweep ran; its ForeignEntrySlack = 512 is a judgement call, the eviction probe is the load-bearing assertion). Sibling absolutes in the same class and in DeploymentWarmThenGateTests fixed too. SiteRuntime.Tests run twice at full parallelism, 633/633 both. The underlying process-global Clear() cross-talk hazard remains → row 100. |
Found by the 2026-08-17 row-85 fix session | Closed 2026-08-17. | — |
| 92 | The event-log cap's size model is sample-based and blind to mid-history width shifts (Low, accuracy limitation — documented in memo §3.1) | Row-81 fix session (2026-08-17) | Row width is sampled 500 rows from each END of site_events; a stream whose row width changes sharply mid-history is mis-sized until the trim reaches that region. Self-correcting per tick (each trim exposes new tail rows to the sample), and the model's deliberate +8–13% high bias absorbs modest shifts, but a pathological width profile could over- or under-trim for several ticks. Reviewed 2026-08-18: accepted by design — inherent to any sampled model, not closable by code short of the row-94 dbstat replacement; the row-93 schema pin does not (and cannot) cover it. |
First field report of the cap trimming visibly wide of its target, or the row-94 dbstat replacement (which retires this row with the model) |
| 93 | site_events schemaSiteEventsSchemaCalibrationTests (SiteEventLogging.Tests) pins the exact shape the calibration assumed: the eight-column list the sampler's octet_length sum covers, the index set as exactly the four secondary indexes IndexedColumnCount = 4 charges plus the PK autoindex (absorbed into the measured fill factors), and each index's single covered column. Any site_events schema migration — new column, fifth index, composite widening — now fails the suite with a message directing to the memo §4 recalibration and the constants + sampling-SQL update, instead of silently mis-sizing the cap. Suite green 110/110. |
Row-81 fix session (2026-08-17); pinned by test 2026-08-18 | Closed 2026-08-18. | — |
| 94 | SqliteCapabilityTests is a deliberate tripwire — do not "fix" it by weakening the assertion (Low, standing note) |
Row-81 fix session (2026-08-17) | The test FAILS BY DESIGN if a future SQLite bump ships dbstat/sqlite_dbpage support. The correct response at that point is to REPLACE the sampled page-layout model with exact dbstat measurement (retiring row 92 — and the row-93 schema pin SiteEventsSchemaCalibrationTests, added 2026-08-18, goes with the model too), not to relax the test. The test's own comment says the same. |
SqliteCapabilityTests fails after a SQLite/provider bump |
| 95 | 106e467d, agent commit 34a8079d). All three classes live in AuditLog.Tests (premise correction: none in SiteCallAudit.Tests). Watch/Stop/ExpectTerminated replicated from da23f2dc at every creation site: 11 in AuditLogPurgeActorTests, 10 in SiteAuditReconciliationActorTests, 2 in OutageReconciliationTests. Second premise correction: OutageReconciliationTests DID call Sys.Stop — but fire-and-forget AND from the last line of the await using block, so an in-flight tick could still be writing to MSSQL during read-back and any earlier assertion failure skipped the stop entirely; stops moved before the read-backs, plus the AuditLogIngestActor that test 2 was discarding outright (holding a disposed provider) is now stopped too. Known lesser sibling habit left in place: EventStream probes are never unsubscribed (bounded by the per-test ActorSystem — noted, not a row). AuditLog.Tests 383/383. Postscript 2026-08-19 (arch-review P8-6): the premise correction was itself WRONG — SiteCallAudit.Tests DOES leak 100 ms-tick actors. SiteCallAuditPurgeTests (4 creation sites incl. a discarded Sys.ActorOf) and SiteCallAuditReconciliationTests (11 sites, two in one test) had ZERO Watch/Stop/ExpectTerminated. Reopened and re-closed 2026-08-19 (agent commit 102e7a2b): row-82 treatment via a per-file StopTicker helper (try/finally, blocked test doubles released first) at every site; the same sweep also fixed PendingDeploymentPurgeActorTests.cs:90 in Communication.Tests (same class), capped Communication.Tests with the row-96 xunit.runner.json (maxParallelThreads: 4 — it was the last uncapped TestKit+Kestrel suite), collection-serialized the two aggregators.active gauge-moving classes (row-102 shape), and reworked 11 sleep-then-assert-absence sites in SiteAlarmAggregatorActorTests per row-40 (AssertHoldsFor polling helper; settle-sleeps before actions left alone). Both suites 3× green (717/717, 66/66), zero flakes observed. |
Row-82 fix session (2026-08-17) | Closed 2026-08-17; reopened + re-closed 2026-08-19 (arch-review P8-6). | — |
| 96 | 106e467d, agent commit d2d4db49) — policy made and shipped: per-suite caps for the resource-heavy suites, pure unit suites stay uncapped. The four MSSQL-fixture users (ConfigurationDatabase.Tests, AuditLog.Tests, SiteCallAudit.Tests, Host.Tests — verified before capping) get xunit.runner.json with maxParallelThreads: 4, deliberately matching row-82's sp_getapplock slot default so the in-process and cross-process limits agree; no collections serialized. IntegrationTests (already fully serialized) and CentralUI.PlaywrightTests (already maxParallelThreads: 1, stricter than needed) left alone. Cap verified live (216% CPU uncapped vs 145% capped on ConfigurationDatabase.Tests); cost is nil — these suites are latency-bound on MSSQL/sockets (14–25% CPU), durations unchanged within noise. Policy documented in docs/test_infra/test_infra.md → Test Parallelism Policy (two-axis model, per-suite table, default for new projects), cross-linked from test_infra_db.md. Found-not-fixed: StoreAndForward.Tests/xunit.runner.json is a no-op (pins the default parallelizeTestCollections: true) — harmless, tidy on next touch. |
Row-82 fix session (2026-08-17) | Closed 2026-08-17. | — |
| 97 | 9dbda4a2, agent commit 7c1268d8) — the memo's §6 sketch shipped, plus two things the sketch missed. EnsureDclConnection now has three outcomes off its one hash slot: unchanged → nothing; new → create; changed → ReconfigureConnectionCommand (arbitration/tombstones byte-for-byte untouched). The manager forwards to the owning actor (unknown name falls back to create); DataConnectionActor handles it in all three lifecycle states and never stashes it, via a new shared SwapAdapter helper now also used by both failover sites. The sketch's two gaps, found in implementation: endpoint selection must be re-armed to Primary (a connection sitting on its old backup would otherwise dial the new backup first), and the deliberate link drop must suppress the unstable-disconnect accounting (else the swap itself could trip a failover). Also a premise-sharpening: "reconfigure while mid-connect" is the primary case, not an edge — a wrong endpoint loops in Connecting forever and never reaches Connected. Fail-before: 5/7 new DataConnectionReconfigureTests + 3 SiteRuntime ensure-path tests fail on the reverted behavior. Live rig probe (post-redeploy, round trip): connection 1's config changed (operationTimeoutMs 15000→16000→back) fired exactly TWO in-place reconfigures on the active node — Reconfiguring DataConnectionActor for OPC PLC Simulator in place → Reconnected successfully on Primary endpoint in 45 ms → Re-subscribing 2 tags — with all rig-probe-1 tags Good after each, the standby silent (0 reconfigure lines, topology as designed), and the old dead-end "has a new configuration" Warning never appearing. Rig config restored to original. |
Found by the 2026-08-17 rows-89/90 fix session; memo docs/plans/2026-08-17-dcl-connection-arbitration-decision.md (§9 postscript appended) |
Closed 2026-08-17. | — |
| 98 | EnforceSiteScopeForInstance is fail-open for nonexistent instance ids152ec85e, agent commit f33f25a1) — premise corrected: the defect had FIVE instances, not two, and the worst three were worse than "harmless". Beyond the two named helpers, the same if (target != null) EnforceSiteScope(...) shape sat inline in HandleGetInstance (:879), HandleGetSite (:1702) and HandleGetDataConnection (:1844) — and those returned ManagementSuccess{JsonData=null} for a nonexistent id vs ManagementUnauthorized for an out-of-scope one, i.e. a working id-enumeration oracle across the whole fleet for a site-scoped caller. All five now route through one shared DenyUnresolvedTargetIfScoped: a scoped caller's unresolvable target is DENIED, indistinguishable from out-of-scope (no existence leak); system-wide callers (Administrator / system-wide Deployer) are untouched. Also checked: Instance.SiteId is non-nullable int, so no secondary fail-open path through EnforceSiteScope(null). Fail-before: 5 of the 16 new ManagementActorTests fail on the old code. |
Found by the 2026-08-17 rows-37/86-88 fix session (Agent A survey) | Closed 2026-08-17. | — |
| 99 | audit query --status client-side validation is case-sensitive152ec85e, agent commit f33f25a1) — premise corrected: "the server would accept it" was FALSE, and removing the client gate alone would have been a regression. The server-side parser (AuditQueryParamParsers.ParseEnumList) is documented-and-tested lax — it silently drops unparseable values and collapses to null (filter unconstrained), i.e. exactly the row-46/88 silent-wrong-answer trap. Fix shipped both halves: seven AcceptOnlyFromAmong gates removed across audit query/audit export (--channel/--kind/--status on both, plus export --format, which rejected CSV the server lowercases anyway), valid sets moved to help text, and a new strict ParseEnumListStrict (throws naming the valid values; LooksNumeric guard rejects ordinals) wired into both ManagementService audit endpoints. The lax parser deliberately survives for the CentralUI drill-in/export query strings (bookmarked fragments must render, not fault — documented as non-reconcilable). Left alone with reasons in code: `--direction up |
down(no server parse to defer to),--format json |
table(CLI-local), NotificationCommands' literals (unsurveyed). Four pre-existing tests were passing *vacuously* (no-credentials exit codes) and now pin the real contract; fail-before 21/26. **Found-and-fixed follow-on (orchestrator,16056f8e):** row 88's own ParseDeploymentStatus accepted ordinals (--status 2→ InProgress) — sameLooksNumericguard applied + 3 tests. **Live rig probes (post-redeploy):**audit query --status deliveredreturns rows;--status Nonsenseand--status 3both fail loud naming the 8 valid values;deploy status --status 2fails loud naming the 4 valid values. **Postscript 2026-08-19 (arch-review P6-3): the strict parser this row shipped was itself fail-open —Enum.TryParsesplits and ORs comma-joined names inside ONE token for any enum (not just[Flags]), and dense 0-based AuditStatuslands composites on defined members:**--status Forwarded,Attempted` = 1 |
2 = 3 = Delivered (silently wrong data), Submitted,Parked = 0 |
| 100 | SiteScriptCompileCache.Clear() is process-global — test classes wipe each other's entries8a9a8d29, agent commit 84b405ff) — premise corrected: there were ZERO production callers of Clear() (the only production entry point is ScriptCompilationService.CompileCore → GetOrAdd; Clear() existed solely for tests), so neither register option was taken as written — Clear() was deleted outright and the cache became an instance-scoped sealed class owned by ScriptCompilationService. The N1 "one compile per body per process" property now rests on AddSiteRuntime registering that service as a DI singleton — a lifetime that became load-bearing, so new SiteRuntimeServiceRegistrationTests pins it (a Scoped/Transient downgrade would reintroduce WP3.1's recompile storms with no behavioral test failing). The 37 test sites doing new ScriptCompilationService(logger) each silently get their own cache — isolation for the whole assembly, not just the row-91 classes. Row-91 workarounds RETIRED, not preserved: serialized collection removed from all three classes, exact Hits/Count assertions restored, guid key token gone, ForeignEntrySlack 512→8; new SeparateCaches_DoNotShareEntriesOrCounters pins the scoping. SiteRuntime.Tests green three consecutive full-parallel runs. |
Found by the 2026-08-17 row-91 fix session | Closed 2026-08-17. | — |
| 101 | ApplyingArtifacts_RemovedDataConnection_StopsLiveDclConnection flaked once — suspected overlay-sweep read racing the artifact-apply write0d3a90a9) — hypothesis VERIFIED, and the production consequence was WORSE than the row guessed. The race is real: the snapshot is read off the actor thread but compared against the overlay only when the piped message lands, so a read starting before an artifact apply's store writes can land after HandleApplyArtifactDataConnectionsToDcl populated the overlay — its stale (even empty) snapshot then "proves" the just-applied names disappeared. Overlay iteration order explains the exact fingerprint (PlcA removed first). Two corrections to the row's guesses: (1) it is NOT "self-healing at the next tick" — the removal evicts the hash slot the overlay's push is gated on, so the next fresh sweep clears the tombstone but re-creates nothing; the stopped connection stays dead until an instance redeploy; (2) the mid-write "momentarily not seeing the row" framing was the lesser half — the whole-snapshot staleness (read-before-write, land-after-apply) is the dominant window. Fix: _definitionsWriteGeneration, bumped on the actor thread by every artifact apply and captured at every snapshot-read dispatch (boot load + sweep); a result carrying a stale generation is discarded WHOLE — additions included, since a stale row pushed at a live connection would reconfigure it backward and flip-flop with the next fresh read. Deterministic repro fails before the fix (StaleSweepSnapshot_RacingAnArtifactApply_DoesNotRemoveJustAppliedConnections); FreshEmptySweepSnapshot_StillRemovesDisappearedConnections pins that row-90's legitimate disappearance still fires. SiteRuntime.Tests 647/647; rig redeployed, zero spurious removals/discards at boot, all rig-probe-1 tags Good. Memo §10 records the mechanism. Postscript 2026-08-18 (merge 505f3569, agent commit f02e1166): extended to the WHOLE snapshot. The instance-config half had the same TOCTOU, and all three hazards were PROVEN real pre-fix: a stale sweep restarts an instance backward (TryRestartStaleInstance feeds the snapshot's old ConfigJson to HandleDeploy — a literal old-config redeploy; pipeline ownership only covers in-flight deploys and the old revision was never claimed), a read-pre-delete/disable resurrects a just-stopped instance (delete even clears the sweep claim), and the boot path skews _deployedInstanceNames. Fix: one _storeWriteGeneration over both tables, bumped at BOTH ends of every actor-thread config mutation (deploy persist, enable, disable, delete, rollback, artifact apply) — both ends because ApplyDeployment/HandleDisable/HandleDelete advance actor state BEFORE the off-thread write, so a completion-only bump (the row-101 original argument, which holds only for the artifact apply) leaves a window. Stale sweep discarded whole; stale boot load re-dispatched, capped at 5 consecutive (then applied with a Warning). CDC/SiteReconciliationActor writes deliberately never bump — discovering them is the sweep's purpose. 3 new fail-before tests + 2 fresh-generation guards pinning that row-72/78 legitimate repairs survive; SiteRuntime.Tests 652/652 x3; full suite green (one unrelated Communication.Tests gauge flake → row 102); rig probe: disable/enable/redeploy churn over multiple sweep ticks produced zero self-heal events on either site-a node, tags Good. Memo §11. |
Observed by the 2026-08-17 row-100 fix session | Closed 2026-08-17; whole-snapshot extension 2026-08-18. | — |
| 102 | SiteStreamSubscribersActiveGauge_GoesToOneOnConnect_AndBackToZeroOnCancel is delta-hardened but still flakes — the baseline can't survive other contributors CLOSINGSiteStreamGrpcServer.SubscribeInstance handler, and exactly two test classes drive it: SiteStreamGrpcServerTests (the gauge test's own class) and CleanupVerificationTests — GrpcOptionsWiringTests instantiates the server but never opens a stream, and the client/aggregator suites subscribe against fakes, so none of those can perturb the counter. Both drivers now share the [Collection("SiteStreamSubscriberGauge")] xunit collection (comments on both classes + at the baseline poll explain why the relative read alone was insufficient), so no other contributor can close a stream between the baseline capture and the assertions. Verified 4/4 consecutive full-suite runs green (717/717), durations unchanged (22–25 s vs the prior 24 s) — the serialization is free. Original evidence: the test polls baseline + 1, but the process-global gauge's baseline is captured ONCE; capture while parallel classes' streams are up → they close → the gauge settles at this test's own 1 and baseline+1 is unreachable forever — a fixed-baseline delta only survives contributors that hold constant or increase. |
Observed 2026-08-18 (one full-suite failure, Expected: 4, Actual: 1; 3/3 green in isolation); fixed same day |
Closed 2026-08-18. | — |
| 103 | c8452136). Both instances closed with the Medium-wave T10 take-once discipline: TransportImport.razor.cs re-checks disposal after TryLoadAsync's await and releases the import slot through exactly-once ownership (dispose-during-start frees the BundleSession slot immediately instead of stranding it for the 30-min TTL), and Topology.razor creates its timer under the same post-await re-check so a disposed page leaves no live timer (asserted event-style, not by sleep). Original note: (arch-review 2026-08-19 P6-6's optional tail, deliberately deferred by the fix). The permanent debug-stream leak was closed at DebugView.razor + DebugStreamHub (dispose-vs-connect take-once pattern, commit 11e623c2), but two same-family instances with a different shape were left: (1) TransportImport.razor.cs TryLoadAsync (MaxConcurrentImportSessions slots until the 30-min BundleSession TTL reaps it (partially defeats row 43); (2) Topology.razor (_disposed re-check + take-once teardown, but routed through BundleSession.ReleaseSession/the wizard-step lifecycle rather than a session-stop call. |
Arch-review 2026-08-19 P6-6 remediation (T10 explicitly scoped them out) | Open. | Next CentralUI hardening pass, or opportunistically when either file is next touched. |
| 104 | auto_vacuum=NONE; PRAGMA incremental_vacuum after a cap trim is a no-op on themdocker/vacuum-site-localdb.sh (stop both nodes of a pair together → PRAGMA auto_vacuum=INCREMENTAL; VACUUM each volume with integrity checks before/after → start both; DRY_RUN=1 reports) plus the topology-guide section Site LocalDb files created before 2026-08-20 need a one-off offline VACUUM. Run on all three rig pairs: all six files auto_vacuum 0 → 2, integrity_check=ok, ~11 % smaller (site-a 20.3 → 18.1 MB), site-a replication back Connected=true/backlog 0 with no watermark or schema change — the VACUUM was replication-invisible as predicted (ledger keys by declared PK; oplog seq is an INTEGER PRIMARY KEY AUTOINCREMENT). The same maintenance surfaced the unrelated row 106. Original: |
Arch-review 2026-08-19 P4-6 (Low-wave T4, 47f03fe6) |
auto_vacuum is only honoured on a table-less database, and LocalDbSchema.EnsureCreated had always created LocalDb's bookkeeping tables before SiteEventLogSchema.Apply ran — so the pragma had never once taken effect on any real file. The fix (SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile, run before AddZbLocalDb's lazy singleton constructs SqliteLocalDb) only helps fresh files; an unconditional VACUUM against a live ~1 GB replicated file was explicitly rejected (long exclusive lock on the active node; the rewrite is CDC-invisible but blocks the sync pump). Every node on the docker rig today predates the fix, so freed site_events pages there are reused, not returned to the OS — the storage cap (row 81) still bounds the data volume, only the file size stops shrinking. |
Operator-scheduled VACUUM in a maintenance window (both nodes of a pair stopped together), or when a site's file is next recreated. |
| 105 | ON DELETE foreign key (CASCADE/SET NULL/SET DEFAULT) from another tableSiteLocalDbReplicatedTableFkGuardTests (Host.Tests) builds the real site schema via OnReady on an unreplicated config (so 0.3.3's own guard can't mask which FK is at fault), walks SiteLocalDbSetup.ReplicatedTables (now internal), asserts each exists and has no inbound row-mutating ON DELETE FK with the same sqlite_master JOIN pragma_foreign_key_list query LocalDb uses, and carries a negative control proving the checker finds a cascading child. The constraint itself stands — the row is kept as its record. Original: |
Arch-review 2026-08-19 P4-7 → LocalDb 0.3.3 (scadaproj 122940c + review follow-up 9663279, Low-wave T5) |
LwwApplier upserts with INSERT OR REPLACE under foreign_keys=ON, so the implicit delete fires the child action while CDC capture is suppressed — the child rows change on the applying node only and the pair diverges silently. 0.3.3 makes RegisterReplicated refuse such a parent at registration (clear exception naming the child FK), which is a BOOT failure for the site node, so this is enforced, not advisory. None of the ten SiteLocalDbSetup.ReplicatedTables has any FK today (zero REFERENCES on the live rig file). Model cross-table cleanup as explicit application deletes on each table (as the 3-table deploy cascade already does), never as SQLite FK actions. |
Whenever a schema change adds a foreign key between site tables, or an eleventh table joins ReplicatedTables. |
| 106 | 75c1897), published and pinned. SnapshotStreamer.SendAsync now stamps as_of_seq = max(MAX(seq), last_acked_seq); pinned by WipedPeer_SnapshotLiftsItsWatermark_SoTheNextReconnectSendsNoSecondSnapshot (fails on 0.3.3 exactly as the rig did: watermark 0, expected 3). Live proof on the rig: the first restart on 0.3.4 sent one last full snapshot stamped as_of_seq 149 (was 0) and node-a's last_applied_remote_seq went 0 → 149; a second restart of the pair exchanged zero snapshots, Connected=true, backlog 0. Original: |
Found 2026-08-20 while verifying row 104 on the rig (site-a-a received the same 127 k-row full snapshot at both the 13:09 deploy restart and the 13:25 VACUUM restart) | Sender-side defect in SnapshotStreamer.SendAsync: as_of_seq = MAX(seq) of the oplog, which is 0 once the pair has converged and ack-pruning emptied it. The receiver's SetLastAppliedRemoteSeqAsync is MAX-guarded (no regression — good), but a peer whose watermark is genuinely 0 (site-a-a was rebuilt after the 2026-08-15 corruption; any wiped/re-imaged node is the same) can never be lifted by an as_of_seq 0 snapshot, so the 0.1.2 empty-oplog horizon rule (streamableFrom = last_acked_seq + 1) sees 0 < 149 as a gap on every handshake and re-streams everything. Rig state: node-b last_acked_seq=149, oplog empty; node-a last_applied_remote_seq=0. Not a divergence (LWW absorbs the re-delivered rows) but a cost/availability bug — ~1 s and 127 k rows here, minutes and a full-file read on a 1 GB production pair, on every reconnect, until the sender happens to write a new row. Fix is sender-side and wire-compatible: stamp as_of_seq = max(MAX(seq), last_acked_seq) (a full snapshot taken now covers every delta ever pruned below last_acked_seq), so the receiver's MAX-guarded watermark lifts to 149 and the loop ends; add a pinning test (converged+pruned sender, zero-watermark receiver → exactly one snapshot across two handshakes). Ships as LocalDb 0.3.4. |
Next LocalDb release (0.3.4) — small, well-understood, and worth doing before any production pair is rebuilt from its peer. |
| 107 | Prod CvdReactor.ProcessRecipeDownload still addresses Children["DelmiaRecv"] after the 2026-06-27 DelmiaRecv → DelmiaReceiver composition rename — inferred from the deployed script (docs/api-specs/Delmia-Recipe-Download-API.md §7 item 1): ChildrenAccessor concatenates the name into a path, an unknown path yields null, so every call answers Receiver ready flag not set to true regardless of the real flag. No call has hit prod since the rename, so this is inferred, not observed. The docker rig's stand-in (docker/seed-delmia-sim.sh) uses Children["DelmiaReceiver"] and the same handshake answers correctly, which is the working reference. |
Found 2026-09-02 while capturing the as-deployed spec; QA copy unread (credentials) | Separate change with its own deployment record: scadabridge template script update on the CvdReactor template + redeploy Z28061/Z28062/Z28061Sim, then a live WWNotifier call. |
Before the first production Delmia recipe download. |
| 108 | ZB.MOM.WW.Auth 0.3.0 (new ZB.MOM.WW.Auth.ApiKeys.SqlServer package) + ScadaBridge ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer. A central pair now shares one row set: both nodes' key stores, admin stores, and audit stores point at the same zbauth schema in the configuration database (connection string defaults to ScadaBridge:Database:ConfigurationDb — never a ${secret:} reference, to avoid a circular dependency on the SQL-backed secrets store), so create/revoke/rotate/scope changes are cluster-wide immediately instead of per-node. Each node's existing SQLite file is imported once at boot (insert-where-absent, so both nodes importing is safe and the result is the union) and renamed *.imported. Provider defaults to Sqlite (unchanged behavior for single-node installs, MxAccessGateway, HistorianGateway); a clustered Central left on Sqlite now gets a boot-time StartupValidator Warning naming this design instead of silently keeping the gap. Both docker rigs (docker/, docker-env2/) run Provider=SqlServer on both centrals; docker/sync-inbound-api-keys.sh is deleted, superseded by the shared store. Rig proof 2026-09-03 (docker/, both centrals redeployed on Provider=SqlServer): both nodes imported their SQLite file at the same second with no PK race — central-a 1 key / 264 audit rows inserted, central-b 0 / 26 inserted, 1 / 264 already present (the union), files renamed *.imported; a key minted via :9001 answered 200 via :9002, --enabled false via :9001 made both nodes answer 401 and :9002's own list showed enabled=False, re-enable via :9002 restored 200 on :9001, delete via :9002 made :9001 answer 401; verify-delmia-notifier.sh with a freshly minted key and NO file copy: A/B/E/F YES, C/G NO, active-central docker kill takeover 28 s, rejoin as standby in 2 s. docker-env2 (second pair, ScadaBridgeConfig2): both centrals booted, zbauth v1 created, empty files imported. QA (vq03/vq04) and prod (vd03) still run Provider=Sqlite — their move is a planned deployment record (docs/deployment/deployment-records/2026-09-XX-zb-servers-inbound-key-store.md), not yet applied. Design: docs/plans/2026-09-02-inbound-api-key-shared-store-design.md. Original: |
docs/api-specs/Delmia-Recipe-Download-API.md §8; docker/sync-inbound-api-keys.sh header |
Fix is an architecture decision, not a patch: move the store to the shared SQL Server (as Secrets 0.5.0 did for its central store) or replicate the file; either way revocation must become cluster-wide. Needs the ZB.MOM.WW.Auth.ApiKeys library (scadaproj), not just this repo. |
Before any 2-node central goes live with inbound API clients; before the next arch-review wave. |
| 109 | ZB.MOM.WW.Secrets.Replicator.SqlServer's store migrator retries its DDL with a flat 100 * attempt backoff and no jitter — a latent concurrent-boot flake, open in Secrets (Low). Porting that migrator's retry loop verbatim for the Auth 0.3.0 SQL Server API-key store (register row 108) reproduced an intermittent exhaustion — about once in sixty 8-way concurrent-boot trials the five flat-backoff retries ran out before a DDL/lock conflict cleared; adding jitter to the Auth port's backoff took it to 0 failures in 180 trials. Secrets' own migrator has no concurrent-migrate test, so the same defect is presumed still open there — it just hasn't been exercised at 8-way concurrency, and a ScadaBridge central pair is only ever 2-way, so this is unlikely to bite in practice. |
Found 2026-09-02 while implementing docs/plans/2026-09-02-inbound-api-key-shared-store.md (Auth 0.3.0 SqlServerApiKeyStoreMigrator) |
Add jitter to SqlServerSecretsStoreMigrator's backoff in ~/Desktop/scadaproj/ZB.MOM.WW.Secrets (mirror the Auth fix) and add a Migrate_Concurrently_FromEightTasks_AllSucceed-shaped live test to catch a regression. Lives entirely in scadaproj, not this repo. |
Next time Secrets' SQL Server migrator is touched, or if a wider-than-2-way concurrent boot of a Secrets-backed cluster is ever planned. |
| 110 | Flow Debugger follow-up: the Vertical trace list toggle (mockup Direction B) (Low). v1 ships the Direction A swimlane only; the mockup's Direction B — the same steps as a vertical, filterable list with the lane as a column — was kept as an alternate for long flows and screen readers. Additive: a list view over the existing FlowGraph. ScriptRundocs/plans/2026-09-09-flow-debugger-step-logs.md) — and deliberately not as this row sketched it. Extra.console on the ScriptRun row was rejected (the 8 KB payload cap would silently drop lines, and helper-step attribution would need every helper to buffer); shipped instead is a leveled Log global on all four script surfaces plus one ScriptTrace.ScriptLog audit row per line, folded onto the step that produced it, with the failing step's full exception on errorDetail. Console.Out capture on a site node is recorded as permanently out of scope in the design's §9: Console.Out is process-global and shared by every concurrent run, so per-run capture is unsound. |
Design docs/plans/2026-09-05-flow-debugger-design.md §11; plan 2026-09-05-flow-debugger.md Task 23 |
The list view was not asked for by an operator; the swimlane + Timing/Writes/Log tabs cover the acceptance flow (Delmia recipe download, rig steps H and J). | First operator request for a list view. |
| 111 | ?correlationId= deep link from the node to the Site Calls / Notification report pages (Low)docs/plans/2026-09-06-flow-debugger-followups.md). FlowStep now carries the row's CorrelationId; the detail panel deep-links to /site-calls/report?correlationId= / /notifications/report?correlationId= (both pages accept it through additive per-id query filters — SiteCallQueryRequest.TrackedOperationIdFilter, NotificationOutboxQueryRequest.NotificationIdFilter — pre-filter to the row and auto-open its detail), and a Parked node offers Retry / Discard in place through IFlowDebugService.RetryOutboundAsync / DiscardOutboundAsync, which reuse the report pages' relay, site-scope re-check and confirm wording, after which the page re-polls Replay until the node leaves Parked. Per-attempt request/response diff is still not shown (the lifecycle rows carry the error summary only). Found on the way: the prediction circuit deadlock (4769e192). |
Plan 2026-09-06 "Out of scope"; FlowStepDetail.razor parked hint |
The Site Calls page owns the retry semantics and its audit rows; duplicating the action on the debugger node would need the same Deployer gate + confirm flow. | First operator who reaches a parked node from the debugger and asks for the button. |
| 112 | Flow Debugger: a SITE-side inline trace truncation is invisible to the operator (Low). ScriptCallResult / RouteToCallResponse carry TraceEvents but no Truncated flag, so the page's event list truncated badge reflects central's FlowTraceCollector only — a site run whose own collector dropped rows returns a silently short list. The new 1 MB byte budget (arch-review F1) makes site truncation reachable sooner than the old 500-event count cap did. The fix is additive: a trailing bool Truncated = false and a LogsTruncated flag on both records (step logs 2026-09-09 gave the site collector a second, low-priority lane whose overflow sets LogsTruncated rather than Truncated, so a site run that shed only log lines is invisible in exactly the same way — and the page already has the badge for it), a new field on RouteToCallResponseDto (regen via docker/regen-proto.sh sitecommand), and FlowDebugRunner ORing both into FlowDebugRunResult.Truncated / FlowDebugRunResult.LogsTruncated. |
Arch-review remediation code review 2026-09-06 | Needs a proto regeneration plus a rig rebuild to prove end-to-end, and nothing is lost meanwhile: the site's audit rows are written in full, so a Replay of the same execution is never truncated. | The first Live run that shows fewer site steps than its Replay of the same execution. |
| 113 | Flow Debugger: the method-entry site scope (F2) gates ROUTED site traffic only (Low). A site-scoped Deployer who Live-runs a method belonging to another plant is still refused at RouteTarget.ResolveSiteAsync, but the method's central-side side effects — its Database.* calls, ExternalSystem calls, notifications — execute before or regardless of that refusal. Only routing is scoped. |
Arch-review remediation code review 2026-09-06 | F2's stated property ("no site traffic outside the operator's scope") holds as written. Going further needs a per-method site affinity, which does not exist in the model — ApiMethods are central objects with no site column — so there is nothing to scope a method against without inventing that concept. | A decision to give ApiMethod a site affinity, or an operator report of a scoped user reaching another plant's central-side effects. |
| 114 | Never issued. The 2026-09-09 step-logs close-out minted two rows numbered 113 — this one and the "other actors' log lines" follow-up. The duplicates were renumbered 115 / 116 at final review (I2) and 114 was left unused rather than reshuffling every number after it; register numbers are permanent identifiers, cited from code comments and CLAUDE.md. |
— | — | — |
| 115 | Other actors' log lines are not captured for an execution — only the script's own Log.* and the runtime lines inside a scoped helper (Low). The Flow Debugger's step logs (2026-09-09) capture what the script logs and what ScriptRuntimeContext's own logger emits while a run is traced, because those two go through one seam. What the Instance Actor, the DCL, the alarm actors or the S&F engine log while working on the same execution never reaches the graph: there are zero LogContext.PushProperty calls in the repo, Enrich.FromLogContext() is not configured on the Serilog pipeline (TraceContextEnricher stamps W3C ids only), and site_events has no execution-id column. Capturing them is a separate enrichment initiative: push ExecutionId (and instance/script) onto the ambient LogContext at run start, enable Enrich.FromLogContext(), add an execution-id column + index to site_events, then either sink matching entries into ScriptLog rows or query the site event log by execution id. |
Step-logs design docs/plans/2026-09-09-flow-debugger-step-logs-design.md §2 + §9 ("Out of scope") |
Out of proportion to the ask: it changes the logging pipeline of every actor on a site node, adds a replicated-table column (site_events is one of LocalDb's ten ReplicatedTables, so the change is a both-nodes-together schema edit), and risks a volume increase on untraced runs unless the enrichment is itself gated. The script-authored lines cover the debugging case the Flow Debugger exists for. |
First investigation where the answer sits in what the Instance Actor or the DCL logged during a traced run and the operator cannot correlate it by hand. |
| 116 | CompileSurfaceParityTests.AccessorMirrorPairs cannot guard the (ScriptLogger, CompileLog) pair — the theory has no per-member exclusion mechanism (Low). The arity theory asserts the compile mirror exposes at least as many overloads as the runtime accessor for every public instance method, and ScriptLogger has one the mirror must NOT have: FlushDropped(), the run-end closing-row flush, which is called by ScriptRunLauncher, not by a script. Adding the pair today would fail the theory for a member that is correctly absent, so the pair is left out and Log's four methods are guarded only by the top-level name-superset check plus the hand-written surface tests. The fix is an exclusion seam on the theory — an optional IReadOnlySet<string> ignoredRuntimeMethods per row (or an [NotScriptReachable] marker attribute on FlushDropped, which also documents the intent at the declaration) — after which the pair joins AccessorMirrorPairs. |
Found while wiring Task 15 of docs/plans/2026-09-09-flow-debugger-step-logs.md; test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/CompileSurfaceParityTests.cs |
No drift is possible without a code change to ScriptLogger, and a missing Log method on the mirror surfaces immediately as a CS1061 at the design-time gate for any script that calls it. The seam is a test-infrastructure change that touches every existing row's signature. |
The next public method added to ScriptLogger (or to any runtime accessor whose mirror must deliberately omit a member) — at that point build the exclusion seam and add the pair. |
| 117 | Flow Debugger site trace pull: FlowDebugRunner.WriteAndPollAsync fetches ONE bounded page with a fixed since (Low). The write-and-poll loop asks the site for FlowDebugRunner.PullTraceEventsMax rows per poll, always since the run start, and the site serves ORDER BY OccurredAtUtc ASC LIMIT — so the page is the OLDEST rows in the window, never a continuation of the previous one. Step logs made that reachable: one traced run can now emit up to 201 ScriptLog rows, so a handful of chatty runs inside the window can fill the page and push a later spawned run's ScriptRun row past the end of it, at which point the loop never sees the awaited run and reports a false NO_TRIGGER_WITHIN_WINDOW for a run that did happen. Final review I1 raised the cap 500 -> 2 000 (the site's own ceiling, SiteStreamGrpcServer.MaxTraceEventsPerPull), which makes it ~10x harder but does not remove it. The real fix is to PAGE — advance since to the last-seen OccurredAtUtc between polls (keeping the already-known ids so nothing is double-counted) — or to exclude ScriptLog from the awaited-run poll entirely and pull the log rows once, after the run has settled. |
Final review of docs/plans/2026-09-09-flow-debugger-step-logs.md (I1); src/ZB.MOM.WW.ScadaBridge.ManagementService/FlowDebug/FlowDebugRunner.cs WriteAndPollAsync |
2 000 is the site's hard ceiling, so the headroom cannot be bought again with a bigger number; paging needs the poll loop to track a cursor and the awaited-run predicate to survive a partial view, which is more than the raise was worth at review time. | The first NO_TRIGGER_WITHIN_WINDOW reported for a run whose Replay of the same root DOES show the spawned ScriptRun. |
| 118 | Pre-existing bUnit flake: CentralUI.Tests.Pages.NotificationReportDetailModalTests.CloseButton_DismissesModal (~10 %) (Low). Fails intermittently with UnknownEventHandlerIdException — the test resolves the close button once and clicks it after a re-render has already retired that element's handler id. Not caused by the step-logs work: the file is untouched since 40944a0d. The fix is a one-liner — re-Find the button immediately before the click (the pattern the other modal tests already use) rather than reusing an element captured earlier in the test. |
Observed during the step-logs full-suite runs, 2026-09-09 | It is a test-only race with a known cause and a known fix; it was out of scope for a remediation pass whose rule was to touch only the files under review. | Next time CentralUI.Tests is worked on, or the first time the flake blocks a gate run. |
| 68 | deploy.sh run from a git worktree silently rehomes every node's bind-mount data dir./site-a-node-a/data), so a deploy from a worktree checkout mounts the worktree's copies — the rig keeps running and looks healthy, but its SQLite stores split from the main checkout's and everything written there is orphaned by the next deploy from main. Bit the rows-46–53 close-out live: the seeding agent's rig-probe-1 deploy persisted into worktree-homed dirs and "vanished" after the orchestrator's redeploy (no tombstone, no product bug — LWW/tombstone forensics ruled out deletion; the row sat in .claude/worktrees/…/docker/site-a-node-a/data/site-localdb.db the whole time). Both docker/deploy.sh and docker-env2/deploy.sh now refuse to run from a .claude/worktrees/ path (override: SCADABRIDGE_ALLOW_WORKTREE_DEPLOY=1). |
Found during the rows-46–53 close-out | Closed 2026-08-15. | — |
| 119 | Port the QA/prod MES receiver scripts onto the reset-aware WriteBatchAndWaitAsync overload (Gitea #36) (Medium). Template scripts CvdReactor MesMoveIn (13) / MesMoveOut (14) / MesSampleData (36) and base MESReceiver MoveIn/MoveOut (15, 16, 25, 26, 37, 38) still reset <X>CompleteFlag with a plain write and rely on Task.Delay(1000); QA carries a config-only WaitAsync(false, 10 s) patch on 13/14 (deployment record 110). Rewrite: var hs = await rx.Attributes.WriteBatchAndWaitAsync(values, "<X>Flag", true, "<X>CompleteFlag", true, 25 s, "<X>CompleteFlag", false, 10 s); if (!hs.Matched) return new { WasSuccessful=false, ErrorText = hs.ResetFailed ? "Timeout waiting for <X>CompleteFlag to reset" : "Timeout waiting for <X> to complete", BatchID=0 }; and delete the delay. Requires the #36 build on the target central + sites first (the overload does not exist on older sites → deploy-gate compile failure, by design). |
Gitea #36, design 2026-09-10-handshake-reset-echo-design.md §8 |
Repo-only scope was the user's decision for the #36 session; the script bodies live only in the QA/prod databases. | Next QA deployment of a build ≥ this change (write a deployment record; then prod). |
| 120 | Data-sourced SCALAR writes reach the OPC UA server as String Variants — every non-String node rejects them with BadTypeMismatch (0x80740000) (Medium). InstanceActor.HandleSetDataAttribute / HandleWriteAttributeBatch decode only List values before the DCL write ("Scalars are unchanged"), so the codec-encoded string ("True", "42.5") reaches RealOpcUaClient.WriteValuesAsync, which wraps it in new Variant(value) — a String variant sent to a Boolean/Double node. Found live 2026-09-10 running docker/seed-handshake-demo.sh against opc-plc Motor.Running/Pump.Running/Motor.Speed (SetAttribute('Complete') failed: OPC UA write failed with status: 0x80740000); the seed was retargeted onto the rig's only two writable String nodes. The MxGateway path is unaffected in practice: prod's Delmia/MES handshakes write the same strings through the gateway today. Fix candidates: (a) decode scalars by declared DataType in the InstanceActor before the DCL write — also changes what MxGateway receives, so it needs a gateway-side compatibility check; (b) adapter-local: RealOpcUaClient coerces to the node's DataType (it already resolves DataTypeIds in EnrichVariableTypeInfoAsync), leaving the MxGateway wire untouched — preferred. Pin with a RealOpcUaClientArrayWriteTests-style test for a Boolean scalar. |
Found live 2026-09-10 during the #36 rig gate | Out of scope for #36 (handshake ordering); touches the production write path and needs the MxGateway compatibility question answered first. | Before any OPC UA-bound template relies on script writes to non-String nodes (none does today — until this gate the rig's script writes to OPC UA were static-only). |
| 121 | No source-level IsMesReportable on native alarm source bindings (Low). TemplateNativeAlarmSource / InstanceNativeAlarmSourceOverride carry no per-alarm MES-relevance flag — "every condition under this binding is reportable" is not expressible, only the 900–999 severity-band convention. |
docs/plans/2026-09-11-mes-reportable-alarm-flag-design.md |
Native alarms have no per-alarm config row, and the design chose not to invent one for this slice. | A native-source machine needs per-alarm MES relevance beyond the severity band. |
| 122 | IsMesReportable does not ride the wire (Low). AlarmStateUpdate (vendored proto) and DebugAlarmStateDto have no IsMesReportable field, so central-side surfaces (Debug View live stream, Alarm Summary) always see false even though the site-side snapshot stamps the real value. Adding it is additive-only: AlarmStateUpdate field 25, DebugAlarmStateDto field 20, via a manual regen-proto.sh toggle-build-copy-untoggle. |
docs/plans/2026-09-11-mes-reportable-alarm-flag-design.md |
No consumer needed it in this slice — the flag exists to serve Alarms.MesReportableAsync() for scripts, not a central UI surface. |
Operators ask to see the MES-reportable flag live on Debug View / Alarm Summary. |
| 123 | Rail nav scroll pane (.rail-scroll + the #theme-rail > .side-rail { height: 100% } fill rule) lives in app-local site.css/NavMenu.razor, not the ZB.MOM.WW.Theme kit (Low). Every other kit consumer (OtOpcUa, MxGateway, HistorianGateway, ignitionoee) has the same unbounded-nav-pushes-the-footer-off-screen shape and would need the identical fix copy-pasted. |
docs/plans/2026-09-14-pending-updates-design.md §3.5 |
Scoped to ScadaBridge for this change; upstreaming into the shared kit is a separate cross-repo release, not a one-app CSS tweak. | A second kit consumer hits the same long-nav clipping and copies the fix instead of consuming it from the kit. |
| 124 | No template-graph-watermark short-circuit on the pending-updates fleet flatten (Low). PendingUpdatesEvaluator.EvaluateAsync re-flattens every deployed instance through DeploymentService.GetInstanceStalenessAsync on every cache-TTL expiry (PendingUpdatesCacheTtl, default 60 s) while any Deployer/Admin circuit is open, even when nothing in the template/instance/artifact graph has changed since the last evaluation. |
docs/plans/2026-09-14-pending-updates-design.md §3.2 ("Not in scope") |
The single-flight TTL cache already bounds the cost to one fleet flatten per TTL fleet-wide (not per viewer); a watermark short-circuit is a further optimization, not a correctness need, and needs a cheap "has anything changed" signal that does not exist yet. | The fleet flatten's cost (instance count x template depth) becomes measurable against PendingUpdatesCacheTtl on a large deployment. |
| 125 | Pending-updates counts have no Health-dashboard tile or CLI surface (Low). The aggregate (PendingUpdatesSnapshot.StaleInstanceCount / StaleArtifactSiteCount) is consumed only by the Central UI Sites page and rail indicator; there is no Health dashboard KPI tile and no deploy/site CLI command surfacing it — site list shows only the raw artifactRevisionHash/artifactsDeployedAt fields on each site, not whether they are current. |
docs/plans/2026-09-14-pending-updates-design.md §3.2 ("Not in scope") |
Out of scope for the UI-focused ask; the CLI/Health surfaces are a separate consumer of the same IPendingUpdatesCache. |
An operator wants the pending-updates counts without opening the Central UI (headless monitoring, CLI scripting). |
| 126 | 2b9a7c3c → b3021e7c, 55 → 57, artifacts deployed (record docs/deployment/deployment-records/2026-09-28-wonder-app-vq03-vq04-b3021e7c.md). (deployment, closed). DEV vd03 DONE 2026-09-28 (fe89363b, 55 → 57, record docs/deployment/deployment-records/2026-09-28-wonder-app-vd03-fe89363b.md — the sb-mig-55to57.sql there is the same script QA needs). QA still runs 2b9a7c3c with the DB at migration 55 (20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration); main (ac833f5a, pushed 2026-09-14) carries two more: 56 20260911135501_AddAlarmMesReportable (computed-alarm IsMesReportable flag) and 57 20260914173147_AddSiteArtifactRevision (Sites.ArtifactRevisionHash/ArtifactsDeployedAt, both nullable). Both are additive AddColumns — no data upgrade, no coordinated step like the 2026-09-10 protocol-config encryption. Recipe (same as the 55 rollout): generate ONE from→to delta script dotnet ef migrations script EnforceGlobalInstanceNamesAndProtectProtocolConfiguration AddSiteArtifactRevision --project src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase --startup-project src/ZB.MOM.WW.ScadaBridge.Host -o sb-mig-55to57.sql (NOT idempotent, identical for vd03 and the QA pair), take the pre-upgrade DB backup, stop the pair TOGETHER, apply via sqlcmd -C -b -I on the SQL host, publish/swap the ac833f5a binaries, start the pair, confirm the Central boot log shows 0 pending migrations and no "Applying migration" (the script already applied them). After upgrade every site reads Artifacts never deployed until one Deploy Artifacts to All Sites — run it as the last step. Write the deployment record docs/deployment/deployment-records/<date>-wonder-app-{vd03,vq03-vq04}-ac833f5a.md. |
Pending-updates finish (2026-09-14) | winrm is re-enabled for deployment activities (CLAUDE.md). vd03 done; QA done 2026-09-28. |
Closed 2026-09-28. |
| 127 | TODO — roll out site secrets via artifact deploy to vd03 + QA (vq03/vq04). The gRPC secrets hub is retired in ScadaBridge as of docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md (merged to main @ 23a139a4 2026-09-28, plus the same-day follow-ups: per-site secret-store outage handling, removal audit rows, per-node mirror status on the health report, artifact-deploy CLI exit codes; rig-only so far); vd03 and QA still run the pre-feature build with Secrets:Replication:Enabled=false, so upgrading them will not fail the boot (the retirement's StartupValidator rejection only fires on Enabled=true). Upgrade impact: once upgraded, any site whose data connections carry secret: references fails its whole artifact deploy (scripts, external systems, DB connections too) and reads Out of date until every referenced secret exists in the central store — so before upgrading vd03/QA, check whether their connection configs use secret: references and whether their site pairs run LocalDb replication, and do the prerequisites in the same window. Prerequisites, in order: (1) provision the zbsecrets schema/store in the environment's configuration SQL Server (the same server ConfigurationDb already points at); (2) set Secrets:Store:Provider=SqlServer on both central nodes of the environment; (3) verify the KEK (ZB_SECRETS_MASTER_KEY / master-key file) is identical across every node — both centrals and every site node — since a mismatch is caught only at deploy time by DeployedSecretsMirror's per-secret verify, not at boot (after the first deploy a mismatch on either node — the standby included — shows as a Secret mirror failing badge on that site's Health card); (4) confirm every site pair that uses secrets runs LocalDb replication — on an unreplicated pair only the node active at deploy time gets the secrets, so a failover leaves the new active node without them until the next Deploy Artifacts; the upgrade also adds two replicated site tables (deployed_secrets, secret_mirror_status), so stop and start each site pair together as usual; (5) recreate, at the new central shared store, any secret an operator previously hand-seeded directly into a site's local secrets.db (those never round-tripped through central and would otherwise be silently absent from the referenced-secrets scan). Then run Deploy Artifacts against every affected site as the rollout's last step. |
docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md §4 Rollout; docs/deployment/topology-guide.md → Secrets: shared store, per-site deploy, and the KEK |
Deployment record, not a code change — vd03/QA config is user-held and their rollout schedule is independent of the branch merging to main. | User asks for the vd03/QA secrets window. |
Resolved (verified against the code 2026-07-10)
Rows removed from the Deferred table above once confirmed shipped. Kept here for traceability.
| # | Item | Resolution |
|---|---|---|
| 10 | Aggregated live alarm stream for Alarm Summary | Shipped 2026-07-10 (docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md): a transient, in-memory per-site central live alarm cache (ISiteAlarmLiveCache/SiteAlarmLiveCacheService + per-site SiteAlarmAggregatorActor) fed by a new site-wide, alarm-only SubscribeSite gRPC stream (SiteStreamManager.SubscribeSiteAlarms), seed-then-stream with dedup + NodeA↔NodeB re-seed + periodic reconcile. Alarm Summary is now live-cache-driven (AlarmSummaryService.BuildFromLiveAlarms) with the 15s poll retained as fallback + NotReporting authority. Honors the [PERM] no-central-store rule — nothing persisted (no EF table/migration). Options on CommunicationOptions (eagerly validated) + two ScadaBridgeTelemetry signals. |
| 7 | SecuredWrite audit rows leave SourceNode NULL | Resolved (PLAN-07): ManagementActor.EmitSecuredWriteAuditAsync routes through ICentralAuditWriter, which stamps SourceNode (central-a/central-b) from INodeIdentityProvider. |
| 12 (CLI/API) | Native-alarm-source-override CSV import | Shipped 2026-07-10: shared CsvLineSplitter, NativeAlarmSourceOverrideCsvParser, bulk all-or-nothing SetInstanceNativeAlarmSourceOverridesCommand + ManagementActor handler (Deployer-gated), CLI instance native-alarm-source import --file, parser/CLI/handler tests. UI upload affordance shipped 2026-08-01 — second InputFile on the InstanceConfigure Native Alarm Source Overrides card reusing the shared parser, mirroring the attribute importer's UX and the server's all-or-nothing merge semantics (InstanceConfigureNativeAlarmCsvImportTests); row 12 removed from the Deferred table. |
| 13 | WaitForAttribute quality-gated ("Good"-only) mode | Already implemented (Commons WaitForAttribute.RequireGoodQuality, enforced in InstanceActor, threaded through ScriptRuntimeContext, tested in InstanceActorWaitForAttributeTests). Stale "planned enhancement" doc line corrected 2026-07-10. |
| 14 | WaitForAttribute in Test-Run sandbox | Shipped 2026-07-10 (full fidelity): sandbox Attributes.WaitAsync/WaitForAsync (value-equality) route to the bound instance via ISandboxInstanceGateway.WaitForAttributeAsync → the existing CommunicationService.RouteToWaitForAttributeAsync cross-site route. Additive RouteToWaitForAttributeRequest.RequireGoodQuality (honored by the site handler) makes quality-gated waits route too. Predicate-form waits stay unsupported (an in-process lambda can't be routed) and throw a labelled ScriptSandboxException. Tests: sandbox accessor routing (CentralUI), site-handler quality-flag threading (SiteRuntime). |
| 15 | BrowseNext final-page signal not surfaced | Already surfaced (M7 browse work): RealOpcUaClient sets Truncated=false/ContinuationToken=null on the last page; BrowseNodeResult carries both; TreeRow.razor renders "Load more" only when a continuation token remains — no wasted BrowseNext. |
| 16 | StubOpcUaClient throws on browse | Already resolved: StubOpcUaClient supports browse + address-space search, covered by StubOpcUaClientBrowseTests/StubOpcUaClientSearchTests. |
| 18 | Folder drag-drop | Closed — permanently deferred ([PERM]) by the M9 decision (docs/plans/2026-06-15-stillpending-completion-design.md:122): menu-based reorder (T23) shipped instead, and the folder-hierarchy design fixed the reorganization UX as "right-click context menus only (no drag-drop)" (2026-05-11-templates-folder-hierarchy-design.md:27). Row removed from the Deferred table 2026-08-01 — nothing left to revisit. |
| 20 | Deployment EXPIRED-row purge | Already resolved (PLAN-04): PendingDeploymentPurgeActor central singleton (spawned in AkkaHostedService) ticks IDeploymentManagerRepository.PurgeExpiredPendingDeploymentsAsync every CommunicationOptions.PendingDeploymentPurgeInterval (default 1h), options-validated, tested. |
| 21 | SiteAuditBacklogReporter threshold consolidation | Shipped 2026-07-10: SqliteAuditWriterOptions.BacklogPollIntervalSeconds (default 30) now drives the reporter's poll cadence; explicit ctor override still wins (tests), non-positive falls back to the 30 s default. Cadence tests added; stale "hard-code / follow-up" class-doc corrected. |
| 22 | KPI history hourly rollups | Shipped 2026-07-10 (docs/plans/2026-07-10-kpi-history-hourly-rollups-plan.md, T1–T8): new KpiRollupHourly table (migration 20260710153953) folded by a third recorder tick (kpi-rollup, RollupInterval default 1h) over a re-folded RollupLookbackHours window via an idempotent, failover-self-healing upsert; per-metric gauge-vs-rate aggregation (KpiMetricAggregationCatalog); a one-shot backfill of the retention window on start; raw-vs-rollup query routing by RollupThresholdHours (default 168h); longer rollup retention (RollupRetentionDays default 365 ≥ RetentionDays, dual daily purge); and 30 d / 90 d trend windows added to the four surfaces. Options + validator, docs (Component-KpiHistory.md), and tests shipped. |
New deferrals from review 08 (this plan)
| Item | Rationale | Revisit trigger |
|---|---|---|
| Communication → HealthMonitoring layering (ICentralHealthAggregator consumed by CentralCommunicationActor.cs:351) | Moving the interface + SiteHealthState to Commons ripples across 5 projects for a cosmetic inversion | Next breaking change to ICentralHealthAggregator |
| docs/components reference docs for ScriptAnalysis, KpiHistory, DelmiaNotifier | Reference docs are substantial (StyleGuide-conformant); README claim scoped instead (PLAN-08 Task 10) | Next doc-writing session touching those components |
| Test-coverage backfill: SiteCallAudit.Tests (31 tests/1.6k LOC), DeploymentManager.Tests | No defect identified; coverage partly lives in ManagementService/Host/Integration suites | First regression escaping either component |
~tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs is no longer a skipped placeholder: it runs as a live [Fact] (Category=Performance) on the real two-node in-process rig (TwoNodeClusterFixture, production BuildHocon) at production timings — 2s heartbeat / 10s failure-detection threshold / 15s stable-after — hard-killing the younger node and timing the survivor's member REMOVAL with singleton continuity asserted on the oldest. Delivered by PLAN-R2-01 Task 4 (archreview/plans/PLAN-R2-01-cluster-host-failover.md:226). The oldest-crash direction is covered behaviorally by SbrFailoverTests.AutoDown_HardCrashOfOldestNode_* and by docker/failover-drill.sh. |
The 2026-07-08 "PLAN-01 rig landing" trigger had fired unnoticed (NF2); PLAN-R2-01 T4 wired the placeholder to the fixture rig rather than recording a blocker. | Closed. |
DefaultRetryInterval (measured 28.9 s) before anything drains, since EnqueueAsync(attemptImmediateDelivery: false) stamps LastAttemptAt and the due-predicate then holds the row for one interval (correct by design; Notify.Send's deferToSweep: true leaves it null and skips the wait). Filed as informational finding F2. (b) Slow-subscriber backpressure: with 5 subscribers on the SAME instance and one reader stalled at 50 ms/event, the four healthy subscribers delivered 100.00% with zero drops while the stalled one lost 197,028 of 200,000 events entirely within its own bounded channel. Isolation is structural and total — Buffer(DropHead) never backpressures the BroadcastHub, Sink.ForEach's Tell never blocks, and StreamRelayActor.WriteToChannel's TryWrite on a DropOldest channel never blocks — so a slow WAN link degrades only its own feed, visibly (per-stream eviction counter + RecordSiteStreamEventDropped). Finding F3. Caveat also recorded: an unpaced burst costs every subscriber, because the publish Source.ActorRef upstream of the hub is shared — that shared stage, not per-subscriber buffering, bounds burst absorption. |
Closed by the row-25 run; no defect found in either half. | Closed 2026-08-15 — docs/plans/2026-08-15-target-scale-load-test-results.md. |
Deferred — operational risk (from the initiative tracker, folded in 2026-07-12)
Two live items previously tracked ONLY in archreview/plans/00-MASTER-TRACKER.md's registry are folded in here (NF5) so this register is the single tracking place. The tracker's narrative subsections remain as the historical evidence.
| # | Item | Where noted | Rationale for deferral | Revisit trigger |
|---|---|---|---|---|
| SBR | keep-oldest to the auto-down downing strategy (Akka AutoDowning, auto-down-unreachable-after = 15s): a hard crash of EITHER node — active/oldest included — now fails over to the survivor in ~25s with no operator action. Accepted trade: a real network partition produces dual-active until an operator restarts one side. Decision record + evidence (live keep-oldest DownReachable … including myself log, Akka.NET 1.5.62 KeepOldest.OldestDecision source, rejected alternatives incl. the static-quorum-1 DownAll trap): docs/plans/2026-07-21-auto-down-availability-decision.md. |
archreview/plans/00-MASTER-TRACKER.md:194 + auto-memory sbr-keep-oldest-2node-active-crash-gap (both now historical) |
— | Closed. Residual: seed-node boot-alone constraint (unchanged, documented in Component-ClusterInfrastructure.md); dual-active recovery is operator-driven. |
| vd03 | deploy/wonder-app-vd03/ overlay edits unapplied — appsettings.Central.json needs AllowSingleNodeCluster: true + phantom-seed removal + NodeName: central-a; install.ps1 needs sc.exe failure recovery actions. The deploy/wonder-app-vd03/ artifact directory is intentionally untracked (production config out of source control), so the repo cannot ship the fix. |
archreview/plans/00-MASTER-TRACKER.md:198 (PLAN-01 T16/T20/T23) |
Needs on-host access; without NodeName that deployment's audit rows stamp NULL SourceNode — partially mitigated once PLAN-R2-08 Task 7 lands: the host now FAILS AT BOOT with a key-naming error instead of silently NULLing, so applying the overlay becomes mandatory at the next upgrade. Owner: whoever maintains the host (user). |
Next wonder-app-vd03 deployment/upgrade — the Task 7 validator makes this row unskippable then. |