Joseph Doherty
51 KiB
51 KiB
Deferred-Work Register (established 2026-07-08, from architecture review 08)
Single tracked list of consciously-deferred work. Rules: every deferral gets a row (rationale + revisit trigger); fix-now items reference the archreview plan that owns them and are removed from this table when that plan's task lands.
Fix-now (owned by archreview plans)
All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round 2, 2026-07-12) — rows removed per the rule above; see the round-2 report §1 for the per-item evidence.
Deferred (with rationale + revisit trigger)
| # | Item | Where noted | Rationale for deferral | Revisit trigger |
|---|---|---|---|---|
| 8 | Hash-chain tamper evidence (T1); CLI verify-chain is a no-op stub | audit-log roadmap :12 | v1.x by locked decision; append-only DB roles are the control | Compliance requirement for cryptographic tamper evidence |
| 9 | Parquet audit archival (T2); endpoint returns 501 | AuditEndpoints.cs:204 | v1.x; 501 + CLI messaging are honest | AuditLog partition volume nears retention ceiling |
| 11 | Central-persisted OPC UA cert-trust audit | m7 follow-ups | Broadcast-to-both-nodes covers HA | Governance/audit requirement for trust decisions |
| 17 | Unified notifications+site-calls outbox page | stillpending :118 | Explicit M9 decision to keep two pages | Operator confusion reports |
| 19 | Bundle signing / cluster-to-cluster pull / differential bundles | transport-design :402 | v1 manifest hash + AES-GCM held sufficient | Non-repudiation requirement across orgs |
| 23 | Live LDAP group-membership re-query for an active session | docs/requirements/Component-Security.md :61-69 (+ :78-79) |
Blocked on an external package. The mid-session refresh re-maps the stored groups against the central DB with no LDAP call, so a directory group-membership change lands only at next login. A live re-query needs a passwordless service-account group-search method on the shared ZB.MOM.WW.Auth.Ldap library — an external NuGet PackageReference (src/ZB.MOM.WW.ScadaBridge.Security/…csproj:23) exposing only AuthenticateAsync(username, password, ct). Central role-mapping/scope changes still apply within ~15 min (RoleRefreshThresholdMinutes). |
ZB.MOM.WW.Auth.Ldap gains a standalone group-search API, or a requirement that a directory-side group revocation take effect mid-session rather than at next login |
| 24 | M8 large-bundle performance hardening | docs/plans/2026-06-15-stillpending-completion-design.md:106 — "Small follow-ups logged (not blocking): … large-bundle/perf hardening" |
Logged as a non-blocking follow-up when M8 shipped and never given an artifact: no plan, no task entry, no perf/load test exists (tests/…Transport.Tests/Import/BundleImporterLoadTests.cs is a LoadAsync unit suite despite the name). No measured problem; the only sizing controls in place are the 5-minute CLI transport timeout, LineDiffer's MaxInputLines=4000 summary-only cap, and MaxConcurrentImportSessions=8. |
First real bundle that times out, exhausts memory, or makes the import wizard's diff step unusable |
| 25 | docker/ rig still running. Headline numbers: tag update latency P50 0.88 ms / P99 4.57 ms / max 37.41 ms end-to-end (DCL boundary to stream subscriber, the emit instant carried verbatim through production code); 0 events dropped at 100 live subscribers; health report collect+ingest P99 0.31 ms with 10/10 sites tracked; debug view snapshot under load P99 2.19 ms, 0 timeouts; 500 instances deployed to a site in 2.6 s; CPU 2.9% of the box; working-set slope +8.83 MB/min. 11 clean passes, 1 pass with a caveat (F1: 20 min with zero gen-2 collections cannot fully settle the leak question — the 1-hour run would), 0 failures. Harness: tests/ZB.MOM.WW.ScadaBridge.LoadHarness/ (standalone Exe — the Category=Performance trait enables a filter but does NOT exclude by default, so a 20-minute test could not live in PerformanceTests) plus a CI-scale smoke [Fact] (PerformanceTests/TargetScale/TargetScaleHarnessSmokeTests.cs). Real: per-site ActorSystem + LocalDb, the real DCL via the documented DataConnectionFactory.RegisterAdapter seam, real InstanceActors, real SiteStreamManager, real StreamRelayActor + production-capacity bounded channel, real S&F, real health collector/aggregator; only the socket hops are stood in for. Results: docs/plans/2026-08-15-target-scale-load-test-results.md; design + deviations + thresholds: docs/plans/2026-08-15-target-scale-load-test-design.md. |
as before | Closed. Residual scope: [xc-3] site-event-log retention volume, [xc-4] audit-vs-central degradation, [xc-5] large-template flattening and [xc-8] UI responsiveness are NOT covered by this harness (central-cluster/browser concerns) and remain open WP-4 scope — see results doc SS6. |
Closed 2026-08-15. |
| 26 | Ipsen MES MoveIn tail: leak-test (-LT) receivers + routing, PLC-output-flag writes, Z28062 BTDB data completeness |
docs/plans/2026-06-16-ipsen-mes-movein.md:409 ("Out of scope (future)"); design 2026-06-16-ipsen-mes-movein-design.md:58-60, :196-198 |
Customer-site scope, not a platform gap. -LT routing needs an MES-receiver child + Galaxy reference that do not exist on the reactor template (any -LT/unknown suffix returns WasSuccessful=false with an "unsupported side/target" message by decision); MoveInComplete/Successful/ErrorText are PLC-owned by locked decision, so ScadaBridge deliberately does not write them; Z28062 completeness is an operational data fix, not code. Note the separate alarm-status path already handles the suffix — _LT is stripped before side-scoping (2026-06-30-mes-alarm-status-api.md:158). |
Ipsen creates the leak-test receiver + Galaxy reference, or asks ScadaBridge to own the PLC-output flags — otherwise a candidate won't-do ([PERM]) at the next Ipsen scope review |
| 27 | External-system per-system retry config (MaxRetries/RetryDelay) never reaches sites, and has no CLI/management surface |
Found live 2026-08-01 (rig session, #11 gRPC live checks) | Two stacked gaps: (a) ExternalSystemArtifact (Commons) carries TimeoutSeconds but NOT MaxRetries/RetryDelay, and the site external_systems table has no such columns — so a centrally-configured retry policy is silently ignored on sites; every cached call buffers with the S&F default (DefaultMaxRetries 50 × DefaultRetryInterval 30s ≈ 25 min to park). (b) Create/UpdateExternalSystemCommand don't expose the fields either — the only way to set them today is a direct DB edit of ExternalSystemDefinitions. Transport bundles DO carry them (arch-review 05 "ES retry config"), which masks the gap in export/import round-trips. Fix is additive: extend the artifact + site schema + apply path, and add --max-retries/--retry-delay to the CLI. |
First operator who tunes retry policy on an external system and expects site cached calls to honor it |
| 28 | Health-dashboard "Trigger failover" confirm dialog's confirm button is labeled "Delete" | Found live 2026-08-01 (rig session, #11 TriggerSiteFailover check) | The DialogService confirmation host's default destructive-action label leaks through — the dialog copy is correct but the red confirm button says "Delete" for a failover. One-line fix: pass an explicit confirm label ("Fail over") at the Health-dashboard call site (and audit other confirm-dialog call sites for the same default). | Next Central UI session |
| 29 | 1-hour sustained run to settle the memory-slope ambiguity (load-test finding F1) | docs/plans/2026-08-15-target-scale-load-test-results.md F1 (Low) |
The row-25 20-minute run PASSED the working-set threshold (+8.83 MB/min < 20) but recorded zero gen-2 collections across 45M events, so the +19.52 MB/min managed-heap drift on an uncompacted gen-2 is ambiguous — the heap demonstrably sawtooths, but 20 min cannot distinguish "gen-2 not yet collected" from a slow leak. Not a defect finding; an evidence gap. The harness exists (tests/ZB.MOM.WW.ScadaBridge.LoadHarness/) and the full 1-hour protocol is one documented command (results doc §7) — deferred only because the measured window was scaled for practicality. |
Before production go-live at target scale, or first field report of unexplained site-node memory growth |
| 30 | WP-4 residual scope: [xc-3] site-event-log retention volume, [xc-4] audit-load-vs-central degradation, [xc-5] large-template flattening time, [xc-8] UI responsiveness under load |
docs/plans/2026-08-15-target-scale-load-test-results.md §6 + row 25 closure note |
The row-25 harness is a site-runtime rig; these four WP-4 sub-criteria are central-cluster/browser concerns it deliberately does not cover. Recorded so the row-25 closure never reads as "WP-4 fully done" — phase-8 WP-4 is closed for the site-runtime data path only. | Before declaring phase-8 WP-4 complete, or production go-live at target scale |
| 31 | Unpaced-burst absorption is bounded by the shared publish stage, not per-subscriber buffers (load-test finding F3 caveat) | docs/plans/2026-08-15-target-scale-load-test-results.md F3 |
Measurement-backed design note, no defect: slow-subscriber isolation is structural and total (a stalled reader loses events only in its own bounded DropOldest channel; healthy subscribers delivered 100.00%), but the publish Source.ActorRef upstream of the BroadcastHub is shared, so an unpaced burst degrades every subscriber equally — that stage, not per-subscriber buffering, is the burst ceiling. |
First field report of event loss on healthy subscribers during a burst, or any redesign of the site stream publish stage |
| 32 | QueueDepthGaugeTests full-suite flake — shared static gauge (load-test finding F4) |
tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/QueueDepthGaugeTests.cs; surfaced by the row-25 verification runs (docs/plans/2026-08-15-target-scale-load-test-results.md F4) |
Pre-existing, test-only: fails when the full suite runs concurrently, passes in isolation — the gauge under test is process-global static state shared across test classes. Zero src/ involvement; needs test isolation (per-test meter/collector instance), not a product fix. |
Next test-infra session, or the first time it fails a CI/full-suite run that matters |
| 33 | LocalDbOplogBacklog has no threshold, KPI history, or dashboard treatment |
Found by R7 (docs/plans/2026-08-15-residual-remediation-plan.md) while documenting the purge burst |
SiteAuditBacklog drives a configurable threshold warning on the site tile, but the LocalDb backlog is reported and Prometheus-exported with nothing consuming it for alerting or trending — it is not among the metrics SiteHealthKpiSampleSource samples into KPI history, so the purge-burst spike R7 documents cannot be reviewed after the fact; an operator must be watching live. Candidate: add it to the KPI sample source and/or a sustained-backlog threshold badge. Pairs naturally with the R1 per-table-snapshot work, which changes snapshot/backlog behavior. |
Next health-dashboard session, or first operator question about a backlog spike they didn't see live |
| 34 | site_events purge burst is unbounded in the oplog and unmeasured at scale |
Found by R7 | Retention deletes are sliced at 1000 rows/statement, but the whole expired set is deleted in one tick, so a replicated site's oplog gains one row per deleted event with no pacing — a first purge after enabling replication (or after a long low-purge period) queues the entire batch at once. The wire path is bounded (MaxBatchBytes), but the oplog write itself is not paced, and no measurement exists for batch size or drain time at target scale. Candidate: measure on the rig or in the load harness; consider a per-tick delete ceiling if drain time is material. |
Before enabling replication on a site with a large existing site_events backlog, or the next load-harness session |
| 35 | DeploymentRecords index does not cover the deployment-list sort key |
Found by R3 (35ce1413) |
The table has a single-column IX DeployedAt; the new list query orders DeployedAt DESC, Id DESC and joins Instance, so SQL Server sorts per page. A composite (DeployedAt DESC, Id DESC) index would serve the ordering directly — deliberately not done in R3 because it needs an EF migration and the terminal-record purge bounds the table. (Low) |
Deployment list measurably slow at a real fleet's retention depth |
| 36 | Deployments free-text search is unindexable by construction | Found by R3 | Four LIKE '%term%' predicates (instance name, deployment id, revision hash, user) — leading wildcards preclude index seeks, so a search is a scan of the join. Acceptable at the purge-bounded row count. Fix if hot: anchor the id/hash legs as prefix matches, or full-text. (Low) |
Search latency complaints on the Deployments page |
| 37 | Two paged reads of DeploymentRecords with different contracts |
Found by R3 | QueryDeploymentSummariesAsync (CLI/ManagementActor: offset, no total, no instance join, instance-id-list scoping) and R3's QueryDeploymentListPageAsync (UI: total + instance join + site-id scoping) duplicate query logic. Consolidating on one contract — giving the CLI a total, both scoping by site — removes the duplication. Genuinely different contracts today, so not urgent. (Low) |
Next time either query's shape changes |
| 38 | OtOpcUa: RoslynVirtualTagEvaluatorTests racing-clear test is load-fragile (OtOpcUa repo, not this one) |
Found by R4 during the full-suite bump verification | Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed fails under full-solution parallel test load, passes 3/3 isolated: its design assumes at most one ClearCompiledScripts lands inside one evaluation (so the product's one-shot retry suffices); under contention two can. Same timing-assumption family as the R5 sweep, but lives in OtOpcUa. Test-side fix: bound clear cadence relative to observed evaluation latency, or assert on retry-exhaustion instead of zero failures. |
Next OtOpcUa test session, or when it fails a run that matters |
| 39 | OtOpcUa: late-enabled replication cannot baseline pre-existing rows (OtOpcUa repo, not this one) | Found by R4; documented in OtOpcUa's CLAUDE.md | LocalDbSetup.OnReady registers all four tables unconditionally on every driver node, so it can never pass baselineExistingRows: true (only correct when gated on replication being configured). Turning replication ON for a long-running default-OFF node converges on subsequent writes only — existing rows are never baselined. ScadaBridge closed this class with conditional registration + boot-time DeregisterReplicated self-heal (WP1.3/WP3.3); porting it is a behavior change beyond a pin bump (registered-set pinned by tests, deregistration must be pair-symmetric). Low current impact: site-a was enabled from fresh. |
Before enabling replication on an established OtOpcUa node |
| 40 | Wall-clock sleeps used as synchronization in tests (absence-assertion class) | Found by the R5 sweep (9fb52153) — a different class from the one swept, deliberately left unfixed |
Tests that sleep, then assert an absence: ScriptDeadlineAtEnqueueTests.cs:105,128 (two Thread.Sleep(2s), with Assert.Equal(0, scheduler.DetachedThreadCount) depending on the sleep out-racing the watchdog — worst offender), DeploymentWarmThenGateTests.cs:117, SiteAlarmLiveCacheServiceTests.cs:195,216, DataConnectionActorTests.cs:435 (Task.Delay(200) to "settle" after an AwaitCondition). These fail toward false-pass rather than flake, so they don't trip CI — but each is a timing pin that weakens what the test proves. Fix pattern: replace the sleep with an explicit observable edge (or assert on the positive event with a bounded wait plus a negative control). |
Next test-infra session, or the first time one of these areas regresses without its test catching it |
| 41 | LocalDb (scadaproj) sync-session tests flake under parallel load (scadaproj repo, not this one) | Found by R1; confirmed pre-existing at base f627cee (1 failure in 3 full runs) |
SyncSessionTests.Handshake_DigestMismatch_FailsClosed and BatchBudgetTests.Pump_RowLargerThanTheBudget_IsSentAloneRatherThanStallingTheStream intermittently time out. Root cause looks structural, not a timing pin: on a handshake fault, RunAsync's catch cancels the writer loop, which can kill the session's OWN Handshake before it transmits — the healthy peer then waits out its full 15 s cap for a message that never comes. Product-adjacent (the same early-cancel could delay a real peer's fail-closed verdict), so worth a look beyond the tests. |
Next LocalDb session |
| 42 | build/push.sh-class publish bug across the family repos |
Found by R1 the hard way (incident: 25 unrelated packages briefly published to the feed, deleted + verified same session; script fixed in scadaproj 0038d46) |
The bug shape: dotnet pack -c Release -o ./artifacts with no project argument and no cd, so the script packs the CALLER'S CWD. Fixed in LocalDb's script (cd to own repo root, name the solution, absolute output path). Peer repos' publish scripts (MxAccessGateway, HistorianGateway, OtOpcUa, others in scadaproj) should be checked for the same shape — the recovery only worked because the token happened to carry delete rights. |
Before anyone runs a family publish script from outside its repo, or next umbrella-repo session |
| 43 | BundleSession slot on every abandoned import — imports lock out for 30 min70838982). New IBundleImporter.Abandon(sessionId) does the same zero-then-remove release ApplyAsync already did (T-007 discipline), idempotent (double/unknown/expired ids are no-ops) and only ever given the caller's own random-GUID id, so it cannot evict a concurrent circuit's session. Wired into EVERY abandonment path: wizard new-upload, TryLoadAsync pre-overwrite, Back-to-Upload/ResetSessionState, 3-strike lockout reset, Dispose — plus two CLI leaks the original row didn't know about: transport preview released nothing at all (every preview burned a slot for 30 min) and transport import stranded the session on pre-apply aborts; both now release in a finally. BundleSessionEvictionService is documented as the TTL backstop, not an abandonment path. Fail-before-fix stash-verified (3 of 4 new bUnit tests fail pre-fix). 10 new tests. |
Found live by R6 | Closed 2026-08-15. | — |
| 44 | BundleUnlockRateLimiter is dead code — the documented per-IP unlock throttle is not enforced5ab26a42). BundleImporter.LoadAsync now registers every unlock attempt against MaxUnlockAttemptsPerIpPerHour (10). Ordering is load-bearing: AFTER the per-bundle 3-strike check (hammering a locked bundle spends no hourly budget) and BEFORE the decrypt (a throttle denial burns no per-bundle strike, so the throttle can't be abused to lock out a legitimate import); successful unlocks count too. Denial = BundleUnlockThrottledException (placed in Commons so the management surface returns a curated message). Client key: Central UI via a scoped CircuitClientAddressProvider snapshotting the IP at circuit establishment; CLI via additive ManagementEnvelope.ClientIp. No attributable IP falls into one shared <unattributed> bucket — stricter, never a bypass; per-client isolation is only real behind a ForwardedHeaders-trusted proxy (same caveat as the login throttle, documented). Fail-before-fix verified (4 of 6 new tests fail with the limiter neutralised; 2 negative controls). |
Found by R6 | Closed 2026-08-15. | — |
| 45 | DbContext across up to 4 parallel deliveries9e769735). _adaptersCache/_adaptersScope/ResolveAdapters() deleted; the typed adapter is resolved inside the per-delivery DI scope DeliverGatedAsync already opens (static ResolveAdapter preserving DI last-wins + the null-adapter park path), so each concurrent delivery gets its own adapter and DbContext. Nothing shared was lost: the Email adapter builds a fresh SMTP client per attempt from an injected factory, the SMS adapter draws from the singleton IHttpClientFactory, and CentralUI's adapter-gated Type selector (NotificationChannelCatalog) is an independent scoped consumer, untouched. Fail-before-fix stash-verified (simulated DbContext throwing EF's exact second-operation exception under a rendezvous-guaranteed overlap: 1 of 2 rows persisted pre-fix; 1-vs-4 distinct adapter instances). Root cause of note: the bug was INTRODUCED by code-review finding NotificationOutbox-006's caching "fix" (2026-05-28), whose own regression test pinned the broken behavior and predates bounded parallel delivery — that finding is marked SUPERSEDED. NotificationOutbox 153, AuditLog 383 green. |
Found by R6; observed live on rig central-b | Closed 2026-08-15. | — |
| 46 | audit-config query returns {} for every filter combination09108018). Not a CLI bug: ManagementActor.HandleQueryAuditLog returned the repository's ValueTuple (Entries, TotalCount) straight to SerializeResult; System.Text.Json writes properties only and a ValueTuple's Item1/Item2 are fields, so the whole payload serialized to the literal {} — every filter combination looked empty with no error anywhere in the stack. Fixed with a named Commons record AuditLogQueryResult (mirroring SecuredWriteListResult) plus a class-wide guard: SerializeResult now throws on any ITuple result, so the next tuple-returning handler fails loudly instead of silently emitting {}. Fail-before verified (test asserted exactly "{}" pre-fix); live re-verified on the rig (totalCount 2725 unfiltered, 712 for --entity-type Template --action Delete). The only other tuple-returning repo method in Commons (INotificationOutboxRepository.QueryAsync) is never serialized. |
Found by R6 against the rig; underlying AuditLogEntries rows verified present |
Closed 2026-08-15. | — |
| 47 | SiteStreamGrpcServer duplicate-stream replacement disposed another handler's CancellationTokenSourcee6842c10). SubscribeInstance's duplicate-prevention path called Cancel() AND Dispose() on the replaced stream's CTS, which belongs to that still-running handler's using var and is still read — escaping as an unhandled ObjectDisposedException under scheduling pressure (confirmed pre-existing on 2b74851f; load-only, passed 10/10 isolated). Fixed to cancel-only (the owner's using disposes exactly once), Cancel wrapped for the converse race, mirroring CancelAllStreams(). Deterministic regression test via a gate inside the setup window, fail-before/pass-after verified by reinstating the Dispose() as a negative control. Recorded here (though closed) because it is a product-bug instance of the disposal-race class, useful precedent for future stream-lifecycle work. |
Closed 2026-08-15. | ||
| 48 | b5e6821c). Working set re-seeded on site-a entirely via the CLI (soak-* rows left alone): template RigProbe + instance rig-probe-1, bound to the OPC PLC sim's built-in moving tags (SpikeData, RandomSignedInt32) plus a 5s heartbeat script. Deploy Success; debug snapshot returns live Good-quality data; health site shows OPC PLC Simulator: Connected, good:2 bad:0 uncertain:0; a 60s debug stream produced 239 events; and scadabridge.site.stream.batch_size recorded live for the first time (count 70 / sum 238, batches to ~10 events) — the R2 batching histogram is now observable on the rig, which was this row's purpose. Reproducible via docker/seed-live-probe.sh. Two diagnostic corrections worth keeping: the custom tags in infra/opcua/nodes.json are declared but not simulated by opc-plc (a healthy-looking connection with zero traffic — bind to the built-ins: SpikeData, RandomSignedInt32, StepUp, FastUInt1..5, SlowUInt1..5); and scadabridge_site_connection_up counts open central→site gRPC subscriber streams, not data connections, so this row's original "0 on all six site nodes" was healthy steady state, not the fault (row 54). Close-out postscript: the seeding agent's live verification ran against worktree-homed data dirs (row 68), so after the orchestrator's redeploy from main the config was absent at boot and re-fetched by the startup reconcile — which exposed row 67 — and rig-probe-1 was re-verified live on the main checkout's data dirs (snapshot Good ×3, heartbeat ticking, sim values moving). |
Found during the R2 close-out live probe (2026-08-15) | Closed 2026-08-15. | — |
| 49 | 75520bee). All 576 resolved findings across the 25 component findings files were re-read for the row-45 generalization (a resolution that caches, hoists, or widens the lifetime of a DI-resolved service, later invalidated by a concurrency/lifetime feature). Nine matched or nearly matched the shape; NotificationOutbox-006 is the only one that ever became a bug and was already fixed as row 45. The other eight verify STILL-SAFE or SUPERSEDED-BENIGN — the two with real concurrency nearby each anticipated it explicitly (DeploymentService's dbGate serialises the one repository touch inside the bounded-parallel Send phase; ArtifactDeploymentService builds all per-site commands serially before fanning out, commenting "DbContext is not thread-safe"). An independent product-code cross-check confirms the pattern is absent: no IServiceScope is stored on a field anywhere in src/ (every long-lived holder takes IServiceScopeFactory and opens a scope per operation), every Task.WhenAll/Parallel.* site either touches no scoped service or opens a per-operation scope, and no scoped-registered service is captured by a singleton. One stale archive corrected: NotificationOutbox-004 was marked Resolved with an _Unresolved._ body; the real resolution is now recorded. Near-miss shapes recorded as rows 58–60. |
Found while fixing row 45 | Closed 2026-08-15. | — |
| 50 | 00a804e6). Both mitigations shipped, and they are complementary. (1) ValidateScopes/ValidateOnBuild in Development on both role branches via Host/ServiceProviderValidation.cs — written out rather than left to CreateBuilder's silent default; production stays unvalidated deliberately (fail-closed at boot). Switching it on surfaced two real Site-role defects: ReconcileService and AuditLogKpiSampleSource were registered on sites but need central-only repositories, so neither could ever be constructed there — one unconstructible descriptor fails ValidateOnBuild for the entire site graph, which is exactly why SiteHealthEndpointTests had quietly pinned UseEnvironment("Production") to dodge validation. Both moved to central-only registrations (AddCentralCommunication() / AddAuditLogCentralKpi()); the workaround is gone and the site graph now boots validated. (2) CaptiveDependencyGuard reflects over all 28 shipped actor classes and fails on any instance field (backing fields included) typed IServiceScope, DbContext, or Scoped-in-the-real-containers — the scoped set is derived from the built Central and Site graphs, so a lifetime change auto-re-scopes the check; IServiceProvider/IServiceScopeFactory fields are allowed (the correct pattern). Proven non-vacuous via permanent positive-control dummy actors. No live violation; the two test-only _injectedRepository ctor fields allowlisted with written safety arguments plus a staleness self-test (row 61). Doc: REQ-HOST-11 in Component-Host.md. |
Found while fixing row 45 | Closed 2026-08-15. | — |
| 51 | BundleLockedException reaches the CLI as "An internal error occurred"b449fe20). Relocated to Commons/Types/Transport/ beside BundleUnlockThrottledException (row 44's pattern); both consumers already carried the namespace, so no call-site edits. ManagementActor.HandlePreviewBundle/HandleImportBundle now rethrow it as ManagementCommandException, so CLI and UI print the lockout reason instead of a correlation id. Fault-mapping tests mirrored from row 44 on both handlers. Follow-up in the same diff (73f518d6): deleted the dead BundleUnlockRateLimitedException (superseded by row 44, zero references — three similarly-named unlock exceptions with one inert was a live trap). |
Found while fixing row 44 | Closed 2026-08-15. | — |
| 52 | 677263af). The trailing-hour window now lives in the central MS SQL BundleUnlockAttempt table — the HA store both central nodes already share, so no new infra, port or consensus. MaxUnlockAttemptsPerIpPerHour is a true fleet cap and survives a node restart; config keys, semantics and the rig's 500 override are unchanged, as are all row-44 behaviours (lockout checked first so locked bundles spend no budget; denials burn no strike and are not recorded; <unattributed> bucket; attempts-not-failures). Three recorded trades: fails open (store error → Warning naming client + cap, attempt permitted — availability of legitimate imports outranks a best-effort cap over an already-authenticated Administrator-gated surface; a process-local fallback was considered and rejected as it would still block an operator's legitimate 11th import); approximate (count-then-insert is not atomic across nodes; overshoot bounded by in-flight count, self-corrects within the window — row 64); opportunistic prune on the unlock path (CAS-claimed ≤ every 15 min per process, ≤1000 rows, retains 2× the window so a prune can't race a count — row 65) rather than a new background service. EF migration 20260815154803_AddBundleUnlockAttemptTable + hand-written production script docs/plans/sql/AddBundleUnlockAttemptTable.sql. Integration test proves two limiter instances over one shared store enforce one combined window, restart durability, and store-failure fail-open. |
Row 44 implementation note; documented in docs/components/Transport.md |
Closed 2026-08-15. | — |
| 53 | MaxConcurrentImportSessions has no observability372ea4df). Gauge scadabridge.central.transport.import_sessions.open on the ZB.MOM.WW.ScadaBridge meter — already in SiteServiceRegistration.ObservedMeters (the single shared allowlist source for both roles; no appsettings pins one), so no allowlist change was needed and the series is not silently dropped. Provider-backed like the S&F queue-depth gauge: registered by BundleSessionEvictionService.StartAsync, identity-checked-cleared on StopAsync. IBundleSessionStore.OpenSessionCount counts only unexpired sessions, so the gauge doesn't read "pinned at the cap" for a sweep interval after a burst drains. BundleSessionStore.Open logs Warning with open count, cap and TTL on a genuine cap hit only (a store full of expired sessions is reclaimed silently). No alertable rejection counter yet — row 66. |
Found while fixing row 43 — R6 could only infer the leak from rotating test failures | Closed 2026-08-15. | — |
| 54 | scadabridge_site_connection_up is misleadingly named (Low, observability) |
Row-48 close-out (2026-08-15) — the row's original mis-diagnosis traced directly to this | It counts open central→site gRPC subscriber streams (SiteStreamGrpcServer), not DCL data connections, and correctly reads 0 whenever no viewer is attached. Rename (e.g. scadabridge.site.stream.subscribers.active) and/or add a real data-connection gauge — the data exists on the health report (dataConnectionEndpoints/dataConnectionTagQuality) but neither is exported as a metric. |
Next observability/metrics-naming session, or the next time someone reads this series as data-connection health |
| 55 | tagResolutionCounts reads {totalSubscribed:0, successfullyResolved:0} for a connection with 2 live Good-quality subscribed tags (Low) |
Row-48 live probe: site-a's health report shows the zero counts while dataConnectionTagQuality on the same report says good:2 |
The "device still booting / tag path unresolved" diagnostic is blind if this stays zero. Code inspection at close-out shows DataConnectionActor calls _healthCollector.UpdateTagResolution with the derived counts at all the right lifecycle points, so this is likely not a 491df111 regression — suspect a reporting-path issue instead (e.g. a standby-node collector's empty entry shadowing the active node's, or a keying/removal race in SiteHealthCollector). Needs live debugging. |
Next health-monitoring or DCL session |
| 56 | The rig's central nodes expose no Prometheus endpoint (Low, rig config) | Row-48 live probe | docker/site-*/appsettings.Site.json set MetricsPort: 8084, but neither docker/central-node-a nor -b does, so central-only series (KPI/outbox counters, the new import-session gauge, alarm-cache gauges) cannot be scraped live and any live probe naming "central metrics" is unsatisfiable. |
Next rig/observability session |
| 57 | audit-config query has no CLI table projection (Low, cosmetic) |
Row-46 fix (2026-08-15) | --format table renders the whole entries array as raw JSON inside one cell because the command passes no tableProjector (unlike e.g. template list). JSON output is correct and is the primary format. |
Next CLI session |
| 58 | IDeploymentStatusNotifier singleton retains delegates targeting circuit-scoped Blazor components (Low) |
Row-49 sweep (CentralUI-006 re-verification) | Safe today because the single subscriber unsubscribes in Dispose (pinned by test) and the notifier isolates each handler. |
A second page subscribes, or any subscriber lands without a matching Dispose unsubscribe |
| 59 | SiteCallAuditActor per-tick scope spans all sites × ≤50 upsert pages (Low) |
Row-49 sweep (SiteCallAudit-009 re-verification) | Safe only because site reconciliation is a serial foreach … await (SiteCallAuditActor.cs:649-653); one scope across a parallel fan-out would be row 45 again. |
ReconcileSiteAsync is ever fanned out with Task.WhenAll |
| 60 | ArtifactDeploymentService.GlobalArtifactSnapshot holds EF-materialised entities across the parallel per-site send (Low) |
Row-49 sweep (DeploymentManager-023 re-verification) | Safe because lazy loading is off and the parallel phase reads only materialised POCOs (per-site commands are built serially first, commented "DbContext is not thread-safe"). | Lazy-loading proxies enabled, or per-site command building moved inside the fan-out |
| 61 | _injectedRepository test-mode constructors are one call site from re-introducing row 45 (Low) |
Row-50 guard allowlist: AuditLogIngestActor / SiteCallAuditActor each expose a constructor taking a scoped repository and cache it for the actor's lifetime |
Safe only because no production call site picks that overload (production wiring uses the IServiceProvider overload; SiteCallAuditActor additionally wraps in SerializedRepository). The CaptiveDependencyGuard allowlists both with a staleness self-test. Preferred fix: fold the test fixtures onto the IServiceProvider overload with a stub provider and delete the overloads. |
Anyone adding a production Props for either actor, or the next time those actors are touched |
| 62 | Five SiteRuntime actors resolve ISiteEventLogger off the root provider without a scope (Low) |
Row-50 guard sweep: AlarmActor (which also caches it in a field), ScriptActor, InstanceActor, NativeAlarmActor, DeploymentManagerActor |
Correct today (ISiteEventLogger is AddSingleton), but it is the one pattern that silently becomes row 45 if that registration ever changes lifetime. The guard catches the AlarmActor field case automatically; the per-call root resolves it cannot see. |
Any change to ISiteEventLogger's lifetime, or a second service pulled this way |
| 63 | Rig site containers run without ASPNETCORE_ENVIRONMENT, so the row-50 Development validation never fires there (Low, rig config) |
Row-50 implementation: central containers set it, site containers don't | The site half of the scope validation is exercised only in CI/tests. Setting it on the six site containers would give live coverage now that the site graph validates clean. | Next rig config pass |
| 64 | The fleet-wide unlock throttle's count-then-insert is not atomic across central nodes (Low, accepted by design) | Row-52 implementation trade, documented in both Transport docs | Simultaneous attempts can overshoot MaxUnlockAttemptsPerIpPerHour by the number in flight; bounded and self-correcting within the window. A conditional single-statement insert would close it at the cost of provider-specific raw SQL. |
A threat model needing a hard cap rather than a rate damper, or evidence of a real brute-force exploiting the overshoot |
| 65 | BundleUnlockAttempt rows are pruned only by nodes actively serving unlocks (Low) |
Row-52 implementation trade | The prune rides the unlock path, so a cluster that stops importing entirely leaves the last rows in place indefinitely (bounded and tiny: one row per permitted attempt, retained 2× the window). | The table appears in a size review, or Transport gains a central scheduler a prune tick could ride |
| 66 | No counter for import-session cap rejections (Low) | Row-53 implementation note | Row 53 shipped the open-sessions gauge and a Warning log, so a rejection is diagnosable from logs, but there is no series to alert on. | Someone wants an alert rule on import-session exhaustion rather than a log search |
| 67 | Startup reconcile stores a fetched instance config but does not start the InstanceActor (Medium-Low, availability) | Proven live during the rows-46–53 close-out (2026-08-15): site-a booted without rig-probe-1's config, SiteReconciliationActor's node-STARTUP fetch stored it ("Reconcile: fetched + stored config"), but no InstanceActor started followed — debug snapshot returned instanceNotFound and the enabled, successfully-deployed instance stayed dead until the NEXT restart, whose SQLite load finally started it |
The reconcile heal path heals the file, not the runtime: an instance whose config is missing at boot (lost volume, first boot after restore, rehomed data dir) needs two restarts to run again. Narrow trigger, self-healing, but silent — nothing surfaces "config stored, actor not running". Fix shape: after a startup-reconcile fetch stores a config for an enabled instance with no running actor, tell the DeploymentManager to start it (or re-run the SQLite load path for the fetched set). | Next SiteRuntime/reconcile session |
| 68 | deploy.sh run from a git worktree silently rehomes every node's bind-mount data dir./site-a-node-a/data), so a deploy from a worktree checkout mounts the worktree's copies — the rig keeps running and looks healthy, but its SQLite stores split from the main checkout's and everything written there is orphaned by the next deploy from main. Bit the rows-46–53 close-out live: the seeding agent's rig-probe-1 deploy persisted into worktree-homed dirs and "vanished" after the orchestrator's redeploy (no tombstone, no product bug — LWW/tombstone forensics ruled out deletion; the row sat in .claude/worktrees/…/docker/site-a-node-a/data/site-localdb.db the whole time). Both docker/deploy.sh and docker-env2/deploy.sh now refuse to run from a .claude/worktrees/ path (override: SCADABRIDGE_ALLOW_WORKTREE_DEPLOY=1). |
Found during the rows-46–53 close-out | Closed 2026-08-15. | — |
Resolved (verified against the code 2026-07-10)
Rows removed from the Deferred table above once confirmed shipped. Kept here for traceability.
| # | Item | Resolution |
|---|---|---|
| 10 | Aggregated live alarm stream for Alarm Summary | Shipped 2026-07-10 (docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md): a transient, in-memory per-site central live alarm cache (ISiteAlarmLiveCache/SiteAlarmLiveCacheService + per-site SiteAlarmAggregatorActor) fed by a new site-wide, alarm-only SubscribeSite gRPC stream (SiteStreamManager.SubscribeSiteAlarms), seed-then-stream with dedup + NodeA↔NodeB re-seed + periodic reconcile. Alarm Summary is now live-cache-driven (AlarmSummaryService.BuildFromLiveAlarms) with the 15s poll retained as fallback + NotReporting authority. Honors the [PERM] no-central-store rule — nothing persisted (no EF table/migration). Options on CommunicationOptions (eagerly validated) + two ScadaBridgeTelemetry signals. |
| 7 | SecuredWrite audit rows leave SourceNode NULL | Resolved (PLAN-07): ManagementActor.EmitSecuredWriteAuditAsync routes through ICentralAuditWriter, which stamps SourceNode (central-a/central-b) from INodeIdentityProvider. |
| 12 (CLI/API) | Native-alarm-source-override CSV import | Shipped 2026-07-10: shared CsvLineSplitter, NativeAlarmSourceOverrideCsvParser, bulk all-or-nothing SetInstanceNativeAlarmSourceOverridesCommand + ManagementActor handler (Deployer-gated), CLI instance native-alarm-source import --file, parser/CLI/handler tests. UI upload affordance shipped 2026-08-01 — second InputFile on the InstanceConfigure Native Alarm Source Overrides card reusing the shared parser, mirroring the attribute importer's UX and the server's all-or-nothing merge semantics (InstanceConfigureNativeAlarmCsvImportTests); row 12 removed from the Deferred table. |
| 13 | WaitForAttribute quality-gated ("Good"-only) mode | Already implemented (Commons WaitForAttribute.RequireGoodQuality, enforced in InstanceActor, threaded through ScriptRuntimeContext, tested in InstanceActorWaitForAttributeTests). Stale "planned enhancement" doc line corrected 2026-07-10. |
| 14 | WaitForAttribute in Test-Run sandbox | Shipped 2026-07-10 (full fidelity): sandbox Attributes.WaitAsync/WaitForAsync (value-equality) route to the bound instance via ISandboxInstanceGateway.WaitForAttributeAsync → the existing CommunicationService.RouteToWaitForAttributeAsync cross-site route. Additive RouteToWaitForAttributeRequest.RequireGoodQuality (honored by the site handler) makes quality-gated waits route too. Predicate-form waits stay unsupported (an in-process lambda can't be routed) and throw a labelled ScriptSandboxException. Tests: sandbox accessor routing (CentralUI), site-handler quality-flag threading (SiteRuntime). |
| 15 | BrowseNext final-page signal not surfaced | Already surfaced (M7 browse work): RealOpcUaClient sets Truncated=false/ContinuationToken=null on the last page; BrowseNodeResult carries both; TreeRow.razor renders "Load more" only when a continuation token remains — no wasted BrowseNext. |
| 16 | StubOpcUaClient throws on browse | Already resolved: StubOpcUaClient supports browse + address-space search, covered by StubOpcUaClientBrowseTests/StubOpcUaClientSearchTests. |
| 18 | Folder drag-drop | Closed — permanently deferred ([PERM]) by the M9 decision (docs/plans/2026-06-15-stillpending-completion-design.md:122): menu-based reorder (T23) shipped instead, and the folder-hierarchy design fixed the reorganization UX as "right-click context menus only (no drag-drop)" (2026-05-11-templates-folder-hierarchy-design.md:27). Row removed from the Deferred table 2026-08-01 — nothing left to revisit. |
| 20 | Deployment EXPIRED-row purge | Already resolved (PLAN-04): PendingDeploymentPurgeActor central singleton (spawned in AkkaHostedService) ticks IDeploymentManagerRepository.PurgeExpiredPendingDeploymentsAsync every CommunicationOptions.PendingDeploymentPurgeInterval (default 1h), options-validated, tested. |
| 21 | SiteAuditBacklogReporter threshold consolidation | Shipped 2026-07-10: SqliteAuditWriterOptions.BacklogPollIntervalSeconds (default 30) now drives the reporter's poll cadence; explicit ctor override still wins (tests), non-positive falls back to the 30 s default. Cadence tests added; stale "hard-code / follow-up" class-doc corrected. |
| 22 | KPI history hourly rollups | Shipped 2026-07-10 (docs/plans/2026-07-10-kpi-history-hourly-rollups-plan.md, T1–T8): new KpiRollupHourly table (migration 20260710153953) folded by a third recorder tick (kpi-rollup, RollupInterval default 1h) over a re-folded RollupLookbackHours window via an idempotent, failover-self-healing upsert; per-metric gauge-vs-rate aggregation (KpiMetricAggregationCatalog); a one-shot backfill of the retention window on start; raw-vs-rollup query routing by RollupThresholdHours (default 168h); longer rollup retention (RollupRetentionDays default 365 ≥ RetentionDays, dual daily purge); and 30 d / 90 d trend windows added to the four surfaces. Options + validator, docs (Component-KpiHistory.md), and tests shipped. |
New deferrals from review 08 (this plan)
| Item | Rationale | Revisit trigger |
|---|---|---|
| Communication → HealthMonitoring layering (ICentralHealthAggregator consumed by CentralCommunicationActor.cs:351) | Moving the interface + SiteHealthState to Commons ripples across 5 projects for a cosmetic inversion | Next breaking change to ICentralHealthAggregator |
| docs/components reference docs for ScriptAnalysis, KpiHistory, DelmiaNotifier | Reference docs are substantial (StyleGuide-conformant); README claim scoped instead (PLAN-08 Task 10) | Next doc-writing session touching those components |
| Test-coverage backfill: SiteCallAudit.Tests (31 tests/1.6k LOC), DeploymentManager.Tests | No defect identified; coverage partly lives in ManagementService/Host/Integration suites | First regression escaping either component |
~tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs is no longer a skipped placeholder: it runs as a live [Fact] (Category=Performance) on the real two-node in-process rig (TwoNodeClusterFixture, production BuildHocon) at production timings — 2s heartbeat / 10s failure-detection threshold / 15s stable-after — hard-killing the younger node and timing the survivor's member REMOVAL with singleton continuity asserted on the oldest. Delivered by PLAN-R2-01 Task 4 (archreview/plans/PLAN-R2-01-cluster-host-failover.md:226). The oldest-crash direction is covered behaviorally by SbrFailoverTests.AutoDown_HardCrashOfOldestNode_* and by docker/failover-drill.sh. |
The 2026-07-08 "PLAN-01 rig landing" trigger had fired unnoticed (NF2); PLAN-R2-01 T4 wired the placeholder to the fixture rig rather than recording a blocker. | Closed. |
DefaultRetryInterval (measured 28.9 s) before anything drains, since EnqueueAsync(attemptImmediateDelivery: false) stamps LastAttemptAt and the due-predicate then holds the row for one interval (correct by design; Notify.Send's deferToSweep: true leaves it null and skips the wait). Filed as informational finding F2. (b) Slow-subscriber backpressure: with 5 subscribers on the SAME instance and one reader stalled at 50 ms/event, the four healthy subscribers delivered 100.00% with zero drops while the stalled one lost 197,028 of 200,000 events entirely within its own bounded channel. Isolation is structural and total — Buffer(DropHead) never backpressures the BroadcastHub, Sink.ForEach's Tell never blocks, and StreamRelayActor.WriteToChannel's TryWrite on a DropOldest channel never blocks — so a slow WAN link degrades only its own feed, visibly (per-stream eviction counter + RecordSiteStreamEventDropped). Finding F3. Caveat also recorded: an unpaced burst costs every subscriber, because the publish Source.ActorRef upstream of the hub is shared — that shared stage, not per-subscriber buffering, bounds burst absorption. |
Closed by the row-25 run; no defect found in either half. | Closed 2026-08-15 — docs/plans/2026-08-15-target-scale-load-test-results.md. |
Deferred — operational risk (from the initiative tracker, folded in 2026-07-12)
Two live items previously tracked ONLY in archreview/plans/00-MASTER-TRACKER.md's registry are folded in here (NF5) so this register is the single tracking place. The tracker's narrative subsections remain as the historical evidence.
| # | Item | Where noted | Rationale for deferral | Revisit trigger |
|---|---|---|---|---|
| SBR | keep-oldest to the auto-down downing strategy (Akka AutoDowning, auto-down-unreachable-after = 15s): a hard crash of EITHER node — active/oldest included — now fails over to the survivor in ~25s with no operator action. Accepted trade: a real network partition produces dual-active until an operator restarts one side. Decision record + evidence (live keep-oldest DownReachable … including myself log, Akka.NET 1.5.62 KeepOldest.OldestDecision source, rejected alternatives incl. the static-quorum-1 DownAll trap): docs/plans/2026-07-21-auto-down-availability-decision.md. |
archreview/plans/00-MASTER-TRACKER.md:194 + auto-memory sbr-keep-oldest-2node-active-crash-gap (both now historical) |
— | Closed. Residual: seed-node boot-alone constraint (unchanged, documented in Component-ClusterInfrastructure.md); dual-active recovery is operator-driven. |
| vd03 | deploy/wonder-app-vd03/ overlay edits unapplied — appsettings.Central.json needs AllowSingleNodeCluster: true + phantom-seed removal + NodeName: central-a; install.ps1 needs sc.exe failure recovery actions. The deploy/wonder-app-vd03/ artifact directory is intentionally untracked (production config out of source control), so the repo cannot ship the fix. |
archreview/plans/00-MASTER-TRACKER.md:198 (PLAN-01 T16/T20/T23) |
Needs on-host access; without NodeName that deployment's audit rows stamp NULL SourceNode — partially mitigated once PLAN-R2-08 Task 7 lands: the host now FAILS AT BOOT with a key-naming error instead of silently NULLing, so applying the overlay becomes mandatory at the next upgrade. Owner: whoever maintains the host (user). |
Next wonder-app-vd03 deployment/upgrade — the Task 7 validator makes this row unskippable then. |