Joseph Doherty
33 KiB
33 KiB
Deferred-Work Register (established 2026-07-08, from architecture review 08)
Single tracked list of consciously-deferred work. Rules: every deferral gets a row (rationale + revisit trigger); fix-now items reference the archreview plan that owns them and are removed from this table when that plan's task lands.
Fix-now (owned by archreview plans)
All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round 2, 2026-07-12) — rows removed per the rule above; see the round-2 report §1 for the per-item evidence.
Deferred (with rationale + revisit trigger)
| # | Item | Where noted | Rationale for deferral | Revisit trigger |
|---|---|---|---|---|
| 8 | Hash-chain tamper evidence (T1); CLI verify-chain is a no-op stub | audit-log roadmap :12 | v1.x by locked decision; append-only DB roles are the control | Compliance requirement for cryptographic tamper evidence |
| 9 | Parquet audit archival (T2); endpoint returns 501 | AuditEndpoints.cs:204 | v1.x; 501 + CLI messaging are honest | AuditLog partition volume nears retention ceiling |
| 11 | Central-persisted OPC UA cert-trust audit | m7 follow-ups | Broadcast-to-both-nodes covers HA | Governance/audit requirement for trust decisions |
| 17 | Unified notifications+site-calls outbox page | stillpending :118 | Explicit M9 decision to keep two pages | Operator confusion reports |
| 19 | Bundle signing / cluster-to-cluster pull / differential bundles | transport-design :402 | v1 manifest hash + AES-GCM held sufficient | Non-repudiation requirement across orgs |
| 23 | Live LDAP group-membership re-query for an active session | docs/requirements/Component-Security.md :61-69 (+ :78-79) |
Blocked on an external package. The mid-session refresh re-maps the stored groups against the central DB with no LDAP call, so a directory group-membership change lands only at next login. A live re-query needs a passwordless service-account group-search method on the shared ZB.MOM.WW.Auth.Ldap library — an external NuGet PackageReference (src/ZB.MOM.WW.ScadaBridge.Security/…csproj:23) exposing only AuthenticateAsync(username, password, ct). Central role-mapping/scope changes still apply within ~15 min (RoleRefreshThresholdMinutes). |
ZB.MOM.WW.Auth.Ldap gains a standalone group-search API, or a requirement that a directory-side group revocation take effect mid-session rather than at next login |
| 24 | M8 large-bundle performance hardening | docs/plans/2026-06-15-stillpending-completion-design.md:106 — "Small follow-ups logged (not blocking): … large-bundle/perf hardening" |
Logged as a non-blocking follow-up when M8 shipped and never given an artifact: no plan, no task entry, no perf/load test exists (tests/…Transport.Tests/Import/BundleImporterLoadTests.cs is a LoadAsync unit suite despite the name). No measured problem; the only sizing controls in place are the 5-minute CLI transport timeout, LineDiffer's MaxInputLines=4000 summary-only cap, and MaxConcurrentImportSessions=8. |
First real bundle that times out, exhausts memory, or makes the import wizard's diff step unusable |
| 25 | docker/ rig still running. Headline numbers: tag update latency P50 0.88 ms / P99 4.57 ms / max 37.41 ms end-to-end (DCL boundary to stream subscriber, the emit instant carried verbatim through production code); 0 events dropped at 100 live subscribers; health report collect+ingest P99 0.31 ms with 10/10 sites tracked; debug view snapshot under load P99 2.19 ms, 0 timeouts; 500 instances deployed to a site in 2.6 s; CPU 2.9% of the box; working-set slope +8.83 MB/min. 11 clean passes, 1 pass with a caveat (F1: 20 min with zero gen-2 collections cannot fully settle the leak question — the 1-hour run would), 0 failures. Harness: tests/ZB.MOM.WW.ScadaBridge.LoadHarness/ (standalone Exe — the Category=Performance trait enables a filter but does NOT exclude by default, so a 20-minute test could not live in PerformanceTests) plus a CI-scale smoke [Fact] (PerformanceTests/TargetScale/TargetScaleHarnessSmokeTests.cs). Real: per-site ActorSystem + LocalDb, the real DCL via the documented DataConnectionFactory.RegisterAdapter seam, real InstanceActors, real SiteStreamManager, real StreamRelayActor + production-capacity bounded channel, real S&F, real health collector/aggregator; only the socket hops are stood in for. Results: docs/plans/2026-08-15-target-scale-load-test-results.md; design + deviations + thresholds: docs/plans/2026-08-15-target-scale-load-test-design.md. |
as before | Closed. Residual scope: [xc-3] site-event-log retention volume, [xc-4] audit-vs-central degradation, [xc-5] large-template flattening and [xc-8] UI responsiveness are NOT covered by this harness (central-cluster/browser concerns) and remain open WP-4 scope — see results doc SS6. |
Closed 2026-08-15. |
| 26 | Ipsen MES MoveIn tail: leak-test (-LT) receivers + routing, PLC-output-flag writes, Z28062 BTDB data completeness |
docs/plans/2026-06-16-ipsen-mes-movein.md:409 ("Out of scope (future)"); design 2026-06-16-ipsen-mes-movein-design.md:58-60, :196-198 |
Customer-site scope, not a platform gap. -LT routing needs an MES-receiver child + Galaxy reference that do not exist on the reactor template (any -LT/unknown suffix returns WasSuccessful=false with an "unsupported side/target" message by decision); MoveInComplete/Successful/ErrorText are PLC-owned by locked decision, so ScadaBridge deliberately does not write them; Z28062 completeness is an operational data fix, not code. Note the separate alarm-status path already handles the suffix — _LT is stripped before side-scoping (2026-06-30-mes-alarm-status-api.md:158). |
Ipsen creates the leak-test receiver + Galaxy reference, or asks ScadaBridge to own the PLC-output flags — otherwise a candidate won't-do ([PERM]) at the next Ipsen scope review |
| 27 | External-system per-system retry config (MaxRetries/RetryDelay) never reaches sites, and has no CLI/management surface |
Found live 2026-08-01 (rig session, #11 gRPC live checks) | Two stacked gaps: (a) ExternalSystemArtifact (Commons) carries TimeoutSeconds but NOT MaxRetries/RetryDelay, and the site external_systems table has no such columns — so a centrally-configured retry policy is silently ignored on sites; every cached call buffers with the S&F default (DefaultMaxRetries 50 × DefaultRetryInterval 30s ≈ 25 min to park). (b) Create/UpdateExternalSystemCommand don't expose the fields either — the only way to set them today is a direct DB edit of ExternalSystemDefinitions. Transport bundles DO carry them (arch-review 05 "ES retry config"), which masks the gap in export/import round-trips. Fix is additive: extend the artifact + site schema + apply path, and add --max-retries/--retry-delay to the CLI. |
First operator who tunes retry policy on an external system and expects site cached calls to honor it |
| 28 | Health-dashboard "Trigger failover" confirm dialog's confirm button is labeled "Delete" | Found live 2026-08-01 (rig session, #11 TriggerSiteFailover check) | The DialogService confirmation host's default destructive-action label leaks through — the dialog copy is correct but the red confirm button says "Delete" for a failover. One-line fix: pass an explicit confirm label ("Fail over") at the Health-dashboard call site (and audit other confirm-dialog call sites for the same default). | Next Central UI session |
| 29 | 1-hour sustained run to settle the memory-slope ambiguity (load-test finding F1) | docs/plans/2026-08-15-target-scale-load-test-results.md F1 (Low) |
The row-25 20-minute run PASSED the working-set threshold (+8.83 MB/min < 20) but recorded zero gen-2 collections across 45M events, so the +19.52 MB/min managed-heap drift on an uncompacted gen-2 is ambiguous — the heap demonstrably sawtooths, but 20 min cannot distinguish "gen-2 not yet collected" from a slow leak. Not a defect finding; an evidence gap. The harness exists (tests/ZB.MOM.WW.ScadaBridge.LoadHarness/) and the full 1-hour protocol is one documented command (results doc §7) — deferred only because the measured window was scaled for practicality. |
Before production go-live at target scale, or first field report of unexplained site-node memory growth |
| 30 | WP-4 residual scope: [xc-3] site-event-log retention volume, [xc-4] audit-load-vs-central degradation, [xc-5] large-template flattening time, [xc-8] UI responsiveness under load |
docs/plans/2026-08-15-target-scale-load-test-results.md §6 + row 25 closure note |
The row-25 harness is a site-runtime rig; these four WP-4 sub-criteria are central-cluster/browser concerns it deliberately does not cover. Recorded so the row-25 closure never reads as "WP-4 fully done" — phase-8 WP-4 is closed for the site-runtime data path only. | Before declaring phase-8 WP-4 complete, or production go-live at target scale |
| 31 | Unpaced-burst absorption is bounded by the shared publish stage, not per-subscriber buffers (load-test finding F3 caveat) | docs/plans/2026-08-15-target-scale-load-test-results.md F3 |
Measurement-backed design note, no defect: slow-subscriber isolation is structural and total (a stalled reader loses events only in its own bounded DropOldest channel; healthy subscribers delivered 100.00%), but the publish Source.ActorRef upstream of the BroadcastHub is shared, so an unpaced burst degrades every subscriber equally — that stage, not per-subscriber buffering, is the burst ceiling. |
First field report of event loss on healthy subscribers during a burst, or any redesign of the site stream publish stage |
| 32 | QueueDepthGaugeTests full-suite flake — shared static gauge (load-test finding F4) |
tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/QueueDepthGaugeTests.cs; surfaced by the row-25 verification runs (docs/plans/2026-08-15-target-scale-load-test-results.md F4) |
Pre-existing, test-only: fails when the full suite runs concurrently, passes in isolation — the gauge under test is process-global static state shared across test classes. Zero src/ involvement; needs test isolation (per-test meter/collector instance), not a product fix. |
Next test-infra session, or the first time it fails a CI/full-suite run that matters |
| 33 | LocalDbOplogBacklog has no threshold, KPI history, or dashboard treatment |
Found by R7 (docs/plans/2026-08-15-residual-remediation-plan.md) while documenting the purge burst |
SiteAuditBacklog drives a configurable threshold warning on the site tile, but the LocalDb backlog is reported and Prometheus-exported with nothing consuming it for alerting or trending — it is not among the metrics SiteHealthKpiSampleSource samples into KPI history, so the purge-burst spike R7 documents cannot be reviewed after the fact; an operator must be watching live. Candidate: add it to the KPI sample source and/or a sustained-backlog threshold badge. Pairs naturally with the R1 per-table-snapshot work, which changes snapshot/backlog behavior. |
Next health-dashboard session, or first operator question about a backlog spike they didn't see live |
| 34 | site_events purge burst is unbounded in the oplog and unmeasured at scale |
Found by R7 | Retention deletes are sliced at 1000 rows/statement, but the whole expired set is deleted in one tick, so a replicated site's oplog gains one row per deleted event with no pacing — a first purge after enabling replication (or after a long low-purge period) queues the entire batch at once. The wire path is bounded (MaxBatchBytes), but the oplog write itself is not paced, and no measurement exists for batch size or drain time at target scale. Candidate: measure on the rig or in the load harness; consider a per-tick delete ceiling if drain time is material. |
Before enabling replication on a site with a large existing site_events backlog, or the next load-harness session |
| 35 | DeploymentRecords index does not cover the deployment-list sort key |
Found by R3 (35ce1413) |
The table has a single-column IX DeployedAt; the new list query orders DeployedAt DESC, Id DESC and joins Instance, so SQL Server sorts per page. A composite (DeployedAt DESC, Id DESC) index would serve the ordering directly — deliberately not done in R3 because it needs an EF migration and the terminal-record purge bounds the table. (Low) |
Deployment list measurably slow at a real fleet's retention depth |
| 36 | Deployments free-text search is unindexable by construction | Found by R3 | Four LIKE '%term%' predicates (instance name, deployment id, revision hash, user) — leading wildcards preclude index seeks, so a search is a scan of the join. Acceptable at the purge-bounded row count. Fix if hot: anchor the id/hash legs as prefix matches, or full-text. (Low) |
Search latency complaints on the Deployments page |
| 37 | Two paged reads of DeploymentRecords with different contracts |
Found by R3 | QueryDeploymentSummariesAsync (CLI/ManagementActor: offset, no total, no instance join, instance-id-list scoping) and R3's QueryDeploymentListPageAsync (UI: total + instance join + site-id scoping) duplicate query logic. Consolidating on one contract — giving the CLI a total, both scoping by site — removes the duplication. Genuinely different contracts today, so not urgent. (Low) |
Next time either query's shape changes |
| 38 | OtOpcUa: RoslynVirtualTagEvaluatorTests racing-clear test is load-fragile (OtOpcUa repo, not this one) |
Found by R4 during the full-suite bump verification | Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed fails under full-solution parallel test load, passes 3/3 isolated: its design assumes at most one ClearCompiledScripts lands inside one evaluation (so the product's one-shot retry suffices); under contention two can. Same timing-assumption family as the R5 sweep, but lives in OtOpcUa. Test-side fix: bound clear cadence relative to observed evaluation latency, or assert on retry-exhaustion instead of zero failures. |
Next OtOpcUa test session, or when it fails a run that matters |
| 39 | OtOpcUa: late-enabled replication cannot baseline pre-existing rows (OtOpcUa repo, not this one) | Found by R4; documented in OtOpcUa's CLAUDE.md | LocalDbSetup.OnReady registers all four tables unconditionally on every driver node, so it can never pass baselineExistingRows: true (only correct when gated on replication being configured). Turning replication ON for a long-running default-OFF node converges on subsequent writes only — existing rows are never baselined. ScadaBridge closed this class with conditional registration + boot-time DeregisterReplicated self-heal (WP1.3/WP3.3); porting it is a behavior change beyond a pin bump (registered-set pinned by tests, deregistration must be pair-symmetric). Low current impact: site-a was enabled from fresh. |
Before enabling replication on an established OtOpcUa node |
| 40 | Wall-clock sleeps used as synchronization in tests (absence-assertion class) | Found by the R5 sweep (9fb52153) — a different class from the one swept, deliberately left unfixed |
Tests that sleep, then assert an absence: ScriptDeadlineAtEnqueueTests.cs:105,128 (two Thread.Sleep(2s), with Assert.Equal(0, scheduler.DetachedThreadCount) depending on the sleep out-racing the watchdog — worst offender), DeploymentWarmThenGateTests.cs:117, SiteAlarmLiveCacheServiceTests.cs:195,216, DataConnectionActorTests.cs:435 (Task.Delay(200) to "settle" after an AwaitCondition). These fail toward false-pass rather than flake, so they don't trip CI — but each is a timing pin that weakens what the test proves. Fix pattern: replace the sleep with an explicit observable edge (or assert on the positive event with a bounded wait plus a negative control). |
Next test-infra session, or the first time one of these areas regresses without its test catching it |
| 41 | LocalDb (scadaproj) sync-session tests flake under parallel load (scadaproj repo, not this one) | Found by R1; confirmed pre-existing at base f627cee (1 failure in 3 full runs) |
SyncSessionTests.Handshake_DigestMismatch_FailsClosed and BatchBudgetTests.Pump_RowLargerThanTheBudget_IsSentAloneRatherThanStallingTheStream intermittently time out. Root cause looks structural, not a timing pin: on a handshake fault, RunAsync's catch cancels the writer loop, which can kill the session's OWN Handshake before it transmits — the healthy peer then waits out its full 15 s cap for a message that never comes. Product-adjacent (the same early-cancel could delay a real peer's fail-closed verdict), so worth a look beyond the tests. |
Next LocalDb session |
| 42 | build/push.sh-class publish bug across the family repos |
Found by R1 the hard way (incident: 25 unrelated packages briefly published to the feed, deleted + verified same session; script fixed in scadaproj 0038d46) |
The bug shape: dotnet pack -c Release -o ./artifacts with no project argument and no cd, so the script packs the CALLER'S CWD. Fixed in LocalDb's script (cd to own repo root, name the solution, absolute output path). Peer repos' publish scripts (MxAccessGateway, HistorianGateway, OtOpcUa, others in scadaproj) should be checked for the same shape — the recovery only worked because the token happened to carry delete rights. |
Before anyone runs a family publish script from outside its repo, or next umbrella-repo session |
| 43 | Central UI import wizard leaks a BundleSession slot on every abandoned import — imports lock out for 30 min (High — availability + secret lifetime) |
Found live by R6 (reproduced: 3 back-to-back suite runs → 6 of 8 slots held, rotating pair of TransportImportTests failures; central-b restart cleared it) |
TransportImport.razor.cs: ResetSessionState() nulls the local field and Dispose() deletes only the staged temp file — neither calls IBundleSessionStore.Remove(sessionId); only BundleImporter.ApplyAsync removes sessions. An operator who unlocks a bundle then navigates away burns one of MaxConcurrentImportSessions (8) for the full BundleSessionTtlMinutes (30); eight abandonments lock out ALL imports on the active node. The leaked session also holds the decrypted bundle content that T-007 deliberately zeroes on the Apply paths — so this is a secret-lifetime hole too. The class doc's "an abandoned wizard does not leak" claim is true of the plaintext file only. Fix: remove the session (and zero its content) from Dispose/ResetSessionState/back-navigation. |
Fix-now candidate — next session touching Transport or Central UI |
| 44 | BundleUnlockRateLimiter is dead code — the documented per-IP unlock throttle is not enforced (Medium — security) |
Found by R6 | Registered, injected into BundleImporter, unit-tested, documented on TransportOptions.MaxUnlockAttemptsPerIpPerHour = 10, pinned in appsettings.Central.json — but _unlockRateLimiter is never invoked anywhere in src/. A brute-force passphrase attacker is bounded only by the per-bundle 3-strike lockout, not the advertised 10/hour/IP cap. Fix: call it on the unlock path (or delete it and the config surface honestly). |
Fix-now candidate — next Transport/security session |
| 45 | Notification Outbox dispatcher shares one DbContext across up to 4 parallel deliveries (Medium — correctness under load) |
Found by R6; observed continuously on rig central-b: InvalidOperationException: A second operation was started on this context instance on dispatch |
NotificationOutboxActor.ResolveAdapters() caches adapter instances in an actor-lifetime IServiceScope while the Email/Sms adapters are AddScoped and hold a scoped INotificationRepository (ScadaBridgeDbContext); MaxParallelDeliveries (4) then shares one context across concurrent deliveries. The per-delivery scope added in DeliverGatedAsync for exactly this hazard covers only INotificationOutboxRepository — the adapter escapes it. Effect today: failed attempts + retries + error noise; sustained contention could burn a row to Parked. Fix: resolve adapters inside the per-delivery scope. |
Fix-now candidate — next Notification Outbox session, or first Parked row attributed to it |
| 46 | CLI audit-config query returns {} for every filter combination (Low) |
Found by R6 against the rig; underlying AuditLogEntries rows verified present |
Possibly a CLI output-shape/serialization issue; not investigated (out of R6 scope). | Next CLI session |
| 47 | SiteStreamGrpcServer duplicate-stream replacement disposed another handler's CancellationTokenSourcee6842c10). SubscribeInstance's duplicate-prevention path called Cancel() AND Dispose() on the replaced stream's CTS, which belongs to that still-running handler's using var and is still read — escaping as an unhandled ObjectDisposedException under scheduling pressure (confirmed pre-existing on 2b74851f; load-only, passed 10/10 isolated). Fixed to cancel-only (the owner's using disposes exactly once), Cancel wrapped for the converse race, mirroring CancelAllStreams(). Deterministic regression test via a gate inside the setup window, fail-before/pass-after verified by reinstating the Dispose() as a negative control. Recorded here (though closed) because it is a product-bug instance of the disposal-race class, useful precedent for future stream-lifecycle work. |
Closed 2026-08-15. |
Resolved (verified against the code 2026-07-10)
Rows removed from the Deferred table above once confirmed shipped. Kept here for traceability.
| # | Item | Resolution |
|---|---|---|
| 10 | Aggregated live alarm stream for Alarm Summary | Shipped 2026-07-10 (docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md): a transient, in-memory per-site central live alarm cache (ISiteAlarmLiveCache/SiteAlarmLiveCacheService + per-site SiteAlarmAggregatorActor) fed by a new site-wide, alarm-only SubscribeSite gRPC stream (SiteStreamManager.SubscribeSiteAlarms), seed-then-stream with dedup + NodeA↔NodeB re-seed + periodic reconcile. Alarm Summary is now live-cache-driven (AlarmSummaryService.BuildFromLiveAlarms) with the 15s poll retained as fallback + NotReporting authority. Honors the [PERM] no-central-store rule — nothing persisted (no EF table/migration). Options on CommunicationOptions (eagerly validated) + two ScadaBridgeTelemetry signals. |
| 7 | SecuredWrite audit rows leave SourceNode NULL | Resolved (PLAN-07): ManagementActor.EmitSecuredWriteAuditAsync routes through ICentralAuditWriter, which stamps SourceNode (central-a/central-b) from INodeIdentityProvider. |
| 12 (CLI/API) | Native-alarm-source-override CSV import | Shipped 2026-07-10: shared CsvLineSplitter, NativeAlarmSourceOverrideCsvParser, bulk all-or-nothing SetInstanceNativeAlarmSourceOverridesCommand + ManagementActor handler (Deployer-gated), CLI instance native-alarm-source import --file, parser/CLI/handler tests. UI upload affordance shipped 2026-08-01 — second InputFile on the InstanceConfigure Native Alarm Source Overrides card reusing the shared parser, mirroring the attribute importer's UX and the server's all-or-nothing merge semantics (InstanceConfigureNativeAlarmCsvImportTests); row 12 removed from the Deferred table. |
| 13 | WaitForAttribute quality-gated ("Good"-only) mode | Already implemented (Commons WaitForAttribute.RequireGoodQuality, enforced in InstanceActor, threaded through ScriptRuntimeContext, tested in InstanceActorWaitForAttributeTests). Stale "planned enhancement" doc line corrected 2026-07-10. |
| 14 | WaitForAttribute in Test-Run sandbox | Shipped 2026-07-10 (full fidelity): sandbox Attributes.WaitAsync/WaitForAsync (value-equality) route to the bound instance via ISandboxInstanceGateway.WaitForAttributeAsync → the existing CommunicationService.RouteToWaitForAttributeAsync cross-site route. Additive RouteToWaitForAttributeRequest.RequireGoodQuality (honored by the site handler) makes quality-gated waits route too. Predicate-form waits stay unsupported (an in-process lambda can't be routed) and throw a labelled ScriptSandboxException. Tests: sandbox accessor routing (CentralUI), site-handler quality-flag threading (SiteRuntime). |
| 15 | BrowseNext final-page signal not surfaced | Already surfaced (M7 browse work): RealOpcUaClient sets Truncated=false/ContinuationToken=null on the last page; BrowseNodeResult carries both; TreeRow.razor renders "Load more" only when a continuation token remains — no wasted BrowseNext. |
| 16 | StubOpcUaClient throws on browse | Already resolved: StubOpcUaClient supports browse + address-space search, covered by StubOpcUaClientBrowseTests/StubOpcUaClientSearchTests. |
| 18 | Folder drag-drop | Closed — permanently deferred ([PERM]) by the M9 decision (docs/plans/2026-06-15-stillpending-completion-design.md:122): menu-based reorder (T23) shipped instead, and the folder-hierarchy design fixed the reorganization UX as "right-click context menus only (no drag-drop)" (2026-05-11-templates-folder-hierarchy-design.md:27). Row removed from the Deferred table 2026-08-01 — nothing left to revisit. |
| 20 | Deployment EXPIRED-row purge | Already resolved (PLAN-04): PendingDeploymentPurgeActor central singleton (spawned in AkkaHostedService) ticks IDeploymentManagerRepository.PurgeExpiredPendingDeploymentsAsync every CommunicationOptions.PendingDeploymentPurgeInterval (default 1h), options-validated, tested. |
| 21 | SiteAuditBacklogReporter threshold consolidation | Shipped 2026-07-10: SqliteAuditWriterOptions.BacklogPollIntervalSeconds (default 30) now drives the reporter's poll cadence; explicit ctor override still wins (tests), non-positive falls back to the 30 s default. Cadence tests added; stale "hard-code / follow-up" class-doc corrected. |
| 22 | KPI history hourly rollups | Shipped 2026-07-10 (docs/plans/2026-07-10-kpi-history-hourly-rollups-plan.md, T1–T8): new KpiRollupHourly table (migration 20260710153953) folded by a third recorder tick (kpi-rollup, RollupInterval default 1h) over a re-folded RollupLookbackHours window via an idempotent, failover-self-healing upsert; per-metric gauge-vs-rate aggregation (KpiMetricAggregationCatalog); a one-shot backfill of the retention window on start; raw-vs-rollup query routing by RollupThresholdHours (default 168h); longer rollup retention (RollupRetentionDays default 365 ≥ RetentionDays, dual daily purge); and 30 d / 90 d trend windows added to the four surfaces. Options + validator, docs (Component-KpiHistory.md), and tests shipped. |
New deferrals from review 08 (this plan)
| Item | Rationale | Revisit trigger |
|---|---|---|
| Communication → HealthMonitoring layering (ICentralHealthAggregator consumed by CentralCommunicationActor.cs:351) | Moving the interface + SiteHealthState to Commons ripples across 5 projects for a cosmetic inversion | Next breaking change to ICentralHealthAggregator |
| docs/components reference docs for ScriptAnalysis, KpiHistory, DelmiaNotifier | Reference docs are substantial (StyleGuide-conformant); README claim scoped instead (PLAN-08 Task 10) | Next doc-writing session touching those components |
| Test-coverage backfill: SiteCallAudit.Tests (31 tests/1.6k LOC), DeploymentManager.Tests | No defect identified; coverage partly lives in ManagementService/Host/Integration suites | First regression escaping either component |
~tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs is no longer a skipped placeholder: it runs as a live [Fact] (Category=Performance) on the real two-node in-process rig (TwoNodeClusterFixture, production BuildHocon) at production timings — 2s heartbeat / 10s failure-detection threshold / 15s stable-after — hard-killing the younger node and timing the survivor's member REMOVAL with singleton continuity asserted on the oldest. Delivered by PLAN-R2-01 Task 4 (archreview/plans/PLAN-R2-01-cluster-host-failover.md:226). The oldest-crash direction is covered behaviorally by SbrFailoverTests.AutoDown_HardCrashOfOldestNode_* and by docker/failover-drill.sh. |
The 2026-07-08 "PLAN-01 rig landing" trigger had fired unnoticed (NF2); PLAN-R2-01 T4 wired the placeholder to the fixture rig rather than recording a blocker. | Closed. |
DefaultRetryInterval (measured 28.9 s) before anything drains, since EnqueueAsync(attemptImmediateDelivery: false) stamps LastAttemptAt and the due-predicate then holds the row for one interval (correct by design; Notify.Send's deferToSweep: true leaves it null and skips the wait). Filed as informational finding F2. (b) Slow-subscriber backpressure: with 5 subscribers on the SAME instance and one reader stalled at 50 ms/event, the four healthy subscribers delivered 100.00% with zero drops while the stalled one lost 197,028 of 200,000 events entirely within its own bounded channel. Isolation is structural and total — Buffer(DropHead) never backpressures the BroadcastHub, Sink.ForEach's Tell never blocks, and StreamRelayActor.WriteToChannel's TryWrite on a DropOldest channel never blocks — so a slow WAN link degrades only its own feed, visibly (per-stream eviction counter + RecordSiteStreamEventDropped). Finding F3. Caveat also recorded: an unpaced burst costs every subscriber, because the publish Source.ActorRef upstream of the hub is shared — that shared stage, not per-subscriber buffering, bounds burst absorption. |
Closed by the row-25 run; no defect found in either half. | Closed 2026-08-15 — docs/plans/2026-08-15-target-scale-load-test-results.md. |
Deferred — operational risk (from the initiative tracker, folded in 2026-07-12)
Two live items previously tracked ONLY in archreview/plans/00-MASTER-TRACKER.md's registry are folded in here (NF5) so this register is the single tracking place. The tracker's narrative subsections remain as the historical evidence.
| # | Item | Where noted | Rationale for deferral | Revisit trigger |
|---|---|---|---|---|
| SBR | keep-oldest to the auto-down downing strategy (Akka AutoDowning, auto-down-unreachable-after = 15s): a hard crash of EITHER node — active/oldest included — now fails over to the survivor in ~25s with no operator action. Accepted trade: a real network partition produces dual-active until an operator restarts one side. Decision record + evidence (live keep-oldest DownReachable … including myself log, Akka.NET 1.5.62 KeepOldest.OldestDecision source, rejected alternatives incl. the static-quorum-1 DownAll trap): docs/plans/2026-07-21-auto-down-availability-decision.md. |
archreview/plans/00-MASTER-TRACKER.md:194 + auto-memory sbr-keep-oldest-2node-active-crash-gap (both now historical) |
— | Closed. Residual: seed-node boot-alone constraint (unchanged, documented in Component-ClusterInfrastructure.md); dual-active recovery is operator-driven. |
| vd03 | deploy/wonder-app-vd03/ overlay edits unapplied — appsettings.Central.json needs AllowSingleNodeCluster: true + phantom-seed removal + NodeName: central-a; install.ps1 needs sc.exe failure recovery actions. The deploy/wonder-app-vd03/ artifact directory is intentionally untracked (production config out of source control), so the repo cannot ship the fix. |
archreview/plans/00-MASTER-TRACKER.md:198 (PLAN-01 T16/T20/T23) |
Needs on-host access; without NodeName that deployment's audit rows stamp NULL SourceNode — partially mitigated once PLAN-R2-08 Task 7 lands: the host now FAILS AT BOOT with a key-naming error instead of silently NULLing, so applying the overlay becomes mandatory at the next upgrade. Owner: whoever maintains the host (user). |
Next wonder-app-vd03 deployment/upgrade — the Task 7 validator makes this row unskippable then. |