Joseph Doherty
cf3bd52f93
Two-node keep-oldest could NEVER survive a crash of the oldest/active node:
Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a
side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs
ITSELF — proven live on the rig ('SBR took decision ... including myself')
before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll);
keep-majority just re-keys the fatal crash to the lowest address.
SplitBrainResolverStrategy gains 'auto-down' (new default): BuildHocon emits
Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter.
The leader among the REACHABLE members downs the unreachable peer, so the
survivor takes over singletons and /health/active in ~25s regardless of which
node died. Accepted trade (explicit owner decision): a real network partition
runs dual-active until an operator restarts one side. keep-oldest remains
supported; DownIfAlone validation is now scoped to it.
Live drill on the rebuilt rig: active-crash TAKEOVER in 28s (victim still
down; all 7 singletons Younger->Oldest), standby-crash removal 27s with 0
routing blips; victims rejoin as standby in 2s. New real-cluster tests pin
both directions (SbrFailoverTests.AutoDown_*); TwoNodeClusterFixture gains a
strategy knob. All 16 appsettings flipped (src, docker, docker-env2, and the
gitignored wonder-app-vd03 overlay on disk — owner must sync to the host).
Docs: decision record docs/plans/2026-07-21-auto-down-availability-decision.md,
Component-ClusterInfrastructure downing section rewritten, drill + README
reworked (active mode now asserts takeover), deferred-work SBR row resolved.
10 KiB
10 KiB
Deferred-Work Register (established 2026-07-08, from architecture review 08)
Single tracked list of consciously-deferred work. Rules: every deferral gets a row (rationale + revisit trigger); fix-now items reference the archreview plan that owns them and are removed from this table when that plan's task lands.
Fix-now (owned by archreview plans)
All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round 2, 2026-07-12) — rows removed per the rule above; see the round-2 report §1 for the per-item evidence.
Deferred (with rationale + revisit trigger)
| # | Item | Where noted | Rationale for deferral | Revisit trigger |
|---|---|---|---|---|
| 8 | Hash-chain tamper evidence (T1); CLI verify-chain is a no-op stub | audit-log roadmap :12 | v1.x by locked decision; append-only DB roles are the control | Compliance requirement for cryptographic tamper evidence |
| 9 | Parquet audit archival (T2); endpoint returns 501 | AuditEndpoints.cs:204 | v1.x; 501 + CLI messaging are honest | AuditLog partition volume nears retention ceiling |
| 11 | Central-persisted OPC UA cert-trust audit | m7 follow-ups | Broadcast-to-both-nodes covers HA | Governance/audit requirement for trust decisions |
| 12 | Native-alarm-source-override CSV import — Central UI InstanceConfigure upload affordance only (CLI + Management API + parser shipped 2026-07-10, see Resolved) |
m7 follow-ups | CLI/API path closes the operator parity gap; the Blazor upload button is polish | First request to bulk-import native sources from the UI rather than the CLI |
| 17 | Unified notifications+site-calls outbox page | stillpending :118 | Explicit M9 decision to keep two pages | Operator confusion reports |
| 18 | Folder drag-drop | same, [PERM] | Permanently closed; menu reorder shipped | — (closed) |
| 19 | Bundle signing / cluster-to-cluster pull / differential bundles | transport-design :402 | v1 manifest hash + AES-GCM held sufficient | Non-repudiation requirement across orgs |
Resolved (verified against the code 2026-07-10)
Rows removed from the Deferred table above once confirmed shipped. Kept here for traceability.
| # | Item | Resolution |
|---|---|---|
| 10 | Aggregated live alarm stream for Alarm Summary | Shipped 2026-07-10 (docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md): a transient, in-memory per-site central live alarm cache (ISiteAlarmLiveCache/SiteAlarmLiveCacheService + per-site SiteAlarmAggregatorActor) fed by a new site-wide, alarm-only SubscribeSite gRPC stream (SiteStreamManager.SubscribeSiteAlarms), seed-then-stream with dedup + NodeA↔NodeB re-seed + periodic reconcile. Alarm Summary is now live-cache-driven (AlarmSummaryService.BuildFromLiveAlarms) with the 15s poll retained as fallback + NotReporting authority. Honors the [PERM] no-central-store rule — nothing persisted (no EF table/migration). Options on CommunicationOptions (eagerly validated) + two ScadaBridgeTelemetry signals. |
| 7 | SecuredWrite audit rows leave SourceNode NULL | Resolved (PLAN-07): ManagementActor.EmitSecuredWriteAuditAsync routes through ICentralAuditWriter, which stamps SourceNode (central-a/central-b) from INodeIdentityProvider. |
| 12 (CLI/API) | Native-alarm-source-override CSV import | Shipped 2026-07-10: shared CsvLineSplitter, NativeAlarmSourceOverrideCsvParser, bulk all-or-nothing SetInstanceNativeAlarmSourceOverridesCommand + ManagementActor handler (Deployer-gated), CLI instance native-alarm-source import --file, parser/CLI/handler tests. UI upload affordance still pending — see row 12 above. |
| 13 | WaitForAttribute quality-gated ("Good"-only) mode | Already implemented (Commons WaitForAttribute.RequireGoodQuality, enforced in InstanceActor, threaded through ScriptRuntimeContext, tested in InstanceActorWaitForAttributeTests). Stale "planned enhancement" doc line corrected 2026-07-10. |
| 14 | WaitForAttribute in Test-Run sandbox | Shipped 2026-07-10 (full fidelity): sandbox Attributes.WaitAsync/WaitForAsync (value-equality) route to the bound instance via ISandboxInstanceGateway.WaitForAttributeAsync → the existing CommunicationService.RouteToWaitForAttributeAsync cross-site route. Additive RouteToWaitForAttributeRequest.RequireGoodQuality (honored by the site handler) makes quality-gated waits route too. Predicate-form waits stay unsupported (an in-process lambda can't be routed) and throw a labelled ScriptSandboxException. Tests: sandbox accessor routing (CentralUI), site-handler quality-flag threading (SiteRuntime). |
| 15 | BrowseNext final-page signal not surfaced | Already surfaced (M7 browse work): RealOpcUaClient sets Truncated=false/ContinuationToken=null on the last page; BrowseNodeResult carries both; TreeRow.razor renders "Load more" only when a continuation token remains — no wasted BrowseNext. |
| 16 | StubOpcUaClient throws on browse | Already resolved: StubOpcUaClient supports browse + address-space search, covered by StubOpcUaClientBrowseTests/StubOpcUaClientSearchTests. |
| 20 | Deployment EXPIRED-row purge | Already resolved (PLAN-04): PendingDeploymentPurgeActor central singleton (spawned in AkkaHostedService) ticks IDeploymentManagerRepository.PurgeExpiredPendingDeploymentsAsync every CommunicationOptions.PendingDeploymentPurgeInterval (default 1h), options-validated, tested. |
| 21 | SiteAuditBacklogReporter threshold consolidation | Shipped 2026-07-10: SqliteAuditWriterOptions.BacklogPollIntervalSeconds (default 30) now drives the reporter's poll cadence; explicit ctor override still wins (tests), non-positive falls back to the 30 s default. Cadence tests added; stale "hard-code / follow-up" class-doc corrected. |
| 22 | KPI history hourly rollups | Shipped 2026-07-10 (docs/plans/2026-07-10-kpi-history-hourly-rollups-plan.md, T1–T8): new KpiRollupHourly table (migration 20260710153953) folded by a third recorder tick (kpi-rollup, RollupInterval default 1h) over a re-folded RollupLookbackHours window via an idempotent, failover-self-healing upsert; per-metric gauge-vs-rate aggregation (KpiMetricAggregationCatalog); a one-shot backfill of the retention window on start; raw-vs-rollup query routing by RollupThresholdHours (default 168h); longer rollup retention (RollupRetentionDays default 365 ≥ RetentionDays, dual daily purge); and 30 d / 90 d trend windows added to the four surfaces. Options + validator, docs (Component-KpiHistory.md), and tests shipped. |
New deferrals from review 08 (this plan)
| Item | Rationale | Revisit trigger |
|---|---|---|
| Communication → HealthMonitoring layering (ICentralHealthAggregator consumed by CentralCommunicationActor.cs:351) | Moving the interface + SiteHealthState to Commons ripples across 5 projects for a cosmetic inversion | Next breaking change to ICentralHealthAggregator |
| docs/components reference docs for ScriptAnalysis, KpiHistory, DelmiaNotifier | Reference docs are substantial (StyleGuide-conformant); README claim scoped instead (PLAN-08 Task 10) | Next doc-writing session touching those components |
| Test-coverage backfill: SiteCallAudit.Tests (31 tests/1.6k LOC), DeploymentManager.Tests | No defect identified; coverage partly lives in ManagementService/Host/Integration suites | First regression escaping either component |
| Failover-timing + broader perf envelope (S&F drain rate, per-subscriber backpressure) | Trigger fired 2026-07-08 — the "PLAN-01 rig landing" trigger fired when tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixture.cs landed (before the placeholder harness even shipped), and went unnoticed (NF2). Owner: PLAN-R2-01 — archreview/plans/PLAN-R2-01-*.md (failover-envelope measurement task) owns wiring FailoverTimingTests to the fixture rig or recording the true blocker. |
Owned by PLAN-R2-01; this register row tracks the handoff only |
Deferred — operational risk (from the initiative tracker, folded in 2026-07-12)
Two live items previously tracked ONLY in archreview/plans/00-MASTER-TRACKER.md's registry are folded in here (NF5) so this register is the single tracking place. The tracker's narrative subsections remain as the historical evidence.
| # | Item | Where noted | Rationale for deferral | Revisit trigger |
|---|---|---|---|---|
| SBR | keep-oldest to the auto-down downing strategy (Akka AutoDowning, auto-down-unreachable-after = 15s): a hard crash of EITHER node — active/oldest included — now fails over to the survivor in ~25s with no operator action. Accepted trade: a real network partition produces dual-active until an operator restarts one side. Decision record + evidence (live keep-oldest DownReachable … including myself log, Akka.NET 1.5.62 KeepOldest.OldestDecision source, rejected alternatives incl. the static-quorum-1 DownAll trap): docs/plans/2026-07-21-auto-down-availability-decision.md. |
archreview/plans/00-MASTER-TRACKER.md:194 + auto-memory sbr-keep-oldest-2node-active-crash-gap (both now historical) |
— | Closed. Residual: seed-node boot-alone constraint (unchanged, documented in Component-ClusterInfrastructure.md); dual-active recovery is operator-driven. |
| vd03 | deploy/wonder-app-vd03/ overlay edits unapplied — appsettings.Central.json needs AllowSingleNodeCluster: true + phantom-seed removal + NodeName: central-a; install.ps1 needs sc.exe failure recovery actions. The deploy/wonder-app-vd03/ artifact directory is intentionally untracked (production config out of source control), so the repo cannot ship the fix. |
archreview/plans/00-MASTER-TRACKER.md:198 (PLAN-01 T16/T20/T23) |
Needs on-host access; without NodeName that deployment's audit rows stamp NULL SourceNode — partially mitigated once PLAN-R2-08 Task 7 lands: the host now FAILS AT BOOT with a key-naming error instead of silently NULLing, so applying the overlay becomes mandatory at the next upgrade. Owner: whoever maintains the host (user). |
Next wonder-app-vd03 deployment/upgrade — the Task 7 validator makes this row unskippable then. |