diff --git a/archreview/plans/00-MASTER-TRACKER.md b/archreview/plans/00-MASTER-TRACKER.md index 6f125065..184bc935 100644 --- a/archreview/plans/00-MASTER-TRACKER.md +++ b/archreview/plans/00-MASTER-TRACKER.md @@ -10,7 +10,7 @@ Every plan follows the TDD bite-sized-task format and ships a co-located `.tasks | Plan | Domain | Tasks | Done | Status | Findings coverage | |------|--------|------:|-----:|--------|-------------------| -| [PLAN-01](PLAN-01-cluster-host-failover.md) | Cluster, Host & Failover | 23 | 12 | ๐ŸŸจ In progress (P0: T1,T3,T4; Wave 1: T2,T5โ€“T12) | 28 findings: 20 fixed, 3 โ†’ other plans, 3 accepted, 2 roadmap-deferred | +| [PLAN-01](PLAN-01-cluster-host-failover.md) | Cluster, Host & Failover | 23 | 23 | โœ… Complete (all 23 tasks; branch not yet merged) | 28 findings: 20 fixed, 3 โ†’ other plans, 3 accepted, 2 roadmap-deferred | | [PLAN-02](PLAN-02-communication-store-and-forward.md) | Communication & S&F | 24 | 2 | ๐ŸŸจ In progress (P0: T1, T2) | 28 findings: 25 fixed, 1 โ†’ plan 01, 1 deferred (serializer swap), 1 won't-fix | | [PLAN-03](PLAN-03-site-runtime-dcl.md) | Site Runtime & DCL | 26 | 0 | โฌœ Not started | 28 actionable: 25 fixed, 3 deferred with rationale | | [PLAN-04](PLAN-04-data-audit-backbone.md) | Data & Audit Backbone | 26 | 0 | โฌœ Not started | 27 items: 22 fixed, 4 deferred, 1 won't-fix (incl. NodeB reconcile from report 08) | @@ -23,7 +23,7 @@ Status values: โฌœ Not started ยท ๐ŸŸจ In progress ยท โœ… Complete ยท โธ Blocke **Progress:** P0 (Wave 0) complete โ€” **10 of 192 tasks done** (2026-07-08): PLAN-01 T1/T3/T4, PLAN-02 T1/T2, PLAN-05 T1, PLAN-07 T1/T2/T3, PLAN-08 T1. Merged to `main` (`3f8e18c6`). One follow-up surfaced during execution โ€” see the [active-node-crash SBR gap](#follow-up-discovered-during-p0-execution-2026-07-08) in the deferred registry. -**Wave 1 in progress โ€” PLAN-01 (2026-07-08):** T2, T5โ€“T12 delivered on worktree branch `worktree-archreview-wave1-plan01` (NOT yet merged/pushed). Active-node semantics unified on oldest-member (`ClusterActivityEvaluator`; T5โ€“T8, incl. partition-safe `/health/active` + graceful-leave divergence proof), all seven central singletons routed through `CentralSingletonRegistrar` with drains (T2, T9, T10), and end-to-end health-report acking (`SiteHealthReportAck`; T11โ€“T12). PLAN-01 now **12 / 23**. Remaining PLAN-01: T13โ€“T23. Initiative total: **19 of 192**. Each task TDD + pathspec-committed; full solution builds clean. +**Wave 1 โ€” PLAN-01 COMPLETE (2026-07-08):** all 23 tasks delivered on worktree branch `worktree-archreview-wave1-plan01` (**NOT yet merged/pushed**). Beyond the P0/early-Wave-1 spine (SBR downing + two-node rig T1/T3/T4; oldest-member active-node unification via `ClusterActivityEvaluator` T5โ€“T8; `CentralSingletonRegistrar` drains T2/T9/T10; health-report acking T11โ€“T12), Wave 1 also landed: metrics-staleness + status-transition state machine and its dashboard surface (T13/T14), deleted-site eviction from the aggregator (T18), seed-vs-metrics-port validator + `AllowSingleNodeCluster` + docker `stop_grace_period` (T15โ€“T17), rate-limited dead-letter warnings (T19), the down-if-alone process-exit recovery loop (T20), the `failover-drill.sh` SIGKILL drill (T21), and the REQ-HOST-6 + Traefik/Host doc reconciliation with unused-Akka.Hosting-package removal (T22/T23). PLAN-01 now **23 / 23**. Initiative total: **30 of 192**. Each task TDD + pathspec-committed; full solution builds clean (0 warnings). Three deploy-artifact sub-edits (`deploy/wonder-app-vd03/` appsettings + install.ps1, Tasks 16/20/23) were deferred because that production artifact is not tracked in this repo โ€” logged in the deferred register below. ## P0 โ€” Do These First (regardless of plan order) @@ -90,3 +90,7 @@ Consolidated in **PLAN-08 Task 11** โ†’ `docs/plans/2026-07-08-deferred-work-reg ### Follow-up discovered during P0 execution (2026-07-08) - **Active(oldest)-node hard crash is NOT covered by two-node `keep-oldest` โ€” total central outage until restart.** PLAN-01 Task 1's `downing-provider-class` fix is verified active (a hard-crashed node now transitions `Unreachable โ†’ Down โ†’ Removed`, impossible under the pre-fix NoDowning default). **But** while building PLAN-01 Task 4's behavioral proof, the live two-node rig showed (Akka 1.5.62) that `keep-oldest` downs the partition *without* the oldest: crashing the **younger** node is survivable (oldest keeps its singletons), but crashing the **oldest** node makes the younger survivor **down itself** (`Down` ~8s, terminated ~10s) โ†’ the whole central cluster is lost until the crashed node restarts. `down-if-alone = on` does not help on a hard crash (the alone-oldest is dead and cannot down itself). Because ScadaBridge treats **oldest = active node** (PLAN-01 Task 5 `ClusterActivityEvaluator`), this means a crash of the *active* central node does not fail over to the standby โ€” the scenario operators care most about. Task 4's test was rewritten to prove the achievable guarantee (younger-crash โ†’ SBR downs it โ†’ oldest keeps singleton) and documents this asymmetry in its XML doc. **Remedy is a topology/strategy decision (deferred, owner: user):** e.g. `keep-majority` + a 3rd/lighthouse seed node, `static-quorum`, or an explicit accepted-risk note. Not fixed in P0 to avoid silently changing the production SBR strategy. + +### Deferred during Wave 1 PLAN-01 execution (2026-07-08) + +- **`deploy/wonder-app-vd03/` overlay edits not applied โ€” that production deploy artifact is not tracked in this repo.** PLAN-01 Tasks 16, 20, and 23 each specified a small edit to the on-host deployment overlay (`appsettings.Central.json`: add `AllowSingleNodeCluster: true` + drop the phantom seed, add `NodeName: central-a`; `install.ps1`: add `sc.exe failure` service-recovery actions for the down-if-alone restart contract). Only the *deployment-record* doc (`docs/deployment/deployment-records/2026-06-27-wonder-app-vd03.md`) is version-controlled; the `deploy/wonder-app-vd03/` artifact directory itself is not present in the repo (production config kept out of source control). The **code/validator/doc** halves of all three tasks shipped and are what make the features usable (`AllowSingleNodeCluster` flag + validator, the `IHostApplicationLifetime` process-exit watchdog, the oldest-member Traefik/Host doc reconciliation + TLS roadmap note). **Owner: whoever maintains the on-host artifact** โ€” apply the three overlay edits out-of-band on `wonder-app-vd03` (see the referenced task steps); without `NodeName` every audit row from that deployment stamps a NULL `SourceNode`, and without the service-recovery actions a self-downed node will not auto-restart. diff --git a/archreview/plans/PLAN-01-cluster-host-failover.md.tasks.json b/archreview/plans/PLAN-01-cluster-host-failover.md.tasks.json index 169cc6c3..9a335f25 100644 --- a/archreview/plans/PLAN-01-cluster-host-failover.md.tasks.json +++ b/archreview/plans/PLAN-01-cluster-host-failover.md.tasks.json @@ -25,7 +25,7 @@ }, { "id": 4, - "subject": "Task 4: SBR behavioral proof โ€” DELIVERED as younger-node-crash (SBR downs it, oldest keeps singleton); the planned oldest-crash 'singleton migration' is impossible under 2-node keep-oldest โ€” see master-tracker follow-up", + "subject": "Task 4: SBR behavioral proof \u2014 DELIVERED as younger-node-crash (SBR downs it, oldest keeps singleton); the planned oldest-crash 'singleton migration' is impossible under 2-node keep-oldest \u2014 see master-tracker follow-up", "status": "completed", "blockedBy": [ 3 @@ -33,7 +33,7 @@ }, { "id": 5, - "subject": "Task 5: ClusterActivityEvaluator โ€” single oldest-member \"active node\" definition", + "subject": "Task 5: ClusterActivityEvaluator \u2014 single oldest-member \"active node\" definition", "status": "completed", "blockedBy": [] }, @@ -55,7 +55,7 @@ }, { "id": 8, - "subject": "Task 8: Divergence proof โ€” restarted first node must NOT become active again", + "subject": "Task 8: Divergence proof \u2014 restarted first node must NOT become active again", "status": "completed", "blockedBy": [ 3, @@ -95,13 +95,13 @@ { "id": 13, "subject": "Task 13: Metrics-staleness signal + status-transition timestamps in the aggregator", - "status": "pending", + "status": "completed", "blockedBy": [] }, { "id": 14, "subject": "Task 14: Surface metrics-stale + offline-since on the Health dashboard", - "status": "pending", + "status": "completed", "blockedBy": [ 13 ] @@ -109,25 +109,25 @@ { "id": 15, "subject": "Task 15: Fix the metrics-port seed in appsettings.Site.json + close the validator gap", - "status": "pending", + "status": "completed", "blockedBy": [] }, { "id": 16, - "subject": "Task 16: Single-node installs โ€” explicit AllowSingleNodeCluster instead of phantom seeds", - "status": "pending", + "subject": "Task 16: Single-node installs \u2014 explicit AllowSingleNodeCluster instead of phantom seeds", + "status": "completed", "blockedBy": [] }, { "id": 17, "subject": "Task 17: stop_grace_period for all app containers (both topologies)", - "status": "pending", + "status": "completed", "blockedBy": [] }, { "id": 18, "subject": "Task 18: Evict deleted sites from the health aggregator", - "status": "pending", + "status": "completed", "blockedBy": [ 13 ] @@ -135,13 +135,13 @@ { "id": 19, "subject": "Task 19: Rate-limit dead-letter warnings", - "status": "pending", + "status": "completed", "blockedBy": [] }, { "id": 20, - "subject": "Task 20: Down-if-alone recovery โ€” exit on unexpected ActorSystem termination + service restart actions", - "status": "pending", + "subject": "Task 20: Down-if-alone recovery \u2014 exit on unexpected ActorSystem termination + service restart actions", + "status": "completed", "blockedBy": [ 10 ] @@ -149,7 +149,7 @@ { "id": 21, "subject": "Task 21: Docker failover drill script (manual/CI kill test against the real topology)", - "status": "pending", + "status": "completed", "blockedBy": [ 1, 17 @@ -158,7 +158,7 @@ { "id": 22, "subject": "Task 22: Reconcile REQ-HOST-6 (Akka.Hosting) with the hand-rolled bootstrap", - "status": "pending", + "status": "completed", "blockedBy": [ 1 ] @@ -166,7 +166,7 @@ { "id": 23, "subject": "Task 23: Docs & deploy-config cleanup batch (Traefik doc, NodeName overlays, TLS roadmap note)", - "status": "pending", + "status": "completed", "blockedBy": [ 7, 16