ScadaBridge/archreview/plans/00-MASTER-TRACKER.md

# Architecture Review Fix Plans — Master Tracker

**Created:** 2026-07-08
**Source:** [`archreview/00-OVERALL.md`](../00-OVERALL.md) and domain reports 01–08
**Plans:** 8 implementation plans, **192 tasks** (191 net of one cross-plan duplicate — see Dedupe Rulings)

Every plan follows the TDD bite-sized-task format and ships a co-located `.tasks.json` manifest. Execute a plan with the `superpowers-extended-cc:executing-plans` skill (pass the plan path); it reads/updates the `.tasks.json`. **When a plan's status changes, update the table below** — this file is the single source of truth for initiative progress.

## Status Board

| Plan | Domain | Tasks | Done | Status | Findings coverage |
|------|--------|------:|-----:|--------|-------------------|
| [PLAN-01](PLAN-01-cluster-host-failover.md) | Cluster, Host & Failover | 23 | 23 | ✅ Complete (all 23 tasks; branch not yet merged) | 28 findings: 20 fixed, 3 → other plans, 3 accepted, 2 roadmap-deferred |
| [PLAN-02](PLAN-02-communication-store-and-forward.md) | Communication & S&F | 24 | 24 | ✅ Complete (all 24 tasks; merged to `main`) | 28 findings: 25 fixed, 1 → plan 01, 1 deferred (serializer swap), 1 won't-fix |
| [PLAN-03](PLAN-03-site-runtime-dcl.md) | Site Runtime & DCL | 26 | 0 | 🟨 In progress (Wave 1) | 28 actionable: 25 fixed, 3 deferred with rationale |
| [PLAN-04](PLAN-04-data-audit-backbone.md) | Data & Audit Backbone | 26 | 0 | ⬜ Not started | 27 items: 22 fixed, 4 deferred, 1 won't-fix (incl. NodeB reconcile from report 08) |
| [PLAN-05](PLAN-05-templates-deployment-transport.md) | Templates, Deployment & Transport | 25 | 1 | 🟨 In progress (P0: T1) | 33 line-items: 27 fixed, 6 deferred with rationale |
| [PLAN-06](PLAN-06-edge-integrations.md) | Edge Integrations | 24 | 0 | ⬜ Not started | 24 findings: 21 fixed, 3 deferred/won't-fix |
| [PLAN-07](PLAN-07-ui-management-security.md) | UI, Management & Security | 33 | 3 | 🟨 In progress (P0: T1, T2, T3) | 24 items: 20 fixed, 3 partial (deferrals documented), 1 won't-fix |
| [PLAN-08](PLAN-08-conventions-tests.md) | Conventions & Tests | 11 | 1 | 🟨 In progress (P0: T1) | 13 covered, 4 owned by other plans, ~20 triaged into deferred-work register |

Status values: ⬜ Not started · 🟨 In progress · ✅ Complete · ⏸ Blocked

**Progress:** P0 (Wave 0) complete — **10 of 192 tasks done** (2026-07-08): PLAN-01 T1/T3/T4, PLAN-02 T1/T2, PLAN-05 T1, PLAN-07 T1/T2/T3, PLAN-08 T1. Merged to `main` (`3f8e18c6`). One follow-up surfaced during execution — see the [active-node-crash SBR gap](#follow-up-discovered-during-p0-execution-2026-07-08) in the deferred registry.

**Wave 1 — PLAN-02 COMPLETE (2026-07-08):** all 24 tasks delivered and **merged to `main`** (`ce239f6f`). Store-and-Forward: active-node delivery gate (`Func<bool>` seam), bounded/short-circuiting retry sweep, upsert-based standby applies + anti-entropy peer resync, WAL+busy_timeout, epoch-ms due-check, ordered single-reader audit-observer channel; `Notify.Send` enqueue-only (`deferToSweep`, `maxRetries:0`, corrupt-payload parks not discards); gRPC channel keying + DropOldest debug-stream eviction counting + 5s application heartbeat; oldest-parked-age health field. Docker cluster rebuilt on merged image (Healthy, 5 singletons, corrected active-node semantics) and 3 sites seeded. Initiative total: **52 of 192** (sum of the Done column above).

**Wave 1 — PLAN-01 COMPLETE (2026-07-08):** all 23 tasks delivered on worktree branch `worktree-archreview-wave1-plan01` (**NOT yet merged/pushed**). Beyond the P0/early-Wave-1 spine (SBR downing + two-node rig T1/T3/T4; oldest-member active-node unification via `ClusterActivityEvaluator` T5–T8; `CentralSingletonRegistrar` drains T2/T9/T10; health-report acking T11–T12), Wave 1 also landed: metrics-staleness + status-transition state machine and its dashboard surface (T13/T14), deleted-site eviction from the aggregator (T18), seed-vs-metrics-port validator + `AllowSingleNodeCluster` + docker `stop_grace_period` (T15–T17), rate-limited dead-letter warnings (T19), the down-if-alone process-exit recovery loop (T20), the `failover-drill.sh` SIGKILL drill (T21), and the REQ-HOST-6 + Traefik/Host doc reconciliation with unused-Akka.Hosting-package removal (T22/T23). PLAN-01 now **23 / 23**. Initiative total: **30 of 192**. Each task TDD + pathspec-committed; full solution builds clean (0 warnings). Three deploy-artifact sub-edits (`deploy/wonder-app-vd03/` appsettings + install.ps1, Tasks 16/20/23) were deferred because that production artifact is not tracked in this repo — logged in the deferred register below.

## P0 — Do These First (regardless of plan order)

**✅ All six delivered 2026-07-08** on branch `archreview-p0` (worktree). TDD throughout; full solution builds clean; touched suites green (Host, Communication 244, Security 163, Transport 127+63, IntegrationTests cluster rig). One finding surfaced — see the active-node-crash SBR gap in the deferred registry below.

1. ✅ **PLAN-01 Task 1** — enable the SBR downing provider (the system-wide Critical; one line + parsed-config test). *Provider verified active via the live two-node rig.*
2. ✅ **PLAN-01 Tasks 3–4** — the two-node kill-test rig (`TwoNodeClusterFixture`); integration-proves Task 1 and later PLAN-02 work. *Task 4 rewritten to the achievable keep-oldest guarantee (younger-crash → SBR downs it → oldest keeps singleton); oldest-crash gap registered as follow-up.*
3. ✅ **PLAN-07 Tasks 1–3** — `DisableLoginGuard` + scrub `deploy/wonder-app-vd03` (Critical; unauthenticated prod surface). *Guard + Host wiring committed; `deploy/` is gitignored so its scrub is on-disk-only (not version-controlled), effective on next redeploy.*
4. ✅ **PLAN-02 Tasks 1–2** — ClusterClient generation-suffixed names (Critical crash-loop dropping routing to all sites).
5. ✅ **PLAN-05 Task 1** — bundle import consumes `BaseTemplateName` (Critical silent data loss).
6. ✅ **PLAN-08 Task 1** — CLI.Tests into the slnx (protects every subsequent plan's test runs). *341 CLI tests now run at the solution level.*

## Cross-Plan Dependency Map

**Hard-ish sequencing (execute earlier):**
- **PLAN-01 Tasks 5–7 → PLAN-02 Tasks 3–4.** PLAN-02's standby S&F delivery gate is a `Func<bool>` seam with marked swap points; it works standalone but its final wiring consumes PLAN-01's `ClusterActivityEvaluator` / `SelfIsPrimary` (singleton-host semantics).
- **PLAN-01 Tasks 3–4 → PLAN-02 integration proof.** The two-node rig is how PLAN-02's Critical fixes get integration-verified.
- **PLAN-04 Task 13 (additive `RequestedBy` message change) before any PLAN-07 sender refactors** touching the same messages.

**Soft seams (either order works, coordinate):**
- **PLAN-05 Task 14 (`IScriptArtifactChangeBus`) ↔ PLAN-06 Task 3 (`InboundScriptExecutor.InvalidateMethod`).** PLAN-06 Task 1's per-request revision check is the designed correctness fallback, so there is no hard sequencing; when both land, wire the bus to the invalidate seam.
- **PLAN-03's `DeployCompileValidator` should adopt PLAN-05's compile-cache abstraction** once it exists (later refactor, not a blocker).
- **PLAN-03 Task 8's stuck-thread watchdog pattern is reusable by PLAN-06 Task 4** (inbound orphan counter).

**EF Core migrations — serialize globally.** Migration-adding tasks exist in PLAN-04 (Tasks 6, 8), PLAN-06 (Tasks 8, 19), and possibly PLAN-07 (secured-write expiry). All share one model snapshot: **never run two migration tasks concurrently**, and always build first (never `--no-build` on `dotnet ef migrations add` — known empty-scaffold gotcha).

**Shared-file merge surfaces (don't run these plans' overlapping tasks concurrently):**
| File / area | Plans touching it |
|---|---|
| `ManagementActor.cs` | PLAN-07 (15 tasks, serialized within plan), PLAN-05 (management-path publish), PLAN-06 (method-update paths) |
| `deploy/wonder-app-vd03/appsettings.Central.json` | PLAN-07 (auth), PLAN-01 (Cluster/Node sections) |
| `NotificationOutboxActor.cs` | PLAN-04, PLAN-06 (SMS retry policy) |
| `StoreAndForwardService.cs` / site tracking store | PLAN-02 (primary owner), PLAN-04 Task 15 (keyset query) |
| `Host/Program.cs` + service registration | PLAN-01, PLAN-08 (options validation) |
| `SiteHealthReport` (Commons record, additive fields) | PLAN-03 Task 6, PLAN-04 health-event fields |
| Transport `BundleImporter.cs` | PLAN-05 (primary, serialized chain), PLAN-07 (import idempotency note) |

## Dedupe Rulings (one owner per finding)

| Finding | Owner | Others must skip |
|---|---|---|
| CLI.Tests into slnx | **PLAN-08 Task 1** (also retires CLAUDE.md gotcha + memory note) | PLAN-07 Task 33 — **skip as duplicate**; mark `skipped-duplicate` in its tasks.json |
| `DisableLogin: true` artifact | **PLAN-07 Tasks 1–3** | PLAN-01 references only |
| AuditLog reconciliation NodeB failover | **PLAN-04 Task 18** | PLAN-08 pointer only |
| `SmsConfiguration.MaxRetryCount` unused | **PLAN-06 Task 18** | PLAN-08 pointer only |
| Transport `AreaName: null` | **PLAN-05 Task 7** | PLAN-08 pointer only |
| Role-casing normalization | **PLAN-07 Task 10 chain** | PLAN-08 pointer only |
| Two-node failover rig | **PLAN-01 Tasks 3–4** | PLAN-08 Task 8 is a documented placeholder; PLAN-02 consumes |
| Script-artifact invalidation contract | **PLAN-05 Task 14** (design + publisher) | PLAN-06 implements the inbound consumer |

## Recommended Execution Waves

- **Wave 0 (P0 list above).** Independent files; all six items can run in parallel. ~1 focused session.
- **Wave 1 — stability spine.** Rest of PLAN-01 (active-node semantics, drains); PLAN-02 Tasks 3–9 (standby gate, gRPC channel keying, sweep bounding); PLAN-03 Tasks 1–4 (reconnect leak, compile gate); PLAN-04 Tasks 1–5 (purge chains); PLAN-05 import chain Tasks 2–6; PLAN-06 Tasks 1–4 + 8; PLAN-07 Tasks 4–9 (deploy scoping, secret elision).
- **Wave 2 — remainder.** All other tasks per intra-plan `blockedBy` edges; migration tasks serialized globally; docs-update tasks land with their code tasks (repo rule: doc + code + tests together).
- **Wave 3 — closeout.** PLAN-08 Task 11 (deferred-work register), each plan's doc-sweep task, then a full `dotnet build ZB.MOM.WW.ScadaBridge.slnx` + `dotnet test` (which now includes CLI.Tests) + `bash docker/deploy.sh` + the two-node drill from PLAN-01.

**Execution hygiene:** work in a dedicated worktree (not the primary checkout); one plan per executor session where possible; for parallel implementers use pathspec-scoped commits and verify every commit landed on HEAD afterward (both are known failure modes in this repo's history). Re-verify a finding's line numbers before editing — reports and plans cite the code as of 2026-07-08.

## Deferred / Won't-Fix Registry (initiative-level)

Consolidated in **PLAN-08 Task 11** → `docs/plans/2026-07-08-deferred-work-register.md` (triages the 23-item inventory from report 08 plus each plan's coverage-table deferrals: serializer swap, hash-chain/Parquet, aggregated live alarm stream, central cert-trust persistence, runtime script sandbox, clustered-GUID benchmark, TLS profile, Dockerfile HEALTHCHECK, etc.). Nothing from the review is silently dropped: every report finding is either a task, an ownership pointer, or a registered deferral with rationale.

### Follow-up discovered during P0 execution (2026-07-08)

- **Active(oldest)-node hard crash is NOT covered by two-node `keep-oldest` — total central outage until restart.** PLAN-01 Task 1's `downing-provider-class` fix is verified active (a hard-crashed node now transitions `Unreachable → Down → Removed`, impossible under the pre-fix NoDowning default). **But** while building PLAN-01 Task 4's behavioral proof, the live two-node rig showed (Akka 1.5.62) that `keep-oldest` downs the partition *without* the oldest: crashing the **younger** node is survivable (oldest keeps its singletons), but crashing the **oldest** node makes the younger survivor **down itself** (`Down` ~8s, terminated ~10s) → the whole central cluster is lost until the crashed node restarts. `down-if-alone = on` does not help on a hard crash (the alone-oldest is dead and cannot down itself). Because ScadaBridge treats **oldest = active node** (PLAN-01 Task 5 `ClusterActivityEvaluator`), this means a crash of the *active* central node does not fail over to the standby — the scenario operators care most about. Task 4's test was rewritten to prove the achievable guarantee (younger-crash → SBR downs it → oldest keeps singleton) and documents this asymmetry in its XML doc. **Remedy is a topology/strategy decision (deferred, owner: user):** e.g. `keep-majority` + a 3rd/lighthouse seed node, `static-quorum`, or an explicit accepted-risk note. Not fixed in P0 to avoid silently changing the production SBR strategy.

### Deferred during Wave 1 PLAN-01 execution (2026-07-08)

- **`deploy/wonder-app-vd03/` overlay edits not applied — that production deploy artifact is not tracked in this repo.** PLAN-01 Tasks 16, 20, and 23 each specified a small edit to the on-host deployment overlay (`appsettings.Central.json`: add `AllowSingleNodeCluster: true` + drop the phantom seed, add `NodeName: central-a`; `install.ps1`: add `sc.exe failure` service-recovery actions for the down-if-alone restart contract). Only the *deployment-record* doc (`docs/deployment/deployment-records/2026-06-27-wonder-app-vd03.md`) is version-controlled; the `deploy/wonder-app-vd03/` artifact directory itself is not present in the repo (production config kept out of source control). The **code/validator/doc** halves of all three tasks shipped and are what make the features usable (`AllowSingleNodeCluster` flag + validator, the `IHostApplicationLifetime` process-exit watchdog, the oldest-member Traefik/Host doc reconciliation + TLS roadmap note). **Owner: whoever maintains the on-host artifact** — apply the three overlay edits out-of-band on `wonder-app-vd03` (see the referenced task steps); without `NodeName` every audit row from that deployment stamps a NULL `SourceNode`, and without the service-recovery actions a self-downed node will not auto-restart.

### Plan deviations during Wave 1 PLAN-03 execution (2026-07-09)

- **Task 15 (S6) — deploy-success reporting redesigned: readiness handshake, not swallow-only.** The plan's Task 15 relied on the `Terminated` signal beating the persistence result, so a stale persistence `Success` could simply be swallowed once the init-failure path had already replied `Failed`. **In practice the store commits FIRST** (the optimistic `StoreDeployedConfigAsync` completes before the child's async `PreStart` throws), so the swallow-only guard still reported `Success` for an actor that then died — the exact S6 bug. Fix as shipped makes the join deterministic: `InstanceActor.PreStart` now sends a new `InstanceActorInitialized` readiness message to the DeploymentManager as its final init step, and deployment `Success` requires **both** the persistence commit **and** that signal (`InFlightDeploy.Persisted && Initialized`); an actor that dies during init never signals readiness, so the `Terminated` fallback fails the deployment and rolls back the optimistic state. A second, related bug was fixed at the same time: the optimistic `Store…` and the rollback `Remove…` were independent `Task.Run`s with no ordering guarantee, so the persisted-row rollback is now **deferred until the store commits** (`_initFailedPendingRowRemoval`) and can never race ahead of it. New public surface: `Messages/InstanceActorInitialized.cs` (additive). **Follow-up owner: PLAN-03 Task 26** — the design-doc updates must describe the two-signal deploy-success join + the readiness message (alongside Task 4's already-flagged synchronous-compile-gate deviation). No further code work outstanding; this is a doc-capture follow-up.