ScadaBridge/archreview/plans/00-MASTER-TRACKER.md

Architecture Review Fix Plans — Master Tracker

Created: 2026-07-08 Source: archreview/00-OVERALL.md and domain reports 01–08 Plans: 8 implementation plans, 192 tasks (191 net of one cross-plan duplicate — see Dedupe Rulings)

Every plan follows the TDD bite-sized-task format and ships a co-located .tasks.json manifest. Execute a plan with the superpowers-extended-cc:executing-plans skill (pass the plan path); it reads/updates the .tasks.json. When a plan's status changes, update the table below — this file is the single source of truth for initiative progress.

Status Board

Plan	Domain	Tasks	Done	Status	Findings coverage
PLAN-01	Cluster, Host & Failover	23	23	✅ Complete (all 23 tasks; branch not yet merged)	28 findings: 20 fixed, 3 → other plans, 3 accepted, 2 roadmap-deferred
PLAN-02	Communication & S&F	24	24	✅ Complete (all 24 tasks; merged to main)	28 findings: 25 fixed, 1 → plan 01, 1 deferred (serializer swap), 1 won't-fix
PLAN-03	Site Runtime & DCL	26	0	🟨 In progress (Wave 1)	28 actionable: 25 fixed, 3 deferred with rationale
PLAN-04	Data & Audit Backbone	26	0	⬜ Not started	27 items: 22 fixed, 4 deferred, 1 won't-fix (incl. NodeB reconcile from report 08)
PLAN-05	Templates, Deployment & Transport	25	1	🟨 In progress (P0: T1)	33 line-items: 27 fixed, 6 deferred with rationale
PLAN-06	Edge Integrations	24	0	⬜ Not started	24 findings: 21 fixed, 3 deferred/won't-fix
PLAN-07	UI, Management & Security	33	3	🟨 In progress (P0: T1, T2, T3)	24 items: 20 fixed, 3 partial (deferrals documented), 1 won't-fix
PLAN-08	Conventions & Tests	11	1	🟨 In progress (P0: T1)	13 covered, 4 owned by other plans, ~20 triaged into deferred-work register

Status values: ⬜ Not started · 🟨 In progress · ✅ Complete · ⏸ Blocked

Progress: P0 (Wave 0) complete — 10 of 192 tasks done (2026-07-08): PLAN-01 T1/T3/T4, PLAN-02 T1/T2, PLAN-05 T1, PLAN-07 T1/T2/T3, PLAN-08 T1. Merged to main (3f8e18c6). One follow-up surfaced during execution — see the active-node-crash SBR gap in the deferred registry.

Wave 1 — PLAN-02 COMPLETE (2026-07-08): all 24 tasks delivered and merged to main (ce239f6f). Store-and-Forward: active-node delivery gate (Func<bool> seam), bounded/short-circuiting retry sweep, upsert-based standby applies + anti-entropy peer resync, WAL+busy_timeout, epoch-ms due-check, ordered single-reader audit-observer channel; Notify.Send enqueue-only (deferToSweep, maxRetries:0, corrupt-payload parks not discards); gRPC channel keying + DropOldest debug-stream eviction counting + 5s application heartbeat; oldest-parked-age health field. Docker cluster rebuilt on merged image (Healthy, 5 singletons, corrected active-node semantics) and 3 sites seeded. Initiative total: 52 of 192 (sum of the Done column above).

Wave 1 — PLAN-01 COMPLETE (2026-07-08): all 23 tasks delivered on worktree branch worktree-archreview-wave1-plan01 (NOT yet merged/pushed). Beyond the P0/early-Wave-1 spine (SBR downing + two-node rig T1/T3/T4; oldest-member active-node unification via ClusterActivityEvaluator T5–T8; CentralSingletonRegistrar drains T2/T9/T10; health-report acking T11–T12), Wave 1 also landed: metrics-staleness + status-transition state machine and its dashboard surface (T13/T14), deleted-site eviction from the aggregator (T18), seed-vs-metrics-port validator + AllowSingleNodeCluster + docker stop_grace_period (T15–T17), rate-limited dead-letter warnings (T19), the down-if-alone process-exit recovery loop (T20), the failover-drill.sh SIGKILL drill (T21), and the REQ-HOST-6 + Traefik/Host doc reconciliation with unused-Akka.Hosting-package removal (T22/T23). PLAN-01 now 23 / 23. Initiative total: 30 of 192. Each task TDD + pathspec-committed; full solution builds clean (0 warnings). Three deploy-artifact sub-edits (deploy/wonder-app-vd03/ appsettings + install.ps1, Tasks 16/20/23) were deferred because that production artifact is not tracked in this repo — logged in the deferred register below.

P0 — Do These First (regardless of plan order)

✅ All six delivered 2026-07-08 on branch archreview-p0 (worktree). TDD throughout; full solution builds clean; touched suites green (Host, Communication 244, Security 163, Transport 127+63, IntegrationTests cluster rig). One finding surfaced — see the active-node-crash SBR gap in the deferred registry below.

1. ✅ PLAN-01 Task 1 — enable the SBR downing provider (the system-wide Critical; one line + parsed-config test). Provider verified active via the live two-node rig.
2. ✅ PLAN-01 Tasks 3–4 — the two-node kill-test rig (TwoNodeClusterFixture); integration-proves Task 1 and later PLAN-02 work. Task 4 rewritten to the achievable keep-oldest guarantee (younger-crash → SBR downs it → oldest keeps singleton); oldest-crash gap registered as follow-up.
3. ✅ PLAN-07 Tasks 1–3 — DisableLoginGuard + scrub deploy/wonder-app-vd03 (Critical; unauthenticated prod surface). Guard + Host wiring committed; deploy/ is gitignored so its scrub is on-disk-only (not version-controlled), effective on next redeploy.
4. ✅ PLAN-02 Tasks 1–2 — ClusterClient generation-suffixed names (Critical crash-loop dropping routing to all sites).
5. ✅ PLAN-05 Task 1 — bundle import consumes BaseTemplateName (Critical silent data loss).
6. ✅ PLAN-08 Task 1 — CLI.Tests into the slnx (protects every subsequent plan's test runs). 341 CLI tests now run at the solution level.

Cross-Plan Dependency Map

Hard-ish sequencing (execute earlier):

• PLAN-01 Tasks 5–7 → PLAN-02 Tasks 3–4. PLAN-02's standby S&F delivery gate is a Func<bool> seam with marked swap points; it works standalone but its final wiring consumes PLAN-01's ClusterActivityEvaluator / SelfIsPrimary (singleton-host semantics).
• PLAN-01 Tasks 3–4 → PLAN-02 integration proof. The two-node rig is how PLAN-02's Critical fixes get integration-verified.
• PLAN-04 Task 13 (additive RequestedBy message change) before any PLAN-07 sender refactors touching the same messages.

Soft seams (either order works, coordinate):

• PLAN-05 Task 14 (IScriptArtifactChangeBus) ↔ PLAN-06 Task 3 (InboundScriptExecutor.InvalidateMethod). PLAN-06 Task 1's per-request revision check is the designed correctness fallback, so there is no hard sequencing; when both land, wire the bus to the invalidate seam.
• PLAN-03's DeployCompileValidator should adopt PLAN-05's compile-cache abstraction once it exists (later refactor, not a blocker).
• PLAN-03 Task 8's stuck-thread watchdog pattern is reusable by PLAN-06 Task 4 (inbound orphan counter).

EF Core migrations — serialize globally. Migration-adding tasks exist in PLAN-04 (Tasks 6, 8), PLAN-06 (Tasks 8, 19), and possibly PLAN-07 (secured-write expiry). All share one model snapshot: never run two migration tasks concurrently, and always build first (never --no-build on dotnet ef migrations add — known empty-scaffold gotcha).

Shared-file merge surfaces (don't run these plans' overlapping tasks concurrently):

File / area	Plans touching it
ManagementActor.cs	PLAN-07 (15 tasks, serialized within plan), PLAN-05 (management-path publish), PLAN-06 (method-update paths)
deploy/wonder-app-vd03/appsettings.Central.json	PLAN-07 (auth), PLAN-01 (Cluster/Node sections)
NotificationOutboxActor.cs	PLAN-04, PLAN-06 (SMS retry policy)
StoreAndForwardService.cs / site tracking store	PLAN-02 (primary owner), PLAN-04 Task 15 (keyset query)
Host/Program.cs + service registration	PLAN-01, PLAN-08 (options validation)
SiteHealthReport (Commons record, additive fields)	PLAN-03 Task 6, PLAN-04 health-event fields
Transport BundleImporter.cs	PLAN-05 (primary, serialized chain), PLAN-07 (import idempotency note)

Dedupe Rulings (one owner per finding)

Finding	Owner	Others must skip
CLI.Tests into slnx	PLAN-08 Task 1 (also retires CLAUDE.md gotcha + memory note)	PLAN-07 Task 33 — skip as duplicate; mark skipped-duplicate in its tasks.json
DisableLogin: true artifact	PLAN-07 Tasks 1–3	PLAN-01 references only
AuditLog reconciliation NodeB failover	PLAN-04 Task 18	PLAN-08 pointer only
SmsConfiguration.MaxRetryCount unused	PLAN-06 Task 18	PLAN-08 pointer only
Transport AreaName: null	PLAN-05 Task 7	PLAN-08 pointer only
Role-casing normalization	PLAN-07 Task 10 chain	PLAN-08 pointer only
Two-node failover rig	PLAN-01 Tasks 3–4	PLAN-08 Task 8 is a documented placeholder; PLAN-02 consumes
Script-artifact invalidation contract	PLAN-05 Task 14 (design + publisher)	PLAN-06 implements the inbound consumer

Recommended Execution Waves

• Wave 0 (P0 list above). Independent files; all six items can run in parallel. ~1 focused session.
• Wave 1 — stability spine. Rest of PLAN-01 (active-node semantics, drains); PLAN-02 Tasks 3–9 (standby gate, gRPC channel keying, sweep bounding); PLAN-03 Tasks 1–4 (reconnect leak, compile gate); PLAN-04 Tasks 1–5 (purge chains); PLAN-05 import chain Tasks 2–6; PLAN-06 Tasks 1–4 + 8; PLAN-07 Tasks 4–9 (deploy scoping, secret elision).
• Wave 2 — remainder. All other tasks per intra-plan blockedBy edges; migration tasks serialized globally; docs-update tasks land with their code tasks (repo rule: doc + code + tests together).
• Wave 3 — closeout. PLAN-08 Task 11 (deferred-work register), each plan's doc-sweep task, then a full dotnet build ZB.MOM.WW.ScadaBridge.slnx + dotnet test (which now includes CLI.Tests) + bash docker/deploy.sh + the two-node drill from PLAN-01.

Execution hygiene: work in a dedicated worktree (not the primary checkout); one plan per executor session where possible; for parallel implementers use pathspec-scoped commits and verify every commit landed on HEAD afterward (both are known failure modes in this repo's history). Re-verify a finding's line numbers before editing — reports and plans cite the code as of 2026-07-08.

Deferred / Won't-Fix Registry (initiative-level)

Consolidated in PLAN-08 Task 11 → docs/plans/2026-07-08-deferred-work-register.md (triages the 23-item inventory from report 08 plus each plan's coverage-table deferrals: serializer swap, hash-chain/Parquet, aggregated live alarm stream, central cert-trust persistence, runtime script sandbox, clustered-GUID benchmark, TLS profile, Dockerfile HEALTHCHECK, etc.). Nothing from the review is silently dropped: every report finding is either a task, an ownership pointer, or a registered deferral with rationale.

Follow-up discovered during P0 execution (2026-07-08)

• Active(oldest)-node hard crash is NOT covered by two-node keep-oldest — total central outage until restart. PLAN-01 Task 1's downing-provider-class fix is verified active (a hard-crashed node now transitions Unreachable → Down → Removed, impossible under the pre-fix NoDowning default). But while building PLAN-01 Task 4's behavioral proof, the live two-node rig showed (Akka 1.5.62) that keep-oldest downs the partition without the oldest: crashing the younger node is survivable (oldest keeps its singletons), but crashing the oldest node makes the younger survivor down itself (Down ~8s, terminated ~10s) → the whole central cluster is lost until the crashed node restarts. down-if-alone = on does not help on a hard crash (the alone-oldest is dead and cannot down itself). Because ScadaBridge treats oldest = active node (PLAN-01 Task 5 ClusterActivityEvaluator), this means a crash of the active central node does not fail over to the standby — the scenario operators care most about. Task 4's test was rewritten to prove the achievable guarantee (younger-crash → SBR downs it → oldest keeps singleton) and documents this asymmetry in its XML doc. Remedy is a topology/strategy decision (deferred, owner: user): e.g. keep-majority + a 3rd/lighthouse seed node, static-quorum, or an explicit accepted-risk note. Not fixed in P0 to avoid silently changing the production SBR strategy.

Deferred during Wave 1 PLAN-01 execution (2026-07-08)

• deploy/wonder-app-vd03/ overlay edits not applied — that production deploy artifact is not tracked in this repo. PLAN-01 Tasks 16, 20, and 23 each specified a small edit to the on-host deployment overlay (appsettings.Central.json: add AllowSingleNodeCluster: true + drop the phantom seed, add NodeName: central-a; install.ps1: add sc.exe failure service-recovery actions for the down-if-alone restart contract). Only the deployment-record doc (docs/deployment/deployment-records/2026-06-27-wonder-app-vd03.md) is version-controlled; the deploy/wonder-app-vd03/ artifact directory itself is not present in the repo (production config kept out of source control). The code/validator/doc halves of all three tasks shipped and are what make the features usable (AllowSingleNodeCluster flag + validator, the IHostApplicationLifetime process-exit watchdog, the oldest-member Traefik/Host doc reconciliation + TLS roadmap note). Owner: whoever maintains the on-host artifact — apply the three overlay edits out-of-band on wonder-app-vd03 (see the referenced task steps); without NodeName every audit row from that deployment stamps a NULL SourceNode, and without the service-recovery actions a self-downed node will not auto-restart.

Plan deviations during Wave 1 PLAN-03 execution (2026-07-09)

• Task 15 (S6) — deploy-success reporting redesigned: readiness handshake, not swallow-only. The plan's Task 15 relied on the Terminated signal beating the persistence result, so a stale persistence Success could simply be swallowed once the init-failure path had already replied Failed. In practice the store commits FIRST (the optimistic StoreDeployedConfigAsync completes before the child's async PreStart throws), so the swallow-only guard still reported Success for an actor that then died — the exact S6 bug. Fix as shipped makes the join deterministic: InstanceActor.PreStart now sends a new InstanceActorInitialized readiness message to the DeploymentManager as its final init step, and deployment Success requires both the persistence commit and that signal (InFlightDeploy.Persisted && Initialized); an actor that dies during init never signals readiness, so the Terminated fallback fails the deployment and rolls back the optimistic state. A second, related bug was fixed at the same time: the optimistic Store… and the rollback Remove… were independent Task.Runs with no ordering guarantee, so the persisted-row rollback is now deferred until the store commits (_initFailedPendingRowRemoval) and can never race ahead of it. New public surface: Messages/InstanceActorInitialized.cs (additive). Follow-up owner: PLAN-03 Task 26 — the design-doc updates must describe the two-signal deploy-success join + the readiness message (alongside Task 4's already-flagged synchronous-compile-gate deviation). No further code work outstanding; this is a doc-capture follow-up.

Pre-existing failure discovered during Wave 1 PLAN-03 execution — owner PLAN-07 (2026-07-09)

• ✅ RESOLVED 2026-07-09 — 3 CentralUI.Tests / SecuredWritesTests render tests fixed by registering a stub IBrowseService in the bUnit harness; suite now 7/7 green. Original report retained below for history. IBrowseService not registered in the page's bUnit harness (PLAN-07 / Secured Writes, NOT PLAN-03). PLAN-03's final full-solution dotnet test surfaced 3 failures in SecuredWritesTests (Operator_SeesSubmitForm, Operator_Submit_CallsServiceWithEnteredValues, Verifier_OwnSubmission_HasApproveAndRejectDisabled), all InvalidOperationException: Cannot provide a value for property 'BrowseService' on type 'NodeBrowserDialog' — no registered service of type 'IBrowseService'. Root cause: the Secured Writes page (Components/Pages/Operations/SecuredWrites.razor) renders Components/Dialogs/NodeBrowserDialog.razor, which @injects IBrowseService; the test fixture (SecuredWritesTests) registers _service/_siteRepo/_dialog/auth but not a fake IBrowseService, so rendering the child dialog throws. Confirmed pre-existing and out of PLAN-03 scope — the failing files were last touched by commits outside the PLAN-03 session (1a7e7351 page/test, 9cff87fe dialog), and PLAN-03 never touches Secured Writes / CentralUI. Merged to main (c7a0c922) with this red present because it is not a regression from this branch. Owner: PLAN-07 — register a stub IBrowseService in the SecuredWritesTests Services collection before RenderPage() (or in the shared bUnit fixture). See PLAN-07 "Post-plan known issues".