209 lines
43 KiB
Markdown
209 lines
43 KiB
Markdown
# Architecture Review Fix Plans — Master Tracker
|
||
|
||
**Created:** 2026-07-08
|
||
**Source:** [`archreview/00-OVERALL.md`](../00-OVERALL.md) and domain reports 01–08
|
||
**Plans:** 8 implementation plans, **192 tasks** (191 net of one cross-plan duplicate — see Dedupe Rulings)
|
||
|
||
Every plan follows the TDD bite-sized-task format and ships a co-located `.tasks.json` manifest. Execute a plan with the `superpowers-extended-cc:executing-plans` skill (pass the plan path); it reads/updates the `.tasks.json`. **When a plan's status changes, update the table below** — this file is the single source of truth for initiative progress.
|
||
|
||
---
|
||
|
||
# ROUND 2 (2026-07-12)
|
||
|
||
**Source:** the round-2 re-review (all 8 `archreview/0X-*.md` reports rewritten 2026-07-12 at HEAD `8c888f13`): every round-1 finding source-verified (168 fixed, 0 regressions, 0 false claims), plus **56 new findings** (1 Critical / 4 High / 15 Medium / 36 Low) concentrated in post-baseline code. Round-2 plans: **8 plans, 86 tasks.**
|
||
|
||
## Round-2 Status Board
|
||
|
||
| Plan | Domain | Tasks | Done | Status | Findings coverage |
|
||
|------|--------|------:|-----:|--------|-------------------|
|
||
| [PLAN-R2-01](PLAN-R2-01-cluster-host-failover.md) | Cluster, Host & Failover | 11 | 0 | ⬜ Not started | N1→T1–T4 (incl. live drill run + envelope measurement, covers R2-08's NF2); N2→T5–T7 (`needs-user`-adjacent: deploy overlay edits, no git add); N3→T8; N4→T9; N5→T10/T11; N6→R2-08 |
|
||
| [PLAN-R2-02](PLAN-R2-02-communication-store-and-forward.md) | Communication & S&F + live alarm stream | 15 | 0 | ⬜ Not started | **N1 Critical→T1–T4** (shared oldest-Up predicate, failing-first repro); **N2 High→T5–T7** (chunked resync protocol); N3→T8; N4→T9; N5→T6; N6→T10 (MUTEX w/ R2-07); N7→T11/T12; N8→T13/T14; N9→T15 |
|
||
| [PLAN-R2-03](PLAN-R2-03-site-runtime-dcl.md) | Site Runtime & DCL | 7 | 0 | ⬜ Not started | N1→T1; N2→T2/T3; N3→T4; N4→T5/T6 (full compile-cache adoption); N5/N6→T7 |
|
||
| [PLAN-R2-04](PLAN-R2-04-data-audit-backbone.md) | Data & Audit Backbone + KPI rollups | 13 | 0 | ⬜ Not started | **R1 High→T2–T4** (sliced backfill + watermark fast-path); R2→T1; R3→T5–T7; R4→T8; R5→T9; R6→T10/T11 (1 EF migration, build-first gotcha noted); R7→T12; final verify T13 |
|
||
| [PLAN-R2-05](PLAN-R2-05-templates-deployment-transport.md) | Templates, Deployment & Transport | 9 | 0 | ⬜ Not started | N1→T2–T4 (verdict-cache globals-surface keying); N2→T1; N3→T5 (publisher; subscriber = R2-06); N4→T7; N5→T6 (trust-gate extension, hard-error); N6→T8; docs T9 |
|
||
| [PLAN-R2-06](PLAN-R2-06-edge-integrations.md) | Edge Integrations | 6 | 0 | ⬜ Not started | N1→T1/T2 (**wires the change-bus subscriber**, closes round-1 U2 partial); N2→T3; N3→T5 (typed SITE_UNREACHABLE); N4→T4; N5→T6 |
|
||
| [PLAN-R2-07](PLAN-R2-07-ui-management-security.md) | UI, Management & Security | 12 | 0 | ⬜ Not started | N1→T1 (hub throttle); N2→T2/T3 (ForwardedHeaders); N3→T4–T6 (secured-write site scope, ManagementActor lane); N4→T7; N5→T8; N6→T10 (MUTEX w/ R2-02); N7→T9; N8→T11; N9→T12 |
|
||
| [PLAN-R2-08](PLAN-R2-08-conventions-hygiene.md) | Conventions & Hygiene | 13 | 0 | ⬜ Not started | **NF1 High P0→T1–T3** (`needs-user`: key rotation + secret-file deletion — ldap/sql login files hold LIVE secrets too); NF3→T4; NF4→T5–T8 (T8 = possible live `IOperationTrackingStore` DI gap, verify-then-fix); NF5→T10/T11; NF6→T12; NF7→T13; NF8→T9 (Host duplicates are the LIVE bindings — canonicalize SCE sections first) |
|
||
|
||
**Round-2 progress: 0 of 86 tasks.**
|
||
|
||
## Round-2 P0 (do first, any order)
|
||
|
||
1. **R2-08 T1/T2** — rotate the exposed wonder-app-vd03 API key; delete `test.txt` + the three root credential files (ALL contain live secrets incl. a production sysadmin password). `needs-user`.
|
||
2. **R2-02 T1–T4** — the Critical: unify the active-node predicate; failing-first buffer-wipe repro then fix.
|
||
3. **R2-02 T5–T7** — the resync frame-size High (chunked protocol).
|
||
4. **R2-04 T2–T4** — bound the rollup backfill before the next central failover at data volume.
|
||
|
||
## Round-2 Cross-Plan Mutexes & Dedupe Rulings
|
||
|
||
| Surface / finding | Ruling |
|
||
|---|---|
|
||
| `SiteAlarmLiveCacheService.cs` | R2-02 T10 (coalescing) and R2-07 T10 (aggregator deathwatch/sticky IsLive) must NOT run concurrently — serialize, either order. R2-02 T12/T15 also touch adjacent files; keep the whole live-alarm slice single-lane across plans. |
|
||
| `ManagementActor.cs` | Single-writer lane (round-1 rule stands): R2-07 T4/T6 and R2-02 T13 (`HandleDeleteSite`) serialize against each other. |
|
||
| `IScriptArtifactChangeBus` | Publisher Add-fix = R2-05 T5; subscriber wiring + Host comment = R2-06 T1/T2. Independent, no cross-plan blockedBy. |
|
||
| Failover envelope measurement (report-08 NF2) | Owned by R2-01 T4; R2-08 T10 only repoints the register row. |
|
||
| Deferral-register consolidation (reports 01 N6 / 08 NF5) | Owned by R2-08 T10/T11. |
|
||
| Options-validation tail | Split by owner: SiteRuntime knobs = R2-03 T4; Transport knob = R2-05 T8; S&F sweep knobs = R2-02 T9; AuditLog/Host = R2-08 T5–T8. |
|
||
| SBR keep-oldest active-crash topology gap | Still a registered deferred USER decision — no round-2 plan fixes it; R2-01 documents/measures it, R2-08 T10 moves it into the canonical register. |
|
||
|
||
## Round-2 Recommended Waves
|
||
|
||
- **Wave 0 (P0):** R2-08 T1–T3 (`needs-user`) ∥ R2-02 T1–T7 ∥ R2-04 T1–T4.
|
||
- **Wave 1:** rest of R2-02 ∥ R2-03 ∥ R2-05 ∥ R2-06 (all file-disjoint domains); R2-01 T1–T4 (drill + measurement, needs live docker cluster).
|
||
- **Wave 2:** R2-07 (after R2-02's live-alarm slice lands, honoring the mutex) ∥ rest of R2-04 ∥ rest of R2-01 ∥ rest of R2-08.
|
||
- Full-suite sweep + docker redeploy at the end, per round-1 practice (watch for stale enum-count lock-in tests — bit PLAN-04 and PLAN-07).
|
||
|
||
---
|
||
|
||
# ROUND 1 (2026-07-08) — COMPLETE
|
||
|
||
## Status Board
|
||
|
||
| Plan | Domain | Tasks | Done | Status | Findings coverage |
|
||
|------|--------|------:|-----:|--------|-------------------|
|
||
| [PLAN-01](PLAN-01-cluster-host-failover.md) | Cluster, Host & Failover | 23 | 23 | ✅ Complete (all 23 tasks; merged to `main` `ec47cb56`) | 28 findings: 20 fixed, 3 → other plans, 3 accepted, 2 roadmap-deferred |
|
||
| [PLAN-02](PLAN-02-communication-store-and-forward.md) | Communication & S&F | 24 | 24 | ✅ Complete (all 24 tasks; merged to `main`) | 28 findings: 25 fixed, 1 → plan 01, 1 deferred (serializer swap), 1 won't-fix |
|
||
| [PLAN-03](PLAN-03-site-runtime-dcl.md) | Site Runtime & DCL | 26 | 26 | ✅ Complete (all 26 tasks; merged to `main` `c7a0c922`) | 28 actionable: 25 fixed, 3 deferred with rationale |
|
||
| [PLAN-04](PLAN-04-data-audit-backbone.md) | Data & Audit Backbone | 26 | 26 | ✅ Complete (all 26 tasks; merged to `main` `dbd3ed2f` via fast-forward; branch removed) | 27 items: 22 fixed, 4 deferred, 1 won't-fix (incl. NodeB reconcile from report 08) |
|
||
| [PLAN-05](PLAN-05-templates-deployment-transport.md) | Templates, Deployment & Transport | 25 | 25 | ✅ Complete (all 25 tasks; merged to `main` `ec23db35` via fast-forward; worktree removed, branch deleted) | 33 line-items: 27 fixed, 6 deferred with rationale |
|
||
| [PLAN-06](PLAN-06-edge-integrations.md) | Edge Integrations | 24 | 24 | ✅ Complete (all 24 tasks; merged to `main` `3e964ac2` via fast-forward; worktree removed, branch deleted) | 24 findings: 21 fixed, 3 deferred/won't-fix |
|
||
| [PLAN-07](PLAN-07-ui-management-security.md) | UI, Management & Security | 33 | 32 | ✅ Complete (all 32 active tasks; T33 `skipped-duplicate` — CLI.Tests already in slnx via PLAN-08 T1; merged to `main` via fast-forward; worktree removed, branch deleted) | 24 items: 20 fixed, 3 partial (deferrals documented), 1 won't-fix |
|
||
| [PLAN-08](PLAN-08-conventions-tests.md) | Conventions & Tests | 11 | 11 | ✅ Complete (all 11 tasks; T1 in P0, T2–T11 in Wave 6; merged to `main` via fast-forward; worktrees removed, branches deleted) | 13 covered, 4 owned by other plans, ~20 triaged into deferred-work register |
|
||
|
||
Status values: ⬜ Not started · 🟨 In progress · ✅ Complete · ⏸ Blocked
|
||
|
||
**Progress:** P0 (Wave 0) complete — **10 of 192 tasks done** (2026-07-08): PLAN-01 T1/T3/T4, PLAN-02 T1/T2, PLAN-05 T1, PLAN-07 T1/T2/T3, PLAN-08 T1. Merged to `main` (`3f8e18c6`). One follow-up surfaced during execution — see the [active-node-crash SBR gap](#follow-up-discovered-during-p0-execution-2026-07-08) in the deferred registry.
|
||
|
||
**Wave 1 — PLAN-02 COMPLETE (2026-07-08):** all 24 tasks delivered and **merged to `main`** (`ce239f6f`). Store-and-Forward: active-node delivery gate (`Func<bool>` seam), bounded/short-circuiting retry sweep, upsert-based standby applies + anti-entropy peer resync, WAL+busy_timeout, epoch-ms due-check, ordered single-reader audit-observer channel; `Notify.Send` enqueue-only (`deferToSweep`, `maxRetries:0`, corrupt-payload parks not discards); gRPC channel keying + DropOldest debug-stream eviction counting + 5s application heartbeat; oldest-parked-age health field. Docker cluster rebuilt on merged image (Healthy, 5 singletons, corrected active-node semantics) and 3 sites seeded. Initiative total: **52 of 192** (sum of the Done column above).
|
||
|
||
**Wave 1 — PLAN-01 COMPLETE (2026-07-08):** all 23 tasks delivered on worktree branch `worktree-archreview-wave1-plan01` (**merged to `main` `ec47cb56` 2026-07-09; branch since removed**). Beyond the P0/early-Wave-1 spine (SBR downing + two-node rig T1/T3/T4; oldest-member active-node unification via `ClusterActivityEvaluator` T5–T8; `CentralSingletonRegistrar` drains T2/T9/T10; health-report acking T11–T12), Wave 1 also landed: metrics-staleness + status-transition state machine and its dashboard surface (T13/T14), deleted-site eviction from the aggregator (T18), seed-vs-metrics-port validator + `AllowSingleNodeCluster` + docker `stop_grace_period` (T15–T17), rate-limited dead-letter warnings (T19), the down-if-alone process-exit recovery loop (T20), the `failover-drill.sh` SIGKILL drill (T21), and the REQ-HOST-6 + Traefik/Host doc reconciliation with unused-Akka.Hosting-package removal (T22/T23). PLAN-01 now **23 / 23**. Initiative total: **30 of 192**. Each task TDD + pathspec-committed; full solution builds clean (0 warnings). Three deploy-artifact sub-edits (`deploy/wonder-app-vd03/` appsettings + install.ps1, Tasks 16/20/23) were deferred because that production artifact is not tracked in this repo — logged in the deferred register below.
|
||
|
||
**Wave 1 — PLAN-03 COMPLETE (2026-07-09):** all 26 tasks delivered and **merged to `main`** (`c7a0c922`). Site Runtime & DCL hardening across nine TDD batches: reconnect/backoff leak fixes and the `DeployCompileValidator` compile gate, DCL adapter lifecycle + alarm-filter overwrite warning (S10) + observed adapter dispose on stop (S9), site-node cert reconcile (CertStoreActor trusted-store export → `DeploymentManagerActor` MemberUp-driven additive push), replicated config-fetch retry (`ConfigFetchRetryCount`), invariant-culture conditional-trigger fallback (C6), bounded/DropOldest event-channel comment fix (C2), and the SiteRuntime/DCL requirements-doc sync. Three low-severity items deferred with rationale (see PLAN-03 coverage table). Full CentralUI/SiteRuntime/DCL suites green. Initiative total: **78 of 192** (sum of the Done column above). Two out-of-band fixes also landed post-merge: PLAN-07's SecuredWrites `IBrowseService` bUnit-harness gap (resolved 2026-07-09, `9049690b`) — a pre-existing failure surfaced during PLAN-03's full-suite run, not a plan task.
|
||
|
||
**Wave 2 — PLAN-04 COMPLETE (2026-07-09):** all 26 tasks delivered on worktree branch `worktree-archreview-wave2-plan04` (Batches 1–3 fast-forwarded to `main`+`origin`; Batches 4–8 + T24 + the T26 fix still on the branch, pending `finishing-a-development-branch`). Site SQLite retention chain + partition-switch hardening + purger role-grants/index migrations (T1–T9), backlog KPI + SiteCalls freshness upsert + EF `EnableRetryOnFailure` (T10–T12), operator-identity audit on Retry/Discard relays + reconciliation composite-keyset cursor/abandoned-row audit/NodeB failover (T13–T18), time-sliced KpiSample purge + single-query NotificationOutbox KPI + SiteCalls `OPTION (RECOMPILE)` (T19–T21), short-first-tick purge timers + data-layer cleanups + bounded parallel notification delivery (T22, T23, T25), the design-doc reconciliation pass (T24). **T26 (final `dotnet build`/`dotnet test` sweep):** build 0 warn/0 err; all 30 unit + integration test assemblies green after fixing one stale enum-count lock-in test (`22d52390` — Task 17 added `AuditKind.ReconciliationAbandoned` but left `AuditEnumTests` asserting 14 members; the per-task filtered runs never hit that Commons guard, so the full sweep caught it). The Playwright E2E residue (8–9 fails) is environmental — a 17h-old docker cluster whose site containers fail `instance deploy` with `SQLite Error 14: unable to open database file`; the failure set is non-deterministic and PLAN-04 changed zero files in that project or the deployment-DB path. Initiative total: **104 of 192** (sum of the Done column).
|
||
|
||
**Wave 3 — PLAN-05 COMPLETE (2026-07-10):** all 25 tasks delivered on worktree branch `worktree-archreview-wave3-plan05` and **merged to `main` `ec23db35` via fast-forward** (main was still at PLAN-04's `dbd3ed2f`, zero divergence; worktree removed, branch deleted). Templates/Deployment/Transport hardening across seven report-and-wait waves: import correctness spine (BundleImporter consumes `BaseTemplateName`/composition/inheritance edges + import-time acyclicity, `MinTimeBetweenRuns`/`ExecutionTimeoutSeconds`/`LockedInDerived`/native-alarm-sources/real `AreaName` carried through — T1–T7), a reflection round-trip equivalence guard that surfaced and fixed 5 silent-data-loss holes (ES retry+methods-on-Add, notification recipients, folder parent edges — T8), flattener/collision-detector repeated-composition fixes (T9/T10), revision-hash + DiffService native-source coverage (T11/T12), native-source lock enforcement (T13), the `IScriptArtifactChangeBus` invalidation contract + Host in-process bus + Transport post-commit publisher (T14, plan-06 handoff), compile-verdict cache + read-only-path compile skip + post-success audit isolation + central NotDeployed-delete (T15–T18), and the closing security/quality chain: **import blocker severity split** (template-script findings advisory, ApiMethod hard — T19), **the script trust gate on bundle import** (5th `ScriptTrustValidator` call site, over script bodies *and* Expression-trigger bodies — T20, given a code-reviewer security pass that caught + fixed 2 Important gaps: expression-trigger coverage and per-origin local-declaration scoping), ScriptTrustPolicy deny-list hardening (T21), LineDiffer input cap (T22), ArtifactDiff placeholder-identity resolution (T23), the low-severity batch (session cap / full error lists / leaf-fallback warning — T24), and the design-doc + CLAUDE.md sweep (T25). **Terminal verification:** full slnx build 0 warn/0 err; the entire `dotnet test` sweep green (~5,900 unit + integration tests across every suite) plus a direct `CLI.Tests` run (341). Only residue: the same environmental 8 `CentralUI.PlaywrightTests` Deployment E2E failures (`SQLite Error 14` on `instance deploy` — needs a live docker cluster; PLAN-05 touched zero files in that project's deployment-DB path). Initiative total: **128 of 192** (sum of the Done column).
|
||
|
||
**Wave 4 — PLAN-06 COMPLETE (2026-07-10):** all 24 tasks delivered and **merged to `main` `3e964ac2` via fast-forward** (main was still at PLAN-05's `9b0cf44c`, zero divergence; worktree removed, branch deleted). Edge Integrations hardening across six report-and-wait waves of file-disjoint parallel subagents: the **InboundAPI** spine — revision-checked compiled-handler cache (heals failover / direct-SQL / known-bad staleness, DB row now authoritative — T1/T2), `InvalidateMethod` seam consuming PLAN-05's `IScriptArtifactChangeBus` contract (T3), deferred DI-scope disposal + orphan-execution telemetry (T4), machine-readable error `code`s incl. `SITE_UNREACHABLE` + 415-on-non-JSON (T5), `IOptionsMonitor` hot-reload (T6), parallel startup compile (T7); the **ESG** per-system `TimeoutSeconds` full vertical slice (entity→EF migration→artifact→site SQLite→client honor→mgmt→CLI→UI→doc — T8–T12) plus `{param}` path-template substitution (T13), bounded outbound response buffering (T14), structured JSON `AuthConfiguration` with legacy-colon fallback (T15), cancellation-aware `IsTransient` (T16); the **Notification** batch — SMS first-transient short-circuit (T17), wired the dead `SmsConfiguration.MaxRetries/RetryDelay` per-type retry policy (T18), configurable OAuth2 authority/scope (entity→EF migration→token service→mgmt/CLI/UI surfaces — T19/T20), deterministic lowest-Id SMTP selection + atomic OAuth2 token cache (T21), Twilio `AccountSid` format validation (T22); and the docs (DelmiaNotifier at-least-once semantics — T23; dual-key API-key rotation procedure — T24). Two EF migrations added (`AddExternalSystemTimeoutSeconds` T8, `AddSmtpOAuth2AuthorityScope` T19, serialized on the shared snapshot). **Terminal verification:** full slnx build 0 warn/0 err; every unit + integration suite green (InboundAPI 253, ESG 132, NotificationOutbox 132, NotificationService 57, ManagementService 245, CLI 346, CentralUI 893, SiteRuntime 494, Commons 597, ConfigDB 338, Host 260, + all others). Only residue: the same environmental 8 `CentralUI.PlaywrightTests` Deployment E2E failures (`SQLite Error 14` on `instance deploy` — needs a live docker cluster; PLAN-06 touched zero files in that project's deployment-DB path). Execution note: `isolation:"worktree"` subagents proved non-isolating here (shared index → commit races); the working model was pre-created dedicated `git worktree`s per task + cherry-pick onto an integration branch + per-wave combined build/impacted-suite barrier. Initiative total: **152 of 192** (sum of the Done column).
|
||
|
||
**Wave 5 — PLAN-07 COMPLETE (2026-07-10):** all 32 active tasks (T4–T32; T1–T3 landed in P0; **T33 `skipped-duplicate`** — CLI.Tests already in the slnx via PLAN-08 T1) delivered on worktree branch `plan07` and **merged to `main` via fast-forward** (main was still at PLAN-06's `4fb754f9`, zero divergence; all worktrees removed, branch deleted). UI/Management/Security hardening across six report-and-wait waves of file-disjoint parallel subagents against the dominant **ManagementActor single-writer lane** (15 tasks serialized in one integration worktree; the file-disjoint remainder ran in dedicated per-task worktrees, cherry-picked in with a per-wave combined build + impacted-suite barrier): **C1** auth-disabled artifact killed + `DisableLoginGuard` fail-fast (T1–T3, P0); **C2** `DeployToSiteAsync` single-site deploy + fleet-wide site-scope guard (T4/T5); **C3** connection/external-system/DB-connection secret elision via `ConfigSecretScrubber` + sentinel-preserving update (T6–T9); **UA1/C6/C5** any-of role gates + `ListSecuredWrites` gating + area-gate reconciliation (Designer|Deployer) + role-casing canonicalization (T10–T12); **UA2** the frozen `GetRequiredRoles` authorization matrix over all **138** registered commands + dispatch-coverage guard (T13/T24); **S2/P3** real server-side secured-write TTL + expiry-at-approve/reject + opportunistic list sweep + list paging + UI history pager & submission-age dialog (T14–T18); **S1** 5-min transport/deploy Ask timeout + shipped config (T19/T20); **C4** additive `SiteIdentifier` on browse/verify/cert commands + actor dispatch (Designer browse/verify, Admin cert-trust) + dispatch-coverage + CLI parity (T21–T25); **P1/UA5** `LoginThrottle` per-username+IP LDAP-bind lockout wired into every bind surface (T26/T27); **P2** additive Skip/Take paging on template/instance lists (T28); **S3** `PollGate` reentrancy guard on Health/AlarmSummary timers (T29); **S4** transport single-flight + export base64-copy elimination (T30); **S5/C7/C8/P4/UA6** debug-stream fault logging + event-log-filter doc fix + AlarmSummary memoization/a11y + the stale CLAUDE.md `SourceNode` note corrected (C7/C8 both verified true — T31/T32). Two message-contract additions (`AuditKind.SecuredWriteExpire`, secured-write list paging params) landed additively; no EF migrations needed. **Terminal verification:** full slnx build 0 warn/0 err; 28 of 30 test assemblies green (ManagementService 448 incl. the frozen matrix + dispatch-coverage, CentralUI 898, CLI 359, Security 175, Commons 609, Communication 263, Transport 138, ConfigDB 341, SiteRuntime 494, Host 260, + all others), after fixing one stale enum-count lock-in test (`AuditEnumTests` 15→16 for T15's additive `SecuredWriteExpire` — the per-task filtered runs never hit that Commons guard, so the full sweep caught it, exactly as in PLAN-04). Only residue: the same environmental 8 `CentralUI.PlaywrightTests` Deployment E2E failures (`SQLite Error 14` on `instance deploy` — needs a live docker cluster; PLAN-07 touched zero files in that project's deployment-DB path). Initiative total: **181 of 192** (sum of the Done column; PLAN-07's max is 32/33 with T33 a documented skip).
|
||
|
||
**Wave 6 — PLAN-08 COMPLETE (2026-07-10): INITIATIVE COMPLETE.** All 10 remaining tasks (T2–T11; T1 landed in P0) delivered on integration branch `plan08` and **merged to `main` via fast-forward** (main was still at PLAN-07's `187fbd7c`, zero divergence; all worktrees removed, branches deleted). Conventions/tests/underdeveloped-areas hardening across **two** report-and-wait waves of file-disjoint parallel subagents — the initiative's single highest-parallelism slice (8 concurrent implementers in Wave 1): **§1.5 eager options validation** — new `OptionsValidatorBase<T>` validators wired via `.ValidateOnStart()` + `TryAddEnumerable` across **12 components** in four tasks: Communication + DataConnectionLayer (T2, + 3 extra TimeSpan fields beyond the plan; string-only global-options classes correctly skipped), the site pipeline SiteRuntime/StoreAndForward/SiteEventLog bound in Host (T3), edge/management InboundAPI/ESG/Notification/ManagementService (T4, correctly leaving `SecuredWritePendingTtl`≤0 and empty `ApiKeyPepper` valid as legitimate disable/dev defaults), central Transport/SiteCallAudit/DeploymentManager (T5, incl. the security-sensitive `Pbkdf2Iterations >= 100_000` floor; validating base not `Resolved*` properties); **§1.3/#23** excised the vestigial site-side notification-list surface — deleted `SiteNotificationRepository`, its DI registration, and the dead `StoreNotificationListAsync`/`StoreSmtpConfigurationAsync` write paths, keeping `PurgeCentralOnlyNotificationConfigAsync` as the security cleanup (T6, with a `Site_NotificationRepository_IsNotRegistered` design-invariant lock replacing the old registration test); **§2.3** the first real perf-envelope test — site-wide stream throughput (100k events, ≥10k/s + ≥90% delivery assertions — T7) plus a skipped failover-timing harness documenting the 25s-envelope protocol pending PLAN-01's two-node rig (T8); **§1.8** contract-lock tests for the 7 highest-traffic ClusterClient records enforcing additive-only evolution under version skew (T9, which surfaced a real wire-behavior finding: Newtonsoft ignores C# ctor optional-param defaults, so a missing trailing field deserializes to `default(T)` not the C# default — the test now locks that reality); **§1.1/§1.3/§1.4/§4** the docs-drift batch — Commons `Observability/` folder, the site-repo `NotSupportedException` exception, the `..\`→`../` csproj-separator fix, the scoped "24 of 27 reference docs" README claim, and the DelmiaNotifier convention-exception bullet in CLAUDE.md (T10); and **§3** the deferred-work register triaging the 23-item inventory into fix-now (plan-owned) vs deferred-with-rationale+revisit-trigger (T11). No EF migrations, no wire-contract changes, no new runtime components — all convention-hardening around existing seams. **Terminal verification:** full slnx build 0 warn/0 err; the entire `dotnet test` sweep green across all 30 test assemblies — now including **CLI.Tests (359) at the solution level** for the first time (T1's slnx addition) — except two documented non-PLAN-08 residues: (1) a **pre-existing** `Communication.Tests` gRPC observable-gauge timing flake (`SiteStreamGrpcServerTests.SiteConnectionUpGauge_*`, proven to flake 3/6 on base `main` `187fbd7c`; PLAN-08 touched zero `Grpc/` files), and (2) the same environmental 8–9 `CentralUI.PlaywrightTests` Deployment E2E failures (`SQLite Error 14` on `instance deploy` — needs a live docker cluster). Initiative total: **191 of 192** (sum of the Done column) — the single remaining gap is PLAN-07 T33, a documented `skipped-duplicate` (CLI.Tests slnx addition, delivered instead by PLAN-08 T1). **All 8 archreview plans complete.**
|
||
|
||
## Parallelization Map — Incomplete Tasks (2026-07-09)
|
||
|
||
Covers the plans with open tasks: **none — all 8 plans complete as of 2026-07-10 (Wave 6, PLAN-08).** PLAN-01/02/03/04/05/06/07/08 are complete; the section below is retained for its cross-plan mutex analysis and the record of how PLAN-07's ManagementActor lane and PLAN-08's highest-parallelism validator slice were run. **"Ready"** = every `blockedBy` entry is already `completed` in the plan's `.tasks.json`, so the task can start now. Ready sets are recomputed below from the live manifests; re-derive after each merge.
|
||
|
||
### Cross-plan serialization mutexes (a task holding one must be the ONLY task touching that surface, anywhere)
|
||
|
||
| Shared surface | Tasks that touch it (P0n = PLAN-0n) | Rule |
|
||
|---|---|---|
|
||
| `ManagementService/ManagementActor.cs` | P05 T13; P06 T2, T3, T11, T20, T22; P07 T5, T7, T8, T9, T10, T11, T12, T15, T16, T17, T22, T23, T28, T30, T31 | **Global single-writer.** At most one ManagementActor task in flight across ALL plans. PLAN-07 already serializes its 15 internally — extend the same mutex to P05 T13 and the five P06 tasks. |
|
||
| EF migration model-snapshot (`ConfigurationDatabase/Migrations/*` + `*ModelSnapshot`) | P06 T8 (ready), T19 | **Never two migration tasks concurrently.** Build first, never `--no-build` (empty-scaffold gotcha). Only P06 T8 is ready now. |
|
||
| `Host/Program.cs` | P06 T7; P08 T4 | Single-writer — don't run P06 T7 and P08 T4 at the same time. |
|
||
| `DeploymentManager/ArtifactDeploymentService.cs` | P06 T10; P07 T4 | Single-writer (P06 T10 not yet ready; P07 T4 is). |
|
||
| `SiteRuntime/Persistence/SiteStorageService.cs` | P06 T10; P08 T6 | Single-writer (neither ready yet). |
|
||
| `NotificationOutbox/NotificationOutboxActor.cs` | P06 T18, T21 | Intra-PLAN-06 (PLAN-04's delivery work is done and no longer touches it). |
|
||
| `CentralUI/.../Monitoring/AlarmSummary.razor` | P07 T29, T32 | Intra-PLAN-07 — serialize. |
|
||
| Docs (low-risk, coordinate — pathspec-commit + last-writer reconcile): `CLAUDE.md` (P05 T25, P06 T18, P07 T15/T31/T33, P08 T10), `Component-ManagementService.md`, `Component-InboundAPI.md`, `Component-ExternalSystemGateway.md`, `Component-NotificationService.md` | multiple | Not a hard blocker; do not co-edit the same file in the same wall-clock window. |
|
||
|
||
**Dedupe reminder:** **P07 T33 is a skip** — CLI.Tests is already in the slnx via P08 T1 (done). Mark it `skipped-duplicate` in `PLAN-07-…tasks.json`; do not execute it.
|
||
|
||
### Ready set + concurrent lanes per plan
|
||
|
||
- **PLAN-04** — ✅ COMPLETE (2026-07-09); all 26 tasks done, merged to `main` `dbd3ed2f`. No longer a parallelization lane.
|
||
- **PLAN-05** — ✅ COMPLETE (2026-07-10); all 25 tasks done, merged to `main` `ec23db35` (fast-forward). No longer a parallelization lane. (Historical: the BundleImporter serial chain T1→2→…→6→19→20→23 ran in-worktree; T14's `IScriptArtifactChangeBus` contract is shipped for PLAN-06 to consume.)
|
||
- **PLAN-06** — ✅ COMPLETE (2026-07-10); all 24 tasks done, merged to `main` `3e964ac2` (fast-forward). No longer a parallelization lane. (Historical: ran as 6 waves — W1{1,6,7,8,17,23} W2{2,9,10,12,18,19} W3{3,13,21} W4{4,11,14} W5{5,15,20} W6{16,22,24}; the ExternalSystemClient chain T9→13→14→15→16 and inbound-executor chain T1→3→4→5 were the two serial spines.)
|
||
- **PLAN-07** — ✅ COMPLETE (2026-07-10); all 32 active tasks done, merged to `main` (fast-forward); T33 `skipped-duplicate`. No longer a parallelization lane. (Historical: ran as 6 waves — W1{4,6,10,14,19,21,26,29} W2{7,8,9,27,32} W3{11,20,22,23} W4{13,15,16,17,24} W5{5,12,18,25,28} W6{30,31}; Lane M = the ManagementActor single-writer chain T10→7→8→9→11→22→23→15→16→17→5→12→28→30→31 driven serially in the integration worktree, with the file-disjoint remainder in dedicated per-task worktrees cherry-picked in per wave.)
|
||
- **PLAN-08** — ready: **{2, 3, 4, 5, 7, 8, 9, 11}** — each is a new per-component validator file or an isolated test/doc, so **mutually parallel**, with one caveat: **T4 shares `Host/Program.cs` with P06 T7** (sequence them). This is the initiative's single highest-parallelism slice (independent validator files). T6, T10 stay blocked.
|
||
|
||
### Recommended concurrent execution
|
||
|
||
Run up to **five plan-lane executor sessions concurrently** (one each for PLAN-04/05/06/07/08 — PLAN-01/02/03 are done), with these coordination rules layered on top:
|
||
|
||
1. **One ManagementActor task at a time, initiative-wide** — route P05 T13 and every ready P07 ManagementActor task ({8,9,10,12,28,30,31}) through a single serialized "Lane M"; start with **P07 T10**.
|
||
2. **One migration at a time** — only **P06 T8** is a ready migration; hold the snapshot until it lands.
|
||
3. **`Host/Program.cs` mutex** — **P06 T7 ⟂ P08 T4**.
|
||
4. Inside each session, dispatch that plan's ready, file-disjoint tasks as concurrent implementers per the lanes above; use **pathspec-scoped commits and verify each landed on HEAD** (known repo failure mode — see Execution hygiene).
|
||
5. Highest-value immediate fan-out: **PLAN-08 T2/T3/T5/T7/T8/T9/T11** (independent validator + test + doc files) and **PLAN-07's non-ManagementActor octet** can run as many concurrent implementers today; the throughput ceiling is Lane M (ManagementActor) and the single migration lane.
|
||
|
||
## P0 — Do These First (regardless of plan order)
|
||
|
||
**✅ All six delivered 2026-07-08** on branch `archreview-p0` (worktree). TDD throughout; full solution builds clean; touched suites green (Host, Communication 244, Security 163, Transport 127+63, IntegrationTests cluster rig). One finding surfaced — see the active-node-crash SBR gap in the deferred registry below.
|
||
|
||
1. ✅ **PLAN-01 Task 1** — enable the SBR downing provider (the system-wide Critical; one line + parsed-config test). *Provider verified active via the live two-node rig.*
|
||
2. ✅ **PLAN-01 Tasks 3–4** — the two-node kill-test rig (`TwoNodeClusterFixture`); integration-proves Task 1 and later PLAN-02 work. *Task 4 rewritten to the achievable keep-oldest guarantee (younger-crash → SBR downs it → oldest keeps singleton); oldest-crash gap registered as follow-up.*
|
||
3. ✅ **PLAN-07 Tasks 1–3** — `DisableLoginGuard` + scrub `deploy/wonder-app-vd03` (Critical; unauthenticated prod surface). *Guard + Host wiring committed; `deploy/` is gitignored so its scrub is on-disk-only (not version-controlled), effective on next redeploy.*
|
||
4. ✅ **PLAN-02 Tasks 1–2** — ClusterClient generation-suffixed names (Critical crash-loop dropping routing to all sites).
|
||
5. ✅ **PLAN-05 Task 1** — bundle import consumes `BaseTemplateName` (Critical silent data loss).
|
||
6. ✅ **PLAN-08 Task 1** — CLI.Tests into the slnx (protects every subsequent plan's test runs). *341 CLI tests now run at the solution level.*
|
||
|
||
## Cross-Plan Dependency Map
|
||
|
||
**Hard-ish sequencing (execute earlier):**
|
||
- **PLAN-01 Tasks 5–7 → PLAN-02 Tasks 3–4.** PLAN-02's standby S&F delivery gate is a `Func<bool>` seam with marked swap points; it works standalone but its final wiring consumes PLAN-01's `ClusterActivityEvaluator` / `SelfIsPrimary` (singleton-host semantics).
|
||
- **PLAN-01 Tasks 3–4 → PLAN-02 integration proof.** The two-node rig is how PLAN-02's Critical fixes get integration-verified.
|
||
- **PLAN-04 Task 13 (additive `RequestedBy` message change) before any PLAN-07 sender refactors** touching the same messages.
|
||
|
||
**Soft seams (either order works, coordinate):**
|
||
- **PLAN-05 Task 14 (`IScriptArtifactChangeBus`) ↔ PLAN-06 Task 3 (`InboundScriptExecutor.InvalidateMethod`).** PLAN-06 Task 1's per-request revision check is the designed correctness fallback, so there is no hard sequencing; when both land, wire the bus to the invalidate seam.
|
||
- **PLAN-03's `DeployCompileValidator` should adopt PLAN-05's compile-cache abstraction** once it exists (later refactor, not a blocker).
|
||
- **PLAN-03 Task 8's stuck-thread watchdog pattern is reusable by PLAN-06 Task 4** (inbound orphan counter).
|
||
|
||
**EF Core migrations — serialize globally.** Migration-adding tasks exist in PLAN-04 (Tasks 6, 8), PLAN-06 (Tasks 8, 19), and possibly PLAN-07 (secured-write expiry). All share one model snapshot: **never run two migration tasks concurrently**, and always build first (never `--no-build` on `dotnet ef migrations add` — known empty-scaffold gotcha).
|
||
|
||
**Shared-file merge surfaces (don't run these plans' overlapping tasks concurrently):**
|
||
| File / area | Plans touching it |
|
||
|---|---|
|
||
| `ManagementActor.cs` | PLAN-07 (15 tasks, serialized within plan), PLAN-05 (management-path publish), PLAN-06 (method-update paths) |
|
||
| `deploy/wonder-app-vd03/appsettings.Central.json` | PLAN-07 (auth), PLAN-01 (Cluster/Node sections) |
|
||
| `NotificationOutboxActor.cs` | PLAN-04, PLAN-06 (SMS retry policy) |
|
||
| `StoreAndForwardService.cs` / site tracking store | PLAN-02 (primary owner), PLAN-04 Task 15 (keyset query) |
|
||
| `Host/Program.cs` + service registration | PLAN-01, PLAN-08 (options validation) |
|
||
| `SiteHealthReport` (Commons record, additive fields) | PLAN-03 Task 6, PLAN-04 health-event fields |
|
||
| Transport `BundleImporter.cs` | PLAN-05 (primary, serialized chain), PLAN-07 (import idempotency note) |
|
||
|
||
## Dedupe Rulings (one owner per finding)
|
||
|
||
| Finding | Owner | Others must skip |
|
||
|---|---|---|
|
||
| CLI.Tests into slnx | **PLAN-08 Task 1** (also retires CLAUDE.md gotcha + memory note) | PLAN-07 Task 33 — **skip as duplicate**; mark `skipped-duplicate` in its tasks.json |
|
||
| `DisableLogin: true` artifact | **PLAN-07 Tasks 1–3** | PLAN-01 references only |
|
||
| AuditLog reconciliation NodeB failover | **PLAN-04 Task 18** | PLAN-08 pointer only |
|
||
| `SmsConfiguration.MaxRetryCount` unused | **PLAN-06 Task 18** | PLAN-08 pointer only |
|
||
| Transport `AreaName: null` | **PLAN-05 Task 7** | PLAN-08 pointer only |
|
||
| Role-casing normalization | **PLAN-07 Task 10 chain** | PLAN-08 pointer only |
|
||
| Two-node failover rig | **PLAN-01 Tasks 3–4** | PLAN-08 Task 8 is a documented placeholder; PLAN-02 consumes |
|
||
| Script-artifact invalidation contract | **PLAN-05 Task 14** (design + publisher) | PLAN-06 implements the inbound consumer |
|
||
|
||
## Recommended Execution Waves
|
||
|
||
- **Wave 0 (P0 list above).** Independent files; all six items can run in parallel. ~1 focused session.
|
||
- **Wave 1 — stability spine.** Rest of PLAN-01 (active-node semantics, drains); PLAN-02 Tasks 3–9 (standby gate, gRPC channel keying, sweep bounding); PLAN-03 Tasks 1–4 (reconnect leak, compile gate); PLAN-04 Tasks 1–5 (purge chains); PLAN-05 import chain Tasks 2–6; PLAN-06 Tasks 1–4 + 8; PLAN-07 Tasks 4–9 (deploy scoping, secret elision).
|
||
- **Wave 2 — remainder.** All other tasks per intra-plan `blockedBy` edges; migration tasks serialized globally; docs-update tasks land with their code tasks (repo rule: doc + code + tests together).
|
||
- **Wave 3 — closeout.** PLAN-08 Task 11 (deferred-work register), each plan's doc-sweep task, then a full `dotnet build ZB.MOM.WW.ScadaBridge.slnx` + `dotnet test` (which now includes CLI.Tests) + `bash docker/deploy.sh` + the two-node drill from PLAN-01.
|
||
|
||
**Execution hygiene:** work in a dedicated worktree (not the primary checkout); one plan per executor session where possible; for parallel implementers use pathspec-scoped commits and verify every commit landed on HEAD afterward (both are known failure modes in this repo's history). Re-verify a finding's line numbers before editing — reports and plans cite the code as of 2026-07-08.
|
||
|
||
## Deferred / Won't-Fix Registry (initiative-level)
|
||
|
||
> 2026-07-12 (plan R2-08 T10): the two live items below (SBR oldest-crash gap, wonder-app-vd03 overlay) now have canonical rows in `docs/plans/2026-07-08-deferred-work-register.md` — that register is the single tracking place; the subsections below remain as the historical narrative.
|
||
|
||
Consolidated in **PLAN-08 Task 11** → `docs/plans/2026-07-08-deferred-work-register.md` (triages the 23-item inventory from report 08 plus each plan's coverage-table deferrals: serializer swap, hash-chain/Parquet, aggregated live alarm stream, central cert-trust persistence, runtime script sandbox, clustered-GUID benchmark, TLS profile, Dockerfile HEALTHCHECK, etc.). Nothing from the review is silently dropped: every report finding is either a task, an ownership pointer, or a registered deferral with rationale.
|
||
|
||
### Follow-up discovered during P0 execution (2026-07-08)
|
||
|
||
- **Active(oldest)-node hard crash is NOT covered by two-node `keep-oldest` — total central outage until restart.** PLAN-01 Task 1's `downing-provider-class` fix is verified active (a hard-crashed node now transitions `Unreachable → Down → Removed`, impossible under the pre-fix NoDowning default). **But** while building PLAN-01 Task 4's behavioral proof, the live two-node rig showed (Akka 1.5.62) that `keep-oldest` downs the partition *without* the oldest: crashing the **younger** node is survivable (oldest keeps its singletons), but crashing the **oldest** node makes the younger survivor **down itself** (`Down` ~8s, terminated ~10s) → the whole central cluster is lost until the crashed node restarts. `down-if-alone = on` does not help on a hard crash (the alone-oldest is dead and cannot down itself). Because ScadaBridge treats **oldest = active node** (PLAN-01 Task 5 `ClusterActivityEvaluator`), this means a crash of the *active* central node does not fail over to the standby — the scenario operators care most about. Task 4's test was rewritten to prove the achievable guarantee (younger-crash → SBR downs it → oldest keeps singleton) and documents this asymmetry in its XML doc. **Remedy is a topology/strategy decision (deferred, owner: user):** e.g. `keep-majority` + a 3rd/lighthouse seed node, `static-quorum`, or an explicit accepted-risk note. Not fixed in P0 to avoid silently changing the production SBR strategy.
|
||
|
||
### Deferred during Wave 1 PLAN-01 execution (2026-07-08)
|
||
|
||
- **`deploy/wonder-app-vd03/` overlay edits not applied — that production deploy artifact is not tracked in this repo.** PLAN-01 Tasks 16, 20, and 23 each specified a small edit to the on-host deployment overlay (`appsettings.Central.json`: add `AllowSingleNodeCluster: true` + drop the phantom seed, add `NodeName: central-a`; `install.ps1`: add `sc.exe failure` service-recovery actions for the down-if-alone restart contract). Only the *deployment-record* doc (`docs/deployment/deployment-records/2026-06-27-wonder-app-vd03.md`) is version-controlled; the `deploy/wonder-app-vd03/` artifact directory itself is not present in the repo (production config kept out of source control). The **code/validator/doc** halves of all three tasks shipped and are what make the features usable (`AllowSingleNodeCluster` flag + validator, the `IHostApplicationLifetime` process-exit watchdog, the oldest-member Traefik/Host doc reconciliation + TLS roadmap note). **Owner: whoever maintains the on-host artifact** — apply the three overlay edits out-of-band on `wonder-app-vd03` (see the referenced task steps); without `NodeName` every audit row from that deployment stamps a NULL `SourceNode`, and without the service-recovery actions a self-downed node will not auto-restart.
|
||
|
||
### Plan deviations during Wave 1 PLAN-03 execution (2026-07-09)
|
||
|
||
- **Task 15 (S6) — deploy-success reporting redesigned: readiness handshake, not swallow-only.** The plan's Task 15 relied on the `Terminated` signal beating the persistence result, so a stale persistence `Success` could simply be swallowed once the init-failure path had already replied `Failed`. **In practice the store commits FIRST** (the optimistic `StoreDeployedConfigAsync` completes before the child's async `PreStart` throws), so the swallow-only guard still reported `Success` for an actor that then died — the exact S6 bug. Fix as shipped makes the join deterministic: `InstanceActor.PreStart` now sends a new `InstanceActorInitialized` readiness message to the DeploymentManager as its final init step, and deployment `Success` requires **both** the persistence commit **and** that signal (`InFlightDeploy.Persisted && Initialized`); an actor that dies during init never signals readiness, so the `Terminated` fallback fails the deployment and rolls back the optimistic state. A second, related bug was fixed at the same time: the optimistic `Store…` and the rollback `Remove…` were independent `Task.Run`s with no ordering guarantee, so the persisted-row rollback is now **deferred until the store commits** (`_initFailedPendingRowRemoval`) and can never race ahead of it. New public surface: `Messages/InstanceActorInitialized.cs` (additive). **Follow-up owner: PLAN-03 Task 26** — the design-doc updates must describe the two-signal deploy-success join + the readiness message (alongside Task 4's already-flagged synchronous-compile-gate deviation). No further code work outstanding; this is a doc-capture follow-up.
|
||
|
||
### Pre-existing failure discovered during Wave 1 PLAN-03 execution — owner PLAN-07 (2026-07-09)
|
||
|
||
- **✅ RESOLVED 2026-07-09 — 3 `CentralUI.Tests / SecuredWritesTests` render tests fixed by registering a stub `IBrowseService` in the bUnit harness; suite now 7/7 green. Original report retained below for history.** ~~`IBrowseService` not registered in the page's bUnit harness (PLAN-07 / Secured Writes, NOT PLAN-03).~~ PLAN-03's final full-solution `dotnet test` surfaced 3 failures in `SecuredWritesTests` (`Operator_SeesSubmitForm`, `Operator_Submit_CallsServiceWithEnteredValues`, `Verifier_OwnSubmission_HasApproveAndRejectDisabled`), all `InvalidOperationException: Cannot provide a value for property 'BrowseService' on type 'NodeBrowserDialog' — no registered service of type 'IBrowseService'`. Root cause: the Secured Writes page (`Components/Pages/Operations/SecuredWrites.razor`) renders `Components/Dialogs/NodeBrowserDialog.razor`, which `@inject`s `IBrowseService`; the test fixture (`SecuredWritesTests`) registers `_service`/`_siteRepo`/`_dialog`/auth but **not** a fake `IBrowseService`, so rendering the child dialog throws. **Confirmed pre-existing and out of PLAN-03 scope** — the failing files were last touched by commits outside the PLAN-03 session (`1a7e7351` page/test, `9cff87fe` dialog), and PLAN-03 never touches Secured Writes / CentralUI. Merged to `main` (`c7a0c922`) with this red present because it is not a regression from this branch. **Owner: PLAN-07** — register a stub `IBrowseService` in the `SecuredWritesTests` `Services` collection before `RenderPage()` (or in the shared bUnit fixture). See PLAN-07 "Post-plan known issues".
|