diff --git a/README.md b/README.md index 5c2e8236..44c2c8d8 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ This repository is the full **implementation** project for ScadaBridge — the C | `docker/` | Primary 8-node cluster topology (2 central + 3 sites × 2 nodes + Traefik) + `deploy.sh`. | | `docker-env2/` | Minimal second cluster (2 central + 1 site) for exercising Transport (#24) against a real second environment. | | `infra/` | Local test services (MS SQL, LDAP, OPC UA, SMTP, REST API, Traefik). | -| `deploy/` | Production/on-host deployment artifacts (e.g. `wonder-app-vd03/`). | +| `deploy/` | On-host (DEV/QA) deployment artifacts (e.g. `wonder-app-vd03/`). | | `AkkaDotNet/` | Akka.NET reference notes. | ## Build, Test & Run diff --git a/archreview/00-OVERALL.md b/archreview/00-OVERALL.md index 6a3d5d59..52b13e43 100644 --- a/archreview/00-OVERALL.md +++ b/archreview/00-OVERALL.md @@ -15,7 +15,7 @@ | 05 | [Templates, Deployment & Transport](05-templates-deployment-transport.md) | TemplateEngine, DeploymentManager, Transport, ScriptAnalysis | All 26 actionable round-1 findings — including the Critical inheritance-edge loss and the whole silent-data-loss family — fixed in code with zero regressions; residue is 1 Medium perf leftover and 5 Lows. | | 06 | [Edge Integrations](06-edge-integrations.md) | InboundAPI, ExternalSystemGateway, NotificationService, DelmiaNotifier | All round-1 risk genuinely fixed and source-verified (20/25 fixed, 0 regressions); new residual risk is 1 Medium + 4 Low — the Medium being the script-artifact invalidation bus publishing to zero subscribers. | | 07 | [UI, Management & Security](07-ui-management-security.md) | CentralUI, ManagementService, CLI, Security | Both round-1 Criticals dead, 17/24 fully fixed, 0 regressions; remaining new findings are Medium-and-below — mostly security seams the fixes themselves exposed (proxy-blind throttle keying, unthrottled hub bind, unscoped secured writes). | -| 08 | [Conventions, Tests & Underdeveloped](08-conventions-tests-underdeveloped.md) | Cross-cutting | Every actionable round-1 finding genuinely fixed and conventions held through 268 commits of churn; new residue is small and mostly hygiene — except a live production API key sitting in an untracked working-tree file. | +| 08 | [Conventions, Tests & Underdeveloped](08-conventions-tests-underdeveloped.md) | Cross-cutting | Every actionable round-1 finding genuinely fixed and conventions held through 268 commits of churn; new residue is small and mostly hygiene — except a live DEV-host (vd03) API key sitting in an untracked working-tree file. | ## Round-1 Finding Disposition (verified against source, not plan claims) @@ -66,7 +66,7 @@ The round-2 risk profile is different in kind: **the new findings live almost ex **H1 (new). The resync snapshot is undeliverable at exactly the scale it exists for.** Up to 10,000 full rows travel as ONE Akka remoting message with no `maximum-frame-size` override (default 128 KB) — silently dropped for any realistic backlog. *(Report 02 N2; `SiteReplicationActor.cs:40,407-410,460`)* -**H2 (new). A live production Inbound API key sits in untracked `test.txt` at the repo root** (`sbk_eb5acc…` for `wonder-app-vd03.zmr.zimmer.com:8085`) — one `git add -A` from history. Rotate the key and delete the file **today**. *(Report 08 NF1; `test.txt:3`)* +**H2 (new). A live DEV-host (vd03) Inbound API key sits in untracked `test.txt` at the repo root** (`sbk_eb5acc…` for `wonder-app-vd03.zmr.zimmer.com:8085`) — one `git add -A` from history. Rotate the key and delete the file **today**. *(Report 08 NF1; `test.txt:3`)* **H3 (new). The failover drill codifies a recovery the topology cannot deliver.** `failover-drill.sh` kills the ACTIVE(=oldest) central node — the one crash keep-oldest cannot survive (registered deferred decision) — and the restart recovery loop cannot converge when central-b is the survivor because both nodes list central-a first in `SeedNodes` (Akka's first-seed rule blocks a non-first seed from self-bootstrapping). The README admits the drill was never run live yet promises ~25s failover; the ~25s envelope also remains the last unmeasured headline design claim (the perf harness's skip precondition — the two-node rig — landed 2026-07-08). *(Reports 01 N1, 08 NF2)* diff --git a/archreview/01-cluster-host-failover.md b/archreview/01-cluster-host-failover.md index 14475e03..14efe8ca 100644 --- a/archreview/01-cluster-host-failover.md +++ b/archreview/01-cluster-host-failover.md @@ -72,9 +72,9 @@ The keep-oldest asymmetry itself is a *registered deferred user decision* (maste ### [Medium] N2 — The wonder-app-vd03 overlay edits remain unapplied while the artifact sits editable in the working tree; the deferral rationale is factually wrong -Three PLAN-01 sub-edits (Tasks 16/20/23) were deferred with the rationale "that production deploy artifact is not tracked in this repo … the artifact directory itself is not present" (master tracker, Deferred during Wave 1). The directory **is** present at `deploy/wonder-app-vd03/` — it is merely gitignored (`.gitignore:48: /deploy/`) — and was clearly edited during this initiative (its `_comment_Security` at `appsettings.Central.json:40` describes the PLAN-07 DisableLogin removal and the new bundled GLAuth). So the artifact is being maintained in this working tree, yet still lacks: +Three PLAN-01 sub-edits (Tasks 16/20/23) were deferred with the rationale "that DEV-host (vd03) deploy artifact is not tracked in this repo … the artifact directory itself is not present" (master tracker, Deferred during Wave 1). The directory **is** present at `deploy/wonder-app-vd03/` — it is merely gitignored (`.gitignore:48: /deploy/`) — and was clearly edited during this initiative (its `_comment_Security` at `appsettings.Central.json:40` describes the PLAN-07 DisableLogin removal and the new bundled GLAuth). So the artifact is being maintained in this working tree, yet still lacks: -- `Node.NodeName` → every audit row from the **only production install** stamps a NULL `SourceNode`, defeating `IX_AuditLog_Node_Occurred` and the per-node stuck KPIs (round-1 U5, unchanged); +- `Node.NodeName` → every audit row from the **only live install (DEV vd03)** stamps a NULL `SourceNode`, defeating `IX_AuditLog_Node_Occurred` and the per-node stuck KPIs (round-1 U5, unchanged); - `Cluster.AllowSingleNodeCluster: true` + removal of the phantom seed `localhost:8091` (round-1 S12's artifact half — the node still dials a nonexistent seed forever); - the `install.ps1` `sc.exe failure` service-recovery actions that the down-if-alone watchdog contract *requires* ("the Host process must exit **so the service supervisor restarts it**" — `Component-ClusterInfrastructure.md:106-110`); without them, a self-downed single node on that host stays down. diff --git a/archreview/07-ui-management-security.md b/archreview/07-ui-management-security.md index c8e7b0d9..103ba230 100644 --- a/archreview/07-ui-management-security.md +++ b/archreview/07-ui-management-security.md @@ -13,7 +13,7 @@ ## Round-1 vs Round-2 Verdict -**Round 1:** mature surface with sharp asymmetries — an auth-disabled production artifact, a silently-ignored deploy option, inconsistent secret projection, a fictional secured-write "Expired" state. **Round 2:** the fix plan genuinely landed — 17 of 24 findings fully fixed and verified in source (including both criticals), 6 partially fixed with the residue consciously deferred and documented, 1 was a false premise; the new code (frozen 139-entry authz matrix, dispatch-coverage guard, TTL'd secured writes, live alarm cache) is of high quality, and the remaining new findings are Medium-and-below — mostly seams the fixes themselves exposed (an unthrottled debug-hub bind the docs claim is covered, throttle keying behind Traefik, and site-scope not enforced on secured writes despite the doc saying scoping "is layered on at the LDAP-mapping level"). +**Round 1:** mature surface with sharp asymmetries — an auth-disabled DEV-host (vd03) artifact, a silently-ignored deploy option, inconsistent secret projection, a fictional secured-write "Expired" state. **Round 2:** the fix plan genuinely landed — 17 of 24 findings fully fixed and verified in source (including both criticals), 6 partially fixed with the residue consciously deferred and documented, 1 was a false premise; the new code (frozen 139-entry authz matrix, dispatch-coverage guard, TTL'd secured writes, live alarm cache) is of high quality, and the remaining new findings are Medium-and-below — mostly seams the fixes themselves exposed (an unthrottled debug-hub bind the docs claim is covered, throttle keying behind Traefik, and site-scope not enforced on secured writes despite the doc saying scoping "is layered on at the LDAP-mapping level"). --- @@ -21,7 +21,7 @@ | Finding | R1 Severity | Disposition | Evidence (file:line) | |---|---|---|---| -| C1 — `DisableLogin: true` shipped in production deploy artifact; no environment guard | Critical | **Fixed (verified)** | `Security/Auth/DisableLoginGuard.cs:17-28` (refuses outside Development without `AllowOutsideDevelopment` ack); wired fail-fast at the composition root `Host/Program.cs:149-154` *before* `AddSecurity`; `deploy/wonder-app-vd03/appsettings.Central.json` no longer contains `DisableLogin` (repo-wide grep: no non-`false` occurrence under `deploy/`, `docker/`, `docker-env2/`), and its `_comment_Security` documents the new posture | +| C1 — `DisableLogin: true` shipped in the DEV-host (vd03) deploy artifact; no environment guard | Critical | **Fixed (verified)** | `Security/Auth/DisableLoginGuard.cs:17-28` (refuses outside Development without `AllowOutsideDevelopment` ack); wired fail-fast at the composition root `Host/Program.cs:149-154` *before* `AddSecurity`; `deploy/wonder-app-vd03/appsettings.Central.json` no longer contains `DisableLogin` (repo-wide grep: no non-`false` occurrence under `deploy/`, `docker/`, `docker-env2/`), and its `_comment_Security` documents the new posture | | C2 — `deploy artifacts --site-id` silently ignored; fleet-wide deploy with no scope check | High | **Fixed (verified)** | `ArtifactDeploymentService.cs:228-255` (`DeployToAllSitesAsync`/`DeployToSiteAsync` over shared `DeployCoreAsync`); `ManagementActor.cs:2295-2316` honors `cmd.SiteId`, adds the explicit fleet-wide guard for site-scoped non-admins (`SiteScopeViolationException`, :2300-2306) **and** `EnforceSiteScope` (:2307) | | C3 — DataConnection / ExternalSystem / DatabaseConnection secrets leak in List/Get + audit | High | **Fixed (verified)** | `ConfigSecretScrubber.cs` (recursive JSON elision + sentinel merge); `DataConnectionPublicShape` `ManagementActor.cs:1632-1647` applied to List/Get/Create/Update **and** audit afterState (:1653-1654, :1663, :1677-1678, :1697-1698), sentinel-preserving update :1692-1693; `ExternalSystemPublicShape` :1847-1857 + preserve-if-null :1897; `DatabaseConnectionPublicShape` :2835-2841 applied :2845-2878. Note: UI editors write via repositories directly (e.g. `ExternalSystemForm.razor`), so UI round-tripping was never at risk; CLI round-trips are sentinel-protected | | S1 — 30 s server Ask vs 5-min bundle operations; import keeps applying after 504 | High | **Partially fixed (verified; residue deferred)** | `ManagementEndpoints.cs:41-58` — `ResolveAskTimeout(options, commandType)` gives Import/Preview/Export/DeployArtifacts/DeployInstance a 5-min default (`ManagementServiceOptions.cs:9`), call site :126; `ManagementService` timeout config shipped to all three topologies (`docker/central-node-a/appsettings.Central.json:51`, `docker-env2/.../:48`, `deploy/wonder-app-vd03/.../:49`); `ManagementServiceOptionsValidator` fail-fast. **Still open (accepted, plan-05 domain):** `ImportBundleCommand` (`TransportCommands.cs:121-128`) has no client-supplied idempotency id — a timeout-then-retry can still double-apply; the `TransportGate` single-flight (:3203) only narrows the window on one node | diff --git a/archreview/08-conventions-tests-underdeveloped.md b/archreview/08-conventions-tests-underdeveloped.md index 4b0ef9d5..9b2983c1 100644 --- a/archreview/08-conventions-tests-underdeveloped.md +++ b/archreview/08-conventions-tests-underdeveloped.md @@ -16,7 +16,7 @@ Method: targeted greps and direct file reads only (static review; no builds/test ## Verdict -**Round 1 vs round 2 in one line:** every actionable round-1 finding was genuinely fixed (17 fixed, 3 partially, 6 consciously deferred with register rows, 0 regressed) and the conventions that were already clean stayed clean through 268 commits of fix-plan churn; the new residue is small and mostly hygiene — one real embarrassment (a live production API key sitting in an untracked `test.txt` at the repo root), a failover-perf placeholder whose stated precondition has already been satisfied, and an abandoned CHANGELOG that is now factually wrong. +**Round 1 vs round 2 in one line:** every actionable round-1 finding was genuinely fixed (17 fixed, 3 partially, 6 consciously deferred with register rows, 0 regressed) and the conventions that were already clean stayed clean through 268 commits of fix-plan churn; the new residue is small and mostly hygiene — one real embarrassment (a live DEV-host (vd03) API key sitting in an untracked `test.txt` at the repo root), a failover-perf placeholder whose stated precondition has already been satisfied, and an abandoned CHANGELOG that is now factually wrong. --- @@ -132,8 +132,8 @@ Two register-integrity problems, though: ## 5. New Findings (round 2) -### NF1 [High] — Live production API key in plaintext at the repo root -`test.txt` (untracked, repo root) contains a working Inbound API key for the production host `wonder-app-vd03.zmr.zimmer.com:8085` (`sbk_eb5acc0b…`, full key present). It is one `git add -A` (or one run of the `pushit` skill, which stages everything) away from entering history, and it is a live credential regardless. **Rotate/revoke the key** (the dual-key rotation procedure from PLAN-06 T24 exists for exactly this), delete the file, and consider a `.gitignore`/pre-commit secret-scan guard for the root. (File: `/Users/dohertj2/Desktop/ScadaBridge/test.txt`.) +### NF1 [High] — Live DEV-host (vd03) API key in plaintext at the repo root +`test.txt` (untracked, repo root) contains a working Inbound API key for the DEV host `wonder-app-vd03.zmr.zimmer.com:8085` (`sbk_eb5acc0b…`, full key present). It is one `git add -A` (or one run of the `pushit` skill, which stages everything) away from entering history, and it is a live credential regardless. **Rotate/revoke the key** (the dual-key rotation procedure from PLAN-06 T24 exists for exactly this), delete the file, and consider a `.gitignore`/pre-commit secret-scan guard for the root. (File: `/Users/dohertj2/Desktop/ScadaBridge/test.txt`.) ### NF2 [Medium] — Failover-timing test's skip precondition is already satisfied; the 25s envelope remains unmeasured `tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs:26-28` skips with "Requires the real two-node failover rig delivered by PLAN-01 … wire it to the rig when PLAN-01 lands" — but PLAN-01's rig landed 2026-07-08 (`tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/TwoNodeClusterFixture.cs`), two days before this placeholder shipped. The design's single most-quoted performance claim (~25s failover) is still unverified by any test, the register's revisit trigger for this row has fired, and the skip message is now misleading. Wire the protocol to the fixture (or update the skip reason + register row with the real blocker, e.g. "in-proc rig can't hard-kill a process"). @@ -182,6 +182,6 @@ Beyond `test.txt` (NF1): `docs/plans/2026-06-30-mes-alarm-status-api.md` is a 12 | Severity | Count | Findings | |---|---|---| | Critical | 0 | — | -| High | 1 | NF1 (production API key in working tree) | +| High | 1 | NF1 (DEV-host API key in working tree) | | Medium | 2 | NF2 (failover envelope unmeasured, stale skip), NF3 (CHANGELOG abandoned + wrong) | | Low | 5 | NF4 (options residue), NF5 (deferral scattering), NF6 (stale Transport comment), NF7 (untracked plan doc + clutter), NF8 (duplicate bindings) | diff --git a/archreview/plans/00-MASTER-TRACKER.md b/archreview/plans/00-MASTER-TRACKER.md index 9a71fa88..466c89a3 100644 --- a/archreview/plans/00-MASTER-TRACKER.md +++ b/archreview/plans/00-MASTER-TRACKER.md @@ -31,7 +31,7 @@ Every plan follows the TDD bite-sized-task format and ships a co-located `.tasks ## Round-2 P0 (do first, any order) -1. **R2-08 T1/T2** (→ #74) — rotate the exposed wonder-app-vd03 API key; delete `test.txt` + the three root credential files (ALL contain live secrets incl. a production sysadmin password). `needs-user`. +1. **R2-08 T1/T2** (→ #74) — rotate the exposed wonder-app-vd03 API key; delete `test.txt` + the three root credential files (ALL contain live secrets incl. a DEV-host (vd03) sysadmin password). `needs-user`. 2. **R2-02 T1–T4** — the Critical: unify the active-node predicate; failing-first buffer-wipe repro then fix. 3. **R2-02 T5–T7** — the resync frame-size High (chunked protocol). 4. **R2-04 T2–T4** — bound the rollup backfill before the next central failover at data volume. @@ -135,7 +135,7 @@ Run up to **five plan-lane executor sessions concurrently** (one each for PLAN-0 1. ✅ **PLAN-01 Task 1** — enable the SBR downing provider (the system-wide Critical; one line + parsed-config test). *Provider verified active via the live two-node rig.* 2. ✅ **PLAN-01 Tasks 3–4** — the two-node kill-test rig (`TwoNodeClusterFixture`); integration-proves Task 1 and later PLAN-02 work. *Task 4 rewritten to the achievable keep-oldest guarantee (younger-crash → SBR downs it → oldest keeps singleton); oldest-crash gap registered as follow-up.* -3. ✅ **PLAN-07 Tasks 1–3** — `DisableLoginGuard` + scrub `deploy/wonder-app-vd03` (Critical; unauthenticated prod surface). *Guard + Host wiring committed; `deploy/` is gitignored so its scrub is on-disk-only (not version-controlled), effective on next redeploy.* +3. ✅ **PLAN-07 Tasks 1–3** — `DisableLoginGuard` + scrub `deploy/wonder-app-vd03` (Critical; unauthenticated DEV-host surface). *Guard + Host wiring committed; `deploy/` is gitignored so its scrub is on-disk-only (not version-controlled), effective on next redeploy.* 4. ✅ **PLAN-02 Tasks 1–2** — ClusterClient generation-suffixed names (Critical crash-loop dropping routing to all sites). 5. ✅ **PLAN-05 Task 1** — bundle import consumes `BaseTemplateName` (Critical silent data loss). 6. ✅ **PLAN-08 Task 1** — CLI.Tests into the slnx (protects every subsequent plan's test runs). *341 CLI tests now run at the solution level.* diff --git a/archreview/plans/PLAN-07-ui-management-security.md b/archreview/plans/PLAN-07-ui-management-security.md index 95cf190f..5b96b5d9 100644 --- a/archreview/plans/PLAN-07-ui-management-security.md +++ b/archreview/plans/PLAN-07-ui-management-security.md @@ -2,7 +2,7 @@ > **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. -**Goal** — Close every finding from architecture review 07 (`archreview/07-ui-management-security.md`): kill the auth-disabled production artifact, honor per-site artifact deployment with a site-scope check, elide connection/external-system/DB secrets everywhere SMTP/SMS already are, make secured-write expiry real server-side, fix the 30s-vs-5min Ask-timeout mismatch, give the CLI actor-side browse/verify/cert-trust parity, canonicalize role casing, throttle LDAP binds, and freeze the authorization table with a matrix test. +**Goal** — Close every finding from architecture review 07 (`archreview/07-ui-management-security.md`): kill the auth-disabled DEV-host (vd03) artifact, honor per-site artifact deployment with a site-scope check, elide connection/external-system/DB secrets everywhere SMTP/SMS already are, make secured-write expiry real server-side, fix the 30s-vs-5min Ask-timeout mismatch, give the CLI actor-side browse/verify/cert-trust parity, canonicalize role casing, throttle LDAP binds, and freeze the authorization table with a matrix test. **Architecture** — All fixes stay inside the existing management/presentation seam: the `ManagementActor` (single dispatch switch + `GetRequiredRole` gate + `PipeTo`), the `ManagementAuthenticator` Basic→LDAP flow shared by every CLI surface, the Security component's cookie/JWT/policy layer, and the Blazor Central UI. New primitives are small and testable in isolation (`DisableLoginGuard`, `ConfigSecretScrubber`, `LoginThrottle`, `PollGate`), wired at the existing composition points. Message-contract changes are strictly additive (optional trailing record params, new enum members) per the repo's contract-evolution rule; every spec-affecting change updates `docs/requirements/Component-Security.md` / `Component-ManagementService.md` / `Component-CentralUI.md` / `Component-CLI.md` in the same task as the code. @@ -27,7 +27,7 @@ See [00-MASTER-TRACKER.md § Parallelization Map](00-MASTER-TRACKER.md#paralleli | Finding | Severity | Disposition | |---|---|---| -| C1 — `DisableLogin: true` in production deploy artifact, no environment guard | Critical | Tasks 1, 2, 3 | +| C1 — `DisableLogin: true` in the DEV-host (vd03) deploy artifact, no environment guard | Critical | Tasks 1, 2, 3 | | C2 — `deploy artifacts --site-id` silently ignored; no site-scope check | High | Tasks 4, 5 | | C3 — DataConnection/ExternalSystem/DB-connection secrets leak in List/Get + audit | High | Tasks 6, 7, 8, 9 | | S1 — 30s server Ask vs 5-min bundle/deploy operations; import keeps applying after 504 | High | Tasks 19, 20. Client-supplied import-id idempotency: **Deferred** — Transport apply-path change, plan 05's domain (coordinate; single-flight in Task 30 also narrows the double-apply window) | diff --git a/archreview/plans/PLAN-R2-01-cluster-host-failover.md b/archreview/plans/PLAN-R2-01-cluster-host-failover.md index 377d83b2..c78a2996 100644 --- a/archreview/plans/PLAN-R2-01-cluster-host-failover.md +++ b/archreview/plans/PLAN-R2-01-cluster-host-failover.md @@ -335,7 +335,7 @@ git commit -m "test(failover): unskip FailoverTimingTests on the two-node rig at - Modify: `deploy/wonder-app-vd03/appsettings.Central.json` (`Node` lines 3-7, `Cluster` lines 8-18, `_comment_Cluster` line 19) - Modify: `deploy/wonder-app-vd03/appsettings.Site.json` (`Node` lines 3-9, `Cluster` lines 10-20, `_comment_Cluster` line 21) -Round-2 N2: the round-1 deferral claimed the artifact "is not tracked in this repo … the artifact directory itself is not present" — **the directory IS present** (merely gitignored, `.gitignore:48: /deploy/`) and has been actively maintained in this working tree (its `_comment_Security` documents the PLAN-07 edit). Apply the two appsettings edits from PLAN-01 Tasks 16/23 directly. Without `NodeName`, every audit row from the only production install stamps a NULL `SourceNode` (defeats `IX_AuditLog_Node_Occurred` + the per-node stuck KPIs); with the phantom seeds, each node dials a nonexistent peer (`localhost:8091` / `localhost:8092`) forever. +Round-2 N2: the round-1 deferral claimed the artifact "is not tracked in this repo … the artifact directory itself is not present" — **the directory IS present** (merely gitignored, `.gitignore:48: /deploy/`) and has been actively maintained in this working tree (its `_comment_Security` documents the PLAN-07 edit). Apply the two appsettings edits from PLAN-01 Tasks 16/23 directly. Without `NodeName`, every audit row from the only live install (DEV vd03) stamps a NULL `SourceNode` (defeats `IX_AuditLog_Node_Occurred` + the per-node stuck KPIs); with the phantom seeds, each node dials a nonexistent peer (`localhost:8091` / `localhost:8092`) forever. 1. `appsettings.Central.json`: - `Node` (after `"Role": "Central",`): add `"NodeName": "central-a",` (matches the docker convention, `docker/central-node-a/appsettings.Central.json`). @@ -390,10 +390,10 @@ foreach ($svc in 'ScadaBridge-LDAP', 'ScadaBridge-Central', 'ScadaBridge-Site') **Files:** - Modify: `archreview/plans/00-MASTER-TRACKER.md` (line 28 trailing sentence; "Deferred during Wave 1 PLAN-01 execution (2026-07-08)" bullet at lines 145-147) -The deferral rationale — "that production deploy artifact is not tracked in this repo … the artifact directory itself is not present in the repo" — is factually wrong (round-2 N2): `deploy/wonder-app-vd03/` exists in the working tree, is merely gitignored (`.gitignore:48: /deploy/`), and was edited during this initiative (PLAN-07's `_comment_Security`). Depends on Tasks 5-6 having applied the edits. +The deferral rationale — "that DEV-host (vd03) deploy artifact is not tracked in this repo … the artifact directory itself is not present in the repo" — is factually wrong (round-2 N2): `deploy/wonder-app-vd03/` exists in the working tree, is merely gitignored (`.gitignore:48: /deploy/`), and was edited during this initiative (PLAN-07's `_comment_Security`). Depends on Tasks 5-6 having applied the edits. 1. Rewrite the line-147 bullet: keep the history (what was deferred and why it mattered), then correct the record: the directory **is present but gitignored** (production config kept out of version control — the *tracking* claim was wrong, the *presence* claim was false); the three overlay edits were **applied on-disk 2026-07-12 by PLAN-R2-01 Tasks 5-6** (`NodeName` central-a/node-a, `AllowSingleNodeCluster` + phantom seeds dropped, `sc.exe failureflag` completing the already-present `sc.exe failure` actions). Remaining owner action: sync the artifact to `wonder-app-vd03` and restart services (appsettings take effect on restart; failureflag on re-running install.ps1 §7 or the two `sc.exe` lines manually) — per the RUNBOOK row added by T6. -2. Trim the stale trailing sentence of the line-28 Wave-1 summary ("were deferred because that production artifact is not tracked in this repo") → "were deferred on a mistaken not-in-repo rationale; corrected and applied on-disk by PLAN-R2-01 (2026-07-12) — see the Deferred-during-Wave-1 entry." +2. Trim the stale trailing sentence of the line-28 Wave-1 summary ("were deferred because that DEV-host (vd03) artifact is not tracked in this repo") → "were deferred on a mistaken not-in-repo rationale; corrected and applied on-disk by PLAN-R2-01 (2026-07-12) — see the Deferred-during-Wave-1 entry." 3. Do NOT add rows to `docs/plans/2026-07-08-deferred-work-register.md` — register consolidation (round-2 N6) is owned by **PLAN-R2-08**; this task only stops the tracker from asserting a falsehood. 4. Verify: `grep -n "not tracked in this repo" archreview/plans/00-MASTER-TRACKER.md` → 0 hits; `grep -n "PLAN-R2-01" archreview/plans/00-MASTER-TRACKER.md` → ≥2 hits. 5. Commit: diff --git a/archreview/plans/PLAN-R2-08-conventions-hygiene.md b/archreview/plans/PLAN-R2-08-conventions-hygiene.md index 32a9b07f..d6cd12ea 100644 --- a/archreview/plans/PLAN-R2-08-conventions-hygiene.md +++ b/archreview/plans/PLAN-R2-08-conventions-hygiene.md @@ -4,7 +4,7 @@ > **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. -**Goal:** Close the eight NEW findings from architecture review 08 round 2 (`archreview/08-conventions-tests-underdeveloped.md`, 2026-07-12) — rotate the live production Inbound API key sitting in `test.txt` at the repo root and purge the other plaintext credential captures (NF1, High), refresh the abandoned-and-now-wrong CHANGELOG (NF3), finish the options-validation tail (AuditLog's 6 sub-options + Host Node/Database/Logging + OperationTracking — NF4), re-consolidate deferral tracking into the canonical register (NF5), fix the stale Transport area comment (NF6), commit the orphaned MES plan doc + clean root clutter (NF7), and kill the duplicate/drifted Host options bindings (NF8). NF2 (failover envelope measurement) is **owned by PLAN-R2-01** — this plan only repairs the register row whose revisit trigger fired. +**Goal:** Close the eight NEW findings from architecture review 08 round 2 (`archreview/08-conventions-tests-underdeveloped.md`, 2026-07-12) — rotate the live DEV-host (vd03) Inbound API key sitting in `test.txt` at the repo root and purge the other plaintext credential captures (NF1, High), refresh the abandoned-and-now-wrong CHANGELOG (NF3), finish the options-validation tail (AuditLog's 6 sub-options + Host Node/Database/Logging + OperationTracking — NF4), re-consolidate deferral tracking into the canonical register (NF5), fix the stale Transport area comment (NF6), commit the orphaned MES plan doc + clean root clutter (NF7), and kill the duplicate/drifted Host options bindings (NF8). NF2 (failover envelope measurement) is **owned by PLAN-R2-01** — this plan only repairs the register row whose revisit trigger fired. **Architecture:** No new runtime components, no schema changes, no wire-contract changes. The options work copies the shipped `OptionsValidatorBase` + `AddValidatedOptions` pattern (`ZB.MOM.WW.Configuration`; see `AuditLog/ServiceCollectionExtensions.cs:81` and `KpiHistory/ServiceCollectionExtensions.cs:40`). The credential work is operational (dual-key rotation per `docs/requirements/Component-InboundAPI.md` §Key rotation, shipped by PLAN-06 T24) plus working-tree hygiene with a `.gitignore`/pre-commit guard. One discovery made while verifying NF4 is folded in as Task 8: the site-only `IOperationTrackingStore` appears to have **no DI registration anywhere in `src/`**, despite two comments claiming `AddSiteRuntime` registers it — Task 8 verifies and fixes that seam alongside the missing `OperationTrackingOptions` binding. @@ -19,19 +19,19 @@ - **Task 1 FIRST** (NF1 is the P0; it needs the user/host, so start it immediately and let the wait overlap other tasks). Task 2 follows Task 1 (rotate before destroying the local record of what was exposed). - **Mutually parallel (file-disjoint):** 1, 3, 4, 5, 7, 10, 12, 13. - **Serial chains (shared files):** 5 → 6 (both edit `AuditLog/ServiceCollectionExtensions.cs`); 7 → 8 → 9 (all edit `Host/SiteServiceRegistration.cs`); 10 → 11 (delete the snapshot only after the register has absorbed its content). -- Tasks 1, 2, 11, and the deletion half of 13 are flagged **`needs-user`** — they require production-host access and/or destroy untracked (unrecoverable) files, so the executor must show the user what is being done and get confirmation. +- Tasks 1, 2, 11, and the deletion half of 13 are flagged **`needs-user`** — they require DEV-host (vd03) access and/or destroy untracked (unrecoverable) files, so the executor must show the user what is being done and get confirmation. --- ### Task 1: Rotate the exposed wonder-app-vd03 Inbound API key (dual-key flow) — `needs-user` → #74 -**Classification:** NF1 — high-risk security hygiene; **`needs-user`** (production environment access; the executor runs the commands only with the user present/consenting, and cannot complete step 3's caller-cutover without the user) +**Classification:** NF1 — high-risk security hygiene; **`needs-user`** (DEV-host (vd03) environment access; the executor runs the commands only with the user present/consenting, and cannot complete step 3's caller-cutover without the user) **Estimated implement time:** ~5 min of active work (cutover/quarantine waits excluded) **Parallelizable with:** 3, 4, 5, 7, 10, 12, 13 **Files:** - Modify: `docs/deployment/deployment-records/2026-06-27-wonder-app-vd03.md` (append a dated rotation record — the only repo artifact this task produces) -**Context (verified):** `test.txt` (untracked, repo root) contains a **working production Inbound API key** for `http://wonder-app-vd03.zmr.zimmer.com:8085` — `sbk_eb5acc0b…` (full token on line 3) scoped at least to `MesMoveOut`. The key is live regardless of whether the file ever enters git history, so it must be **rotated**, not merely deleted. The zero-downtime dual-key procedure is already documented (`docs/requirements/Component-InboundAPI.md:75-86`, PLAN-06 T24): create → cutover → disable → quarantine → delete. This task is that procedure instantiated against wonder-app-vd03 as an executable checklist. Host facts (auto-memory `wonder-app-vd03-host-access`): the deployment runs with `DisableLogin=true`, so `:8085/management` is unauthenticated and the CLI works with just `--url`; host shell access (if needed) is servecli SSH port 2222. +**Context (verified):** `test.txt` (untracked, repo root) contains a **working DEV-host (vd03) Inbound API key** for `http://wonder-app-vd03.zmr.zimmer.com:8085` — `sbk_eb5acc0b…` (full token on line 3) scoped at least to `MesMoveOut`. The key is live regardless of whether the file ever enters git history, so it must be **rotated**, not merely deleted. The zero-downtime dual-key procedure is already documented (`docs/requirements/Component-InboundAPI.md:75-86`, PLAN-06 T24): create → cutover → disable → quarantine → delete. This task is that procedure instantiated against wonder-app-vd03 as an executable checklist. Host facts (auto-memory `wonder-app-vd03-host-access`): the deployment runs with `DisableLogin=true`, so `:8085/management` is unauthenticated and the CLI works with just `--url`; host shell access (if needed) is servecli SSH port 2222. **Checklist (run each step, record output):** @@ -76,9 +76,9 @@ If the user cannot reach wonder-app-vd03 in this session, record the checklist s - Delete (untracked+ignored, user-confirmed): `ldap_login.txt`, `sql_login.txt`, `sqllogin.txt` **Context (verified by reading each file):** -- `test.txt` — the live `sbk_eb5acc0b…` production API key (Task 1's subject) plus an ad-hoc test note. Untracked and NOT gitignored — one `git add -A` (or the `pushit` skill, which stages everything) from entering history. +- `test.txt` — the live `sbk_eb5acc0b…` DEV-host (vd03) API key (Task 1's subject) plus an ad-hoc test note. Untracked and NOT gitignored — one `git add -A` (or the `pushit` skill, which stages everything) from entering history. - `ldap_login.txt` — **live AD credentials**: `zimdom.zimmer.com:3268` bind with the `SVC-DNC` service-account password AND the user's personal domain password. Ignored via `.gitignore:45` (`*login*.txt`) so unstageable, but plaintext on disk. -- `sql_login.txt` — the wonder-app-vd03 SQL Server `wwadmin` password (the production ScadaBridge config DB). Ignored, plaintext. +- `sql_login.txt` — the wonder-app-vd03 SQL Server `wwadmin` password (the DEV ScadaBridge config DB). Ignored, plaintext. - `sqllogin.txt` — a provisioning transcript that creates `wwadmin` **with sysadmin** using the same password. Ignored, plaintext. None of the four are in git history (verified: untracked/ignored; `git log` shows no commits touching them), so this is disk cleanup, not history surgery. @@ -496,7 +496,7 @@ Four edits, all specified by NF5 + the register's own rules: | Report finding (08-conventions-tests-underdeveloped.md, round 2) | Severity | Coverage | |---|---|---| -| NF1 — live production Inbound API key in `test.txt` at the repo root (plus, found on triage: live AD/SQL credentials in the gitignored `ldap_login.txt`/`sql_login.txt`/`sqllogin.txt`) | High | Tasks 1 (rotate via the shipped dual-key flow, `needs-user`), 2 (delete the credential files, `needs-user`), 3 (`.gitignore` + pre-commit guard) | +| NF1 — live DEV-host (vd03) Inbound API key in `test.txt` at the repo root (plus, found on triage: live AD/SQL credentials in the gitignored `ldap_login.txt`/`sql_login.txt`/`sqllogin.txt`) | High | Tasks 1 (rotate via the shipped dual-key flow, `needs-user`), 2 (delete the credential files, `needs-user`), 3 (`.gitignore` + pre-commit guard) | | NF2 — failover envelope unmeasured; `FailoverTimingTests` skip precondition already satisfied | Medium | **Owned by PLAN-R2-01** (rig wiring / measurement). This plan: Task 10 step 3 repairs the register row whose "PLAN-01 rig landing" trigger fired unnoticed, repointing it at PLAN-R2-01's measurement task | | NF3 — CHANGELOG.md abandoned and factually wrong (`Operator`/`Engineer` claim false since M7) | Medium | Task 4 | | NF4 — options-validation residue: 6 AuditLog sub-options, Host Node/Database/Logging, unbound `OperationTrackingOptions` | Low | Tasks 5, 6 (AuditLog site + central sub-options), 7 (Host options; empty `NodeName` fails fast), 8 (OperationTracking binding + the discovered missing site `IOperationTrackingStore` registration) | diff --git a/docs/api-specs/Delmia-Recipe-Download-API.md b/docs/api-specs/Delmia-Recipe-Download-API.md index 5862fec0..4a5216d4 100644 --- a/docs/api-specs/Delmia-Recipe-Download-API.md +++ b/docs/api-specs/Delmia-Recipe-Download-API.md @@ -1,9 +1,9 @@ # DelmiaRecipeDownload — ScadaBridge Inbound API spec (as deployed) -**Captured:** 2026-09-02 from the live **prod** management API on `wonder-app-vd03` (method id 4, +**Captured:** 2026-09-02 from the live **DEV** management API on `wonder-app-vd03` (method id 4, API keys, the `CvdReactor` template's `ProcessRecipeDownload` script, the `DelmiaReceiver` template, and the inbound audit log). **QA (`vq03`/`vq04`) was NOT read** — its management API requires real -LDAP credentials (prod runs with `DisableLogin`); re-capture QA with `scadabridge api-method get --id …` +LDAP credentials (DEV vd03 runs with `DisableLogin`); re-capture QA with `scadabridge api-method get --id …` once credentials are to hand. The definition is config-as-data in the central ConfigDb (not in git), so this file is the durable record; the legacy predecessor is documented in [`../former-api-specs/dnc/Delmia-Integration-API.md`](../former-api-specs/dnc/Delmia-Integration-API.md) @@ -18,7 +18,7 @@ this file is the durable record; the legacy predecessor is documented in | Auth | `X-API-Key: sbk__` **or** `Authorization: Bearer sbk__` (Authorization wins if both) | | Content type | `application/json`, flat PascalCase object | | Server timeout | **45 s** (`TimeoutSeconds` on the method); the site-side handshake wait inside it is **25 s** | -| Prod base URL | `http://wonder-app-vd03.zmr.zimmer.com:8085` (single central node) | +| DEV (vd03) base URL | `http://wonder-app-vd03.zmr.zimmer.com:8085` (single central node) | | QA base URLs | `http://10.220.158.2:8085` (vq03), `http://10.220.158.3:8085` (vq04) — list **both** in the notifier's `BaseUrls`; a standby central relays to the active peer | Unauthenticated / bad-key calls return **401** `{"error":"Invalid or missing API key"}` (audit kind @@ -44,14 +44,14 @@ script itself rejects a missing/blank `MachineCode`): | Field | Notifier flag | Meaning | Where it lands on the site | |---|---|---|---| -| `MachineCode` | `-m` | **Routing key** — must equal an instance `UniqueName` exactly (trimmed). Prod today: `Z28061`, `Z28062`, `Z28061Sim`. | `Route.To(MachineCode)` | +| `MachineCode` | `-m` | **Routing key** — must equal an instance `UniqueName` exactly (trimmed). DEV (vd03) today: `Z28061`, `Z28062`, `Z28061Sim`. | `Route.To(MachineCode)` | | `DownloadPath` | `-d` | Path where the recipe/NC file was placed | `DelmiaReceiver.DownloadPath` | | `WorkOrderNumber` | `-w` | Work order | `DelmiaReceiver.WorkOrderNumber` | | `PartNumber` | `-p` | Part / item number | `DelmiaReceiver.PartNumber` | | `JobStepNumber` | `-s` | Job step / sequence op | `DelmiaReceiver.JobStepNumber` | | `Username` | `-u` | Operator (audit only) | `DelmiaReceiver.Username` | -Example (a real call taken from the prod audit log): +Example (a real call taken from the DEV (vd03) audit log): ```bash curl -s -X POST http://wonder-app-vd03.zmr.zimmer.com:8085/api/DelmiaRecipeDownload \ @@ -93,7 +93,7 @@ Delmia ─► WWNotifier.exe ─► POST /api/DelmiaRecipeDownload (central, act 4. read RecipeProcessResult / RecipeProcessResultText → {Result, ResultText} ``` -Both scripts as deployed on prod (verbatim): +Both scripts as deployed on DEV vd03 (verbatim):
Central inbound method script (id 4, timeout 45 s) @@ -154,12 +154,12 @@ try { > the PREVIOUS download's cached `true`. The runtime fix is the reset-aware overload — > `WriteBatchAndWaitAsync(data, "RecipeDownloadFlag", true, "RecipeProcessedFlag", true, 25 s, > "RecipeProcessedFlag", false, 10 s)` returning a `HandshakeResult` — which replaces steps 2–3. -> The deployed prod/QA script bodies are NOT yet ported (deferred-work register row 119). +> The deployed DEV/QA script bodies are NOT yet ported (deferred-work register row 119). ## 5. Galaxy-side contract — the `DelmiaReceiver` template (id 12; composed clone id 26) All ten attributes are **data-sourced over the site `MxGateway` connection**, bound per instance to the -reactor-level Galaxy object (prod: `DelmiaReceiver_038` on Z28061, `_039` on Z28062, `_037` on +reactor-level Galaxy object (DEV vd03: `DelmiaReceiver_038` on Z28061, `_039` on Z28062, `_037` on Z28061Sim; `Left_00x.DelmiaReceiver` / `Right_00x.DelmiaReceiver` back the unused side compositions). | Attribute | Type | Direction | Role in the handshake | @@ -171,17 +171,17 @@ Z28061Sim; `Left_00x.DelmiaReceiver` / `Right_00x.DelmiaReceiver` back the unuse | `RecipeProcessResultText` | String | Galaxy → SB | Outcome text → `ResultText` | | `DownloadPath`, `WorkOrderNumber`, `PartNumber`, `JobStepNumber`, `Username` | String | SB → Galaxy | Payload, written as one batch before the trigger | -## 6. API keys scoped to this method (prod, names only — secrets are shown once at creation) +## 6. API keys scoped to this method (DEV vd03, names only — secrets are shown once at creation) `mes`, `mes api key`, `delmia-mes-receivers`, `TestKey`, `JOE_TEST_KEY` — all enabled, all also scoped to the MES methods (`MesMoveIn`, `MesMoveOut`, `MesSampleData`, `SimpleAlarmStatusRequest`, most also `IpsenMESMoveIn`). `delmia-mes-receivers` is the one whose name says it is the Delmia caller's key; the audit `actor` column shows which key served each call. -## 7. Observed behaviour and open issues (from the prod audit log, 2026-06-17 → 2026-06-26) +## 7. Observed behaviour and open issues (from the DEV vd03 audit log, 2026-06-17 → 2026-06-26) Nine calls total, all test traffic (actors `delmia-mes-receivers`, `delmia-sim-test`), none since -2026-06-26 — **no production Delmia traffic has hit this method yet.** Every 200 answered +2026-06-26 — **no real Delmia traffic has hit this method yet.** Every 200 answered `Result=false`: mostly `Receiver ready flag not set to true`, one `Script 'ProcessRecipeDownload' … The given key 'side' was not present` from an earlier script revision, and one `MachineCode must end with A (Left) or B (Right)` from a retired left/right variant @@ -277,7 +277,7 @@ README states throughout. configuration database, so a key exists on both nodes the instant it's created and a revoke is cluster-wide immediately. `docker/sync-inbound-api-keys.sh` is deleted; a re-run of this drill needs no file copy before the failover step. Re-measured 2026-09-03 without the - workaround: A/B/E/F `YES`, C/G `NO`, active-central `docker kill` takeover **28 s** (the earlier run measured 27 s with the file copy), victim rejoined as standby 2 s after restart. Prod `wonder-app-vd03` is single-node and unaffected; the QA pair - (`vq03`/`vq04`) and prod still run `Provider=Sqlite` pending a deployment record. + workaround: A/B/E/F `YES`, C/G `NO`, active-central `docker kill` takeover **28 s** (the earlier run measured 27 s with the file copy), victim rejoined as standby 2 s after restart. DEV `wonder-app-vd03` is single-node and unaffected; the QA pair + (`vq03`/`vq04`) and DEV (vd03) still run `Provider=Sqlite` pending a deployment record. -Deferred-work register row 107 records §7 item 1 (`Children["DelmiaRecv"]`) as the prod follow-up. +Deferred-work register row 107 records §7 item 1 (`Children["DelmiaRecv"]`) as the DEV (vd03) follow-up. diff --git a/docs/api-specs/README.md b/docs/api-specs/README.md index 2261dc0d..18d1737e 100644 --- a/docs/api-specs/README.md +++ b/docs/api-specs/README.md @@ -6,4 +6,4 @@ definitions are config-as-data in the central ConfigDb (not source-controlled), durable record; each states which environment it was captured from and when. Legacy/predecessor system specs live in [`../former-api-specs/`](../former-api-specs). -- [`Delmia-Recipe-Download-API.md`](Delmia-Recipe-Download-API.md) — `POST /api/DelmiaRecipeDownload` (DELMIA Apriso recipe-download notification), prod `wonder-app-vd03`, 2026-09-02. +- [`Delmia-Recipe-Download-API.md`](Delmia-Recipe-Download-API.md) — `POST /api/DelmiaRecipeDownload` (DELMIA Apriso recipe-download notification), DEV `wonder-app-vd03`, 2026-09-02. diff --git a/docs/deployment/deployment-records/2026-08-20-wonder-app-vd03.md b/docs/deployment/deployment-records/2026-08-20-wonder-app-vd03.md index bf8d30f1..8401ea1e 100644 --- a/docs/deployment/deployment-records/2026-08-20-wonder-app-vd03.md +++ b/docs/deployment/deployment-records/2026-08-20-wonder-app-vd03.md @@ -1,4 +1,4 @@ -# Deployment record — wonder-app-vd03 (prod single host) — 2026-08-20 +# Deployment record — wonder-app-vd03 (DEV single host) — 2026-08-20 Binary + schema upgrade of the single-host install at `wonder-app-vd03.zmr.zimmer.com` (`E:\ApiInstall\ScadaBridge`; `ScadaBridge-Central` :8085, `ScadaBridge-Site`, `ScadaBridge-LDAP`; diff --git a/docs/deployment/deployment-records/2026-08-20-wonder-app-vq03-vq04.md b/docs/deployment/deployment-records/2026-08-20-wonder-app-vq03-vq04.md index 4f5983be..c7b3c916 100644 --- a/docs/deployment/deployment-records/2026-08-20-wonder-app-vq03-vq04.md +++ b/docs/deployment/deployment-records/2026-08-20-wonder-app-vq03-vq04.md @@ -76,6 +76,6 @@ restart cleared it incidentally. A graceful gateway shutdown takes the same path (pinned by `MxGatewayDataConnectionReconnectTests.EventStreamGracefulCompletion_*`). Deployed to all four QA nodes as a binary-only swap (no migrations, no config): backups `app.bak.20260820T165357` / `cli.bak.20260820T165357` on both boxes; `/health/ready` 200 -Healthy, 2/2 members, site replication connected. **Not yet on prod vd03.** Live gate still to +Healthy, 2/2 members, site replication connected. **Not yet on DEV vd03.** Live gate still to run: close the site's session from the gateway dashboard and confirm the site reconnects (`MxGateway event stream ended without a fault … signalling disconnect` → reconnect → values resume). diff --git a/docs/deployment/deployment-records/2026-08-21-zb-servers-691a2f2f.md b/docs/deployment/deployment-records/2026-08-21-zb-servers-691a2f2f.md index 89dcc1e3..ab418c67 100644 --- a/docs/deployment/deployment-records/2026-08-21-zb-servers-691a2f2f.md +++ b/docs/deployment/deployment-records/2026-08-21-zb-servers-691a2f2f.md @@ -1,4 +1,4 @@ -# Deployment record — QA (wonder-app-vq03/vq04) + prod (wonder-app-vd03) — 2026-08-21 +# Deployment record — QA (wonder-app-vq03/vq04) + DEV (wonder-app-vd03) — 2026-08-21 Binary-only upgrade of all three ZB hosts from `da0bfe2b` to `main` @ **`691a2f2f`** (create-time connection bindings: Create Instance page binds module groups via `IFlatteningPipeline.PreviewAsync`, @@ -12,7 +12,7 @@ schema step and no DB backup. Driven entirely from psbridge on vq04; vq03 and vd | Publish | Host (`app\`, 520 files) + CLI (`cli\`, 223 files) `-c Release -r win-x64 --self-contained`, `scadabridge-691a2f2f.zip` 125 MB, `psbridge push --compress` (SHA-256 `c11bcf15…`). | | Staging | `E:\psrem\releases\691a2f2f\unpacked` on vq04; robocopy `/MIR` to `\\wonder-app-vq03\E$\psrem-stage\691a2f2f` and `\\wonder-app-vd03\E$\psrem-stage\691a2f2f` (1246 files each, ProductVersion verified before the swap). | | QA swap | 06:30:40 — Site pair stopped, Central pair stopped, `app`/`cli` → `app.bak.20260821T063040` / `cli.bak.20260821T063040` on both boxes, stage copied, each box's three `appsettings*.json` copied back, Central pair started, +20 s Site pair started. | -| Prod swap | 06:33:42 — same on vd03 (single host): `app.bak.20260821T063342`, `cli.bak.20260821T063342`. | +| DEV swap | 06:33:42 — same on vd03 (single host): `app.bak.20260821T063342`, `cli.bak.20260821T063342`. | | Preserved | `appsettings*.json`, service registry environment, `central\data`, `site\data` on every host. | ### Verification diff --git a/docs/deployment/deployment-records/2026-08-21-zb-servers-afb27e77.md b/docs/deployment/deployment-records/2026-08-21-zb-servers-afb27e77.md index 8867eabd..431d1684 100644 --- a/docs/deployment/deployment-records/2026-08-21-zb-servers-afb27e77.md +++ b/docs/deployment/deployment-records/2026-08-21-zb-servers-afb27e77.md @@ -1,4 +1,4 @@ -# 2026-08-21 — QA (vq03/vq04) + prod (vd03) → `afb27e77` (template picker) +# 2026-08-21 — QA (vq03/vq04) + DEV (vd03) → `afb27e77` (template picker) **Build:** `main` @ `afb27e77` (= `origin/main`), ProductVersion `1.0.0+afb27e77803b0ef775ce7202df1a04fe36e7889e`. **Delta from the boxes' previous build `691a2f2f`:** Central UI only — Create Instance picks its @@ -16,7 +16,7 @@ confirm verb pinned by test (plan `docs/plans/2026-08-21-template-picker.md`). * 2. **QA 08:29** — Site pair stopped, Central pair stopped (both boxes together), `app`/`cli` renamed to `*.bak.20260821T082917`, staged copy in, each box's three `appsettings*.json` restored, Central pair started (+20 s), Site pair started (+45 s). -3. **Prod 08:32** — same on vd03, backups `*.bak.20260821T083203`. +3. **DEV 08:32** — same on vd03, backups `*.bak.20260821T083203`. ## Verification diff --git a/docs/deployment/deployment-records/2026-09-28-wonder-app-vd03-fe89363b.md b/docs/deployment/deployment-records/2026-09-28-wonder-app-vd03-fe89363b.md index 45daff69..6a20085b 100644 --- a/docs/deployment/deployment-records/2026-09-28-wonder-app-vd03-fe89363b.md +++ b/docs/deployment/deployment-records/2026-09-28-wonder-app-vd03-fe89363b.md @@ -64,8 +64,8 @@ modified except the `SimpleAlarmStatusRequest` stub (below); CvdReactor and its | Object | Id | Notes | |---|---|---| -| Template `TestMESReceiver` | 28 | 26 STATIC attributes with the prod MESReceiver names; ready flags default true | -| Template `TestDelmiaReceiver` | 29 | 10 STATIC attributes with the prod DelmiaReceiver names; ReadyFlag true | +| Template `TestMESReceiver` | 28 | 26 STATIC attributes with the real MESReceiver names; ready flags default true | +| Template `TestDelmiaReceiver` | 29 | 10 STATIC attributes with the real DelmiaReceiver names; ReadyFlag true | | Template `TestMesMachine` | 30 | standalone base (not CvdReactor-derived); composes `LeftMESReceiver`/`RightMESReceiver`/`DelmiaReceiver`; Call scripts `MesMoveIn`, `MesMoveOut`, `ProcessRecipeDownload`, `AlarmStatus`; simulators `SimLeft/RightMoveIn`, `SimLeft/RightMoveOut`, `SimDelmiaRecipe` | | Instance `SBTEST01` | 9 | Dev Site, deployed | | Instance `SBTEST02` | 10 | Dev Site, deployed; alarm override `SensorFault` → MES-reportable | diff --git a/docs/deployment/deployment-records/2026-09-XX-zb-servers-inbound-key-store.md b/docs/deployment/deployment-records/2026-09-XX-zb-servers-inbound-key-store.md index 29be205f..6cd350cb 100644 --- a/docs/deployment/deployment-records/2026-09-XX-zb-servers-inbound-key-store.md +++ b/docs/deployment/deployment-records/2026-09-XX-zb-servers-inbound-key-store.md @@ -1,4 +1,4 @@ -# 2026-09-XX — QA (vq03/vq04) + prod (vd03) → shared inbound API key store (PLANNED) +# 2026-09-XX — QA (vq03/vq04) + DEV (vd03) → shared inbound API key store (PLANNED) **Status: PLANNED — not yet run.** This record is a script for the operator to execute on the ZB servers and then fill in (date/time, log lines, results). Everything below the header table @@ -45,7 +45,7 @@ new secret is distributed. both boxes, staged copy in, each box's three `appsettings*.json` restored **and then edited** to add the `ApiKeyStore.Provider` block above, Central pair started (+20 s), Site pair started (+45 s). -4. **Prod `_____`** — same on vd03 (single node): backup `*.bak.`, same appsettings +4. **DEV `_____`** — same on vd03 (single node): backup `*.bak.`, same appsettings edit, Central started, no site-pair step (`vd03` central is single-node; its site pair is unaffected by this change and follows the same stop/start cadence as any other binary swap on that box). @@ -65,8 +65,8 @@ steps 5–6: | `inbound-api-keys.sqlite.imported` (+ `-wal.imported`/`-shm.imported`) present next to each QA node's data | present on both boxes | `_____` | | `scadabridge security api-key list` against `.2:8085` | matches `.3:8085` (same key set) | `_____` | | `scadabridge security api-key list` against `.3:8085` | matches `.2:8085` | `_____` | -| Prod boot log | `Inbound API key store: imported … KeysInserted=5` for the five existing keys (`mes`, `mes api key`, `delmia-mes-receivers`, `TestKey`, `JOE_TEST_KEY`) | `_____` | -| Prod `security api-key list` before vs. after | unchanged | `_____` | +| DEV boot log | `Inbound API key store: imported … KeysInserted=5` for the five existing keys (`mes`, `mes api key`, `delmia-mes-receivers`, `TestKey`, `JOE_TEST_KEY`) | `_____` | +| DEV `security api-key list` before vs. after | unchanged | `_____` | | Boot warning **absent** on QA post-change | (was present pre-change: `Startup configuration warning: Inbound API keys use the per-node SQLite store on a 2-node central cluster; …`) — confirm it no longer logs | `_____` | | Failover check, no file copy: mint a key via the active node, `curl -X POST /api/DelmiaRecipeDownload` with it aimed at the **non-active** node's port | 200, not 401 (standby relays; both nodes now verify against the same SQL rows even without a relay) | `_____` | | Optional real failover: stop the active node, repeat the same call against the new active | 200 | `_____` | @@ -91,7 +91,7 @@ on SQLite. ## Quarantine Leave the `*.imported` files in place for **30 days** after the change is confirmed stable on -both QA and prod, then delete them (all three: `.imported`, `-wal.imported`, `-shm.imported`) +both QA and DEV, then delete them (all three: `.imported`, `-wal.imported`, `-shm.imported`) from every node's data volume. Do not delete them before that window — they are the only rollback path for keys as of the cutover moment. @@ -107,10 +107,10 @@ rollback path for keys as of the cutover moment. | QA start (Site, both boxes) | `_____` | vq03 + vq04 | `_____` | — | | QA import — first node | `_____` | `_____` | `_____` | `_____` | | QA import — second node | `_____` | `_____` | `_____` | `_____` | -| Prod stop | `_____` | vd03 | `_____` | — | -| Prod swap + appsettings edit | `_____` | vd03 | `_____` | — | -| Prod start | `_____` | vd03 | `_____` | `_____` | -| Prod import | `_____` | vd03 | `_____` | `_____` | +| DEV stop | `_____` | vd03 | `_____` | — | +| DEV swap + appsettings edit | `_____` | vd03 | `_____` | — | +| DEV start | `_____` | vd03 | `_____` | `_____` | +| DEV import | `_____` | vd03 | `_____` | `_____` | | Failover relay check (no file copy) | `_____` | `_____` | `_____` | — | | Optional real failover | `_____` | `_____` | `_____` takeover `_____ s` | — | diff --git a/docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md b/docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md index 734b6297..29cf6d95 100644 --- a/docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md +++ b/docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md @@ -23,7 +23,7 @@ still fetches over HTTP, so that half of the original fix stands. **The successor ceiling is different in kind.** The gRPC sync stream has a 4 MB default receive limit. Until LocalDb 0.2.0 the library batched by ROW COUNT only (`LocalDb:Replication:MaxBatchSize`, -default 500): a ~70 KB `config_json` — the largest measured in production — times 500 rows is +default 500): a ~70 KB `config_json` — the largest measured on the DEV host (vd03) — times 500 rows is ~35 MB, which would exceed the limit, so the rig pinned `MaxBatchSize` to **16** (~1.1 MB worst case). See the Phase 2 plan (D6) and `docs/plans/2026-07-19-localdb-phase2-live-gate.md`. diff --git a/docs/plans/2026-05-31-folder-repo-rename-scadabridge-design.md b/docs/plans/2026-05-31-folder-repo-rename-scadabridge-design.md index 023cad7b..925f452d 100644 --- a/docs/plans/2026-05-31-folder-repo-rename-scadabridge-design.md +++ b/docs/plans/2026-05-31-folder-repo-rename-scadabridge-design.md @@ -85,7 +85,7 @@ The existing `~/Desktop/ScadaBridge` (non-git, older project containing `akka.md ## Section 3a — Git-ignored `deploy/` tree (added during implementation) -`/deploy/` is git-ignored, so the `git grep`-based scrub script (and its completeness gate) structurally could not reach it — a gap surfaced by code review. The `deploy/wonder-app-vd03/` production artifacts (`install.ps1`, `appsettings.Central.json`, `appsettings.Site.json`, `RUNBOOK.md`, `ldap/glauth.cfg`) still held old references that would break against the renamed binary: `SCADALINK_*` env vars, `ScadaLink__*` config keys, the `"ScadaLink"` config-section root, `ScadaLink.Host.exe` (now `ZB.MOM.WW.ScadaBridge.Host.exe`), `scadalink.exe` (CLI `AssemblyName` is `scadabridge`), and `akka.tcp://scadalink@…` seed URIs (the actor system name in code is `scadabridge`). +`/deploy/` is git-ignored, so the `git grep`-based scrub script (and its completeness gate) structurally could not reach it — a gap surfaced by code review. The `deploy/wonder-app-vd03/` DEV artifacts (`install.ps1`, `appsettings.Central.json`, `appsettings.Site.json`, `RUNBOOK.md`, `ldap/glauth.cfg`) still held old references that would break against the renamed binary: `SCADALINK_*` env vars, `ScadaLink__*` config keys, the `"ScadaLink"` config-section root, `ScadaLink.Host.exe` (now `ZB.MOM.WW.ScadaBridge.Host.exe`), `scadalink.exe` (CLI `AssemblyName` is `scadabridge`), and `akka.tcp://scadalink@…` seed URIs (the actor system name in code is `scadabridge`). Per user decision, this tree was scrubbed by hand with the **same** substitution set, **including** the internal LDAP directory domain (`dc=scadalink,dc=local` → `dc=scadabridge,dc=local` and `@scadalink.local` → `@scadabridge.local`) applied in lockstep across both `appsettings` and `glauth.cfg` so LDAP login stays consistent. Because `deploy/` is git-ignored, these edits are **local-only / not committed** — they fix the on-disk artifacts for the next `wonder-app-vd03` deploy. Verified clean with `grep -rniI scadalink deploy/`. diff --git a/docs/plans/2026-06-16-ipsen-mes-movein-design.md b/docs/plans/2026-06-16-ipsen-mes-movein-design.md index 2a6c047a..a1eaaf9c 100644 --- a/docs/plans/2026-06-16-ipsen-mes-movein-design.md +++ b/docs/plans/2026-06-16-ipsen-mes-movein-design.md @@ -5,7 +5,7 @@ **Scope:** ScadaBridge — inbound API `POST /api/IpsenMESMoveIn`, a small inbound-script DB capability, and a new template script on the reactor template that writes a MoveIn to the correct MES-receiver child. -**Target environment:** production `wonder-app-vd03` (Central + Site, ConfigDb `ScadaBridge` +**Target environment:** DEV `wonder-app-vd03` (Central + Site, ConfigDb `ScadaBridge` on `wonder-app-vd03,1433`; reactors live on Site 2). BTDB = db-connection id 1 (`dbo.Machine` on `wonder-sql-vd03/BT`). @@ -96,7 +96,7 @@ receiver is ready to accept a MoveIn. > `QuerySingle`/`Query` sketched in the original draft above, and **writes are supported** > (InboundAPI-026) rather than read-only. Scripts MUST call `await Database.QuerySingleAsync(...)`. > Three deployed methods (`IpsenMESMoveIn`, `MesMoveIn`, `MesMoveOut`) were authored from the old -> `Database.QuerySingle` spelling and failed Roslyn compilation in production until corrected +> `Database.QuerySingle` spelling and failed Roslyn compilation on DEV (vd03) until corrected > (2026-06-25). The authoritative surface lives in `docs/requirements/Component-InboundAPI.md`. ### B. Rewrite inbound `/api/IpsenMESMoveIn` script (data, via management API) diff --git a/docs/plans/2026-06-16-ipsen-mes-movein.md b/docs/plans/2026-06-16-ipsen-mes-movein.md index cb867522..6794c8b1 100644 --- a/docs/plans/2026-06-16-ipsen-mes-movein.md +++ b/docs/plans/2026-06-16-ipsen-mes-movein.md @@ -260,7 +260,7 @@ Add `using Microsoft.Extensions.DependencyInjection;` at the top. No `ScriptOpti ## Task 5: Deploy ScadaBridge Central (Component A) to wonder-app-vd03 -**Classification:** high-risk (production deploy) +**Classification:** high-risk (live DEV-host deploy) **Estimated implement time:** ~6 min **Parallelizable with:** none (depends on Tasks 3–4) **Requires explicit go-ahead before running.** @@ -278,7 +278,7 @@ Add `using Microsoft.Extensions.DependencyInjection;` at the top. No `ScriptOpti ## Task 6: Apply Component B — update the inbound `IpsenMESMoveIn` script -**Classification:** high-risk (production data) +**Classification:** high-risk (live DEV-host data) **Estimated implement time:** ~3 min **Parallelizable with:** Task 7 (different surfaces; both depend on Task 5) **Requires explicit go-ahead.** @@ -335,7 +335,7 @@ Preserve the current `ParameterDefinitions`/`ReturnDefinition` (read them first; ## Task 7: Apply Component C — `IpsenMoveIn` template script on T1 -**Classification:** high-risk (production data) +**Classification:** high-risk (live DEV-host data) **Estimated implement time:** ~4 min **Parallelizable with:** Task 6 (depends on Task 5) **Requires explicit go-ahead.** @@ -389,7 +389,7 @@ return new { WasSuccessful = true, ErrorText = "", BatchID = batchId }; ## Task 8: On-box end-to-end verification -**Classification:** standard (production verify) +**Classification:** standard (live DEV-host verify) **Estimated implement time:** ~5 min **Parallelizable with:** none (depends on Tasks 6, 7) diff --git a/docs/plans/2026-06-17-waitasync-compile-surface-mirror-gaps.md b/docs/plans/2026-06-17-waitasync-compile-surface-mirror-gaps.md index 61564559..bd219bc9 100644 --- a/docs/plans/2026-06-17-waitasync-compile-surface-mirror-gaps.md +++ b/docs/plans/2026-06-17-waitasync-compile-surface-mirror-gaps.md @@ -16,7 +16,7 @@ updated to mirror the new methods. This was hit live: deploying a template scrip > `'ScriptCompileSurface.CompileAttributeAccessor' does not contain a definition for 'WaitAsync'` -…which forced a bounded-poll-loop workaround in the DELMIA/MES receiver scripts on prod. Once this plan +…which forced a bounded-poll-loop workaround in the DELMIA/MES receiver scripts on DEV (vd03). Once this plan ships, those scripts can be switched back to the cheaper event-driven `WaitAsync`. **Tech Stack:** C#/.NET 10, Roslyn (`Microsoft.CodeAnalysis.CSharp.Scripting`), xUnit. No new dependencies. diff --git a/docs/plans/2026-07-08-deferred-work-register.md b/docs/plans/2026-07-08-deferred-work-register.md index 5acdff88..0935a2a0 100644 --- a/docs/plans/2026-07-08-deferred-work-register.md +++ b/docs/plans/2026-07-08-deferred-work-register.md @@ -106,8 +106,8 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round | 104 | ~~**Existing site LocalDb files stay `auto_vacuum=NONE`; `PRAGMA incremental_vacuum` after a cap trim is a no-op on them**~~ **RESOLVED 2026-08-20 — recipe shipped and applied to the rig.** New `docker/vacuum-site-localdb.sh` (stop both nodes of a pair together → `PRAGMA auto_vacuum=INCREMENTAL; VACUUM` each volume with integrity checks before/after → start both; `DRY_RUN=1` reports) plus the topology-guide section *Site LocalDb files created before 2026-08-20 need a one-off offline VACUUM*. Run on all three rig pairs: all six files `auto_vacuum 0 → 2`, `integrity_check=ok`, ~11 % smaller (site-a 20.3 → 18.1 MB), site-a replication back `Connected=true`/backlog 0 with no watermark or schema change — the VACUUM was replication-invisible as predicted (ledger keys by declared PK; oplog `seq` is an `INTEGER PRIMARY KEY AUTOINCREMENT`). The same maintenance surfaced the unrelated row 106. Original: | Arch-review 2026-08-19 P4-6 (Low-wave T4, `47f03fe6`) | `auto_vacuum` is only honoured on a table-less database, and `LocalDbSchema.EnsureCreated` had always created LocalDb's bookkeeping tables before `SiteEventLogSchema.Apply` ran — so the pragma had never once taken effect on any real file. The fix (`SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile`, run before `AddZbLocalDb`'s lazy singleton constructs `SqliteLocalDb`) only helps **fresh** files; an unconditional `VACUUM` against a live ~1 GB replicated file was explicitly rejected (long exclusive lock on the active node; the rewrite is CDC-invisible but blocks the sync pump). Every node on the docker rig today predates the fix, so freed `site_events` pages there are reused, not returned to the OS — the storage cap (row 81) still bounds the data volume, only the file size stops shrinking. | Operator-scheduled `VACUUM` in a maintenance window (both nodes of a pair stopped together), or when a site's file is next recreated. | | 105 | ~~**Design constraint (not a defect): a LocalDb-replicated table must never be the parent of a row-mutating `ON DELETE` foreign key (`CASCADE`/`SET NULL`/`SET DEFAULT`) from another table**~~ **PINNED 2026-08-20.** Enforcement moved from the first replicated boot to the build: `SiteLocalDbReplicatedTableFkGuardTests` (Host.Tests) builds the real site schema via `OnReady` on an unreplicated config (so 0.3.3's own guard can't mask which FK is at fault), walks `SiteLocalDbSetup.ReplicatedTables` (now `internal`), asserts each exists and has no inbound row-mutating `ON DELETE` FK with the same `sqlite_master JOIN pragma_foreign_key_list` query LocalDb uses, and carries a negative control proving the checker finds a cascading child. The constraint itself stands — the row is kept as its record. Original: | Arch-review 2026-08-19 P4-7 → LocalDb 0.3.3 (scadaproj `122940c` + review follow-up `9663279`, Low-wave T5) | `LwwApplier` upserts with `INSERT OR REPLACE` under `foreign_keys=ON`, so the implicit delete fires the child action while CDC capture is suppressed — the child rows change on the applying node only and the pair diverges silently. 0.3.3 makes `RegisterReplicated` refuse such a parent at registration (clear exception naming the child FK), which is a BOOT failure for the site node, so this is enforced, not advisory. None of the ten `SiteLocalDbSetup.ReplicatedTables` has any FK today (zero `REFERENCES` on the live rig file). Model cross-table cleanup as explicit application deletes on each table (as the 3-table deploy cascade already does), never as SQLite FK actions. | Whenever a schema change adds a foreign key between site tables, or an eleventh table joins `ReplicatedTables`. | | 106 | ~~**LocalDb: a converged pair re-streams a FULL snapshot on EVERY reconnect once the sender's oplog has been ack-pruned to empty and the receiver's watermark is 0**~~ **RESOLVED 2026-08-20 — LocalDb 0.3.4 (scadaproj `75c1897`), published and pinned.** `SnapshotStreamer.SendAsync` now stamps `as_of_seq = max(MAX(seq), last_acked_seq)`; pinned by `WipedPeer_SnapshotLiftsItsWatermark_SoTheNextReconnectSendsNoSecondSnapshot` (fails on 0.3.3 exactly as the rig did: watermark 0, expected 3). Live proof on the rig: the first restart on 0.3.4 sent one last full snapshot stamped `as_of_seq 149` (was 0) and node-a's `last_applied_remote_seq` went 0 → 149; a second restart of the pair exchanged **zero** snapshots, `Connected=true`, backlog 0. Original: | Found 2026-08-20 while verifying row 104 on the rig (site-a-a received the same 127 k-row full snapshot at both the 13:09 deploy restart and the 13:25 VACUUM restart) | Sender-side defect in `SnapshotStreamer.SendAsync`: `as_of_seq = MAX(seq)` of the oplog, which is **0** once the pair has converged and ack-pruning emptied it. The receiver's `SetLastAppliedRemoteSeqAsync` is MAX-guarded (no regression — good), but a peer whose watermark is genuinely 0 (site-a-a was rebuilt after the 2026-08-15 corruption; any wiped/re-imaged node is the same) can never be lifted by an `as_of_seq 0` snapshot, so the 0.1.2 empty-oplog horizon rule (`streamableFrom = last_acked_seq + 1`) sees `0 < 149` as a gap on every handshake and re-streams everything. Rig state: node-b `last_acked_seq=149`, oplog empty; node-a `last_applied_remote_seq=0`. Not a divergence (LWW absorbs the re-delivered rows) but a cost/availability bug — ~1 s and 127 k rows here, minutes and a full-file read on a 1 GB production pair, on every reconnect, until the sender happens to write a new row. Fix is sender-side and wire-compatible: stamp `as_of_seq = max(MAX(seq), last_acked_seq)` (a full snapshot taken now covers every delta ever pruned below `last_acked_seq`), so the receiver's MAX-guarded watermark lifts to 149 and the loop ends; add a pinning test (converged+pruned sender, zero-watermark receiver → exactly one snapshot across two handshakes). Ships as LocalDb 0.3.4. | Next LocalDb release (0.3.4) — small, well-understood, and worth doing before any production pair is rebuilt from its peer. | -| 107 | **→ #52** **Prod `CvdReactor.ProcessRecipeDownload` still addresses `Children["DelmiaRecv"]` after the 2026-06-27 `DelmiaRecv` → `DelmiaReceiver` composition rename** — inferred from the deployed script (`docs/api-specs/Delmia-Recipe-Download-API.md` §7 item 1): `ChildrenAccessor` concatenates the name into a path, an unknown path yields null, so every call answers `Receiver ready flag not set to true` regardless of the real flag. No call has hit prod since the rename, so this is inferred, not observed. The docker rig's stand-in (`docker/seed-delmia-sim.sh`) uses `Children["DelmiaReceiver"]` and the same handshake answers correctly, which is the working reference. | Found 2026-09-02 while capturing the as-deployed spec; QA copy unread (credentials) | Separate change with its own deployment record: `scadabridge template script update` on the `CvdReactor` template + redeploy Z28061/Z28062/Z28061Sim, then a live `WWNotifier` call. | Before the first production Delmia recipe download. | -| 108 | ~~**Inbound API keys are a PER-NODE SQLite store — a central failover invalidates every inbound client's key**~~ **RESOLVED 2026-09-02 — `ZB.MOM.WW.Auth` 0.3.0 (new `ZB.MOM.WW.Auth.ApiKeys.SqlServer` package) + ScadaBridge `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer`.** A central pair now shares one row set: both nodes' key stores, admin stores, and audit stores point at the same `zbauth` schema in the configuration database (connection string defaults to `ScadaBridge:Database:ConfigurationDb` — never a `${secret:}` reference, to avoid a circular dependency on the SQL-backed secrets store), so create/revoke/rotate/scope changes are cluster-wide immediately instead of per-node. Each node's existing SQLite file is imported once at boot (insert-where-absent, so both nodes importing is safe and the result is the union) and renamed `*.imported`. `Provider` defaults to `Sqlite` (unchanged behavior for single-node installs, MxAccessGateway, HistorianGateway); a clustered Central left on `Sqlite` now gets a boot-time `StartupValidator` Warning naming this design instead of silently keeping the gap. Both docker rigs (`docker/`, `docker-env2/`) run `Provider=SqlServer` on both centrals; `docker/sync-inbound-api-keys.sh` is deleted, superseded by the shared store. Rig proof 2026-09-03 (`docker/`, both centrals redeployed on `Provider=SqlServer`): both nodes imported their SQLite file at the same second with no PK race — central-a `1 key / 264 audit rows inserted`, central-b `0 / 26 inserted, 1 / 264 already present` (the union), files renamed `*.imported`; a key minted via `:9001` answered `200` via `:9002`, `--enabled false` via `:9001` made both nodes answer `401` and `:9002`'s own list showed `enabled=False`, re-enable via `:9002` restored `200` on `:9001`, delete via `:9002` made `:9001` answer `401`; `verify-delmia-notifier.sh` with a freshly minted key and NO file copy: A/B/E/F `YES`, C/G `NO`, active-central `docker kill` takeover **28 s**, rejoin as standby in 2 s. `docker-env2` (second pair, `ScadaBridgeConfig2`): both centrals booted, `zbauth` v1 created, empty files imported. QA (`vq03`/`vq04`) and prod (`vd03`) still run `Provider=Sqlite` — their move is a planned deployment record (`docs/deployment/deployment-records/2026-09-XX-zb-servers-inbound-key-store.md`), not yet applied. Design: `docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`. Original: | `docs/api-specs/Delmia-Recipe-Download-API.md` §8; `docker/sync-inbound-api-keys.sh` header | Fix is an architecture decision, not a patch: move the store to the shared SQL Server (as Secrets 0.5.0 did for its central store) or replicate the file; either way revocation must become cluster-wide. Needs the `ZB.MOM.WW.Auth.ApiKeys` library (scadaproj), not just this repo. | Before any 2-node central goes live with inbound API clients; before the next arch-review wave. | +| 107 | **→ #52** **DEV (vd03) `CvdReactor.ProcessRecipeDownload` still addresses `Children["DelmiaRecv"]` after the 2026-06-27 `DelmiaRecv` → `DelmiaReceiver` composition rename** — inferred from the deployed script (`docs/api-specs/Delmia-Recipe-Download-API.md` §7 item 1): `ChildrenAccessor` concatenates the name into a path, an unknown path yields null, so every call answers `Receiver ready flag not set to true` regardless of the real flag. No call has hit DEV (vd03) since the rename, so this is inferred, not observed. The docker rig's stand-in (`docker/seed-delmia-sim.sh`) uses `Children["DelmiaReceiver"]` and the same handshake answers correctly, which is the working reference. | Found 2026-09-02 while capturing the as-deployed spec; QA copy unread (credentials) | Separate change with its own deployment record: `scadabridge template script update` on the `CvdReactor` template + redeploy Z28061/Z28062/Z28061Sim, then a live `WWNotifier` call. | Before the first production Delmia recipe download. | +| 108 | ~~**Inbound API keys are a PER-NODE SQLite store — a central failover invalidates every inbound client's key**~~ **RESOLVED 2026-09-02 — `ZB.MOM.WW.Auth` 0.3.0 (new `ZB.MOM.WW.Auth.ApiKeys.SqlServer` package) + ScadaBridge `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer`.** A central pair now shares one row set: both nodes' key stores, admin stores, and audit stores point at the same `zbauth` schema in the configuration database (connection string defaults to `ScadaBridge:Database:ConfigurationDb` — never a `${secret:}` reference, to avoid a circular dependency on the SQL-backed secrets store), so create/revoke/rotate/scope changes are cluster-wide immediately instead of per-node. Each node's existing SQLite file is imported once at boot (insert-where-absent, so both nodes importing is safe and the result is the union) and renamed `*.imported`. `Provider` defaults to `Sqlite` (unchanged behavior for single-node installs, MxAccessGateway, HistorianGateway); a clustered Central left on `Sqlite` now gets a boot-time `StartupValidator` Warning naming this design instead of silently keeping the gap. Both docker rigs (`docker/`, `docker-env2/`) run `Provider=SqlServer` on both centrals; `docker/sync-inbound-api-keys.sh` is deleted, superseded by the shared store. Rig proof 2026-09-03 (`docker/`, both centrals redeployed on `Provider=SqlServer`): both nodes imported their SQLite file at the same second with no PK race — central-a `1 key / 264 audit rows inserted`, central-b `0 / 26 inserted, 1 / 264 already present` (the union), files renamed `*.imported`; a key minted via `:9001` answered `200` via `:9002`, `--enabled false` via `:9001` made both nodes answer `401` and `:9002`'s own list showed `enabled=False`, re-enable via `:9002` restored `200` on `:9001`, delete via `:9002` made `:9001` answer `401`; `verify-delmia-notifier.sh` with a freshly minted key and NO file copy: A/B/E/F `YES`, C/G `NO`, active-central `docker kill` takeover **28 s**, rejoin as standby in 2 s. `docker-env2` (second pair, `ScadaBridgeConfig2`): both centrals booted, `zbauth` v1 created, empty files imported. QA (`vq03`/`vq04`) and DEV (`vd03`) still run `Provider=Sqlite` — their move is a planned deployment record (`docs/deployment/deployment-records/2026-09-XX-zb-servers-inbound-key-store.md`), not yet applied. Design: `docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`. Original: | `docs/api-specs/Delmia-Recipe-Download-API.md` §8; `docker/sync-inbound-api-keys.sh` header | Fix is an architecture decision, not a patch: move the store to the shared SQL Server (as Secrets 0.5.0 did for its central store) or replicate the file; either way revocation must become cluster-wide. Needs the `ZB.MOM.WW.Auth.ApiKeys` library (scadaproj), not just this repo. | Before any 2-node central goes live with inbound API clients; before the next arch-review wave. | | 109 | **→ scadaproj#18** **`ZB.MOM.WW.Secrets.Replicator.SqlServer`'s store migrator retries its DDL with a flat `100 * attempt` backoff and no jitter — a latent concurrent-boot flake, open in Secrets** *(Low)*. Porting that migrator's retry loop verbatim for the Auth 0.3.0 SQL Server API-key store (register row 108) reproduced an intermittent exhaustion — about once in sixty 8-way concurrent-boot trials the five flat-backoff retries ran out before a DDL/lock conflict cleared; adding jitter to the Auth port's backoff took it to 0 failures in 180 trials. Secrets' own migrator has no concurrent-migrate test, so the same defect is presumed still open there — it just hasn't been exercised at 8-way concurrency, and a ScadaBridge central pair is only ever 2-way, so this is unlikely to bite in practice. | Found 2026-09-02 while implementing `docs/plans/2026-09-02-inbound-api-key-shared-store.md` (Auth 0.3.0 `SqlServerApiKeyStoreMigrator`) | Add jitter to `SqlServerSecretsStoreMigrator`'s backoff in `~/Desktop/scadaproj/ZB.MOM.WW.Secrets` (mirror the Auth fix) and add a `Migrate_Concurrently_FromEightTasks_AllSucceed`-shaped live test to catch a regression. Lives entirely in scadaproj, not this repo. | Next time Secrets' SQL Server migrator is touched, or if a wider-than-2-way concurrent boot of a Secrets-backed cluster is ever planned. | | 110 | **→ #53** **Flow Debugger follow-up: the *Vertical trace* list toggle (mockup Direction B)** *(Low)*. v1 ships the Direction A swimlane only; the mockup's Direction B — the same steps as a vertical, filterable list with the lane as a column — was kept as an alternate for long flows and screen readers. Additive: a list view over the existing `FlowGraph`. ~~and console capture on `ScriptRun`~~ **The console-capture half is RESOLVED 2026-09-09 (plan `docs/plans/2026-09-09-flow-debugger-step-logs.md`)** — and deliberately not as this row sketched it. `Extra.console` on the `ScriptRun` row was rejected (the 8 KB payload cap would silently drop lines, and helper-step attribution would need every helper to buffer); shipped instead is a leveled `Log` global on all four script surfaces plus one `ScriptTrace.ScriptLog` audit row per line, folded onto the step that produced it, with the failing step's full exception on `errorDetail`. `Console.Out` capture on a site node is recorded as permanently out of scope in the design's §9: `Console.Out` is process-global and shared by every concurrent run, so per-run capture is unsound. | Design `docs/plans/2026-09-05-flow-debugger-design.md` §11; plan `2026-09-05-flow-debugger.md` Task 23 | The list view was not asked for by an operator; the swimlane + Timing/Writes/Log tabs cover the acceptance flow (Delmia recipe download, rig steps H and J). | First operator request for a list view. | | 111 | ~~**Flow Debugger outbound nodes: no Retry/Discard on a parked node, and no `?correlationId=` deep link from the node to the Site Calls / Notification report pages** *(Low)*~~ **RESOLVED 2026-09-06 (plan `docs/plans/2026-09-06-flow-debugger-followups.md`).** `FlowStep` now carries the row's `CorrelationId`; the detail panel deep-links to `/site-calls/report?correlationId=` / `/notifications/report?correlationId=` (both pages accept it through additive per-id query filters — `SiteCallQueryRequest.TrackedOperationIdFilter`, `NotificationOutboxQueryRequest.NotificationIdFilter` — pre-filter to the row and auto-open its detail), and a `Parked` node offers Retry / Discard in place through `IFlowDebugService.RetryOutboundAsync` / `DiscardOutboundAsync`, which reuse the report pages' relay, site-scope re-check and confirm wording, after which the page re-polls Replay until the node leaves Parked. Per-attempt request/response diff is still not shown (the lifecycle rows carry the error summary only). Found on the way: the prediction circuit deadlock (`4769e192`). | Plan 2026-09-06 "Out of scope"; `FlowStepDetail.razor` parked hint | The Site Calls page owns the retry semantics and its audit rows; duplicating the action on the debugger node would need the same Deployer gate + confirm flow. | First operator who reaches a parked node from the debugger and asks for the button. | @@ -119,8 +119,8 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round | 117 | ~~**Flow Debugger site trace pull: `FlowDebugRunner.WriteAndPollAsync` fetches ONE bounded page with a fixed `since`**~~ **RESOLVED 2026-09-09 (`9ee402bb`, arch-review 2026-09-09 MG-6; closed in this register 2026-10-21 — found stale by the 2026-10-21 system review).** `WriteAndPollAsync` now advances a `(SinceUtc, AfterEventId)` keyset cursor between polls and only treats a page as progress when the cursor moved, so a chatty run can no longer push a spawned run past the page end. Original description kept for traceability: *(Low)*. The write-and-poll loop asks the site for `FlowDebugRunner.PullTraceEventsMax` rows per poll, always `since` the run start, and the site serves `ORDER BY OccurredAtUtc ASC LIMIT` — so the page is the OLDEST rows in the window, never a continuation of the previous one. Step logs made that reachable: one traced run can now emit up to 201 `ScriptLog` rows, so a handful of chatty runs inside the window can fill the page and push a later spawned run's `ScriptRun` row past the end of it, at which point the loop never sees the awaited run and reports a false `NO_TRIGGER_WITHIN_WINDOW` for a run that did happen. Final review I1 raised the cap 500 -> 2 000 (the site's own ceiling, `SiteStreamGrpcServer.MaxTraceEventsPerPull`), which makes it ~10x harder but does not remove it. The real fix is to PAGE — advance `since` to the last-seen `OccurredAtUtc` between polls (keeping the already-known ids so nothing is double-counted) — or to exclude `ScriptLog` from the awaited-run poll entirely and pull the log rows once, after the run has settled. | Final review of `docs/plans/2026-09-09-flow-debugger-step-logs.md` (I1); `src/ZB.MOM.WW.ScadaBridge.ManagementService/FlowDebug/FlowDebugRunner.cs` `WriteAndPollAsync` | 2 000 is the site's hard ceiling, so the headroom cannot be bought again with a bigger number; paging needs the poll loop to track a cursor and the awaited-run predicate to survive a partial view, which is more than the raise was worth at review time. | The first `NO_TRIGGER_WITHIN_WINDOW` reported for a run whose Replay of the same root DOES show the spawned `ScriptRun`. | | 118 | **Pre-existing bUnit flake: `CentralUI.Tests.Pages.NotificationReportDetailModalTests.CloseButton_DismissesModal` (~10 %)** *(Low)*. Fails intermittently with `UnknownEventHandlerIdException` — the test resolves the close button once and clicks it after a re-render has already retired that element's handler id. Not caused by the step-logs work: the file is untouched since `40944a0d`. The fix is a one-liner — re-`Find` the button immediately before the click (the pattern the other modal tests already use) rather than reusing an element captured earlier in the test. | Observed during the step-logs full-suite runs, 2026-09-09 | It is a test-only race with a known cause and a known fix; it was out of scope for a remediation pass whose rule was to touch only the files under review. | Next time `CentralUI.Tests` is worked on, or the first time the flake blocks a gate run. | | 68 | ~~**`deploy.sh` run from a git worktree silently rehomes every node's bind-mount data dir**~~ **RESOLVED 2026-08-15 (guard added in the same close-out).** The compose files' bind mounts are relative (`./site-a-node-a/data`), so a deploy from a worktree checkout mounts the worktree's copies — the rig keeps running and looks healthy, but its SQLite stores split from the main checkout's and everything written there is orphaned by the next deploy from main. Bit the rows-46–53 close-out live: the seeding agent's `rig-probe-1` deploy persisted into worktree-homed dirs and "vanished" after the orchestrator's redeploy (no tombstone, no product bug — LWW/tombstone forensics ruled out deletion; the row sat in `.claude/worktrees/…/docker/site-a-node-a/data/site-localdb.db` the whole time). Both `docker/deploy.sh` and `docker-env2/deploy.sh` now refuse to run from a `.claude/worktrees/` path (override: `SCADABRIDGE_ALLOW_WORKTREE_DEPLOY=1`). | Found during the rows-46–53 close-out | Closed 2026-08-15. | — | -| 119 | **→ #36** **Port the QA/prod MES receiver scripts onto the reset-aware `WriteBatchAndWaitAsync` overload (Gitea #36)** *(Medium)*. Template scripts CvdReactor MesMoveIn (13) / MesMoveOut (14) / MesSampleData (36) and base MESReceiver MoveIn/MoveOut (15, 16, 25, 26, 37, 38) still reset `CompleteFlag` with a plain write and rely on `Task.Delay(1000)`; QA carries a config-only `WaitAsync(false, 10 s)` patch on 13/14 (deployment record 110). Rewrite: `var hs = await rx.Attributes.WriteBatchAndWaitAsync(values, "Flag", true, "CompleteFlag", true, 25 s, "CompleteFlag", false, 10 s); if (!hs.Matched) return new { WasSuccessful=false, ErrorText = hs.ResetFailed ? "Timeout waiting for CompleteFlag to reset" : "Timeout waiting for to complete", BatchID=0 };` and delete the delay. Requires the #36 build on the target central + sites first (the overload does not exist on older sites → deploy-gate compile failure, by design). | Gitea #36, design `2026-09-10-handshake-reset-echo-design.md` §8 | Repo-only scope was the user's decision for the #36 session; the script bodies live only in the QA/prod databases. | Next QA deployment of a build ≥ this change (write a deployment record; then prod). | -| 120 | **→ #58** **Data-sourced SCALAR writes reach the OPC UA server as String Variants — every non-String node rejects them with BadTypeMismatch (0x80740000)** *(Medium)*. `InstanceActor.HandleSetDataAttribute` / `HandleWriteAttributeBatch` decode only List values before the DCL write ("Scalars are unchanged"), so the codec-encoded string (`"True"`, `"42.5"`) reaches `RealOpcUaClient.WriteValuesAsync`, which wraps it in `new Variant(value)` — a String variant sent to a Boolean/Double node. Found live 2026-09-10 running `docker/seed-handshake-demo.sh` against opc-plc `Motor.Running`/`Pump.Running`/`Motor.Speed` (`SetAttribute('Complete') failed: OPC UA write failed with status: 0x80740000`); the seed was retargeted onto the rig's only two writable String nodes. The MxGateway path is unaffected in practice: prod's Delmia/MES handshakes write the same strings through the gateway today. Fix candidates: (a) decode scalars by declared `DataType` in the InstanceActor before the DCL write — also changes what MxGateway receives, so it needs a gateway-side compatibility check; (b) adapter-local: `RealOpcUaClient` coerces to the node's DataType (it already resolves `DataTypeIds` in `EnrichVariableTypeInfoAsync`), leaving the MxGateway wire untouched — preferred. Pin with a `RealOpcUaClientArrayWriteTests`-style test for a Boolean scalar. | Found live 2026-09-10 during the #36 rig gate | Out of scope for #36 (handshake ordering); touches the production write path and needs the MxGateway compatibility question answered first. | Before any OPC UA-bound template relies on script writes to non-String nodes (none does today — until this gate the rig's script writes to OPC UA were static-only). | +| 119 | **→ #36** **Port the QA/DEV MES receiver scripts onto the reset-aware `WriteBatchAndWaitAsync` overload (Gitea #36)** *(Medium)*. Template scripts CvdReactor MesMoveIn (13) / MesMoveOut (14) / MesSampleData (36) and base MESReceiver MoveIn/MoveOut (15, 16, 25, 26, 37, 38) still reset `CompleteFlag` with a plain write and rely on `Task.Delay(1000)`; QA carries a config-only `WaitAsync(false, 10 s)` patch on 13/14 (deployment record 110). Rewrite: `var hs = await rx.Attributes.WriteBatchAndWaitAsync(values, "Flag", true, "CompleteFlag", true, 25 s, "CompleteFlag", false, 10 s); if (!hs.Matched) return new { WasSuccessful=false, ErrorText = hs.ResetFailed ? "Timeout waiting for CompleteFlag to reset" : "Timeout waiting for to complete", BatchID=0 };` and delete the delay. Requires the #36 build on the target central + sites first (the overload does not exist on older sites → deploy-gate compile failure, by design). | Gitea #36, design `2026-09-10-handshake-reset-echo-design.md` §8 | Repo-only scope was the user's decision for the #36 session; the script bodies live only in the QA/DEV databases. | Next QA deployment of a build ≥ this change (write a deployment record; then DEV). | +| 120 | **→ #58** **Data-sourced SCALAR writes reach the OPC UA server as String Variants — every non-String node rejects them with BadTypeMismatch (0x80740000)** *(Medium)*. `InstanceActor.HandleSetDataAttribute` / `HandleWriteAttributeBatch` decode only List values before the DCL write ("Scalars are unchanged"), so the codec-encoded string (`"True"`, `"42.5"`) reaches `RealOpcUaClient.WriteValuesAsync`, which wraps it in `new Variant(value)` — a String variant sent to a Boolean/Double node. Found live 2026-09-10 running `docker/seed-handshake-demo.sh` against opc-plc `Motor.Running`/`Pump.Running`/`Motor.Speed` (`SetAttribute('Complete') failed: OPC UA write failed with status: 0x80740000`); the seed was retargeted onto the rig's only two writable String nodes. The MxGateway path is unaffected in practice: DEV's (vd03) Delmia/MES handshakes write the same strings through the gateway today. Fix candidates: (a) decode scalars by declared `DataType` in the InstanceActor before the DCL write — also changes what MxGateway receives, so it needs a gateway-side compatibility check; (b) adapter-local: `RealOpcUaClient` coerces to the node's DataType (it already resolves `DataTypeIds` in `EnrichVariableTypeInfoAsync`), leaving the MxGateway wire untouched — preferred. Pin with a `RealOpcUaClientArrayWriteTests`-style test for a Boolean scalar. | Found live 2026-09-10 during the #36 rig gate | Out of scope for #36 (handshake ordering); touches the production write path and needs the MxGateway compatibility question answered first. | Before any OPC UA-bound template relies on script writes to non-String nodes (none does today — until this gate the rig's script writes to OPC UA were static-only). | | 121 | **→ #59** **No source-level `IsMesReportable` on native alarm source bindings** *(Low)*. `TemplateNativeAlarmSource` / `InstanceNativeAlarmSourceOverride` carry no per-alarm MES-relevance flag — "every condition under this binding is reportable" is not expressible, only the 900–999 severity-band convention. | `docs/plans/2026-09-11-mes-reportable-alarm-flag-design.md` | Native alarms have no per-alarm config row, and the design chose not to invent one for this slice. | A native-source machine needs per-alarm MES relevance beyond the severity band. | | 122 | **→ #60** **`IsMesReportable` does not ride the wire** *(Low)*. `AlarmStateUpdate` (vendored proto) and `DebugAlarmStateDto` have no `IsMesReportable` field, so central-side surfaces (Debug View live stream, Alarm Summary) always see `false` even though the site-side snapshot stamps the real value. Adding it is additive-only: `AlarmStateUpdate` field 25, `DebugAlarmStateDto` field 20, via a manual `regen-proto.sh` toggle-build-copy-untoggle. | `docs/plans/2026-09-11-mes-reportable-alarm-flag-design.md` | No consumer needed it in this slice — the flag exists to serve `Alarms.MesReportableAsync()` for scripts, not a central UI surface. | Operators ask to see the MES-reportable flag live on Debug View / Alarm Summary. | | 123 | **→ #61** **Rail nav scroll pane (`.rail-scroll` + the `#theme-rail > .side-rail { height: 100% }` fill rule) lives in app-local `site.css`/`NavMenu.razor`, not the `ZB.MOM.WW.Theme` kit** *(Low)*. Every other kit consumer (OtOpcUa, MxGateway, HistorianGateway, ignitionoee) has the same unbounded-nav-pushes-the-footer-off-screen shape and would need the identical fix copy-pasted. | `docs/plans/2026-09-14-pending-updates-design.md` §3.5 | Scoped to ScadaBridge for this change; upstreaming into the shared kit is a separate cross-repo release, not a one-app CSS tweak. | A second kit consumer hits the same long-nav clipping and copies the fix instead of consuming it from the kit. | @@ -135,7 +135,7 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round | 132 | **→ #68** **Site compile-cache eviction counter is not exported as a Meter** *(Low)*. `SiteScriptCompileCache` counts evictions and logs a rate-limited Warning (review 01/SR-M2) but the counter is not a `System.Diagnostics.Metrics` instrument, and `ZbTelemetryOptions.Meters` is an allowlist (`SiteServiceRegistration.ObservedMeters`), so exporting it also needs the Host allowlist entry. | 2026-10-21 system review (`docs/plans/2026-10-21-system-review-remediation.md`) (lane B) | The Warning log already surfaces thrash; a Meter is observability polish. | Wanting to alert on compile-cache thrash from Prometheus. | | 133 | **→ #69** **Dedicated health field for site audit rejected rows** *(Low)*. Permanent push rejections (`IngestAck.rejected_event_ids`) are counted only by the `scadabridge.site.audit.rejected_rows` counter and the `Rejected` `ForwardState`; there is no `SiteHealthReport` field/dashboard badge. | 2026-10-21 system review (`docs/plans/2026-10-21-system-review-remediation.md`) (lane H) | Rejections only come from row-intrinsic SQL errors and should be near-zero; the counter is sufficient for now. | The first non-zero `rejected_rows` seen in production, or an operator asking for it on the Health dashboard. | | 134 | **An oversize `AuditLog` row (larger than the gRPC message cap) wedges both push and pull** *(Low, accepted)*. A single row bigger than the 4 MB gRPC cap cannot be shipped by either the telemetry push or the reconciliation pull, so the queue head stalls. Unreachable at the default caps (8 KB / 64 KB payload caps, `InboundMaxBytes` 1 MiB) — it needs an operator to raise caps past the transport limit. | 2026-10-21 system review (`docs/plans/2026-10-21-system-review-remediation.md`) (lane H) | Defaults keep every row far below the limit; guarding means a skip/park state for unshippable rows. | Any deployment that raises `InboundMaxBytes` or the payload caps toward 4 MB. | -| 135 | **→ #70** **Config-audit single-column indexes are possibly redundant** *(Low)*. `ReviewRemediationIndexes` added composite `AuditLogEntries` indexes `(EntityType, Timestamp DESC)`, `(User, Timestamp DESC)` and `(Timestamp DESC, Id DESC)`; the older single-column indexes on timestamp/user/entity type may now be dead weight on a write-heavy table. | 2026-10-21 system review (`docs/plans/2026-10-21-system-review-remediation.md`) (lane P / O) | Dropping needs a measured before/after on a populated table, not an assumption. | Measure index usage (`sys.dm_db_index_usage_stats`) after the migration has run in QA/prod for a few weeks, then drop the unused ones. | +| 135 | **→ #70** **Config-audit single-column indexes are possibly redundant** *(Low)*. `ReviewRemediationIndexes` added composite `AuditLogEntries` indexes `(EntityType, Timestamp DESC)`, `(User, Timestamp DESC)` and `(Timestamp DESC, Id DESC)`; the older single-column indexes on timestamp/user/entity type may now be dead weight on a write-heavy table. | 2026-10-21 system review (`docs/plans/2026-10-21-system-review-remediation.md`) (lane P / O) | Dropping needs a measured before/after on a populated table, not an assumption. | Measure index usage (`sys.dm_db_index_usage_stats`) after the migration has run in QA/DEV for a few weeks, then drop the unused ones. | ## Resolved (verified against the code 2026-07-10) Rows removed from the Deferred table above once confirmed shipped. Kept here for traceability. diff --git a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json index 8803467b..95fbcfd6 100644 --- a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json +++ b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json @@ -87,7 +87,7 @@ {"id": 16, "subject": "Task 16: Clean up DeploymentManagerActor and AkkaHostedService", "status": "completed", "classification": "standard", "blockedBy": [15], "note": "LANDED AS ONE COMMIT with tasks 15+16 - PLAN DEFECT: they cannot compile separately (SiteReplicationActor takes a ReplicationService + calls ReplaceAllAsync; DeploymentManagerActor Tells ReplicationMessages types; AkkaHostedService constructs the actor). Combining also strengthens Task 14's own invariant: never both mechanisms, never neither. Registered 8 tables; notification_lists + smtp_configurations deliberately NOT registered. Deleted ReplaceAllAsync as UNSAFE (a mass DELETE on a replicated table would be captured and shipped), and its test replaced by a comment explaining that. The positional-arg hazard was REAL: 4 DeploymentManagerActor test call sites bound wrong args; converted to named where possible - Props.Create is an expression tree and rejects OUT-OF-POSITION named args, so the rest are padded positionally. Task 7's 'not yet' test INVERTED (exact in both directions) + new security-named test for the SMTP tables + composite-PK test. Harness's temporary registration deleted, so the convergence suites now prove the cutover. My expected table list was mis-sorted: ordinal puts '_' (0x5F) before 'b', so data_connection_definitions precedes database_connections.", "commit": "037798b3"}, {"id": 17, "subject": "Task 17: Config-key cleanup (incl. ConfigFetchRetryCount; 10 appsettings files)", "status": "completed", "classification": "standard", "blockedBy": [16], "note": "As planned. DEVIATION: deploy/wonder-app-vd03/appsettings.Site.json sits under a GITIGNORED deploy/ tree, so its edit is local-only and must be repeated on the box at deploy time. Comment style is // (JSONC) rather than \"_comment_\" keys: every one of these files already contains // comments and .NET's json config reader accepts them, and a _comment_ key inside a bound section is a phantom config entry. Both relaxations pinned by the INVERSE of the test they replace (Site_MissingSiteDbPath_IsAccepted..., EmptySqliteDbPath_IsAccepted...), each verified red with the old rule restored. DatabaseOptionsValidator needed no change - it was already null-tolerant/blank-rejecting, which is exactly migration-only semantics.", "commit": "605e5682"}, {"id": 18, "subject": "Task 18: Two-node convergence suite for the Phase 2 tables", "status": "completed", "classification": "high-risk", "blockedBy": [17], "note": "DEVIATION: landed as a NEW file LocalDbPhase2ConvergenceTests.cs rather than extending Phase 1's LocalDbSitePairConvergenceTests.cs, and drives the REAL SiteStorageService instead of hand-written SQL - possible only post-cutover, and what makes the cascade scenario test the shipped transaction rather than a re-creation of it. Scenario 4 was retargeted onto shared_scripts/external_systems/static_attribute_overrides because the plan's version overlapped LocalDbConfigConvergenceTests' N1 scenario almost exactly; the union-survives property is per-table, so re-proving it on untouched tables is the non-redundant half. Cascade test carries a never-removed control instance (absence assertions otherwise cannot distinguish 'cascade converged' from 'node B lost these tables'). Non-vacuity PROVEN as mandated: with the 8 RegisterReplicated calls commented out, 4 failed / 0 passed; restored, 20/20 across the three LocalDb suites.", "commit": "15013156"}, - {"id": 19, "subject": "Task 19: Rig configuration (MaxOplogRows/MaxOplogAge + MaxBatchSize per D6)", "status": "completed", "classification": "small", "blockedBy": [17], "note": "MaxBatchSize 500->16 (D6: row-count batching x ~70 KB production config_json vs the 4 MB gRPC cap; 16 => ~1.1 MB worst case). MaxOplogRows 1M->250,000 and MaxOplogAge 7d->2d from the soak's 0.80 rows/sec (~69k/day). Tighter-than-default is SAFE because a cap breach prunes + sets needs_snapshot (graceful snapshot resync), not data loss - the Task 1 finding that the plan's stop condition was weaker than written. site-b/site-c left unreplicated so default-OFF stays proven side by side.", "commit": "921edab4"}, + {"id": 19, "subject": "Task 19: Rig configuration (MaxOplogRows/MaxOplogAge + MaxBatchSize per D6)", "status": "completed", "classification": "small", "blockedBy": [17], "note": "MaxBatchSize 500->16 (D6: row-count batching x ~70 KB DEV-host (vd03) config_json vs the 4 MB gRPC cap; 16 => ~1.1 MB worst case). MaxOplogRows 1M->250,000 and MaxOplogAge 7d->2d from the soak's 0.80 rows/sec (~69k/day). Tighter-than-default is SAFE because a cap breach prunes + sets needs_snapshot (graceful snapshot resync), not data loss - the Task 1 finding that the plan's stop condition was weaker than written. site-b/site-c left unreplicated so default-OFF stays proven side by side.", "commit": "921edab4"}, {"id": 20, "subject": "Task 20: Live gate on the docker rig", "status": "completed", "classification": "high-risk", "blockedBy": [18, 19], "note": "ALL 10 CHECKS PASS. Evidence: docs/plans/2026-07-19-localdb-phase2-live-gate.md. Key blocker found and fixed mid-run: external systems reach a site ONLY via ArtifactDeploymentService, which `instance deploy` never invokes - `deploy artifacts` was needed both to deliver the probe harness and to propagate the owed ExternalSystemDefinitions restore. THREE METHOD CORRECTIONS to the plan: (1) its instruction to run DB checks host-side against the bind mounts is UNSAFE - host sqlite3 poisons the container WAL; copy the db/-wal/-shm triplet and query the copy. (2) `docker exec ... curl` cannot scrape metrics (no curl in aspnet:10.0) and with 2>/dev/null the failure is silent - it nearly became a false 'metrics missing' finding; use a network-sharing curl sidecar. (3) checks needing S&F load need CachedCall, not Call. CAVEATS recorded not glossed: check 2's zero-count is vacuous alone (legacy source was also empty) and rests on the ABSENCE of CDC triggers; check 7's native_alarm_state leg was empty live and is covered only offline; check 10's sampling is coarse and the real rise/drain evidence comes from check 6's 0->4->0.", "commit": "158e79bb"}, {"id": 21, "subject": "Task 21: Documentation truth pass", "status": "completed", "classification": "standard", "blockedBy": [20], "note": "Both CLAUDE.md files + Component-StoreAndForward.md:83 (normative resync paragraph rewritten for CDC, stating the duplicate-delivery bound explicitly: limited to messages the OLD primary delivered whose status change had not yet replicated when the gate flipped - one flush interval plus in-flight ack, and it does NOT grow with backlog depth or absence duration) + components/{StoreAndForward,SiteRuntime,Host}.md + the frame-size known-issue (amended: Phase 2 deleted notify-and-fetch itself, so the 128KB Akka frame constraint is gone from the intra-site hop entirely; successor ceiling is the 4MB gRPC cap via MaxBatchSize, and note the failure mode differs - oversized gRPC is REJECTED, not silently dropped) + deployment topology-guide.md and installation-guide.md (D5 stop-both-together, D2 TombstoneRetention resurrection bound). DoD closed: build 0 warnings, all 10 suites green (3509 tests, 0 failures). DoD grep nuance recorded in the plan: 4 matches remain in src/ and are all deliberate COMMENT prose explaining what was deleted - a literal 'no matches' would delete the explanations that stop someone re-introducing the old design.", "commit": "7b5a5a6f"} ], @@ -102,5 +102,5 @@ } ], "lastUpdated": "2026-07-20", - "phase2Status": "COMPLETE - all 21 tasks done, live gate PASS (docs/plans/2026-07-19-localdb-phase2-live-gate.md), MERGED to main as 28ca04d7 ('LocalDb adoption Phase 1 + 2 ... (#23)'), which carried Phase 1 in the same merge. History below retained for context. Task 1 gate CLOSED (verdict PROCEED) and Task 2 DONE, 2026-07-20. Task 1's original STOP verdict is SUPERSEDED: the 'Phase 1 disk I/O defect' was OBSERVER-INDUCED (host-side sqlite3 against live bind-mounted WAL files resets the WAL across virtiofs and permanently poisons the container's connections) - NOT a product defect. See docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md. Both earlier isolation claims were confounded: one sampling pass had already poisoned BOTH nodes, and a poisoned standby looks healthy only because it issues almost no statements; 'LocalDb-specific' was sampling-selection bias (only the LocalDb file had ever been host-read). Clean re-run 2026-07-20 with the copy-based snap() helper on restarted nodes, 6 consecutive 60s intervals: sf_messages 48 rows/min = 0.80 rows/sec dead steady, retry-UPDATE rate 0/sec (SUM(retry_count) flat at 200), oplog 0, native_alarm_state 0, max payload_json 76 B, max config_json 721 B, ZERO SQLite errors across 30 min, both site-a nodes converged at 3564 rows. Honest gap: 0.80/s is ~1.6% of the 50/s ceiling and the retry-UPDATE path was never exercised - acceptable because that ceiling is structural (SweepBatchLimit/RetryTimerInterval), not empirical. Rig config rows (721 B) are NOT representative; D6 sizing rests on the documented ~60-70 KB production config_json. Plan-premise corrections stand: D6 (MaxBatchSize 500 -> 16, the one firmly evidence-backed number; Task 19 sets it), D4 (alarm writes bounded by per-SourceReference coalescing at a 100 ms flush, NOT unbounded), sf_messages hard ceiling 50 rows/sec, oplog cap overrun = graceful snapshot resync (needs_snapshot), not data loss. Tasks 3-21 were subsequently executed and all landed. Rig cleanup that was owed before Task 20 is DONE: ExternalSystemDefinitions id 1 restored to http://scadabridge-restapi:5200 (propagation to both site nodes confirmed), and SoakGenerator template 2021 + instances soakgen-1..4 deleted - see 'Rig state as left' in the live-gate doc." + "phase2Status": "COMPLETE - all 21 tasks done, live gate PASS (docs/plans/2026-07-19-localdb-phase2-live-gate.md), MERGED to main as 28ca04d7 ('LocalDb adoption Phase 1 + 2 ... (#23)'), which carried Phase 1 in the same merge. History below retained for context. Task 1 gate CLOSED (verdict PROCEED) and Task 2 DONE, 2026-07-20. Task 1's original STOP verdict is SUPERSEDED: the 'Phase 1 disk I/O defect' was OBSERVER-INDUCED (host-side sqlite3 against live bind-mounted WAL files resets the WAL across virtiofs and permanently poisons the container's connections) - NOT a product defect. See docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md. Both earlier isolation claims were confounded: one sampling pass had already poisoned BOTH nodes, and a poisoned standby looks healthy only because it issues almost no statements; 'LocalDb-specific' was sampling-selection bias (only the LocalDb file had ever been host-read). Clean re-run 2026-07-20 with the copy-based snap() helper on restarted nodes, 6 consecutive 60s intervals: sf_messages 48 rows/min = 0.80 rows/sec dead steady, retry-UPDATE rate 0/sec (SUM(retry_count) flat at 200), oplog 0, native_alarm_state 0, max payload_json 76 B, max config_json 721 B, ZERO SQLite errors across 30 min, both site-a nodes converged at 3564 rows. Honest gap: 0.80/s is ~1.6% of the 50/s ceiling and the retry-UPDATE path was never exercised - acceptable because that ceiling is structural (SweepBatchLimit/RetryTimerInterval), not empirical. Rig config rows (721 B) are NOT representative; D6 sizing rests on the documented ~60-70 KB DEV-host (vd03) config_json. Plan-premise corrections stand: D6 (MaxBatchSize 500 -> 16, the one firmly evidence-backed number; Task 19 sets it), D4 (alarm writes bounded by per-SourceReference coalescing at a 100 ms flush, NOT unbounded), sf_messages hard ceiling 50 rows/sec, oplog cap overrun = graceful snapshot resync (needs_snapshot), not data loss. Tasks 3-21 were subsequently executed and all landed. Rig cleanup that was owed before Task 20 is DONE: ExternalSystemDefinitions id 1 restored to http://scadabridge-restapi:5200 (propagation to both site nodes confirmed), and SoakGenerator template 2021 + instances soakgen-1..4 deleted - see 'Rig state as left' in the live-gate doc." } diff --git a/docs/plans/2026-07-19-localdb-phase2-soak.md b/docs/plans/2026-07-19-localdb-phase2-soak.md index 237022a6..09bd0f1b 100644 --- a/docs/plans/2026-07-19-localdb-phase2-soak.md +++ b/docs/plans/2026-07-19-localdb-phase2-soak.md @@ -182,10 +182,10 @@ says the *escaped Akka envelope* exceeded the 128 KB frame, and that the default double-escapes the payload, so **"the raw flattened JSON only needs to be ~60-70 KB to blow the 128 KB frame."** -So the largest known real production `config_json` is on the order of **60–70 KB**. +So the largest known real-world `config_json` (DEV host vd03) is on the order of **60–70 KB**. Measured on the rig (4 deployed configs): **max 715 B, avg 714 B** — trivially small, as the plan -predicted, hence the production figure above is the one to size against. (A direct measurement +predicted, hence the DEV-host figure above is the one to size against. (A direct measurement against wonder was attempted; `wonder-app-vd03.zmr.zimmer.com` resolves over the VPN but the servecli SSH service on :2222 refuses connections, so the box was unreachable.) @@ -323,7 +323,7 @@ exercised at all. This re-run therefore confirms *steady-state health and the ab Finding 1 pathology*; it does **not** probe the ceiling. That is acceptable because the ceiling is **structural rather than empirical** (Finding 4: `SweepBatchLimit` ÷ `RetryTimerInterval`), so sizing does not depend on observing it. Likewise the rig's 721 B config rows are **not** -representative — D6's sizing rests on the documented ~60–70 KB production `config_json`, not on +representative — D6's sizing rests on the documented ~60–70 KB DEV-host (vd03) `config_json`, not on this number. ## 2. What still owes measurement @@ -338,7 +338,7 @@ Carry forward (items 1–2 partially discharged by §1b above): answer ConditionRefresh with a `SnapshotComplete`; the missing piece is ongoing transitions and a seeded `TemplateNativeAlarmSource`. (The test's own doc comment claiming the simulator "does not reliably expose A&C" is stale.) -3. A production-representative `config_json` from wonder, to replace the ~60–70 KB inference. +3. A real-world (DEV-host) `config_json` from wonder, to replace the ~60–70 KB inference. 4. The empirical oplog drain-under-churn check — Task 20 evidence item 10. ## 3. Rig state left behind diff --git a/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md b/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md index d0815009..440dd90e 100644 --- a/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md +++ b/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md @@ -469,7 +469,7 @@ Add `"SelfFormAfter": "00:00:10"` next to `"SplitBrainResolverStrategy"` in the Then `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` and commit `config(cluster): SelfFormAfter=10s explicit in all node appsettings`. -> Ops note (do NOT edit here): the gitignored `deploy/wonder-app-vd03/` overlay gets the same key on the next production deploy. +> Ops note (do NOT edit here): the gitignored `deploy/wonder-app-vd03/` overlay gets the same key on the next DEV (vd03) deploy. --- diff --git a/docs/plans/2026-08-12-script-compile-metadata-resolver-cache.md b/docs/plans/2026-08-12-script-compile-metadata-resolver-cache.md index a1f9032e..b5b909e5 100644 --- a/docs/plans/2026-08-12-script-compile-metadata-resolver-cache.md +++ b/docs/plans/2026-08-12-script-compile-metadata-resolver-cache.md @@ -20,7 +20,7 @@ **Empirical proof** (scratch repro, Roslyn 5.0.0, options mirroring the site's): a 5-assembly explicit set has a 74-assembly transitive closure; compiling the *same code* three times drove `ResolveMissingAssembly` 74 times **per compile** (222 total) and returned 222 **distinct** `PortableExecutableReference` instances — i.e. 74 fresh `AssemblyMetadata`+`PEReader`+native-block sets per compile. With a memoizing decorator: compile #1 = 74 inner resolutions, compiles #2/#3 = **0**, and the script still evaluates correctly. -This matches the production gcdump signature exactly: +This matches the DEV-host (vd03) gcdump signature exactly: - Per-script scaling, untouched by the `5a781c70` fix: Site (21 scripts) 6,640 triple-objects ≈ 21 × ~105 refs × 3 counted types; Central (6 scripts) 2,699 ≈ 6 × ~150 × 3. - Bit-identical counts pre/post fix on an identical workload (`NativeHeapMemoryBlock` 2,681 both times): the minting is deterministic per compiled script, and the explicit sets the fix de-duplicated were a small fraction of the closure. - Invisible to source greps: no ScadaBridge code calls anything per compile — the `CreateFromFile` calls happen inside `RuntimeMetadataReferenceResolver`. @@ -555,7 +555,7 @@ git commit -m "docs(scripts): record the shared caching metadata resolver invari ### Task 9 (operator-gated): Live verification, keyed to compile count — not elapsed time -**Classification:** high-risk (production measurement; no code) +**Classification:** high-risk (live DEV-host measurement; no code) **Estimated implement time:** operator session Run on the docker rig first, then wonder-app-vd03 after deploy (tooling at `E:\ApiInstall\_tools\diag\`, baselines at `E:\ApiInstall\_leakcheck-post5a781c70\`): @@ -590,7 +590,7 @@ Measured as a fixed-compile-count A/B (both nodes compile an identical startup s | Site / Central working set | 2,814 / 1,374 MB | 612 / 663 MB (post-fix read at ~5 min uptime — not uptime-matched; the metadata counts are the controlled result) | | Site / Central working set — **uptime-matched re-read** (26.3 min, same `20f2ae05` boot) | 2,814 / 1,374 MB (24+ min) | **620 / 695 MB** — flat vs the 5-min read (+8 / +32 MB), so the comparison is now clean: **4.5× / 2.0× working-set reduction** | -`CachingScriptMetadataResolver` shows exactly 1 instance per process in the gcdump — active in production. Calibration note: the steady-state residual is a **fixed** cost of roughly one shared closure set + statics (~800 for 21 scripts ≈ 38/script amortized), not the per-compile ≤~10 growth figure — the growth figure applies to *additional* compiles (rig-verified at +2/compile), the fixed cost to the startup set. Follow-up lead — **re-tested 2026-08-12, no longer reproduces**: the complex `AddTemplateScript` OOM (historically a deterministic 3–8 min hang → `OutOfMemoryException` on a ~20-line handshake body, all three authored forms, reproducible on a throwaway template) was re-run on `20f2ae05` against a fresh throwaway template (id 27, created and deleted for the test) with a faithful re-authoring of the `WriteBatchAndWaitAsync` form AND the `Attributes.WaitAsync` form, driven from a clean off-box client: both returned **HTTP 200 in <1 s**, rows persisted, Central healthy. Recorded as *no longer reproduces* rather than attributed to this fix — four deploys (`3c9b101d`→`20f2ae05`) intervened since the last confirmed repro and are not distinguishable after the fact. +`CachingScriptMetadataResolver` shows exactly 1 instance per process in the gcdump — active on the DEV host. Calibration note: the steady-state residual is a **fixed** cost of roughly one shared closure set + statics (~800 for 21 scripts ≈ 38/script amortized), not the per-compile ≤~10 growth figure — the growth figure applies to *additional* compiles (rig-verified at +2/compile), the fixed cost to the startup set. Follow-up lead — **re-tested 2026-08-12, no longer reproduces**: the complex `AddTemplateScript` OOM (historically a deterministic 3–8 min hang → `OutOfMemoryException` on a ~20-line handshake body, all three authored forms, reproducible on a throwaway template) was re-run on `20f2ae05` against a fresh throwaway template (id 27, created and deleted for the test) with a faithful re-authoring of the `WriteBatchAndWaitAsync` form AND the `Attributes.WaitAsync` form, driven from a clean off-box client: both returned **HTTP 200 in <1 s**, rows persisted, Central healthy. Recorded as *no longer reproduces* rather than attributed to this fix — four deploys (`3c9b101d`→`20f2ae05`) intervened since the last confirmed repro and are not distinguishable after the fact. ## Out of scope / follow-ups diff --git a/docs/plans/2026-09-02-delmia-sim-rig-verification.md b/docs/plans/2026-09-02-delmia-sim-rig-verification.md index 09887181..0e49179b 100644 --- a/docs/plans/2026-09-02-delmia-sim-rig-verification.md +++ b/docs/plans/2026-09-02-delmia-sim-rig-verification.md @@ -4,7 +4,7 @@ **Goal:** Stand up an in-memory (static-attribute) stand-in for a vd03-style machine with a composed `DelmiaReceiver` on the local docker rig, wire the `DelmiaRecipeDownload` inbound method and a scoped API key to it, let `DelmiaNotifier` read that key from `appsettings.json`, and prove the notifier end-to-end on the 2-node central cluster including an active-node crash and rejoin. -**Architecture:** Two templates seeded by CLI only (no DB edits): `DelmiaReceiverSim` (the ten receiver attributes, all static — no data connection) composed into `SimMachine` as `DelmiaReceiver`. The Galaxy side of the handshake is replaced by a root-scope `ValueChange` script on `SimMachine` that reacts to `DelmiaReceiver.RecipeDownloadFlag`, so the same call script shape as prod runs against pure in-memory tags. The central inbound method is the prod script verbatim (minus its dead `suf` line). The notifier gains an optional `ScadaBridge:ApiKey` config field with the existing `SCADABRIDGE_API_KEY` env var taking precedence, and a bash verification script drives the whole thing against `docker/` (central-a `:9001`, central-b `:9002`, Traefik `:9000`). +**Architecture:** Two templates seeded by CLI only (no DB edits): `DelmiaReceiverSim` (the ten receiver attributes, all static — no data connection) composed into `SimMachine` as `DelmiaReceiver`. The Galaxy side of the handshake is replaced by a root-scope `ValueChange` script on `SimMachine` that reacts to `DelmiaReceiver.RecipeDownloadFlag`, so the same call script shape as DEV (vd03) runs against pure in-memory tags. The central inbound method is the DEV (vd03) script verbatim (minus its dead `suf` line). The notifier gains an optional `ScadaBridge:ApiKey` config field with the existing `SCADABRIDGE_API_KEY` env var taking precedence, and a bash verification script drives the whole thing against `docker/` (central-a `:9001`, central-b `:9002`, Traefik `:9000`). **Tech Stack:** ScadaBridge CLI (`scadabridge`), bash, .NET 10 (`DelmiaNotifier` + xUnit tests), docker compose rig under `docker/`. @@ -18,7 +18,7 @@ - `security api-key create --format=json` returns `{ "keyId": …, "token": "sbk_…" }`; the token is shown once. - Rig: site-a is `--site-id 1`; Traefik `http://localhost:9000` → active central; direct nodes `:9001` (central-a) and `:9002` (central-b); both centrals have `StandbyForwardPeerEndpoints` set, so a standby **relays** inbound calls to the active peer. Auto-down failover of a crashed active takes ~25–30 s (`docker/failover-drill.sh`). - The notifier's failover is **connect-failure-only**: any HTTP answer (incl. 503) is final. During the takeover window a standby that cannot reach its peer answers 503 `ACTIVE_NODE_UNREACHABLE` → the notifier prints `NO`. That is expected and must be documented, not "fixed". -- Prod's deployed method + site script are recorded in `docs/api-specs/Delmia-Recipe-Download-API.md`; prod's site script still references `Children["DelmiaRecv"]` (a known, separate defect). The rig script uses `Children["DelmiaReceiver"]`. +- DEV's (vd03) deployed method + site script are recorded in `docs/api-specs/Delmia-Recipe-Download-API.md`; DEV's site script still references `Children["DelmiaRecv"]` (a known, separate defect). The rig script uses `Children["DelmiaReceiver"]`. **Model assignment summary** (see each task's `Model:` line): Sonnet for scripted CLI seeding, config plumbing, and docs; Opus for the live rig verification and the failover drill (judgment under a live system) and for reviewing the secret-handling change; Fable (this session) for the final integration review. @@ -200,10 +200,10 @@ set -euo pipefail # site-a, so DelmiaNotifier can be exercised end-to-end on the docker rig without a Galaxy. # # What it creates (all static attributes — no data connection, no bindings): -# - Template DelmiaReceiverSim — the ten DelmiaReceiver attributes from prod +# - Template DelmiaReceiverSim — the ten DelmiaReceiver attributes from DEV (vd03) # (docs/api-specs/Delmia-Recipe-Download-API.md §5), ReadyFlag defaulting to true. # - Template SimMachine — composes DelmiaReceiverSim as `DelmiaReceiver` and carries -# * ProcessRecipeDownload (Call) — prod's handshake ported to per-attribute +# * ProcessRecipeDownload (Call) — DEV's (vd03) handshake ported to per-attribute # SetAsync/WaitAsync (WriteBatchAndWaitAsync rejects static attributes). # * DelmiaReceiverSimulator (ValueChange on DelmiaReceiver.RecipeDownloadFlag) — # plays the Galaxy object: accepts unless DownloadPath ends in ".bad" or the work @@ -245,9 +245,9 @@ MACHINE_ID=$("$CLI" template create "${A[@]}" --name SimMachine \ "$CLI" template composition add "${A[@]}" --template-id "$MACHINE_ID" --composed-template-id "$RECV_ID" --instance-name DelmiaReceiver > /dev/null echo " id=$MACHINE_ID" -echo "4/6 Adding ProcessRecipeDownload (Call) — prod handshake on static attributes..." +echo "4/6 Adding ProcessRecipeDownload (Call) — DEV (vd03) handshake on static attributes..." read -r -d '' PROCESS_CODE <<'CS' || true -// Port of prod CvdReactor.ProcessRecipeDownload for STATIC attributes: WriteBatchAndWaitAsync only +// Port of DEV (vd03) CvdReactor.ProcessRecipeDownload for STATIC attributes: WriteBatchAndWaitAsync only // accepts data-sourced attributes, so the payload is written with SetAsync and the handshake waited // with WaitAsync (same trigger/response flags, same 25 s budget, same result mapping). var recv = Children["DelmiaReceiver"]; @@ -354,7 +354,7 @@ git commit -m "chore(docker): seed-delmia-sim.sh — in-memory SimMachine + comp #!/bin/bash set -euo pipefail -# Seed the DelmiaRecipeDownload inbound method (prod's script verbatim, minus its dead `suf` +# Seed the DelmiaRecipeDownload inbound method (DEV vd03's script verbatim, minus its dead `suf` # line — see docs/api-specs/Delmia-Recipe-Download-API.md §4) and one API key scoped to it. # Prints the one-time token as DELMIA_API_KEY=... — capture it; it cannot be retrieved again. # Not idempotent (method name must be unique): delete with `api-method delete --id`. @@ -415,7 +415,7 @@ Expected: `{"Result":true,"ResultText":"Simulated download accepted for WO-1"}`. ```bash git add docker/seed-delmia-api.sh docker/README.md -git commit -m "chore(docker): seed-delmia-api.sh — DelmiaRecipeDownload inbound method (prod script) + scoped delmia-notifier API key" +git commit -m "chore(docker): seed-delmia-api.sh — DelmiaRecipeDownload inbound method (DEV vd03 script) + scoped delmia-notifier API key" ``` --- @@ -546,8 +546,8 @@ git commit -m "test(docker): verify-delmia-notifier.sh — notifier end-to-end o **Files:** - Modify: `docs/api-specs/Delmia-Recipe-Download-API.md` (new §8 "Docker rig stand-in", pointing at the two seed scripts, the static-attribute port, and the verified outcomes A–G with the measured takeover time) - Modify: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md` (`## Manual smoke test` → add the rig recipe: run the three scripts, `export DELMIA_API_KEY`, expected YES/NO table) -- Modify: `docs/plans/2026-07-08-deferred-work-register.md` (one new row: "prod `ProcessRecipeDownload` still addresses `Children[\"DelmiaRecv\"]` after the 2026-06-27 rename — fix + redeploy Z28061/Z28062/Z28061Sim; QA unverified" — only if a row for it does not already exist) -- Modify: `CLAUDE.md` — in the `### Templates & Deployment` / DelmiaNotifier context, one sentence: "Rig stand-in: `docker/seed-delmia-sim.sh` + `seed-delmia-api.sh` + `verify-delmia-notifier.sh` (static-attribute port of the prod handshake — `WriteBatchAndWaitAsync` rejects static attributes, use `SetAsync`/`WaitAsync`)." +- Modify: `docs/plans/2026-07-08-deferred-work-register.md` (one new row: "DEV (vd03) `ProcessRecipeDownload` still addresses `Children[\"DelmiaRecv\"]` after the 2026-06-27 rename — fix + redeploy Z28061/Z28062/Z28061Sim; QA unverified" — only if a row for it does not already exist) +- Modify: `CLAUDE.md` — in the `### Templates & Deployment` / DelmiaNotifier context, one sentence: "Rig stand-in: `docker/seed-delmia-sim.sh` + `seed-delmia-api.sh` + `verify-delmia-notifier.sh` (static-attribute port of the DEV (vd03) handshake — `WriteBatchAndWaitAsync` rejects static attributes, use `SetAsync`/`WaitAsync`)." **Step 1: Write the sections** using the numbers from Task 4's log (takeover seconds, D's outcome, audit row count). @@ -560,7 +560,7 @@ Expected: no hits (Task 1 removed the README/env ones; anything left here is sta ```bash git add docs/api-specs/Delmia-Recipe-Download-API.md src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md docs/plans/2026-07-08-deferred-work-register.md CLAUDE.md -git commit -m "docs: Delmia rig stand-in + notifier verification results (A-G incl. active-central takeover); register row for prod DelmiaRecv path defect" +git commit -m "docs: Delmia rig stand-in + notifier verification results (A-G incl. active-central takeover); register row for DEV (vd03) DelmiaRecv path defect" ``` --- @@ -576,7 +576,7 @@ git commit -m "docs: Delmia rig stand-in + notifier verification results (A-G in ## Out of scope (deliberately) -- Fixing prod vd03's `Children["DelmiaRecv"]` defect or making the prod method idempotent — recorded in the deferred-work register (Task 5); separate change with its own deployment record. +- Fixing DEV vd03's `Children["DelmiaRecv"]` defect or making the DEV (vd03) method idempotent — recorded in the deferred-work register (Task 5); separate change with its own deployment record. - Extending `WriteBatchAndWaitAsync` to static attributes. - Site-pair failover (the user asked for the 2-node **central** cluster); `docker/failover-drill.sh` already covers the general case. - Native AOT publish of the notifier (Windows-only); the managed build is what the rig exercises. diff --git a/docs/plans/2026-09-02-inbound-api-key-shared-store-design.md b/docs/plans/2026-09-02-inbound-api-key-shared-store-design.md index c4ab7e28..01160792 100644 --- a/docs/plans/2026-09-02-inbound-api-key-shared-store-design.md +++ b/docs/plans/2026-09-02-inbound-api-key-shared-store-design.md @@ -16,7 +16,7 @@ exactly one node**. Consequences, all observed or verified on 2026-09-02: to copy the file by hand before a failover drill can pass. - A key **revoked** on one node stays valid on the other. Revocation is not cluster-wide. - The standby-relay path only works because the standby forwards *before* authenticating. -- Prod `wonder-app-vd03` is single-node and unaffected; the QA pair `vq03`/`vq04` has the gap today. +- DEV `wonder-app-vd03` is single-node and unaffected; the QA pair `vq03`/`vq04` has the gap today. - `Component-InboundAPI.md` §Storage still says keys live in the configuration database — stale since the 2026-06-02 re-arch (`RetireInboundApiKeyStore` dropped the SQL table and moved keys to SQLite). @@ -97,7 +97,7 @@ Additive for every existing consumer; the four existing packages bump in lockste `docker/sync-inbound-api-keys.sh` is **deleted** once `verify-delmia-notifier.sh` passes without it. - **Windows installs**: appsettings are user-held. QA (`vq03`/`vq04`): stop both centrals **together**, add `Provider`, start both — the first to boot creates `zbauth` and imports its file, the second - imports its own (union). Prod `vd03`: same change for consistency (single node, import once). If the + imports its own (union). DEV `vd03`: same change for consistency (single node, import once). If the app login lacks DDL rights, a DBA runs `docs/operations/sql/zbauth-schema.sql` first; the migrator finds the version row and skips DDL. Recorded per environment in a deployment record. diff --git a/docs/plans/2026-09-02-inbound-api-key-shared-store.md b/docs/plans/2026-09-02-inbound-api-key-shared-store.md index 3e9efc1b..15d59fd7 100644 --- a/docs/plans/2026-09-02-inbound-api-key-shared-store.md +++ b/docs/plans/2026-09-02-inbound-api-key-shared-store.md @@ -547,7 +547,7 @@ Commit — `chore(docker): both rigs' centrals on the shared SQL Server inbound - Create: `docs/operations/sql/zbauth-schema.sql` (copy of the library's DDL for DBAs whose app login lacks DDL rights; the migrator then finds the version row and skips). - Modify: `docs/deployment/topology-guide.md` (central-pair section: the key store is shared under SqlServer; SQLite on a pair is a documented misconfiguration the validator warns about). - Modify: `env_variables.md` (rows for `ScadaBridge__InboundApi__ApiKeyStore__Provider`, `…__SqlServer__ConnectionString`, `…__ImportFromSqlitePath`). -- Modify: `CLAUDE.md` → *Security & Auth*: one bullet — inbound key store per node by default; central pairs MUST run `Provider=SqlServer` (shared `zbauth` schema in the config DB, import once at boot, revoke cluster-wide); rig does; QA/prod appsettings are user-held (deployment record). +- Modify: `CLAUDE.md` → *Security & Auth*: one bullet — inbound key store per node by default; central pairs MUST run `Provider=SqlServer` (shared `zbauth` schema in the config DB, import once at boot, revoke cluster-wide); rig does; QA/DEV appsettings are user-held (deployment record). - Modify: `docs/plans/2026-07-08-deferred-work-register.md` row 108 → resolved (library 0.3.0 + host provider + rig proof; QA/vd03 rollout pending as a deployment record). - Modify: `docs/api-specs/Delmia-Recipe-Download-API.md` §8 (the note about the sync script → superseded). - Modify: `../scadaproj/CLAUDE.md` ScadaBridge entry (inbound keys on the shared store; Auth 0.3.0 consumer). @@ -556,7 +556,7 @@ Check: `grep -rn "sync-inbound-api-keys" docs docker CLAUDE.md` → only histori --- -### Task 13: Deployment record template for QA (vq03/vq04) and prod (vd03) +### Task 13: Deployment record template for QA (vq03/vq04) and DEV (vd03) **Classification:** trivial **Estimated implement time:** ~3 min @@ -566,9 +566,9 @@ Check: `grep -rn "sync-inbound-api-keys" docs docker CLAUDE.md` → only histori **Files:** - Create: `docs/deployment/deployment-records/2026-09-XX-zb-servers-inbound-key-store.md` (a **planned** record with the exact steps and blanks for results — the user runs it; appsettings on those boxes are user-held) -Content: build/stage per the 2026-08-21 recipe; **QA:** stop both centrals together → add `"ApiKeyStore": { "Provider": "SqlServer" }` under `ScadaBridge:InboundApi` in both `appsettings.Central.json` → if `scadabridge_app` lacks DDL rights run `docs/operations/sql/zbauth-schema.sql` on the QA config DB first → start both → expect one `KeysInserted=N` line and one `KeysSkipped` line → `security api-key list` from `10.220.158.2:8085` and `.3:8085` agree → `sync`-free failover check with a curl against `DelmiaRecipeDownload` from the non-active node. **Prod vd03:** single node, same change (`KeysInserted=5` expected: `mes`, `mes api key`, `delmia-mes-receivers`, `TestKey`, `JOE_TEST_KEY`). Rollback: revert `Provider`, rename `.imported` back — no SQL rows need removing (the SQLite path ignores them). +Content: build/stage per the 2026-08-21 recipe; **QA:** stop both centrals together → add `"ApiKeyStore": { "Provider": "SqlServer" }` under `ScadaBridge:InboundApi` in both `appsettings.Central.json` → if `scadabridge_app` lacks DDL rights run `docs/operations/sql/zbauth-schema.sql` on the QA config DB first → start both → expect one `KeysInserted=N` line and one `KeysSkipped` line → `security api-key list` from `10.220.158.2:8085` and `.3:8085` agree → `sync`-free failover check with a curl against `DelmiaRecipeDownload` from the non-active node. **DEV vd03:** single node, same change (`KeysInserted=5` expected: `mes`, `mes api key`, `delmia-mes-receivers`, `TestKey`, `JOE_TEST_KEY`). Rollback: revert `Provider`, rename `.imported` back — no SQL rows need removing (the SQLite path ignores them). -Commit — `docs(deployment): planned record for moving QA/prod central inbound key stores to SQL Server` +Commit — `docs(deployment): planned record for moving QA/DEV central inbound key stores to SQL Server` --- @@ -588,4 +588,4 @@ Check: (1) scadaproj `git log` shows Tasks 1-7 committed and `artifacts/` holds - Site nodes, MxAccessGateway, HistorianGateway (SQLite stays their store). - Hard-failing a clustered central on `Provider=Sqlite` (warning only; revisit after QA is on SqlServer). - Encrypting `secret_hash` at rest; the pepper never enters SQL. -- Fixing register row 107 (prod `DelmiaRecv`) — separate change. +- Fixing register row 107 (DEV vd03 `DelmiaRecv`) — separate change. diff --git a/docs/plans/2026-09-02-inbound-api-key-shared-store.md.tasks.json b/docs/plans/2026-09-02-inbound-api-key-shared-store.md.tasks.json index e2e9d1d6..ad20ead4 100644 --- a/docs/plans/2026-09-02-inbound-api-key-shared-store.md.tasks.json +++ b/docs/plans/2026-09-02-inbound-api-key-shared-store.md.tasks.json @@ -109,7 +109,7 @@ }, { "id": 13, - "subject": "Task 13: Planned deployment record for QA vq03/vq04 and prod vd03", + "subject": "Task 13: Planned deployment record for QA vq03/vq04 and DEV vd03", "status": "completed", "blockedBy": [ 12 diff --git a/docs/plans/2026-09-09-arch-review.md b/docs/plans/2026-09-09-arch-review.md index 9d4d8886..bb5a6745 100644 --- a/docs/plans/2026-09-09-arch-review.md +++ b/docs/plans/2026-09-09-arch-review.md @@ -13,7 +13,7 @@ rechecked all High source paths, ran build/test/format/package checks, and execu probes. Each area revalidated its existing IDs and searched for additional defects. Source-traced mechanisms, runtime probes, latent risks, and unproven operational impact are distinguished below. This is a focused whole-system review, not a claim that every line or production topology was -exercised. The review phase ran no rig rebuild, device writes, failover drill, or remote QA/production inspection. +exercised. The review phase ran no rig rebuild, device writes, failover drill, or remote QA/DEV inspection. Sections 1–8 preserve that reviewed baseline; §§9–10 record subsequent implementation and local rig verification. **Second-pass changes:** 11 new findings; UI-3 withdrawn as unsupported; XC-6 merged into CM-5; @@ -479,7 +479,7 @@ source on each adapter. Medium. **DC-6 — `deploy site` exits 0 when its domain result reports failed instances `[Medium][CONFIRMED]`** — `CLI/Commands/CommandHelpers.cs:169-193` returns 0 for any 2xx; `DeployCommands.cs:68-84`. A CI `scadabridge deploy site … || rollback` never fires; other command-specific success payloads need separate contract checks; do not assume every JSON command has the same failure shape. Fix: an `onSuccess` handler inspecting `failureCount` → exit 1; document. Small. -**DC-7 — DelmiaNotifier treats a standby central's 503 as final and never fails over `[Medium][CONFIRMED]`** — `DelmiaNotifier/Notifier.cs:31-52` rolls over only on `ConnectFailed`. Its HA therefore depends entirely on `StandbyForwardPeerEndpoints` being configured on both centrals — which QA (`vq03`/`vq04`) and prod (`vd03`) still lack (CLAUDE.md). Fix: distinguish a known pre-execution standby rejection from an ambiguous post-send failure before retrying; generic retry-on-any-503 can duplicate effects. Document and test the notifier HA contract. QA/prod peer settings quoted here were not remotely rechecked in this pass. Small. +**DC-7 — DelmiaNotifier treats a standby central's 503 as final and never fails over `[Medium][CONFIRMED]`** — `DelmiaNotifier/Notifier.cs:31-52` rolls over only on `ConnectFailed`. Its HA therefore depends entirely on `StandbyForwardPeerEndpoints` being configured on both centrals — which QA (`vq03`/`vq04`) and DEV (`vd03`) still lack (CLAUDE.md). Fix: distinguish a known pre-execution standby rejection from an ambiguous post-send failure before retrying; generic retry-on-any-503 can duplicate effects. Document and test the notifier HA contract. QA/DEV peer settings quoted here were not remotely rechecked in this pass. Small. **DC-8 — `secured-write` is a shipping CLI group absent from the README and from `CommandTreeTests` `[Low][CONFIRMED]`** — `CLI/Program.cs:42`; `CommandTreeTests.cs:26-45, 63-71` asserts 18 groups vs 19 in `Program.cs` and builds `data-connection` without `ConnectionBrowseCommands.AddTo`, so browse/search/verify-endpoint/certs are uncovered too. Fix: add to the test (19), `AddTo` in the test build, README section. Small. @@ -773,7 +773,7 @@ Repeatable regression recipes are attached to each finding and the §5 acceptanc | XC-1 | Use actual listener/local port, not spoofable Host/authority matching, for port isolation | All previous SR/DC/MG/UI/DL/EI/CM/XC IDs and ORCH-1 were source-revisited or, for inventory/doc -observations, checked against their current files. Historical production/QA statements, performance +observations, checked against their current files. Historical DEV/QA statements, performance estimates, and migration/deferred-register inventories are not newly measured simply because source is unchanged. The original twelve design commitments describe implementation structure; they are not evidence that the script trust gate is a complete security sandbox or that per-site diff --git a/docs/plans/2026-09-11-mes-reportable-alarm-flag.md b/docs/plans/2026-09-11-mes-reportable-alarm-flag.md index bf9e0493..c9d8ff4e 100644 --- a/docs/plans/2026-09-11-mes-reportable-alarm-flag.md +++ b/docs/plans/2026-09-11-mes-reportable-alarm-flag.md @@ -667,6 +667,6 @@ Expected: 0 errors; every project `Passed!` (the known env-only Playwright role- **Step 2: `git diff main --stat` review**, then `git log --oneline main..HEAD` — one commit per task above. -**Step 3: Merge** via `superpowers-extended-cc:finishing-a-development-branch` (fast-forward `main`, delete the worktree). Do NOT push unless the user says so; QA/prod (`vq03`/`vq04`/`vd03`) are NOT deployed (remote exec is on hold — CLAUDE.md 2026-09-11 note). +**Step 3: Merge** via `superpowers-extended-cc:finishing-a-development-branch` (fast-forward `main`, delete the worktree). Do NOT push unless the user says so; QA/DEV (`vq03`/`vq04`/`vd03`) are NOT deployed (remote exec is on hold — CLAUDE.md 2026-09-11 note). -**Step 4: Memory** — write `mes-reportable-alarm-flag-state.md` (status, merged SHA, rig seeded, key name, QA/prod not deployed, follow-ups = register rows 121/122) and add its line to `MEMORY.md`. +**Step 4: Memory** — write `mes-reportable-alarm-flag-state.md` (status, merged SHA, rig seeded, key name, QA/DEV not deployed, follow-ups = register rows 121/122) and add its line to `MEMORY.md`. diff --git a/docs/plans/2026-10-21-system-review-remediation.md b/docs/plans/2026-10-21-system-review-remediation.md index 1d6cf0ad..65640695 100644 --- a/docs/plans/2026-10-21-system-review-remediation.md +++ b/docs/plans/2026-10-21-system-review-remediation.md @@ -47,8 +47,8 @@ Rows 128-135 of `docs/plans/2026-07-08-deferred-work-register.md`: TLS productio ## Rollout notes -- **Not deployed:** the docker rig is rebuilt from main; QA (vq03/vq04) and prod (vd03) are not deployed. -- **Migration:** `ReviewRemediationIndexes` adds `AuditLog` `IX_AuditLog_Kind_Occurred` and `IX_AuditLog_Actor_Occurred` (partition-aligned) plus three composite `AuditLogEntries` indexes. Production applies `docs/plans/sql/ReviewRemediationIndexes.sql` by hand (index builds on a large partitioned `AuditLog`: schedule off-hours). Not yet applied to QA/prod. +- **Not deployed:** the docker rig is rebuilt from main; QA (vq03/vq04) and DEV (vd03) are not deployed. +- **Migration:** `ReviewRemediationIndexes` adds `AuditLog` `IX_AuditLog_Kind_Occurred` and `IX_AuditLog_Actor_Occurred` (partition-aligned) plus three composite `AuditLogEntries` indexes. Production applies `docs/plans/sql/ReviewRemediationIndexes.sql` by hand (index builds on a large partitioned `AuditLog`: schedule off-hours). Not yet applied to QA/DEV. - **Mixed version:** upgrade central first (new `IngestAck` field 2 and proto additions are additive; an old central ignores `rejected_event_ids` and an old site never rejects). Pair upgrades stay both-nodes-together for LocalDb sites. - **Config changes:** CLI hosts need a central that serves `/auth/token` (older centrals fall back to Basic automatically); Central must have a >= 32-byte `JwtSigningKey` or fails startup validation; remove `Database:MachineDataDb` and `Transport:SchemaVersionMajor` from appsettings (ignored if left). - **New keys (all defaulted):** `ScadaBridge:ConfigAudit:{RetentionDays 0, PurgeBatchSize 5000, MaxBatchesPerTick 200, PurgeInterval 1d}`; `Security:RoleMappingCacheSeconds 30`; `SiteEventLog:{KeywordSearchMaxWindow 7d, CapTrimCountRefreshBatches 10}`; `AuditLog` site writer/reconciler `{MaxRejectedPushAttempts 10, PurgeBatchSize 5000, PurgeVacuumPagesPerBatch 1000, MaxPagesPerTick 50}`; `SiteRuntime:ScriptCompileCacheMaxEntries 1024`; `NotificationOutbox:DispatchSweepBudget 2m`; `ManagementService:{SecuredWriteTerminalRetention 90d, SecuredWriteTerminalPurgeInterval 1h}`; ESG `DefinitionCacheTtl 10s` (central); DCL `MxAlarmStreamReopenDebounce 200ms`; `DeploymentManager:TerminalDeploymentRecordRetention` governs system artifact deployment record purge. diff --git a/schemalib.md b/schemalib.md index 4c678c9e..ad56706e 100644 --- a/schemalib.md +++ b/schemalib.md @@ -158,7 +158,7 @@ Queried live against `scadabridge-mssql` on 2026-08-11: Not only has the library never been used — most scripts don't declare a typed parameter shape at all, which is the precondition for ever wanting a reusable one. -*(Production `wonder-app-vd03` was deliberately not queried; see §7.)* +*(DEV `wonder-app-vd03` was deliberately not queried; see §7.)* ### 6.2 Origin: speculative, not requirement-driven @@ -203,7 +203,7 @@ months, take option 1 rather than leaving permanent unexercised surface in the c - The CLI has no schema-library commands either (`src/ZB.MOM.WW.ScadaBridge.CLI/` has no `SharedSchema` reference), so there is no scriptable authoring path as a workaround. This is why §3's trap is currently unreachable: hand-editing the database is the only way to create a ref. -- The §6.1 counts are from the two **local docker** environments only. Production +- The §6.1 counts are from the two **local docker** environments only. DEV (`wonder-app-vd03`) was not queried — that is an operator action, not one to take unasked. If that box carries inbound methods with `$ref` definitions, §3 is live there and the §6.4 recommendation changes.