deploy(vd03): fe89363b + migrations 56/57; in-memory MES/Delmia/alarm-API test machines
vd03 (DEV) upgraded2b9a7c3c->fe89363bvia the deployment-only winrm channel: COPY_ONLY backup, 55->57 delta script (AddAlarmMesReportable, AddSiteArtifactRevision), ~53 s window, snapshots identical to baseline, artifacts deployed. Register row 126 now covers the QA pair only. docker/seed-mes-test-machine.sh (idempotent) builds a standalone TestMesMachine base template - Left/Right TestMESReceiver + TestDelmiaReceiver (prod attribute names, STATIC), computed alarms with IsMesReportable, CvdReactor-compatible MesMoveIn/MesMoveOut/ProcessRecipeDownload/AlarmStatus scripts ported to SetAsync+WaitAsync, and Galaxy-side simulators with reject/no-response knobs - plus instances SBTEST01/02 and the SimpleAlarmStatusRequest (was a stub) / AlarmStatusRequest inbound methods in the legacy /mes/(simple)alarmstatus shapes, resolving machines through BTDB. BTDB rows SBTEST01=990001 / SBTEST02=990002 added on vd03. docker/verify-mes-test-machine.sh drives the real inbound methods via flow-debug: PASS 25/25.
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,126 @@
|
||||
# 2026-09-28 — DEV (wonder-app-vd03) → `fe89363b` + in-memory MES/Delmia/alarm-API test machines
|
||||
|
||||
**Build:** `main` @ `fe89363b` (= `origin/main`; code tip `3a21cfc2`), ProductVersion
|
||||
`1.0.0+fe89363b1a2dbc07c7578cd33c88b7162a1b19e7`, published from a detached worktree at that SHA
|
||||
(`dotnet publish` Host + CLI `-c Release -r win-x64 --self-contained true -p:PublishSingleFile=false`;
|
||||
1029 app files, 223 cli files, zip 125 593 314 B, SHA-256 `bb821df1…87167033`).
|
||||
**Delta from the box's previous build `2b9a7c3c`:** 62 commits — MES-reportable computed-alarm flag
|
||||
(`IsMesReportable`, `Alarms.MesReportableAsync()`), site artifact revision + pending-updates indicator,
|
||||
Gitea #36 reset-aware handshake overload, Gitea #37 Flow Debugger structured method parameters.
|
||||
**QA (vq03/vq04) NOT touched** — still `2b9a7c3c` at migration 55 (register row 126).
|
||||
|
||||
**Transport:** Mac-side `winrm` CLI — WinRM 5985/NTLM for the two on-box scripts, `winrm smb push`
|
||||
over 445 for the zip/SQL/scripts. First deploy under the 2026-09-28 *deployment-only* WinRM scope
|
||||
(CLAUDE.md); every non-deploy read (baseline, logs, verification) went over HTTP or SMB.
|
||||
|
||||
## Schema: 55 → 57 migrations (additive only)
|
||||
|
||||
| Migration | Effect |
|
||||
|---|---|
|
||||
| `20260911135501_AddAlarmMesReportable` | `TemplateAlarms.IsMesReportable bit NOT NULL DEFAULT 0`; `InstanceAlarmOverrides.IsMesReportableOverride bit NULL` |
|
||||
| `20260914173147_AddSiteArtifactRevision` | `Sites.ArtifactRevisionHash nvarchar(100) NULL`; `Sites.ArtifactsDeployedAt datetimeoffset NULL` |
|
||||
|
||||
From→to script `dotnet ef migrations script 20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration
|
||||
20260914173147_AddSiteArtifactRevision` (`sb-mig-55to57.sql`, NOT idempotent — reusable as-is for the QA
|
||||
pair), applied with `sqlcmd -C -b -I` while both services were stopped. No data upgrade; the Central boot
|
||||
logged no "Applying migration" (the script had already applied both).
|
||||
|
||||
## Config
|
||||
|
||||
**No appsettings change.** The only new option in the delta, `ScadaBridge:DeploymentManager:PendingUpdatesCacheTtl`,
|
||||
defaults to 60 s. The box's three `appsettings*.json` were copied out, restored after the swap and SHA-256-matched
|
||||
(`A6B6CF05…` / `3382DD83…` / `1A4333DA…`). Service registry environment untouched.
|
||||
|
||||
## Procedure (`E:\psrem-stage\fe89363b\`, log `E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-20260928T095846.txt`)
|
||||
|
||||
1. Baseline over HTTP from the Mac: `/health/ready` 200/200; `debug snapshot` Z28061(1) Bad 126 / Good 144,
|
||||
Z28062(2) Bad 124 / Good 146, Z28061Sim(8) Bad 28 / Good 242 (270 attrs, 7 alarms each).
|
||||
2. `sb-prep.ps1` (09:58:25): zip SHA-256 re-verified on the box, unpacked, staged Host/CLI ProductVersion =
|
||||
`fe89363b`; `reg export` of both service keys; `BACKUP DATABASE [ScadaBridge] … COPY_ONLY, CHECKSUM,
|
||||
COMPRESSION` → `E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-fe89363b_20260928T095825.bak`
|
||||
(140 795 392 B), `RESTORE VERIFYONLY … WITH CHECKSUM` valid; DB at 55 migrations.
|
||||
3. `sb-swap.ps1` (the 2026-09-10 script with the SHA and 55→57 step swapped in): **09:58:46** Site stopped
|
||||
(10 s), Central stopped (10 s), no `Host.exe` left; 09:59:08 migrations 55 → 57; `app`/`cli` →
|
||||
`*.bak.20260928T095846`, staged copy in, appsettings restored (SHA match); 09:59:14 Central started →
|
||||
ready 09:59:30; Site started → ready **09:59:39**. **~53 s total downtime.**
|
||||
4. `deploy artifacts` (last step per register row 126): `Dev Site` success; the site now reports
|
||||
`artifactRevisionHash sha256:9e054ef0…`, `artifactsDeployedAt 2026-09-28T14:01:04Z`.
|
||||
|
||||
## Verification
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| `:8085` / `:8084 /health/ready` | 200 Healthy / 200 Healthy |
|
||||
| ProductVersion | live Host + CLI `fe89363b` |
|
||||
| DB | 57 migrations |
|
||||
| Snapshots after (settled) | Z28061 Bad 126 / Good 144, Z28062 Bad 124 / Good 146, Z28061Sim Bad 28 / Good 242 — **identical to baseline** (the first read ~20 s after start showed 20 tags Uncertain while subscriptions settled) |
|
||||
| Logs after boot (pulled over SMB) | central + site: only the usual first-probe `akka-cluster … member status: Removed` ERR lines during startup; no other ERR/FTL |
|
||||
|
||||
## In-memory MES / Delmia / alarm-API test machines (same session)
|
||||
|
||||
Seeded with **`docker/seed-mes-test-machine.sh`** (idempotent; second run reused every id and the gate
|
||||
re-passed) and gated with **`docker/verify-mes-test-machine.sh` — PASS 25/25**. Nothing existing was
|
||||
modified except the `SimpleAlarmStatusRequest` stub (below); CvdReactor and its instances are untouched.
|
||||
|
||||
| Object | Id | Notes |
|
||||
|---|---|---|
|
||||
| Template `TestMESReceiver` | 28 | 26 STATIC attributes with the prod MESReceiver names; ready flags default true |
|
||||
| Template `TestDelmiaReceiver` | 29 | 10 STATIC attributes with the prod DelmiaReceiver names; ReadyFlag true |
|
||||
| Template `TestMesMachine` | 30 | standalone base (not CvdReactor-derived); composes `LeftMESReceiver`/`RightMESReceiver`/`DelmiaReceiver`; Call scripts `MesMoveIn`, `MesMoveOut`, `ProcessRecipeDownload`, `AlarmStatus`; simulators `SimLeft/RightMoveIn`, `SimLeft/RightMoveOut`, `SimDelmiaRecipe` |
|
||||
| Instance `SBTEST01` | 9 | Dev Site, deployed |
|
||||
| Instance `SBTEST02` | 10 | Dev Site, deployed; alarm override `SensorFault` → MES-reportable |
|
||||
| Inbound `SimpleAlarmStatusRequest` | 9 | **was a stub returning `[]`** — now real: SAPID → BTDB → `AlarmStatus`, flagged-only, acked included |
|
||||
| Inbound `AlarmStatusRequest` | 12 | new: legacy `/mes/alarmstatus` shape — `MachineFilter` (SAPID → Code → ZTag → MachineID, via BTDB) + `AlarmFilter` |
|
||||
| BTDB `dbo.Machine` | 203, 204 | `SBTEST01` SAPID `990001`, `SBTEST02` SAPID `990002` (Code = Name = ZTag = instance name, `Eastern Standard Time`, `LastUpdateBy = scadabridge-test-seed`), inserted through a temporary inbound method (`zzTmpBtdbSchema`, id 11) that was deleted afterwards |
|
||||
|
||||
**Alarms** (static source → computed alarm): `VacuumPressure` → `HighVacuumFault` (HiLo hi 5 / hihi 10, 800,
|
||||
MES), `ArgonPressure` → `LowArgonPressure` (lo 20 / lolo 10, 600, MES), `FurnaceTemp` → `FurnaceOverTemp`
|
||||
(hi 1050 / hihi 1100, 900, MES), `DoorOpen` → `DoorOpenAlarm` (true, 300, not MES), `SensorStatus` →
|
||||
`SensorFault` (−1, 700, not MES on the template; MES on SBTEST02 via override).
|
||||
|
||||
**Simulator knobs** (root static attributes): `SimNextBatchID` (5000, increments per move-in),
|
||||
`SimMoveInReject`, `SimMoveOutReject`, `SimRecipeReject`, `SimNoResponse` (never answer → the 25 s timeout
|
||||
path). Input-driven rejects: a work order containing `FAIL`; a Delmia path ending `.bad` or a blank work
|
||||
order; a move-out with nothing moved in on that side. Ready flags on the receivers can be set false to hit
|
||||
the "ready flag not set" paths.
|
||||
|
||||
### How to test (real HTTP)
|
||||
|
||||
```bash
|
||||
H=http://wonder-app-vd03.zmr.zimmer.com:8085/api; K='X-API-Key: <key with the method in scope>'
|
||||
curl -s -X POST $H/MesMoveIn -H "$K" -H 'Content-Type: application/json' -d '{"SAPID":"990001_A","ContainerNumber":"CN-1","JobSequenceNumber":"10","WorkOrders":[{"WorkOrderNumber":"WO-1","PartNumber":"PN-1"}],"OperatorName":"me"}'
|
||||
curl -s -X POST $H/MesMoveOut -H "$K" -H 'Content-Type: application/json' -d '{"SAPID":"990001_A","ContainerNumber":"CN-1","NumberWorkOrders":1,"WorkOrders":[{"WorkOrderNumber":"WO-1","PartNumber":"PN-1"}],"OperatorName":"me"}'
|
||||
curl -s -X POST $H/DelmiaRecipeDownload -H "$K" -H 'Content-Type: application/json' -d '{"MachineCode":"SBTEST01","DownloadPath":"\\\\share\\r1.xml","WorkOrderNumber":"WO-1","PartNumber":"PN-1","JobStepNumber":"20","Username":"me"}'
|
||||
curl -s -X POST $H/SimpleAlarmStatusRequest -H "$K" -H 'Content-Type: application/json' -d '{"SAPID":"990001"}'
|
||||
curl -s -X POST $H/AlarmStatusRequest -H "$K" -H 'Content-Type: application/json' -d '{"MachineFilter":{"Code":"SBTEST01"},"AlarmFilter":{"MinSeverity":500}}'
|
||||
```
|
||||
|
||||
`_A`/`_B` picks the Left/Right receiver, `_LT` makes it a leak-test move (MoveOut then returns the six `LT*`
|
||||
rows instead of the 16 cycle rows). Trip an alarm without a key via the Flow Debugger (CLI
|
||||
`flow-debug run-value-change --instance-id 9 --attribute VacuumPressure --value 12`, clear with `0.5`), or
|
||||
from the Central UI Debug View. The gate needs no key — it drives the methods through `flow-debug run-method`.
|
||||
|
||||
### Known semantics (not bugs)
|
||||
|
||||
- Computed alarms are always `Acknowledged` → `StatusCode = "Triggered.Acked"`, `AckDT = null`, and
|
||||
`AlarmFilter.IncludeAcked = false` returns nothing for this machine (it only bites on native mirrors).
|
||||
- `Description` falls back to the alarm name (computed alarms carry no message/type text).
|
||||
- The alarm methods accept an exact BTDB SAPID first, then its leading digits (so `990001_A` works), and
|
||||
strip a trailing `A`/`B` from the BTDB Code to find the instance (MesMoveOut's convention);
|
||||
`HierarchicalName` keeps the BTDB Code, as the legacy service did.
|
||||
- A machine without an `AlarmStatus` script (e.g. CvdReactor's Z28061) answers
|
||||
`Alarm status is not supported on machine '<code>'`.
|
||||
|
||||
## Rollback
|
||||
|
||||
Upgrade: stop Site, stop Central; `RESTORE DATABASE [ScadaBridge] FROM DISK =
|
||||
'E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-fe89363b_20260928T095825.bak' WITH REPLACE`;
|
||||
rename `app` → `app.failed`, `app.bak.20260928T095846` → `app`, same for `cli`; start Central, then Site.
|
||||
(The migrations are additive, so `2b9a7c3c` would also run against the 57-migration DB — the restore just
|
||||
makes it exact.) Test machines: `instance delete` 9/10, `template delete` 30/29/28, restore the stub body of
|
||||
method 9 (`return new { WasSuccessful = true, Alarms = Array.Empty<Object>() };`) and delete method 12;
|
||||
`DELETE FROM dbo.Machine WHERE MachineID IN (203, 204) AND LastUpdateBy = N'scadabridge-test-seed'`.
|
||||
|
||||
Staging left on the box under `E:\psrem-stage\fe89363b\` (zip, unpacked tree, migration SQL, prep + swap
|
||||
scripts, `.reg` exports). The `_appsettings-keep-20260928T095846` copy and `*.bak.20260928T095846`
|
||||
directories can go once the build has soaked.
|
||||
@@ -119,7 +119,7 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
|
||||
| 123 | **Rail nav scroll pane (`.rail-scroll` + the `#theme-rail > .side-rail { height: 100% }` fill rule) lives in app-local `site.css`/`NavMenu.razor`, not the `ZB.MOM.WW.Theme` kit** *(Low)*. Every other kit consumer (OtOpcUa, MxGateway, HistorianGateway, ignitionoee) has the same unbounded-nav-pushes-the-footer-off-screen shape and would need the identical fix copy-pasted. | `docs/plans/2026-09-14-pending-updates-design.md` §3.5 | Scoped to ScadaBridge for this change; upstreaming into the shared kit is a separate cross-repo release, not a one-app CSS tweak. | A second kit consumer hits the same long-nav clipping and copies the fix instead of consuming it from the kit. |
|
||||
| 124 | **No template-graph-watermark short-circuit on the pending-updates fleet flatten** *(Low)*. `PendingUpdatesEvaluator.EvaluateAsync` re-flattens every deployed instance through `DeploymentService.GetInstanceStalenessAsync` on every cache-TTL expiry (`PendingUpdatesCacheTtl`, default 60 s) while any Deployer/Admin circuit is open, even when nothing in the template/instance/artifact graph has changed since the last evaluation. | `docs/plans/2026-09-14-pending-updates-design.md` §3.2 ("Not in scope") | The single-flight TTL cache already bounds the cost to one fleet flatten per TTL fleet-wide (not per viewer); a watermark short-circuit is a further optimization, not a correctness need, and needs a cheap "has anything changed" signal that does not exist yet. | The fleet flatten's cost (instance count x template depth) becomes measurable against `PendingUpdatesCacheTtl` on a large deployment. |
|
||||
| 125 | **Pending-updates counts have no Health-dashboard tile or CLI surface** *(Low)*. The aggregate (`PendingUpdatesSnapshot.StaleInstanceCount` / `StaleArtifactSiteCount`) is consumed only by the Central UI Sites page and rail indicator; there is no Health dashboard KPI tile and no `deploy`/`site` CLI command surfacing it — `site list` shows only the raw `artifactRevisionHash`/`artifactsDeployedAt` fields on each site, not whether they are current. | `docs/plans/2026-09-14-pending-updates-design.md` §3.2 ("Not in scope") | Out of scope for the UI-focused ask; the CLI/Health surfaces are a separate consumer of the same `IPendingUpdatesCache`. | An operator wants the pending-updates counts without opening the Central UI (headless monitoring, CLI scripting). |
|
||||
| 126 | **TODO — roll the two outstanding migrations to DEV vd03 and the QA pair vq03/vq04** *(deployment, blocked)*. Those boxes run `2b9a7c3c` with the DB at migration 55 (`20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration`); `main` (`ac833f5a`, pushed 2026-09-14) carries two more: **56** `20260911135501_AddAlarmMesReportable` (computed-alarm `IsMesReportable` flag) and **57** `20260914173147_AddSiteArtifactRevision` (`Sites.ArtifactRevisionHash`/`ArtifactsDeployedAt`, both nullable). Both are additive `AddColumn`s — no data upgrade, no coordinated step like the 2026-09-10 protocol-config encryption. Recipe (same as the 55 rollout): generate ONE from→to delta script `dotnet ef migrations script EnforceGlobalInstanceNamesAndProtectProtocolConfiguration AddSiteArtifactRevision --project src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase --startup-project src/ZB.MOM.WW.ScadaBridge.Host -o sb-mig-55to57.sql` (NOT idempotent, identical for vd03 and the QA pair), take the pre-upgrade DB backup, stop the pair TOGETHER, apply via `sqlcmd -C -b -I` on the SQL host, publish/swap the `ac833f5a` binaries, start the pair, confirm the Central boot log shows 0 pending migrations and no "Applying migration" (the script already applied them). After upgrade every site reads *Artifacts never deployed* until one *Deploy Artifacts to All Sites* — run it as the last step. Write the deployment record `docs/deployment/deployment-records/<date>-wonder-app-{vd03,vq03-vq04}-ac833f5a.md`. | Pending-updates finish (2026-09-14) | **Blocked** by the wonder-box remote-access hold (CLAUDE.md 2026-09-11: no `psbridge`/WinRM) — the SQL step is allowed (direct SQL login), the service stop/publish/start is not. | The user clears a remote-exec channel for vd03/vq03/vq04, or hands the box steps to someone on-site. |
|
||||
| 126 | **TODO — roll the two outstanding migrations to the QA pair vq03/vq04** *(deployment, open)*. **DEV vd03 DONE 2026-09-28** (`fe89363b`, 55 → 57, record `docs/deployment/deployment-records/2026-09-28-wonder-app-vd03-fe89363b.md` — the `sb-mig-55to57.sql` there is the same script QA needs). QA still runs `2b9a7c3c` with the DB at migration 55 (`20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration`); `main` (`ac833f5a`, pushed 2026-09-14) carries two more: **56** `20260911135501_AddAlarmMesReportable` (computed-alarm `IsMesReportable` flag) and **57** `20260914173147_AddSiteArtifactRevision` (`Sites.ArtifactRevisionHash`/`ArtifactsDeployedAt`, both nullable). Both are additive `AddColumn`s — no data upgrade, no coordinated step like the 2026-09-10 protocol-config encryption. Recipe (same as the 55 rollout): generate ONE from→to delta script `dotnet ef migrations script EnforceGlobalInstanceNamesAndProtectProtocolConfiguration AddSiteArtifactRevision --project src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase --startup-project src/ZB.MOM.WW.ScadaBridge.Host -o sb-mig-55to57.sql` (NOT idempotent, identical for vd03 and the QA pair), take the pre-upgrade DB backup, stop the pair TOGETHER, apply via `sqlcmd -C -b -I` on the SQL host, publish/swap the `ac833f5a` binaries, start the pair, confirm the Central boot log shows 0 pending migrations and no "Applying migration" (the script already applied them). After upgrade every site reads *Artifacts never deployed* until one *Deploy Artifacts to All Sites* — run it as the last step. Write the deployment record `docs/deployment/deployment-records/<date>-wonder-app-{vd03,vq03-vq04}-ac833f5a.md`. | Pending-updates finish (2026-09-14) | ~~Blocked by the 2026-09-11 remote-access hold~~ — **unblocked 2026-09-28**: `winrm` is re-enabled for deployment activities (CLAUDE.md). vd03 done; QA waits only for the user to schedule its window. | The user asks for the QA (vq03/vq04) window. |
|
||||
|
||||
## Resolved (verified against the code 2026-07-10)
|
||||
Rows removed from the Deferred table above once confirmed shipped. Kept here for traceability.
|
||||
|
||||
Reference in New Issue
Block a user