Joseph Doherty
82 lines
7.0 KiB
Markdown
82 lines
7.0 KiB
Markdown
# Deployment record — wonder-app-vq03 + wonder-app-vq04 (QA) — 2026-08-20
|
|
|
|
Point-in-time record of a ScadaBridge binary + schema upgrade of the **QA two-host
|
|
topology**: a real central pair (`central-a` = wonder-app-vq03 `10.220.158.2`, `central-b` =
|
|
wonder-app-vq04 `10.220.158.3`, :8085) and a site-a pair (`node-a`/`node-b` on the same two
|
|
boxes, gRPC :8083, metrics :8084), each box running `ScadaBridge-Central`, `ScadaBridge-Site`
|
|
and `ScadaBridge-LDAP` out of `E:\ApiInstall\ScadaBridge`. Shared ConfigDb on
|
|
`wonder-sql-vq03`. Performed from the Mac via **psbridge** on vq04 (`:6150`, root `E:\psrem`);
|
|
vq03 reached from vq04 over WinRM (`Invoke-Command`) and the `E$` admin share.
|
|
|
|
## 1. Binary + schema upgrade
|
|
|
|
| Item | Value |
|
|
|---|---|
|
|
| Build deployed | `main` @ **`a24fa38e`** (from `d4bb4e17`, 2026-08-12; 246 commits) — all four nodes |
|
|
| Headline change | **Grouped connection bindings** (`docs/plans/2026-08-20-grouped-connection-bindings-design.md`): the Configure page sources its binding rows from the flattened configuration, grouped per module with a per-group Object apply; CLI `instance bind-module`. Plus everything since 08-12 (arch-review Medium/Low waves, LocalDb 0.3.2→0.3.4, gRPC stream batching, bulk-deploy up-to-date skip, …). |
|
|
| Publish | `dotnet publish -c Release -r win-x64 --self-contained true` for Host (→ `app\`, 520 files) and CLI (→ `cli\`), zipped and pushed via `psbridge push` (SHA-256 verified), unpacked under `E:\psrem\releases\a24fa38e\`, mirrored to `\\wonder-app-vq03\E$\psrem-stage\a24fa38e` with robocopy. |
|
|
| EF migrations applied | 6, via `sqlcmd -C -b -I` on vq03 from an **idempotent delta script** (`dotnet ef migrations script 20260810100836_AddSmtpConfigurationTransport --idempotent`): `20260814235335_AddAuditLogAndNotificationCoveringIndexes`, `20260815004957_AlignAuditLogEventIdUniqueness`, `20260815154803_AddBundleUnlockAttemptTable`, `20260815165452_AddDeploymentRecordListSortIndex`, `20260819183326_AddSiteCallsListSortIndex`, `20260819201210_AddNotificationsListSortIndex`. All additive (indexes + one table + one index drop). |
|
|
| New required config | **`AuditLog:SiteWriter:DatabasePath`** (top-level section, Site nodes; `StartupValidator` refuses to boot without it since `2e4e41a8`) — added to `appsettings.Site.json` on both boxes as `E:\ApiInstall\ScadaBridge\site\data\auditlog.db`. The only appsettings delta across the 246 commits. |
|
|
| Data relocated | The site audit SQLite file had been written CWD-relative, i.e. to **`C:\Windows\System32\auditlog.db`** (the exact hazard `2e4e41a8` closed). Moved, with its `-wal`/`-shm`, into `site\data\` on both boxes while the Site service was stopped. |
|
|
| Preserved across swap | box `appsettings.json` / `appsettings.Central.json` / `appsettings.Site.json` (copied back from the backup; service env vars untouched), `central\data`, `site\data` (secrets.db, site-localdb.db). |
|
|
| DB backup | `ScadaBridge_pre-a24fa38e_20260820T162535.bak` (COPY_ONLY, SQL Server default backup dir on wonder-sql-vq03) |
|
|
| Binary backup | `E:\ApiInstall\ScadaBridge\app.bak.20260820T162659` + `cli.bak.20260820T162659` on each box (old `d4bb4e17` binaries + appsettings) |
|
|
|
|
### Sequence
|
|
Stop Site pair (both boxes) → stop Central pair → swap `app\`/`cli\` on vq04 then vq03 →
|
|
start Central pair → wait 20 s → start Site pair. Pairs were stopped and started **together**
|
|
(required since LocalDb Phase 2 — no rolling upgrade of a site pair).
|
|
|
|
### Pitfalls hit (for next time)
|
|
- A **full-history** `--idempotent` script fails on SQL Server batch-compile (`Invalid column
|
|
name 'Configuration'` from an early migration) — script **from the DB's current migration**.
|
|
- `sqlcmd` needs **`-I`** (QUOTED_IDENTIFIER ON) or the filtered/covering index migrations fail
|
|
with Msg 1934.
|
|
- Inside a `psbridge exec` single-quoted script, doubled `''` quotes are eaten — use `[char]39`.
|
|
- `Invoke-Command` output must be piped through `Out-String` to come back through psbridge.
|
|
|
|
### Verification
|
|
- `/health/ready` = **200 Healthy** on both centrals (database + akka-cluster Up, memberCount 2,
|
|
unreachable 0, all 5 required singletons) and both sites (akka-cluster Up, memberCount 2,
|
|
`localdb` Healthy with `replicationConnected=true`, backlog 0).
|
|
- Zero `[ERR]`/`[WRN]`/`[FTL]` on any of the four node logs after the new boot (the `[FTL]
|
|
host terminated unexpectedly` at 16:27:39 is the *old* build's stop).
|
|
- **Feature proven on the QA data that motivated it:** `/deployment/instances/1/configure`
|
|
(Z28061, CvdReactor) now lists **264 data-sourced attributes in 8 groups** — `(instance)` 16,
|
|
`DelmiaReceiver` 10, `LeftDelmiaReceiver` 10, `LeftMESReceiver` 26, `LeftReactorSide` 83
|
|
(incl. `LeakTest`), `RightDelmiaReceiver` 10, `RightMESReceiver` 26, `RightReactorSide` 83 —
|
|
all bound, every group's Object inferred and pre-filled (`Z28061`, `DelmiaReceiver_043`,
|
|
`Left_002.DelmiaReceiver`, `MESReceiver_023`, `Left_002`, `Right_002.DelmiaReceiver`,
|
|
`MESReceiver_022`, `Right_002`), `Test Bindings (264)`. Before this build the page showed
|
|
only the 16 direct attributes.
|
|
|
|
## 2. Rollback
|
|
- Binaries: on each box stop both services, rename `app` → `app.failed`, `app.bak.20260820T162659`
|
|
→ `app` (same for `cli`), start Central then Site. The old build tolerates the new (additive)
|
|
schema and ignores the added `AuditLog` key — but note it will again write `auditlog.db`
|
|
CWD-relative into System32.
|
|
- Schema: restore `ScadaBridge_pre-a24fa38e_20260820T162535.bak` (not needed for a binary rollback).
|
|
|
|
## 3. Outstanding
|
|
- None from this deploy. ZIPSEN (id 7) remains the known non-deployable draft instance.
|
|
|
|
## 4. Hotfix `da0bfe2b` — same day, 16:53 (QA only so far)
|
|
|
|
**Incident (reported on vq03 Z28061 MoveIn):** `SetAttribute('RightMESReceiver.MoveInCompleteFlag')
|
|
failed: Session session-736b2d47… was not found` while the attribute showed GOOD quality.
|
|
**Root cause (from both logs):** at 15:29:03 the gateway dashboard user `admin` (client
|
|
`10.181.200.169`) closed ScadaBridge's MxGateway session. The gateway completes a closed
|
|
session's event stream *gracefully* (OK status), and `MxGatewayDataConnection.RunEventLoopAsync`
|
|
only raised `Disconnected` on a *fault* — so the adapter stayed Connected on a dead session for
|
|
an hour (values frozen at last GOOD, every write `NotFound`; the 10 s `NotFound` heartbeat was
|
|
the unresolved-tag resolution probe re-subscribing on the dead session). The 16:28 upgrade
|
|
restart cleared it incidentally. A graceful gateway shutdown takes the same path.
|
|
|
|
**Fix:** `da0bfe2b` — a non-cancelled end of the event stream now signals `Disconnected`
|
|
(pinned by `MxGatewayDataConnectionReconnectTests.EventStreamGracefulCompletion_*`).
|
|
Deployed to all four QA nodes as a binary-only swap (no migrations, no config): backups
|
|
`app.bak.20260820T165357` / `cli.bak.20260820T165357` on both boxes; `/health/ready` 200
|
|
Healthy, 2/2 members, site replication connected. **Not yet on prod vd03.** Live gate still to
|
|
run: close the site's session from the gateway dashboard and confirm the site reconnects
|
|
(`MxGateway event stream ended without a fault … signalling disconnect` → reconnect → values resume).
|