docs(deployment): planned record for moving QA (vq03/vq04) and prod (vd03) central inbound key stores to SQL Server; plan doc no longer spells out the dev sa password
Claude-Session: https://claude.ai/code/session_013L3XdmVii9i5sY4FChU8pB
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,117 @@
|
||||
# 2026-09-XX — QA (vq03/vq04) + prod (vd03) → shared inbound API key store (PLANNED)
|
||||
|
||||
**Status: PLANNED — not yet run.** This record is a script for the operator to execute on the
|
||||
ZB servers and then fill in (date/time, log lines, results). Everything below the header table
|
||||
is what is *expected* to happen, per the design and the runbook it must agree with; blanks are
|
||||
marked `_____`.
|
||||
|
||||
**Build:** branch `inbound-api-key-shared-store` @ `<merge commit>` (not yet merged to `main`).
|
||||
**Delta from the boxes' previous build:** library-only + one config key — pins
|
||||
`ZB.MOM.WW.Auth` 0.2.1 → **0.3.0** (adds the `ZB.MOM.WW.Auth.ApiKeys.SqlServer` package) and adds
|
||||
`ScadaBridge:InboundApi:ApiKeyStore:Provider`. `git diff --stat <prev>..<merge commit> -- '**/Migrations/*'`
|
||||
expected empty — no ScadaBridge EF migration; the new `zbauth` schema is created by the library's
|
||||
own migrator (`SqlServerApiKeyStoreMigrator`), not `dotnet ef`. Design: `docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`
|
||||
· register row 108 (`docs/plans/2026-07-08-deferred-work-register.md`). Runbook this record must
|
||||
agree with: `docs/operations/inbound-api-key-reissue.md` → *Central pairs*.
|
||||
|
||||
## Config change (both central nodes on each pair)
|
||||
|
||||
`appsettings.Central.json`, under `ScadaBridge:InboundApi`, add:
|
||||
|
||||
```json
|
||||
"ApiKeyStore": {
|
||||
"Provider": "SqlServer"
|
||||
}
|
||||
```
|
||||
|
||||
No other key changes — `SqlServer:ConnectionString` is deliberately **left unset**, so it
|
||||
defaults to `ScadaBridge:Database:ConfigurationDb` (never a `${secret:}` reference — the
|
||||
connection string is the same one already used for method definitions and the config DB). No
|
||||
new secret is distributed.
|
||||
|
||||
## Procedure (build/stage recipe per the 2026-08-21 records)
|
||||
|
||||
1. `dotnet publish` Host + CLI `-c Release -r win-x64 --self-contained true`; zip; `psbridge push
|
||||
--compress` to vq04 (digest verified); `Expand-Archive`; robocopy `/MIR` to
|
||||
`\\wonder-app-vq03\E$\psrem-stage\<merge commit>` and `\\wonder-app-vd03\E$\psrem-stage\<merge commit>`
|
||||
(file count + ProductVersion checked, per the existing recipe).
|
||||
2. **Pre-check (QA only, before the swap):** confirm the app's SQL login has `CREATE SCHEMA`/
|
||||
`CREATE TABLE` rights on the QA configuration database. If not, have a DBA run
|
||||
`docs/operations/sql/zbauth-schema.sql` against it now — the migrator then finds the
|
||||
`zbauth.schema_version` row already present and skips DDL on both nodes' first boot.
|
||||
3. **QA `_____`** — Site pair stopped, Central pair stopped (both boxes **together** — a
|
||||
clustered Central left with one node on `SqlServer` and one still importing would leave the
|
||||
pair briefly disagreeing on where keys live), `app`/`cli` renamed to `*.bak.<timestamp>` on
|
||||
both boxes, staged copy in, each box's three `appsettings*.json` restored **and then edited**
|
||||
to add the `ApiKeyStore.Provider` block above, Central pair started (+20 s), Site pair started
|
||||
(+45 s).
|
||||
4. **Prod `_____`** — same on vd03 (single node): backup `*.bak.<timestamp>`, same appsettings
|
||||
edit, Central started, no site-pair step (`vd03` central is single-node; its site pair is
|
||||
unaffected by this change and follows the same stop/start cadence as any other binary swap on
|
||||
that box).
|
||||
|
||||
## Verification
|
||||
|
||||
Expect the following, matching `docs/operations/inbound-api-key-reissue.md` §*Central pairs*
|
||||
steps 5–6:
|
||||
|
||||
| Check | Expected | Result |
|
||||
|---|---|---|
|
||||
| 10.220.158.2:8085 / :8084 (vq03) `/health/ready` | 200 Healthy, members=2 | `_____` |
|
||||
| 10.220.158.3:8085 / :8084 (vq04) `/health/ready` | 200 Healthy, members=2 | `_____` |
|
||||
| vd03 localhost:8085 / :8084 `/health/ready` | 200 Healthy, members=1 | `_____` |
|
||||
| Boot log, QA node that imports first | `Inbound API key store: imported … N keys inserted` (or the docs/operations wording, `grep -i "api-key store import\|zbauth"`) | `_____` |
|
||||
| Boot log, QA other node | `… keys already present` (0 inserted — its own file's keys already landed via the first node's insert-where-absent import, or it imports its own disjoint set) | `_____` |
|
||||
| `inbound-api-keys.sqlite.imported` (+ `-wal.imported`/`-shm.imported`) present next to each QA node's data | present on both boxes | `_____` |
|
||||
| `scadabridge security api-key list` against `.2:8085` | matches `.3:8085` (same key set) | `_____` |
|
||||
| `scadabridge security api-key list` against `.3:8085` | matches `.2:8085` | `_____` |
|
||||
| Prod boot log | `Inbound API key store: imported … KeysInserted=5` for the five existing keys (`mes`, `mes api key`, `delmia-mes-receivers`, `TestKey`, `JOE_TEST_KEY`) | `_____` |
|
||||
| Prod `security api-key list` before vs. after | unchanged | `_____` |
|
||||
| Boot warning **absent** on QA post-change | (was present pre-change: `Startup configuration warning: Inbound API keys use the per-node SQLite store on a 2-node central cluster; …`) — confirm it no longer logs | `_____` |
|
||||
| Failover check, no file copy: mint a key via the active node, `curl -X POST /api/DelmiaRecipeDownload` with it aimed at the **non-active** node's port | 200, not 401 (standby relays; both nodes now verify against the same SQL rows even without a relay) | `_____` |
|
||||
| Optional real failover: stop the active node, repeat the same call against the new active | 200 | `_____` |
|
||||
| QA takeover time during the optional real failover | rig reference: `docker kill` takeover 28 s | `_____ s` |
|
||||
|
||||
Post-swap ERR/FTL log scan (all boxes, post-boot window): `_____`
|
||||
|
||||
## Rollback
|
||||
|
||||
Per host: stop the pair (QA: both boxes together) → revert `Provider` (delete the `ApiKeyStore`
|
||||
block from `appsettings.Central.json`, or set it back to `Sqlite`) → rename
|
||||
`inbound-api-keys.sqlite.imported` (and its `-wal.imported`/`-shm.imported` siblings) back to
|
||||
`inbound-api-keys.sqlite` (`-wal`/`-shm`) on both nodes → start the pair. No SQL rows need
|
||||
removing — the SQLite provider never reads the `zbauth` schema.
|
||||
|
||||
**Caveat:** any key minted, revoked, or rotated *while* on `SqlServer` is not in the SQLite
|
||||
file and is lost on rollback. Before rolling back, list `zbauth.api_keys` (via
|
||||
`scadabridge security api-key list` against either node, or a direct query using the
|
||||
`SqlServer:Schema` above) and re-mint anything created after the cutover once the pair is back
|
||||
on SQLite.
|
||||
|
||||
## Quarantine
|
||||
|
||||
Leave the `*.imported` files in place for **30 days** after the change is confirmed stable on
|
||||
both QA and prod, then delete them (all three: `.imported`, `-wal.imported`, `-shm.imported`)
|
||||
from every node's data volume. Do not delete them before that window — they are the only
|
||||
rollback path for keys as of the cutover moment.
|
||||
|
||||
## Results
|
||||
|
||||
| Step | Date/time (local) | Node | Outcome | Log line |
|
||||
|---|---|---|---|---|
|
||||
| Pre-check: SQL login DDL rights | `_____` | — | `_____` | `_____` |
|
||||
| QA stop (Site + Central, both boxes) | `_____` | vq03 + vq04 | `_____` | — |
|
||||
| QA swap + appsettings edit | `_____` | vq03 | `_____` | — |
|
||||
| QA swap + appsettings edit | `_____` | vq04 | `_____` | — |
|
||||
| QA start (Central, both boxes) | `_____` | vq03 + vq04 | `_____` | `_____` |
|
||||
| QA start (Site, both boxes) | `_____` | vq03 + vq04 | `_____` | — |
|
||||
| QA import — first node | `_____` | `_____` | `_____` | `_____` |
|
||||
| QA import — second node | `_____` | `_____` | `_____` | `_____` |
|
||||
| Prod stop | `_____` | vd03 | `_____` | — |
|
||||
| Prod swap + appsettings edit | `_____` | vd03 | `_____` | — |
|
||||
| Prod start | `_____` | vd03 | `_____` | `_____` |
|
||||
| Prod import | `_____` | vd03 | `_____` | `_____` |
|
||||
| Failover relay check (no file copy) | `_____` | `_____` | `_____` | — |
|
||||
| Optional real failover | `_____` | `_____` | `_____` takeover `_____ s` | — |
|
||||
|
||||
**Sign-off:** `_____` (name, date, "verified per §Verification above, no rollback needed")
|
||||
@@ -15,7 +15,7 @@
|
||||
- Secrets' SQL Server pattern to copy: `~/Desktop/scadaproj/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/` — `SecretsSqlServerConnectionFactory`, `SqlServerSecretsSchema` (`CREATE SCHEMA` via `EXEC`, `IF NOT EXISTS` tables), `SqlServerSecretsStoreMigrator` (retry on 1205/2714/1913/2627, `MaxMigrationRetries = 5`), and the DI extension's "register before `AddZbSecrets`" comment. Live tests: `tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/Live/LiveSqlServerFixture.cs` gated on `SECRETS_SQLSERVER_CONNSTR`.
|
||||
- HMAC-SHA256 hashes are 32 bytes (`ApiKeySecretHasher.Hash`). Scopes are serialised by `ScopeSerializer.Serialize/Deserialize` — reuse it, do not invent a second format.
|
||||
- ScadaBridge touchpoints: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs:427-470` (store defaults + `AddZbApiKeyAuth` + `ApiKeyAdminCommands` composition), `Directory.Packages.props:101-104` (four `0.2.1` pins), `tests/ZB.MOM.WW.ScadaBridge.Security.Tests/LibraryInboundApiKeyAdminTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/EndpointExtensionsTests.cs:753-758` and `EndpointContentTypeTests.cs:157` (construct `ApiKeyAdminCommands` with the SQLite migrator — keep compiling via the retained ctor).
|
||||
- Rig: both centrals reach `scadabridge-mssql` with `ScadaBridge__Database__ConfigurationDb` (env in `docker/docker-compose.yml`, user `scadabridge_app`, which already created the `zbsecrets` schema in `ZbSecretsHub` on 2026-08-07, so it holds DDL rights). Local MSSQL for library tests: `Server=localhost,1433;User Id=sa;Password=ScadaBridge_Dev1#;TrustServerCertificate=true` (`infra/docker-compose.yml`).
|
||||
- Rig: both centrals reach `scadabridge-mssql` with `ScadaBridge__Database__ConfigurationDb` (env in `docker/docker-compose.yml`, user `scadabridge_app`, which already created the `zbsecrets` schema in `ZbSecretsHub` on 2026-08-07, so it holds DDL rights). Local MSSQL for library tests: `Server=localhost,1433;User Id=sa;Password=<the infra/ MSSQL_SA_PASSWORD>;TrustServerCertificate=true` (`infra/docker-compose.yml`).
|
||||
- Register row 108 already records the defect; row 107 is the unrelated `DelmiaRecv` item. Do not add a new row — close 108.
|
||||
- The executing session's `docker/sync-inbound-api-keys.sh` and the preflight in `docker/verify-delmia-notifier.sh` exist because of this defect; Task 14 removes the script and the preflight's dependency on it.
|
||||
|
||||
@@ -213,7 +213,7 @@ git commit -m "feat(auth.apikeys.sqlserver): package skeleton, options (schema d
|
||||
- `Migrate_Concurrently_FromEightTasks_AllSucceed` — `Task.WhenAll` of 8 migrators on a virgin database; this is the case that bit Secrets live.
|
||||
- `Migrate_WhenStoredVersionIsNewer_Throws` — write `version = 99`, expect `AuthStoreMigrationException` (reuse the existing exception type from `SqliteAuthStoreMigrator.cs`).
|
||||
|
||||
**Step 2: Run** with `AUTH_SQLSERVER_CONNSTR="Server=localhost,1433;User Id=sa;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"` (start `infra/` first: `docker compose -f ~/Desktop/ScadaBridge/infra/docker-compose.yml up -d scadabridge-mssql`). Expected: compile failure (types missing). Without the variable the tests must report **Skipped**, not failed.
|
||||
**Step 2: Run** with `AUTH_SQLSERVER_CONNSTR="Server=localhost,1433;User Id=sa;Password=<the infra/ MSSQL_SA_PASSWORD>;TrustServerCertificate=true"` (start `infra/` first: `docker compose -f ~/Desktop/ScadaBridge/infra/docker-compose.yml up -d scadabridge-mssql`). Expected: compile failure (types missing). Without the variable the tests must report **Skipped**, not failed.
|
||||
|
||||
**Step 3: Implement**
|
||||
|
||||
|
||||
@@ -92,7 +92,7 @@
|
||||
{
|
||||
"id": 11,
|
||||
"subject": "Task 11: Docker rigs on the shared store; delete sync-inbound-api-keys.sh; drill without copying",
|
||||
"status": "in_progress",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
9
|
||||
],
|
||||
@@ -101,7 +101,7 @@
|
||||
{
|
||||
"id": 12,
|
||||
"subject": "Task 12: Docs, runbook, DBA SQL, env inventory, CLAUDE.md, register row 108, umbrella index",
|
||||
"status": "in_progress",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
9
|
||||
],
|
||||
@@ -110,7 +110,7 @@
|
||||
{
|
||||
"id": 13,
|
||||
"subject": "Task 13: Planned deployment record for QA vq03/vq04 and prod vd03",
|
||||
"status": "pending",
|
||||
"status": "in_progress",
|
||||
"blockedBy": [
|
||||
12
|
||||
],
|
||||
|
||||
Reference in New Issue
Block a user