diff --git a/docs/deployment/deployment-records/2026-09-XX-zb-servers-inbound-key-store.md b/docs/deployment/deployment-records/2026-09-XX-zb-servers-inbound-key-store.md new file mode 100644 index 00000000..29be205f --- /dev/null +++ b/docs/deployment/deployment-records/2026-09-XX-zb-servers-inbound-key-store.md @@ -0,0 +1,117 @@ +# 2026-09-XX — QA (vq03/vq04) + prod (vd03) → shared inbound API key store (PLANNED) + +**Status: PLANNED — not yet run.** This record is a script for the operator to execute on the +ZB servers and then fill in (date/time, log lines, results). Everything below the header table +is what is *expected* to happen, per the design and the runbook it must agree with; blanks are +marked `_____`. + +**Build:** branch `inbound-api-key-shared-store` @ `` (not yet merged to `main`). +**Delta from the boxes' previous build:** library-only + one config key — pins +`ZB.MOM.WW.Auth` 0.2.1 → **0.3.0** (adds the `ZB.MOM.WW.Auth.ApiKeys.SqlServer` package) and adds +`ScadaBridge:InboundApi:ApiKeyStore:Provider`. `git diff --stat .. -- '**/Migrations/*'` +expected empty — no ScadaBridge EF migration; the new `zbauth` schema is created by the library's +own migrator (`SqlServerApiKeyStoreMigrator`), not `dotnet ef`. Design: `docs/plans/2026-09-02-inbound-api-key-shared-store-design.md` +· register row 108 (`docs/plans/2026-07-08-deferred-work-register.md`). Runbook this record must +agree with: `docs/operations/inbound-api-key-reissue.md` → *Central pairs*. + +## Config change (both central nodes on each pair) + +`appsettings.Central.json`, under `ScadaBridge:InboundApi`, add: + +```json +"ApiKeyStore": { + "Provider": "SqlServer" +} +``` + +No other key changes — `SqlServer:ConnectionString` is deliberately **left unset**, so it +defaults to `ScadaBridge:Database:ConfigurationDb` (never a `${secret:}` reference — the +connection string is the same one already used for method definitions and the config DB). No +new secret is distributed. + +## Procedure (build/stage recipe per the 2026-08-21 records) + +1. `dotnet publish` Host + CLI `-c Release -r win-x64 --self-contained true`; zip; `psbridge push + --compress` to vq04 (digest verified); `Expand-Archive`; robocopy `/MIR` to + `\\wonder-app-vq03\E$\psrem-stage\` and `\\wonder-app-vd03\E$\psrem-stage\` + (file count + ProductVersion checked, per the existing recipe). +2. **Pre-check (QA only, before the swap):** confirm the app's SQL login has `CREATE SCHEMA`/ + `CREATE TABLE` rights on the QA configuration database. If not, have a DBA run + `docs/operations/sql/zbauth-schema.sql` against it now — the migrator then finds the + `zbauth.schema_version` row already present and skips DDL on both nodes' first boot. +3. **QA `_____`** — Site pair stopped, Central pair stopped (both boxes **together** — a + clustered Central left with one node on `SqlServer` and one still importing would leave the + pair briefly disagreeing on where keys live), `app`/`cli` renamed to `*.bak.` on + both boxes, staged copy in, each box's three `appsettings*.json` restored **and then edited** + to add the `ApiKeyStore.Provider` block above, Central pair started (+20 s), Site pair started + (+45 s). +4. **Prod `_____`** — same on vd03 (single node): backup `*.bak.`, same appsettings + edit, Central started, no site-pair step (`vd03` central is single-node; its site pair is + unaffected by this change and follows the same stop/start cadence as any other binary swap on + that box). + +## Verification + +Expect the following, matching `docs/operations/inbound-api-key-reissue.md` §*Central pairs* +steps 5–6: + +| Check | Expected | Result | +|---|---|---| +| 10.220.158.2:8085 / :8084 (vq03) `/health/ready` | 200 Healthy, members=2 | `_____` | +| 10.220.158.3:8085 / :8084 (vq04) `/health/ready` | 200 Healthy, members=2 | `_____` | +| vd03 localhost:8085 / :8084 `/health/ready` | 200 Healthy, members=1 | `_____` | +| Boot log, QA node that imports first | `Inbound API key store: imported … N keys inserted` (or the docs/operations wording, `grep -i "api-key store import\|zbauth"`) | `_____` | +| Boot log, QA other node | `… keys already present` (0 inserted — its own file's keys already landed via the first node's insert-where-absent import, or it imports its own disjoint set) | `_____` | +| `inbound-api-keys.sqlite.imported` (+ `-wal.imported`/`-shm.imported`) present next to each QA node's data | present on both boxes | `_____` | +| `scadabridge security api-key list` against `.2:8085` | matches `.3:8085` (same key set) | `_____` | +| `scadabridge security api-key list` against `.3:8085` | matches `.2:8085` | `_____` | +| Prod boot log | `Inbound API key store: imported … KeysInserted=5` for the five existing keys (`mes`, `mes api key`, `delmia-mes-receivers`, `TestKey`, `JOE_TEST_KEY`) | `_____` | +| Prod `security api-key list` before vs. after | unchanged | `_____` | +| Boot warning **absent** on QA post-change | (was present pre-change: `Startup configuration warning: Inbound API keys use the per-node SQLite store on a 2-node central cluster; …`) — confirm it no longer logs | `_____` | +| Failover check, no file copy: mint a key via the active node, `curl -X POST /api/DelmiaRecipeDownload` with it aimed at the **non-active** node's port | 200, not 401 (standby relays; both nodes now verify against the same SQL rows even without a relay) | `_____` | +| Optional real failover: stop the active node, repeat the same call against the new active | 200 | `_____` | +| QA takeover time during the optional real failover | rig reference: `docker kill` takeover 28 s | `_____ s` | + +Post-swap ERR/FTL log scan (all boxes, post-boot window): `_____` + +## Rollback + +Per host: stop the pair (QA: both boxes together) → revert `Provider` (delete the `ApiKeyStore` +block from `appsettings.Central.json`, or set it back to `Sqlite`) → rename +`inbound-api-keys.sqlite.imported` (and its `-wal.imported`/`-shm.imported` siblings) back to +`inbound-api-keys.sqlite` (`-wal`/`-shm`) on both nodes → start the pair. No SQL rows need +removing — the SQLite provider never reads the `zbauth` schema. + +**Caveat:** any key minted, revoked, or rotated *while* on `SqlServer` is not in the SQLite +file and is lost on rollback. Before rolling back, list `zbauth.api_keys` (via +`scadabridge security api-key list` against either node, or a direct query using the +`SqlServer:Schema` above) and re-mint anything created after the cutover once the pair is back +on SQLite. + +## Quarantine + +Leave the `*.imported` files in place for **30 days** after the change is confirmed stable on +both QA and prod, then delete them (all three: `.imported`, `-wal.imported`, `-shm.imported`) +from every node's data volume. Do not delete them before that window — they are the only +rollback path for keys as of the cutover moment. + +## Results + +| Step | Date/time (local) | Node | Outcome | Log line | +|---|---|---|---|---| +| Pre-check: SQL login DDL rights | `_____` | — | `_____` | `_____` | +| QA stop (Site + Central, both boxes) | `_____` | vq03 + vq04 | `_____` | — | +| QA swap + appsettings edit | `_____` | vq03 | `_____` | — | +| QA swap + appsettings edit | `_____` | vq04 | `_____` | — | +| QA start (Central, both boxes) | `_____` | vq03 + vq04 | `_____` | `_____` | +| QA start (Site, both boxes) | `_____` | vq03 + vq04 | `_____` | — | +| QA import — first node | `_____` | `_____` | `_____` | `_____` | +| QA import — second node | `_____` | `_____` | `_____` | `_____` | +| Prod stop | `_____` | vd03 | `_____` | — | +| Prod swap + appsettings edit | `_____` | vd03 | `_____` | — | +| Prod start | `_____` | vd03 | `_____` | `_____` | +| Prod import | `_____` | vd03 | `_____` | `_____` | +| Failover relay check (no file copy) | `_____` | `_____` | `_____` | — | +| Optional real failover | `_____` | `_____` | `_____` takeover `_____ s` | — | + +**Sign-off:** `_____` (name, date, "verified per §Verification above, no rollback needed") diff --git a/docs/plans/2026-09-02-inbound-api-key-shared-store.md b/docs/plans/2026-09-02-inbound-api-key-shared-store.md index 001460f2..3e9efc1b 100644 --- a/docs/plans/2026-09-02-inbound-api-key-shared-store.md +++ b/docs/plans/2026-09-02-inbound-api-key-shared-store.md @@ -15,7 +15,7 @@ - Secrets' SQL Server pattern to copy: `~/Desktop/scadaproj/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/` — `SecretsSqlServerConnectionFactory`, `SqlServerSecretsSchema` (`CREATE SCHEMA` via `EXEC`, `IF NOT EXISTS` tables), `SqlServerSecretsStoreMigrator` (retry on 1205/2714/1913/2627, `MaxMigrationRetries = 5`), and the DI extension's "register before `AddZbSecrets`" comment. Live tests: `tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/Live/LiveSqlServerFixture.cs` gated on `SECRETS_SQLSERVER_CONNSTR`. - HMAC-SHA256 hashes are 32 bytes (`ApiKeySecretHasher.Hash`). Scopes are serialised by `ScopeSerializer.Serialize/Deserialize` — reuse it, do not invent a second format. - ScadaBridge touchpoints: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs:427-470` (store defaults + `AddZbApiKeyAuth` + `ApiKeyAdminCommands` composition), `Directory.Packages.props:101-104` (four `0.2.1` pins), `tests/ZB.MOM.WW.ScadaBridge.Security.Tests/LibraryInboundApiKeyAdminTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.InboundAPI.Tests/EndpointExtensionsTests.cs:753-758` and `EndpointContentTypeTests.cs:157` (construct `ApiKeyAdminCommands` with the SQLite migrator — keep compiling via the retained ctor). -- Rig: both centrals reach `scadabridge-mssql` with `ScadaBridge__Database__ConfigurationDb` (env in `docker/docker-compose.yml`, user `scadabridge_app`, which already created the `zbsecrets` schema in `ZbSecretsHub` on 2026-08-07, so it holds DDL rights). Local MSSQL for library tests: `Server=localhost,1433;User Id=sa;Password=ScadaBridge_Dev1#;TrustServerCertificate=true` (`infra/docker-compose.yml`). +- Rig: both centrals reach `scadabridge-mssql` with `ScadaBridge__Database__ConfigurationDb` (env in `docker/docker-compose.yml`, user `scadabridge_app`, which already created the `zbsecrets` schema in `ZbSecretsHub` on 2026-08-07, so it holds DDL rights). Local MSSQL for library tests: `Server=localhost,1433;User Id=sa;Password=;TrustServerCertificate=true` (`infra/docker-compose.yml`). - Register row 108 already records the defect; row 107 is the unrelated `DelmiaRecv` item. Do not add a new row — close 108. - The executing session's `docker/sync-inbound-api-keys.sh` and the preflight in `docker/verify-delmia-notifier.sh` exist because of this defect; Task 14 removes the script and the preflight's dependency on it. @@ -213,7 +213,7 @@ git commit -m "feat(auth.apikeys.sqlserver): package skeleton, options (schema d - `Migrate_Concurrently_FromEightTasks_AllSucceed` — `Task.WhenAll` of 8 migrators on a virgin database; this is the case that bit Secrets live. - `Migrate_WhenStoredVersionIsNewer_Throws` — write `version = 99`, expect `AuthStoreMigrationException` (reuse the existing exception type from `SqliteAuthStoreMigrator.cs`). -**Step 2: Run** with `AUTH_SQLSERVER_CONNSTR="Server=localhost,1433;User Id=sa;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"` (start `infra/` first: `docker compose -f ~/Desktop/ScadaBridge/infra/docker-compose.yml up -d scadabridge-mssql`). Expected: compile failure (types missing). Without the variable the tests must report **Skipped**, not failed. +**Step 2: Run** with `AUTH_SQLSERVER_CONNSTR="Server=localhost,1433;User Id=sa;Password=;TrustServerCertificate=true"` (start `infra/` first: `docker compose -f ~/Desktop/ScadaBridge/infra/docker-compose.yml up -d scadabridge-mssql`). Expected: compile failure (types missing). Without the variable the tests must report **Skipped**, not failed. **Step 3: Implement** diff --git a/docs/plans/2026-09-02-inbound-api-key-shared-store.md.tasks.json b/docs/plans/2026-09-02-inbound-api-key-shared-store.md.tasks.json index 91283d10..3a185dc8 100644 --- a/docs/plans/2026-09-02-inbound-api-key-shared-store.md.tasks.json +++ b/docs/plans/2026-09-02-inbound-api-key-shared-store.md.tasks.json @@ -92,7 +92,7 @@ { "id": 11, "subject": "Task 11: Docker rigs on the shared store; delete sync-inbound-api-keys.sh; drill without copying", - "status": "in_progress", + "status": "completed", "blockedBy": [ 9 ], @@ -101,7 +101,7 @@ { "id": 12, "subject": "Task 12: Docs, runbook, DBA SQL, env inventory, CLAUDE.md, register row 108, umbrella index", - "status": "in_progress", + "status": "completed", "blockedBy": [ 9 ], @@ -110,7 +110,7 @@ { "id": 13, "subject": "Task 13: Planned deployment record for QA vq03/vq04 and prod vd03", - "status": "pending", + "status": "in_progress", "blockedBy": [ 12 ],