# 2026-09-XX — QA (vq03/vq04) + prod (vd03) → shared inbound API key store (PLANNED)

**Status: PLANNED — not yet run.** This record is a script for the operator to execute on the
ZB servers and then fill in (date/time, log lines, results). Everything below the header table
is what is *expected* to happen, per the design and the runbook it must agree with; blanks are
marked `_____`.

**Build:** branch `inbound-api-key-shared-store` @ `<merge commit>` (not yet merged to `main`).
**Delta from the boxes' previous build:** library-only + one config key — pins
`ZB.MOM.WW.Auth` 0.2.1 → **0.3.0** (adds the `ZB.MOM.WW.Auth.ApiKeys.SqlServer` package) and adds
`ScadaBridge:InboundApi:ApiKeyStore:Provider`. `git diff --stat <prev>..<merge commit> -- '**/Migrations/*'`
expected empty — no ScadaBridge EF migration; the new `zbauth` schema is created by the library's
own migrator (`SqlServerApiKeyStoreMigrator`), not `dotnet ef`. Design: `docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`
· register row 108 (`docs/plans/2026-07-08-deferred-work-register.md`). Runbook this record must
agree with: `docs/operations/inbound-api-key-reissue.md` → *Central pairs*.

## Config change (both central nodes on each pair)

`appsettings.Central.json`, under `ScadaBridge:InboundApi`, add:

```json
"ApiKeyStore": {
  "Provider": "SqlServer"
}
```

No other key changes — `SqlServer:ConnectionString` is deliberately **left unset**, so it
defaults to `ScadaBridge:Database:ConfigurationDb` (never a `${secret:}` reference — the
connection string is the same one already used for method definitions and the config DB). No
new secret is distributed.

## Procedure (build/stage recipe per the 2026-08-21 records)

1. `dotnet publish` Host + CLI `-c Release -r win-x64 --self-contained true`; zip; `psbridge push
   --compress` to vq04 (digest verified); `Expand-Archive`; robocopy `/MIR` to
   `\\wonder-app-vq03\E$\psrem-stage\<merge commit>` and `\\wonder-app-vd03\E$\psrem-stage\<merge commit>`
   (file count + ProductVersion checked, per the existing recipe).
2. **Pre-check (QA only, before the swap):** confirm the app's SQL login has `CREATE SCHEMA`/
   `CREATE TABLE` rights on the QA configuration database. If not, have a DBA run
   `docs/operations/sql/zbauth-schema.sql` against it now — the migrator then finds the
   `zbauth.schema_version` row already present and skips DDL on both nodes' first boot.
3. **QA `_____`** — Site pair stopped, Central pair stopped (both boxes **together** — a
   clustered Central left with one node on `SqlServer` and one still importing would leave the
   pair briefly disagreeing on where keys live), `app`/`cli` renamed to `*.bak.<timestamp>` on
   both boxes, staged copy in, each box's three `appsettings*.json` restored **and then edited**
   to add the `ApiKeyStore.Provider` block above, Central pair started (+20 s), Site pair started
   (+45 s).
4. **Prod `_____`** — same on vd03 (single node): backup `*.bak.<timestamp>`, same appsettings
   edit, Central started, no site-pair step (`vd03` central is single-node; its site pair is
   unaffected by this change and follows the same stop/start cadence as any other binary swap on
   that box).

## Verification

Expect the following, matching `docs/operations/inbound-api-key-reissue.md` §*Central pairs*
steps 5–6:

| Check | Expected | Result |
|---|---|---|
| 10.220.158.2:8085 / :8084 (vq03) `/health/ready` | 200 Healthy, members=2 | `_____` |
| 10.220.158.3:8085 / :8084 (vq04) `/health/ready` | 200 Healthy, members=2 | `_____` |
| vd03 localhost:8085 / :8084 `/health/ready` | 200 Healthy, members=1 | `_____` |
| Boot log, QA node that imports first | `Inbound API key store: imported … N keys inserted` (or the docs/operations wording, `grep -i "api-key store import\|zbauth"`) | `_____` |
| Boot log, QA other node | `… keys already present` (0 inserted — its own file's keys already landed via the first node's insert-where-absent import, or it imports its own disjoint set) | `_____` |
| `inbound-api-keys.sqlite.imported` (+ `-wal.imported`/`-shm.imported`) present next to each QA node's data | present on both boxes | `_____` |
| `scadabridge security api-key list` against `.2:8085` | matches `.3:8085` (same key set) | `_____` |
| `scadabridge security api-key list` against `.3:8085` | matches `.2:8085` | `_____` |
| Prod boot log | `Inbound API key store: imported … KeysInserted=5` for the five existing keys (`mes`, `mes api key`, `delmia-mes-receivers`, `TestKey`, `JOE_TEST_KEY`) | `_____` |
| Prod `security api-key list` before vs. after | unchanged | `_____` |
| Boot warning **absent** on QA post-change | (was present pre-change: `Startup configuration warning: Inbound API keys use the per-node SQLite store on a 2-node central cluster; …`) — confirm it no longer logs | `_____` |
| Failover check, no file copy: mint a key via the active node, `curl -X POST /api/DelmiaRecipeDownload` with it aimed at the **non-active** node's port | 200, not 401 (standby relays; both nodes now verify against the same SQL rows even without a relay) | `_____` |
| Optional real failover: stop the active node, repeat the same call against the new active | 200 | `_____` |
| QA takeover time during the optional real failover | rig reference: `docker kill` takeover 28 s | `_____ s` |

Post-swap ERR/FTL log scan (all boxes, post-boot window): `_____`

## Rollback

Per host: stop the pair (QA: both boxes together) → revert `Provider` (delete the `ApiKeyStore`
block from `appsettings.Central.json`, or set it back to `Sqlite`) → rename
`inbound-api-keys.sqlite.imported` (and its `-wal.imported`/`-shm.imported` siblings) back to
`inbound-api-keys.sqlite` (`-wal`/`-shm`) on both nodes → start the pair. No SQL rows need
removing — the SQLite provider never reads the `zbauth` schema.

**Caveat:** any key minted, revoked, or rotated *while* on `SqlServer` is not in the SQLite
file and is lost on rollback. Before rolling back, list `zbauth.api_keys` (via
`scadabridge security api-key list` against either node, or a direct query using the
`SqlServer:Schema` above) and re-mint anything created after the cutover once the pair is back
on SQLite.

## Quarantine

Leave the `*.imported` files in place for **30 days** after the change is confirmed stable on
both QA and prod, then delete them (all three: `.imported`, `-wal.imported`, `-shm.imported`)
from every node's data volume. Do not delete them before that window — they are the only
rollback path for keys as of the cutover moment.

## Results

| Step | Date/time (local) | Node | Outcome | Log line |
|---|---|---|---|---|
| Pre-check: SQL login DDL rights | `_____` | — | `_____` | `_____` |
| QA stop (Site + Central, both boxes) | `_____` | vq03 + vq04 | `_____` | — |
| QA swap + appsettings edit | `_____` | vq03 | `_____` | — |
| QA swap + appsettings edit | `_____` | vq04 | `_____` | — |
| QA start (Central, both boxes) | `_____` | vq03 + vq04 | `_____` | `_____` |
| QA start (Site, both boxes) | `_____` | vq03 + vq04 | `_____` | — |
| QA import — first node | `_____` | `_____` | `_____` | `_____` |
| QA import — second node | `_____` | `_____` | `_____` | `_____` |
| Prod stop | `_____` | vd03 | `_____` | — |
| Prod swap + appsettings edit | `_____` | vd03 | `_____` | — |
| Prod start | `_____` | vd03 | `_____` | `_____` |
| Prod import | `_____` | vd03 | `_____` | `_____` |
| Failover relay check (no file copy) | `_____` | `_____` | `_____` | — |
| Optional real failover | `_____` | `_____` | `_____` takeover `_____ s` | — |

**Sign-off:** `_____` (name, date, "verified per §Verification above, no rollback needed")