ScadaBridge/docs/deployment/deployment-records/2026-09-XX-zb-servers-inbound-key-store.md

2026-09-XX — QA (vq03/vq04) + prod (vd03) → shared inbound API key store (PLANNED)

Status: PLANNED — not yet run. This record is a script for the operator to execute on the ZB servers and then fill in (date/time, log lines, results). Everything below the header table is what is expected to happen, per the design and the runbook it must agree with; blanks are marked _____.

Build: branch inbound-api-key-shared-store @ <merge commit> (not yet merged to main). Delta from the boxes' previous build: library-only + one config key — pins ZB.MOM.WW.Auth 0.2.1 → 0.3.0 (adds the ZB.MOM.WW.Auth.ApiKeys.SqlServer package) and adds ScadaBridge:InboundApi:ApiKeyStore:Provider. git diff --stat <prev>..<merge commit> -- '**/Migrations/*' expected empty — no ScadaBridge EF migration; the new zbauth schema is created by the library's own migrator (SqlServerApiKeyStoreMigrator), not dotnet ef. Design: docs/plans/2026-09-02-inbound-api-key-shared-store-design.md · register row 108 (docs/plans/2026-07-08-deferred-work-register.md). Runbook this record must agree with: docs/operations/inbound-api-key-reissue.md → Central pairs.

Config change (both central nodes on each pair)

appsettings.Central.json, under ScadaBridge:InboundApi, add:

"ApiKeyStore": {
  "Provider": "SqlServer"
}

No other key changes — SqlServer:ConnectionString is deliberately left unset, so it defaults to ScadaBridge:Database:ConfigurationDb (never a ${secret:} reference — the connection string is the same one already used for method definitions and the config DB). No new secret is distributed.

Procedure (build/stage recipe per the 2026-08-21 records)

1. dotnet publish Host + CLI -c Release -r win-x64 --self-contained true; zip; psbridge push --compress to vq04 (digest verified); Expand-Archive; robocopy /MIR to \\wonder-app-vq03\E$\psrem-stage\<merge commit> and \\wonder-app-vd03\E$\psrem-stage\<merge commit> (file count + ProductVersion checked, per the existing recipe).
2. Pre-check (QA only, before the swap): confirm the app's SQL login has CREATE SCHEMA/ CREATE TABLE rights on the QA configuration database. If not, have a DBA run docs/operations/sql/zbauth-schema.sql against it now — the migrator then finds the zbauth.schema_version row already present and skips DDL on both nodes' first boot.
3. QA _____ — Site pair stopped, Central pair stopped (both boxes together — a clustered Central left with one node on SqlServer and one still importing would leave the pair briefly disagreeing on where keys live), app/cli renamed to *.bak.<timestamp> on both boxes, staged copy in, each box's three appsettings*.json restored and then edited to add the ApiKeyStore.Provider block above, Central pair started (+20 s), Site pair started (+45 s).
4. Prod _____ — same on vd03 (single node): backup *.bak.<timestamp>, same appsettings edit, Central started, no site-pair step (vd03 central is single-node; its site pair is unaffected by this change and follows the same stop/start cadence as any other binary swap on that box).

Verification

Expect the following, matching docs/operations/inbound-api-key-reissue.md §Central pairs steps 5–6:

Check	Expected	Result
10.220.158.2:8085 / :8084 (vq03) /health/ready	200 Healthy, members=2	_____
10.220.158.3:8085 / :8084 (vq04) /health/ready	200 Healthy, members=2	_____
vd03 localhost:8085 / :8084 /health/ready	200 Healthy, members=1	_____
Boot log, QA node that imports first	Inbound API key store: imported … N keys inserted (or the docs/operations wording, grep -i "api-key store import|zbauth")	_____
Boot log, QA other node	… keys already present (0 inserted — its own file's keys already landed via the first node's insert-where-absent import, or it imports its own disjoint set)	_____
inbound-api-keys.sqlite.imported (+ -wal.imported/-shm.imported) present next to each QA node's data	present on both boxes	_____
scadabridge security api-key list against .2:8085	matches .3:8085 (same key set)	_____
scadabridge security api-key list against .3:8085	matches .2:8085	_____
Prod boot log	Inbound API key store: imported … KeysInserted=5 for the five existing keys (mes, mes api key, delmia-mes-receivers, TestKey, JOE_TEST_KEY)	_____
Prod security api-key list before vs. after	unchanged	_____
Boot warning absent on QA post-change	(was present pre-change: Startup configuration warning: Inbound API keys use the per-node SQLite store on a 2-node central cluster; …) — confirm it no longer logs	_____
Failover check, no file copy: mint a key via the active node, curl -X POST /api/DelmiaRecipeDownload with it aimed at the non-active node's port	200, not 401 (standby relays; both nodes now verify against the same SQL rows even without a relay)	_____
Optional real failover: stop the active node, repeat the same call against the new active	200	_____
QA takeover time during the optional real failover	rig reference: docker kill takeover 28 s	_____ s

Post-swap ERR/FTL log scan (all boxes, post-boot window): _____

Rollback

Per host: stop the pair (QA: both boxes together) → revert Provider (delete the ApiKeyStore block from appsettings.Central.json, or set it back to Sqlite) → rename inbound-api-keys.sqlite.imported (and its -wal.imported/-shm.imported siblings) back to inbound-api-keys.sqlite (-wal/-shm) on both nodes → start the pair. No SQL rows need removing — the SQLite provider never reads the zbauth schema.

Caveat: any key minted, revoked, or rotated while on SqlServer is not in the SQLite file and is lost on rollback. Before rolling back, list zbauth.api_keys (via scadabridge security api-key list against either node, or a direct query using the SqlServer:Schema above) and re-mint anything created after the cutover once the pair is back on SQLite.

Quarantine

Leave the *.imported files in place for 30 days after the change is confirmed stable on both QA and prod, then delete them (all three: .imported, -wal.imported, -shm.imported) from every node's data volume. Do not delete them before that window — they are the only rollback path for keys as of the cutover moment.

Results

Step	Date/time (local)	Node	Outcome	Log line
Pre-check: SQL login DDL rights	_____	—	_____	_____
QA stop (Site + Central, both boxes)	_____	vq03 + vq04	_____	—
QA swap + appsettings edit	_____	vq03	_____	—
QA swap + appsettings edit	_____	vq04	_____	—
QA start (Central, both boxes)	_____	vq03 + vq04	_____	_____
QA start (Site, both boxes)	_____	vq03 + vq04	_____	—
QA import — first node	_____	_____	_____	_____
QA import — second node	_____	_____	_____	_____
Prod stop	_____	vd03	_____	—
Prod swap + appsettings edit	_____	vd03	_____	—
Prod start	_____	vd03	_____	_____
Prod import	_____	vd03	_____	_____
Failover relay check (no file copy)	_____	_____	_____	—
Optional real failover	_____	_____	_____ takeover _____ s	—

Sign-off: _____ (name, date, "verified per §Verification above, no rollback needed")