docs: QA central pair moved to shared SQL key + secret stores; record the Windows import-rename defect

2026-10-22 15:56:25 -04:00
parent 6ed6800b84
commit 4bc124d73e
2 changed files with 48 additions and 10 deletions
@@ -85,14 +85,52 @@ The deploy ran from the Mac with the `winrm` CLI. The zip and SQL went over SMB,

 The indexes are additive, so `b3021e7c` runs against 58 migrations. For an exact DB, `RESTORE DATABASE [ScadaBridge] FROM DISK = 'F:\MSSQL\BACKUP\ScadaBridge_pre-3808b49f_20261022T152457.bak' WITH REPLACE` with everything stopped.

+## Follow-up, same day: shared key and secret stores on the central pair (15:45–15:50)
+
+**Pre-check (SMB copies of the per-node files):**
+- Inbound keys: vq03 held 3 keys (`MESTestKey`, `MES_Key`, `delmia-notifier-qa`); vq04 held **none**. Before this change, a failover to vq04 would 401 every inbound client.
+- Secrets: both central SQLite stores held **0** secrets, so there was nothing to migrate (the secrets store has no automatic importer).
+- SQL login: `db_owner` on `ScadaBridge`.
+
+**Change:**
+
+| Where | Change | How |
+|---|---|---|
+| `appsettings.Central.json` (both boxes) | `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer` (connection string defaults to the ConfigurationDb) and `Secrets:Store:Provider=SqlServer`, plus a `_comment_SharedStores` note | Edited on the Mac, pushed over SMB while Central was stopped |
+| `appsettings.json` (both boxes) | Removed the stale `Secrets:Replication` (incl. `Mode`) and `Secrets:GrpcHub` blocks and the dead `SqlServer` sync keys | Edited on the Mac, pushed over SMB while Central was stopped |
+| Central service registry env (both) | Added `Secrets__SqlServer__ConnectionString` | Copied on the box from `ScadaBridge__Database__ConfigurationDb`; the value never left the box. Site services untouched |
+
+Backups: `E:\psrem-stage\3808b49f\_envbak-ScadaBridge-Central-pre-sharedstores-<ts>.reg` and `E:\ApiInstall\ScadaBridge\_appsettings-keep-sharedstores-<ts>`, on each box.
+
+**Sequence:**
+1. Both centrals stopped together (15:45:42 / 15:45:55).
+2. vq03 started → ready 15:46:26; vq04 started → ready 15:46:39. The `zbauth` and `zbsecrets` schemas were created in `ScadaBridge`.
+3. vq03 imported **3 keys + 3 audit rows**; vq04 imported 0.
+
+**Defect found and fixed:** on both nodes the import's rename to `*.imported` failed with `IOException: … being used by another process`.
+- Cause: the library importer reads the file through a pooled `SqliteConnection`, whose handle outlives the connection. Windows refuses to rename an open file (Linux doesn't mind, so the rig never showed it).
+- Reproduced on the Windows build box: the rename fails without `SqliteConnection.ClearAllPools()` and succeeds with it.
+- Code fix: `6ed6800b`, which clears the pools before the rename, plus a regression test and a runbook note.
+- On QA the files were renamed by hand with each Central stopped:
+  - vq04 first. Its first attempt hit the same lock because its Host process was still exiting, and the script stopped before restarting it, so **vq04's Central was down 15:48:55–15:50:24**.
+  - vq03 at 15:49:07–15:49:28.
+  - These two windows overlapped, so **the central API was unavailable for about 20 s**. The sites were unaffected.
+
+**Verified:**
+- Both centrals ready; 2 members, 0 unreachable; vq03 active.
+- `security api-key list` returns the same 3 keys on **both** nodes.
+- An inbound call with a bogus key gets 401 `UNAUTHORIZED` on the active node.
+- No startup warnings after the last start (the Secrets-SQLite, inbound-key-SQLite and stale-key warnings are gone).
+- `central\data` on each box now holds `inbound-api-keys.sqlite{,-wal,-shm}.imported` and the (empty) `secrets.db`.
+
+**Rollback:**
+1. Restore the two appsettings files from `_appsettings-keep-sharedstores-<ts>` and import the `.reg` backup, or remove the new env value.
+2. Rename the `*.imported` files back.
+3. Restart both centrals.
+
+The `zbauth` and `zbsecrets` rows can stay; the SQLite path never reads them.
+
 ## Outstanding

-These pre-existing items now carry startup Warnings:
- **Shared stores.** Set both, then recreate:
-  - `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer` for inbound keys.
-  - `Secrets:Store:Provider=SqlServer` plus `Secrets:SqlServer:ConnectionString` for secrets.
-
-  Neither is needed for current QA operation: the site holds no deployed secrets.
- **Stale keys.** Remove the `Secrets:Replication` `Mode`/`GrpcHub` keys from both boxes' appsettings.json.
-
-Housekeeping after soak: older `app.bak.*`, `cli.bak.*`, `_appsettings-keep-*` and `E:\psrem-stage\*` folders on both boxes.
+- **Standby relay:** QA still has no `ScadaBridge:InboundApi:StandbyForwardPeerEndpoints`, so an inbound call sent to the standby answers 503 `STANDBY_NODE` instead of being relayed to the active node.
+- **Housekeeping after soak:** delete the `*.imported` key files; also the older `app.bak.*`, `cli.bak.*`, `_appsettings-keep-*` and `E:\psrem-stage\*` folders on both boxes.