docs: QA central pair moved to shared SQL key + secret stores; record the Windows import-rename defect
This commit is contained in:
Joseph Doherty
@@ -193,7 +193,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- LDAP failure: new logins fail; active sessions continue with current roles.
|
||||
- Load balancer in front of central UI; cookie-embedded JWT + shared Data Protection keys for failover transparency.
|
||||
- Two-person MxGateway secured writes (M7): two new global roles — `Operator` (initiates) + `Verifier` (approves) — added alongside the canonical `Administrator`/`Designer`/`Deployer`/`Viewer`, with `RequireOperator`/`RequireVerifier` policies. An Operator submits a secured write from the Central UI Secured Writes page (`/operations/secured-writes`); it stays a `Pending` `PendingSecuredWrite` row until a *distinct* Verifier approves it (no-self-approval enforced server-side in the ManagementActor, plus a compare-and-swap race guard). Approval relays a `WriteTagRequest` to the site MxGateway; MxGateway-protocol connections only; each lifecycle event (submit/approve/reject/execute) emits a best-effort `AuditChannel.SecuredWrite` / `AuditKind.SecuredWrite*` central-direct-write row sharing the row id as `CorrelationId`. (SecuredWrite audit rows stamp `SourceNode` via `ICentralAuditWriter`/`INodeIdentityProvider`.) Pending secured writes expire server-side after a configurable TTL (`ManagementServiceOptions.SecuredWritePendingTtl`, default 24 h): an overdue `Pending` row is CAS'd to `Expired` (never relayed) — enforced at approve/reject and swept opportunistically on list (arch-review S2, `AuditKind.SecuredWriteExpire`).
|
||||
- **Inbound API key store — per node by default, shared on central pairs (2026-09-02, register row 108).** The shared `ZB.MOM.WW.Auth.ApiKeys` library stores inbound keys in **SQLite per node** unless `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer`, which points both central nodes at the same `zbauth` schema in the configuration database (library `ZB.MOM.WW.Auth.ApiKeys.SqlServer` 0.3.0) — under `SqlServer`, create/revoke/rotate/scope changes are cluster-wide immediately; a node's existing SQLite file is imported once at boot (insert-where-absent, safe on both nodes of a pair) and renamed `*.imported`. Central pairs **MUST** run `Provider=SqlServer` — a clustered Central left on `Sqlite` boots fine but `StartupValidator` logs a Warning, because a failover otherwise 401s every inbound client until keys are re-issued on the survivor. Both docker rigs (`docker/`, `docker-env2/`) run `SqlServer`; QA (`vq03`/`vq04`) and prod (`vd03`) appsettings are user-held and their rollout is a deployment record, not yet applied. Design: `docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`; runbook: `docs/operations/inbound-api-key-reissue.md` → *Central pairs*.
|
||||
- **Inbound API key store — per node by default, shared on central pairs (2026-09-02, register row 108).** The shared `ZB.MOM.WW.Auth.ApiKeys` library stores inbound keys in **SQLite per node** unless `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer`, which points both central nodes at the same `zbauth` schema in the configuration database (library `ZB.MOM.WW.Auth.ApiKeys.SqlServer` 0.3.0) — under `SqlServer`, create/revoke/rotate/scope changes are cluster-wide immediately; a node's existing SQLite file is imported once at boot (insert-where-absent, safe on both nodes of a pair) and renamed `*.imported`. Central pairs **MUST** run `Provider=SqlServer` — a clustered Central left on `Sqlite` boots fine but `StartupValidator` logs a Warning, because a failover otherwise 401s every inbound client until keys are re-issued on the survivor. Both docker rigs (`docker/`, `docker-env2/`) run `SqlServer`; QA (`vq03`/`vq04`) runs `SqlServer` for both inbound keys and secrets since 2026-10-22 (deployment record `2026-10-22-wonder-app-vq03-vq04-3808b49f.md`); prod (`vd03`, single node) is not switched. The boot import's `*.imported` rename needs the importer's pooled SQLite handle released first — Windows refuses to rename an open file (fixed `6ed6800b`; pre-fix Windows builds need the manual rename in the runbook). Design: `docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`; runbook: `docs/operations/inbound-api-key-reissue.md` → *Central pairs*.
|
||||
|
||||
### Cluster & Failover
|
||||
- **`auto-down` downing strategy (decision 2026-07-21 — availability over partition-safety).** Akka's `AutoDowning` provider, `auto-down-unreachable-after` = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. `keep-oldest` remains a supported `SplitBrainResolverStrategy` value (partition-safe, but an oldest-crash is a total outage — Akka's `down-if-alone` only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: `docs/plans/2026-07-21-auto-down-availability-decision.md`. **Ignore the boot-time Akka warning claiming "keep-majority will be used instead"** — drill-verified benign (2026-08-15, decision-doc postscript): the substitution does not happen with the explicit `AutoDowning` provider-class `BuildHocon` sets. But `AutoDowning` IS deprecated, so **every Akka.NET version bump must re-run `DRILL_MODE=active bash docker/failover-drill.sh`** (register row 70).
|
||||
|
||||
@@ -85,14 +85,52 @@ The deploy ran from the Mac with the `winrm` CLI. The zip and SQL went over SMB,
|
||||
|
||||
The indexes are additive, so `b3021e7c` runs against 58 migrations. For an exact DB, `RESTORE DATABASE [ScadaBridge] FROM DISK = 'F:\MSSQL\BACKUP\ScadaBridge_pre-3808b49f_20261022T152457.bak' WITH REPLACE` with everything stopped.
|
||||
|
||||
## Follow-up, same day: shared key and secret stores on the central pair (15:45–15:50)
|
||||
|
||||
**Pre-check (SMB copies of the per-node files):**
|
||||
- Inbound keys: vq03 held 3 keys (`MESTestKey`, `MES_Key`, `delmia-notifier-qa`); vq04 held **none**. Before this change, a failover to vq04 would 401 every inbound client.
|
||||
- Secrets: both central SQLite stores held **0** secrets, so there was nothing to migrate (the secrets store has no automatic importer).
|
||||
- SQL login: `db_owner` on `ScadaBridge`.
|
||||
|
||||
**Change:**
|
||||
|
||||
| Where | Change | How |
|
||||
|---|---|---|
|
||||
| `appsettings.Central.json` (both boxes) | `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer` (connection string defaults to the ConfigurationDb) and `Secrets:Store:Provider=SqlServer`, plus a `_comment_SharedStores` note | Edited on the Mac, pushed over SMB while Central was stopped |
|
||||
| `appsettings.json` (both boxes) | Removed the stale `Secrets:Replication` (incl. `Mode`) and `Secrets:GrpcHub` blocks and the dead `SqlServer` sync keys | Edited on the Mac, pushed over SMB while Central was stopped |
|
||||
| Central service registry env (both) | Added `Secrets__SqlServer__ConnectionString` | Copied on the box from `ScadaBridge__Database__ConfigurationDb`; the value never left the box. Site services untouched |
|
||||
|
||||
Backups: `E:\psrem-stage\3808b49f\_envbak-ScadaBridge-Central-pre-sharedstores-<ts>.reg` and `E:\ApiInstall\ScadaBridge\_appsettings-keep-sharedstores-<ts>`, on each box.
|
||||
|
||||
**Sequence:**
|
||||
1. Both centrals stopped together (15:45:42 / 15:45:55).
|
||||
2. vq03 started → ready 15:46:26; vq04 started → ready 15:46:39. The `zbauth` and `zbsecrets` schemas were created in `ScadaBridge`.
|
||||
3. vq03 imported **3 keys + 3 audit rows**; vq04 imported 0.
|
||||
|
||||
**Defect found and fixed:** on both nodes the import's rename to `*.imported` failed with `IOException: … being used by another process`.
|
||||
- Cause: the library importer reads the file through a pooled `SqliteConnection`, whose handle outlives the connection. Windows refuses to rename an open file (Linux doesn't mind, so the rig never showed it).
|
||||
- Reproduced on the Windows build box: the rename fails without `SqliteConnection.ClearAllPools()` and succeeds with it.
|
||||
- Code fix: `6ed6800b`, which clears the pools before the rename, plus a regression test and a runbook note.
|
||||
- On QA the files were renamed by hand with each Central stopped:
|
||||
- vq04 first. Its first attempt hit the same lock because its Host process was still exiting, and the script stopped before restarting it, so **vq04's Central was down 15:48:55–15:50:24**.
|
||||
- vq03 at 15:49:07–15:49:28.
|
||||
- These two windows overlapped, so **the central API was unavailable for about 20 s**. The sites were unaffected.
|
||||
|
||||
**Verified:**
|
||||
- Both centrals ready; 2 members, 0 unreachable; vq03 active.
|
||||
- `security api-key list` returns the same 3 keys on **both** nodes.
|
||||
- An inbound call with a bogus key gets 401 `UNAUTHORIZED` on the active node.
|
||||
- No startup warnings after the last start (the Secrets-SQLite, inbound-key-SQLite and stale-key warnings are gone).
|
||||
- `central\data` on each box now holds `inbound-api-keys.sqlite{,-wal,-shm}.imported` and the (empty) `secrets.db`.
|
||||
|
||||
**Rollback:**
|
||||
1. Restore the two appsettings files from `_appsettings-keep-sharedstores-<ts>` and import the `.reg` backup, or remove the new env value.
|
||||
2. Rename the `*.imported` files back.
|
||||
3. Restart both centrals.
|
||||
|
||||
The `zbauth` and `zbsecrets` rows can stay; the SQLite path never reads them.
|
||||
|
||||
## Outstanding
|
||||
|
||||
These pre-existing items now carry startup Warnings:
|
||||
- **Shared stores.** Set both, then recreate:
|
||||
- `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer` for inbound keys.
|
||||
- `Secrets:Store:Provider=SqlServer` plus `Secrets:SqlServer:ConnectionString` for secrets.
|
||||
|
||||
Neither is needed for current QA operation: the site holds no deployed secrets.
|
||||
- **Stale keys.** Remove the `Secrets:Replication` `Mode`/`GrpcHub` keys from both boxes' appsettings.json.
|
||||
|
||||
Housekeeping after soak: older `app.bak.*`, `cli.bak.*`, `_appsettings-keep-*` and `E:\psrem-stage\*` folders on both boxes.
|
||||
- **Standby relay:** QA still has no `ScadaBridge:InboundApi:StandbyForwardPeerEndpoints`, so an inbound call sent to the standby answers 503 `STANDBY_NODE` instead of being relayed to the active node.
|
||||
- **Housekeeping after soak:** delete the `*.imported` key files; also the older `app.bak.*`, `cli.bak.*`, `_appsettings-keep-*` and `E:\psrem-stage\*` folders on both boxes.
|
||||
|
||||
Reference in New Issue
Block a user