docs: inbound API key store — shared SQL Server provider on central pairs (spec, runbook + central-pairs procedure, DBA DDL copy, topology, env inventory, CLAUDE.md, register row 108 closed with rig proof, row 109 Secrets backoff follow-up, Delmia spec §8)

Claude-Session: https://claude.ai/code/session_013L3XdmVii9i5sY4FChU8pB
This commit is contained in:
Joseph Doherty
2026-09-02 23:31:21 -04:00
parent a83cc8613d
commit f47ceb55b1
11 changed files with 1004 additions and 12 deletions
+18 -7
View File
@@ -219,7 +219,11 @@ Galaxy. Seeded by CLI only, no DB edits:
configs are not prefixed at flattening, and `ScriptActor` matches `ValueChange` by exact canonical name.
- `docker/seed-delmia-api.sh` — the §4 method verbatim (minus its dead `suf` line), timeout 45 s, plus one key
(`delmia-notifier`) scoped to it; the token is printed once.
- `docker/sync-inbound-api-keys.sh` — **required before any central failover test**, see below.
- ~~`docker/sync-inbound-api-keys.sh` — required before any central failover test, see below.~~
**Superseded 2026-09-02 by register row 108 / the shared SQL Server key store**
(`ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer`, now set on both rig centrals): the script
is deleted and `verify-delmia-notifier.sh` needs no file copy before a failover run — both nodes
read the same `zbauth` rows.
- `docker/verify-delmia-notifier.sh` — runs the managed `WWNotifier.dll` from a scratch copy whose
`appsettings.json` carries the key (`ScadaBridge:ApiKey`), with `BaseUrls` listing the two central nodes
directly (not Traefik) so the notifier's own connect-failure failover is what gets exercised.
@@ -254,11 +258,18 @@ README states throughout.
active node. After the first run's takeover, central-b answered **401 `KeyNotFound`** to the key
central-a had minted; its `security api-key list` was `[]`. The standby-relay case (B) passes only
because the standby forwards **before** authenticating. The same asymmetry means a key revoked on one
node stays valid on the other. Rig workaround: `docker/sync-inbound-api-keys.sh` checkpoints central-a's
store on the host (`sqlite3`) and replaces central-b's (target stopped for the copy; both rig nodes share
one pepper, so the hashes verify on either node); the verify script's preflight refuses to start unless
both nodes list the key. The product fix is an architecture decision (shared SQL Server store or file
replication, with cluster-wide revocation) and needs the `ZB.MOM.WW.Auth.ApiKeys` library — deferred-work
register row 108. Prod `wonder-app-vd03` is single-node and unaffected; the QA pair has this gap today.
node stays valid on the other. Rig workaround at the time: `docker/sync-inbound-api-keys.sh`
checkpointed central-a's store on the host (`sqlite3`) and replaced central-b's (target stopped for the
copy; both rig nodes share one pepper, so the hashes verify on either node); the verify script's
preflight refused to start unless both nodes listed the key.
**Superseded 2026-09-02 (same day) by register row 108.** `ZB.MOM.WW.Auth` 0.3.0 adds a SQL Server
provider (`ZB.MOM.WW.Auth.ApiKeys.SqlServer`); both rig centrals now run
`ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer`, sharing one `zbauth` row set in the
configuration database, so a key exists on both nodes the instant it's created and a revoke is
cluster-wide immediately. `docker/sync-inbound-api-keys.sh` is deleted; a re-run of this drill needs
no file copy before the failover step. Re-measured 2026-09-03 without the
workaround: A/B/E/F `YES`, C/G `NO`, active-central `docker kill` takeover **28 s** (the earlier run measured 27 s with the file copy), victim rejoined as standby 2 s after restart. Prod `wonder-app-vd03` is single-node and unaffected; the QA pair
(`vq03`/`vq04`) and prod still run `Provider=Sqlite` pending a deployment record.
Deferred-work register row 107 records §7 item 1 (`Children["DelmiaRecv"]`) as the prod follow-up.