docs: Delmia rig stand-in + notifier verification results (A-G incl. active-central kill/takeover 27 s, rejoin); register rows 107 (prod DelmiaRecv path) + 108 (per-node inbound API key store breaks central failover); CLAUDE.md bullet; plan + task file

Claude-Session: https://claude.ai/code/session_013L3XdmVii9i5sY4FChU8pB
2026-09-02 12:54:58 -04:00
parent f4363e0b4b
commit 081e57f166
6 changed files with 744 additions and 0 deletions
@@ -197,3 +197,68 @@ revision, and one `MachineCode must end with A (Left) or B (Right)` from a retir
 3. `required` is empty in the parameter schema; only `MachineCode` is enforced, in script. Missing
   payload fields are written to the receiver as null/empty.
 4. QA definitions unread (credentials) — assumed identical (same build lineage), not verified.
+
+## 8. Docker rig stand-in and notifier verification (2026-09-02)
+
+The local rig (`docker/`, central-a `:9001` / central-b `:9002` / Traefik `:9000`, site-a = site id 1)
+carries an in-memory stand-in for a vd03-style machine so the whole path can be exercised without a
+Galaxy. Seeded by CLI only, no DB edits:
+
+- `docker/seed-delmia-sim.sh` — template **`DelmiaReceiverSim`** (the ten §5 attributes, all **static**,
+  `ReadyFlag` defaulting to `true`) composed into **`SimMachine`** as `DelmiaReceiver`; instance
+  **`DelmiaSim01`** on site-a, deployed. `SimMachine` carries two scripts:
+  - `ProcessRecipeDownload` (`Call`) — the §4 handshake ported to static attributes. `WriteBatchAndWaitAsync`
+    **rejects static attributes** (`InstanceActor.HandleWriteAttributeBatch`: "not a data-sourced
+    attribute"), so the port writes the payload with per-attribute `SetAsync` and waits with `WaitAsync`
+    — same trigger/response flags, same 25 s budget, same result mapping. It addresses
+    `Children["DelmiaReceiver"]` (the post-rename name; contrast §7 item 1).
+  - `DelmiaReceiverSimulator` (`ValueChange` on `DelmiaReceiver.RecipeDownloadFlag`) — plays the Galaxy
+    object: on the rising edge it accepts unless `DownloadPath` ends in `.bad` or the work order is blank,
+    writes `RecipeProcessResult`/`RecipeProcessResultText`, clears the trigger flag, then sets
+    `RecipeProcessedFlag = true`. It lives on the **root** template because composed scripts' trigger
+    configs are not prefixed at flattening, and `ScriptActor` matches `ValueChange` by exact canonical name.
+- `docker/seed-delmia-api.sh` — the §4 method verbatim (minus its dead `suf` line), timeout 45 s, plus one key
+  (`delmia-notifier`) scoped to it; the token is printed once.
+- `docker/sync-inbound-api-keys.sh` — **required before any central failover test**, see below.
+- `docker/verify-delmia-notifier.sh` — runs the managed `WWNotifier.dll` from a scratch copy whose
+  `appsettings.json` carries the key (`ScadaBridge:ApiKey`), with `BaseUrls` listing the two central nodes
+  directly (not Traefik) so the notifier's own connect-failure failover is what gets exercised.
+
+Whole-path latency through Traefik: a `curl` call returns in under 100 ms (simulator fires on the same
+node, no device round-trip).
+
+### Verified outcomes (second run, 2026-09-02 16:51 UTC)
+
+| Case | Setup | Result | Audit row (`ApiInbound`) |
+|------|-------|--------|--------------------------|
+| A | active (central-a) first in `BaseUrls` | `YES` | `InboundRequest` on `central-a` |
+| B | standby (central-b) first — standby relays | `YES` | **one** row, on `central-a` (the relay does not double-audit) |
+| C | bad key | `NO` + `HTTP 401` | `InboundAuthFailure` on `central-a` |
+| D | active `docker kill`ed, attempt inside the takeover window | `NO` + `HTTP 503` (`ACTIVE_NODE_UNREACHABLE` from the not-yet-active survivor) — expected, connect-failure-only failover | none (infrastructure answer) |
+| E | survivor `/health/active` = 200 — takeover measured **27 s** after the kill | `YES` (`:9001` refused → `:9002` 200) | `InboundRequest` on `central-b` |
+| F | victim restarted, `memberCount=2`, victim `/health/active` = 503 (rejoined as standby) | `YES` (victim relays) | `InboundRequest` on `central-b` |
+| G | `DownloadPath` = `C:\recipes\x.bad` | `NO` + `Simulated reject: bad path or missing work order` | `InboundRequest` on `central-b`, `Result=false` |
+
+The notifier's first-line contract, exit codes (`0` / `-1`), and the reason line on `NO` behaved as the
+README states throughout.
+
+### Two findings from the first run (kept, not "fixed")
+
+1. **`docker stop` is not a crash.** SIGTERM triggers CoordinatedShutdown's graceful singleton handover —
+   the survivor was active **0 s** after the stop. The drill uses `docker kill` so auto-down's ~25–30 s
+   takeover is what gets measured (`docker/failover-drill.sh` already does the same).
+2. **Inbound API keys are a per-node SQLite store, so a central failover invalidates every client's key.**
+   Keys live in `ScadaBridge:InboundApi:ApiKeyStore:SqlitePath` (`/app/data/inbound-api-keys.sqlite`, on
+   each central's own data volume), and key admin (`ManagementActor`, a per-node `ActorOf` reached from that
+   node's `/management` endpoints) writes only the node that serves the call — through Traefik that is the
+   active node. After the first run's takeover, central-b answered **401 `KeyNotFound`** to the key
+   central-a had minted; its `security api-key list` was `[]`. The standby-relay case (B) passes only
+   because the standby forwards **before** authenticating. The same asymmetry means a key revoked on one
+   node stays valid on the other. Rig workaround: `docker/sync-inbound-api-keys.sh` checkpoints central-a's
+   store on the host (`sqlite3`) and replaces central-b's (target stopped for the copy; both rig nodes share
+   one pepper, so the hashes verify on either node); the verify script's preflight refuses to start unless
+   both nodes list the key. The product fix is an architecture decision (shared SQL Server store or file
+   replication, with cluster-wide revocation) and needs the `ZB.MOM.WW.Auth.ApiKeys` library — deferred-work
+   register row 108. Prod `wonder-app-vd03` is single-node and unaffected; the QA pair has this gap today.
+
+Deferred-work register row 107 records §7 item 1 (`Children["DelmiaRecv"]`) as the prod follow-up.