diff --git a/CLAUDE.md b/CLAUDE.md index f3c5fe36..f2c34be3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -128,7 +128,7 @@ spec for each is `docs/requirements/Component-.md`, and `README.md` carrie - `Notify.Send` is async and **enqueue-only** — it buffers the notification into the local SQLite S&F store and returns a `NotificationId` (GUID, idempotency key) status handle immediately; it never runs the forwarder's central Ask inline on the script thread (`deferToSweep: true` buffers due-immediately + kicks a background sweep), so its worst-case latency is the local insert whether central is up or down. It enqueues with `maxRetries: 0` (the "no limit" escape hatch), so notifications retry until central acks and are **never parked for retry exhaustion** — only a corrupt payload parks them (arch-review 02, Tasks 13/14). `Notify.Status(notificationId)` returns a status record (status, retry count, last error, key timestamps); answered site-locally as `Forwarding` while still in the site S&F buffer, otherwise round-trips to central. - SMS delivery adapter (T9/T10, 2026-06-19): `SmsNotificationDeliveryAdapter` — Twilio REST, no SDK, HTTP Basic auth (`AccountSid:AuthToken`), one `POST .../Messages.json` per recipient. `SmsConfiguration` entity (`AccountSid`, `AuthToken` encrypted via Data Protection, `FromNumber`/`MessagingServiceSid`, `ApiBaseUrl`, timeout/retry) stored centrally; managed Admin-only via CLI `notification sms list|update` + Central UI `/notifications/sms`. Central-only (never deployed to sites). Per-recipient rollup: all-accepted → Delivered; any-transient → retry/park; mix → delivered-to-good + note; all-permanent → Park. `SmsConfiguration` (Auth Token in SecretsBlock) travels in Transport bundles via `--sms-configs`. - Inbound API: `POST /api/{methodName}`, `X-API-Key` header, flat JSON, extended type system (Object, List). -- **Inbound API on a STANDBY central node RELAYS to the active peer (2026-08-28)** instead of answering 503 `STANDBY_NODE`. Method scripts/`Route.To()` still run only on the active node (`IActiveNodeGate`); the standby's `InboundApiEndpointFilter` hands the request to `IStandbyRequestForwarder` (`HttpStandbyRequestForwarder`, plain HTTP, verbatim method/path/query/body/headers, peer response replayed unchanged) when `ScadaBridge:InboundApi:StandbyForwardPeerEndpoints` lists the *other* node's HTTP base URL — a static appsettings list, because the Akka member address carries the remoting port, not the HTTP one. Empty list = old 503 behaviour. Peers are tried in order; a peer 503 stamped `X-ScadaBridge-Standby: 1`, a connection failure or a `StandbyForwardTimeout` (60 s, must exceed the longest method timeout) expiry each move to the next, all failing → 503 `ACTIVE_NODE_UNREACHABLE`. **One-hop loop guard:** relayed requests carry `X-ScadaBridge-Forwarded` and are never relayed again. The 413 body cap runs *before* the relay, and the standby's `AuditWriteMiddleware` skips its own row for a relayed call (`InboundApiEndpointFilter.RelayedToActiveItemKey`) so it is audited once, on the node that executed it. Both rigs (`docker/`, `docker-env2/`) configure it on both central nodes; QA/prod appsettings are user-held and need the key added by hand. Spec: `Component-InboundAPI.md` → *Standby forwarding*. +- **Inbound API on a STANDBY central node RELAYS to the active peer (2026-08-28)** instead of answering 503 `STANDBY_NODE`. Method scripts/`Route.To()` still run only on the active node (`IActiveNodeGate`); the standby's `InboundApiEndpointFilter` hands the request to `IStandbyRequestForwarder` (`HttpStandbyRequestForwarder`, plain HTTP, verbatim method/path/query/body/headers, peer response replayed unchanged) when `ScadaBridge:InboundApi:StandbyForwardPeerEndpoints` lists the *other* node's HTTP base URL — a static appsettings list, because the Akka member address carries the remoting port, not the HTTP one. Empty list = old 503 behaviour. Peers are tried in order; a peer 503 stamped `X-ScadaBridge-Standby: 1`, a connection failure or a `StandbyForwardTimeout` (60 s, must exceed the longest method timeout) expiry each move to the next, all failing → 503 `ACTIVE_NODE_UNREACHABLE`. **One-hop loop guard:** relayed requests carry `X-ScadaBridge-Forwarded` and are never relayed again. The 413 body cap runs *before* the relay, and the standby's `AuditWriteMiddleware` skips its own row for a relayed call (`InboundApiEndpointFilter.RelayedToActiveItemKey`) so it is audited once, on the node that executed it. Both rigs (`docker/`, `docker-env2/`) configure it on both central nodes; QA (`vq03`/`vq04`) has it since 2026-10-22; prod `vd03` is a single central and needs none. Spec: `Component-InboundAPI.md` → *Standby forwarding*. - **DelmiaNotifier rig stand-in + the per-node inbound-key-store trap (2026-09-02).** `docker/seed-delmia-sim.sh` (in-memory `SimMachine` composing `DelmiaReceiverSim` as `DelmiaReceiver`, a `ValueChange` script on the ROOT template playing the Galaxy side) + `seed-delmia-api.sh` (prod's `DelmiaRecipeDownload` method + a scoped key) + `verify-delmia-notifier.sh` (notifier A–G incl. an active-central `docker kill`, takeover measured 27 s, rejoin as standby). The site script is a **static-attribute port** of the prod handshake — `WriteBatchAndWaitAsync` rejects static attributes, so it uses per-attribute `SetAsync` + `WaitAsync`. Two facts the drill established: (1) `docker stop` is a graceful CoordinatedShutdown handover (takeover 0 s) — crash drills must `docker kill`; (2) **inbound API keys live in a PER-NODE SQLite file** (`ScadaBridge:InboundApi:ApiKeyStore:SqlitePath`, each central's own volume), written by whichever node serves the management call, so a key exists on one node only and **a central failover makes every inbound client 401** — the standby-relay path only works because the standby forwards before authenticating — resolved by register row 108 / the shared store; `sync-inbound-api-keys.sh` is deleted. Results: `docs/api-specs/Delmia-Recipe-Download-API.md` §8. ### Templates & Deployment diff --git a/docs/deployment/deployment-records/2026-10-22-wonder-app-vq03-vq04-3808b49f.md b/docs/deployment/deployment-records/2026-10-22-wonder-app-vq03-vq04-3808b49f.md index cbecd64f..4e014153 100644 --- a/docs/deployment/deployment-records/2026-10-22-wonder-app-vq03-vq04-3808b49f.md +++ b/docs/deployment/deployment-records/2026-10-22-wonder-app-vq03-vq04-3808b49f.md @@ -130,7 +130,25 @@ Backups: `E:\psrem-stage\3808b49f\_envbak-ScadaBridge-Central-pre-sharedstores-< The `zbauth` and `zbsecrets` rows can stay; the SQLite path never reads them. +## Follow-up 2: inbound API standby relay (16:57) + +`ScadaBridge:InboundApi:StandbyForwardPeerEndpoints` was added to each box's `appsettings.Central.json`: +- vq03 → `["http://10.220.158.3:8085"]` +- vq04 → `["http://10.220.158.2:8085"]` + +`StandbyForwardTimeout` stays at its 60 s default, which is above QA's longest method timeout (`MesMoveIn` 46 s). The relay signature uses `ApiKeyPepper`, which is identical on both centrals. + +**Rolling restart, standby first:** +1. vq04: stop 16:57:09, service process exited, file pushed over SMB, ready 16:57:28. +2. Waited for 2 members / 0 unreachable. +3. vq03: stop 16:57:33, ready 16:57:51. + +The cluster never dropped below one serving central. **vq04 is now the active node** (failover on vq03's restart); nothing depends on which node is active. + +Backups: `E:\ApiInstall\ScadaBridge\_appsettings.Central.json.pre-relay-` on each box. + +**Verified:** a bogus-key inbound call answers **401 `UNAUTHORIZED` on both nodes**. The standby (vq03) used to answer 503 `STANDBY_NODE`; it now relays to the active peer, and the peer's answer is replayed. + ## Outstanding -- **Standby relay:** QA still has no `ScadaBridge:InboundApi:StandbyForwardPeerEndpoints`, so an inbound call sent to the standby answers 503 `STANDBY_NODE` instead of being relayed to the active node. - **Housekeeping after soak:** delete the `*.imported` key files; also the older `app.bak.*`, `cli.bak.*`, `_appsettings-keep-*` and `E:\psrem-stage\*` folders on both boxes.