Joseph Doherty
265 lines
17 KiB
Markdown
265 lines
17 KiB
Markdown
# DelmiaRecipeDownload — ScadaBridge Inbound API spec (as deployed)
|
||
|
||
**Captured:** 2026-09-02 from the live **prod** management API on `wonder-app-vd03` (method id 4,
|
||
API keys, the `CvdReactor` template's `ProcessRecipeDownload` script, the `DelmiaReceiver` template,
|
||
and the inbound audit log). **QA (`vq03`/`vq04`) was NOT read** — its management API requires real
|
||
LDAP credentials (prod runs with `DisableLogin`); re-capture QA with `scadabridge api-method get --id …`
|
||
once credentials are to hand. The definition is config-as-data in the central ConfigDb (not in git), so
|
||
this file is the durable record; the legacy predecessor is documented in
|
||
[`../former-api-specs/dnc/Delmia-Integration-API.md`](../former-api-specs/dnc/Delmia-Integration-API.md)
|
||
(Surface B), and the client that calls it is the `WWNotifier.exe` drop-in
|
||
([`DelmiaNotifier`](../../src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md)).
|
||
|
||
## 1. Endpoint
|
||
|
||
| Item | Value |
|
||
|---|---|
|
||
| Method / path | `POST /api/DelmiaRecipeDownload` |
|
||
| Auth | `X-API-Key: sbk_<keyId>_<secret>` **or** `Authorization: Bearer sbk_<keyId>_<secret>` (Authorization wins if both) |
|
||
| Content type | `application/json`, flat PascalCase object |
|
||
| Server timeout | **45 s** (`TimeoutSeconds` on the method); the site-side handshake wait inside it is **25 s** |
|
||
| Prod base URL | `http://wonder-app-vd03.zmr.zimmer.com:8085` (single central node) |
|
||
| QA base URLs | `http://10.220.158.2:8085` (vq03), `http://10.220.158.3:8085` (vq04) — list **both** in the notifier's `BaseUrls`; a standby central relays to the active peer |
|
||
|
||
Unauthenticated / bad-key calls return **401** `{"error":"Invalid or missing API key"}` (audit kind
|
||
`InboundAuthFailure`). General Inbound API status semantics (413 body cap, 503 `STANDBY_NODE` /
|
||
`ACTIVE_NODE_UNREACHABLE`, unknown method) are in
|
||
[`Component-InboundAPI.md`](../requirements/Component-InboundAPI.md).
|
||
|
||
## 2. Request
|
||
|
||
Deployed `ParameterDefinitions` (JSON Schema, verbatim — note `required` is **empty**; the
|
||
script itself rejects a missing/blank `MachineCode`):
|
||
|
||
```json
|
||
{"type":"object","properties":{
|
||
"MachineCode":{"type":"string"},
|
||
"DownloadPath":{"type":"string"},
|
||
"WorkOrderNumber":{"type":"string"},
|
||
"PartNumber":{"type":"string"},
|
||
"JobStepNumber":{"type":"string"},
|
||
"Username":{"type":"string"}},
|
||
"required":[]}
|
||
```
|
||
|
||
| Field | Notifier flag | Meaning | Where it lands on the site |
|
||
|---|---|---|---|
|
||
| `MachineCode` | `-m` | **Routing key** — must equal an instance `UniqueName` exactly (trimmed). Prod today: `Z28061`, `Z28062`, `Z28061Sim`. | `Route.To(MachineCode)` |
|
||
| `DownloadPath` | `-d` | Path where the recipe/NC file was placed | `DelmiaReceiver.DownloadPath` |
|
||
| `WorkOrderNumber` | `-w` | Work order | `DelmiaReceiver.WorkOrderNumber` |
|
||
| `PartNumber` | `-p` | Part / item number | `DelmiaReceiver.PartNumber` |
|
||
| `JobStepNumber` | `-s` | Job step / sequence op | `DelmiaReceiver.JobStepNumber` |
|
||
| `Username` | `-u` | Operator (audit only) | `DelmiaReceiver.Username` |
|
||
|
||
Example (a real call taken from the prod audit log):
|
||
|
||
```bash
|
||
curl -s -X POST http://wonder-app-vd03.zmr.zimmer.com:8085/api/DelmiaRecipeDownload \
|
||
-H "X-API-Key: $SCADABRIDGE_API_KEY" -H "Content-Type: application/json" \
|
||
-d '{"MachineCode":"Z28061Sim","DownloadPath":"C:\\recipes\\test.nc","WorkOrderNumber":"WO-TEST-001","PartNumber":"PN-TEST","JobStepNumber":"0100","Username":"delmia-sim-test"}'
|
||
```
|
||
|
||
## 3. Response
|
||
|
||
Deployed `ReturnDefinition`: `{"type":"object","properties":{"Result":{"type":"boolean"},"ResultText":{"type":"string"}},"required":["Result","ResultText"]}`
|
||
|
||
**HTTP 200 is returned for every business outcome** — success is `Result == true`, never the status
|
||
code. `ResultText` values the deployed scripts can produce:
|
||
|
||
| `Result` | `ResultText` | Produced by |
|
||
|---|---|---|
|
||
| `true` | echo of `DelmiaReceiver.RecipeProcessResultText` (often `""`) | Galaxy object acknowledged the download |
|
||
| `false` | `DelmiaReceiver.RecipeProcessResultText` | Galaxy object rejected the download (`RecipeProcessResult` false) |
|
||
| `false` | `MachineCode is required` | central script, blank `MachineCode` |
|
||
| `false` | `Receiver ready flag not set to true` | site script, `ReadyFlag` not true (**or unreadable — see §7**) |
|
||
| `false` | `Timeout waiting for recipe to be processed` | site script, `RecipeProcessedFlag` not true within 25 s |
|
||
| `false` | `RecipeDownload failed: <exception>` | either script — e.g. unknown instance, site unreachable, `Script 'ProcessRecipeDownload' on instance 'X' failed: …` |
|
||
|
||
## 4. Processing flow
|
||
|
||
```
|
||
Delmia ─► WWNotifier.exe ─► POST /api/DelmiaRecipeDownload (central, active node)
|
||
│ inbound method script (id 4): validate MachineCode
|
||
▼
|
||
Route.To(MachineCode).Call("ProcessRecipeDownload", {5 fields})
|
||
│ gRPC central→site, lands on the instance actor
|
||
▼
|
||
CvdReactor.ProcessRecipeDownload (site Call script) on the composed DelmiaReceiver child:
|
||
1. read ReadyFlag → must be true, else "Receiver ready flag not set to true"
|
||
2. write RecipeProcessedFlag = false
|
||
3. WriteBatchAndWaitAsync({DownloadPath, WorkOrderNumber, PartNumber, JobStepNumber, Username},
|
||
trigger RecipeDownloadFlag = true,
|
||
wait RecipeProcessedFlag == true, timeout 25 s)
|
||
4. read RecipeProcessResult / RecipeProcessResultText → {Result, ResultText}
|
||
```
|
||
|
||
Both scripts as deployed on prod (verbatim):
|
||
|
||
<details><summary>Central inbound method script (id 4, timeout 45 s)</summary>
|
||
|
||
```csharp
|
||
try {
|
||
System.Func<string, object> P = k => Parameters.ContainsKey(k) ? Parameters[k] : null;
|
||
var code = (P("MachineCode") as string ?? "").Trim();
|
||
if (string.IsNullOrEmpty(code)) return new { Result = false, ResultText = "MachineCode is required" };
|
||
var suf = code.Substring(code.Length - 1);
|
||
|
||
return await Route.To(code).Call("ProcessRecipeDownload", new {
|
||
DownloadPath = P("DownloadPath"),
|
||
WorkOrderNumber = P("WorkOrderNumber"),
|
||
PartNumber = P("PartNumber"),
|
||
JobStepNumber = P("JobStepNumber"),
|
||
Username = P("Username"),
|
||
});
|
||
} catch (Exception ex) {
|
||
return new { Result = false, ResultText = "RecipeDownload failed: " + ex.Message };
|
||
}
|
||
```
|
||
</details>
|
||
|
||
<details><summary>Site script <code>CvdReactor.ProcessRecipeDownload</code> (template 1, script id 4, trigger <code>call</code>)</summary>
|
||
|
||
```csharp
|
||
// Generic DELMIA recipe-download — performed DIRECTLY on the composed DelmiaReceiver child
|
||
// (mirrors how MesMoveIn writes the MESReceiver child) instead of delegating via CallScript.
|
||
// No side logic; the receiver's own DelmiaReceiver.Download method holds the same handshake.
|
||
var recv = Children["DelmiaRecv"];
|
||
System.Func<object,bool> IsTrue = v => { var s=v?.ToString(); return (v is bool b && b)||s=="1"||string.Equals(s,"true",System.StringComparison.OrdinalIgnoreCase); };
|
||
System.Func<string,object?> P = k => Parameters.ContainsKey(k) ? Parameters[k] : null;
|
||
if (!IsTrue(recv.Attributes["ReadyFlag"]))
|
||
return new { Result=false, ResultText="Receiver ready flag not set to true" };
|
||
try {
|
||
recv.Attributes["RecipeProcessedFlag"] = false;
|
||
var data = new System.Collections.Generic.Dictionary<string, object?> {
|
||
{ "DownloadPath", P("DownloadPath") },
|
||
{ "WorkOrderNumber", P("WorkOrderNumber") },
|
||
{ "PartNumber", P("PartNumber") },
|
||
{ "JobStepNumber", P("JobStepNumber") },
|
||
{ "Username", P("Username") },
|
||
};
|
||
if (!await recv.Attributes.WriteBatchAndWaitAsync(data, "RecipeDownloadFlag", true, "RecipeProcessedFlag", true, System.TimeSpan.FromSeconds(25)))
|
||
return new { Result=false, ResultText="Timeout waiting for recipe to be processed" };
|
||
return new {
|
||
Result = IsTrue(recv.Attributes["RecipeProcessResult"]),
|
||
ResultText = recv.Attributes["RecipeProcessResultText"]?.ToString() ?? ""
|
||
};
|
||
} catch (Exception ex) {
|
||
return new { Result=false, ResultText="RecipeDownload failed: " + ex.Message };
|
||
}
|
||
```
|
||
</details>
|
||
|
||
## 5. Galaxy-side contract — the `DelmiaReceiver` template (id 12; composed clone id 26)
|
||
|
||
All ten attributes are **data-sourced over the site `MxGateway` connection**, bound per instance to the
|
||
reactor-level Galaxy object (prod: `DelmiaReceiver_038` on Z28061, `_039` on Z28062, `_037` on
|
||
Z28061Sim; `Left_00x.DelmiaReceiver` / `Right_00x.DelmiaReceiver` back the unused side compositions).
|
||
|
||
| Attribute | Type | Direction | Role in the handshake |
|
||
|---|---|---|---|
|
||
| `ReadyFlag` | Boolean | Galaxy → SB | Precondition; must be true or the call is refused |
|
||
| `RecipeDownloadFlag` | Boolean | SB → Galaxy | **Trigger** — set true after the five data fields are written |
|
||
| `RecipeProcessedFlag` | Boolean | both | SB clears to false first; Galaxy sets true when done (waited ≤ 25 s) |
|
||
| `RecipeProcessResult` | Boolean | Galaxy → SB | Outcome → `Result` |
|
||
| `RecipeProcessResultText` | String | Galaxy → SB | Outcome text → `ResultText` |
|
||
| `DownloadPath`, `WorkOrderNumber`, `PartNumber`, `JobStepNumber`, `Username` | String | SB → Galaxy | Payload, written as one batch before the trigger |
|
||
|
||
## 6. API keys scoped to this method (prod, names only — secrets are shown once at creation)
|
||
|
||
`mes`, `mes api key`, `delmia-mes-receivers`, `TestKey`, `JOE_TEST_KEY` — all enabled, all also
|
||
scoped to the MES methods (`MesMoveIn`, `MesMoveOut`, `MesSampleData`, `SimpleAlarmStatusRequest`,
|
||
most also `IpsenMESMoveIn`). `delmia-mes-receivers` is the one whose name says it is the Delmia
|
||
caller's key; the audit `actor` column shows which key served each call.
|
||
|
||
## 7. Observed behaviour and open issues (from the prod audit log, 2026-06-17 → 2026-06-26)
|
||
|
||
Nine calls total, all test traffic (actors `delmia-mes-receivers`, `delmia-sim-test`), none since
|
||
2026-06-26 — **no production Delmia traffic has hit this method yet.** Every 200 answered
|
||
`Result=false`: mostly `Receiver ready flag not set to true`, one
|
||
`Script 'ProcessRecipeDownload' … The given key 'side' was not present` from an earlier script
|
||
revision, and one `MachineCode must end with A (Left) or B (Right)` from a retired left/right variant
|
||
(the current script routes on the whole `MachineCode`; the leftover `suf` variable is unused).
|
||
|
||
1. **Likely broken since the 2026-06-27 rename — verify before go-live.** The site script still
|
||
addresses the child as `Children["DelmiaRecv"]`, but the `CvdReactor` composition was renamed
|
||
`DelmiaRecv` → `DelmiaReceiver` that day (`../deployment/deployment-records/2026-06-27-wonder-app-vd03.md`
|
||
§2b), so the flattened paths are `DelmiaReceiver.*`. `ChildrenAccessor` just concatenates the
|
||
name into a path and `GetAttribute` on an unknown path logs a warning and yields null, so
|
||
`IsTrue(null)` is false and **every call will answer `Receiver ready flag not set to true`
|
||
regardless of the real flag**. No call has been made since the rename, so this is inferred from
|
||
the code, not observed. Fix = change that one line to `Children["DelmiaReceiver"]` on the
|
||
template (`scadabridge template script update`) and redeploy the three CvdReactor instances;
|
||
check QA's copy of the script for the same string.
|
||
2. **Not idempotent.** The design requires idempotency on `(MachineCode, DownloadPath, WorkOrderNumber)`
|
||
because the notifier retries on timeout/connect failure (at-least-once). The deployed script
|
||
re-runs the full handshake on a duplicate, so a retry after a slow first attempt will trigger a
|
||
second download on the Galaxy object.
|
||
3. `required` is empty in the parameter schema; only `MachineCode` is enforced, in script. Missing
|
||
payload fields are written to the receiver as null/empty.
|
||
4. QA definitions unread (credentials) — assumed identical (same build lineage), not verified.
|
||
|
||
## 8. Docker rig stand-in and notifier verification (2026-09-02)
|
||
|
||
The local rig (`docker/`, central-a `:9001` / central-b `:9002` / Traefik `:9000`, site-a = site id 1)
|
||
carries an in-memory stand-in for a vd03-style machine so the whole path can be exercised without a
|
||
Galaxy. Seeded by CLI only, no DB edits:
|
||
|
||
- `docker/seed-delmia-sim.sh` — template **`DelmiaReceiverSim`** (the ten §5 attributes, all **static**,
|
||
`ReadyFlag` defaulting to `true`) composed into **`SimMachine`** as `DelmiaReceiver`; instance
|
||
**`DelmiaSim01`** on site-a, deployed. `SimMachine` carries two scripts:
|
||
- `ProcessRecipeDownload` (`Call`) — the §4 handshake ported to static attributes. `WriteBatchAndWaitAsync`
|
||
**rejects static attributes** (`InstanceActor.HandleWriteAttributeBatch`: "not a data-sourced
|
||
attribute"), so the port writes the payload with per-attribute `SetAsync` and waits with `WaitAsync`
|
||
— same trigger/response flags, same 25 s budget, same result mapping. It addresses
|
||
`Children["DelmiaReceiver"]` (the post-rename name; contrast §7 item 1).
|
||
- `DelmiaReceiverSimulator` (`ValueChange` on `DelmiaReceiver.RecipeDownloadFlag`) — plays the Galaxy
|
||
object: on the rising edge it accepts unless `DownloadPath` ends in `.bad` or the work order is blank,
|
||
writes `RecipeProcessResult`/`RecipeProcessResultText`, clears the trigger flag, then sets
|
||
`RecipeProcessedFlag = true`. It lives on the **root** template because composed scripts' trigger
|
||
configs are not prefixed at flattening, and `ScriptActor` matches `ValueChange` by exact canonical name.
|
||
- `docker/seed-delmia-api.sh` — the §4 method verbatim (minus its dead `suf` line), timeout 45 s, plus one key
|
||
(`delmia-notifier`) scoped to it; the token is printed once.
|
||
- `docker/sync-inbound-api-keys.sh` — **required before any central failover test**, see below.
|
||
- `docker/verify-delmia-notifier.sh` — runs the managed `WWNotifier.dll` from a scratch copy whose
|
||
`appsettings.json` carries the key (`ScadaBridge:ApiKey`), with `BaseUrls` listing the two central nodes
|
||
directly (not Traefik) so the notifier's own connect-failure failover is what gets exercised.
|
||
|
||
Whole-path latency through Traefik: a `curl` call returns in under 100 ms (simulator fires on the same
|
||
node, no device round-trip).
|
||
|
||
### Verified outcomes (second run, 2026-09-02 16:51 UTC)
|
||
|
||
| Case | Setup | Result | Audit row (`ApiInbound`) |
|
||
|------|-------|--------|--------------------------|
|
||
| A | active (central-a) first in `BaseUrls` | `YES` | `InboundRequest` on `central-a` |
|
||
| B | standby (central-b) first — standby relays | `YES` | **one** row, on `central-a` (the relay does not double-audit) |
|
||
| C | bad key | `NO` + `HTTP 401` | `InboundAuthFailure` on `central-a` |
|
||
| D | active `docker kill`ed, attempt inside the takeover window | `NO` + `HTTP 503` (`ACTIVE_NODE_UNREACHABLE` from the not-yet-active survivor) — expected, connect-failure-only failover | none (infrastructure answer) |
|
||
| E | survivor `/health/active` = 200 — takeover measured **27 s** after the kill | `YES` (`:9001` refused → `:9002` 200) | `InboundRequest` on `central-b` |
|
||
| F | victim restarted, `memberCount=2`, victim `/health/active` = 503 (rejoined as standby) | `YES` (victim relays) | `InboundRequest` on `central-b` |
|
||
| G | `DownloadPath` = `C:\recipes\x.bad` | `NO` + `Simulated reject: bad path or missing work order` | `InboundRequest` on `central-b`, `Result=false` |
|
||
|
||
The notifier's first-line contract, exit codes (`0` / `-1`), and the reason line on `NO` behaved as the
|
||
README states throughout.
|
||
|
||
### Two findings from the first run (kept, not "fixed")
|
||
|
||
1. **`docker stop` is not a crash.** SIGTERM triggers CoordinatedShutdown's graceful singleton handover —
|
||
the survivor was active **0 s** after the stop. The drill uses `docker kill` so auto-down's ~25–30 s
|
||
takeover is what gets measured (`docker/failover-drill.sh` already does the same).
|
||
2. **Inbound API keys are a per-node SQLite store, so a central failover invalidates every client's key.**
|
||
Keys live in `ScadaBridge:InboundApi:ApiKeyStore:SqlitePath` (`/app/data/inbound-api-keys.sqlite`, on
|
||
each central's own data volume), and key admin (`ManagementActor`, a per-node `ActorOf` reached from that
|
||
node's `/management` endpoints) writes only the node that serves the call — through Traefik that is the
|
||
active node. After the first run's takeover, central-b answered **401 `KeyNotFound`** to the key
|
||
central-a had minted; its `security api-key list` was `[]`. The standby-relay case (B) passes only
|
||
because the standby forwards **before** authenticating. The same asymmetry means a key revoked on one
|
||
node stays valid on the other. Rig workaround: `docker/sync-inbound-api-keys.sh` checkpoints central-a's
|
||
store on the host (`sqlite3`) and replaces central-b's (target stopped for the copy; both rig nodes share
|
||
one pepper, so the hashes verify on either node); the verify script's preflight refuses to start unless
|
||
both nodes list the key. The product fix is an architecture decision (shared SQL Server store or file
|
||
replication, with cluster-wide revocation) and needs the `ZB.MOM.WW.Auth.ApiKeys` library — deferred-work
|
||
register row 108. Prod `wonder-app-vd03` is single-node and unaffected; the QA pair has this gap today.
|
||
|
||
Deferred-work register row 107 records §7 item 1 (`Children["DelmiaRecv"]`) as the prod follow-up.
|