# DelmiaRecipeDownload — ScadaBridge Inbound API spec (as deployed)

**Captured:** 2026-09-02 from the live **prod** management API on `wonder-app-vd03` (method id 4,
API keys, the `CvdReactor` template's `ProcessRecipeDownload` script, the `DelmiaReceiver` template,
and the inbound audit log). **QA (`vq03`/`vq04`) was NOT read** — its management API requires real
LDAP credentials (prod runs with `DisableLogin`); re-capture QA with `scadabridge api-method get --id …`
once credentials are to hand. The definition is config-as-data in the central ConfigDb (not in git), so
this file is the durable record; the legacy predecessor is documented in
[`../former-api-specs/dnc/Delmia-Integration-API.md`](../former-api-specs/dnc/Delmia-Integration-API.md)
(Surface B), and the client that calls it is the `WWNotifier.exe` drop-in
([`DelmiaNotifier`](../../src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md)).

## 1. Endpoint

| Item | Value |
|---|---|
| Method / path | `POST /api/DelmiaRecipeDownload` |
| Auth | `X-API-Key: sbk_<keyId>_<secret>` **or** `Authorization: Bearer sbk_<keyId>_<secret>` (Authorization wins if both) |
| Content type | `application/json`, flat PascalCase object |
| Server timeout | **45 s** (`TimeoutSeconds` on the method); the site-side handshake wait inside it is **25 s** |
| Prod base URL | `http://wonder-app-vd03.zmr.zimmer.com:8085` (single central node) |
| QA base URLs | `http://10.220.158.2:8085` (vq03), `http://10.220.158.3:8085` (vq04) — list **both** in the notifier's `BaseUrls`; a standby central relays to the active peer |

Unauthenticated / bad-key calls return **401** `{"error":"Invalid or missing API key"}` (audit kind
`InboundAuthFailure`). General Inbound API status semantics (413 body cap, 503 `STANDBY_NODE` /
`ACTIVE_NODE_UNREACHABLE`, unknown method) are in
[`Component-InboundAPI.md`](../requirements/Component-InboundAPI.md).

## 2. Request

Deployed `ParameterDefinitions` (JSON Schema, verbatim — note `required` is **empty**; the
script itself rejects a missing/blank `MachineCode`):

```json
{"type":"object","properties":{
  "MachineCode":{"type":"string"},
  "DownloadPath":{"type":"string"},
  "WorkOrderNumber":{"type":"string"},
  "PartNumber":{"type":"string"},
  "JobStepNumber":{"type":"string"},
  "Username":{"type":"string"}},
 "required":[]}
```

| Field | Notifier flag | Meaning | Where it lands on the site |
|---|---|---|---|
| `MachineCode` | `-m` | **Routing key** — must equal an instance `UniqueName` exactly (trimmed). Prod today: `Z28061`, `Z28062`, `Z28061Sim`. | `Route.To(MachineCode)` |
| `DownloadPath` | `-d` | Path where the recipe/NC file was placed | `DelmiaReceiver.DownloadPath` |
| `WorkOrderNumber` | `-w` | Work order | `DelmiaReceiver.WorkOrderNumber` |
| `PartNumber` | `-p` | Part / item number | `DelmiaReceiver.PartNumber` |
| `JobStepNumber` | `-s` | Job step / sequence op | `DelmiaReceiver.JobStepNumber` |
| `Username` | `-u` | Operator (audit only) | `DelmiaReceiver.Username` |

Example (a real call taken from the prod audit log):

```bash
curl -s -X POST http://wonder-app-vd03.zmr.zimmer.com:8085/api/DelmiaRecipeDownload \
  -H "X-API-Key: $SCADABRIDGE_API_KEY" -H "Content-Type: application/json" \
  -d '{"MachineCode":"Z28061Sim","DownloadPath":"C:\\recipes\\test.nc","WorkOrderNumber":"WO-TEST-001","PartNumber":"PN-TEST","JobStepNumber":"0100","Username":"delmia-sim-test"}'
```

## 3. Response

Deployed `ReturnDefinition`: `{"type":"object","properties":{"Result":{"type":"boolean"},"ResultText":{"type":"string"}},"required":["Result","ResultText"]}`

**HTTP 200 is returned for every business outcome** — success is `Result == true`, never the status
code. `ResultText` values the deployed scripts can produce:

| `Result` | `ResultText` | Produced by |
|---|---|---|
| `true` | echo of `DelmiaReceiver.RecipeProcessResultText` (often `""`) | Galaxy object acknowledged the download |
| `false` | `DelmiaReceiver.RecipeProcessResultText` | Galaxy object rejected the download (`RecipeProcessResult` false) |
| `false` | `MachineCode is required` | central script, blank `MachineCode` |
| `false` | `Receiver ready flag not set to true` | site script, `ReadyFlag` not true (**or unreadable — see §7**) |
| `false` | `Timeout waiting for recipe to be processed` | site script, `RecipeProcessedFlag` not true within 25 s |
| `false` | `RecipeDownload failed: <exception>` | either script — e.g. unknown instance, site unreachable, `Script 'ProcessRecipeDownload' on instance 'X' failed: …` |

## 4. Processing flow

```
Delmia ─► WWNotifier.exe ─► POST /api/DelmiaRecipeDownload (central, active node)
                              │ inbound method script (id 4): validate MachineCode
                              ▼
                     Route.To(MachineCode).Call("ProcessRecipeDownload", {5 fields})
                              │ gRPC central→site, lands on the instance actor
                              ▼
      CvdReactor.ProcessRecipeDownload (site Call script) on the composed DelmiaReceiver child:
        1. read  ReadyFlag            → must be true, else "Receiver ready flag not set to true"
        2. write RecipeProcessedFlag  = false
        3. WriteBatchAndWaitAsync({DownloadPath, WorkOrderNumber, PartNumber, JobStepNumber, Username},
                                  trigger RecipeDownloadFlag = true,
                                  wait RecipeProcessedFlag == true, timeout 25 s)
        4. read  RecipeProcessResult / RecipeProcessResultText → {Result, ResultText}
```

Both scripts as deployed on prod (verbatim):

<details><summary>Central inbound method script (id 4, timeout 45 s)</summary>

```csharp
try {
  System.Func<string, object> P = k => Parameters.ContainsKey(k) ? Parameters[k] : null;
  var code = (P("MachineCode") as string ?? "").Trim();
  if (string.IsNullOrEmpty(code)) return new { Result = false, ResultText = "MachineCode is required" };
  var suf = code.Substring(code.Length - 1);

  return await Route.To(code).Call("ProcessRecipeDownload", new {
      DownloadPath    = P("DownloadPath"),
      WorkOrderNumber = P("WorkOrderNumber"),
      PartNumber      = P("PartNumber"),
      JobStepNumber   = P("JobStepNumber"),
      Username        = P("Username"),
  });
} catch (Exception ex) {
  return new { Result = false, ResultText = "RecipeDownload failed: " + ex.Message };
}
```
</details>

<details><summary>Site script <code>CvdReactor.ProcessRecipeDownload</code> (template 1, script id 4, trigger <code>call</code>)</summary>

```csharp
// Generic DELMIA recipe-download — performed DIRECTLY on the composed DelmiaReceiver child
// (mirrors how MesMoveIn writes the MESReceiver child) instead of delegating via CallScript.
// No side logic; the receiver's own DelmiaReceiver.Download method holds the same handshake.
var recv = Children["DelmiaRecv"];
System.Func<object,bool> IsTrue = v => { var s=v?.ToString(); return (v is bool b && b)||s=="1"||string.Equals(s,"true",System.StringComparison.OrdinalIgnoreCase); };
System.Func<string,object?> P = k => Parameters.ContainsKey(k) ? Parameters[k] : null;
if (!IsTrue(recv.Attributes["ReadyFlag"]))
    return new { Result=false, ResultText="Receiver ready flag not set to true" };
try {
    recv.Attributes["RecipeProcessedFlag"] = false;
    var data = new System.Collections.Generic.Dictionary<string, object?> {
        { "DownloadPath", P("DownloadPath") },
        { "WorkOrderNumber", P("WorkOrderNumber") },
        { "PartNumber", P("PartNumber") },
        { "JobStepNumber", P("JobStepNumber") },
        { "Username", P("Username") },
    };
    if (!await recv.Attributes.WriteBatchAndWaitAsync(data, "RecipeDownloadFlag", true, "RecipeProcessedFlag", true, System.TimeSpan.FromSeconds(25)))
        return new { Result=false, ResultText="Timeout waiting for recipe to be processed" };
    return new {
        Result = IsTrue(recv.Attributes["RecipeProcessResult"]),
        ResultText = recv.Attributes["RecipeProcessResultText"]?.ToString() ?? ""
    };
} catch (Exception ex) {
    return new { Result=false, ResultText="RecipeDownload failed: " + ex.Message };
}
```
</details>

## 5. Galaxy-side contract — the `DelmiaReceiver` template (id 12; composed clone id 26)

All ten attributes are **data-sourced over the site `MxGateway` connection**, bound per instance to the
reactor-level Galaxy object (prod: `DelmiaReceiver_038` on Z28061, `_039` on Z28062, `_037` on
Z28061Sim; `Left_00x.DelmiaReceiver` / `Right_00x.DelmiaReceiver` back the unused side compositions).

| Attribute | Type | Direction | Role in the handshake |
|---|---|---|---|
| `ReadyFlag` | Boolean | Galaxy → SB | Precondition; must be true or the call is refused |
| `RecipeDownloadFlag` | Boolean | SB → Galaxy | **Trigger** — set true after the five data fields are written |
| `RecipeProcessedFlag` | Boolean | both | SB clears to false first; Galaxy sets true when done (waited ≤ 25 s) |
| `RecipeProcessResult` | Boolean | Galaxy → SB | Outcome → `Result` |
| `RecipeProcessResultText` | String | Galaxy → SB | Outcome text → `ResultText` |
| `DownloadPath`, `WorkOrderNumber`, `PartNumber`, `JobStepNumber`, `Username` | String | SB → Galaxy | Payload, written as one batch before the trigger |

## 6. API keys scoped to this method (prod, names only — secrets are shown once at creation)

`mes`, `mes api key`, `delmia-mes-receivers`, `TestKey`, `JOE_TEST_KEY` — all enabled, all also
scoped to the MES methods (`MesMoveIn`, `MesMoveOut`, `MesSampleData`, `SimpleAlarmStatusRequest`,
most also `IpsenMESMoveIn`). `delmia-mes-receivers` is the one whose name says it is the Delmia
caller's key; the audit `actor` column shows which key served each call.

## 7. Observed behaviour and open issues (from the prod audit log, 2026-06-17 → 2026-06-26)

Nine calls total, all test traffic (actors `delmia-mes-receivers`, `delmia-sim-test`), none since
2026-06-26 — **no production Delmia traffic has hit this method yet.** Every 200 answered
`Result=false`: mostly `Receiver ready flag not set to true`, one
`Script 'ProcessRecipeDownload' … The given key 'side' was not present` from an earlier script
revision, and one `MachineCode must end with A (Left) or B (Right)` from a retired left/right variant
(the current script routes on the whole `MachineCode`; the leftover `suf` variable is unused).

1. **Likely broken since the 2026-06-27 rename — verify before go-live.** The site script still
   addresses the child as `Children["DelmiaRecv"]`, but the `CvdReactor` composition was renamed
   `DelmiaRecv` → `DelmiaReceiver` that day (`../deployment/deployment-records/2026-06-27-wonder-app-vd03.md`
   §2b), so the flattened paths are `DelmiaReceiver.*`. `ChildrenAccessor` just concatenates the
   name into a path and `GetAttribute` on an unknown path logs a warning and yields null, so
   `IsTrue(null)` is false and **every call will answer `Receiver ready flag not set to true`
   regardless of the real flag**. No call has been made since the rename, so this is inferred from
   the code, not observed. Fix = change that one line to `Children["DelmiaReceiver"]` on the
   template (`scadabridge template script update`) and redeploy the three CvdReactor instances;
   check QA's copy of the script for the same string.
2. **Not idempotent.** The design requires idempotency on `(MachineCode, DownloadPath, WorkOrderNumber)`
   because the notifier retries on timeout/connect failure (at-least-once). The deployed script
   re-runs the full handshake on a duplicate, so a retry after a slow first attempt will trigger a
   second download on the Galaxy object.
3. `required` is empty in the parameter schema; only `MachineCode` is enforced, in script. Missing
   payload fields are written to the receiver as null/empty.
4. QA definitions unread (credentials) — assumed identical (same build lineage), not verified.

## 8. Docker rig stand-in and notifier verification (2026-09-02)

The local rig (`docker/`, central-a `:9001` / central-b `:9002` / Traefik `:9000`, site-a = site id 1)
carries an in-memory stand-in for a vd03-style machine so the whole path can be exercised without a
Galaxy. Seeded by CLI only, no DB edits:

- `docker/seed-delmia-sim.sh` — template **`DelmiaReceiverSim`** (the ten §5 attributes, all **static**,
  `ReadyFlag` defaulting to `true`) composed into **`SimMachine`** as `DelmiaReceiver`; instance
  **`DelmiaSim01`** on site-a, deployed. `SimMachine` carries two scripts:
  - `ProcessRecipeDownload` (`Call`) — the §4 handshake ported to static attributes. `WriteBatchAndWaitAsync`
    **rejects static attributes** (`InstanceActor.HandleWriteAttributeBatch`: "not a data-sourced
    attribute"), so the port writes the payload with per-attribute `SetAsync` and waits with `WaitAsync`
    — same trigger/response flags, same 25 s budget, same result mapping. It addresses
    `Children["DelmiaReceiver"]` (the post-rename name; contrast §7 item 1).
  - `DelmiaReceiverSimulator` (`ValueChange` on `DelmiaReceiver.RecipeDownloadFlag`) — plays the Galaxy
    object: on the rising edge it accepts unless `DownloadPath` ends in `.bad` or the work order is blank,
    writes `RecipeProcessResult`/`RecipeProcessResultText`, clears the trigger flag, then sets
    `RecipeProcessedFlag = true`. It lives on the **root** template because composed scripts' trigger
    configs are not prefixed at flattening, and `ScriptActor` matches `ValueChange` by exact canonical name.
- `docker/seed-delmia-api.sh` — the §4 method verbatim (minus its dead `suf` line), timeout 45 s, plus one key
  (`delmia-notifier`) scoped to it; the token is printed once.
- ~~`docker/sync-inbound-api-keys.sh` — required before any central failover test, see below.~~
  **Superseded 2026-09-02 by register row 108 / the shared SQL Server key store**
  (`ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer`, now set on both rig centrals): the script
  is deleted and `verify-delmia-notifier.sh` needs no file copy before a failover run — both nodes
  read the same `zbauth` rows.
- `docker/verify-delmia-notifier.sh` — runs the managed `WWNotifier.dll` from a scratch copy whose
  `appsettings.json` carries the key (`ScadaBridge:ApiKey`), with `BaseUrls` listing the two central nodes
  directly (not Traefik) so the notifier's own connect-failure failover is what gets exercised.

Whole-path latency through Traefik: a `curl` call returns in under 100 ms (simulator fires on the same
node, no device round-trip).

### Verified outcomes (second run, 2026-09-02 16:51 UTC; row H from the third run, 2026-09-06 04:48 UTC — A–G re-passed, takeover 29 s, rejoin 3 s)

| Case | Setup | Result | Audit row (`ApiInbound`) |
|------|-------|--------|--------------------------|
| A | active (central-a) first in `BaseUrls` | `YES` | `InboundRequest` on `central-a` |
| B | standby (central-b) first — standby relays | `YES` | **one** row, on `central-a` (the relay does not double-audit) |
| C | bad key | `NO` + `HTTP 401` | `InboundAuthFailure` on `central-a` |
| D | active `docker kill`ed, attempt inside the takeover window | `NO` + `HTTP 503` (`ACTIVE_NODE_UNREACHABLE` from the not-yet-active survivor) — expected, connect-failure-only failover | none (infrastructure answer) |
| E | survivor `/health/active` = 200 — takeover measured **27 s** after the kill | `YES` (`:9001` refused → `:9002` 200) | `InboundRequest` on `central-b` |
| F | victim restarted, `memberCount=2`, victim `/health/active` = 503 (rejoined as standby) | `YES` (victim relays) | `InboundRequest` on `central-b` |
| G | `DownloadPath` = `C:\recipes\x.bad` | `NO` + `Simulated reject: bad path or missing work order` | `InboundRequest` on `central-b`, `Result=false` |
| H | Flow Debugger (2026-09-06 third run, branch `feat/flow-debugger`): `flow-debug run-method` on `DelmiaRecipeDownload`, then `flow-debug replay` on the returned root id | `YES` — live run `completed`, **11** events inline (`InboundRequest`, `RouteCall`, `ScriptRun`, 7 × `AttributeWrite`, `AttributeWait`); replay rebuilt the same root with **17** events on the first attempt (no telemetry lag hit) | `ApiInbound`/`InboundRequest` (actor = operator) + `ScriptTrace` rows; the six extra replay rows are the simulator's two cascaded `ValueChange` runs (rising + clearing edge of `RecipeDownloadFlag`, 2 × `ScriptRun` + 4 × `AttributeWrite`) — inline `TraceEvents` carry only the routed run's own steps, replay walks the whole execution tree |

The notifier's first-line contract, exit codes (`0` / `-1`), and the reason line on `NO` behaved as the
README states throughout.

### Two findings from the first run (kept, not "fixed")

1. **`docker stop` is not a crash.** SIGTERM triggers CoordinatedShutdown's graceful singleton handover —
   the survivor was active **0 s** after the stop. The drill uses `docker kill` so auto-down's ~25–30 s
   takeover is what gets measured (`docker/failover-drill.sh` already does the same).
2. **Inbound API keys are a per-node SQLite store, so a central failover invalidates every client's key.**
   Keys live in `ScadaBridge:InboundApi:ApiKeyStore:SqlitePath` (`/app/data/inbound-api-keys.sqlite`, on
   each central's own data volume), and key admin (`ManagementActor`, a per-node `ActorOf` reached from that
   node's `/management` endpoints) writes only the node that serves the call — through Traefik that is the
   active node. After the first run's takeover, central-b answered **401 `KeyNotFound`** to the key
   central-a had minted; its `security api-key list` was `[]`. The standby-relay case (B) passes only
   because the standby forwards **before** authenticating. The same asymmetry means a key revoked on one
   node stays valid on the other. Rig workaround at the time: `docker/sync-inbound-api-keys.sh`
   checkpointed central-a's store on the host (`sqlite3`) and replaced central-b's (target stopped for the
   copy; both rig nodes share one pepper, so the hashes verify on either node); the verify script's
   preflight refused to start unless both nodes listed the key.

   **Superseded 2026-09-02 (same day) by register row 108.** `ZB.MOM.WW.Auth` 0.3.0 adds a SQL Server
   provider (`ZB.MOM.WW.Auth.ApiKeys.SqlServer`); both rig centrals now run
   `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer`, sharing one `zbauth` row set in the
   configuration database, so a key exists on both nodes the instant it's created and a revoke is
   cluster-wide immediately. `docker/sync-inbound-api-keys.sh` is deleted; a re-run of this drill needs
   no file copy before the failover step. Re-measured 2026-09-03 without the
   workaround: A/B/E/F `YES`, C/G `NO`, active-central `docker kill` takeover **28 s** (the earlier run measured 27 s with the file copy), victim rejoined as standby 2 s after restart. Prod `wonder-app-vd03` is single-node and unaffected; the QA pair
   (`vq03`/`vq04`) and prod still run `Provider=Sqlite` pending a deployment record.

Deferred-work register row 107 records §7 item 1 (`Children["DelmiaRecv"]`) as the prod follow-up.