# Delmia Simulated Machine + Notifier Rig Verification — Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
**Goal:** Stand up an in-memory (static-attribute) stand-in for a vd03-style machine with a composed `DelmiaReceiver` on the local docker rig, wire the `DelmiaRecipeDownload` inbound method and a scoped API key to it, let `DelmiaNotifier` read that key from `appsettings.json`, and prove the notifier end-to-end on the 2-node central cluster including an active-node crash and rejoin.
**Architecture:** Two templates seeded by CLI only (no DB edits): `DelmiaReceiverSim` (the ten receiver attributes, all static — no data connection) composed into `SimMachine` as `DelmiaReceiver`. The Galaxy side of the handshake is replaced by a root-scope `ValueChange` script on `SimMachine` that reacts to `DelmiaReceiver.RecipeDownloadFlag`, so the same call script shape as prod runs against pure in-memory tags. The central inbound method is the prod script verbatim (minus its dead `suf` line). The notifier gains an optional `ScadaBridge:ApiKey` config field with the existing `SCADABRIDGE_API_KEY` env var taking precedence, and a bash verification script drives the whole thing against `docker/` (central-a `:9001`, central-b `:9002`, Traefik `:9000`).
**Tech Stack:** ScadaBridge CLI (`scadabridge`), bash, .NET 10 (`DelmiaNotifier` + xUnit tests), docker compose rig under `docker/`.
**Facts the implementer must not re-derive** (verified 2026-09-02):
- `WriteBatchAndWaitAsync` **rejects static attributes** (`InstanceActor.HandleWriteAttributeBatch`: "Attribute 'X' is not a data-sourced attribute"). The rig script therefore uses `SetAsync` per attribute + `WaitAsync`, which both work on static attributes. Do not "fix" this in the runtime — out of scope.
- A static write always publishes `AttributeValueChanged` (no equality short-circuit), and `ScriptActor` matches a `ValueChange` trigger by **exact** canonical name (`valueTrigger.AttributeName == changed.AttributeName`). Composed scripts' trigger configs are **not** prefixed at flattening (alarms are), so the simulator script lives on the **root** template with `attributeName` = `DelmiaReceiver.RecipeDownloadFlag`.
- `WaitAsync` has a fast path when the current value already matches, so the simulator finishing before the wait registers is not a race.
- Trigger types are matched case-insensitively as strings: `Call`, `ValueChange`, `Interval`, `Conditional`, `Expression`. Trigger config key is `attributeName`.
- CLI global options (`--url`, `--username`, `--password`, `--format=json`) must come **after** the subcommand. Rig creds: `--username multi-role --password password`. Write loops in **bash** (zsh does not word-split unquoted variables).
- `security api-key create --format=json` returns `{ "keyId": …, "token": "sbk_…" }`; the token is shown once.
- Rig: site-a is `--site-id 1`; Traefik `http://localhost:9000` → active central; direct nodes `:9001` (central-a) and `:9002` (central-b); both centrals have `StandbyForwardPeerEndpoints` set, so a standby **relays** inbound calls to the active peer. Auto-down failover of a crashed active takes ~25–30 s (`docker/failover-drill.sh`).
- The notifier's failover is **connect-failure-only**: any HTTP answer (incl. 503) is final. During the takeover window a standby that cannot reach its peer answers 503 `ACTIVE_NODE_UNREACHABLE` → the notifier prints `NO`. That is expected and must be documented, not "fixed".
- Prod's deployed method + site script are recorded in `docs/api-specs/Delmia-Recipe-Download-API.md`; prod's site script still references `Children["DelmiaRecv"]` (a known, separate defect). The rig script uses `Children["DelmiaReceiver"]`.
**Model assignment summary** (see each task's `Model:` line): Sonnet for scripted CLI seeding, config plumbing, and docs; Opus for the live rig verification and the failover drill (judgment under a live system) and for reviewing the secret-handling change; Fable (this session) for the final integration review.
---
### Task 0: Preflight — rig healthy, CLI built, notifier tests green
**Classification:** trivial
**Estimated implement time:** ~3 min
**Parallelizable with:** none (everything depends on it)
**Model:** sonnet
**Files:**
- (none edited)
**Step 1: Confirm the rig is up and which node is active**
Run:
```bash
for p in 9000 9001 9002; do printf "$p ready="; curl -s -m 5 -o /dev/null -w "%{http_code}" http://localhost:$p/health/ready; printf " active="; curl -s -m 5 -o /dev/null -w "%{http_code}\n" http://localhost:$p/health/active; done
```
Expected: all three `ready=200`; exactly one of `9001`/`9002` shows `active=200`, the other `503`; `9000 active=200`. If a node is down, run `bash docker/deploy.sh` first and re-check.
**Step 2: Build the CLI and the notifier, run the notifier tests**
Run:
```bash
cd /Users/dohertj2/Desktop/ScadaBridge
dotnet build src/ZB.MOM.WW.ScadaBridge.CLI -c Debug 2>&1 | tail -3
dotnet test tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests 2>&1 | tail -5
```
Expected: `Build succeeded` and `Passed!` with 0 failed. Note the CLI binary path used by every later task: `src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge`.
**Step 3: Confirm the names this plan creates are free on the rig**
Run:
```bash
CLI=src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge; A="--url http://localhost:9000 --format=json --username multi-role --password password"
bash -c "$CLI template list $A | jq -r '.[].name' | grep -E '^(DelmiaReceiverSim|SimMachine)$'; $CLI api-method list $A | jq -r '.[].name' | grep -E '^DelmiaRecipeDownload$'; $CLI instance list $A | jq -r '.[].uniqueName' | grep -E '^DelmiaSim01$'; echo done"
```
Expected: only `done` printed. If any name is taken, delete it (`instance delete --id`, `template delete --id`, `api-method delete --id`) before continuing — the seed script is not idempotent by design.
---
### Task 1: `DelmiaNotifier` — optional `ApiKey` in `appsettings.json`, env var wins
**Classification:** small (single-project logic change, but it relaxes a documented secret-handling decision — reviewer must weigh in)
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 2, Task 3
**Model:** sonnet (implement) → **opus** code review, with the explicit question "is the env-over-file precedence and the README wording adequate for a key that now may live on disk?"
**Design note (state this in the commit):** the 2026-06-26 design deliberately kept the key env-only. The user has asked for the key in `appsettings.json`. Keep env-only as the *recommended* production posture, add the file as a fallback, and keep the precedence deterministic: `SCADABRIDGE_API_KEY` (if non-blank) → `ScadaBridge:ApiKey` (if non-blank) → not configured. The failure message names both sources.
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/NotifierConfig.cs`
- Modify: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/ConfigLoader.cs`
- Modify: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/Program.cs:42-47`
- Modify: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/appsettings.json`
- Modify: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md` (the `## Secret — SCADABRIDGE_API_KEY` section and the config JSON block)
- Modify: `env_variables.md` (§5 `SCADABRIDGE_API_KEY` row — "env-only by design" is no longer true)
- Modify: `docs/plans/2026-06-26-delmia-recipe-notifier-design.md` (the `Secret` row of the key-decisions table: add "superseded 2026-09-02: file fallback added, env still wins")
- Test: `tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests/ConfigLoaderTests.cs`
**Step 1: Write the failing tests** (append to `ConfigLoaderTests.cs`)
```csharp
[Fact]
public void Load_reads_ApiKey_from_ScadaBridge_section()
{
var cfg = ConfigLoader.Load("{\"ScadaBridge\":{\"BaseUrls\":\"http://x\",\"ApiKey\":\"sbk_file\"}}");
Assert.Equal("sbk_file", cfg.ScadaBridge.ApiKey);
}
[Fact]
public void ResolveApiKey_prefers_env_over_config()
{
var cfg = ConfigLoader.Load("{\"ScadaBridge\":{\"ApiKey\":\"sbk_file\"}}");
Assert.Equal("sbk_env", ConfigLoader.ResolveApiKey(n => n == "SCADABRIDGE_API_KEY" ? "sbk_env" : null, cfg));
}
[Fact]
public void ResolveApiKey_falls_back_to_config_when_env_unset_or_blank()
{
var cfg = ConfigLoader.Load("{\"ScadaBridge\":{\"ApiKey\":\" sbk_file \"}}");
Assert.Equal("sbk_file", ConfigLoader.ResolveApiKey(_ => null, cfg));
Assert.Equal("sbk_file", ConfigLoader.ResolveApiKey(_ => " ", cfg));
}
[Fact]
public void ResolveApiKey_returns_null_when_neither_source_set()
{
var cfg = ConfigLoader.Load("{\"ScadaBridge\":{\"ApiKey\":\" \"}}");
Assert.Null(ConfigLoader.ResolveApiKey(_ => null, cfg));
}
```
Also update the two existing `ResolveApiKey_*` tests to pass a config: `ConfigLoader.ResolveApiKey(_ => null, new NotifierConfig())` etc. (the one-argument overload is removed, not kept — YAGNI).
**Step 2: Run tests to verify they fail**
Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests 2>&1 | tail -5`
Expected: build error `'ScadaBridgeSection' does not contain a definition for 'ApiKey'` / no overload for `ResolveApiKey` taking 2 arguments.
**Step 3: Implement**
`NotifierConfig.cs` — add to `ScadaBridgeSection`:
```csharp
/// Optional inbound API key. SCADABRIDGE_API_KEY in the environment takes precedence; this is the file fallback (design 2026-06-26, relaxed 2026-09-02).
public string? ApiKey { get; set; }
```
`ConfigLoader.cs` — replace `ResolveApiKey`:
```csharp
/// Resolve the API key: SCADABRIDGE_API_KEY wins when non-blank, otherwise ScadaBridge:ApiKey from (trimmed); null when neither is set. Env accessor is injected for testability.
/// Accessor used to read an environment variable by name.
/// The loaded configuration, consulted only when the environment variable is unset/blank.
/// The resolved API key, or null if neither source yields a non-blank value.
public static string? ResolveApiKey(Func envGet, NotifierConfig cfg)
{
var env = envGet(ApiKeyEnvVar);
if (!string.IsNullOrWhiteSpace(env)) return env;
var file = cfg.ScadaBridge.ApiKey;
return string.IsNullOrWhiteSpace(file) ? null : file.Trim();
}
```
`Program.cs` lines 42–47 become:
```csharp
var apiKey = ConfigLoader.ResolveApiKey(Environment.GetEnvironmentVariable, cfg);
if (apiKey is null)
{
log.Write("API key not configured (SCADABRIDGE_API_KEY or ScadaBridge:ApiKey)");
return Reporter.Report(false, "API key not configured (SCADABRIDGE_API_KEY or ScadaBridge:ApiKey)", stdout);
}
```
`appsettings.json` — add `"ApiKey": ""` after `LogPath` (empty = not set; **never commit a real token**).
README: in the config JSON block add `"ApiKey": ""`; rewrite the `## Secret` section:
> The inbound API key is sent as `X-API-Key`. It is resolved from **`SCADABRIDGE_API_KEY`** first (recommended for production — the key never touches disk), falling back to **`ScadaBridge:ApiKey`** in `appsettings.json` when the variable is unset or blank. If neither is set → `NO` + `API key not configured (SCADABRIDGE_API_KEY or ScadaBridge:ApiKey)`, exit `-1`, no HTTP attempt. If you use the file, restrict its ACL to the account DELMIA runs under.
`env_variables.md` §5: change the "In appsettings?" cell to `Yes — optional \`ScadaBridge:ApiKey\` fallback (2026-09-02); env wins` and the Purpose cell's "Read only from the environment, never a file" to "Environment first, file fallback".
Design doc table row `| Secret | API key from env var … — never in a file |` → append ` **(relaxed 2026-09-02: optional \`ScadaBridge:ApiKey\` file fallback; env still wins)**`.
**Step 4: Run tests to verify they pass**
Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests 2>&1 | tail -5`
Expected: `Passed!`, 0 failed, total = previous count + 4. Also `dotnet build src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier -warnaserror` succeeds (the project treats warnings as errors; the XML doc comments above are required).
**Step 5: Commit**
```bash
git add src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests env_variables.md docs/plans/2026-06-26-delmia-recipe-notifier-design.md
git commit -m "feat(delmia-notifier): optional ScadaBridge:ApiKey in appsettings.json as a fallback to SCADABRIDGE_API_KEY (env still wins); README/env inventory/design row updated"
```
---
### Task 2: `docker/seed-delmia-sim.sh` — templates, composition, scripts, instance, deploy
**Classification:** standard (new script + two template scripts that must compile under the trust policy)
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 1, Task 3
**Model:** sonnet
**Files:**
- Create: `docker/seed-delmia-sim.sh`
- Modify: `docker/README.md:96-97` (add the new script to the tree listing, one line)
**Step 1: Write the script**
```bash
#!/bin/bash
set -euo pipefail
# Seed an IN-MEMORY stand-in for a vd03-style machine with a composed DelmiaReceiver on
# site-a, so DelmiaNotifier can be exercised end-to-end on the docker rig without a Galaxy.
#
# What it creates (all static attributes — no data connection, no bindings):
# - Template DelmiaReceiverSim — the ten DelmiaReceiver attributes from prod
# (docs/api-specs/Delmia-Recipe-Download-API.md §5), ReadyFlag defaulting to true.
# - Template SimMachine — composes DelmiaReceiverSim as `DelmiaReceiver` and carries
# * ProcessRecipeDownload (Call) — prod's handshake ported to per-attribute
# SetAsync/WaitAsync (WriteBatchAndWaitAsync rejects static attributes).
# * DelmiaReceiverSimulator (ValueChange on DelmiaReceiver.RecipeDownloadFlag) —
# plays the Galaxy object: accepts unless DownloadPath ends in ".bad" or the work
# order is blank, writes RecipeProcessResult/Text, clears the trigger flag, then
# sets RecipeProcessedFlag=true.
# - Instance DelmiaSim01 on site-a (id 1), deployed.
#
# Not idempotent: names must be unique. To re-seed:
# scadabridge instance delete --id ; scadabridge template delete --id ;
# scadabridge template delete --id
# Prerequisites: bash docker/deploy.sh; docker/seed-sites.sh; central healthy on :9000.
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
CLI="$PROJECT_ROOT/src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge"
A=(--url "${SCADABRIDGE_URL:-http://localhost:9000}" --format=json --username "${SCADABRIDGE_USERNAME:-multi-role}" --password "${SCADABRIDGE_PASSWORD:-password}")
SITE_ID="${SITE_ID:-1}"
INSTANCE_NAME="${INSTANCE_NAME:-DelmiaSim01}"
json_id() { python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])'; }
echo "1/6 Creating template DelmiaReceiverSim..."
RECV_ID=$("$CLI" template create "${A[@]}" --name DelmiaReceiverSim \
--description "In-memory stand-in for the Galaxy DelmiaReceiver object (ten static attributes)" | json_id)
echo " id=$RECV_ID"
echo "2/6 Adding the ten receiver attributes (static)..."
"$CLI" template attribute add "${A[@]}" --template-id "$RECV_ID" --name ReadyFlag --data-type Boolean --value true --description "Precondition; true = receiver accepts downloads" > /dev/null
"$CLI" template attribute add "${A[@]}" --template-id "$RECV_ID" --name RecipeDownloadFlag --data-type Boolean --value false --description "Trigger: set true after the payload is written" > /dev/null
"$CLI" template attribute add "${A[@]}" --template-id "$RECV_ID" --name RecipeProcessedFlag --data-type Boolean --value false --description "Cleared by the caller, set true by the receiver when done" > /dev/null
"$CLI" template attribute add "${A[@]}" --template-id "$RECV_ID" --name RecipeProcessResult --data-type Boolean --value false --description "Outcome -> Result" > /dev/null
for s in DownloadPath WorkOrderNumber PartNumber JobStepNumber Username RecipeProcessResultText; do
"$CLI" template attribute add "${A[@]}" --template-id "$RECV_ID" --name "$s" --data-type String --description "Recipe-download payload / outcome text" > /dev/null
done
echo "3/6 Creating template SimMachine + composing DelmiaReceiver..."
MACHINE_ID=$("$CLI" template create "${A[@]}" --name SimMachine \
--description "vd03-style machine stand-in: composes DelmiaReceiverSim as DelmiaReceiver; in-memory only" | json_id)
"$CLI" template composition add "${A[@]}" --template-id "$MACHINE_ID" --composed-template-id "$RECV_ID" --instance-name DelmiaReceiver > /dev/null
echo " id=$MACHINE_ID"
echo "4/6 Adding ProcessRecipeDownload (Call) — prod handshake on static attributes..."
read -r -d '' PROCESS_CODE <<'CS' || true
// Port of prod CvdReactor.ProcessRecipeDownload for STATIC attributes: WriteBatchAndWaitAsync only
// accepts data-sourced attributes, so the payload is written with SetAsync and the handshake waited
// with WaitAsync (same trigger/response flags, same 25 s budget, same result mapping).
var recv = Children["DelmiaReceiver"];
System.Func IsTrue = v => { var s = v?.ToString(); return (v is bool b && b) || s == "1" || string.Equals(s, "true", System.StringComparison.OrdinalIgnoreCase); };
System.Func P = k => Parameters.ContainsKey(k) ? Parameters[k] : null;
if (!IsTrue(await recv.Attributes.GetAsync("ReadyFlag")))
return new { Result = false, ResultText = "Receiver ready flag not set to true" };
try {
await recv.Attributes.SetAsync("RecipeProcessedFlag", false);
await recv.Attributes.SetAsync("DownloadPath", P("DownloadPath"));
await recv.Attributes.SetAsync("WorkOrderNumber", P("WorkOrderNumber"));
await recv.Attributes.SetAsync("PartNumber", P("PartNumber"));
await recv.Attributes.SetAsync("JobStepNumber", P("JobStepNumber"));
await recv.Attributes.SetAsync("Username", P("Username"));
await recv.Attributes.SetAsync("RecipeDownloadFlag", true);
if (!await recv.Attributes.WaitAsync("RecipeProcessedFlag", true, System.TimeSpan.FromSeconds(25)))
return new { Result = false, ResultText = "Timeout waiting for recipe to be processed" };
return new {
Result = IsTrue(await recv.Attributes.GetAsync("RecipeProcessResult")),
ResultText = (await recv.Attributes.GetAsync("RecipeProcessResultText"))?.ToString() ?? ""
};
} catch (Exception ex) {
return new { Result = false, ResultText = "RecipeDownload failed: " + ex.Message };
}
CS
"$CLI" template script add "${A[@]}" --template-id "$MACHINE_ID" --name ProcessRecipeDownload --trigger-type Call \
--parameters '{"type":"object","properties":{"DownloadPath":{"type":"string"},"WorkOrderNumber":{"type":"string"},"PartNumber":{"type":"string"},"JobStepNumber":{"type":"string"},"Username":{"type":"string"}},"required":[]}' \
--return-def '{"type":"object","properties":{"Result":{"type":"boolean"},"ResultText":{"type":"string"}},"required":["Result","ResultText"]}' \
--code "$PROCESS_CODE" > /dev/null
echo "5/6 Adding DelmiaReceiverSimulator (ValueChange on DelmiaReceiver.RecipeDownloadFlag)..."
read -r -d '' SIM_CODE <<'CS' || true
// Plays the Galaxy DelmiaReceiver object. Fires on every write to RecipeDownloadFlag (root-scope
// trigger, canonical name) and acts only on the rising edge; its own reset to false re-triggers
// this script, which then returns at the guard.
var recv = Children["DelmiaReceiver"];
var flag = await recv.Attributes.GetAsync("RecipeDownloadFlag");
var fs = flag?.ToString();
if (!((flag is bool b && b) || fs == "1" || string.Equals(fs, "true", System.StringComparison.OrdinalIgnoreCase))) return null;
var path = (await recv.Attributes.GetAsync("DownloadPath"))?.ToString() ?? "";
var wo = (await recv.Attributes.GetAsync("WorkOrderNumber"))?.ToString() ?? "";
var ok = path.Length > 0 && wo.Length > 0 && !path.EndsWith(".bad", System.StringComparison.OrdinalIgnoreCase);
await recv.Attributes.SetAsync("RecipeProcessResult", ok);
await recv.Attributes.SetAsync("RecipeProcessResultText", ok ? "Simulated download accepted for " + wo : "Simulated reject: bad path or missing work order");
await recv.Attributes.SetAsync("RecipeDownloadFlag", false);
await recv.Attributes.SetAsync("RecipeProcessedFlag", true);
return null;
CS
"$CLI" template script add "${A[@]}" --template-id "$MACHINE_ID" --name DelmiaReceiverSimulator --trigger-type ValueChange \
--trigger-config '{"attributeName":"DelmiaReceiver.RecipeDownloadFlag"}' --code "$SIM_CODE" > /dev/null
echo "6/6 Creating + deploying instance $INSTANCE_NAME on site $SITE_ID..."
INSTANCE_ID=$("$CLI" instance create "${A[@]}" --name "$INSTANCE_NAME" --template-id "$MACHINE_ID" --site-id "$SITE_ID" | json_id)
"$CLI" instance deploy "${A[@]}" --id "$INSTANCE_ID"
echo ""
echo "DELMIA_SIM_TEMPLATE_ID=$RECV_ID SIM_MACHINE_TEMPLATE_ID=$MACHINE_ID INSTANCE_ID=$INSTANCE_ID INSTANCE_NAME=$INSTANCE_NAME"
echo "Verify: $CLI debug snapshot --id $INSTANCE_ID --url http://localhost:9000 --username multi-role --password password"
```
**Step 2: Make it executable and run it against the rig**
Run: `chmod +x docker/seed-delmia-sim.sh && bash docker/seed-delmia-sim.sh 2>&1 | tail -15`
Expected: six numbered lines, a deploy reply without `"error"`, and the final `DELMIA_SIM_TEMPLATE_ID=… INSTANCE_ID=…` line. Record `INSTANCE_ID` for Task 4.
If `template script add` fails with a trust-policy or compile error, the message names the offending symbol — fix the script text, do **not** widen the trust policy. If `instance deploy` reports a pre-deployment validation failure mentioning bindings, a data-source was set by mistake: static attributes must be created **without** `--data-source`.
**Step 3: Prove the handshake works site-side before any HTTP is involved**
Run (bash):
```bash
CLI=src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge
$CLI debug snapshot --id --url http://localhost:9000 --format=json --username multi-role --password password | jq '.attributes // . | with_entries(select(.key|startswith("DelmiaReceiver.")))'
```
Expected: ten `DelmiaReceiver.*` entries, `ReadyFlag` true, `RecipeProcessedFlag` false. (The snapshot JSON shape is the CLI's — adapt the `jq` path to what comes back; the point is that all ten attributes exist under the `DelmiaReceiver.` prefix.)
**Step 4: Add the script to the docker tree listing**
`docker/README.md` line 97: after the `seed-live-probe.sh` row add
`├── seed-delmia-sim.sh # Seed the in-memory Delmia machine stand-in (DelmiaSim01) on site-a`
**Step 5: Commit**
```bash
git add docker/seed-delmia-sim.sh docker/README.md
git commit -m "chore(docker): seed-delmia-sim.sh — in-memory SimMachine + composed DelmiaReceiver stand-in on site-a with a ValueChange simulator playing the Galaxy side"
```
---
### Task 3: `docker/seed-delmia-api.sh` — inbound method + scoped API key
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 1, Task 2
**Model:** sonnet
**Files:**
- Create: `docker/seed-delmia-api.sh`
- Modify: `docker/README.md` (one tree-listing line under the Task 2 line)
**Step 1: Write the script**
```bash
#!/bin/bash
set -euo pipefail
# Seed the DelmiaRecipeDownload inbound method (prod's script verbatim, minus its dead `suf`
# line — see docs/api-specs/Delmia-Recipe-Download-API.md §4) and one API key scoped to it.
# Prints the one-time token as DELMIA_API_KEY=... — capture it; it cannot be retrieved again.
# Not idempotent (method name must be unique): delete with `api-method delete --id`.
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
CLI="$PROJECT_ROOT/src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge"
A=(--url "${SCADABRIDGE_URL:-http://localhost:9000}" --format=json --username "${SCADABRIDGE_USERNAME:-multi-role}" --password "${SCADABRIDGE_PASSWORD:-password}")
KEY_NAME="${KEY_NAME:-delmia-notifier}"
read -r -d '' METHOD_CODE <<'CS' || true
try {
System.Func P = k => Parameters.ContainsKey(k) ? Parameters[k] : null;
var code = (P("MachineCode") as string ?? "").Trim();
if (string.IsNullOrEmpty(code)) return new { Result = false, ResultText = "MachineCode is required" };
return await Route.To(code).Call("ProcessRecipeDownload", new {
DownloadPath = P("DownloadPath"),
WorkOrderNumber = P("WorkOrderNumber"),
PartNumber = P("PartNumber"),
JobStepNumber = P("JobStepNumber"),
Username = P("Username"),
});
} catch (Exception ex) {
return new { Result = false, ResultText = "RecipeDownload failed: " + ex.Message };
}
CS
echo "1/2 Creating inbound method DelmiaRecipeDownload (timeout 45 s)..."
METHOD_JSON=$("$CLI" api-method create "${A[@]}" --name DelmiaRecipeDownload --timeout 45 \
--parameters '{"type":"object","properties":{"MachineCode":{"type":"string"},"DownloadPath":{"type":"string"},"WorkOrderNumber":{"type":"string"},"PartNumber":{"type":"string"},"JobStepNumber":{"type":"string"},"Username":{"type":"string"}},"required":[]}' \
--return-def '{"type":"object","properties":{"Result":{"type":"boolean"},"ResultText":{"type":"string"}},"required":["Result","ResultText"]}' \
--script "$METHOD_CODE")
echo " $(echo "$METHOD_JSON" | python3 -c 'import json,sys; d=json.load(sys.stdin); print("id=%s name=%s" % (d["id"], d["name"]))')"
echo "2/2 Creating API key '$KEY_NAME' scoped to DelmiaRecipeDownload..."
KEY_JSON=$("$CLI" security api-key create "${A[@]}" --name "$KEY_NAME" --methods DelmiaRecipeDownload 2>/dev/null)
TOKEN=$(echo "$KEY_JSON" | python3 -c 'import json,sys; print(json.load(sys.stdin)["token"])')
echo "DELMIA_API_KEY=$TOKEN"
```
> Note: `security api-key create` prints the token via `PrintCreatedKey` in table mode; with `--format=json` the raw `{keyId, token}` document is emitted and parsed above. If the json branch still prints the human lines, grep the `sbk_` token instead: `TOKEN=$(echo "$KEY_JSON" | grep -o 'sbk_[A-Za-z0-9_-]*' | head -1)`.
**Step 2: Run it and keep the token OUT of git**
Run: `chmod +x docker/seed-delmia-api.sh && bash docker/seed-delmia-api.sh`
Expected: `id= name=DelmiaRecipeDownload` then `DELMIA_API_KEY=sbk_…`. Save the token to the scratchpad only (e.g. `$SCRATCH/delmia.env` with `export DELMIA_API_KEY=…`). Never write it into `src/**/appsettings.json`.
**Step 3: Smoke the method with curl (no notifier yet)**
```bash
curl -s -X POST http://localhost:9000/api/DelmiaRecipeDownload -H "X-API-Key: $DELMIA_API_KEY" -H "Content-Type: application/json" \
-d '{"MachineCode":"DelmiaSim01","DownloadPath":"C:\\recipes\\wo1.nc","WorkOrderNumber":"WO-1","PartNumber":"PN-1","JobStepNumber":"0100","Username":"curl"}'
```
Expected: `{"Result":true,"ResultText":"Simulated download accepted for WO-1"}`. Then the reject path (`"DownloadPath":"C:\\recipes\\x.bad"`) → `Result:false`, `Simulated reject…`; and a bogus key → HTTP 401 `{"error":"Invalid or missing API key"}`. If the first call returns `Timeout waiting for recipe to be processed`, the simulator did not fire: check `scadabridge debug snapshot` for `DelmiaReceiver.RecipeDownloadFlag` (should have been reset to false) and the site-a-a container log for a compile error on `DelmiaReceiverSimulator`.
**Step 4: Commit**
```bash
git add docker/seed-delmia-api.sh docker/README.md
git commit -m "chore(docker): seed-delmia-api.sh — DelmiaRecipeDownload inbound method (prod script) + scoped delmia-notifier API key"
```
---
### Task 4: `docker/verify-delmia-notifier.sh` — notifier end-to-end + central failover drill
**Classification:** high-risk (drives `docker stop` on a live rig node; interprets failover timing)
**Estimated implement time:** ~5 min to write, ~6 min to run (two drills × ~1 min settle)
**Parallelizable with:** none (needs Tasks 1–3)
**Model:** **opus** — run interactively, read the outputs, and decide whether a `NO` is an expected takeover-window answer or a real fault before proceeding.
**Files:**
- Create: `docker/verify-delmia-notifier.sh`
- Modify: `docker/README.md` (one tree-listing line)
**Step 1: Write the script**
```bash
#!/bin/bash
set -euo pipefail
# End-to-end DelmiaNotifier verification against the docker rig, including an ACTIVE central
# crash + rejoin. Requires: seed-delmia-sim.sh + seed-delmia-api.sh already run, and
# DELMIA_API_KEY exported (the token printed by seed-delmia-api.sh).
#
# The notifier is run as the managed build (dotnet WWNotifier.dll) from a scratch copy so the
# API key is written to THAT copy's appsettings.json (ScadaBridge:ApiKey — Task 1) and never
# into the repo. BaseUrls lists the two central nodes DIRECTLY (not Traefik) so the notifier's
# own connect-failure failover is what gets exercised.
#
# Expected outcomes:
# A baseline (active listed first) -> YES
# B standby listed first (standby relays) -> YES (StandbyForwardPeerEndpoints)
# C bad key -> NO (HTTP 401, no failover: 401 is an answer)
# D active STOPPED, before takeover completes -> NO or YES — informational only. A NO here is the
# documented connect-failure-only design (standby answers 503 ACTIVE_NODE_UNREACHABLE).
# E active STOPPED, after survivor /health/active=200 -> YES (first URL refused -> failover)
# F victim restarted and rejoined (members=2) -> YES
# G reject path (DownloadPath *.bad) -> NO + "Simulated reject"
: "${DELMIA_API_KEY:?export DELMIA_API_KEY=sbk_... first}"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"; PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
RUN_DIR="${RUN_DIR:-${TMPDIR:-/tmp}/delmia-notifier-run}"
MACHINE="${MACHINE:-DelmiaSim01}"
A_URL=http://localhost:9001; B_URL=http://localhost:9002
echo "== build notifier + stage a runnable copy in $RUN_DIR"
dotnet build "$PROJECT_ROOT/src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier" -c Debug -nologo -v q
rm -rf "$RUN_DIR"; mkdir -p "$RUN_DIR"; cp -R "$PROJECT_ROOT/src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/bin/Debug/net10.0/." "$RUN_DIR/"
write_cfg() { # $1 = BaseUrls, $2 = ApiKey
printf '{"ScadaBridge":{"BaseUrls":"%s","TimeoutSeconds":30,"LogPath":"delmia-notifier.log","ApiKey":"%s"}}\n' "$1" "$2" > "$RUN_DIR/appsettings.json"
}
notify() { # $1 = label, $2 = expected YES|NO, rest = extra args; prints stdout + exit code
local label=$1 expect=$2; shift 2
local out rc; set +e; out=$(env -u SCADABRIDGE_API_KEY dotnet "$RUN_DIR/WWNotifier.dll" -m "$MACHINE" -d 'C:\recipes\wo1.nc' -w "WO-$label" -p PN-1 -s 0100 -u drill "$@" 2>/dev/null); rc=$?; set -e
local first; first=$(echo "$out" | head -1)
printf '%-3s stdout=%-4s exit=%-3s | %s\n' "$label" "$first" "$rc" "$(echo "$out" | sed -n 2p)"
[ "$first" = "$expect" ] || { echo " ^^ EXPECTED $expect"; return 1; }
}
active() { curl -sf -o /dev/null "$A_URL/health/active" && echo a || { curl -sf -o /dev/null "$B_URL/health/active" && echo b; }; }
url_of() { [ "$1" = a ] && echo "$A_URL" || echo "$B_URL"; }
container_of() { [ "$1" = a ] && echo scadabridge-central-a || echo scadabridge-central-b; }
other() { [ "$1" = a ] && echo b || echo a; }
wait_for() { # $1 url $2 path $3 seconds
local i; for i in $(seq 1 "$3"); do curl -sf -o /dev/null "$1$2" && return 0; sleep 1; done; return 1; }
ACT=$(active); STB=$(other "$ACT")
echo "== active=central-$ACT standby=central-$STB"
echo "== A baseline (active first)"; write_cfg "$(url_of "$ACT"),$(url_of "$STB")" "$DELMIA_API_KEY"; notify A YES
echo "== B standby first (relay)"; write_cfg "$(url_of "$STB"),$(url_of "$ACT")" "$DELMIA_API_KEY"; notify B YES
echo "== C bad key"; write_cfg "$(url_of "$ACT"),$(url_of "$STB")" "sbk_bogus_bogus"; notify C NO || true
write_cfg "$(url_of "$ACT"),$(url_of "$STB")" "$DELMIA_API_KEY"
echo "== D/E stop the ACTIVE central ($(container_of "$ACT"))"; docker stop "$(container_of "$ACT")" > /dev/null
T0=$(date +%s)
echo "-- D immediate attempt (informational):"; notify D NO || notify D YES || true
echo "-- waiting for survivor $(url_of "$STB")/health/active (auto-down ~25-30 s)..."
wait_for "$(url_of "$STB")" /health/active 90 || { echo "FAIL: survivor never became active"; exit 1; }
echo " takeover after $(( $(date +%s) - T0 )) s"
echo "== E after takeover (first URL refused -> failover)"; notify E YES
echo "== F restart victim + wait for rejoin"; docker start "$(container_of "$ACT")" > /dev/null
wait_for "$(url_of "$ACT")" /health/ready 120 || { echo "FAIL: victim never ready"; exit 1; }
for i in $(seq 1 60); do curl -s "$(url_of "$STB")/health/ready" | grep -q '"members":2\|members=2\|Up.*Up' && break; sleep 2; done
notify F YES
echo "== G reject path"; notify G NO -d 'C:\recipes\x.bad' || true
echo "== done. Notifier log: $RUN_DIR/delmia-notifier.log"
```
> The `members` probe in F is deliberately loose — check what `/health/ready` actually prints on this build (`curl -s localhost:9001/health/ready`) and tighten the grep to the real field before relying on it.
**Step 2: Run it**
```bash
chmod +x docker/verify-delmia-notifier.sh
export DELMIA_API_KEY=sbk_… # from Task 3
bash docker/verify-delmia-notifier.sh 2>&1 | tee "$SCRATCH/verify-delmia.log"
```
Expected: A, B, E, F print `stdout=YES exit=0`; C and G print `stdout=NO`; D is informational (record which it was and the takeover seconds). The victim rejoins as **standby** (`/health/active` 503 on it, 200 on the survivor). Any other outcome is a real finding — capture the notifier log line (`attempt : …`) and the central log (`docker logs --since 5m scadabridge-central- | grep -i delmia`) before changing anything.
**Step 3: Confirm the audit trail**
```bash
CLI=src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge
$CLI audit query --channel ApiInbound --since 15m --url http://localhost:9000 --format=json --username multi-role --password password | grep DelmiaRecipeDownload | jq -c '{t:.occurredAtUtc, kind, status, http:.httpStatus, node:.sourceNode, actor, resp:.responseSummary}'
```
Expected: one `InboundRequest` row per YES/NO business answer, **exactly one row for B** (the standby relay must not double-audit — `InboundApiEndpointFilter.RelayedToActiveItemKey`), an `InboundAuthFailure` for C, and `sourceNode` showing the node that executed each call (B and E should show the node that was active at the time).
**Step 4: Commit the script**
```bash
git add docker/verify-delmia-notifier.sh docker/README.md
git commit -m "test(docker): verify-delmia-notifier.sh — notifier end-to-end on the rig incl. standby relay, active-central crash/takeover and rejoin"
```
---
### Task 5: Record results + spec/doc updates
**Classification:** trivial (docs only)
**Estimated implement time:** ~4 min
**Parallelizable with:** none (needs Task 4's numbers)
**Model:** sonnet
**Files:**
- Modify: `docs/api-specs/Delmia-Recipe-Download-API.md` (new §8 "Docker rig stand-in", pointing at the two seed scripts, the static-attribute port, and the verified outcomes A–G with the measured takeover time)
- Modify: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md` (`## Manual smoke test` → add the rig recipe: run the three scripts, `export DELMIA_API_KEY`, expected YES/NO table)
- Modify: `docs/plans/2026-07-08-deferred-work-register.md` (one new row: "prod `ProcessRecipeDownload` still addresses `Children[\"DelmiaRecv\"]` after the 2026-06-27 rename — fix + redeploy Z28061/Z28062/Z28061Sim; QA unverified" — only if a row for it does not already exist)
- Modify: `CLAUDE.md` — in the `### Templates & Deployment` / DelmiaNotifier context, one sentence: "Rig stand-in: `docker/seed-delmia-sim.sh` + `seed-delmia-api.sh` + `verify-delmia-notifier.sh` (static-attribute port of the prod handshake — `WriteBatchAndWaitAsync` rejects static attributes, use `SetAsync`/`WaitAsync`)."
**Step 1: Write the sections** using the numbers from Task 4's log (takeover seconds, D's outcome, audit row count).
**Step 2: Check no stale claims remain**
Run: `grep -rn "never read from or written to a file\|env-only by design" src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier docs env_variables.md CLAUDE.md`
Expected: no hits (Task 1 removed the README/env ones; anything left here is stale and must be updated).
**Step 3: Commit**
```bash
git add docs/api-specs/Delmia-Recipe-Download-API.md src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md docs/plans/2026-07-08-deferred-work-register.md CLAUDE.md
git commit -m "docs: Delmia rig stand-in + notifier verification results (A-G incl. active-central takeover); register row for prod DelmiaRecv path defect"
```
---
### Task 6: Final integration review
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Model:** **fable** (this session) — read `git log --oneline main..HEAD` + `git diff main`, re-run `dotnet test tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests`, and check specifically: (1) no `sbk_` token anywhere in the diff (`git diff main | grep -n 'sbk_[A-Za-z0-9]'` must return only the placeholder in the bad-key test), (2) the env-over-file precedence is asserted by a test, (3) the seed scripts leave the rig in a state `docker/README.md` describes, (4) Task 4's log shows E and F as YES with the victim rejoined as standby. Then decide with the user whether to push (`/pushit`) — the plan does not push.
---
## Out of scope (deliberately)
- Fixing prod vd03's `Children["DelmiaRecv"]` defect or making the prod method idempotent — recorded in the deferred-work register (Task 5); separate change with its own deployment record.
- Extending `WriteBatchAndWaitAsync` to static attributes.
- Site-pair failover (the user asked for the 2-node **central** cluster); `docker/failover-drill.sh` already covers the general case.
- Native AOT publish of the notifier (Windows-only); the managed build is what the rig exercises.