ScadaBridge/docs/plans/2026-09-02-delmia-sim-rig-verification.md

Delmia Simulated Machine + Notifier Rig Verification — Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.

Goal: Stand up an in-memory (static-attribute) stand-in for a vd03-style machine with a composed DelmiaReceiver on the local docker rig, wire the DelmiaRecipeDownload inbound method and a scoped API key to it, let DelmiaNotifier read that key from appsettings.json, and prove the notifier end-to-end on the 2-node central cluster including an active-node crash and rejoin.

Architecture: Two templates seeded by CLI only (no DB edits): DelmiaReceiverSim (the ten receiver attributes, all static — no data connection) composed into SimMachine as DelmiaReceiver. The Galaxy side of the handshake is replaced by a root-scope ValueChange script on SimMachine that reacts to DelmiaReceiver.RecipeDownloadFlag, so the same call script shape as prod runs against pure in-memory tags. The central inbound method is the prod script verbatim (minus its dead suf line). The notifier gains an optional ScadaBridge:ApiKey config field with the existing SCADABRIDGE_API_KEY env var taking precedence, and a bash verification script drives the whole thing against docker/ (central-a :9001, central-b :9002, Traefik :9000).

Tech Stack: ScadaBridge CLI (scadabridge), bash, .NET 10 (DelmiaNotifier + xUnit tests), docker compose rig under docker/.

Facts the implementer must not re-derive (verified 2026-09-02):

• WriteBatchAndWaitAsync rejects static attributes (InstanceActor.HandleWriteAttributeBatch: "Attribute 'X' is not a data-sourced attribute"). The rig script therefore uses SetAsync per attribute + WaitAsync, which both work on static attributes. Do not "fix" this in the runtime — out of scope.
• A static write always publishes AttributeValueChanged (no equality short-circuit), and ScriptActor matches a ValueChange trigger by exact canonical name (valueTrigger.AttributeName == changed.AttributeName). Composed scripts' trigger configs are not prefixed at flattening (alarms are), so the simulator script lives on the root template with attributeName = DelmiaReceiver.RecipeDownloadFlag.
• WaitAsync has a fast path when the current value already matches, so the simulator finishing before the wait registers is not a race.
• Trigger types are matched case-insensitively as strings: Call, ValueChange, Interval, Conditional, Expression. Trigger config key is attributeName.
• CLI global options (--url, --username, --password, --format=json) must come after the subcommand. Rig creds: --username multi-role --password password. Write loops in bash (zsh does not word-split unquoted variables).
• security api-key create --format=json returns { "keyId": …, "token": "sbk_…" }; the token is shown once.
• Rig: site-a is --site-id 1; Traefik http://localhost:9000 → active central; direct nodes :9001 (central-a) and :9002 (central-b); both centrals have StandbyForwardPeerEndpoints set, so a standby relays inbound calls to the active peer. Auto-down failover of a crashed active takes ~25–30 s (docker/failover-drill.sh).
• The notifier's failover is connect-failure-only: any HTTP answer (incl. 503) is final. During the takeover window a standby that cannot reach its peer answers 503 ACTIVE_NODE_UNREACHABLE → the notifier prints NO. That is expected and must be documented, not "fixed".
• Prod's deployed method + site script are recorded in docs/api-specs/Delmia-Recipe-Download-API.md; prod's site script still references Children["DelmiaRecv"] (a known, separate defect). The rig script uses Children["DelmiaReceiver"].

Model assignment summary (see each task's Model: line): Sonnet for scripted CLI seeding, config plumbing, and docs; Opus for the live rig verification and the failover drill (judgment under a live system) and for reviewing the secret-handling change; Fable (this session) for the final integration review.

---

Task 0: Preflight — rig healthy, CLI built, notifier tests green

Classification: trivial Estimated implement time: ~3 min Parallelizable with: none (everything depends on it) Model: sonnet

Files:

• (none edited)

Step 1: Confirm the rig is up and which node is active

Run:

for p in 9000 9001 9002; do printf "$p ready="; curl -s -m 5 -o /dev/null -w "%{http_code}" http://localhost:$p/health/ready; printf " active="; curl -s -m 5 -o /dev/null -w "%{http_code}\n" http://localhost:$p/health/active; done

Expected: all three ready=200; exactly one of 9001/9002 shows active=200, the other 503; 9000 active=200. If a node is down, run bash docker/deploy.sh first and re-check.

Step 2: Build the CLI and the notifier, run the notifier tests

Run:

cd /Users/dohertj2/Desktop/ScadaBridge
dotnet build src/ZB.MOM.WW.ScadaBridge.CLI -c Debug 2>&1 | tail -3
dotnet test tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests 2>&1 | tail -5

Expected: Build succeeded and Passed! with 0 failed. Note the CLI binary path used by every later task: src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge.

Step 3: Confirm the names this plan creates are free on the rig

Run:

CLI=src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge; A="--url http://localhost:9000 --format=json --username multi-role --password password"
bash -c "$CLI template list $A | jq -r '.[].name' | grep -E '^(DelmiaReceiverSim|SimMachine)$'; $CLI api-method list $A | jq -r '.[].name' | grep -E '^DelmiaRecipeDownload$'; $CLI instance list $A | jq -r '.[].uniqueName' | grep -E '^DelmiaSim01$'; echo done"

Expected: only done printed. If any name is taken, delete it (instance delete --id, template delete --id, api-method delete --id) before continuing — the seed script is not idempotent by design.

---

Task 1: DelmiaNotifier — optional ApiKey in appsettings.json, env var wins

Classification: small (single-project logic change, but it relaxes a documented secret-handling decision — reviewer must weigh in) Estimated implement time: ~5 min Parallelizable with: Task 2, Task 3 Model: sonnet (implement) → opus code review, with the explicit question "is the env-over-file precedence and the README wording adequate for a key that now may live on disk?"

Design note (state this in the commit): the 2026-06-26 design deliberately kept the key env-only. The user has asked for the key in appsettings.json. Keep env-only as the recommended production posture, add the file as a fallback, and keep the precedence deterministic: SCADABRIDGE_API_KEY (if non-blank) → ScadaBridge:ApiKey (if non-blank) → not configured. The failure message names both sources.

Files:

• Modify: src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/NotifierConfig.cs
• Modify: src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/ConfigLoader.cs
• Modify: src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/Program.cs:42-47
• Modify: src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/appsettings.json
• Modify: src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md (the ## Secret — SCADABRIDGE_API_KEY section and the config JSON block)
• Modify: env_variables.md (§5 SCADABRIDGE_API_KEY row — "env-only by design" is no longer true)
• Modify: docs/plans/2026-06-26-delmia-recipe-notifier-design.md (the Secret row of the key-decisions table: add "superseded 2026-09-02: file fallback added, env still wins")
• Test: tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests/ConfigLoaderTests.cs

Step 1: Write the failing tests (append to ConfigLoaderTests.cs)

    [Fact]
    public void Load_reads_ApiKey_from_ScadaBridge_section()
    {
        var cfg = ConfigLoader.Load("{\"ScadaBridge\":{\"BaseUrls\":\"http://x\",\"ApiKey\":\"sbk_file\"}}");
        Assert.Equal("sbk_file", cfg.ScadaBridge.ApiKey);
    }

    [Fact]
    public void ResolveApiKey_prefers_env_over_config()
    {
        var cfg = ConfigLoader.Load("{\"ScadaBridge\":{\"ApiKey\":\"sbk_file\"}}");
        Assert.Equal("sbk_env", ConfigLoader.ResolveApiKey(n => n == "SCADABRIDGE_API_KEY" ? "sbk_env" : null, cfg));
    }

    [Fact]
    public void ResolveApiKey_falls_back_to_config_when_env_unset_or_blank()
    {
        var cfg = ConfigLoader.Load("{\"ScadaBridge\":{\"ApiKey\":\"  sbk_file \"}}");
        Assert.Equal("sbk_file", ConfigLoader.ResolveApiKey(_ => null, cfg));
        Assert.Equal("sbk_file", ConfigLoader.ResolveApiKey(_ => "   ", cfg));
    }

    [Fact]
    public void ResolveApiKey_returns_null_when_neither_source_set()
    {
        var cfg = ConfigLoader.Load("{\"ScadaBridge\":{\"ApiKey\":\"   \"}}");
        Assert.Null(ConfigLoader.ResolveApiKey(_ => null, cfg));
    }

Also update the two existing ResolveApiKey_* tests to pass a config: ConfigLoader.ResolveApiKey(_ => null, new NotifierConfig()) etc. (the one-argument overload is removed, not kept — YAGNI).

Step 2: Run tests to verify they fail

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests 2>&1 | tail -5 Expected: build error 'ScadaBridgeSection' does not contain a definition for 'ApiKey' / no overload for ResolveApiKey taking 2 arguments.

Step 3: Implement

NotifierConfig.cs — add to ScadaBridgeSection:

    /// <summary>Optional inbound API key. <c>SCADABRIDGE_API_KEY</c> in the environment takes precedence; this is the file fallback (design 2026-06-26, relaxed 2026-09-02).</summary>
    public string? ApiKey { get; set; }

ConfigLoader.cs — replace ResolveApiKey:

    /// <summary>Resolve the API key: <c>SCADABRIDGE_API_KEY</c> wins when non-blank, otherwise <c>ScadaBridge:ApiKey</c> from <paramref name="cfg"/> (trimmed); null when neither is set. Env accessor is injected for testability.</summary>
    /// <param name="envGet">Accessor used to read an environment variable by name.</param>
    /// <param name="cfg">The loaded configuration, consulted only when the environment variable is unset/blank.</param>
    /// <returns>The resolved API key, or <c>null</c> if neither source yields a non-blank value.</returns>
    public static string? ResolveApiKey(Func<string, string?> envGet, NotifierConfig cfg)
    {
        var env = envGet(ApiKeyEnvVar);
        if (!string.IsNullOrWhiteSpace(env)) return env;
        var file = cfg.ScadaBridge.ApiKey;
        return string.IsNullOrWhiteSpace(file) ? null : file.Trim();
    }

Program.cs lines 42–47 become:

            var apiKey = ConfigLoader.ResolveApiKey(Environment.GetEnvironmentVariable, cfg);
            if (apiKey is null)
            {
                log.Write("API key not configured (SCADABRIDGE_API_KEY or ScadaBridge:ApiKey)");
                return Reporter.Report(false, "API key not configured (SCADABRIDGE_API_KEY or ScadaBridge:ApiKey)", stdout);
            }

appsettings.json — add "ApiKey": "" after LogPath (empty = not set; never commit a real token).

README: in the config JSON block add "ApiKey": ""; rewrite the ## Secret section:

The inbound API key is sent as X-API-Key. It is resolved from SCADABRIDGE_API_KEY first (recommended for production — the key never touches disk), falling back to ScadaBridge:ApiKey in appsettings.json when the variable is unset or blank. If neither is set → NO + API key not configured (SCADABRIDGE_API_KEY or ScadaBridge:ApiKey), exit -1, no HTTP attempt. If you use the file, restrict its ACL to the account DELMIA runs under.

env_variables.md §5: change the "In appsettings?" cell to Yes — optional \ScadaBridge:ApiKey` fallback (2026-09-02); env wins` and the Purpose cell's "Read only from the environment, never a file" to "Environment first, file fallback".

Design doc table row | Secret | API key from env var … — never in a file | → append **(relaxed 2026-09-02: optional \ScadaBridge:ApiKey` file fallback; env still wins)**`.

Step 4: Run tests to verify they pass

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests 2>&1 | tail -5 Expected: Passed!, 0 failed, total = previous count + 4. Also dotnet build src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier -warnaserror succeeds (the project treats warnings as errors; the XML doc comments above are required).

Step 5: Commit

git add src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests env_variables.md docs/plans/2026-06-26-delmia-recipe-notifier-design.md
git commit -m "feat(delmia-notifier): optional ScadaBridge:ApiKey in appsettings.json as a fallback to SCADABRIDGE_API_KEY (env still wins); README/env inventory/design row updated"

---

Task 2: docker/seed-delmia-sim.sh — templates, composition, scripts, instance, deploy

Classification: standard (new script + two template scripts that must compile under the trust policy) Estimated implement time: ~5 min Parallelizable with: Task 1, Task 3 Model: sonnet

Files:

• Create: docker/seed-delmia-sim.sh
• Modify: docker/README.md:96-97 (add the new script to the tree listing, one line)

Step 1: Write the script

#!/bin/bash
set -euo pipefail

# Seed an IN-MEMORY stand-in for a vd03-style machine with a composed DelmiaReceiver on
# site-a, so DelmiaNotifier can be exercised end-to-end on the docker rig without a Galaxy.
#
# What it creates (all static attributes — no data connection, no bindings):
#   - Template DelmiaReceiverSim — the ten DelmiaReceiver attributes from prod
#       (docs/api-specs/Delmia-Recipe-Download-API.md §5), ReadyFlag defaulting to true.
#   - Template SimMachine — composes DelmiaReceiverSim as `DelmiaReceiver` and carries
#       * ProcessRecipeDownload (Call)  — prod's handshake ported to per-attribute
#         SetAsync/WaitAsync (WriteBatchAndWaitAsync rejects static attributes).
#       * DelmiaReceiverSimulator (ValueChange on DelmiaReceiver.RecipeDownloadFlag) —
#         plays the Galaxy object: accepts unless DownloadPath ends in ".bad" or the work
#         order is blank, writes RecipeProcessResult/Text, clears the trigger flag, then
#         sets RecipeProcessedFlag=true.
#   - Instance DelmiaSim01 on site-a (id 1), deployed.
#
# Not idempotent: names must be unique. To re-seed:
#   scadabridge instance delete --id <id>; scadabridge template delete --id <SimMachine id>;
#   scadabridge template delete --id <DelmiaReceiverSim id>
# Prerequisites: bash docker/deploy.sh; docker/seed-sites.sh; central healthy on :9000.

SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
CLI="$PROJECT_ROOT/src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge"
A=(--url "${SCADABRIDGE_URL:-http://localhost:9000}" --format=json --username "${SCADABRIDGE_USERNAME:-multi-role}" --password "${SCADABRIDGE_PASSWORD:-password}")
SITE_ID="${SITE_ID:-1}"
INSTANCE_NAME="${INSTANCE_NAME:-DelmiaSim01}"

json_id() { python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])'; }

echo "1/6 Creating template DelmiaReceiverSim..."
RECV_ID=$("$CLI" template create "${A[@]}" --name DelmiaReceiverSim \
  --description "In-memory stand-in for the Galaxy DelmiaReceiver object (ten static attributes)" | json_id)
echo "  id=$RECV_ID"

echo "2/6 Adding the ten receiver attributes (static)..."
"$CLI" template attribute add "${A[@]}" --template-id "$RECV_ID" --name ReadyFlag           --data-type Boolean --value true  --description "Precondition; true = receiver accepts downloads" > /dev/null
"$CLI" template attribute add "${A[@]}" --template-id "$RECV_ID" --name RecipeDownloadFlag  --data-type Boolean --value false --description "Trigger: set true after the payload is written" > /dev/null
"$CLI" template attribute add "${A[@]}" --template-id "$RECV_ID" --name RecipeProcessedFlag --data-type Boolean --value false --description "Cleared by the caller, set true by the receiver when done" > /dev/null
"$CLI" template attribute add "${A[@]}" --template-id "$RECV_ID" --name RecipeProcessResult --data-type Boolean --value false --description "Outcome -> Result" > /dev/null
for s in DownloadPath WorkOrderNumber PartNumber JobStepNumber Username RecipeProcessResultText; do
  "$CLI" template attribute add "${A[@]}" --template-id "$RECV_ID" --name "$s" --data-type String --description "Recipe-download payload / outcome text" > /dev/null
done

echo "3/6 Creating template SimMachine + composing DelmiaReceiver..."
MACHINE_ID=$("$CLI" template create "${A[@]}" --name SimMachine \
  --description "vd03-style machine stand-in: composes DelmiaReceiverSim as DelmiaReceiver; in-memory only" | json_id)
"$CLI" template composition add "${A[@]}" --template-id "$MACHINE_ID" --composed-template-id "$RECV_ID" --instance-name DelmiaReceiver > /dev/null
echo "  id=$MACHINE_ID"

echo "4/6 Adding ProcessRecipeDownload (Call) — prod handshake on static attributes..."
read -r -d '' PROCESS_CODE <<'CS' || true
// Port of prod CvdReactor.ProcessRecipeDownload for STATIC attributes: WriteBatchAndWaitAsync only
// accepts data-sourced attributes, so the payload is written with SetAsync and the handshake waited
// with WaitAsync (same trigger/response flags, same 25 s budget, same result mapping).
var recv = Children["DelmiaReceiver"];
System.Func<object?,bool> IsTrue = v => { var s = v?.ToString(); return (v is bool b && b) || s == "1" || string.Equals(s, "true", System.StringComparison.OrdinalIgnoreCase); };
System.Func<string,object?> P = k => Parameters.ContainsKey(k) ? Parameters[k] : null;
if (!IsTrue(await recv.Attributes.GetAsync("ReadyFlag")))
    return new { Result = false, ResultText = "Receiver ready flag not set to true" };
try {
    await recv.Attributes.SetAsync("RecipeProcessedFlag", false);
    await recv.Attributes.SetAsync("DownloadPath", P("DownloadPath"));
    await recv.Attributes.SetAsync("WorkOrderNumber", P("WorkOrderNumber"));
    await recv.Attributes.SetAsync("PartNumber", P("PartNumber"));
    await recv.Attributes.SetAsync("JobStepNumber", P("JobStepNumber"));
    await recv.Attributes.SetAsync("Username", P("Username"));
    await recv.Attributes.SetAsync("RecipeDownloadFlag", true);
    if (!await recv.Attributes.WaitAsync("RecipeProcessedFlag", true, System.TimeSpan.FromSeconds(25)))
        return new { Result = false, ResultText = "Timeout waiting for recipe to be processed" };
    return new {
        Result = IsTrue(await recv.Attributes.GetAsync("RecipeProcessResult")),
        ResultText = (await recv.Attributes.GetAsync("RecipeProcessResultText"))?.ToString() ?? ""
    };
} catch (Exception ex) {
    return new { Result = false, ResultText = "RecipeDownload failed: " + ex.Message };
}
CS
"$CLI" template script add "${A[@]}" --template-id "$MACHINE_ID" --name ProcessRecipeDownload --trigger-type Call \
  --parameters '{"type":"object","properties":{"DownloadPath":{"type":"string"},"WorkOrderNumber":{"type":"string"},"PartNumber":{"type":"string"},"JobStepNumber":{"type":"string"},"Username":{"type":"string"}},"required":[]}' \
  --return-def '{"type":"object","properties":{"Result":{"type":"boolean"},"ResultText":{"type":"string"}},"required":["Result","ResultText"]}' \
  --code "$PROCESS_CODE" > /dev/null

echo "5/6 Adding DelmiaReceiverSimulator (ValueChange on DelmiaReceiver.RecipeDownloadFlag)..."
read -r -d '' SIM_CODE <<'CS' || true
// Plays the Galaxy DelmiaReceiver object. Fires on every write to RecipeDownloadFlag (root-scope
// trigger, canonical name) and acts only on the rising edge; its own reset to false re-triggers
// this script, which then returns at the guard.
var recv = Children["DelmiaReceiver"];
var flag = await recv.Attributes.GetAsync("RecipeDownloadFlag");
var fs = flag?.ToString();
if (!((flag is bool b && b) || fs == "1" || string.Equals(fs, "true", System.StringComparison.OrdinalIgnoreCase))) return null;
var path = (await recv.Attributes.GetAsync("DownloadPath"))?.ToString() ?? "";
var wo = (await recv.Attributes.GetAsync("WorkOrderNumber"))?.ToString() ?? "";
var ok = path.Length > 0 && wo.Length > 0 && !path.EndsWith(".bad", System.StringComparison.OrdinalIgnoreCase);
await recv.Attributes.SetAsync("RecipeProcessResult", ok);
await recv.Attributes.SetAsync("RecipeProcessResultText", ok ? "Simulated download accepted for " + wo : "Simulated reject: bad path or missing work order");
await recv.Attributes.SetAsync("RecipeDownloadFlag", false);
await recv.Attributes.SetAsync("RecipeProcessedFlag", true);
return null;
CS
"$CLI" template script add "${A[@]}" --template-id "$MACHINE_ID" --name DelmiaReceiverSimulator --trigger-type ValueChange \
  --trigger-config '{"attributeName":"DelmiaReceiver.RecipeDownloadFlag"}' --code "$SIM_CODE" > /dev/null

echo "6/6 Creating + deploying instance $INSTANCE_NAME on site $SITE_ID..."
INSTANCE_ID=$("$CLI" instance create "${A[@]}" --name "$INSTANCE_NAME" --template-id "$MACHINE_ID" --site-id "$SITE_ID" | json_id)
"$CLI" instance deploy "${A[@]}" --id "$INSTANCE_ID"
echo ""
echo "DELMIA_SIM_TEMPLATE_ID=$RECV_ID SIM_MACHINE_TEMPLATE_ID=$MACHINE_ID INSTANCE_ID=$INSTANCE_ID INSTANCE_NAME=$INSTANCE_NAME"
echo "Verify: $CLI debug snapshot --id $INSTANCE_ID --url http://localhost:9000 --username multi-role --password password"

Step 2: Make it executable and run it against the rig

Run: chmod +x docker/seed-delmia-sim.sh && bash docker/seed-delmia-sim.sh 2>&1 | tail -15 Expected: six numbered lines, a deploy reply without "error", and the final DELMIA_SIM_TEMPLATE_ID=… INSTANCE_ID=… line. Record INSTANCE_ID for Task 4.

If template script add fails with a trust-policy or compile error, the message names the offending symbol — fix the script text, do not widen the trust policy. If instance deploy reports a pre-deployment validation failure mentioning bindings, a data-source was set by mistake: static attributes must be created without --data-source.

Step 3: Prove the handshake works site-side before any HTTP is involved

Run (bash):

CLI=src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge
$CLI debug snapshot --id <INSTANCE_ID> --url http://localhost:9000 --format=json --username multi-role --password password | jq '.attributes // . | with_entries(select(.key|startswith("DelmiaReceiver.")))'

Expected: ten DelmiaReceiver.* entries, ReadyFlag true, RecipeProcessedFlag false. (The snapshot JSON shape is the CLI's — adapt the jq path to what comes back; the point is that all ten attributes exist under the DelmiaReceiver. prefix.)

Step 4: Add the script to the docker tree listing

docker/README.md line 97: after the seed-live-probe.sh row add ├── seed-delmia-sim.sh # Seed the in-memory Delmia machine stand-in (DelmiaSim01) on site-a

Step 5: Commit

git add docker/seed-delmia-sim.sh docker/README.md
git commit -m "chore(docker): seed-delmia-sim.sh — in-memory SimMachine + composed DelmiaReceiver stand-in on site-a with a ValueChange simulator playing the Galaxy side"

---

Task 3: docker/seed-delmia-api.sh — inbound method + scoped API key

Classification: small Estimated implement time: ~4 min Parallelizable with: Task 1, Task 2 Model: sonnet

Files:

• Create: docker/seed-delmia-api.sh
• Modify: docker/README.md (one tree-listing line under the Task 2 line)

Step 1: Write the script

#!/bin/bash
set -euo pipefail

# Seed the DelmiaRecipeDownload inbound method (prod's script verbatim, minus its dead `suf`
# line — see docs/api-specs/Delmia-Recipe-Download-API.md §4) and one API key scoped to it.
# Prints the one-time token as DELMIA_API_KEY=... — capture it; it cannot be retrieved again.
# Not idempotent (method name must be unique): delete with `api-method delete --id`.

SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
CLI="$PROJECT_ROOT/src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge"
A=(--url "${SCADABRIDGE_URL:-http://localhost:9000}" --format=json --username "${SCADABRIDGE_USERNAME:-multi-role}" --password "${SCADABRIDGE_PASSWORD:-password}")
KEY_NAME="${KEY_NAME:-delmia-notifier}"

read -r -d '' METHOD_CODE <<'CS' || true
try {
  System.Func<string, object> P = k => Parameters.ContainsKey(k) ? Parameters[k] : null;
  var code = (P("MachineCode") as string ?? "").Trim();
  if (string.IsNullOrEmpty(code)) return new { Result = false, ResultText = "MachineCode is required" };

  return await Route.To(code).Call("ProcessRecipeDownload", new {
      DownloadPath    = P("DownloadPath"),
      WorkOrderNumber = P("WorkOrderNumber"),
      PartNumber      = P("PartNumber"),
      JobStepNumber   = P("JobStepNumber"),
      Username        = P("Username"),
  });
} catch (Exception ex) {
  return new { Result = false, ResultText = "RecipeDownload failed: " + ex.Message };
}
CS

echo "1/2 Creating inbound method DelmiaRecipeDownload (timeout 45 s)..."
METHOD_JSON=$("$CLI" api-method create "${A[@]}" --name DelmiaRecipeDownload --timeout 45 \
  --parameters '{"type":"object","properties":{"MachineCode":{"type":"string"},"DownloadPath":{"type":"string"},"WorkOrderNumber":{"type":"string"},"PartNumber":{"type":"string"},"JobStepNumber":{"type":"string"},"Username":{"type":"string"}},"required":[]}' \
  --return-def '{"type":"object","properties":{"Result":{"type":"boolean"},"ResultText":{"type":"string"}},"required":["Result","ResultText"]}' \
  --script "$METHOD_CODE")
echo "  $(echo "$METHOD_JSON" | python3 -c 'import json,sys; d=json.load(sys.stdin); print("id=%s name=%s" % (d["id"], d["name"]))')"

echo "2/2 Creating API key '$KEY_NAME' scoped to DelmiaRecipeDownload..."
KEY_JSON=$("$CLI" security api-key create "${A[@]}" --name "$KEY_NAME" --methods DelmiaRecipeDownload 2>/dev/null)
TOKEN=$(echo "$KEY_JSON" | python3 -c 'import json,sys; print(json.load(sys.stdin)["token"])')
echo "DELMIA_API_KEY=$TOKEN"

Note: security api-key create prints the token via PrintCreatedKey in table mode; with --format=json the raw {keyId, token} document is emitted and parsed above. If the json branch still prints the human lines, grep the sbk_ token instead: TOKEN=$(echo "$KEY_JSON" | grep -o 'sbk_[A-Za-z0-9_-]*' | head -1).

Step 2: Run it and keep the token OUT of git

Run: chmod +x docker/seed-delmia-api.sh && bash docker/seed-delmia-api.sh Expected: id=<n> name=DelmiaRecipeDownload then DELMIA_API_KEY=sbk_…. Save the token to the scratchpad only (e.g. $SCRATCH/delmia.env with export DELMIA_API_KEY=…). Never write it into src/**/appsettings.json.

Step 3: Smoke the method with curl (no notifier yet)

curl -s -X POST http://localhost:9000/api/DelmiaRecipeDownload -H "X-API-Key: $DELMIA_API_KEY" -H "Content-Type: application/json" \
  -d '{"MachineCode":"DelmiaSim01","DownloadPath":"C:\\recipes\\wo1.nc","WorkOrderNumber":"WO-1","PartNumber":"PN-1","JobStepNumber":"0100","Username":"curl"}'

Expected: {"Result":true,"ResultText":"Simulated download accepted for WO-1"}. Then the reject path ("DownloadPath":"C:\\recipes\\x.bad") → Result:false, Simulated reject…; and a bogus key → HTTP 401 {"error":"Invalid or missing API key"}. If the first call returns Timeout waiting for recipe to be processed, the simulator did not fire: check scadabridge debug snapshot for DelmiaReceiver.RecipeDownloadFlag (should have been reset to false) and the site-a-a container log for a compile error on DelmiaReceiverSimulator.

Step 4: Commit

git add docker/seed-delmia-api.sh docker/README.md
git commit -m "chore(docker): seed-delmia-api.sh — DelmiaRecipeDownload inbound method (prod script) + scoped delmia-notifier API key"

---

Task 4: docker/verify-delmia-notifier.sh — notifier end-to-end + central failover drill

Classification: high-risk (drives docker stop on a live rig node; interprets failover timing) Estimated implement time: ~5 min to write, ~6 min to run (two drills × ~1 min settle) Parallelizable with: none (needs Tasks 1–3) Model: opus — run interactively, read the outputs, and decide whether a NO is an expected takeover-window answer or a real fault before proceeding.

Files:

• Create: docker/verify-delmia-notifier.sh
• Modify: docker/README.md (one tree-listing line)

Step 1: Write the script

#!/bin/bash
set -euo pipefail

# End-to-end DelmiaNotifier verification against the docker rig, including an ACTIVE central
# crash + rejoin. Requires: seed-delmia-sim.sh + seed-delmia-api.sh already run, and
# DELMIA_API_KEY exported (the token printed by seed-delmia-api.sh).
#
# The notifier is run as the managed build (dotnet WWNotifier.dll) from a scratch copy so the
# API key is written to THAT copy's appsettings.json (ScadaBridge:ApiKey — Task 1) and never
# into the repo. BaseUrls lists the two central nodes DIRECTLY (not Traefik) so the notifier's
# own connect-failure failover is what gets exercised.
#
# Expected outcomes:
#   A  baseline (active listed first)            -> YES
#   B  standby listed first (standby relays)     -> YES   (StandbyForwardPeerEndpoints)
#   C  bad key                                   -> NO    (HTTP 401, no failover: 401 is an answer)
#   D  active STOPPED, before takeover completes -> NO or YES — informational only. A NO here is the
#      documented connect-failure-only design (standby answers 503 ACTIVE_NODE_UNREACHABLE).
#   E  active STOPPED, after survivor /health/active=200 -> YES (first URL refused -> failover)
#   F  victim restarted and rejoined (members=2)  -> YES
#   G  reject path (DownloadPath *.bad)           -> NO + "Simulated reject"

: "${DELMIA_API_KEY:?export DELMIA_API_KEY=sbk_... first}"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"; PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
RUN_DIR="${RUN_DIR:-${TMPDIR:-/tmp}/delmia-notifier-run}"
MACHINE="${MACHINE:-DelmiaSim01}"
A_URL=http://localhost:9001; B_URL=http://localhost:9002

echo "== build notifier + stage a runnable copy in $RUN_DIR"
dotnet build "$PROJECT_ROOT/src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier" -c Debug -nologo -v q
rm -rf "$RUN_DIR"; mkdir -p "$RUN_DIR"; cp -R "$PROJECT_ROOT/src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/bin/Debug/net10.0/." "$RUN_DIR/"

write_cfg() { # $1 = BaseUrls, $2 = ApiKey
  printf '{"ScadaBridge":{"BaseUrls":"%s","TimeoutSeconds":30,"LogPath":"delmia-notifier.log","ApiKey":"%s"}}\n' "$1" "$2" > "$RUN_DIR/appsettings.json"
}
notify() { # $1 = label, $2 = expected YES|NO, rest = extra args; prints stdout + exit code
  local label=$1 expect=$2; shift 2
  local out rc; set +e; out=$(env -u SCADABRIDGE_API_KEY dotnet "$RUN_DIR/WWNotifier.dll" -m "$MACHINE" -d 'C:\recipes\wo1.nc' -w "WO-$label" -p PN-1 -s 0100 -u drill "$@" 2>/dev/null); rc=$?; set -e
  local first; first=$(echo "$out" | head -1)
  printf '%-3s stdout=%-4s exit=%-3s | %s\n' "$label" "$first" "$rc" "$(echo "$out" | sed -n 2p)"
  [ "$first" = "$expect" ] || { echo "   ^^ EXPECTED $expect"; return 1; }
}
active() { curl -sf -o /dev/null "$A_URL/health/active" && echo a || { curl -sf -o /dev/null "$B_URL/health/active" && echo b; }; }
url_of() { [ "$1" = a ] && echo "$A_URL" || echo "$B_URL"; }
container_of() { [ "$1" = a ] && echo scadabridge-central-a || echo scadabridge-central-b; }
other() { [ "$1" = a ] && echo b || echo a; }
wait_for() { # $1 url $2 path $3 seconds
  local i; for i in $(seq 1 "$3"); do curl -sf -o /dev/null "$1$2" && return 0; sleep 1; done; return 1; }

ACT=$(active); STB=$(other "$ACT")
echo "== active=central-$ACT standby=central-$STB"

echo "== A baseline (active first)";        write_cfg "$(url_of "$ACT"),$(url_of "$STB")" "$DELMIA_API_KEY"; notify A YES
echo "== B standby first (relay)";           write_cfg "$(url_of "$STB"),$(url_of "$ACT")" "$DELMIA_API_KEY"; notify B YES
echo "== C bad key";                         write_cfg "$(url_of "$ACT"),$(url_of "$STB")" "sbk_bogus_bogus"; notify C NO || true
write_cfg "$(url_of "$ACT"),$(url_of "$STB")" "$DELMIA_API_KEY"

echo "== D/E stop the ACTIVE central ($(container_of "$ACT"))"; docker stop "$(container_of "$ACT")" > /dev/null
T0=$(date +%s)
echo "-- D immediate attempt (informational):"; notify D NO || notify D YES || true
echo "-- waiting for survivor $(url_of "$STB")/health/active (auto-down ~25-30 s)..."
wait_for "$(url_of "$STB")" /health/active 90 || { echo "FAIL: survivor never became active"; exit 1; }
echo "   takeover after $(( $(date +%s) - T0 )) s"
echo "== E after takeover (first URL refused -> failover)"; notify E YES

echo "== F restart victim + wait for rejoin"; docker start "$(container_of "$ACT")" > /dev/null
wait_for "$(url_of "$ACT")" /health/ready 120 || { echo "FAIL: victim never ready"; exit 1; }
for i in $(seq 1 60); do curl -s "$(url_of "$STB")/health/ready" | grep -q '"members":2\|members=2\|Up.*Up' && break; sleep 2; done
notify F YES

echo "== G reject path"; notify G NO -d 'C:\recipes\x.bad' || true
echo "== done. Notifier log: $RUN_DIR/delmia-notifier.log"

The members probe in F is deliberately loose — check what /health/ready actually prints on this build (curl -s localhost:9001/health/ready) and tighten the grep to the real field before relying on it.

Step 2: Run it

chmod +x docker/verify-delmia-notifier.sh
export DELMIA_API_KEY=sbk_…   # from Task 3
bash docker/verify-delmia-notifier.sh 2>&1 | tee "$SCRATCH/verify-delmia.log"

Expected: A, B, E, F print stdout=YES exit=0; C and G print stdout=NO; D is informational (record which it was and the takeover seconds). The victim rejoins as standby (/health/active 503 on it, 200 on the survivor). Any other outcome is a real finding — capture the notifier log line (attempt <url>: …) and the central log (docker logs --since 5m scadabridge-central-<x> | grep -i delmia) before changing anything.

Step 3: Confirm the audit trail

CLI=src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge
$CLI audit query --channel ApiInbound --since 15m --url http://localhost:9000 --format=json --username multi-role --password password | grep DelmiaRecipeDownload | jq -c '{t:.occurredAtUtc, kind, status, http:.httpStatus, node:.sourceNode, actor, resp:.responseSummary}'

Expected: one InboundRequest row per YES/NO business answer, exactly one row for B (the standby relay must not double-audit — InboundApiEndpointFilter.RelayedToActiveItemKey), an InboundAuthFailure for C, and sourceNode showing the node that executed each call (B and E should show the node that was active at the time).

Step 4: Commit the script

git add docker/verify-delmia-notifier.sh docker/README.md
git commit -m "test(docker): verify-delmia-notifier.sh — notifier end-to-end on the rig incl. standby relay, active-central crash/takeover and rejoin"

---

Task 5: Record results + spec/doc updates

Classification: trivial (docs only) Estimated implement time: ~4 min Parallelizable with: none (needs Task 4's numbers) Model: sonnet

Files:

• Modify: docs/api-specs/Delmia-Recipe-Download-API.md (new §8 "Docker rig stand-in", pointing at the two seed scripts, the static-attribute port, and the verified outcomes A–G with the measured takeover time)
• Modify: src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md (## Manual smoke test → add the rig recipe: run the three scripts, export DELMIA_API_KEY, expected YES/NO table)
• Modify: docs/plans/2026-07-08-deferred-work-register.md (one new row: "prod ProcessRecipeDownload still addresses Children[\"DelmiaRecv\"] after the 2026-06-27 rename — fix + redeploy Z28061/Z28062/Z28061Sim; QA unverified" — only if a row for it does not already exist)
• Modify: CLAUDE.md — in the ### Templates & Deployment / DelmiaNotifier context, one sentence: "Rig stand-in: docker/seed-delmia-sim.sh + seed-delmia-api.sh + verify-delmia-notifier.sh (static-attribute port of the prod handshake — WriteBatchAndWaitAsync rejects static attributes, use SetAsync/WaitAsync)."

Step 1: Write the sections using the numbers from Task 4's log (takeover seconds, D's outcome, audit row count).

Step 2: Check no stale claims remain

Run: grep -rn "never read from or written to a file\|env-only by design" src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier docs env_variables.md CLAUDE.md Expected: no hits (Task 1 removed the README/env ones; anything left here is stale and must be updated).

Step 3: Commit

git add docs/api-specs/Delmia-Recipe-Download-API.md src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md docs/plans/2026-07-08-deferred-work-register.md CLAUDE.md
git commit -m "docs: Delmia rig stand-in + notifier verification results (A-G incl. active-central takeover); register row for prod DelmiaRecv path defect"

---

Task 6: Final integration review

Classification: standard Estimated implement time: ~5 min Parallelizable with: none Model: fable (this session) — read git log --oneline main..HEAD + git diff main, re-run dotnet test tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests, and check specifically: (1) no sbk_ token anywhere in the diff (git diff main | grep -n 'sbk_[A-Za-z0-9]' must return only the placeholder in the bad-key test), (2) the env-over-file precedence is asserted by a test, (3) the seed scripts leave the rig in a state docker/README.md describes, (4) Task 4's log shows E and F as YES with the victim rejoined as standby. Then decide with the user whether to push (/pushit) — the plan does not push.

---

Out of scope (deliberately)

• Fixing prod vd03's Children["DelmiaRecv"] defect or making the prod method idempotent — recorded in the deferred-work register (Task 5); separate change with its own deployment record.
• Extending WriteBatchAndWaitAsync to static attributes.
• Site-pair failover (the user asked for the 2-node central cluster); docker/failover-drill.sh already covers the general case.
• Native AOT publish of the notifier (Windows-only); the managed build is what the rig exercises.