Joseph Doherty
583 lines
37 KiB
Markdown
583 lines
37 KiB
Markdown
# Delmia Simulated Machine + Notifier Rig Verification — Implementation Plan
|
||
|
||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
|
||
|
||
**Goal:** Stand up an in-memory (static-attribute) stand-in for a vd03-style machine with a composed `DelmiaReceiver` on the local docker rig, wire the `DelmiaRecipeDownload` inbound method and a scoped API key to it, let `DelmiaNotifier` read that key from `appsettings.json`, and prove the notifier end-to-end on the 2-node central cluster including an active-node crash and rejoin.
|
||
|
||
**Architecture:** Two templates seeded by CLI only (no DB edits): `DelmiaReceiverSim` (the ten receiver attributes, all static — no data connection) composed into `SimMachine` as `DelmiaReceiver`. The Galaxy side of the handshake is replaced by a root-scope `ValueChange` script on `SimMachine` that reacts to `DelmiaReceiver.RecipeDownloadFlag`, so the same call script shape as prod runs against pure in-memory tags. The central inbound method is the prod script verbatim (minus its dead `suf` line). The notifier gains an optional `ScadaBridge:ApiKey` config field with the existing `SCADABRIDGE_API_KEY` env var taking precedence, and a bash verification script drives the whole thing against `docker/` (central-a `:9001`, central-b `:9002`, Traefik `:9000`).
|
||
|
||
**Tech Stack:** ScadaBridge CLI (`scadabridge`), bash, .NET 10 (`DelmiaNotifier` + xUnit tests), docker compose rig under `docker/`.
|
||
|
||
**Facts the implementer must not re-derive** (verified 2026-09-02):
|
||
|
||
- `WriteBatchAndWaitAsync` **rejects static attributes** (`InstanceActor.HandleWriteAttributeBatch`: "Attribute 'X' is not a data-sourced attribute"). The rig script therefore uses `SetAsync` per attribute + `WaitAsync`, which both work on static attributes. Do not "fix" this in the runtime — out of scope.
|
||
- A static write always publishes `AttributeValueChanged` (no equality short-circuit), and `ScriptActor` matches a `ValueChange` trigger by **exact** canonical name (`valueTrigger.AttributeName == changed.AttributeName`). Composed scripts' trigger configs are **not** prefixed at flattening (alarms are), so the simulator script lives on the **root** template with `attributeName` = `DelmiaReceiver.RecipeDownloadFlag`.
|
||
- `WaitAsync` has a fast path when the current value already matches, so the simulator finishing before the wait registers is not a race.
|
||
- Trigger types are matched case-insensitively as strings: `Call`, `ValueChange`, `Interval`, `Conditional`, `Expression`. Trigger config key is `attributeName`.
|
||
- CLI global options (`--url`, `--username`, `--password`, `--format=json`) must come **after** the subcommand. Rig creds: `--username multi-role --password password`. Write loops in **bash** (zsh does not word-split unquoted variables).
|
||
- `security api-key create --format=json` returns `{ "keyId": …, "token": "sbk_…" }`; the token is shown once.
|
||
- Rig: site-a is `--site-id 1`; Traefik `http://localhost:9000` → active central; direct nodes `:9001` (central-a) and `:9002` (central-b); both centrals have `StandbyForwardPeerEndpoints` set, so a standby **relays** inbound calls to the active peer. Auto-down failover of a crashed active takes ~25–30 s (`docker/failover-drill.sh`).
|
||
- The notifier's failover is **connect-failure-only**: any HTTP answer (incl. 503) is final. During the takeover window a standby that cannot reach its peer answers 503 `ACTIVE_NODE_UNREACHABLE` → the notifier prints `NO`. That is expected and must be documented, not "fixed".
|
||
- Prod's deployed method + site script are recorded in `docs/api-specs/Delmia-Recipe-Download-API.md`; prod's site script still references `Children["DelmiaRecv"]` (a known, separate defect). The rig script uses `Children["DelmiaReceiver"]`.
|
||
|
||
**Model assignment summary** (see each task's `Model:` line): Sonnet for scripted CLI seeding, config plumbing, and docs; Opus for the live rig verification and the failover drill (judgment under a live system) and for reviewing the secret-handling change; Fable (this session) for the final integration review.
|
||
|
||
---
|
||
|
||
### Task 0: Preflight — rig healthy, CLI built, notifier tests green
|
||
|
||
**Classification:** trivial
|
||
**Estimated implement time:** ~3 min
|
||
**Parallelizable with:** none (everything depends on it)
|
||
**Model:** sonnet
|
||
|
||
**Files:**
|
||
- (none edited)
|
||
|
||
**Step 1: Confirm the rig is up and which node is active**
|
||
|
||
Run:
|
||
```bash
|
||
for p in 9000 9001 9002; do printf "$p ready="; curl -s -m 5 -o /dev/null -w "%{http_code}" http://localhost:$p/health/ready; printf " active="; curl -s -m 5 -o /dev/null -w "%{http_code}\n" http://localhost:$p/health/active; done
|
||
```
|
||
Expected: all three `ready=200`; exactly one of `9001`/`9002` shows `active=200`, the other `503`; `9000 active=200`. If a node is down, run `bash docker/deploy.sh` first and re-check.
|
||
|
||
**Step 2: Build the CLI and the notifier, run the notifier tests**
|
||
|
||
Run:
|
||
```bash
|
||
cd /Users/dohertj2/Desktop/ScadaBridge
|
||
dotnet build src/ZB.MOM.WW.ScadaBridge.CLI -c Debug 2>&1 | tail -3
|
||
dotnet test tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests 2>&1 | tail -5
|
||
```
|
||
Expected: `Build succeeded` and `Passed!` with 0 failed. Note the CLI binary path used by every later task: `src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge`.
|
||
|
||
**Step 3: Confirm the names this plan creates are free on the rig**
|
||
|
||
Run:
|
||
```bash
|
||
CLI=src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge; A="--url http://localhost:9000 --format=json --username multi-role --password password"
|
||
bash -c "$CLI template list $A | jq -r '.[].name' | grep -E '^(DelmiaReceiverSim|SimMachine)$'; $CLI api-method list $A | jq -r '.[].name' | grep -E '^DelmiaRecipeDownload$'; $CLI instance list $A | jq -r '.[].uniqueName' | grep -E '^DelmiaSim01$'; echo done"
|
||
```
|
||
Expected: only `done` printed. If any name is taken, delete it (`instance delete --id`, `template delete --id`, `api-method delete --id`) before continuing — the seed script is not idempotent by design.
|
||
|
||
---
|
||
|
||
### Task 1: `DelmiaNotifier` — optional `ApiKey` in `appsettings.json`, env var wins
|
||
|
||
**Classification:** small (single-project logic change, but it relaxes a documented secret-handling decision — reviewer must weigh in)
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** Task 2, Task 3
|
||
**Model:** sonnet (implement) → **opus** code review, with the explicit question "is the env-over-file precedence and the README wording adequate for a key that now may live on disk?"
|
||
|
||
**Design note (state this in the commit):** the 2026-06-26 design deliberately kept the key env-only. The user has asked for the key in `appsettings.json`. Keep env-only as the *recommended* production posture, add the file as a fallback, and keep the precedence deterministic: `SCADABRIDGE_API_KEY` (if non-blank) → `ScadaBridge:ApiKey` (if non-blank) → not configured. The failure message names both sources.
|
||
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/NotifierConfig.cs`
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/ConfigLoader.cs`
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/Program.cs:42-47`
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/appsettings.json`
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md` (the `## Secret — SCADABRIDGE_API_KEY` section and the config JSON block)
|
||
- Modify: `env_variables.md` (§5 `SCADABRIDGE_API_KEY` row — "env-only by design" is no longer true)
|
||
- Modify: `docs/plans/2026-06-26-delmia-recipe-notifier-design.md` (the `Secret` row of the key-decisions table: add "superseded 2026-09-02: file fallback added, env still wins")
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests/ConfigLoaderTests.cs`
|
||
|
||
**Step 1: Write the failing tests** (append to `ConfigLoaderTests.cs`)
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void Load_reads_ApiKey_from_ScadaBridge_section()
|
||
{
|
||
var cfg = ConfigLoader.Load("{\"ScadaBridge\":{\"BaseUrls\":\"http://x\",\"ApiKey\":\"sbk_file\"}}");
|
||
Assert.Equal("sbk_file", cfg.ScadaBridge.ApiKey);
|
||
}
|
||
|
||
[Fact]
|
||
public void ResolveApiKey_prefers_env_over_config()
|
||
{
|
||
var cfg = ConfigLoader.Load("{\"ScadaBridge\":{\"ApiKey\":\"sbk_file\"}}");
|
||
Assert.Equal("sbk_env", ConfigLoader.ResolveApiKey(n => n == "SCADABRIDGE_API_KEY" ? "sbk_env" : null, cfg));
|
||
}
|
||
|
||
[Fact]
|
||
public void ResolveApiKey_falls_back_to_config_when_env_unset_or_blank()
|
||
{
|
||
var cfg = ConfigLoader.Load("{\"ScadaBridge\":{\"ApiKey\":\" sbk_file \"}}");
|
||
Assert.Equal("sbk_file", ConfigLoader.ResolveApiKey(_ => null, cfg));
|
||
Assert.Equal("sbk_file", ConfigLoader.ResolveApiKey(_ => " ", cfg));
|
||
}
|
||
|
||
[Fact]
|
||
public void ResolveApiKey_returns_null_when_neither_source_set()
|
||
{
|
||
var cfg = ConfigLoader.Load("{\"ScadaBridge\":{\"ApiKey\":\" \"}}");
|
||
Assert.Null(ConfigLoader.ResolveApiKey(_ => null, cfg));
|
||
}
|
||
```
|
||
|
||
Also update the two existing `ResolveApiKey_*` tests to pass a config: `ConfigLoader.ResolveApiKey(_ => null, new NotifierConfig())` etc. (the one-argument overload is removed, not kept — YAGNI).
|
||
|
||
**Step 2: Run tests to verify they fail**
|
||
|
||
Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests 2>&1 | tail -5`
|
||
Expected: build error `'ScadaBridgeSection' does not contain a definition for 'ApiKey'` / no overload for `ResolveApiKey` taking 2 arguments.
|
||
|
||
**Step 3: Implement**
|
||
|
||
`NotifierConfig.cs` — add to `ScadaBridgeSection`:
|
||
```csharp
|
||
/// <summary>Optional inbound API key. <c>SCADABRIDGE_API_KEY</c> in the environment takes precedence; this is the file fallback (design 2026-06-26, relaxed 2026-09-02).</summary>
|
||
public string? ApiKey { get; set; }
|
||
```
|
||
|
||
`ConfigLoader.cs` — replace `ResolveApiKey`:
|
||
```csharp
|
||
/// <summary>Resolve the API key: <c>SCADABRIDGE_API_KEY</c> wins when non-blank, otherwise <c>ScadaBridge:ApiKey</c> from <paramref name="cfg"/> (trimmed); null when neither is set. Env accessor is injected for testability.</summary>
|
||
/// <param name="envGet">Accessor used to read an environment variable by name.</param>
|
||
/// <param name="cfg">The loaded configuration, consulted only when the environment variable is unset/blank.</param>
|
||
/// <returns>The resolved API key, or <c>null</c> if neither source yields a non-blank value.</returns>
|
||
public static string? ResolveApiKey(Func<string, string?> envGet, NotifierConfig cfg)
|
||
{
|
||
var env = envGet(ApiKeyEnvVar);
|
||
if (!string.IsNullOrWhiteSpace(env)) return env;
|
||
var file = cfg.ScadaBridge.ApiKey;
|
||
return string.IsNullOrWhiteSpace(file) ? null : file.Trim();
|
||
}
|
||
```
|
||
|
||
`Program.cs` lines 42–47 become:
|
||
```csharp
|
||
var apiKey = ConfigLoader.ResolveApiKey(Environment.GetEnvironmentVariable, cfg);
|
||
if (apiKey is null)
|
||
{
|
||
log.Write("API key not configured (SCADABRIDGE_API_KEY or ScadaBridge:ApiKey)");
|
||
return Reporter.Report(false, "API key not configured (SCADABRIDGE_API_KEY or ScadaBridge:ApiKey)", stdout);
|
||
}
|
||
```
|
||
|
||
`appsettings.json` — add `"ApiKey": ""` after `LogPath` (empty = not set; **never commit a real token**).
|
||
|
||
README: in the config JSON block add `"ApiKey": ""`; rewrite the `## Secret` section:
|
||
|
||
> The inbound API key is sent as `X-API-Key`. It is resolved from **`SCADABRIDGE_API_KEY`** first (recommended for production — the key never touches disk), falling back to **`ScadaBridge:ApiKey`** in `appsettings.json` when the variable is unset or blank. If neither is set → `NO` + `API key not configured (SCADABRIDGE_API_KEY or ScadaBridge:ApiKey)`, exit `-1`, no HTTP attempt. If you use the file, restrict its ACL to the account DELMIA runs under.
|
||
|
||
`env_variables.md` §5: change the "In appsettings?" cell to `Yes — optional \`ScadaBridge:ApiKey\` fallback (2026-09-02); env wins` and the Purpose cell's "Read only from the environment, never a file" to "Environment first, file fallback".
|
||
|
||
Design doc table row `| Secret | API key from env var … — never in a file |` → append ` **(relaxed 2026-09-02: optional \`ScadaBridge:ApiKey\` file fallback; env still wins)**`.
|
||
|
||
**Step 4: Run tests to verify they pass**
|
||
|
||
Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests 2>&1 | tail -5`
|
||
Expected: `Passed!`, 0 failed, total = previous count + 4. Also `dotnet build src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier -warnaserror` succeeds (the project treats warnings as errors; the XML doc comments above are required).
|
||
|
||
**Step 5: Commit**
|
||
|
||
```bash
|
||
git add src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests env_variables.md docs/plans/2026-06-26-delmia-recipe-notifier-design.md
|
||
git commit -m "feat(delmia-notifier): optional ScadaBridge:ApiKey in appsettings.json as a fallback to SCADABRIDGE_API_KEY (env still wins); README/env inventory/design row updated"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 2: `docker/seed-delmia-sim.sh` — templates, composition, scripts, instance, deploy
|
||
|
||
**Classification:** standard (new script + two template scripts that must compile under the trust policy)
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** Task 1, Task 3
|
||
**Model:** sonnet
|
||
|
||
**Files:**
|
||
- Create: `docker/seed-delmia-sim.sh`
|
||
- Modify: `docker/README.md:96-97` (add the new script to the tree listing, one line)
|
||
|
||
**Step 1: Write the script**
|
||
|
||
```bash
|
||
#!/bin/bash
|
||
set -euo pipefail
|
||
|
||
# Seed an IN-MEMORY stand-in for a vd03-style machine with a composed DelmiaReceiver on
|
||
# site-a, so DelmiaNotifier can be exercised end-to-end on the docker rig without a Galaxy.
|
||
#
|
||
# What it creates (all static attributes — no data connection, no bindings):
|
||
# - Template DelmiaReceiverSim — the ten DelmiaReceiver attributes from prod
|
||
# (docs/api-specs/Delmia-Recipe-Download-API.md §5), ReadyFlag defaulting to true.
|
||
# - Template SimMachine — composes DelmiaReceiverSim as `DelmiaReceiver` and carries
|
||
# * ProcessRecipeDownload (Call) — prod's handshake ported to per-attribute
|
||
# SetAsync/WaitAsync (WriteBatchAndWaitAsync rejects static attributes).
|
||
# * DelmiaReceiverSimulator (ValueChange on DelmiaReceiver.RecipeDownloadFlag) —
|
||
# plays the Galaxy object: accepts unless DownloadPath ends in ".bad" or the work
|
||
# order is blank, writes RecipeProcessResult/Text, clears the trigger flag, then
|
||
# sets RecipeProcessedFlag=true.
|
||
# - Instance DelmiaSim01 on site-a (id 1), deployed.
|
||
#
|
||
# Not idempotent: names must be unique. To re-seed:
|
||
# scadabridge instance delete --id <id>; scadabridge template delete --id <SimMachine id>;
|
||
# scadabridge template delete --id <DelmiaReceiverSim id>
|
||
# Prerequisites: bash docker/deploy.sh; docker/seed-sites.sh; central healthy on :9000.
|
||
|
||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||
CLI="$PROJECT_ROOT/src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge"
|
||
A=(--url "${SCADABRIDGE_URL:-http://localhost:9000}" --format=json --username "${SCADABRIDGE_USERNAME:-multi-role}" --password "${SCADABRIDGE_PASSWORD:-password}")
|
||
SITE_ID="${SITE_ID:-1}"
|
||
INSTANCE_NAME="${INSTANCE_NAME:-DelmiaSim01}"
|
||
|
||
json_id() { python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])'; }
|
||
|
||
echo "1/6 Creating template DelmiaReceiverSim..."
|
||
RECV_ID=$("$CLI" template create "${A[@]}" --name DelmiaReceiverSim \
|
||
--description "In-memory stand-in for the Galaxy DelmiaReceiver object (ten static attributes)" | json_id)
|
||
echo " id=$RECV_ID"
|
||
|
||
echo "2/6 Adding the ten receiver attributes (static)..."
|
||
"$CLI" template attribute add "${A[@]}" --template-id "$RECV_ID" --name ReadyFlag --data-type Boolean --value true --description "Precondition; true = receiver accepts downloads" > /dev/null
|
||
"$CLI" template attribute add "${A[@]}" --template-id "$RECV_ID" --name RecipeDownloadFlag --data-type Boolean --value false --description "Trigger: set true after the payload is written" > /dev/null
|
||
"$CLI" template attribute add "${A[@]}" --template-id "$RECV_ID" --name RecipeProcessedFlag --data-type Boolean --value false --description "Cleared by the caller, set true by the receiver when done" > /dev/null
|
||
"$CLI" template attribute add "${A[@]}" --template-id "$RECV_ID" --name RecipeProcessResult --data-type Boolean --value false --description "Outcome -> Result" > /dev/null
|
||
for s in DownloadPath WorkOrderNumber PartNumber JobStepNumber Username RecipeProcessResultText; do
|
||
"$CLI" template attribute add "${A[@]}" --template-id "$RECV_ID" --name "$s" --data-type String --description "Recipe-download payload / outcome text" > /dev/null
|
||
done
|
||
|
||
echo "3/6 Creating template SimMachine + composing DelmiaReceiver..."
|
||
MACHINE_ID=$("$CLI" template create "${A[@]}" --name SimMachine \
|
||
--description "vd03-style machine stand-in: composes DelmiaReceiverSim as DelmiaReceiver; in-memory only" | json_id)
|
||
"$CLI" template composition add "${A[@]}" --template-id "$MACHINE_ID" --composed-template-id "$RECV_ID" --instance-name DelmiaReceiver > /dev/null
|
||
echo " id=$MACHINE_ID"
|
||
|
||
echo "4/6 Adding ProcessRecipeDownload (Call) — prod handshake on static attributes..."
|
||
read -r -d '' PROCESS_CODE <<'CS' || true
|
||
// Port of prod CvdReactor.ProcessRecipeDownload for STATIC attributes: WriteBatchAndWaitAsync only
|
||
// accepts data-sourced attributes, so the payload is written with SetAsync and the handshake waited
|
||
// with WaitAsync (same trigger/response flags, same 25 s budget, same result mapping).
|
||
var recv = Children["DelmiaReceiver"];
|
||
System.Func<object?,bool> IsTrue = v => { var s = v?.ToString(); return (v is bool b && b) || s == "1" || string.Equals(s, "true", System.StringComparison.OrdinalIgnoreCase); };
|
||
System.Func<string,object?> P = k => Parameters.ContainsKey(k) ? Parameters[k] : null;
|
||
if (!IsTrue(await recv.Attributes.GetAsync("ReadyFlag")))
|
||
return new { Result = false, ResultText = "Receiver ready flag not set to true" };
|
||
try {
|
||
await recv.Attributes.SetAsync("RecipeProcessedFlag", false);
|
||
await recv.Attributes.SetAsync("DownloadPath", P("DownloadPath"));
|
||
await recv.Attributes.SetAsync("WorkOrderNumber", P("WorkOrderNumber"));
|
||
await recv.Attributes.SetAsync("PartNumber", P("PartNumber"));
|
||
await recv.Attributes.SetAsync("JobStepNumber", P("JobStepNumber"));
|
||
await recv.Attributes.SetAsync("Username", P("Username"));
|
||
await recv.Attributes.SetAsync("RecipeDownloadFlag", true);
|
||
if (!await recv.Attributes.WaitAsync("RecipeProcessedFlag", true, System.TimeSpan.FromSeconds(25)))
|
||
return new { Result = false, ResultText = "Timeout waiting for recipe to be processed" };
|
||
return new {
|
||
Result = IsTrue(await recv.Attributes.GetAsync("RecipeProcessResult")),
|
||
ResultText = (await recv.Attributes.GetAsync("RecipeProcessResultText"))?.ToString() ?? ""
|
||
};
|
||
} catch (Exception ex) {
|
||
return new { Result = false, ResultText = "RecipeDownload failed: " + ex.Message };
|
||
}
|
||
CS
|
||
"$CLI" template script add "${A[@]}" --template-id "$MACHINE_ID" --name ProcessRecipeDownload --trigger-type Call \
|
||
--parameters '{"type":"object","properties":{"DownloadPath":{"type":"string"},"WorkOrderNumber":{"type":"string"},"PartNumber":{"type":"string"},"JobStepNumber":{"type":"string"},"Username":{"type":"string"}},"required":[]}' \
|
||
--return-def '{"type":"object","properties":{"Result":{"type":"boolean"},"ResultText":{"type":"string"}},"required":["Result","ResultText"]}' \
|
||
--code "$PROCESS_CODE" > /dev/null
|
||
|
||
echo "5/6 Adding DelmiaReceiverSimulator (ValueChange on DelmiaReceiver.RecipeDownloadFlag)..."
|
||
read -r -d '' SIM_CODE <<'CS' || true
|
||
// Plays the Galaxy DelmiaReceiver object. Fires on every write to RecipeDownloadFlag (root-scope
|
||
// trigger, canonical name) and acts only on the rising edge; its own reset to false re-triggers
|
||
// this script, which then returns at the guard.
|
||
var recv = Children["DelmiaReceiver"];
|
||
var flag = await recv.Attributes.GetAsync("RecipeDownloadFlag");
|
||
var fs = flag?.ToString();
|
||
if (!((flag is bool b && b) || fs == "1" || string.Equals(fs, "true", System.StringComparison.OrdinalIgnoreCase))) return null;
|
||
var path = (await recv.Attributes.GetAsync("DownloadPath"))?.ToString() ?? "";
|
||
var wo = (await recv.Attributes.GetAsync("WorkOrderNumber"))?.ToString() ?? "";
|
||
var ok = path.Length > 0 && wo.Length > 0 && !path.EndsWith(".bad", System.StringComparison.OrdinalIgnoreCase);
|
||
await recv.Attributes.SetAsync("RecipeProcessResult", ok);
|
||
await recv.Attributes.SetAsync("RecipeProcessResultText", ok ? "Simulated download accepted for " + wo : "Simulated reject: bad path or missing work order");
|
||
await recv.Attributes.SetAsync("RecipeDownloadFlag", false);
|
||
await recv.Attributes.SetAsync("RecipeProcessedFlag", true);
|
||
return null;
|
||
CS
|
||
"$CLI" template script add "${A[@]}" --template-id "$MACHINE_ID" --name DelmiaReceiverSimulator --trigger-type ValueChange \
|
||
--trigger-config '{"attributeName":"DelmiaReceiver.RecipeDownloadFlag"}' --code "$SIM_CODE" > /dev/null
|
||
|
||
echo "6/6 Creating + deploying instance $INSTANCE_NAME on site $SITE_ID..."
|
||
INSTANCE_ID=$("$CLI" instance create "${A[@]}" --name "$INSTANCE_NAME" --template-id "$MACHINE_ID" --site-id "$SITE_ID" | json_id)
|
||
"$CLI" instance deploy "${A[@]}" --id "$INSTANCE_ID"
|
||
echo ""
|
||
echo "DELMIA_SIM_TEMPLATE_ID=$RECV_ID SIM_MACHINE_TEMPLATE_ID=$MACHINE_ID INSTANCE_ID=$INSTANCE_ID INSTANCE_NAME=$INSTANCE_NAME"
|
||
echo "Verify: $CLI debug snapshot --id $INSTANCE_ID --url http://localhost:9000 --username multi-role --password password"
|
||
```
|
||
|
||
**Step 2: Make it executable and run it against the rig**
|
||
|
||
Run: `chmod +x docker/seed-delmia-sim.sh && bash docker/seed-delmia-sim.sh 2>&1 | tail -15`
|
||
Expected: six numbered lines, a deploy reply without `"error"`, and the final `DELMIA_SIM_TEMPLATE_ID=… INSTANCE_ID=…` line. Record `INSTANCE_ID` for Task 4.
|
||
|
||
If `template script add` fails with a trust-policy or compile error, the message names the offending symbol — fix the script text, do **not** widen the trust policy. If `instance deploy` reports a pre-deployment validation failure mentioning bindings, a data-source was set by mistake: static attributes must be created **without** `--data-source`.
|
||
|
||
**Step 3: Prove the handshake works site-side before any HTTP is involved**
|
||
|
||
Run (bash):
|
||
```bash
|
||
CLI=src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge
|
||
$CLI debug snapshot --id <INSTANCE_ID> --url http://localhost:9000 --format=json --username multi-role --password password | jq '.attributes // . | with_entries(select(.key|startswith("DelmiaReceiver.")))'
|
||
```
|
||
Expected: ten `DelmiaReceiver.*` entries, `ReadyFlag` true, `RecipeProcessedFlag` false. (The snapshot JSON shape is the CLI's — adapt the `jq` path to what comes back; the point is that all ten attributes exist under the `DelmiaReceiver.` prefix.)
|
||
|
||
**Step 4: Add the script to the docker tree listing**
|
||
|
||
`docker/README.md` line 97: after the `seed-live-probe.sh` row add
|
||
`├── seed-delmia-sim.sh # Seed the in-memory Delmia machine stand-in (DelmiaSim01) on site-a`
|
||
|
||
**Step 5: Commit**
|
||
|
||
```bash
|
||
git add docker/seed-delmia-sim.sh docker/README.md
|
||
git commit -m "chore(docker): seed-delmia-sim.sh — in-memory SimMachine + composed DelmiaReceiver stand-in on site-a with a ValueChange simulator playing the Galaxy side"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 3: `docker/seed-delmia-api.sh` — inbound method + scoped API key
|
||
|
||
**Classification:** small
|
||
**Estimated implement time:** ~4 min
|
||
**Parallelizable with:** Task 1, Task 2
|
||
**Model:** sonnet
|
||
|
||
**Files:**
|
||
- Create: `docker/seed-delmia-api.sh`
|
||
- Modify: `docker/README.md` (one tree-listing line under the Task 2 line)
|
||
|
||
**Step 1: Write the script**
|
||
|
||
```bash
|
||
#!/bin/bash
|
||
set -euo pipefail
|
||
|
||
# Seed the DelmiaRecipeDownload inbound method (prod's script verbatim, minus its dead `suf`
|
||
# line — see docs/api-specs/Delmia-Recipe-Download-API.md §4) and one API key scoped to it.
|
||
# Prints the one-time token as DELMIA_API_KEY=... — capture it; it cannot be retrieved again.
|
||
# Not idempotent (method name must be unique): delete with `api-method delete --id`.
|
||
|
||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||
CLI="$PROJECT_ROOT/src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge"
|
||
A=(--url "${SCADABRIDGE_URL:-http://localhost:9000}" --format=json --username "${SCADABRIDGE_USERNAME:-multi-role}" --password "${SCADABRIDGE_PASSWORD:-password}")
|
||
KEY_NAME="${KEY_NAME:-delmia-notifier}"
|
||
|
||
read -r -d '' METHOD_CODE <<'CS' || true
|
||
try {
|
||
System.Func<string, object> P = k => Parameters.ContainsKey(k) ? Parameters[k] : null;
|
||
var code = (P("MachineCode") as string ?? "").Trim();
|
||
if (string.IsNullOrEmpty(code)) return new { Result = false, ResultText = "MachineCode is required" };
|
||
|
||
return await Route.To(code).Call("ProcessRecipeDownload", new {
|
||
DownloadPath = P("DownloadPath"),
|
||
WorkOrderNumber = P("WorkOrderNumber"),
|
||
PartNumber = P("PartNumber"),
|
||
JobStepNumber = P("JobStepNumber"),
|
||
Username = P("Username"),
|
||
});
|
||
} catch (Exception ex) {
|
||
return new { Result = false, ResultText = "RecipeDownload failed: " + ex.Message };
|
||
}
|
||
CS
|
||
|
||
echo "1/2 Creating inbound method DelmiaRecipeDownload (timeout 45 s)..."
|
||
METHOD_JSON=$("$CLI" api-method create "${A[@]}" --name DelmiaRecipeDownload --timeout 45 \
|
||
--parameters '{"type":"object","properties":{"MachineCode":{"type":"string"},"DownloadPath":{"type":"string"},"WorkOrderNumber":{"type":"string"},"PartNumber":{"type":"string"},"JobStepNumber":{"type":"string"},"Username":{"type":"string"}},"required":[]}' \
|
||
--return-def '{"type":"object","properties":{"Result":{"type":"boolean"},"ResultText":{"type":"string"}},"required":["Result","ResultText"]}' \
|
||
--script "$METHOD_CODE")
|
||
echo " $(echo "$METHOD_JSON" | python3 -c 'import json,sys; d=json.load(sys.stdin); print("id=%s name=%s" % (d["id"], d["name"]))')"
|
||
|
||
echo "2/2 Creating API key '$KEY_NAME' scoped to DelmiaRecipeDownload..."
|
||
KEY_JSON=$("$CLI" security api-key create "${A[@]}" --name "$KEY_NAME" --methods DelmiaRecipeDownload 2>/dev/null)
|
||
TOKEN=$(echo "$KEY_JSON" | python3 -c 'import json,sys; print(json.load(sys.stdin)["token"])')
|
||
echo "DELMIA_API_KEY=$TOKEN"
|
||
```
|
||
|
||
> Note: `security api-key create` prints the token via `PrintCreatedKey` in table mode; with `--format=json` the raw `{keyId, token}` document is emitted and parsed above. If the json branch still prints the human lines, grep the `sbk_` token instead: `TOKEN=$(echo "$KEY_JSON" | grep -o 'sbk_[A-Za-z0-9_-]*' | head -1)`.
|
||
|
||
**Step 2: Run it and keep the token OUT of git**
|
||
|
||
Run: `chmod +x docker/seed-delmia-api.sh && bash docker/seed-delmia-api.sh`
|
||
Expected: `id=<n> name=DelmiaRecipeDownload` then `DELMIA_API_KEY=sbk_…`. Save the token to the scratchpad only (e.g. `$SCRATCH/delmia.env` with `export DELMIA_API_KEY=…`). Never write it into `src/**/appsettings.json`.
|
||
|
||
**Step 3: Smoke the method with curl (no notifier yet)**
|
||
|
||
```bash
|
||
curl -s -X POST http://localhost:9000/api/DelmiaRecipeDownload -H "X-API-Key: $DELMIA_API_KEY" -H "Content-Type: application/json" \
|
||
-d '{"MachineCode":"DelmiaSim01","DownloadPath":"C:\\recipes\\wo1.nc","WorkOrderNumber":"WO-1","PartNumber":"PN-1","JobStepNumber":"0100","Username":"curl"}'
|
||
```
|
||
Expected: `{"Result":true,"ResultText":"Simulated download accepted for WO-1"}`. Then the reject path (`"DownloadPath":"C:\\recipes\\x.bad"`) → `Result:false`, `Simulated reject…`; and a bogus key → HTTP 401 `{"error":"Invalid or missing API key"}`. If the first call returns `Timeout waiting for recipe to be processed`, the simulator did not fire: check `scadabridge debug snapshot` for `DelmiaReceiver.RecipeDownloadFlag` (should have been reset to false) and the site-a-a container log for a compile error on `DelmiaReceiverSimulator`.
|
||
|
||
**Step 4: Commit**
|
||
|
||
```bash
|
||
git add docker/seed-delmia-api.sh docker/README.md
|
||
git commit -m "chore(docker): seed-delmia-api.sh — DelmiaRecipeDownload inbound method (prod script) + scoped delmia-notifier API key"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 4: `docker/verify-delmia-notifier.sh` — notifier end-to-end + central failover drill
|
||
|
||
**Classification:** high-risk (drives `docker stop` on a live rig node; interprets failover timing)
|
||
**Estimated implement time:** ~5 min to write, ~6 min to run (two drills × ~1 min settle)
|
||
**Parallelizable with:** none (needs Tasks 1–3)
|
||
**Model:** **opus** — run interactively, read the outputs, and decide whether a `NO` is an expected takeover-window answer or a real fault before proceeding.
|
||
|
||
**Files:**
|
||
- Create: `docker/verify-delmia-notifier.sh`
|
||
- Modify: `docker/README.md` (one tree-listing line)
|
||
|
||
**Step 1: Write the script**
|
||
|
||
```bash
|
||
#!/bin/bash
|
||
set -euo pipefail
|
||
|
||
# End-to-end DelmiaNotifier verification against the docker rig, including an ACTIVE central
|
||
# crash + rejoin. Requires: seed-delmia-sim.sh + seed-delmia-api.sh already run, and
|
||
# DELMIA_API_KEY exported (the token printed by seed-delmia-api.sh).
|
||
#
|
||
# The notifier is run as the managed build (dotnet WWNotifier.dll) from a scratch copy so the
|
||
# API key is written to THAT copy's appsettings.json (ScadaBridge:ApiKey — Task 1) and never
|
||
# into the repo. BaseUrls lists the two central nodes DIRECTLY (not Traefik) so the notifier's
|
||
# own connect-failure failover is what gets exercised.
|
||
#
|
||
# Expected outcomes:
|
||
# A baseline (active listed first) -> YES
|
||
# B standby listed first (standby relays) -> YES (StandbyForwardPeerEndpoints)
|
||
# C bad key -> NO (HTTP 401, no failover: 401 is an answer)
|
||
# D active STOPPED, before takeover completes -> NO or YES — informational only. A NO here is the
|
||
# documented connect-failure-only design (standby answers 503 ACTIVE_NODE_UNREACHABLE).
|
||
# E active STOPPED, after survivor /health/active=200 -> YES (first URL refused -> failover)
|
||
# F victim restarted and rejoined (members=2) -> YES
|
||
# G reject path (DownloadPath *.bad) -> NO + "Simulated reject"
|
||
|
||
: "${DELMIA_API_KEY:?export DELMIA_API_KEY=sbk_... first}"
|
||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"; PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||
RUN_DIR="${RUN_DIR:-${TMPDIR:-/tmp}/delmia-notifier-run}"
|
||
MACHINE="${MACHINE:-DelmiaSim01}"
|
||
A_URL=http://localhost:9001; B_URL=http://localhost:9002
|
||
|
||
echo "== build notifier + stage a runnable copy in $RUN_DIR"
|
||
dotnet build "$PROJECT_ROOT/src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier" -c Debug -nologo -v q
|
||
rm -rf "$RUN_DIR"; mkdir -p "$RUN_DIR"; cp -R "$PROJECT_ROOT/src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/bin/Debug/net10.0/." "$RUN_DIR/"
|
||
|
||
write_cfg() { # $1 = BaseUrls, $2 = ApiKey
|
||
printf '{"ScadaBridge":{"BaseUrls":"%s","TimeoutSeconds":30,"LogPath":"delmia-notifier.log","ApiKey":"%s"}}\n' "$1" "$2" > "$RUN_DIR/appsettings.json"
|
||
}
|
||
notify() { # $1 = label, $2 = expected YES|NO, rest = extra args; prints stdout + exit code
|
||
local label=$1 expect=$2; shift 2
|
||
local out rc; set +e; out=$(env -u SCADABRIDGE_API_KEY dotnet "$RUN_DIR/WWNotifier.dll" -m "$MACHINE" -d 'C:\recipes\wo1.nc' -w "WO-$label" -p PN-1 -s 0100 -u drill "$@" 2>/dev/null); rc=$?; set -e
|
||
local first; first=$(echo "$out" | head -1)
|
||
printf '%-3s stdout=%-4s exit=%-3s | %s\n' "$label" "$first" "$rc" "$(echo "$out" | sed -n 2p)"
|
||
[ "$first" = "$expect" ] || { echo " ^^ EXPECTED $expect"; return 1; }
|
||
}
|
||
active() { curl -sf -o /dev/null "$A_URL/health/active" && echo a || { curl -sf -o /dev/null "$B_URL/health/active" && echo b; }; }
|
||
url_of() { [ "$1" = a ] && echo "$A_URL" || echo "$B_URL"; }
|
||
container_of() { [ "$1" = a ] && echo scadabridge-central-a || echo scadabridge-central-b; }
|
||
other() { [ "$1" = a ] && echo b || echo a; }
|
||
wait_for() { # $1 url $2 path $3 seconds
|
||
local i; for i in $(seq 1 "$3"); do curl -sf -o /dev/null "$1$2" && return 0; sleep 1; done; return 1; }
|
||
|
||
ACT=$(active); STB=$(other "$ACT")
|
||
echo "== active=central-$ACT standby=central-$STB"
|
||
|
||
echo "== A baseline (active first)"; write_cfg "$(url_of "$ACT"),$(url_of "$STB")" "$DELMIA_API_KEY"; notify A YES
|
||
echo "== B standby first (relay)"; write_cfg "$(url_of "$STB"),$(url_of "$ACT")" "$DELMIA_API_KEY"; notify B YES
|
||
echo "== C bad key"; write_cfg "$(url_of "$ACT"),$(url_of "$STB")" "sbk_bogus_bogus"; notify C NO || true
|
||
write_cfg "$(url_of "$ACT"),$(url_of "$STB")" "$DELMIA_API_KEY"
|
||
|
||
echo "== D/E stop the ACTIVE central ($(container_of "$ACT"))"; docker stop "$(container_of "$ACT")" > /dev/null
|
||
T0=$(date +%s)
|
||
echo "-- D immediate attempt (informational):"; notify D NO || notify D YES || true
|
||
echo "-- waiting for survivor $(url_of "$STB")/health/active (auto-down ~25-30 s)..."
|
||
wait_for "$(url_of "$STB")" /health/active 90 || { echo "FAIL: survivor never became active"; exit 1; }
|
||
echo " takeover after $(( $(date +%s) - T0 )) s"
|
||
echo "== E after takeover (first URL refused -> failover)"; notify E YES
|
||
|
||
echo "== F restart victim + wait for rejoin"; docker start "$(container_of "$ACT")" > /dev/null
|
||
wait_for "$(url_of "$ACT")" /health/ready 120 || { echo "FAIL: victim never ready"; exit 1; }
|
||
for i in $(seq 1 60); do curl -s "$(url_of "$STB")/health/ready" | grep -q '"members":2\|members=2\|Up.*Up' && break; sleep 2; done
|
||
notify F YES
|
||
|
||
echo "== G reject path"; notify G NO -d 'C:\recipes\x.bad' || true
|
||
echo "== done. Notifier log: $RUN_DIR/delmia-notifier.log"
|
||
```
|
||
|
||
> The `members` probe in F is deliberately loose — check what `/health/ready` actually prints on this build (`curl -s localhost:9001/health/ready`) and tighten the grep to the real field before relying on it.
|
||
|
||
**Step 2: Run it**
|
||
|
||
```bash
|
||
chmod +x docker/verify-delmia-notifier.sh
|
||
export DELMIA_API_KEY=sbk_… # from Task 3
|
||
bash docker/verify-delmia-notifier.sh 2>&1 | tee "$SCRATCH/verify-delmia.log"
|
||
```
|
||
Expected: A, B, E, F print `stdout=YES exit=0`; C and G print `stdout=NO`; D is informational (record which it was and the takeover seconds). The victim rejoins as **standby** (`/health/active` 503 on it, 200 on the survivor). Any other outcome is a real finding — capture the notifier log line (`attempt <url>: …`) and the central log (`docker logs --since 5m scadabridge-central-<x> | grep -i delmia`) before changing anything.
|
||
|
||
**Step 3: Confirm the audit trail**
|
||
|
||
```bash
|
||
CLI=src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge
|
||
$CLI audit query --channel ApiInbound --since 15m --url http://localhost:9000 --format=json --username multi-role --password password | grep DelmiaRecipeDownload | jq -c '{t:.occurredAtUtc, kind, status, http:.httpStatus, node:.sourceNode, actor, resp:.responseSummary}'
|
||
```
|
||
Expected: one `InboundRequest` row per YES/NO business answer, **exactly one row for B** (the standby relay must not double-audit — `InboundApiEndpointFilter.RelayedToActiveItemKey`), an `InboundAuthFailure` for C, and `sourceNode` showing the node that executed each call (B and E should show the node that was active at the time).
|
||
|
||
**Step 4: Commit the script**
|
||
|
||
```bash
|
||
git add docker/verify-delmia-notifier.sh docker/README.md
|
||
git commit -m "test(docker): verify-delmia-notifier.sh — notifier end-to-end on the rig incl. standby relay, active-central crash/takeover and rejoin"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 5: Record results + spec/doc updates
|
||
|
||
**Classification:** trivial (docs only)
|
||
**Estimated implement time:** ~4 min
|
||
**Parallelizable with:** none (needs Task 4's numbers)
|
||
**Model:** sonnet
|
||
|
||
**Files:**
|
||
- Modify: `docs/api-specs/Delmia-Recipe-Download-API.md` (new §8 "Docker rig stand-in", pointing at the two seed scripts, the static-attribute port, and the verified outcomes A–G with the measured takeover time)
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md` (`## Manual smoke test` → add the rig recipe: run the three scripts, `export DELMIA_API_KEY`, expected YES/NO table)
|
||
- Modify: `docs/plans/2026-07-08-deferred-work-register.md` (one new row: "prod `ProcessRecipeDownload` still addresses `Children[\"DelmiaRecv\"]` after the 2026-06-27 rename — fix + redeploy Z28061/Z28062/Z28061Sim; QA unverified" — only if a row for it does not already exist)
|
||
- Modify: `CLAUDE.md` — in the `### Templates & Deployment` / DelmiaNotifier context, one sentence: "Rig stand-in: `docker/seed-delmia-sim.sh` + `seed-delmia-api.sh` + `verify-delmia-notifier.sh` (static-attribute port of the prod handshake — `WriteBatchAndWaitAsync` rejects static attributes, use `SetAsync`/`WaitAsync`)."
|
||
|
||
**Step 1: Write the sections** using the numbers from Task 4's log (takeover seconds, D's outcome, audit row count).
|
||
|
||
**Step 2: Check no stale claims remain**
|
||
|
||
Run: `grep -rn "never read from or written to a file\|env-only by design" src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier docs env_variables.md CLAUDE.md`
|
||
Expected: no hits (Task 1 removed the README/env ones; anything left here is stale and must be updated).
|
||
|
||
**Step 3: Commit**
|
||
|
||
```bash
|
||
git add docs/api-specs/Delmia-Recipe-Download-API.md src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md docs/plans/2026-07-08-deferred-work-register.md CLAUDE.md
|
||
git commit -m "docs: Delmia rig stand-in + notifier verification results (A-G incl. active-central takeover); register row for prod DelmiaRecv path defect"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 6: Final integration review
|
||
|
||
**Classification:** standard
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** none
|
||
**Model:** **fable** (this session) — read `git log --oneline main..HEAD` + `git diff main`, re-run `dotnet test tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests`, and check specifically: (1) no `sbk_` token anywhere in the diff (`git diff main | grep -n 'sbk_[A-Za-z0-9]'` must return only the placeholder in the bad-key test), (2) the env-over-file precedence is asserted by a test, (3) the seed scripts leave the rig in a state `docker/README.md` describes, (4) Task 4's log shows E and F as YES with the victim rejoined as standby. Then decide with the user whether to push (`/pushit`) — the plan does not push.
|
||
|
||
---
|
||
|
||
## Out of scope (deliberately)
|
||
|
||
- Fixing prod vd03's `Children["DelmiaRecv"]` defect or making the prod method idempotent — recorded in the deferred-work register (Task 5); separate change with its own deployment record.
|
||
- Extending `WriteBatchAndWaitAsync` to static attributes.
|
||
- Site-pair failover (the user asked for the 2-node **central** cluster); `docker/failover-drill.sh` already covers the general case.
|
||
- Native AOT publish of the notifier (Windows-only); the managed build is what the rig exercises.
|