docs: Delmia rig stand-in + notifier verification results (A-G incl. active-central kill/takeover 27 s, rejoin); register rows 107 (prod DelmiaRecv path) + 108 (per-node inbound API key store breaks central failover); CLAUDE.md bullet; plan + task file
Claude-Session: https://claude.ai/code/session_013L3XdmVii9i5sY4FChU8pB
This commit is contained in:
Joseph Doherty
@@ -99,6 +99,8 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
|
||||
| 104 | ~~**Existing site LocalDb files stay `auto_vacuum=NONE`; `PRAGMA incremental_vacuum` after a cap trim is a no-op on them**~~ **RESOLVED 2026-08-20 — recipe shipped and applied to the rig.** New `docker/vacuum-site-localdb.sh` (stop both nodes of a pair together → `PRAGMA auto_vacuum=INCREMENTAL; VACUUM` each volume with integrity checks before/after → start both; `DRY_RUN=1` reports) plus the topology-guide section *Site LocalDb files created before 2026-08-20 need a one-off offline VACUUM*. Run on all three rig pairs: all six files `auto_vacuum 0 → 2`, `integrity_check=ok`, ~11 % smaller (site-a 20.3 → 18.1 MB), site-a replication back `Connected=true`/backlog 0 with no watermark or schema change — the VACUUM was replication-invisible as predicted (ledger keys by declared PK; oplog `seq` is an `INTEGER PRIMARY KEY AUTOINCREMENT`). The same maintenance surfaced the unrelated row 106. Original: | Arch-review 2026-08-19 P4-6 (Low-wave T4, `47f03fe6`) | `auto_vacuum` is only honoured on a table-less database, and `LocalDbSchema.EnsureCreated` had always created LocalDb's bookkeeping tables before `SiteEventLogSchema.Apply` ran — so the pragma had never once taken effect on any real file. The fix (`SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile`, run before `AddZbLocalDb`'s lazy singleton constructs `SqliteLocalDb`) only helps **fresh** files; an unconditional `VACUUM` against a live ~1 GB replicated file was explicitly rejected (long exclusive lock on the active node; the rewrite is CDC-invisible but blocks the sync pump). Every node on the docker rig today predates the fix, so freed `site_events` pages there are reused, not returned to the OS — the storage cap (row 81) still bounds the data volume, only the file size stops shrinking. | Operator-scheduled `VACUUM` in a maintenance window (both nodes of a pair stopped together), or when a site's file is next recreated. |
|
||||
| 105 | ~~**Design constraint (not a defect): a LocalDb-replicated table must never be the parent of a row-mutating `ON DELETE` foreign key (`CASCADE`/`SET NULL`/`SET DEFAULT`) from another table**~~ **PINNED 2026-08-20.** Enforcement moved from the first replicated boot to the build: `SiteLocalDbReplicatedTableFkGuardTests` (Host.Tests) builds the real site schema via `OnReady` on an unreplicated config (so 0.3.3's own guard can't mask which FK is at fault), walks `SiteLocalDbSetup.ReplicatedTables` (now `internal`), asserts each exists and has no inbound row-mutating `ON DELETE` FK with the same `sqlite_master JOIN pragma_foreign_key_list` query LocalDb uses, and carries a negative control proving the checker finds a cascading child. The constraint itself stands — the row is kept as its record. Original: | Arch-review 2026-08-19 P4-7 → LocalDb 0.3.3 (scadaproj `122940c` + review follow-up `9663279`, Low-wave T5) | `LwwApplier` upserts with `INSERT OR REPLACE` under `foreign_keys=ON`, so the implicit delete fires the child action while CDC capture is suppressed — the child rows change on the applying node only and the pair diverges silently. 0.3.3 makes `RegisterReplicated` refuse such a parent at registration (clear exception naming the child FK), which is a BOOT failure for the site node, so this is enforced, not advisory. None of the ten `SiteLocalDbSetup.ReplicatedTables` has any FK today (zero `REFERENCES` on the live rig file). Model cross-table cleanup as explicit application deletes on each table (as the 3-table deploy cascade already does), never as SQLite FK actions. | Whenever a schema change adds a foreign key between site tables, or an eleventh table joins `ReplicatedTables`. |
|
||||
| 106 | ~~**LocalDb: a converged pair re-streams a FULL snapshot on EVERY reconnect once the sender's oplog has been ack-pruned to empty and the receiver's watermark is 0**~~ **RESOLVED 2026-08-20 — LocalDb 0.3.4 (scadaproj `75c1897`), published and pinned.** `SnapshotStreamer.SendAsync` now stamps `as_of_seq = max(MAX(seq), last_acked_seq)`; pinned by `WipedPeer_SnapshotLiftsItsWatermark_SoTheNextReconnectSendsNoSecondSnapshot` (fails on 0.3.3 exactly as the rig did: watermark 0, expected 3). Live proof on the rig: the first restart on 0.3.4 sent one last full snapshot stamped `as_of_seq 149` (was 0) and node-a's `last_applied_remote_seq` went 0 → 149; a second restart of the pair exchanged **zero** snapshots, `Connected=true`, backlog 0. Original: | Found 2026-08-20 while verifying row 104 on the rig (site-a-a received the same 127 k-row full snapshot at both the 13:09 deploy restart and the 13:25 VACUUM restart) | Sender-side defect in `SnapshotStreamer.SendAsync`: `as_of_seq = MAX(seq)` of the oplog, which is **0** once the pair has converged and ack-pruning emptied it. The receiver's `SetLastAppliedRemoteSeqAsync` is MAX-guarded (no regression — good), but a peer whose watermark is genuinely 0 (site-a-a was rebuilt after the 2026-08-15 corruption; any wiped/re-imaged node is the same) can never be lifted by an `as_of_seq 0` snapshot, so the 0.1.2 empty-oplog horizon rule (`streamableFrom = last_acked_seq + 1`) sees `0 < 149` as a gap on every handshake and re-streams everything. Rig state: node-b `last_acked_seq=149`, oplog empty; node-a `last_applied_remote_seq=0`. Not a divergence (LWW absorbs the re-delivered rows) but a cost/availability bug — ~1 s and 127 k rows here, minutes and a full-file read on a 1 GB production pair, on every reconnect, until the sender happens to write a new row. Fix is sender-side and wire-compatible: stamp `as_of_seq = max(MAX(seq), last_acked_seq)` (a full snapshot taken now covers every delta ever pruned below `last_acked_seq`), so the receiver's MAX-guarded watermark lifts to 149 and the loop ends; add a pinning test (converged+pruned sender, zero-watermark receiver → exactly one snapshot across two handshakes). Ships as LocalDb 0.3.4. | Next LocalDb release (0.3.4) — small, well-understood, and worth doing before any production pair is rebuilt from its peer. |
|
||||
| 107 | **Prod `CvdReactor.ProcessRecipeDownload` still addresses `Children["DelmiaRecv"]` after the 2026-06-27 `DelmiaRecv` → `DelmiaReceiver` composition rename** — inferred from the deployed script (`docs/api-specs/Delmia-Recipe-Download-API.md` §7 item 1): `ChildrenAccessor` concatenates the name into a path, an unknown path yields null, so every call answers `Receiver ready flag not set to true` regardless of the real flag. No call has hit prod since the rename, so this is inferred, not observed. The docker rig's stand-in (`docker/seed-delmia-sim.sh`) uses `Children["DelmiaReceiver"]` and the same handshake answers correctly, which is the working reference. | Found 2026-09-02 while capturing the as-deployed spec; QA copy unread (credentials) | Separate change with its own deployment record: `scadabridge template script update` on the `CvdReactor` template + redeploy Z28061/Z28062/Z28061Sim, then a live `WWNotifier` call. | Before the first production Delmia recipe download. |
|
||||
| 108 | **Inbound API keys are a PER-NODE SQLite store — a central failover invalidates every inbound client's key** *(High, availability + security-adjacent)*. `ZB.MOM.WW.Auth.ApiKeys` keeps keys in `ScadaBridge:InboundApi:ApiKeyStore:SqlitePath` (`/app/data/inbound-api-keys.sqlite`, on each central node's OWN data volume), and key admin (`ManagementActor`, a plain per-node `ActorOf` reached in-process from that node's `/management` endpoints) writes only the node that serves the call — via Traefik that is always the active node. So a key exists on exactly one node: after an active-central crash the survivor answers **401 `KeyNotFound`** to every inbound call (found live 2026-09-02 by `docker/verify-delmia-notifier.sh`: the standby-relay case passes only because the standby forwards BEFORE authenticating). The same asymmetry means a key revoked on one node stays valid on the other. Rig workaround: `docker/sync-inbound-api-keys.sh` mirrors the file (both nodes share one pepper); the verify script's preflight refuses to run unless both nodes list the key. Prod `wonder-app-vd03` is single-node; the QA pair (`vq03`/`vq04`) has this gap today. | `docs/api-specs/Delmia-Recipe-Download-API.md` §8; `docker/sync-inbound-api-keys.sh` header | Fix is an architecture decision, not a patch: move the store to the shared SQL Server (as Secrets 0.5.0 did for its central store) or replicate the file; either way revocation must become cluster-wide. Needs the `ZB.MOM.WW.Auth.ApiKeys` library (scadaproj), not just this repo. | Before any 2-node central goes live with inbound API clients; before the next arch-review wave. |
|
||||
| 68 | ~~**`deploy.sh` run from a git worktree silently rehomes every node's bind-mount data dir**~~ **RESOLVED 2026-08-15 (guard added in the same close-out).** The compose files' bind mounts are relative (`./site-a-node-a/data`), so a deploy from a worktree checkout mounts the worktree's copies — the rig keeps running and looks healthy, but its SQLite stores split from the main checkout's and everything written there is orphaned by the next deploy from main. Bit the rows-46–53 close-out live: the seeding agent's `rig-probe-1` deploy persisted into worktree-homed dirs and "vanished" after the orchestrator's redeploy (no tombstone, no product bug — LWW/tombstone forensics ruled out deletion; the row sat in `.claude/worktrees/…/docker/site-a-node-a/data/site-localdb.db` the whole time). Both `docker/deploy.sh` and `docker-env2/deploy.sh` now refuse to run from a `.claude/worktrees/` path (override: `SCADABRIDGE_ALLOW_WORKTREE_DEPLOY=1`). | Found during the rows-46–53 close-out | Closed 2026-08-15. | — |
|
||||
|
||||
## Resolved (verified against the code 2026-07-10)
|
||||
|
||||
@@ -0,0 +1,582 @@
|
||||
# Delmia Simulated Machine + Notifier Rig Verification — Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Stand up an in-memory (static-attribute) stand-in for a vd03-style machine with a composed `DelmiaReceiver` on the local docker rig, wire the `DelmiaRecipeDownload` inbound method and a scoped API key to it, let `DelmiaNotifier` read that key from `appsettings.json`, and prove the notifier end-to-end on the 2-node central cluster including an active-node crash and rejoin.
|
||||
|
||||
**Architecture:** Two templates seeded by CLI only (no DB edits): `DelmiaReceiverSim` (the ten receiver attributes, all static — no data connection) composed into `SimMachine` as `DelmiaReceiver`. The Galaxy side of the handshake is replaced by a root-scope `ValueChange` script on `SimMachine` that reacts to `DelmiaReceiver.RecipeDownloadFlag`, so the same call script shape as prod runs against pure in-memory tags. The central inbound method is the prod script verbatim (minus its dead `suf` line). The notifier gains an optional `ScadaBridge:ApiKey` config field with the existing `SCADABRIDGE_API_KEY` env var taking precedence, and a bash verification script drives the whole thing against `docker/` (central-a `:9001`, central-b `:9002`, Traefik `:9000`).
|
||||
|
||||
**Tech Stack:** ScadaBridge CLI (`scadabridge`), bash, .NET 10 (`DelmiaNotifier` + xUnit tests), docker compose rig under `docker/`.
|
||||
|
||||
**Facts the implementer must not re-derive** (verified 2026-09-02):
|
||||
|
||||
- `WriteBatchAndWaitAsync` **rejects static attributes** (`InstanceActor.HandleWriteAttributeBatch`: "Attribute 'X' is not a data-sourced attribute"). The rig script therefore uses `SetAsync` per attribute + `WaitAsync`, which both work on static attributes. Do not "fix" this in the runtime — out of scope.
|
||||
- A static write always publishes `AttributeValueChanged` (no equality short-circuit), and `ScriptActor` matches a `ValueChange` trigger by **exact** canonical name (`valueTrigger.AttributeName == changed.AttributeName`). Composed scripts' trigger configs are **not** prefixed at flattening (alarms are), so the simulator script lives on the **root** template with `attributeName` = `DelmiaReceiver.RecipeDownloadFlag`.
|
||||
- `WaitAsync` has a fast path when the current value already matches, so the simulator finishing before the wait registers is not a race.
|
||||
- Trigger types are matched case-insensitively as strings: `Call`, `ValueChange`, `Interval`, `Conditional`, `Expression`. Trigger config key is `attributeName`.
|
||||
- CLI global options (`--url`, `--username`, `--password`, `--format=json`) must come **after** the subcommand. Rig creds: `--username multi-role --password password`. Write loops in **bash** (zsh does not word-split unquoted variables).
|
||||
- `security api-key create --format=json` returns `{ "keyId": …, "token": "sbk_…" }`; the token is shown once.
|
||||
- Rig: site-a is `--site-id 1`; Traefik `http://localhost:9000` → active central; direct nodes `:9001` (central-a) and `:9002` (central-b); both centrals have `StandbyForwardPeerEndpoints` set, so a standby **relays** inbound calls to the active peer. Auto-down failover of a crashed active takes ~25–30 s (`docker/failover-drill.sh`).
|
||||
- The notifier's failover is **connect-failure-only**: any HTTP answer (incl. 503) is final. During the takeover window a standby that cannot reach its peer answers 503 `ACTIVE_NODE_UNREACHABLE` → the notifier prints `NO`. That is expected and must be documented, not "fixed".
|
||||
- Prod's deployed method + site script are recorded in `docs/api-specs/Delmia-Recipe-Download-API.md`; prod's site script still references `Children["DelmiaRecv"]` (a known, separate defect). The rig script uses `Children["DelmiaReceiver"]`.
|
||||
|
||||
**Model assignment summary** (see each task's `Model:` line): Sonnet for scripted CLI seeding, config plumbing, and docs; Opus for the live rig verification and the failover drill (judgment under a live system) and for reviewing the secret-handling change; Fable (this session) for the final integration review.
|
||||
|
||||
---
|
||||
|
||||
### Task 0: Preflight — rig healthy, CLI built, notifier tests green
|
||||
|
||||
**Classification:** trivial
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** none (everything depends on it)
|
||||
**Model:** sonnet
|
||||
|
||||
**Files:**
|
||||
- (none edited)
|
||||
|
||||
**Step 1: Confirm the rig is up and which node is active**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
for p in 9000 9001 9002; do printf "$p ready="; curl -s -m 5 -o /dev/null -w "%{http_code}" http://localhost:$p/health/ready; printf " active="; curl -s -m 5 -o /dev/null -w "%{http_code}\n" http://localhost:$p/health/active; done
|
||||
```
|
||||
Expected: all three `ready=200`; exactly one of `9001`/`9002` shows `active=200`, the other `503`; `9000 active=200`. If a node is down, run `bash docker/deploy.sh` first and re-check.
|
||||
|
||||
**Step 2: Build the CLI and the notifier, run the notifier tests**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
cd /Users/dohertj2/Desktop/ScadaBridge
|
||||
dotnet build src/ZB.MOM.WW.ScadaBridge.CLI -c Debug 2>&1 | tail -3
|
||||
dotnet test tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests 2>&1 | tail -5
|
||||
```
|
||||
Expected: `Build succeeded` and `Passed!` with 0 failed. Note the CLI binary path used by every later task: `src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge`.
|
||||
|
||||
**Step 3: Confirm the names this plan creates are free on the rig**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
CLI=src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge; A="--url http://localhost:9000 --format=json --username multi-role --password password"
|
||||
bash -c "$CLI template list $A | jq -r '.[].name' | grep -E '^(DelmiaReceiverSim|SimMachine)$'; $CLI api-method list $A | jq -r '.[].name' | grep -E '^DelmiaRecipeDownload$'; $CLI instance list $A | jq -r '.[].uniqueName' | grep -E '^DelmiaSim01$'; echo done"
|
||||
```
|
||||
Expected: only `done` printed. If any name is taken, delete it (`instance delete --id`, `template delete --id`, `api-method delete --id`) before continuing — the seed script is not idempotent by design.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: `DelmiaNotifier` — optional `ApiKey` in `appsettings.json`, env var wins
|
||||
|
||||
**Classification:** small (single-project logic change, but it relaxes a documented secret-handling decision — reviewer must weigh in)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 2, Task 3
|
||||
**Model:** sonnet (implement) → **opus** code review, with the explicit question "is the env-over-file precedence and the README wording adequate for a key that now may live on disk?"
|
||||
|
||||
**Design note (state this in the commit):** the 2026-06-26 design deliberately kept the key env-only. The user has asked for the key in `appsettings.json`. Keep env-only as the *recommended* production posture, add the file as a fallback, and keep the precedence deterministic: `SCADABRIDGE_API_KEY` (if non-blank) → `ScadaBridge:ApiKey` (if non-blank) → not configured. The failure message names both sources.
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/NotifierConfig.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/ConfigLoader.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/Program.cs:42-47`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/appsettings.json`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md` (the `## Secret — SCADABRIDGE_API_KEY` section and the config JSON block)
|
||||
- Modify: `env_variables.md` (§5 `SCADABRIDGE_API_KEY` row — "env-only by design" is no longer true)
|
||||
- Modify: `docs/plans/2026-06-26-delmia-recipe-notifier-design.md` (the `Secret` row of the key-decisions table: add "superseded 2026-09-02: file fallback added, env still wins")
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests/ConfigLoaderTests.cs`
|
||||
|
||||
**Step 1: Write the failing tests** (append to `ConfigLoaderTests.cs`)
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void Load_reads_ApiKey_from_ScadaBridge_section()
|
||||
{
|
||||
var cfg = ConfigLoader.Load("{\"ScadaBridge\":{\"BaseUrls\":\"http://x\",\"ApiKey\":\"sbk_file\"}}");
|
||||
Assert.Equal("sbk_file", cfg.ScadaBridge.ApiKey);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ResolveApiKey_prefers_env_over_config()
|
||||
{
|
||||
var cfg = ConfigLoader.Load("{\"ScadaBridge\":{\"ApiKey\":\"sbk_file\"}}");
|
||||
Assert.Equal("sbk_env", ConfigLoader.ResolveApiKey(n => n == "SCADABRIDGE_API_KEY" ? "sbk_env" : null, cfg));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ResolveApiKey_falls_back_to_config_when_env_unset_or_blank()
|
||||
{
|
||||
var cfg = ConfigLoader.Load("{\"ScadaBridge\":{\"ApiKey\":\" sbk_file \"}}");
|
||||
Assert.Equal("sbk_file", ConfigLoader.ResolveApiKey(_ => null, cfg));
|
||||
Assert.Equal("sbk_file", ConfigLoader.ResolveApiKey(_ => " ", cfg));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ResolveApiKey_returns_null_when_neither_source_set()
|
||||
{
|
||||
var cfg = ConfigLoader.Load("{\"ScadaBridge\":{\"ApiKey\":\" \"}}");
|
||||
Assert.Null(ConfigLoader.ResolveApiKey(_ => null, cfg));
|
||||
}
|
||||
```
|
||||
|
||||
Also update the two existing `ResolveApiKey_*` tests to pass a config: `ConfigLoader.ResolveApiKey(_ => null, new NotifierConfig())` etc. (the one-argument overload is removed, not kept — YAGNI).
|
||||
|
||||
**Step 2: Run tests to verify they fail**
|
||||
|
||||
Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests 2>&1 | tail -5`
|
||||
Expected: build error `'ScadaBridgeSection' does not contain a definition for 'ApiKey'` / no overload for `ResolveApiKey` taking 2 arguments.
|
||||
|
||||
**Step 3: Implement**
|
||||
|
||||
`NotifierConfig.cs` — add to `ScadaBridgeSection`:
|
||||
```csharp
|
||||
/// <summary>Optional inbound API key. <c>SCADABRIDGE_API_KEY</c> in the environment takes precedence; this is the file fallback (design 2026-06-26, relaxed 2026-09-02).</summary>
|
||||
public string? ApiKey { get; set; }
|
||||
```
|
||||
|
||||
`ConfigLoader.cs` — replace `ResolveApiKey`:
|
||||
```csharp
|
||||
/// <summary>Resolve the API key: <c>SCADABRIDGE_API_KEY</c> wins when non-blank, otherwise <c>ScadaBridge:ApiKey</c> from <paramref name="cfg"/> (trimmed); null when neither is set. Env accessor is injected for testability.</summary>
|
||||
/// <param name="envGet">Accessor used to read an environment variable by name.</param>
|
||||
/// <param name="cfg">The loaded configuration, consulted only when the environment variable is unset/blank.</param>
|
||||
/// <returns>The resolved API key, or <c>null</c> if neither source yields a non-blank value.</returns>
|
||||
public static string? ResolveApiKey(Func<string, string?> envGet, NotifierConfig cfg)
|
||||
{
|
||||
var env = envGet(ApiKeyEnvVar);
|
||||
if (!string.IsNullOrWhiteSpace(env)) return env;
|
||||
var file = cfg.ScadaBridge.ApiKey;
|
||||
return string.IsNullOrWhiteSpace(file) ? null : file.Trim();
|
||||
}
|
||||
```
|
||||
|
||||
`Program.cs` lines 42–47 become:
|
||||
```csharp
|
||||
var apiKey = ConfigLoader.ResolveApiKey(Environment.GetEnvironmentVariable, cfg);
|
||||
if (apiKey is null)
|
||||
{
|
||||
log.Write("API key not configured (SCADABRIDGE_API_KEY or ScadaBridge:ApiKey)");
|
||||
return Reporter.Report(false, "API key not configured (SCADABRIDGE_API_KEY or ScadaBridge:ApiKey)", stdout);
|
||||
}
|
||||
```
|
||||
|
||||
`appsettings.json` — add `"ApiKey": ""` after `LogPath` (empty = not set; **never commit a real token**).
|
||||
|
||||
README: in the config JSON block add `"ApiKey": ""`; rewrite the `## Secret` section:
|
||||
|
||||
> The inbound API key is sent as `X-API-Key`. It is resolved from **`SCADABRIDGE_API_KEY`** first (recommended for production — the key never touches disk), falling back to **`ScadaBridge:ApiKey`** in `appsettings.json` when the variable is unset or blank. If neither is set → `NO` + `API key not configured (SCADABRIDGE_API_KEY or ScadaBridge:ApiKey)`, exit `-1`, no HTTP attempt. If you use the file, restrict its ACL to the account DELMIA runs under.
|
||||
|
||||
`env_variables.md` §5: change the "In appsettings?" cell to `Yes — optional \`ScadaBridge:ApiKey\` fallback (2026-09-02); env wins` and the Purpose cell's "Read only from the environment, never a file" to "Environment first, file fallback".
|
||||
|
||||
Design doc table row `| Secret | API key from env var … — never in a file |` → append ` **(relaxed 2026-09-02: optional \`ScadaBridge:ApiKey\` file fallback; env still wins)**`.
|
||||
|
||||
**Step 4: Run tests to verify they pass**
|
||||
|
||||
Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests 2>&1 | tail -5`
|
||||
Expected: `Passed!`, 0 failed, total = previous count + 4. Also `dotnet build src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier -warnaserror` succeeds (the project treats warnings as errors; the XML doc comments above are required).
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests env_variables.md docs/plans/2026-06-26-delmia-recipe-notifier-design.md
|
||||
git commit -m "feat(delmia-notifier): optional ScadaBridge:ApiKey in appsettings.json as a fallback to SCADABRIDGE_API_KEY (env still wins); README/env inventory/design row updated"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: `docker/seed-delmia-sim.sh` — templates, composition, scripts, instance, deploy
|
||||
|
||||
**Classification:** standard (new script + two template scripts that must compile under the trust policy)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 1, Task 3
|
||||
**Model:** sonnet
|
||||
|
||||
**Files:**
|
||||
- Create: `docker/seed-delmia-sim.sh`
|
||||
- Modify: `docker/README.md:96-97` (add the new script to the tree listing, one line)
|
||||
|
||||
**Step 1: Write the script**
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
# Seed an IN-MEMORY stand-in for a vd03-style machine with a composed DelmiaReceiver on
|
||||
# site-a, so DelmiaNotifier can be exercised end-to-end on the docker rig without a Galaxy.
|
||||
#
|
||||
# What it creates (all static attributes — no data connection, no bindings):
|
||||
# - Template DelmiaReceiverSim — the ten DelmiaReceiver attributes from prod
|
||||
# (docs/api-specs/Delmia-Recipe-Download-API.md §5), ReadyFlag defaulting to true.
|
||||
# - Template SimMachine — composes DelmiaReceiverSim as `DelmiaReceiver` and carries
|
||||
# * ProcessRecipeDownload (Call) — prod's handshake ported to per-attribute
|
||||
# SetAsync/WaitAsync (WriteBatchAndWaitAsync rejects static attributes).
|
||||
# * DelmiaReceiverSimulator (ValueChange on DelmiaReceiver.RecipeDownloadFlag) —
|
||||
# plays the Galaxy object: accepts unless DownloadPath ends in ".bad" or the work
|
||||
# order is blank, writes RecipeProcessResult/Text, clears the trigger flag, then
|
||||
# sets RecipeProcessedFlag=true.
|
||||
# - Instance DelmiaSim01 on site-a (id 1), deployed.
|
||||
#
|
||||
# Not idempotent: names must be unique. To re-seed:
|
||||
# scadabridge instance delete --id <id>; scadabridge template delete --id <SimMachine id>;
|
||||
# scadabridge template delete --id <DelmiaReceiverSim id>
|
||||
# Prerequisites: bash docker/deploy.sh; docker/seed-sites.sh; central healthy on :9000.
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
CLI="$PROJECT_ROOT/src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge"
|
||||
A=(--url "${SCADABRIDGE_URL:-http://localhost:9000}" --format=json --username "${SCADABRIDGE_USERNAME:-multi-role}" --password "${SCADABRIDGE_PASSWORD:-password}")
|
||||
SITE_ID="${SITE_ID:-1}"
|
||||
INSTANCE_NAME="${INSTANCE_NAME:-DelmiaSim01}"
|
||||
|
||||
json_id() { python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])'; }
|
||||
|
||||
echo "1/6 Creating template DelmiaReceiverSim..."
|
||||
RECV_ID=$("$CLI" template create "${A[@]}" --name DelmiaReceiverSim \
|
||||
--description "In-memory stand-in for the Galaxy DelmiaReceiver object (ten static attributes)" | json_id)
|
||||
echo " id=$RECV_ID"
|
||||
|
||||
echo "2/6 Adding the ten receiver attributes (static)..."
|
||||
"$CLI" template attribute add "${A[@]}" --template-id "$RECV_ID" --name ReadyFlag --data-type Boolean --value true --description "Precondition; true = receiver accepts downloads" > /dev/null
|
||||
"$CLI" template attribute add "${A[@]}" --template-id "$RECV_ID" --name RecipeDownloadFlag --data-type Boolean --value false --description "Trigger: set true after the payload is written" > /dev/null
|
||||
"$CLI" template attribute add "${A[@]}" --template-id "$RECV_ID" --name RecipeProcessedFlag --data-type Boolean --value false --description "Cleared by the caller, set true by the receiver when done" > /dev/null
|
||||
"$CLI" template attribute add "${A[@]}" --template-id "$RECV_ID" --name RecipeProcessResult --data-type Boolean --value false --description "Outcome -> Result" > /dev/null
|
||||
for s in DownloadPath WorkOrderNumber PartNumber JobStepNumber Username RecipeProcessResultText; do
|
||||
"$CLI" template attribute add "${A[@]}" --template-id "$RECV_ID" --name "$s" --data-type String --description "Recipe-download payload / outcome text" > /dev/null
|
||||
done
|
||||
|
||||
echo "3/6 Creating template SimMachine + composing DelmiaReceiver..."
|
||||
MACHINE_ID=$("$CLI" template create "${A[@]}" --name SimMachine \
|
||||
--description "vd03-style machine stand-in: composes DelmiaReceiverSim as DelmiaReceiver; in-memory only" | json_id)
|
||||
"$CLI" template composition add "${A[@]}" --template-id "$MACHINE_ID" --composed-template-id "$RECV_ID" --instance-name DelmiaReceiver > /dev/null
|
||||
echo " id=$MACHINE_ID"
|
||||
|
||||
echo "4/6 Adding ProcessRecipeDownload (Call) — prod handshake on static attributes..."
|
||||
read -r -d '' PROCESS_CODE <<'CS' || true
|
||||
// Port of prod CvdReactor.ProcessRecipeDownload for STATIC attributes: WriteBatchAndWaitAsync only
|
||||
// accepts data-sourced attributes, so the payload is written with SetAsync and the handshake waited
|
||||
// with WaitAsync (same trigger/response flags, same 25 s budget, same result mapping).
|
||||
var recv = Children["DelmiaReceiver"];
|
||||
System.Func<object?,bool> IsTrue = v => { var s = v?.ToString(); return (v is bool b && b) || s == "1" || string.Equals(s, "true", System.StringComparison.OrdinalIgnoreCase); };
|
||||
System.Func<string,object?> P = k => Parameters.ContainsKey(k) ? Parameters[k] : null;
|
||||
if (!IsTrue(await recv.Attributes.GetAsync("ReadyFlag")))
|
||||
return new { Result = false, ResultText = "Receiver ready flag not set to true" };
|
||||
try {
|
||||
await recv.Attributes.SetAsync("RecipeProcessedFlag", false);
|
||||
await recv.Attributes.SetAsync("DownloadPath", P("DownloadPath"));
|
||||
await recv.Attributes.SetAsync("WorkOrderNumber", P("WorkOrderNumber"));
|
||||
await recv.Attributes.SetAsync("PartNumber", P("PartNumber"));
|
||||
await recv.Attributes.SetAsync("JobStepNumber", P("JobStepNumber"));
|
||||
await recv.Attributes.SetAsync("Username", P("Username"));
|
||||
await recv.Attributes.SetAsync("RecipeDownloadFlag", true);
|
||||
if (!await recv.Attributes.WaitAsync("RecipeProcessedFlag", true, System.TimeSpan.FromSeconds(25)))
|
||||
return new { Result = false, ResultText = "Timeout waiting for recipe to be processed" };
|
||||
return new {
|
||||
Result = IsTrue(await recv.Attributes.GetAsync("RecipeProcessResult")),
|
||||
ResultText = (await recv.Attributes.GetAsync("RecipeProcessResultText"))?.ToString() ?? ""
|
||||
};
|
||||
} catch (Exception ex) {
|
||||
return new { Result = false, ResultText = "RecipeDownload failed: " + ex.Message };
|
||||
}
|
||||
CS
|
||||
"$CLI" template script add "${A[@]}" --template-id "$MACHINE_ID" --name ProcessRecipeDownload --trigger-type Call \
|
||||
--parameters '{"type":"object","properties":{"DownloadPath":{"type":"string"},"WorkOrderNumber":{"type":"string"},"PartNumber":{"type":"string"},"JobStepNumber":{"type":"string"},"Username":{"type":"string"}},"required":[]}' \
|
||||
--return-def '{"type":"object","properties":{"Result":{"type":"boolean"},"ResultText":{"type":"string"}},"required":["Result","ResultText"]}' \
|
||||
--code "$PROCESS_CODE" > /dev/null
|
||||
|
||||
echo "5/6 Adding DelmiaReceiverSimulator (ValueChange on DelmiaReceiver.RecipeDownloadFlag)..."
|
||||
read -r -d '' SIM_CODE <<'CS' || true
|
||||
// Plays the Galaxy DelmiaReceiver object. Fires on every write to RecipeDownloadFlag (root-scope
|
||||
// trigger, canonical name) and acts only on the rising edge; its own reset to false re-triggers
|
||||
// this script, which then returns at the guard.
|
||||
var recv = Children["DelmiaReceiver"];
|
||||
var flag = await recv.Attributes.GetAsync("RecipeDownloadFlag");
|
||||
var fs = flag?.ToString();
|
||||
if (!((flag is bool b && b) || fs == "1" || string.Equals(fs, "true", System.StringComparison.OrdinalIgnoreCase))) return null;
|
||||
var path = (await recv.Attributes.GetAsync("DownloadPath"))?.ToString() ?? "";
|
||||
var wo = (await recv.Attributes.GetAsync("WorkOrderNumber"))?.ToString() ?? "";
|
||||
var ok = path.Length > 0 && wo.Length > 0 && !path.EndsWith(".bad", System.StringComparison.OrdinalIgnoreCase);
|
||||
await recv.Attributes.SetAsync("RecipeProcessResult", ok);
|
||||
await recv.Attributes.SetAsync("RecipeProcessResultText", ok ? "Simulated download accepted for " + wo : "Simulated reject: bad path or missing work order");
|
||||
await recv.Attributes.SetAsync("RecipeDownloadFlag", false);
|
||||
await recv.Attributes.SetAsync("RecipeProcessedFlag", true);
|
||||
return null;
|
||||
CS
|
||||
"$CLI" template script add "${A[@]}" --template-id "$MACHINE_ID" --name DelmiaReceiverSimulator --trigger-type ValueChange \
|
||||
--trigger-config '{"attributeName":"DelmiaReceiver.RecipeDownloadFlag"}' --code "$SIM_CODE" > /dev/null
|
||||
|
||||
echo "6/6 Creating + deploying instance $INSTANCE_NAME on site $SITE_ID..."
|
||||
INSTANCE_ID=$("$CLI" instance create "${A[@]}" --name "$INSTANCE_NAME" --template-id "$MACHINE_ID" --site-id "$SITE_ID" | json_id)
|
||||
"$CLI" instance deploy "${A[@]}" --id "$INSTANCE_ID"
|
||||
echo ""
|
||||
echo "DELMIA_SIM_TEMPLATE_ID=$RECV_ID SIM_MACHINE_TEMPLATE_ID=$MACHINE_ID INSTANCE_ID=$INSTANCE_ID INSTANCE_NAME=$INSTANCE_NAME"
|
||||
echo "Verify: $CLI debug snapshot --id $INSTANCE_ID --url http://localhost:9000 --username multi-role --password password"
|
||||
```
|
||||
|
||||
**Step 2: Make it executable and run it against the rig**
|
||||
|
||||
Run: `chmod +x docker/seed-delmia-sim.sh && bash docker/seed-delmia-sim.sh 2>&1 | tail -15`
|
||||
Expected: six numbered lines, a deploy reply without `"error"`, and the final `DELMIA_SIM_TEMPLATE_ID=… INSTANCE_ID=…` line. Record `INSTANCE_ID` for Task 4.
|
||||
|
||||
If `template script add` fails with a trust-policy or compile error, the message names the offending symbol — fix the script text, do **not** widen the trust policy. If `instance deploy` reports a pre-deployment validation failure mentioning bindings, a data-source was set by mistake: static attributes must be created **without** `--data-source`.
|
||||
|
||||
**Step 3: Prove the handshake works site-side before any HTTP is involved**
|
||||
|
||||
Run (bash):
|
||||
```bash
|
||||
CLI=src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge
|
||||
$CLI debug snapshot --id <INSTANCE_ID> --url http://localhost:9000 --format=json --username multi-role --password password | jq '.attributes // . | with_entries(select(.key|startswith("DelmiaReceiver.")))'
|
||||
```
|
||||
Expected: ten `DelmiaReceiver.*` entries, `ReadyFlag` true, `RecipeProcessedFlag` false. (The snapshot JSON shape is the CLI's — adapt the `jq` path to what comes back; the point is that all ten attributes exist under the `DelmiaReceiver.` prefix.)
|
||||
|
||||
**Step 4: Add the script to the docker tree listing**
|
||||
|
||||
`docker/README.md` line 97: after the `seed-live-probe.sh` row add
|
||||
`├── seed-delmia-sim.sh # Seed the in-memory Delmia machine stand-in (DelmiaSim01) on site-a`
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add docker/seed-delmia-sim.sh docker/README.md
|
||||
git commit -m "chore(docker): seed-delmia-sim.sh — in-memory SimMachine + composed DelmiaReceiver stand-in on site-a with a ValueChange simulator playing the Galaxy side"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: `docker/seed-delmia-api.sh` — inbound method + scoped API key
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** Task 1, Task 2
|
||||
**Model:** sonnet
|
||||
|
||||
**Files:**
|
||||
- Create: `docker/seed-delmia-api.sh`
|
||||
- Modify: `docker/README.md` (one tree-listing line under the Task 2 line)
|
||||
|
||||
**Step 1: Write the script**
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
# Seed the DelmiaRecipeDownload inbound method (prod's script verbatim, minus its dead `suf`
|
||||
# line — see docs/api-specs/Delmia-Recipe-Download-API.md §4) and one API key scoped to it.
|
||||
# Prints the one-time token as DELMIA_API_KEY=... — capture it; it cannot be retrieved again.
|
||||
# Not idempotent (method name must be unique): delete with `api-method delete --id`.
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
CLI="$PROJECT_ROOT/src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge"
|
||||
A=(--url "${SCADABRIDGE_URL:-http://localhost:9000}" --format=json --username "${SCADABRIDGE_USERNAME:-multi-role}" --password "${SCADABRIDGE_PASSWORD:-password}")
|
||||
KEY_NAME="${KEY_NAME:-delmia-notifier}"
|
||||
|
||||
read -r -d '' METHOD_CODE <<'CS' || true
|
||||
try {
|
||||
System.Func<string, object> P = k => Parameters.ContainsKey(k) ? Parameters[k] : null;
|
||||
var code = (P("MachineCode") as string ?? "").Trim();
|
||||
if (string.IsNullOrEmpty(code)) return new { Result = false, ResultText = "MachineCode is required" };
|
||||
|
||||
return await Route.To(code).Call("ProcessRecipeDownload", new {
|
||||
DownloadPath = P("DownloadPath"),
|
||||
WorkOrderNumber = P("WorkOrderNumber"),
|
||||
PartNumber = P("PartNumber"),
|
||||
JobStepNumber = P("JobStepNumber"),
|
||||
Username = P("Username"),
|
||||
});
|
||||
} catch (Exception ex) {
|
||||
return new { Result = false, ResultText = "RecipeDownload failed: " + ex.Message };
|
||||
}
|
||||
CS
|
||||
|
||||
echo "1/2 Creating inbound method DelmiaRecipeDownload (timeout 45 s)..."
|
||||
METHOD_JSON=$("$CLI" api-method create "${A[@]}" --name DelmiaRecipeDownload --timeout 45 \
|
||||
--parameters '{"type":"object","properties":{"MachineCode":{"type":"string"},"DownloadPath":{"type":"string"},"WorkOrderNumber":{"type":"string"},"PartNumber":{"type":"string"},"JobStepNumber":{"type":"string"},"Username":{"type":"string"}},"required":[]}' \
|
||||
--return-def '{"type":"object","properties":{"Result":{"type":"boolean"},"ResultText":{"type":"string"}},"required":["Result","ResultText"]}' \
|
||||
--script "$METHOD_CODE")
|
||||
echo " $(echo "$METHOD_JSON" | python3 -c 'import json,sys; d=json.load(sys.stdin); print("id=%s name=%s" % (d["id"], d["name"]))')"
|
||||
|
||||
echo "2/2 Creating API key '$KEY_NAME' scoped to DelmiaRecipeDownload..."
|
||||
KEY_JSON=$("$CLI" security api-key create "${A[@]}" --name "$KEY_NAME" --methods DelmiaRecipeDownload 2>/dev/null)
|
||||
TOKEN=$(echo "$KEY_JSON" | python3 -c 'import json,sys; print(json.load(sys.stdin)["token"])')
|
||||
echo "DELMIA_API_KEY=$TOKEN"
|
||||
```
|
||||
|
||||
> Note: `security api-key create` prints the token via `PrintCreatedKey` in table mode; with `--format=json` the raw `{keyId, token}` document is emitted and parsed above. If the json branch still prints the human lines, grep the `sbk_` token instead: `TOKEN=$(echo "$KEY_JSON" | grep -o 'sbk_[A-Za-z0-9_-]*' | head -1)`.
|
||||
|
||||
**Step 2: Run it and keep the token OUT of git**
|
||||
|
||||
Run: `chmod +x docker/seed-delmia-api.sh && bash docker/seed-delmia-api.sh`
|
||||
Expected: `id=<n> name=DelmiaRecipeDownload` then `DELMIA_API_KEY=sbk_…`. Save the token to the scratchpad only (e.g. `$SCRATCH/delmia.env` with `export DELMIA_API_KEY=…`). Never write it into `src/**/appsettings.json`.
|
||||
|
||||
**Step 3: Smoke the method with curl (no notifier yet)**
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://localhost:9000/api/DelmiaRecipeDownload -H "X-API-Key: $DELMIA_API_KEY" -H "Content-Type: application/json" \
|
||||
-d '{"MachineCode":"DelmiaSim01","DownloadPath":"C:\\recipes\\wo1.nc","WorkOrderNumber":"WO-1","PartNumber":"PN-1","JobStepNumber":"0100","Username":"curl"}'
|
||||
```
|
||||
Expected: `{"Result":true,"ResultText":"Simulated download accepted for WO-1"}`. Then the reject path (`"DownloadPath":"C:\\recipes\\x.bad"`) → `Result:false`, `Simulated reject…`; and a bogus key → HTTP 401 `{"error":"Invalid or missing API key"}`. If the first call returns `Timeout waiting for recipe to be processed`, the simulator did not fire: check `scadabridge debug snapshot` for `DelmiaReceiver.RecipeDownloadFlag` (should have been reset to false) and the site-a-a container log for a compile error on `DelmiaReceiverSimulator`.
|
||||
|
||||
**Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add docker/seed-delmia-api.sh docker/README.md
|
||||
git commit -m "chore(docker): seed-delmia-api.sh — DelmiaRecipeDownload inbound method (prod script) + scoped delmia-notifier API key"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: `docker/verify-delmia-notifier.sh` — notifier end-to-end + central failover drill
|
||||
|
||||
**Classification:** high-risk (drives `docker stop` on a live rig node; interprets failover timing)
|
||||
**Estimated implement time:** ~5 min to write, ~6 min to run (two drills × ~1 min settle)
|
||||
**Parallelizable with:** none (needs Tasks 1–3)
|
||||
**Model:** **opus** — run interactively, read the outputs, and decide whether a `NO` is an expected takeover-window answer or a real fault before proceeding.
|
||||
|
||||
**Files:**
|
||||
- Create: `docker/verify-delmia-notifier.sh`
|
||||
- Modify: `docker/README.md` (one tree-listing line)
|
||||
|
||||
**Step 1: Write the script**
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
# End-to-end DelmiaNotifier verification against the docker rig, including an ACTIVE central
|
||||
# crash + rejoin. Requires: seed-delmia-sim.sh + seed-delmia-api.sh already run, and
|
||||
# DELMIA_API_KEY exported (the token printed by seed-delmia-api.sh).
|
||||
#
|
||||
# The notifier is run as the managed build (dotnet WWNotifier.dll) from a scratch copy so the
|
||||
# API key is written to THAT copy's appsettings.json (ScadaBridge:ApiKey — Task 1) and never
|
||||
# into the repo. BaseUrls lists the two central nodes DIRECTLY (not Traefik) so the notifier's
|
||||
# own connect-failure failover is what gets exercised.
|
||||
#
|
||||
# Expected outcomes:
|
||||
# A baseline (active listed first) -> YES
|
||||
# B standby listed first (standby relays) -> YES (StandbyForwardPeerEndpoints)
|
||||
# C bad key -> NO (HTTP 401, no failover: 401 is an answer)
|
||||
# D active STOPPED, before takeover completes -> NO or YES — informational only. A NO here is the
|
||||
# documented connect-failure-only design (standby answers 503 ACTIVE_NODE_UNREACHABLE).
|
||||
# E active STOPPED, after survivor /health/active=200 -> YES (first URL refused -> failover)
|
||||
# F victim restarted and rejoined (members=2) -> YES
|
||||
# G reject path (DownloadPath *.bad) -> NO + "Simulated reject"
|
||||
|
||||
: "${DELMIA_API_KEY:?export DELMIA_API_KEY=sbk_... first}"
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"; PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
RUN_DIR="${RUN_DIR:-${TMPDIR:-/tmp}/delmia-notifier-run}"
|
||||
MACHINE="${MACHINE:-DelmiaSim01}"
|
||||
A_URL=http://localhost:9001; B_URL=http://localhost:9002
|
||||
|
||||
echo "== build notifier + stage a runnable copy in $RUN_DIR"
|
||||
dotnet build "$PROJECT_ROOT/src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier" -c Debug -nologo -v q
|
||||
rm -rf "$RUN_DIR"; mkdir -p "$RUN_DIR"; cp -R "$PROJECT_ROOT/src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/bin/Debug/net10.0/." "$RUN_DIR/"
|
||||
|
||||
write_cfg() { # $1 = BaseUrls, $2 = ApiKey
|
||||
printf '{"ScadaBridge":{"BaseUrls":"%s","TimeoutSeconds":30,"LogPath":"delmia-notifier.log","ApiKey":"%s"}}\n' "$1" "$2" > "$RUN_DIR/appsettings.json"
|
||||
}
|
||||
notify() { # $1 = label, $2 = expected YES|NO, rest = extra args; prints stdout + exit code
|
||||
local label=$1 expect=$2; shift 2
|
||||
local out rc; set +e; out=$(env -u SCADABRIDGE_API_KEY dotnet "$RUN_DIR/WWNotifier.dll" -m "$MACHINE" -d 'C:\recipes\wo1.nc' -w "WO-$label" -p PN-1 -s 0100 -u drill "$@" 2>/dev/null); rc=$?; set -e
|
||||
local first; first=$(echo "$out" | head -1)
|
||||
printf '%-3s stdout=%-4s exit=%-3s | %s\n' "$label" "$first" "$rc" "$(echo "$out" | sed -n 2p)"
|
||||
[ "$first" = "$expect" ] || { echo " ^^ EXPECTED $expect"; return 1; }
|
||||
}
|
||||
active() { curl -sf -o /dev/null "$A_URL/health/active" && echo a || { curl -sf -o /dev/null "$B_URL/health/active" && echo b; }; }
|
||||
url_of() { [ "$1" = a ] && echo "$A_URL" || echo "$B_URL"; }
|
||||
container_of() { [ "$1" = a ] && echo scadabridge-central-a || echo scadabridge-central-b; }
|
||||
other() { [ "$1" = a ] && echo b || echo a; }
|
||||
wait_for() { # $1 url $2 path $3 seconds
|
||||
local i; for i in $(seq 1 "$3"); do curl -sf -o /dev/null "$1$2" && return 0; sleep 1; done; return 1; }
|
||||
|
||||
ACT=$(active); STB=$(other "$ACT")
|
||||
echo "== active=central-$ACT standby=central-$STB"
|
||||
|
||||
echo "== A baseline (active first)"; write_cfg "$(url_of "$ACT"),$(url_of "$STB")" "$DELMIA_API_KEY"; notify A YES
|
||||
echo "== B standby first (relay)"; write_cfg "$(url_of "$STB"),$(url_of "$ACT")" "$DELMIA_API_KEY"; notify B YES
|
||||
echo "== C bad key"; write_cfg "$(url_of "$ACT"),$(url_of "$STB")" "sbk_bogus_bogus"; notify C NO || true
|
||||
write_cfg "$(url_of "$ACT"),$(url_of "$STB")" "$DELMIA_API_KEY"
|
||||
|
||||
echo "== D/E stop the ACTIVE central ($(container_of "$ACT"))"; docker stop "$(container_of "$ACT")" > /dev/null
|
||||
T0=$(date +%s)
|
||||
echo "-- D immediate attempt (informational):"; notify D NO || notify D YES || true
|
||||
echo "-- waiting for survivor $(url_of "$STB")/health/active (auto-down ~25-30 s)..."
|
||||
wait_for "$(url_of "$STB")" /health/active 90 || { echo "FAIL: survivor never became active"; exit 1; }
|
||||
echo " takeover after $(( $(date +%s) - T0 )) s"
|
||||
echo "== E after takeover (first URL refused -> failover)"; notify E YES
|
||||
|
||||
echo "== F restart victim + wait for rejoin"; docker start "$(container_of "$ACT")" > /dev/null
|
||||
wait_for "$(url_of "$ACT")" /health/ready 120 || { echo "FAIL: victim never ready"; exit 1; }
|
||||
for i in $(seq 1 60); do curl -s "$(url_of "$STB")/health/ready" | grep -q '"members":2\|members=2\|Up.*Up' && break; sleep 2; done
|
||||
notify F YES
|
||||
|
||||
echo "== G reject path"; notify G NO -d 'C:\recipes\x.bad' || true
|
||||
echo "== done. Notifier log: $RUN_DIR/delmia-notifier.log"
|
||||
```
|
||||
|
||||
> The `members` probe in F is deliberately loose — check what `/health/ready` actually prints on this build (`curl -s localhost:9001/health/ready`) and tighten the grep to the real field before relying on it.
|
||||
|
||||
**Step 2: Run it**
|
||||
|
||||
```bash
|
||||
chmod +x docker/verify-delmia-notifier.sh
|
||||
export DELMIA_API_KEY=sbk_… # from Task 3
|
||||
bash docker/verify-delmia-notifier.sh 2>&1 | tee "$SCRATCH/verify-delmia.log"
|
||||
```
|
||||
Expected: A, B, E, F print `stdout=YES exit=0`; C and G print `stdout=NO`; D is informational (record which it was and the takeover seconds). The victim rejoins as **standby** (`/health/active` 503 on it, 200 on the survivor). Any other outcome is a real finding — capture the notifier log line (`attempt <url>: …`) and the central log (`docker logs --since 5m scadabridge-central-<x> | grep -i delmia`) before changing anything.
|
||||
|
||||
**Step 3: Confirm the audit trail**
|
||||
|
||||
```bash
|
||||
CLI=src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge
|
||||
$CLI audit query --channel ApiInbound --since 15m --url http://localhost:9000 --format=json --username multi-role --password password | grep DelmiaRecipeDownload | jq -c '{t:.occurredAtUtc, kind, status, http:.httpStatus, node:.sourceNode, actor, resp:.responseSummary}'
|
||||
```
|
||||
Expected: one `InboundRequest` row per YES/NO business answer, **exactly one row for B** (the standby relay must not double-audit — `InboundApiEndpointFilter.RelayedToActiveItemKey`), an `InboundAuthFailure` for C, and `sourceNode` showing the node that executed each call (B and E should show the node that was active at the time).
|
||||
|
||||
**Step 4: Commit the script**
|
||||
|
||||
```bash
|
||||
git add docker/verify-delmia-notifier.sh docker/README.md
|
||||
git commit -m "test(docker): verify-delmia-notifier.sh — notifier end-to-end on the rig incl. standby relay, active-central crash/takeover and rejoin"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Record results + spec/doc updates
|
||||
|
||||
**Classification:** trivial (docs only)
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** none (needs Task 4's numbers)
|
||||
**Model:** sonnet
|
||||
|
||||
**Files:**
|
||||
- Modify: `docs/api-specs/Delmia-Recipe-Download-API.md` (new §8 "Docker rig stand-in", pointing at the two seed scripts, the static-attribute port, and the verified outcomes A–G with the measured takeover time)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md` (`## Manual smoke test` → add the rig recipe: run the three scripts, `export DELMIA_API_KEY`, expected YES/NO table)
|
||||
- Modify: `docs/plans/2026-07-08-deferred-work-register.md` (one new row: "prod `ProcessRecipeDownload` still addresses `Children[\"DelmiaRecv\"]` after the 2026-06-27 rename — fix + redeploy Z28061/Z28062/Z28061Sim; QA unverified" — only if a row for it does not already exist)
|
||||
- Modify: `CLAUDE.md` — in the `### Templates & Deployment` / DelmiaNotifier context, one sentence: "Rig stand-in: `docker/seed-delmia-sim.sh` + `seed-delmia-api.sh` + `verify-delmia-notifier.sh` (static-attribute port of the prod handshake — `WriteBatchAndWaitAsync` rejects static attributes, use `SetAsync`/`WaitAsync`)."
|
||||
|
||||
**Step 1: Write the sections** using the numbers from Task 4's log (takeover seconds, D's outcome, audit row count).
|
||||
|
||||
**Step 2: Check no stale claims remain**
|
||||
|
||||
Run: `grep -rn "never read from or written to a file\|env-only by design" src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier docs env_variables.md CLAUDE.md`
|
||||
Expected: no hits (Task 1 removed the README/env ones; anything left here is stale and must be updated).
|
||||
|
||||
**Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add docs/api-specs/Delmia-Recipe-Download-API.md src/ZB.MOM.WW.ScadaBridge.DelmiaNotifier/README.md docs/plans/2026-07-08-deferred-work-register.md CLAUDE.md
|
||||
git commit -m "docs: Delmia rig stand-in + notifier verification results (A-G incl. active-central takeover); register row for prod DelmiaRecv path defect"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Final integration review
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** none
|
||||
**Model:** **fable** (this session) — read `git log --oneline main..HEAD` + `git diff main`, re-run `dotnet test tests/ZB.MOM.WW.ScadaBridge.DelmiaNotifier.Tests`, and check specifically: (1) no `sbk_` token anywhere in the diff (`git diff main | grep -n 'sbk_[A-Za-z0-9]'` must return only the placeholder in the bad-key test), (2) the env-over-file precedence is asserted by a test, (3) the seed scripts leave the rig in a state `docker/README.md` describes, (4) Task 4's log shows E and F as YES with the victim rejoined as standby. Then decide with the user whether to push (`/pushit`) — the plan does not push.
|
||||
|
||||
---
|
||||
|
||||
## Out of scope (deliberately)
|
||||
|
||||
- Fixing prod vd03's `Children["DelmiaRecv"]` defect or making the prod method idempotent — recorded in the deferred-work register (Task 5); separate change with its own deployment record.
|
||||
- Extending `WriteBatchAndWaitAsync` to static attributes.
|
||||
- Site-pair failover (the user asked for the 2-node **central** cluster); `docker/failover-drill.sh` already covers the general case.
|
||||
- Native AOT publish of the notifier (Windows-only); the managed build is what the rig exercises.
|
||||
@@ -0,0 +1,69 @@
|
||||
{
|
||||
"planPath": "docs/plans/2026-09-02-delmia-sim-rig-verification.md",
|
||||
"tasks": [
|
||||
{
|
||||
"id": 0,
|
||||
"subject": "Task 0: Preflight \u2014 rig healthy, CLI built, notifier tests green",
|
||||
"status": "completed",
|
||||
"model": "sonnet"
|
||||
},
|
||||
{
|
||||
"id": 1,
|
||||
"subject": "Task 1: DelmiaNotifier \u2014 optional ApiKey in appsettings.json, env var wins",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
0
|
||||
],
|
||||
"model": "sonnet",
|
||||
"review": "opus"
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"subject": "Task 2: docker/seed-delmia-sim.sh \u2014 templates, composition, scripts, instance, deploy",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
0
|
||||
],
|
||||
"model": "sonnet"
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"subject": "Task 3: docker/seed-delmia-api.sh \u2014 inbound method + scoped API key",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
0
|
||||
],
|
||||
"model": "sonnet"
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"subject": "Task 4: docker/verify-delmia-notifier.sh \u2014 notifier end-to-end + central failover drill",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
1,
|
||||
2,
|
||||
3
|
||||
],
|
||||
"model": "opus"
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"subject": "Task 5: Record results + spec/doc updates",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
4
|
||||
],
|
||||
"model": "sonnet"
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"subject": "Task 6: Final integration review",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
5
|
||||
],
|
||||
"model": "fable"
|
||||
}
|
||||
],
|
||||
"lastUpdated": "2026-09-02T17:10:00Z"
|
||||
}
|
||||
Reference in New Issue
Block a user