Files
ScadaBridge/docs/deployment/deployment-records/2026-11-15-gate-52-delmiareceiver-fix.md
T

8.3 KiB
Raw Blame History

2026-11-15: #52 CvdReactor.ProcessRecipeDownload child-name fix, DEV vd03 + QA vq03/vq04 (Z28061 excluded)

Owner go: 2026-11-15, "yeah" to applying #52 (DEV first, then QA). Z28061 is excluded on both tiers: another team uses it and no redeploy slot has been agreed.

Change: one line in template CvdReactor (id 1), script ProcessRecipeDownload (id 4): Children["DelmiaRecv"] becomes Children["DelmiaReceiver"]. The body is deploy-prep/2026-11-12-gates-52-131/fixed/CvdReactor.ProcessRecipeDownload.cs. Every other script field is re-sent from original/CvdReactor.ProcessRecipeDownload.meta.json and read back unchanged.

Scope: management API only, through the ScadaBridge CLI (built from main) run from the Mac.

  • No binary, no migration, no service restart, no WinRM. Only the redeployed instance actors restarted.
  • Not touched: Z28061 (template change only; its deployment was left alone), every other template, script and instance, OtOpcUa, mxaccessgw, HistorianGateway and Overview.

Tooling change (apply.sh, same commit): every CLI call now passes --url, --username and --password explicitly. Earlier on 2026-11-15, an env-only SCADABRIDGE_URL pointing at QA had silently reached DEV, because the CLI fell back to its configured default central. Two optional guards were added:

  • SB_EXPECT=<names> refuses to run unless the listed instances exist on the central, which proves the tier. The full instance list is printed first.
  • SB_SKIP=<names> never deploys the listed instances. They are left stale.

Functional check (read-only)

The call is flow-debug run-script --instance-id 2 --script ProcessRecipeDownload --params '{}' on Z28062, made only after confirming DelmiaReceiver.ReadyFlag is False / Good. In that state, the script returns before its try block, so nothing is written to the PLC.

  • The returned text is the same before and after: Receiver ready flag not set to true. That is the correct answer when the real flag is False, so the text alone does not prove the fix.
  • The evidence is the trace instead. Before the fix, the run logged WARN runtime GetAttribute: attribute 'DelmiaRecv.ReadyFlag' not found on instance 'Z28062'. This is the #52 defect: a null child, so the hard-coded text came back regardless of the real flag. After the fix, the warning is gone, and the script read the existing DelmiaReceiver.ReadyFlag.
  • Not exercised: the download path with ReadyFlag True. It writes RecipeDownloadFlag and the recipe fields to the receiver and waits on RecipeProcessedFlag. Running it needs a planned recipe download.
  • On QA, Z28063 has DelmiaReceiver.ReadyFlag True. A call there would take the write path, so it was not called. Its first real DelmiaRecipeDownload will now take the download path, which it did not before.

DEV vd03 (http://wonder-app-vd03.zmr.zimmer.com:8085, DisableLogin)

Step Result
Tier proof Instances: SBTEST01, SBTEST02, Z28061, Z28061Sim, Z28062, ZIPSEN. This is DEV's set
check 52 Live body ORIGINAL. Z28061, Z28061Sim and Z28062 not stale (deployed 2026-10-28)
Negative control The trace logs DelmiaRecv.ReadyFlag' not found
apply 52 (SB_SKIP=Z28061) Script 4 updated; metadata read back unchanged. Deployed Z28061Sim (deployment 97) and Z28062 (98). Z28061 skipped
check 52 after Body FIXED. Z28061Sim and Z28062 not stale (22:23:20–21Z). Z28061 stale=True, as intended
Trace after RouteCall and ScriptRun Delivered, with the same ResultText and no not found warning

QA vq03/vq04 (active central vq04 :8085, login multi-role)

The active central is vq04. /health/active returns vq03 503 and vq04 200. On 2026-11-13 it was vq03. Live flow-debug runs are refused on the standby (STANDBY_NODE), so every QA call went to vq04. Reads and writes go through the active node either way.

Step Result
Safety re-check (22:24Z) The 8 instances to redeploy: CycleRunning False/False on both sides, FurnaceTemp 23–25 °C. All SAFE. Open batches: Z28063 L 29049, Z28065 R 29020, Z28068 L 29081
Tier proof Instances: WarmCanary, Z28061–Z28069, ZIPSEN. This is QA's set
check 52 Live body ORIGINAL. All 9 CvdReactor instances not stale (deployed 2026-11-06)
Negative control Z28062 trace logs DelmiaRecv.ReadyFlag' not found
apply 52 (SB_SKIP=Z28061) Script 4 updated; metadata read back unchanged. Deployed Z28062–Z28069 (deployments 233–240). Z28061 skipped
check 52 after Body FIXED. Z28062–Z28069 not stale (22:24:33–37Z). Z28061 stale=True
Trace after Z28062: RouteCall and ScriptRun Delivered, with no not found warning
Post-redeploy All 8 instances 244/244 attributes Good. CycleRunning False. The open batches and current steps are unchanged (Z28063 L 29049/step 2, Z28065 R 29020/step 2, Z28068 L 29081/step 6). ReadyFlag values unchanged

Follow-up

Z28061 still runs the old script on both tiers. It shows stale=True and still carries the #52 defect. Redeploy it once the other team agrees a slot. The tracking issue is in Gitea (#52 follow-up). The command is either instance deploy --id 1 (explicit --url), or apply.sh apply 52 without SB_SKIP, which is a no-op for the script and deploys only the stale instances.

Rollback

Per tier: SCADABRIDGE_URL=<central> SCADABRIDGE_USERNAME=… SCADABRIDGE_PASSWORD=… SB_SKIP=Z28061 bash apply.sh rollback 52. It refuses unless the live body is FIXED. It writes the original body back, and the instances redeploy onto it.

Z28061 (2026-11-15, #142)

The owner said: "if its idle you can do rest of #52". So on each tier Z28061 was redeployed only if it was provably idle at that moment, which stands in for the slot with its other team. Only the management CLI was used, with an explicit --url. Only Z28061 was deployed. apply.sh apply was not used, because it deploys every CvdReactor instance.

DEV vd03: SKIPPED, idle can't be proven

  • apply.sh check 52 at 23:06Z: the body is FIXED, Z28061 (id 1) is stale=True (deployed 2026-10-28), and Z28061Sim and Z28062 are not stale.
  • debug snapshot at 23:06:03Z: 270 attributes, 16 Good.
    • Every idle-check attribute is Bad, value null: CycleRunning, MachineCycleID, FurnaceTemp and MachineBatchID on both sides, plus Left/RightDelmiaReceiver.RecipeDownloadFlag.
    • Only DelmiaReceiver.RecipeDownloadFlag and ReadyFlag are Good, both False, with the value last changed 2026-08-26.
  • The rule says Bad quality means not proven idle, so it was not deployed. DEV Z28061 still runs the old body (Children["DelmiaRecv"]).
  • To unblock DEV: get Z28061's DEV data source back to Good quality, or have its team confirm it's idle.

QA vq03/vq04: DONE

  • Active central: vq03 (/health/active vq03 200, vq04 503); the #116 stop-and-hold drill had moved it there. Every call went to vq03.
  • apply.sh check 52: body FIXED, SB_EXPECT=Z28063,Z28069,WarmCanary passed. Z28061 (id 1) was the only stale instance; Z28062–Z28069 were already current.

Idle check (snapshots at 23:06:18Z and 23:07:25Z, 67 s apart; 244 attributes, all Good):

Attribute Left Right
CycleRunning False False
MachineCycleID 0 0
FurnaceTemp 25 25
MachineBatchID 0 0
  • DelmiaReceiver.RecipeDownloadFlag and ReadyFlag: both False. QA's Z28061 has no Left/Right Delmia receivers.
  • Between the two snapshots, the only change was TrunkPressure 0.18 → 0.17, which is noise.

Deploy and verification:

  • Deploy: instance deploy --id 1 gave deployment 241, status 2, done at 23:07:36.81Z in 0.26 s; revision sha256:2727a432….
  • instance diff: isStale=False.
  • Snapshot after (23:08:02Z): 244 attributes, 244 Good, no value change except TrunkPressure noise, and no batch, so nothing to preserve.
  • Trace: ReadyFlag re-read as False / Good first, so the script returns before any PLC write. flow-debug run-script --instance-id 1 --script ProcessRecipeDownload --params '{}' at 23:08:13Z gave RouteCall and ScriptRun both Delivered, with no not found warning.

Result

  • QA: all ten CvdReactor instances (Z28061–Z28069) run the fixed body.
  • DEV: Z28062 and Z28061Sim are fixed. Z28061 is still pending on #142.