diff --git a/CLAUDE.md b/CLAUDE.md index fa66860a..2aedf436 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -72,6 +72,7 @@ spec for each is `docs/requirements/Component-.md`, and `README.md` carrie - Staggered Instance Actor startup on failover to prevent reconnection storms. - **Instance Actor tag-subscribe retry is guarded + backed off (Lever 1, 2026-08-28).** One outstanding `SubscribeTagsRequest` per connection until `TagSubscribeInFlightCeilingMs` (45 s); the interval doubles ±20 % jitter from `TagSubscribeRetryIntervalMs` (5 s) to `TagSubscribeRetryMaxIntervalMs` (60 s), resetting on Success or `ConnectionQualityChanged(Good)`. Pre-Lever-1 the actor re-sent a full N-tag subscribe every fixed 5 s with no in-flight check — under a slow MxGateway that stacked ~6 concurrent 244-tag subscribes per 30 s and wedged the session (QA 2026-08-26). Consequence to know: a *lost* reply is now re-sent after the ceiling (45 s), not the 5 s interval — a *failed* reply still retries at the backed-off interval. Levers 2 (bulk supervisory advise, cross-repo) and 3 (session `command_timeout`) are still design-only: `docs/plans/2026-08-28-mxgateway-subscribe-storm-prevention-design.md`. - Explicit supervision strategies: Resume for coordinator actors, Stop for short-lived execution actors. +- **Level-triggered flag handshakes must observe the reset echo before raising the trigger (Gitea #36, 2026-09-10).** `Attributes.WriteBatchAndWaitAsync` has a reset-aware overload — `(values, flagKey, flagValue, responseKey, responseValue, timeout, resetKey, resetValue, resetTimeout?)` → `HandshakeResult` (`Matched`/`TimedOut`/`Phase`/`ResetFailed`) — that writes the reset, waits event-driven for the device to echo it, and only then writes the batch + trigger and waits for the response. The six-argument bool overload is unchanged but **unsafe for a repeated handshake**: the waiter's current-value fast-path answers from the previous run's cached response value while the script's reset write is still in flight (QA measured 0 ms matches and stale `BatchID`s). An edge-triggered wait was rejected — a subscription reports changes from the last *reported* value, so a device flipping reset→done inside one sampling interval never reports the reset and such a wait never resolves. Composition only: `ScriptRuntimeContext.WriteBatchResetAndWait` chains the existing `SetAttribute` → waiter → batch → waiter legs, so no actor/message/proto change, and a traced run shows all four steps. Mirrored on `ScriptCompileSurface`, `SandboxScriptHost` and `FlowShapePredictor`. The QA/prod MES script port is register row 119. Design: `docs/plans/2026-09-10-handshake-reset-echo-design.md`. ### Data & Communication - Data Connection Layer is a clean data pipe — publishes to Instance Actors only. diff --git a/docker/seed-delmia-sim.sh b/docker/seed-delmia-sim.sh index f1d9779b..28d2ff2a 100755 --- a/docker/seed-delmia-sim.sh +++ b/docker/seed-delmia-sim.sh @@ -9,7 +9,10 @@ set -euo pipefail # (docs/api-specs/Delmia-Recipe-Download-API.md §5), ReadyFlag defaulting to true. # - Template SimMachine — composes DelmiaReceiverSim as `DelmiaReceiver` and carries # * ProcessRecipeDownload (Call) — prod's handshake ported to per-attribute -# SetAsync/WaitAsync (WriteBatchAndWaitAsync rejects static attributes). +# SetAsync/WaitAsync (WriteBatchAndWaitAsync rejects static attributes). The #36 +# reset-echo race cannot occur here: a STATIC write updates the cache synchronously, +# so the reset is "echoed" before the trigger is written. The data-sourced proof of +# the reset-aware overload is docker/seed-handshake-demo.sh. # * DelmiaReceiverSimulator (ValueChange on DelmiaReceiver.RecipeDownloadFlag) — # plays the Galaxy object: accepts unless DownloadPath ends in ".bad" or the work # order is blank, writes RecipeProcessResult/Text, clears the trigger flag, then diff --git a/docs/api-specs/Delmia-Recipe-Download-API.md b/docs/api-specs/Delmia-Recipe-Download-API.md index d5c58ff4..5862fec0 100644 --- a/docs/api-specs/Delmia-Recipe-Download-API.md +++ b/docs/api-specs/Delmia-Recipe-Download-API.md @@ -149,6 +149,13 @@ try { ``` +> **Known race (Gitea #36, 2026-09-10).** Step 2's `RecipeProcessedFlag = false` is a device +> write whose echo has not arrived when step 3's wait registers, so the wait can be satisfied by +> the PREVIOUS download's cached `true`. The runtime fix is the reset-aware overload — +> `WriteBatchAndWaitAsync(data, "RecipeDownloadFlag", true, "RecipeProcessedFlag", true, 25 s, +> "RecipeProcessedFlag", false, 10 s)` returning a `HandshakeResult` — which replaces steps 2–3. +> The deployed prod/QA script bodies are NOT yet ported (deferred-work register row 119). + ## 5. Galaxy-side contract — the `DelmiaReceiver` template (id 12; composed clone id 26) All ten attributes are **data-sourced over the site `MxGateway` connection**, bound per instance to the diff --git a/docs/plans/2026-06-16-ipsen-mes-movein-design.md b/docs/plans/2026-06-16-ipsen-mes-movein-design.md index 7b52b5af..2a6c047a 100644 --- a/docs/plans/2026-06-16-ipsen-mes-movein-design.md +++ b/docs/plans/2026-06-16-ipsen-mes-movein-design.md @@ -69,6 +69,11 @@ receiver is ready to accept a MoveIn. Written: `MoveInMesContainerNum`, `MoveInBatchID`, `MoveInJobSequenceNumber`, `MoveInNumberWorkOrders`, `MoveInWorkOrderNumbers[]`, `MoveInPartNumbers[]`, `MoveInOperatorName`, `MoveInFlag`. +4a. **Handshake ordering (Gitea #36, 2026-09-10):** the reset of `CompleteFlag` must be + ECHOED by the Galaxy object before `Flag` is raised, otherwise the completion wait + fast-paths on the previous request's cached `true`. Port the receiver scripts onto the + reset-aware `WriteBatchAndWaitAsync` overload (`docs/plans/2026-09-10-handshake-reset-echo-design.md` + §4) and drop the `Task.Delay(1000)`. 5. **Return BatchID (Q2):** echo the input `MoveInBatchID` (no read-back). ## 4. Components diff --git a/docs/plans/2026-06-17-waitfor-attribute-change-helper-spec.md b/docs/plans/2026-06-17-waitfor-attribute-change-helper-spec.md index 8261e518..7eff085f 100644 --- a/docs/plans/2026-06-17-waitfor-attribute-change-helper-spec.md +++ b/docs/plans/2026-06-17-waitfor-attribute-change-helper-spec.md @@ -222,6 +222,15 @@ type/members don't collide with `ForbiddenIdentifiers` (`dynamic`, `Activator`) actor to evict the orphan waiter promptly (otherwise it self-evicts at its own timeout). - **Concurrency / re-entrancy.** Multiple waiters per instance are fine (keyed by `CorrelationId`). Consider a per-instance cap as a guard against a script leaking waiters in a loop. +- **Handshake reset ordering (Gitea #36, design `2026-09-10-handshake-reset-echo-design.md`).** + The fast-path above is correct ONLY once the caller's reset has been observed. A script that + writes `CompleteFlag=false` and immediately calls the batch helper races the device echo: the + cache still holds the previous run's `true`, the wait fast-paths in 0 ms, and the script reads + stale results. An edge-triggered wait is NOT the fix (a subscription can swallow the reset when + the device flips inside one sampling interval); the reset must be echoed BEFORE the trigger is + written. Use the reset-aware `Attributes.WriteBatchAndWaitAsync(…, resetKey, resetValue, + resetTimeout)` overload, which returns a `HandshakeResult` (`ResetFailed` ⇒ nothing else was + written). --- diff --git a/docs/plans/2026-07-08-deferred-work-register.md b/docs/plans/2026-07-08-deferred-work-register.md index 7b83d60e..f0ea6d18 100644 --- a/docs/plans/2026-07-08-deferred-work-register.md +++ b/docs/plans/2026-07-08-deferred-work-register.md @@ -112,6 +112,7 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round | 117 | **Flow Debugger site trace pull: `FlowDebugRunner.WriteAndPollAsync` fetches ONE bounded page with a fixed `since`** *(Low)*. The write-and-poll loop asks the site for `FlowDebugRunner.PullTraceEventsMax` rows per poll, always `since` the run start, and the site serves `ORDER BY OccurredAtUtc ASC LIMIT` — so the page is the OLDEST rows in the window, never a continuation of the previous one. Step logs made that reachable: one traced run can now emit up to 201 `ScriptLog` rows, so a handful of chatty runs inside the window can fill the page and push a later spawned run's `ScriptRun` row past the end of it, at which point the loop never sees the awaited run and reports a false `NO_TRIGGER_WITHIN_WINDOW` for a run that did happen. Final review I1 raised the cap 500 -> 2 000 (the site's own ceiling, `SiteStreamGrpcServer.MaxTraceEventsPerPull`), which makes it ~10x harder but does not remove it. The real fix is to PAGE — advance `since` to the last-seen `OccurredAtUtc` between polls (keeping the already-known ids so nothing is double-counted) — or to exclude `ScriptLog` from the awaited-run poll entirely and pull the log rows once, after the run has settled. | Final review of `docs/plans/2026-09-09-flow-debugger-step-logs.md` (I1); `src/ZB.MOM.WW.ScadaBridge.ManagementService/FlowDebug/FlowDebugRunner.cs` `WriteAndPollAsync` | 2 000 is the site's hard ceiling, so the headroom cannot be bought again with a bigger number; paging needs the poll loop to track a cursor and the awaited-run predicate to survive a partial view, which is more than the raise was worth at review time. | The first `NO_TRIGGER_WITHIN_WINDOW` reported for a run whose Replay of the same root DOES show the spawned `ScriptRun`. | | 118 | **Pre-existing bUnit flake: `CentralUI.Tests.Pages.NotificationReportDetailModalTests.CloseButton_DismissesModal` (~10 %)** *(Low)*. Fails intermittently with `UnknownEventHandlerIdException` — the test resolves the close button once and clicks it after a re-render has already retired that element's handler id. Not caused by the step-logs work: the file is untouched since `40944a0d`. The fix is a one-liner — re-`Find` the button immediately before the click (the pattern the other modal tests already use) rather than reusing an element captured earlier in the test. | Observed during the step-logs full-suite runs, 2026-09-09 | It is a test-only race with a known cause and a known fix; it was out of scope for a remediation pass whose rule was to touch only the files under review. | Next time `CentralUI.Tests` is worked on, or the first time the flake blocks a gate run. | | 68 | ~~**`deploy.sh` run from a git worktree silently rehomes every node's bind-mount data dir**~~ **RESOLVED 2026-08-15 (guard added in the same close-out).** The compose files' bind mounts are relative (`./site-a-node-a/data`), so a deploy from a worktree checkout mounts the worktree's copies — the rig keeps running and looks healthy, but its SQLite stores split from the main checkout's and everything written there is orphaned by the next deploy from main. Bit the rows-46–53 close-out live: the seeding agent's `rig-probe-1` deploy persisted into worktree-homed dirs and "vanished" after the orchestrator's redeploy (no tombstone, no product bug — LWW/tombstone forensics ruled out deletion; the row sat in `.claude/worktrees/…/docker/site-a-node-a/data/site-localdb.db` the whole time). Both `docker/deploy.sh` and `docker-env2/deploy.sh` now refuse to run from a `.claude/worktrees/` path (override: `SCADABRIDGE_ALLOW_WORKTREE_DEPLOY=1`). | Found during the rows-46–53 close-out | Closed 2026-08-15. | — | +| 119 | **Port the QA/prod MES receiver scripts onto the reset-aware `WriteBatchAndWaitAsync` overload (Gitea #36)** *(Medium)*. Template scripts CvdReactor MesMoveIn (13) / MesMoveOut (14) / MesSampleData (36) and base MESReceiver MoveIn/MoveOut (15, 16, 25, 26, 37, 38) still reset `CompleteFlag` with a plain write and rely on `Task.Delay(1000)`; QA carries a config-only `WaitAsync(false, 10 s)` patch on 13/14 (deployment record 110). Rewrite: `var hs = await rx.Attributes.WriteBatchAndWaitAsync(values, "Flag", true, "CompleteFlag", true, 25 s, "CompleteFlag", false, 10 s); if (!hs.Matched) return new { WasSuccessful=false, ErrorText = hs.ResetFailed ? "Timeout waiting for CompleteFlag to reset" : "Timeout waiting for to complete", BatchID=0 };` and delete the delay. Requires the #36 build on the target central + sites first (the overload does not exist on older sites → deploy-gate compile failure, by design). | Gitea #36, design `2026-09-10-handshake-reset-echo-design.md` §8 | Repo-only scope was the user's decision for the #36 session; the script bodies live only in the QA/prod databases. | Next QA deployment of a build ≥ this change (write a deployment record; then prod). | ## Resolved (verified against the code 2026-07-10) Rows removed from the Deferred table above once confirmed shipped. Kept here for traceability. diff --git a/docs/requirements/Component-ScriptAnalysis.md b/docs/requirements/Component-ScriptAnalysis.md index a94ac132..f2531c50 100644 --- a/docs/requirements/Component-ScriptAnalysis.md +++ b/docs/requirements/Component-ScriptAnalysis.md @@ -179,7 +179,7 @@ active use across an eviction while keeping the same ceiling. A reflection-based parity test in `SiteRuntime.Tests` compares the public member names on `ScriptCompileSurface` against `ScriptGlobals` (and `TriggerCompileSurface` against `TriggerExpressionGlobals`). Any drift between the stub and the real globals causes this test to fail, ensuring the stubs cannot silently fall out of sync. -There is a **third** hand-maintained mirror of the runtime globals: the Central UI Test-Run host `SandboxScriptHost` (see REQ-SA-5 / Interactions). Because Central UI deliberately does not reference Site Runtime, it cannot share the reflection-based parity test above; instead it is guarded by representative-script "diagnose clean" tests in `CentralUI.Tests` (one per non-trivial surface — e.g. the `Attributes.WriteBatchAndWaitAsync` / `WaitAsync` / `WaitForAsync` batch-write-and-wait helpers, the inbound `Database`/`WaitForAttribute` helpers, and the `Notify` outbox shape). A member that drifts out of `SandboxScriptHost` does **not** fail the deploy gate (which compiles against `ScriptCompileSurface`) — it surfaces only as an in-editor `CS1061` false error against otherwise-valid scripts, so these compile-clean tests are the safety net for that surface. +There is a **third** hand-maintained mirror of the runtime globals: the Central UI Test-Run host `SandboxScriptHost` (see REQ-SA-5 / Interactions). Because Central UI deliberately does not reference Site Runtime, it cannot share the reflection-based parity test above; instead it is guarded by representative-script "diagnose clean" tests in `CentralUI.Tests` (one per non-trivial surface — e.g. the `Attributes.WriteBatchAndWaitAsync` (including the reset-aware nine-argument overload) / `WaitAsync` / `WaitForAsync` batch-write-and-wait helpers, the inbound `Database`/`WaitForAttribute` helpers, and the `Notify` outbox shape). A member that drifts out of `SandboxScriptHost` does **not** fail the deploy gate (which compiles against `ScriptCompileSurface`) — it surfaces only as an in-editor `CS1061` false error against otherwise-valid scripts, so these compile-clean tests are the safety net for that surface. --- @@ -228,7 +228,7 @@ The decorator changes **no resolution result**, only object identity: it returns `FlowShapePredictor` (Flow Debugger design `docs/plans/2026-09-05-flow-debugger-design.md` §8) predicts, from script **syntax** alone, the swimlane shape a run would produce before anything executes: `PredictInboundMethod(methodName, methodScript, resolveRouteScript, maxDepth = 4)` and `PredictSiteScript(scriptName, scriptBody, maxDepth = 4)` return a `PredictedFlow` of `PredictedNode(Id, Kind, Lane, Title, InputDependent, Note)` + `PredictedEdge(From, To, CrossesBoundary, Async = false)`. -- **Syntax walk, not semantic.** A `CSharpSyntaxWalker` over `CSharpSyntaxTree.ParseText(code, SourceCodeKind.Script)` matches the script-API member-access chains by whitespace-stripped text, exactly as `ScriptTrustValidator`'s hardening walker does: `Route.To(x).Call(y)` / `.SetAttributes` / `.WaitForAttribute`, `Scripts.Call(y)`, `…Attributes.WriteBatchAndWaitAsync` / `WaitAsync` / `WaitForAsync` / `SetAsync`, `…Attributes[x] = v`, `ExternalSystem.Call/CachedCall`, `Database.Execute/Query/QuerySingle/CachedWrite`, `Notify.Send(list)` / `Notify.To(list).Send(...)` (`Async`-suffixed spellings map identically). Reads (`GetAttributes`, indexer reads) produce no node because the runtime records no trace row for them; nor does `Database.Connection(...)` — the number of commands on a raw handle is not statically knowable. +- **Syntax walk, not semantic.** A `CSharpSyntaxWalker` over `CSharpSyntaxTree.ParseText(code, SourceCodeKind.Script)` matches the script-API member-access chains by whitespace-stripped text, exactly as `ScriptTrustValidator`'s hardening walker does: `Route.To(x).Call(y)` / `.SetAttributes` / `.WaitForAttribute`, `Scripts.Call(y)`, `…Attributes.WriteBatchAndWaitAsync` (the reset-aware overload additionally yields a reset `AttributeWrite` + `AttributeWait` pair ahead of the batch) / `WaitAsync` / `WaitForAsync` / `SetAsync`, `…Attributes[x] = v`, `ExternalSystem.Call/CachedCall`, `Database.Execute/Query/QuerySingle/CachedWrite`, `Notify.Send(list)` / `Notify.To(list).Send(...)` (`Async`-suffixed spellings map identically). Reads (`GetAttributes`, indexer reads) produce no node because the runtime records no trace row for them; nor does `Database.Connection(...)` — the number of commands on a raw handle is not statically knowable. - **Outbound calls land on the External lane (2026-09-06, plan `docs/plans/2026-09-06-flow-debugger-outbound-calls.md` D8).** `ExternalSystem.*` → `ApiCall` titled `{system}.{method}`; `Database.*` → `DbCall` titled `{connection} · {member}`; `Notify` → `Notify` titled `Notify {list}`. A `CachedCall`, `CachedWrite` or `Notify` node carries `Note = "cached"` where applicable and its incoming edge is `Async`; an async edge does **not** advance the chain — the next node links from the previous non-async node — mirroring `FlowGraphBuilder.IsAsync` so a predicted async branch renders like a real one. - **Literals resolve; everything else is `InputDependent`** and titled with the expression's source text (`Route.To(code)`). Branching is not modelled: every recognised call is emitted once in source order, and an `if (…) return/throw` guard becomes a `Note = "early return: "` on the node that **follows** it (the condition is visited first, so a guard whose condition *is* the traced call — Delmia's `if (!await …WriteBatchAndWaitAsync(…)) return …` — attributes correctly). - **DB-free by contract.** Routed and nested script bodies come from the caller-supplied `resolveRouteScript(instanceCode, scriptName)` (Central UI resolves instance → template parent chain → script; called with the literal code when constant, else with the expression text so a resolver may still answer); recursion stops at `maxDepth` with a `depth limit` note.