diff --git a/archreview/plans/R3-03-restart-resubscribe-watchdog-plan.md b/archreview/plans/R3-03-restart-resubscribe-watchdog-plan.md index 10648423..171d0004 100644 --- a/archreview/plans/R3-03-restart-resubscribe-watchdog-plan.md +++ b/archreview/plans/R3-03-restart-resubscribe-watchdog-plan.md @@ -770,3 +770,92 @@ so "the second crash tripped supervision" is proven before the backoff is read. - T9 (live gate) still **pending** — the orchestrator runs it. Worth noting for that session: F1 is directly observable on the rig, by deploying a second alarm-bearing raw tag to a driver that already has one and confirming a transition on the NEW tag reaches `/alerts`. + +--- + +## Live gate result (2026-08-20) — **PASS** on docker-dev (before-fix control reproduced the silence first) + +Rig: `docker-dev/docker-compose.yml` on OrbStack (Mac), the six host containers plus SQL + Traefik, AdminUI on +`Security__Auth__DisableLogin=true` (auto-authenticated admin, no sign-in). Deploys via `./docker-dev/deploy.sh` +(`POST /api/deployments`, `X-Api-Key: docker-dev-deploy-key`, `202` every time). Modbus fixture +`otopcua-pymodbus-standard` on `10.100.0.35:5020` (left running throughout — R3-02's gate owns it). +Client.CLI `subscribe … -i 500` against `opc.tcp://localhost:4840` (central-1). **The seed's `HR200` is a static +scratch register, so a second raw tag `HR100` (`MAIN-modbus-hr100`, the fixture's auto-increment HR[100]) was +added by SQL and deployed — a subscription on a register that never changes cannot distinguish "resumed" from +"silent".** Both image swaps asserted parity on all six containers via `docker inspect --format '{{.Image}}'`. + +### After-fix run — image `otopcua-host:dev` = `b344565cad3d` (master `e280db37`, contains the five R3 fixes) ×6 + +**S16 leg (operator Restart → values resume): PASS.** + +- Baseline: `HR100` streaming at 1/s (`38592 → 38762`, `16:48:22Z → 16:49:48.02Z`). +- AdminUI `/hosts → pymodbus (MAIN-modbus) → Controls → Restart → Confirm restart`. Both centrals logged + `[16:49:48 INF] DriverHost central-N:4053: restarting driver MAIN-modbus by request of multi-role-test`, + then on each: `spawned Modbus driver MAIN-modbus` → `driver MAIN-modbus disconnected — 2 value node(s) marked + BadNoCommunication (value retained)` (the R3-02 mark) → `DriverInstance MAIN-modbus: connected` → + **`subscribed to 2 refs (poll-sub-1)`** — all in the same second. That last line is the T3 spawn-time desired-set + push; it is absent in the control below. +- CLI: last pre-restart value `38762 @ 16:49:48.0216Z`, **first post-restart value `38764 @ 16:49:48.3780Z`** — a + 0.36 s gap — then `38766, 38768, …` at 1/s with fresh server timestamps for the rest of the session (130 lines, + every one `0x00000000`). **Values RESUMED, not merely "driver reads Connected".** +- The R3-02 `BadNoCommunication` flash (brief's note): the mark and the reconnect landed inside one second, and at + the CLI's 500 ms sampling no Bad-status notification was delivered — the client saw only the Good stream with a + 0.36 s gap. Recorded as: no visible Bad flash at this cadence; the server-side mark is in the log. + +**S17 leg (watchdog grace after the restart, log evidence only — stop-the-fixture control deliberately NOT run, R3-02 +owns it): PASS as far as this leg can show.** + +- `grep -ciE "comms lost|DisconnectObserved|Reconnecting"` on both centrals over the window `16:49:48 → 16:53:42` + (> 2 × `DefaultCommsLossTimeout` after the reconnect): **0 and 0**. Health snapshots over the same window + (`DriverHealthChanged … State = Healthy, LastSuccessfulReadUtc = 16:50:47 / 16:51:17`) — reads were succeeding, + so no trip was expected and none happened. This proves the watchdog did not re-trip on a freshly-reconnected + driver; it does **not** exercise the `??`→max change itself (that needs a non-Healthy driver post-reconnect — the + churn/positive-control pair stays owned by R3-02's fixture-stop gate, and by the T1 unit pair). + +**F1 leg (a widened alarm set re-subscribes): PASS, on the establishment line; the transition-to-`/alerts` half could +not be exercised on this rig.** + +- The rig has no native-alarm-capable driver (Modbus is not an `IAlarmSource`; the `IAlarmSource` drivers are + OpcUaClient / AbCip / FOCAS / Galaxy). An **OpcUaClient** driver `MAIN-opcplc` was authored by SQL against the + shared opc-plc fixture `opc.tcp://10.100.0.35:50000` (`TargetNamespaceKind=SystemPlatform`, `AutoAcceptCertificates`) + with one alarm-bearing raw tag (`TagConfig.alarm = {alarmType, severity}`), deployed (`c7267b77…`, Sealed): + `[16:47:46 INF] DriverInstance MAIN-opcplc: native-alarm subscription established for 1 alarm ref(s) (opcua-alarm-sub-1)`. +- Second alarm-bearing tag added, deployed (`72a18d67…`, Sealed 16:53:50.513): both centrals + `[16:53:50 INF] DriverHost: stopped driver child MAIN-opcplc` → `spawned` → `connected` → + **`native-alarm subscription established for 2 alarm ref(s) (opcua-alarm-sub-1)`** — the new set, in one connect + cycle. Before F1 the second deploy would have left the handle on the 1-ref set. +- The alarm-bearing delta forced a full rebuild, yet the Modbus `HR100` CLI stream ran through it unbroken + (`16:53:49.47Z → 16:53:50.50Z → …`, no gap, no Bad). +- **Not shown:** a transition on the new tag reaching `/alerts`. The OpcUaClient driver's event handler filters on + `SourceNode` against the ref set it was handed, and opc-plc's `--alm` simulation raises from its own + `DeviceHealthAlarms` objects, so no upstream event matches an authored tag — nothing reached `/alerts` for either + tag, before or after the widening. That is a fixture/driver-filter mismatch, not an R3-03 regression; the + re-subscribe itself (F1's defect) is what the 1→2 line evidences. `A_deploy_that_widens_the_alarm_set_re_subscribes_the_new_refs` + covers the handle semantics offline. + +**Supervision leg:** not provokable from an operator surface — **not live-gated, by design**; covered by +`Supervision_restart_reinitialises_and_re_subscribes_the_child` and the F2/F3 tests in `DriverHostActorRestartTests` +(10 cases, green at merge). + +### Before-fix control — image `otopcua-host:prefix-r303` = `32a2dcee0285`, built from a scratch worktree at `7c77a8c7` (pre-tranche master) ×6 — **FAILED as required** + +Tagged over `otopcua-host:dev`, all six containers recreated, parity `32a2dcee0285` ×6. `site-b-2` was put in +`ClusterNode.MaintenanceMode=1` for the control only (the R3-04 standing pin makes a pre-R3-04 build mount no driver +stack on that node); the centrals replayed the last applied deployment (`72a18d67…`) from SQL on bootstrap +(`recovered Applied state at rev 57251f4b…`, `SubscribeBulk pushed 2 references`), so no new deploy was needed. + +- Baseline on the control: `HR100` streaming (`39474 → 39596`, `16:56:11Z → 16:57:12.35Z`). +- Same `/hosts` Restart at `16:57:12`. Both centrals: `restarting driver MAIN-modbus by request of multi-role-test` → + `spawned Modbus driver MAIN-modbus (stub=False)` → `DriverInstance MAIN-modbus: connected` — and **nothing else**: + no `subscribed to N refs`, no `SubscribeBulk` (count of both over the following 3 min: **0**). +- CLI: **last value `39596 @ 16:57:12.3468Z`, then silence for the remaining 155 s of the session** (checked at + 16:59:47) — no value, no Bad status (the pre-R3-02 build has no `BadNoCommunication` mark either, so the client held + a stale Good forever — the exact 03/S15+S16 shape). `/hosts` kept showing the driver Healthy throughout. + +### Restore + +`docker tag b344565cad3d otopcua-host:dev`, all six recreated → parity `b344565cad3d` ×6; `MaintenanceMode` cleared +on `site-b-2:4053`; the scratch worktree removed; the `otopcua-host:prefix-r303` image tag left behind for re-runs. +The gate's `MAIN-opcplc` driver + its two alarm tags were deleted again and a final deploy (`1c666be3…`, Sealed) +returned the rig to the seed plus the one extra `HR100` raw tag (kept — it is the only changing Modbus register and +the next gate will want it). Rig left UP on the master image with `DisableLogin=true`. diff --git a/archreview/plans/R3-03-restart-resubscribe-watchdog-plan.md.tasks.json b/archreview/plans/R3-03-restart-resubscribe-watchdog-plan.md.tasks.json index 5e1d2dab..605dded6 100644 --- a/archreview/plans/R3-03-restart-resubscribe-watchdog-plan.md.tasks.json +++ b/archreview/plans/R3-03-restart-resubscribe-watchdog-plan.md.tasks.json @@ -63,7 +63,7 @@ { "id": "T9", "subject": "LIVE GATE: docker-dev restart re-subscribe (before/after) + watchdog churn cadence with stop-the-fixture control", - "status": "pending", + "status": "done", "blockedBy": [ "T3", "T5",