diff --git a/docs/drivers/evidence/galaxy-h6-ack-2026-11-11.md b/docs/drivers/evidence/galaxy-h6-ack-2026-11-11.md new file mode 100644 index 000000000..8d418836d --- /dev/null +++ b/docs/drivers/evidence/galaxy-h6-ack-2026-11-11.md @@ -0,0 +1,170 @@ +# Galaxy H6 (#614): OPC UA ack of a real Galaxy native alarm to AVEVA history, 2026-11-11 + +Gitea #614 item H6: "an OPC UA Acknowledge of a REAL Galaxy native alarm reaches AVEVA event history". + +**Result: FAIL — the item cannot be ticked.** The OPC UA path worked end to end inside OtOpcUa: the +Part 9 `Acknowledge` returned Good, the client saw the condition go `Active, Acknowledged`, and the driver +sent the ack to mxaccessgw as the owner-provided Galaxy account with no error. **The acknowledgement +never reached Galaxy.** After the ack, Galaxy's own `.Acked` was still `False`, `.AckMsg` was still +empty, the gateway feed still showed the alarm as unacknowledged, and AVEVA event history has no ACK +row. So OtOpcUa now shows the condition acknowledged while Galaxy and the Historian do not (a +silent-divergence defect, below). + +Exactly **one** acknowledge was sent (approved write, 2026-11-11T17:37:55Z). It was not retried. No +remote command ran on any wonder box. Galaxy access was through the gateway's gRPC API and read-only +SQL only. + +## Target + +| | | +|---|---| +| Alarm | `Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert` (object `TableAlarms_008`, contained as `Right_004.TableAlarms`), priority 400, "Run Time Exceeded" | +| Gateway | `wonder-app-vd03.zmr.zimmer.com:5120` (h2c) | +| Galaxy security | OS/domain accounts (`ZB.dbo.user_profile`). Every 2026 ack in the Historian before this gate is a `NAM\…` operator from an InTouch node; none came through the gateway. | +| Historian | `wonder-sql-vd03` → `Runtime.dbo.v_AlarmEventHistory2`. Queries need an `EventStamp` range. There is no `A2ALMDB` or `WWALMDB`. | + +## Setup (docker-dev, temporary, all removed afterwards) + +| Item | Value | +|---|---| +| Node | SITE-B `site-b-1` (OPC UA `opc.tcp://localhost:4844`). Its partner `site-b-2` was stopped for the run so site-b-1 was the sole (Primary) node. | +| Driver | `h6gal`, DriverType `GalaxyMxGateway`, seeded straight into the ConfigDb. `Gateway.Endpoint` = the VD03 gateway, `UseTls=false`, `ApiKeySecretRef=env:GALAXY_MXGW_API_KEY`, `MxAccess.ClientName=OtOpcUa-H6-gate`, `Repository.WatchDeployEvents=false`. `AlarmAck.FallbackUser`/`FallbackDomain` = the owner-provided account (domain `NAM`); `FallbackPasswordSecretRef=env:OTOPCUA_GALAXY_ACKPW` | +| Device / tag | device `vd03` (`{}`), tag `RuntimeAlert`, Boolean, Read, `TagConfig` = `{"attributeRef":"Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert","alarm":{"alarmType":"AlarmCondition","severity":400,"historizeToAveva":false}}`. `historizeToAveva:false` keeps OtOpcUa's own alarm historian from writing anything to AVEVA. | +| Secrets | the gateway key and the ack password were injected into the site-b-1 container as runtime env only, through a scratch compose override that held `${VAR}` references, never values. Nothing secret went into the repo, the compose files, the ConfigDb or this file. | +| OPC UA client | `otopcua-cli` (`src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI`), UserName `apikey` + a temporary data-plane key (`h6-gate`, principal `h6gate`, 4 h expiry) with a SITE-B `Cluster` grant: Browse, Read, Subscribe, HistoryRead, AlarmRead and AlarmAcknowledge (399). LDAP was not used. | +| Deploy | `POST /api/deployments`. The first attempt used DriverType `Galaxy`, which is wrong: the registered name is `GalaxyMxGateway`. It was corrected and deployed again. Deployments to SITE-B end `TimedOut` while `site-b-2` is stopped (the same as the Heidenhain gate); `site-b-1` applied each one. | + +Driver start (site-b-1): `GalaxyMxSession connected — clientName=OtOpcUa-H6-gate`, `DriverInstance h6gal: connected`, +`native-alarm subscription established for 1 alarm ref(s)`. + +## Step 1: OPC UA, condition and Acknowledge (PASS within OtOpcUa) + +`alarms --refresh` before the ack: + +``` +[2026-07-29T01:02:46.8300000Z] ALARM h6gal/vd03/RuntimeAlert + Condition: RuntimeAlert + State: Active, Unacknowledged + Severity: 500 + Retain: True + ConditionNodeId: ns=2;s=h6gal/vd03/RuntimeAlert + EventId: 8DB71E793FBD7040993404E30808FADC +``` + +The single ack, at 17:37:55Z: + +``` +otopcua-cli ack -u opc.tcp://localhost:4844 -n "ns=2;s=h6gal/vd03/RuntimeAlert" -e 8DB71E793FBD7040993404E30808FADC -c "OtOpcUa #614 H6 gate" -U apikey -P +Acknowledge successful: ns=2;s=h6gal/vd03/RuntimeAlert +``` + +The live subscriber running during the ack saw: + +``` +[2026-11-11T17:37:55.8648690Z] ALARM h6gal/vd03/RuntimeAlert + State: Active, Acknowledged + Severity: 500 + Retain: True + EventId: 0F15D34C9D0B6D43B7A4B08F3A72D047 +``` + +Driver log (account name redacted): + +``` +[17:37:55 INF] Galaxy driver h6gal: acknowledging Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert as Galaxy user NAM\ (Fallback) for OPC UA principal h6gate (key fa1d95d325ab9ab5) +[17:37:55 INF] DriverInstance h6gal: acknowledged native condition h6gal/vd03/RuntimeAlert by h6gate +``` + +No warning, refusal or revert followed. So the gateway answered `PROTOCOL_STATUS_CODE_OK` with +`hresult` 0 (or absent): a non-OK status or a non-zero hresult throws and is logged. The comment the +driver sent was `OtOpcUa #614 H6 gate [OPC UA: h6gate, key fa1d95d325ab9ab5]`. + +## Step 2: Galaxy and the gateway alarm feed (FAIL) + +`QueryActiveAlarms` (prefix = the alarm reference) about 30 s after the ack, unchanged from before: + +``` +"currentState": "ALARM_CONDITION_STATE_ACTIVE", "severity": 400, +"lastTransitionTimestamp": "2026-07-29T01:02:46.830Z", "operatorComment": "Run Time Exceeded" +``` + +Read-only gateway session (`OpenSession` → `Register` → `ReadBulk` → `CloseSession`), at 17:39:24Z and again at 17:40:18Z: + +| Attribute | Value | Quality | +|---|---|---| +| `TableAlarms_008.RuntimeAlert.Acked` | **False** | 192 | +| `TableAlarms_008.RuntimeAlert.AckMsg` | **""** (empty) | 192 | +| `TableAlarms_008.RuntimeAlert.InAlarm` | True (since 2026-07-28T20:02:46.830Z) | 192 | + +On the windev rig, mxaccessgw's gateway-level verification (`AlarmProbeFindings.md`) showed a +successful ack turning `.Acked` True and leaving the comment in `.AckMsg`. Neither happened here. + +## Step 3: AVEVA event history (FAIL) + +`v_AlarmEventHistory2`, `EventStamp` 2026-11-11 12:00 to 23:59 local, queried at 17:38, 17:39 and 17:46 UTC. +The Historian is live: it stored other events at 17:00 UTC. Nothing for this alarm: + +``` +2026-11-11 12:00:30.922 | 17:00:30.922 | UNACK_ALM | DevWinPlatform.CPULoadHiAlarm | NULL | NULL +2026-11-11 12:00:50.918 | 17:00:50.918 | UNACK_RTN | DevWinPlatform.CPULoadHiAlarm | NULL | NULL +``` + +There is no `ACK_ALM` or `ACK_RTN` row for `TableAlarms_008.RuntimeAlert` and no row with `OperatorNode=MxGateway`. + +## Defect: OtOpcUa reports Acknowledged on an ack Galaxy never applied (reported, not fixed) + +mxaccessgw returns `OK`/`hresult 0` both for "write applied" and for "write sent, no `OnWriteComplete` +within the wait (unconfirmed, with a diagnostic)" (mxaccessgw `docs/Grpc.md`, `AcknowledgeAlarm` +outcome table). Its own docs say that "a client that must be certain should watch for the resulting +transition on `StreamAlarms`". + +`GatewayGalaxyAlarmAcknowledger.EvaluateReply` +(`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Runtime/GatewayGalaxyAlarmAcknowledger.cs`) treats OK + +hresult 0 as success. It never reads or logs `AcknowledgeAlarmReply.diagnostic_message`, and nothing +in the driver or host waits for the upstream Acknowledge transition. So the #760 revert never fires, +and the OPC UA condition stays Acknowledged while Galaxy holds `.Acked=False`. A ScadaBridge or +operator client would believe the alarm was handled. + +Why the gateway-side write did not take effect cannot be determined from here: the gateway's logs are +on wonder-app-vd03, and remote commands are barred. Candidates: + +- an unconfirmed `AckMsg` write that was later dropped or rejected; +- the account lacking acknowledge permission on the object's security group; +- a VD03 gateway build that predates the `AckMsg` path. + +Not attempted again (one-ack rule). Suggested follow-ups (not filed, no Gitea per brief): + +- **OtOpcUa:** log `diagnostic_message` on every ack reply. Treat an unconfirmed ack as pending until + the feed shows the Acknowledge transition, and revert on timeout. +- **mxaccessgw:** expose applied vs unconfirmed as a field rather than only in a diagnostic. +- **Owner:** check the VD03 gateway log around 2026-11-11T17:37:55Z for the ack's diagnostic. + +## Other observations (not filed) + +- **The gateway snapshot timestamp is 5 h off.** It reports `lastTransitionTimestamp + 2026-07-29T01:02:46.830Z`. MXAccess's own `InAlarm` timestamp and the Historian's `EventStampUTC` + both say 2026-07-28T20:02:46.830Z. Cross-repo, mxaccessgw. +- **The gateway snapshot and the Historian disagree on state.** The gateway says ACTIVE since July, + and `.InAlarm` agrees. The Historian logged five later "set off scan" `UNACK_RTN` rows (latest + 2026-10-22) with no new `UNACK_ALM`. +- **Severity differs between the three views.** The OPC UA condition shows Severity 500. The authored + `alarm.severity` was 400, and so is the Galaxy priority. +- **site-b-2 never acknowledged the cleanup deploys.** After cleanup, the standby `site-b-2` (restarted + and restored from its LocalDb at the same revision `e221b179…`) did not ack the re-dispatched + deployment, so the reseal deploy also ended `TimedOut` (site-b-1 Applied). +- **The rig needed a nudge after the OrbStack restart.** `otopcua-dev-sql-1` and + `otopcua-dev-traefik-1` stayed Exited. I started them and restarted central-1 and central-2 (#868 + ordering) before seeding. + +## Cleanup + +- Deleted from the ConfigDb: + - `Tag h6gal-rtalert`, `Device h6gal-vd03` and `DriverInstance h6gal`; + - `NodeAcl h6gate-acl` and `ApiKey h6-gate` (`fa1d95d325ab9ab5`); + - the two gate `Deployment` rows that carried the driver config, and their `NodeDeploymentState` rows, + so the account name is not left in an artifact blob. +- Cleanup deployed (`h6-gate-cleanup`, `h6-gate-cleanup-reseal`). site-b-1 logged `GalaxyDriver h6gal shutting down … disposed`. +- site-b-1 recreated from the plain compose file: the ack password is absent and the gateway key is back + to the committed default. site-b-2 restarted. +- The scratch key file was deleted. The shell env was unset. +- The site-b-1 LocalDb cache may still hold a gate-era artifact until its next replacement. diff --git a/docs/drivers/evidence/heidenhain-live-2026-11-11.md b/docs/drivers/evidence/heidenhain-live-2026-11-11.md new file mode 100644 index 000000000..646827dff --- /dev/null +++ b/docs/drivers/evidence/heidenhain-live-2026-11-11.md @@ -0,0 +1,195 @@ +# Heidenhain LSV2 live commissioning — 2026-11-11 (#829, #853) + +.NET commissioning evidence for the read-only Heidenhain LSV2 driver (`Driver.Heidenhain`) against four +**production** Hermle iTNC 530 mills, through a deployed OtOpcUa server and OPC UA clients. Owner-approved scope: +read-only LSV2 only (INSPECT login with an empty password, plus the driver's default optional DNC login attempt and +its allowlisted reads), no writes to the controls, no DATA login, no controller configuration change, one +concurrent session per control, poll intervals at or above the driver defaults. Outages were simulated only by +stopping a local TCP proxy. + +## Setup + +| Item | Value | +|---|---| +| Server | docker-dev rig, image `otopcua-host:dev` built 2026-11-11 15:45 UTC from master (newest Heidenhain commit at build time: `ddc1f29e`, #839) | +| Node | SITE-B, `site-b-1` (OPC UA `opc.tcp://localhost:4844`). Its pair partner `site-b-2` was **stopped** for the run so each control had exactly one session (a warm standby acquires too) | +| Driver | `hh829`, DriverType `Heidenhain`, seeded straight into the ConfigDb, deployed with `POST /api/deployments` | +| Devices | `Hermle59`, `Hermle49`, `Hermle70`, `Hermle29`, each `host.docker.internal:190xx` → a socat proxy on the Mac → `:19000` | +| Proxies | `socat -x TCP-LISTEN:,bind=127.0.0.1,fork,max-children=1 TCP::19000`; `max-children=1` caps upstream sessions at one, `-x` hex-dumps every byte in both directions (the wire census below is parsed from those dumps) | +| Tags | 46 per control (184 total): the 35 fixed leaves — Controller, Status, Program, Overrides, Axes, Tool, Alarms and 11 Capabilities (timers left off) — plus 11 `ALARM:` conditions (`ALARM:*`, `ALARM:OTHER`, `ALARM:CLASS=1..9`, severity 700). Tag `DataType`s follow `HeidenhainTagRules`; AccessLevel `Read`. Deploy reported 184 authored, 0 skipped | +| Config, phase A (steps 1–3) | `TimeoutMs` 5000; Status/Custom/Slow/Alarm **15000 ms**; MaxAlarms 64; Telemetry on; DNC login on (driver default); DATA off; timers off; reconnect 1000–30000 ms | +| Config, phase B (step 4) | the driver defaults: Status 1000, Custom 1000, Slow 10000, Alarm 2000 ms (about 3 minutes) | +| Clients | `otopcua-cli` (`src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI`) and python-opcua 0.98.13 (bulk typed reads and raw Write service calls) | + +## Controls + +Identity read through OPC UA; it matches the Java/Ignition evidence (`ignitionoee/docs/recipes/evidence/heidenhain-final-0.20.3-2026-11-03`). + +| Control | Endpoint | Model / NC / PLC | Options | +|---|---|---|---| +| Hermle59 | 10.201.29.226:19000 | iTNC530 / 340490 04 SP10 / iC30v7.1_1 | `%0000111100000111` | +| Hermle49 | 10.201.29.225:19000 | iTNC530 / 340490 03 SP10 / iC30V6.1_2SP3 | `%0000111100000111` | +| Hermle70 | 10.201.22.225:19000 | iTNC530 / 606420 03 SP4 / iC22_V20.1_2SP2 | `%0000111100000111` | +| Hermle29 | 10.201.22.215:19000 | iTNC530 / 606420 03 SP4 / iC22_V20.1_2SP2 | `%0000111100000111` | + +## Step 1 — browse and read through OPC UA + +The raw tree browses as `ns=2;s=hh829///`, and every condition carries its Part 9 children. +Every leaf advertises **AccessLevel = 1 and UserAccessLevel = 1** (CurrentRead only). Array leaves have ValueRank 1. + +| Leaf group | Hermle59 | Hermle49 | Hermle70 | Hermle29 | +|---|---|---|---|---| +| Controller/* (4) | Good | Good | Good | Good | +| Status/ExecutionState, ProgramState | Good (1, 7) | Good (4, 7) | Good (4, 0) | Good (4, 0) | +| Program/Main, Current, Line | Good (Line 0) | Good (Line 0) | Good, running (Line 42255 and counting) | Good, running (Line 40474 and counting) | +| Overrides/* (R_RI 25) | BadUserAccessDenied | BadUserAccessDenied | BadUserAccessDenied | BadUserAccessDenied | +| Axes/Names | `X Y Z C a A` | `X Y Z C a A` | `X Y Z C a A` | `X Y Z C a A` | +| Axes/Positions | Good, 6 values | Good, 6 values | Good, 6 values, moving | Good, 6 values, moving | +| Tool/* (R_RI 51) | Good (tool 0) | **BadNotSupported** (all 5) | Good (T49, L 138.1506, R −0.0101) | Good (T17, L 138.303, R −0.0101) | +| Alarms/* | Good, Count 0 | Good, Count 0 | Good, Count 0, then 1 (below) | Good, Count 1 | +| Capabilities | DncLogin `AccessDenied`, Overrides `AccessDenied`, DataLogin/PlcLayout `Disabled`, the rest `Supported` | same, and Tool `Unsupported` | as Hermle59 | as Hermle59 | + +Per control: 32 Good + 3 BadUserAccessDenied (Hermle49: 27 Good + 3 BadUserAccessDenied + 5 BadNotSupported). +This matches the LiveIntegration results and the Java evidence: R_RI 25 refused 23 (access denied) on all four, +R_RI 51 refused 24 (unsupported) on Hermle49, and the DNC login refused 23 on all four. INSPECT alone serves every +other selector. + +### Defect found — Tool Number/Index/AxisCode are published as Double + +`Tool/Number` (node DataType Int64) and `Tool/Index` / `Tool/AxisCode` (Int32) arrive at the client as a +**Double** variant (49.0, 0.0, 2.0) on every control that serves R_RI 51. Other integer leaves are correct +(`Program/Line` Int64, `Status/*` and `Alarms/Count` Int32). Cause: `HeidenhainSession.Leaf`, `case Lsv2Tool t`, +returns a switch expression whose arms are `long`, `int`, `int`, `double?`, `double?`. C# gives the expression the +type `double?`, so the integers are converted before boxing. The `Lsv2Program` arm next to it avoids this with an +`(object)` cast. `HeidenhainSessionTests.Tool_without_geometry_is_not_supported_not_zero` does not catch it: +Shouldly's `ShouldBe(7L)` treats a boxed `7.0` as equal. `otopcua-cli read` also hides it, because it prints `49`. +A strict client, or one that checks the variant type against the node's DataType, gets a type mismatch. Not fixed +here, per the run's read-only brief. + +## Step 2 — OPC UA write attempts are refused; no write frame leaves the driver + +| Attempt | Principal | Result | +|---|---|---| +| `otopcua-cli write` Controller/Model, Status/ExecutionState, Tool/Number × 4 controls | Anonymous and a temporary API key with all permissions on SITE-B | **BadNotWritable** (0x803B0000), all 24 | +| `otopcua-cli write` Overrides/Feed (all), Tool/Number (Hermle49) | both | the CLI's pre-write read fails (BadUserAccessDenied / BadNotSupported) and nothing is sent | +| Raw Write service (python): Overrides/Feed, Tool/Number, Axes/Positions, AlarmConditions/AnyAlarm/ActiveState × 4 controls | both | **BadNotWritable**, all 32 | +| Part 9 Acknowledge, Hermle29 AnyAlarm | API key | Good — applied **server-side only** (AckedState → Acknowledged) | +| Part 9 Acknowledge, Hermle29 Class7 | Anonymous | BadUserAccessDenied (AlarmAck gate) | + +Unlike the Java run, where the anonymous access check rejected every write first, a principal that **holds write +permission** was refused here by the node's access level (BadNotWritable). + +**Wire census.** The proxies' byte dumps, parsed frame by frame for the whole run (all sessions, both phases), +contain only `A_LG INSPECT`, `A_LG DNC` (each once per session, empty password), `R_VR`, `R_RI` 22/23/24/25/26/27/28/51. +There were **no** non-allowlisted commands and no `R_MB`, `R_DP` or `C_*` frames. In the write and acknowledge +window (17:00:45–17:02:20 UTC), each control received only its regular poll reads (`R_VR`, `R_RI` 22–28/51). +Client→control bytes for the whole run: Hermle59 12 876, Hermle49 12 746, Hermle70 13 396, Hermle29 15 526. + +## Step 3 — outage and recovery (Hermle59 proxy) + +An OPC UA read loop sampled once a second: every value leaf of all four controls (35 each), plus each control's +AnyAlarm condition Quality. + +| Event (UTC) | Hermle59 | Others | +|---|---|---| +| Before the stop | 32 Good + 3 BadUserAccessDenied | unchanged | +| 17:06:02.6 proxy stopped | — | — | +| 17:06:05 (next poll, +2.4 s) | **all 24 remote leaves BadNoCommunication**; the 11 Capabilities leaves read `CommunicationLost`; AnyAlarm Quality BadNoCommunication; log: `LSV2 communication lost; reconnecting with backoff`, `alarm list unavailable (CommunicationLost); its 11 alarm condition(s) keep their state` | unchanged, Good | +| 17:06:06–12 | reconnects back off (connection refused at 1, 2, 4… s) | unchanged | +| 17:06:15.8 proxy restarted | — | — | +| 17:06:20 reconnect (backoff) → 17:06:22 | **all Good again** (~6 s after the proxy came back); `alarm list readable again` | unchanged | + +Across 327 of 328 samples, Hermle49, Hermle70 and Hermle29 kept their exact status mix. The one exception was the +17:10:31 phase-B redeploy, which respawned the whole driver. The driver's health read `Degraded` with the connect +error while Hermle59 was down. Detection is bounded by the next poll of that control (Status interval + TimeoutMs). +A second, unplanned leg (below) also showed that a control unreachable at driver start retries and recovers: from +17:04:04 to 17:05:34 Hermle59's leaves were BadNoCommunication with `CommunicationLost` capabilities, and they were +all Good about 1 s after the reconnect at 17:05:35. + +## Step 4 — load at default intervals + +The proxy byte dumps give these per-control figures (RTT = request → reply). The channel is one transaction at a +time. + +| Phase | Control | Requests/s | Bytes/s out / in | RTT median / p95 / max (ms) | Channel busy | +|---|---|---|---|---|---| +| A (all 15 s), 180 s | Hermle59 | 0.53 | 5.2 / 16.4 | 50 / 133 / 175 | — (one ~0.5 s batch per 15 s) | +| | Hermle49 | 0.47 | 4.5 / 14.5 | 48 / 134 / 159 | | +| | Hermle70 | 0.53 | 5.2 / 18.5 | 47 / 135 / 170 | | +| | Hermle29 | 0.67 | 6.5 / 27.5 | 50 / 137 / 160 | (batch ≈ 0.67–0.83 s) | +| B (defaults 1/1/10/2 s), 180 s | Hermle59 | 4.78 | 47.6 / 148.5 | 50 / 133 / 396 | 30.3 % | +| | Hermle49 | 4.68 | 46.6 / 145.6 | 48 / 135 / 474 | 28.6 % | +| | Hermle70 | 4.78 | 47.6 / 176.3 | 47 / 132 / 244 | 28.1 % | +| | Hermle29 | 5.38 | 53.6 / 216.3 | 48 / 128 / 244 | 32.1 % | + +Request mix at the defaults over 180 s, per control: R_RI 22/23/24/26 179 each (the 1 s status group keeps up), +R_VR 18 and R_RI 51 18 (10 s slow group), and **R_RI 27 108**, which is 90 alarm walks (2 s) plus 18 slow-group +walks. R_RI 28 also ran 108 times on Hermle29, the control with an active alarm. R_RI 25, and R_RI 51 on Hermle49, +were sent **once per session** and then suppressed as documented. No control slowed down or refused anything new +under either cadence. + +Observation, not a defect: when a control carries `ALARM:` tags **and** the `Alarms/*` arrays, the alarm walk runs +twice — once per `AlarmMs` for the conditions and again in the Slow group for the arrays. The two are separate reads, +not shared. At the defaults that is about 0.1 extra R_RI 27 per second per control. + +## #853 — alarm class/group → Part 9 condition class + +Every alarm seen during the run (about 16 minutes of walks on all four controls). None were provoked. + +| Control | Wire record (`S_RI 27`) | Class | Group | Number (int32 / hex) | Text | Driver's condition class | +|---|---|---|---|---|---|---| +| Hermle29 | `0007 0003 8100017b …` | 7 Info | 3 Plc | −2130706053 / `0x8100017B` | `702126 Filter wait for spindle cooling (U02) E161` | `BaseConditionClass` (i=11163) | +| Hermle70 (raised 17:01:47, cleared before 17:13) | `0007 0003 8100015e …` | 7 Info | 3 Plc | −2130706082 / `0x8100015E` | `701550 Cabin unlocked! B301` | `BaseConditionClass` (i=11163) | +| Hermle59, Hermle49 | — (R_RI 27 → T_ER 57, empty list throughout) | | | | | | + +- Both conditions behaved as designed. `AnyAlarm` and `Class7` went Active, severity 700, Quality Good, with the + message ` (class 7, group 3)`. The `OtherAlarms` and other class conditions stayed Inactive. + Hermle70's alarm raised live, mid-run. +- **Consistent with the mapping table.** Group 3 (`Plc`) maps to Base. Both texts are machine-builder (Hermle) PLC + messages: a 70xxxx message number and a device tag (`E161`, `B301`). That fits the RemoTools `DNC_ERROR_GROUP` + numbering, where 3 = PLC. Class 7 = Info fits non-fatal status notices, and the one record captured earlier + (−1 "Run test of cutout channels", class 7 / group 4) is consistent too. The live data now corroborates + **class 7, group 3 and group 4**. +- **Still unobserved on hardware:** classes 1–6, 8 and 9 (including 5 EmergencyStop → Safety) and groups 1, 2, 5 + and 6 (the Process/System/Python rows). Confirming them would need an active alarm of those kinds, which this run + did not provoke. +- Note for authors: on these controls the alarm **number** field is not the displayed message number. It reads as + `0x81xxxxxx`, which is negative as int32, and the HMI number is the leading digits of the text. A `NUMBER=` selector + therefore has to use the negative raw value. This is documented ("filter on the raw codes"), but worth knowing. + +## Incidental findings (outside the Heidenhain driver) + +1. **Operator error during the run, now recovered.** When the first outage leg stopped the Hermle59 proxy, + `kill $(lsof -ti tcp:19059)` also matched and killed OrbStack's networking helper, which held the forwarded + connection. The OrbStack VM stopped; every container restarted when `orbctl start` was run (17:03–17:04 UTC), and + `otopcua-dev-sql-1` / `otopcua-dev-traefik-1` were started by hand. The outage leg was then re-run cleanly by + killing only the socat processes. **After this restart the OrbStack containers can no longer reach the LAN host + 10.100.0.35** (the driver fixtures and opc-plc). The Mac itself, internet access and the 10.201.x VPN still work. + Before 17:03 there were no such errors. This most likely needs OrbStack restarted from the app, or its macOS + Local Network permission re-granted. +2. **ConfigPublishCoordinator does not survive a ConfigDb that is unreachable at singleton start.** + `ConfigPublishCoordinator.PreStart` queries the DB synchronously. With SQL still starting, it threw + `ActorInitializationException` (SqlException: name not resolved), the singleton was terminated + (`Singleton actor [akka://otopcua/user/config-publish/config-publish] was terminated`) and it never restarted. + Every deploy then waited forever (`ClusterSingletonProxy failed to find … [config-publish]` every 30 s). Restarting + central-1 moved the singleton to central-2, which recovered it. This is a real resilience defect for any host that + boots before its ConfigDb. +3. The first deployment (`7a47a745`) ended `TimedOut` (5/6 acks) because `site-b-2` was stopped. The later deploys + set `ClusterNode.MaintenanceMode = 1` on `site-b-2` and sealed. The flag was reset to 0 afterwards. + +## Cleanup + +The `hh829` driver, its 4 devices, 36 tag groups and 184 tags were deleted, along with the temporary API key and +SITE-B NodeAcl. Deployment `6c7f859f` sealed at 17:14:13, and the driver was stopped and disposed on `site-b-1` at the +same moment. All four proxy sessions closed, and then the proxies were stopped. `site-b-2` was restarted (standby, +no Heidenhain driver), and `site-b-2` MaintenanceMode was restored to 0. Nothing on the controls was changed. + +## Status against the gates + +- **#829:** browse/read, write refusal (including a write-capable principal), outage and recovery with the other + controls unaffected, bounded traffic at default intervals, and retained evidence are **done** on four iTNC 530 + controls. **Open:** the Tool Number/Index/AxisCode Double-variant defect above, which should be fixed and + re-verified with a strict variant-type assertion. Still unproven as before: other control families + (TNC 320/430/640), successful `R_MB`, units, timers 30–34, and DATA (not exercised under this scope). +- **#853:** partially confirmed — class 7, groups 3 and 4 match the RemoTools numbering and map as documented + (Base). Classes other than 7 (notably 5 → Safety) and groups 1/2/5/6 are not yet observed on hardware. diff --git a/docs/drivers/evidence/twincat-z35474-testconnect-2026-11-11.md b/docs/drivers/evidence/twincat-z35474-testconnect-2026-11-11.md new file mode 100644 index 000000000..47c77b363 --- /dev/null +++ b/docs/drivers/evidence/twincat-z35474-testconnect-2026-11-11.md @@ -0,0 +1,91 @@ +# TwinCAT Test Connect on a real ADS target: Z35474, 2026-11-11 + +Gitea #755 item: "Driver probe (Test Connect) happy path on a real ADS target". + +**Result: the item cannot be ticked. On real hardware the probe returns a false RED.** The ADS +target is healthy and reachable through the router, but `TwinCATDriverProbe` never gets to its +ADS handshake: its TCP preflight dials the wrong endpoint. + +## Target and scope + +| | | +|---|---| +| Press | Z35474 (Elma cleaner), production | +| IP | 10.201.23.227 (reached over the VPN) | +| AmsNetId / port | `5.118.56.93.1.1` : 851 (TC3 PLC runtime 1) | +| Route used | the existing press-side route `otopcua-mac-20261111` → `10.181.201.5.1.1` (no AddRoute sent) | +| Scope | read-only: ReadState, ReadDeviceInfo, symbol upload (Flat), two scalar reads. No writes, no WriteControl, no notifications, no route-table change. One session at a time, a few seconds per run. | + +## Harness + +This is a scratch console app outside the repo. It has a `ProjectReference` to +`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT` and hosts `Beckhoff.TwinCAT.Ads.TcpRouter` 7.0.172 in +process. The router's local NetId is `10.181.201.5.1.1`, with one route `Z35474 → 10.201.23.227 / +5.118.56.93.1.1`. The package's sample `StaticRoutes.xml` is excluded, so the router loads no other +routes. The Beckhoff ADS client is 7.0.172, the same version the driver uses. + +The harness drives the probe the way AdminUI does: + +- `RawTreeService.LoadMergedProbeConfigAsync` builds the config through + `DriverDeviceConfigMerger.Merge("{}", [DeviceRow("Z35474", {"hostAddress":"ads://5.118.56.93.1.1:851"})], [])`. +- `AdminOperationsActor.ProbeAsync` calls `new TwinCATDriverProbe().ProbeAsync(merged, 10 s, cts(10 s))`. + The 10 s budget is `TestConnectBudget`'s TwinCAT default. + +The merged config the probe received: + +```json +{"hostAddress":"ads://5.118.56.93.1.1:851","Devices":[{"hostAddress":"ads://5.118.56.93.1.1:851","DeviceName":"Z35474"}],"RawTags":[]} +``` + +## Observed (two runs, identical) + +``` +[probe] Ok=False Message="Probe timed out after 10s." Latency=- wall=10003ms +[phase2] ReadState Succeeded=True Err=Succeeded AdsState=Run DeviceState=0 189ms (2nd run 109ms) +[phase2] DeviceInfo Succeeded=True Name="Plc30 App" Version=3.1.1940 +[browse] roots=3384 (Flat symbol upload) +[read] .actualOperationMode : INT = 2 +[read] .CFG_NUM_CDA_MAX : INT = 5 +``` + +`[phase2]` is not the probe. It reproduces the probe's own Phase 2 (`AdsClient.Connect(netId, 851)` + +`ReadStateAsync`, `Timeout` = 10 s) in the same process through the same router. It shows that the +handshake the probe would have done succeeds: Run state in about 0.1–0.2 s. + +Device: **"Plc30 App", ADS version 3.1.1940, AdsState Run, DeviceState 0.** The browse found 3,384 Flat +roots, which matches the 2026-10-08 count. + +## Defect: the Phase 1 TCP preflight targets `(NetId[0..4]):(AMS port)` + +`TwinCATDriverProbe.ExtractTarget` builds the preflight target as the first four octets of the AmsNetId +plus the AMS port. It then runs `Socket.ConnectAsync(host, port)` before it tries ADS. That target is +wrong in two separate ways: + +1. **The AmsNetId is not the IP.** For Z35474 the preflight dialled **`5.118.56.93:851`**. That is a + public internet address unrelated to the press; on this Mac it routes out of the LAN gateway, not the + VPN. It black-holes, so the probe spends its whole 10 s budget and returns "Probe timed out after + 10s." The press is at 10.201.23.227. The driver config has no IP field, so the probe cannot know it. + Only the AMS router knows it. +2. **851 is an AMS port, not a TCP port.** ADS/TCP is 48898. On a press where the NetId happens to equal + the IP (for example Z33856 `10.201.45.236.1.1`), the preflight would dial `10.201.45.236:801/851`. + That should be refused and return "Connect failed: ConnectionRefused", also a RED. I inferred this + from the code and did not run it against that press. + +Consequence: Test Connect cannot go green against a real TwinCAT runtime. It stops at Phase 1 before +the ADS handshake, which works. The unit tests (`TwinCATDriverProbeTests`) pin the current behaviour: +they open a loopback `TcpListener` on the "AMS port" and use `ads://127.0.0.1.1.1:{port}`, so the +mismatch never shows up there. The probe also sends a TCP SYN to whatever internet host the NetId's +first four octets name. + +The fix is not made here: tracked source was left unedited, as the task asked. The fix direction is +to drop the remote preflight, or point it at the local AMS router (`127.0.0.1:48898`), which is the +only TCP endpoint the ADS client actually uses. Phase 2's `ReadStateAsync`, bounded by the timeout, +then becomes the real reachability check. The unit tests would need to change with it. + +## #755 status + +- "Test Connect happy path on a real ADS target": **not tickable.** The ADS path works (router, route, + ReadState = Run, device info, browse, reads), but the probe as AdminUI runs it returns a false RED. + Re-run this gate after the preflight fix. +- The harness lives in session scratch, not in the repo: + `scratchpad/twincat/probeharness/` (`Program.cs`, modes `probe` | `read`).