docs(evidence): 2026-11-11 live gates — Heidenhain iTNC 530 (#829/#853), TwinCAT Z35474 Test Connect (#755/#863), Galaxy H6 ack (#614/#869)

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk

docs/drivers/evidence/galaxy-h6-ack-2026-11-11.md

@@ -0,0 +1,170 @@
+# Galaxy H6 (#614): OPC UA ack of a real Galaxy native alarm to AVEVA history, 2026-11-11
+
+Gitea #614 item H6: "an OPC UA Acknowledge of a REAL Galaxy native alarm reaches AVEVA event history".
+
+**Result: FAIL — the item cannot be ticked.** The OPC UA path worked end to end inside OtOpcUa: the
+Part 9 `Acknowledge` returned Good, the client saw the condition go `Active, Acknowledged`, and the driver
+sent the ack to mxaccessgw as the owner-provided Galaxy account with no error. **The acknowledgement
+never reached Galaxy.** After the ack, Galaxy's own `.Acked` was still `False`, `.AckMsg` was still
+empty, the gateway feed still showed the alarm as unacknowledged, and AVEVA event history has no ACK
+row. So OtOpcUa now shows the condition acknowledged while Galaxy and the Historian do not (a
+silent-divergence defect, below).
+
+Exactly **one** acknowledge was sent (approved write, 2026-11-11T17:37:55Z). It was not retried. No
+remote command ran on any wonder box. Galaxy access was through the gateway's gRPC API and read-only
+SQL only.
+
+## Target
+
+| | |
+|---|---|
+| Alarm | `Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert` (object `TableAlarms_008`, contained as `Right_004.TableAlarms`), priority 400, "Run Time Exceeded" |
+| Gateway | `wonder-app-vd03.zmr.zimmer.com:5120` (h2c) |
+| Galaxy security | OS/domain accounts (`ZB.dbo.user_profile`). Every 2026 ack in the Historian before this gate is a `NAM\…` operator from an InTouch node; none came through the gateway. |
+| Historian | `wonder-sql-vd03` → `Runtime.dbo.v_AlarmEventHistory2`. Queries need an `EventStamp` range. There is no `A2ALMDB` or `WWALMDB`. |
+
+## Setup (docker-dev, temporary, all removed afterwards)
+
+| Item | Value |
+|---|---|
+| Node | SITE-B `site-b-1` (OPC UA `opc.tcp://localhost:4844`). Its partner `site-b-2` was stopped for the run so site-b-1 was the sole (Primary) node. |
+| Driver | `h6gal`, DriverType `GalaxyMxGateway`, seeded straight into the ConfigDb. `Gateway.Endpoint` = the VD03 gateway, `UseTls=false`, `ApiKeySecretRef=env:GALAXY_MXGW_API_KEY`, `MxAccess.ClientName=OtOpcUa-H6-gate`, `Repository.WatchDeployEvents=false`. `AlarmAck.FallbackUser`/`FallbackDomain` = the owner-provided account (domain `NAM`); `FallbackPasswordSecretRef=env:OTOPCUA_GALAXY_ACKPW` |
+| Device / tag | device `vd03` (`{}`), tag `RuntimeAlert`, Boolean, Read, `TagConfig` = `{"attributeRef":"Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert","alarm":{"alarmType":"AlarmCondition","severity":400,"historizeToAveva":false}}`. `historizeToAveva:false` keeps OtOpcUa's own alarm historian from writing anything to AVEVA. |
+| Secrets | the gateway key and the ack password were injected into the site-b-1 container as runtime env only, through a scratch compose override that held `${VAR}` references, never values. Nothing secret went into the repo, the compose files, the ConfigDb or this file. |
+| OPC UA client | `otopcua-cli` (`src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI`), UserName `apikey` + a temporary data-plane key (`h6-gate`, principal `h6gate`, 4 h expiry) with a SITE-B `Cluster` grant: Browse, Read, Subscribe, HistoryRead, AlarmRead and AlarmAcknowledge (399). LDAP was not used. |
+| Deploy | `POST /api/deployments`. The first attempt used DriverType `Galaxy`, which is wrong: the registered name is `GalaxyMxGateway`. It was corrected and deployed again. Deployments to SITE-B end `TimedOut` while `site-b-2` is stopped (the same as the Heidenhain gate); `site-b-1` applied each one. |
+
+Driver start (site-b-1): `GalaxyMxSession connected — clientName=OtOpcUa-H6-gate`, `DriverInstance h6gal: connected`,
+`native-alarm subscription established for 1 alarm ref(s)`.
+
+## Step 1: OPC UA, condition and Acknowledge (PASS within OtOpcUa)
+
+`alarms --refresh` before the ack:
+
+```
+[2026-07-29T01:02:46.8300000Z] ALARM  h6gal/vd03/RuntimeAlert
+  Condition: RuntimeAlert
+  State:     Active, Unacknowledged
+  Severity:  500
+  Retain:    True
+  ConditionNodeId: ns=2;s=h6gal/vd03/RuntimeAlert
+  EventId:   8DB71E793FBD7040993404E30808FADC
+```
+
+The single ack, at 17:37:55Z:
+
+```
+otopcua-cli ack -u opc.tcp://localhost:4844 -n "ns=2;s=h6gal/vd03/RuntimeAlert" -e 8DB71E793FBD7040993404E30808FADC -c "OtOpcUa #614 H6 gate" -U apikey -P <temp key>
+Acknowledge successful: ns=2;s=h6gal/vd03/RuntimeAlert
+```
+
+The live subscriber running during the ack saw:
+
+```
+[2026-11-11T17:37:55.8648690Z] ALARM  h6gal/vd03/RuntimeAlert
+  State:     Active, Acknowledged
+  Severity:  500
+  Retain:    True
+  EventId:   0F15D34C9D0B6D43B7A4B08F3A72D047
+```
+
+Driver log (account name redacted):
+
+```
+[17:37:55 INF] Galaxy driver h6gal: acknowledging Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert as Galaxy user NAM\<ack account> (Fallback) for OPC UA principal h6gate (key fa1d95d325ab9ab5)
+[17:37:55 INF] DriverInstance h6gal: acknowledged native condition h6gal/vd03/RuntimeAlert by h6gate
+```
+
+No warning, refusal or revert followed. So the gateway answered `PROTOCOL_STATUS_CODE_OK` with
+`hresult` 0 (or absent): a non-OK status or a non-zero hresult throws and is logged. The comment the
+driver sent was `OtOpcUa #614 H6 gate [OPC UA: h6gate, key fa1d95d325ab9ab5]`.
+
+## Step 2: Galaxy and the gateway alarm feed (FAIL)
+
+`QueryActiveAlarms` (prefix = the alarm reference) about 30 s after the ack, unchanged from before:
+
+```
+"currentState": "ALARM_CONDITION_STATE_ACTIVE", "severity": 400,
+"lastTransitionTimestamp": "2026-07-29T01:02:46.830Z", "operatorComment": "Run Time Exceeded"
+```
+
+Read-only gateway session (`OpenSession` → `Register` → `ReadBulk` → `CloseSession`), at 17:39:24Z and again at 17:40:18Z:
+
+| Attribute | Value | Quality |
+|---|---|---|
+| `TableAlarms_008.RuntimeAlert.Acked` | **False** | 192 |
+| `TableAlarms_008.RuntimeAlert.AckMsg` | **""** (empty) | 192 |
+| `TableAlarms_008.RuntimeAlert.InAlarm` | True (since 2026-07-28T20:02:46.830Z) | 192 |
+
+On the windev rig, mxaccessgw's gateway-level verification (`AlarmProbeFindings.md`) showed a
+successful ack turning `.Acked` True and leaving the comment in `.AckMsg`. Neither happened here.
+
+## Step 3: AVEVA event history (FAIL)
+
+`v_AlarmEventHistory2`, `EventStamp` 2026-11-11 12:00 to 23:59 local, queried at 17:38, 17:39 and 17:46 UTC.
+The Historian is live: it stored other events at 17:00 UTC. Nothing for this alarm:
+
+```
+2026-11-11 12:00:30.922 | 17:00:30.922 | UNACK_ALM | DevWinPlatform.CPULoadHiAlarm | NULL | NULL
+2026-11-11 12:00:50.918 | 17:00:50.918 | UNACK_RTN | DevWinPlatform.CPULoadHiAlarm | NULL | NULL
+```
+
+There is no `ACK_ALM` or `ACK_RTN` row for `TableAlarms_008.RuntimeAlert` and no row with `OperatorNode=MxGateway`.
+
+## Defect: OtOpcUa reports Acknowledged on an ack Galaxy never applied (reported, not fixed)
+
+mxaccessgw returns `OK`/`hresult 0` both for "write applied" and for "write sent, no `OnWriteComplete`
+within the wait (unconfirmed, with a diagnostic)" (mxaccessgw `docs/Grpc.md`, `AcknowledgeAlarm`
+outcome table). Its own docs say that "a client that must be certain should watch for the resulting
+transition on `StreamAlarms`".
+
+`GatewayGalaxyAlarmAcknowledger.EvaluateReply`
+(`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Runtime/GatewayGalaxyAlarmAcknowledger.cs`) treats OK +
+hresult 0 as success. It never reads or logs `AcknowledgeAlarmReply.diagnostic_message`, and nothing
+in the driver or host waits for the upstream Acknowledge transition. So the #760 revert never fires,
+and the OPC UA condition stays Acknowledged while Galaxy holds `.Acked=False`. A ScadaBridge or
+operator client would believe the alarm was handled.
+
+Why the gateway-side write did not take effect cannot be determined from here: the gateway's logs are
+on wonder-app-vd03, and remote commands are barred. Candidates:
+
+- an unconfirmed `AckMsg` write that was later dropped or rejected;
+- the account lacking acknowledge permission on the object's security group;
+- a VD03 gateway build that predates the `AckMsg` path.
+
+Not attempted again (one-ack rule). Suggested follow-ups (not filed, no Gitea per brief):
+
+- **OtOpcUa:** log `diagnostic_message` on every ack reply. Treat an unconfirmed ack as pending until
+  the feed shows the Acknowledge transition, and revert on timeout.
+- **mxaccessgw:** expose applied vs unconfirmed as a field rather than only in a diagnostic.
+- **Owner:** check the VD03 gateway log around 2026-11-11T17:37:55Z for the ack's diagnostic.
+
+## Other observations (not filed)
+
+- **The gateway snapshot timestamp is 5 h off.** It reports `lastTransitionTimestamp
+  2026-07-29T01:02:46.830Z`. MXAccess's own `InAlarm` timestamp and the Historian's `EventStampUTC`
+  both say 2026-07-28T20:02:46.830Z. Cross-repo, mxaccessgw.
+- **The gateway snapshot and the Historian disagree on state.** The gateway says ACTIVE since July,
+  and `.InAlarm` agrees. The Historian logged five later "set off scan" `UNACK_RTN` rows (latest
+  2026-10-22) with no new `UNACK_ALM`.
+- **Severity differs between the three views.** The OPC UA condition shows Severity 500. The authored
+  `alarm.severity` was 400, and so is the Galaxy priority.
+- **site-b-2 never acknowledged the cleanup deploys.** After cleanup, the standby `site-b-2` (restarted
+  and restored from its LocalDb at the same revision `e221b179…`) did not ack the re-dispatched
+  deployment, so the reseal deploy also ended `TimedOut` (site-b-1 Applied).
+- **The rig needed a nudge after the OrbStack restart.** `otopcua-dev-sql-1` and
+  `otopcua-dev-traefik-1` stayed Exited. I started them and restarted central-1 and central-2 (#868
+  ordering) before seeding.
+
+## Cleanup
+
+- Deleted from the ConfigDb:
+  - `Tag h6gal-rtalert`, `Device h6gal-vd03` and `DriverInstance h6gal`;
+  - `NodeAcl h6gate-acl` and `ApiKey h6-gate` (`fa1d95d325ab9ab5`);
+  - the two gate `Deployment` rows that carried the driver config, and their `NodeDeploymentState` rows,
+    so the account name is not left in an artifact blob.
+- Cleanup deployed (`h6-gate-cleanup`, `h6-gate-cleanup-reseal`). site-b-1 logged `GalaxyDriver h6gal shutting down … disposed`.
+- site-b-1 recreated from the plain compose file: the ack password is absent and the gateway key is back
+  to the committed default. site-b-2 restarted.
+- The scratch key file was deleted. The shell env was unset.
+- The site-b-1 LocalDb cache may still hold a gate-era artifact until its next replacement.

docs/drivers/evidence/heidenhain-live-2026-11-11.md

@@ -0,0 +1,195 @@
+# Heidenhain LSV2 live commissioning — 2026-11-11 (#829, #853)
+
+.NET commissioning evidence for the read-only Heidenhain LSV2 driver (`Driver.Heidenhain`) against four
+**production** Hermle iTNC 530 mills, through a deployed OtOpcUa server and OPC UA clients. Owner-approved scope:
+read-only LSV2 only (INSPECT login with an empty password, plus the driver's default optional DNC login attempt and
+its allowlisted reads), no writes to the controls, no DATA login, no controller configuration change, one
+concurrent session per control, poll intervals at or above the driver defaults. Outages were simulated only by
+stopping a local TCP proxy.
+
+## Setup
+
+| Item | Value |
+|---|---|
+| Server | docker-dev rig, image `otopcua-host:dev` built 2026-11-11 15:45 UTC from master (newest Heidenhain commit at build time: `ddc1f29e`, #839) |
+| Node | SITE-B, `site-b-1` (OPC UA `opc.tcp://localhost:4844`). Its pair partner `site-b-2` was **stopped** for the run so each control had exactly one session (a warm standby acquires too) |
+| Driver | `hh829`, DriverType `Heidenhain`, seeded straight into the ConfigDb, deployed with `POST /api/deployments` |
+| Devices | `Hermle59`, `Hermle49`, `Hermle70`, `Hermle29`, each `host.docker.internal:190xx` → a socat proxy on the Mac → `<control>:19000` |
+| Proxies | `socat -x TCP-LISTEN:<port>,bind=127.0.0.1,fork,max-children=1 TCP:<control>:19000`; `max-children=1` caps upstream sessions at one, `-x` hex-dumps every byte in both directions (the wire census below is parsed from those dumps) |
+| Tags | 46 per control (184 total): the 35 fixed leaves — Controller, Status, Program, Overrides, Axes, Tool, Alarms and 11 Capabilities (timers left off) — plus 11 `ALARM:` conditions (`ALARM:*`, `ALARM:OTHER`, `ALARM:CLASS=1..9`, severity 700). Tag `DataType`s follow `HeidenhainTagRules`; AccessLevel `Read`. Deploy reported 184 authored, 0 skipped |
+| Config, phase A (steps 1–3) | `TimeoutMs` 5000; Status/Custom/Slow/Alarm **15000 ms**; MaxAlarms 64; Telemetry on; DNC login on (driver default); DATA off; timers off; reconnect 1000–30000 ms |
+| Config, phase B (step 4) | the driver defaults: Status 1000, Custom 1000, Slow 10000, Alarm 2000 ms (about 3 minutes) |
+| Clients | `otopcua-cli` (`src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI`) and python-opcua 0.98.13 (bulk typed reads and raw Write service calls) |
+
+## Controls
+
+Identity read through OPC UA; it matches the Java/Ignition evidence (`ignitionoee/docs/recipes/evidence/heidenhain-final-0.20.3-2026-11-03`).
+
+| Control | Endpoint | Model / NC / PLC | Options |
+|---|---|---|---|
+| Hermle59 | 10.201.29.226:19000 | iTNC530 / 340490 04 SP10 / iC30v7.1_1 | `%0000111100000111` |
+| Hermle49 | 10.201.29.225:19000 | iTNC530 / 340490 03 SP10 / iC30V6.1_2SP3 | `%0000111100000111` |
+| Hermle70 | 10.201.22.225:19000 | iTNC530 / 606420 03 SP4 / iC22_V20.1_2SP2 | `%0000111100000111` |
+| Hermle29 | 10.201.22.215:19000 | iTNC530 / 606420 03 SP4 / iC22_V20.1_2SP2 | `%0000111100000111` |
+
+## Step 1 — browse and read through OPC UA
+
+The raw tree browses as `ns=2;s=hh829/<Control>/<Folder>/<Leaf>`, and every condition carries its Part 9 children.
+Every leaf advertises **AccessLevel = 1 and UserAccessLevel = 1** (CurrentRead only). Array leaves have ValueRank 1.
+
+| Leaf group | Hermle59 | Hermle49 | Hermle70 | Hermle29 |
+|---|---|---|---|---|
+| Controller/* (4) | Good | Good | Good | Good |
+| Status/ExecutionState, ProgramState | Good (1, 7) | Good (4, 7) | Good (4, 0) | Good (4, 0) |
+| Program/Main, Current, Line | Good (Line 0) | Good (Line 0) | Good, running (Line 42255 and counting) | Good, running (Line 40474 and counting) |
+| Overrides/* (R_RI 25) | BadUserAccessDenied | BadUserAccessDenied | BadUserAccessDenied | BadUserAccessDenied |
+| Axes/Names | `X Y Z C a A` | `X Y Z C a A` | `X Y Z C a A` | `X Y Z C a A` |
+| Axes/Positions | Good, 6 values | Good, 6 values | Good, 6 values, moving | Good, 6 values, moving |
+| Tool/* (R_RI 51) | Good (tool 0) | **BadNotSupported** (all 5) | Good (T49, L 138.1506, R −0.0101) | Good (T17, L 138.303, R −0.0101) |
+| Alarms/* | Good, Count 0 | Good, Count 0 | Good, Count 0, then 1 (below) | Good, Count 1 |
+| Capabilities | DncLogin `AccessDenied`, Overrides `AccessDenied`, DataLogin/PlcLayout `Disabled`, the rest `Supported` | same, and Tool `Unsupported` | as Hermle59 | as Hermle59 |
+
+Per control: 32 Good + 3 BadUserAccessDenied (Hermle49: 27 Good + 3 BadUserAccessDenied + 5 BadNotSupported).
+This matches the LiveIntegration results and the Java evidence: R_RI 25 refused 23 (access denied) on all four,
+R_RI 51 refused 24 (unsupported) on Hermle49, and the DNC login refused 23 on all four. INSPECT alone serves every
+other selector.
+
+### Defect found — Tool Number/Index/AxisCode are published as Double
+
+`Tool/Number` (node DataType Int64) and `Tool/Index` / `Tool/AxisCode` (Int32) arrive at the client as a
+**Double** variant (49.0, 0.0, 2.0) on every control that serves R_RI 51. Other integer leaves are correct
+(`Program/Line` Int64, `Status/*` and `Alarms/Count` Int32). Cause: `HeidenhainSession.Leaf`, `case Lsv2Tool t`,
+returns a switch expression whose arms are `long`, `int`, `int`, `double?`, `double?`. C# gives the expression the
+type `double?`, so the integers are converted before boxing. The `Lsv2Program` arm next to it avoids this with an
+`(object)` cast. `HeidenhainSessionTests.Tool_without_geometry_is_not_supported_not_zero` does not catch it:
+Shouldly's `ShouldBe(7L)` treats a boxed `7.0` as equal. `otopcua-cli read` also hides it, because it prints `49`.
+A strict client, or one that checks the variant type against the node's DataType, gets a type mismatch. Not fixed
+here, per the run's read-only brief.
+
+## Step 2 — OPC UA write attempts are refused; no write frame leaves the driver
+
+| Attempt | Principal | Result |
+|---|---|---|
+| `otopcua-cli write` Controller/Model, Status/ExecutionState, Tool/Number × 4 controls | Anonymous and a temporary API key with all permissions on SITE-B | **BadNotWritable** (0x803B0000), all 24 |
+| `otopcua-cli write` Overrides/Feed (all), Tool/Number (Hermle49) | both | the CLI's pre-write read fails (BadUserAccessDenied / BadNotSupported) and nothing is sent |
+| Raw Write service (python): Overrides/Feed, Tool/Number, Axes/Positions, AlarmConditions/AnyAlarm/ActiveState × 4 controls | both | **BadNotWritable**, all 32 |
+| Part 9 Acknowledge, Hermle29 AnyAlarm | API key | Good — applied **server-side only** (AckedState → Acknowledged) |
+| Part 9 Acknowledge, Hermle29 Class7 | Anonymous | BadUserAccessDenied (AlarmAck gate) |
+
+Unlike the Java run, where the anonymous access check rejected every write first, a principal that **holds write
+permission** was refused here by the node's access level (BadNotWritable).
+
+**Wire census.** The proxies' byte dumps, parsed frame by frame for the whole run (all sessions, both phases),
+contain only `A_LG INSPECT`, `A_LG DNC` (each once per session, empty password), `R_VR`, `R_RI` 22/23/24/25/26/27/28/51.
+There were **no** non-allowlisted commands and no `R_MB`, `R_DP` or `C_*` frames. In the write and acknowledge
+window (17:00:45–17:02:20 UTC), each control received only its regular poll reads (`R_VR`, `R_RI` 22–28/51).
+Client→control bytes for the whole run: Hermle59 12 876, Hermle49 12 746, Hermle70 13 396, Hermle29 15 526.
+
+## Step 3 — outage and recovery (Hermle59 proxy)
+
+An OPC UA read loop sampled once a second: every value leaf of all four controls (35 each), plus each control's
+AnyAlarm condition Quality.
+
+| Event (UTC) | Hermle59 | Others |
+|---|---|---|
+| Before the stop | 32 Good + 3 BadUserAccessDenied | unchanged |
+| 17:06:02.6 proxy stopped | — | — |
+| 17:06:05 (next poll, +2.4 s) | **all 24 remote leaves BadNoCommunication**; the 11 Capabilities leaves read `CommunicationLost`; AnyAlarm Quality BadNoCommunication; log: `LSV2 communication lost; reconnecting with backoff`, `alarm list unavailable (CommunicationLost); its 11 alarm condition(s) keep their state` | unchanged, Good |
+| 17:06:06–12 | reconnects back off (connection refused at 1, 2, 4… s) | unchanged |
+| 17:06:15.8 proxy restarted | — | — |
+| 17:06:20 reconnect (backoff) → 17:06:22 | **all Good again** (~6 s after the proxy came back); `alarm list readable again` | unchanged |
+
+Across 327 of 328 samples, Hermle49, Hermle70 and Hermle29 kept their exact status mix. The one exception was the
+17:10:31 phase-B redeploy, which respawned the whole driver. The driver's health read `Degraded` with the connect
+error while Hermle59 was down. Detection is bounded by the next poll of that control (Status interval + TimeoutMs).
+A second, unplanned leg (below) also showed that a control unreachable at driver start retries and recovers: from
+17:04:04 to 17:05:34 Hermle59's leaves were BadNoCommunication with `CommunicationLost` capabilities, and they were
+all Good about 1 s after the reconnect at 17:05:35.
+
+## Step 4 — load at default intervals
+
+The proxy byte dumps give these per-control figures (RTT = request → reply). The channel is one transaction at a
+time.
+
+| Phase | Control | Requests/s | Bytes/s out / in | RTT median / p95 / max (ms) | Channel busy |
+|---|---|---|---|---|---|
+| A (all 15 s), 180 s | Hermle59 | 0.53 | 5.2 / 16.4 | 50 / 133 / 175 | — (one ~0.5 s batch per 15 s) |
+| | Hermle49 | 0.47 | 4.5 / 14.5 | 48 / 134 / 159 | |
+| | Hermle70 | 0.53 | 5.2 / 18.5 | 47 / 135 / 170 | |
+| | Hermle29 | 0.67 | 6.5 / 27.5 | 50 / 137 / 160 | (batch ≈ 0.67–0.83 s) |
+| B (defaults 1/1/10/2 s), 180 s | Hermle59 | 4.78 | 47.6 / 148.5 | 50 / 133 / 396 | 30.3 % |
+| | Hermle49 | 4.68 | 46.6 / 145.6 | 48 / 135 / 474 | 28.6 % |
+| | Hermle70 | 4.78 | 47.6 / 176.3 | 47 / 132 / 244 | 28.1 % |
+| | Hermle29 | 5.38 | 53.6 / 216.3 | 48 / 128 / 244 | 32.1 % |
+
+Request mix at the defaults over 180 s, per control: R_RI 22/23/24/26 179 each (the 1 s status group keeps up),
+R_VR 18 and R_RI 51 18 (10 s slow group), and **R_RI 27 108**, which is 90 alarm walks (2 s) plus 18 slow-group
+walks. R_RI 28 also ran 108 times on Hermle29, the control with an active alarm. R_RI 25, and R_RI 51 on Hermle49,
+were sent **once per session** and then suppressed as documented. No control slowed down or refused anything new
+under either cadence.
+
+Observation, not a defect: when a control carries `ALARM:` tags **and** the `Alarms/*` arrays, the alarm walk runs
+twice — once per `AlarmMs` for the conditions and again in the Slow group for the arrays. The two are separate reads,
+not shared. At the defaults that is about 0.1 extra R_RI 27 per second per control.
+
+## #853 — alarm class/group → Part 9 condition class
+
+Every alarm seen during the run (about 16 minutes of walks on all four controls). None were provoked.
+
+| Control | Wire record (`S_RI 27`) | Class | Group | Number (int32 / hex) | Text | Driver's condition class |
+|---|---|---|---|---|---|---|
+| Hermle29 | `0007 0003 8100017b …` | 7 Info | 3 Plc | −2130706053 / `0x8100017B` | `702126 Filter wait for spindle cooling  (U02) E161` | `BaseConditionClass` (i=11163) |
+| Hermle70 (raised 17:01:47, cleared before 17:13) | `0007 0003 8100015e …` | 7 Info | 3 Plc | −2130706082 / `0x8100015E` | `701550 Cabin unlocked! B301` | `BaseConditionClass` (i=11163) |
+| Hermle59, Hermle49 | — (R_RI 27 → T_ER 57, empty list throughout) | | | | | |
+
+- Both conditions behaved as designed. `AnyAlarm` and `Class7` went Active, severity 700, Quality Good, with the
+  message `<number> <text> (class 7, group 3)`. The `OtherAlarms` and other class conditions stayed Inactive.
+  Hermle70's alarm raised live, mid-run.
+- **Consistent with the mapping table.** Group 3 (`Plc`) maps to Base. Both texts are machine-builder (Hermle) PLC
+  messages: a 70xxxx message number and a device tag (`E161`, `B301`). That fits the RemoTools `DNC_ERROR_GROUP`
+  numbering, where 3 = PLC. Class 7 = Info fits non-fatal status notices, and the one record captured earlier
+  (−1 "Run test of cutout channels", class 7 / group 4) is consistent too. The live data now corroborates
+  **class 7, group 3 and group 4**.
+- **Still unobserved on hardware:** classes 1–6, 8 and 9 (including 5 EmergencyStop → Safety) and groups 1, 2, 5
+  and 6 (the Process/System/Python rows). Confirming them would need an active alarm of those kinds, which this run
+  did not provoke.
+- Note for authors: on these controls the alarm **number** field is not the displayed message number. It reads as
+  `0x81xxxxxx`, which is negative as int32, and the HMI number is the leading digits of the text. A `NUMBER=` selector
+  therefore has to use the negative raw value. This is documented ("filter on the raw codes"), but worth knowing.
+
+## Incidental findings (outside the Heidenhain driver)
+
+1. **Operator error during the run, now recovered.** When the first outage leg stopped the Hermle59 proxy,
+   `kill $(lsof -ti tcp:19059)` also matched and killed OrbStack's networking helper, which held the forwarded
+   connection. The OrbStack VM stopped; every container restarted when `orbctl start` was run (17:03–17:04 UTC), and
+   `otopcua-dev-sql-1` / `otopcua-dev-traefik-1` were started by hand. The outage leg was then re-run cleanly by
+   killing only the socat processes. **After this restart the OrbStack containers can no longer reach the LAN host
+   10.100.0.35** (the driver fixtures and opc-plc). The Mac itself, internet access and the 10.201.x VPN still work.
+   Before 17:03 there were no such errors. This most likely needs OrbStack restarted from the app, or its macOS
+   Local Network permission re-granted.
+2. **ConfigPublishCoordinator does not survive a ConfigDb that is unreachable at singleton start.**
+   `ConfigPublishCoordinator.PreStart` queries the DB synchronously. With SQL still starting, it threw
+   `ActorInitializationException` (SqlException: name not resolved), the singleton was terminated
+   (`Singleton actor [akka://otopcua/user/config-publish/config-publish] was terminated`) and it never restarted.
+   Every deploy then waited forever (`ClusterSingletonProxy failed to find … [config-publish]` every 30 s). Restarting
+   central-1 moved the singleton to central-2, which recovered it. This is a real resilience defect for any host that
+   boots before its ConfigDb.
+3. The first deployment (`7a47a745`) ended `TimedOut` (5/6 acks) because `site-b-2` was stopped. The later deploys
+   set `ClusterNode.MaintenanceMode = 1` on `site-b-2` and sealed. The flag was reset to 0 afterwards.
+
+## Cleanup
+
+The `hh829` driver, its 4 devices, 36 tag groups and 184 tags were deleted, along with the temporary API key and
+SITE-B NodeAcl. Deployment `6c7f859f` sealed at 17:14:13, and the driver was stopped and disposed on `site-b-1` at the
+same moment. All four proxy sessions closed, and then the proxies were stopped. `site-b-2` was restarted (standby,
+no Heidenhain driver), and `site-b-2` MaintenanceMode was restored to 0. Nothing on the controls was changed.
+
+## Status against the gates
+
+- **#829:** browse/read, write refusal (including a write-capable principal), outage and recovery with the other
+  controls unaffected, bounded traffic at default intervals, and retained evidence are **done** on four iTNC 530
+  controls. **Open:** the Tool Number/Index/AxisCode Double-variant defect above, which should be fixed and
+  re-verified with a strict variant-type assertion. Still unproven as before: other control families
+  (TNC 320/430/640), successful `R_MB`, units, timers 30–34, and DATA (not exercised under this scope).
+- **#853:** partially confirmed — class 7, groups 3 and 4 match the RemoTools numbering and map as documented
+  (Base). Classes other than 7 (notably 5 → Safety) and groups 1/2/5/6 are not yet observed on hardware.

docs/drivers/evidence/twincat-z35474-testconnect-2026-11-11.md

@@ -0,0 +1,91 @@
+# TwinCAT Test Connect on a real ADS target: Z35474, 2026-11-11
+
+Gitea #755 item: "Driver probe (Test Connect) happy path on a real ADS target".
+
+**Result: the item cannot be ticked. On real hardware the probe returns a false RED.** The ADS
+target is healthy and reachable through the router, but `TwinCATDriverProbe` never gets to its
+ADS handshake: its TCP preflight dials the wrong endpoint.
+
+## Target and scope
+
+| | |
+|---|---|
+| Press | Z35474 (Elma cleaner), production |
+| IP | 10.201.23.227 (reached over the VPN) |
+| AmsNetId / port | `5.118.56.93.1.1` : 851 (TC3 PLC runtime 1) |
+| Route used | the existing press-side route `otopcua-mac-20261111` → `10.181.201.5.1.1` (no AddRoute sent) |
+| Scope | read-only: ReadState, ReadDeviceInfo, symbol upload (Flat), two scalar reads. No writes, no WriteControl, no notifications, no route-table change. One session at a time, a few seconds per run. |
+
+## Harness
+
+This is a scratch console app outside the repo. It has a `ProjectReference` to
+`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT` and hosts `Beckhoff.TwinCAT.Ads.TcpRouter` 7.0.172 in
+process. The router's local NetId is `10.181.201.5.1.1`, with one route `Z35474 → 10.201.23.227 /
+5.118.56.93.1.1`. The package's sample `StaticRoutes.xml` is excluded, so the router loads no other
+routes. The Beckhoff ADS client is 7.0.172, the same version the driver uses.
+
+The harness drives the probe the way AdminUI does:
+
+- `RawTreeService.LoadMergedProbeConfigAsync` builds the config through
+  `DriverDeviceConfigMerger.Merge("{}", [DeviceRow("Z35474", {"hostAddress":"ads://5.118.56.93.1.1:851"})], [])`.
+- `AdminOperationsActor.ProbeAsync` calls `new TwinCATDriverProbe().ProbeAsync(merged, 10 s, cts(10 s))`.
+  The 10 s budget is `TestConnectBudget`'s TwinCAT default.
+
+The merged config the probe received:
+
+```json
+{"hostAddress":"ads://5.118.56.93.1.1:851","Devices":[{"hostAddress":"ads://5.118.56.93.1.1:851","DeviceName":"Z35474"}],"RawTags":[]}
+```
+
+## Observed (two runs, identical)
+
+```
+[probe]  Ok=False Message="Probe timed out after 10s." Latency=- wall=10003ms
+[phase2] ReadState Succeeded=True Err=Succeeded AdsState=Run DeviceState=0 189ms   (2nd run 109ms)
+[phase2] DeviceInfo Succeeded=True Name="Plc30 App" Version=3.1.1940
+[browse] roots=3384                                  (Flat symbol upload)
+[read]   .actualOperationMode : INT = 2
+[read]   .CFG_NUM_CDA_MAX     : INT = 5
+```
+
+`[phase2]` is not the probe. It reproduces the probe's own Phase 2 (`AdsClient.Connect(netId, 851)` +
+`ReadStateAsync`, `Timeout` = 10 s) in the same process through the same router. It shows that the
+handshake the probe would have done succeeds: Run state in about 0.1–0.2 s.
+
+Device: **"Plc30 App", ADS version 3.1.1940, AdsState Run, DeviceState 0.** The browse found 3,384 Flat
+roots, which matches the 2026-10-08 count.
+
+## Defect: the Phase 1 TCP preflight targets `(NetId[0..4]):(AMS port)`
+
+`TwinCATDriverProbe.ExtractTarget` builds the preflight target as the first four octets of the AmsNetId
+plus the AMS port. It then runs `Socket.ConnectAsync(host, port)` before it tries ADS. That target is
+wrong in two separate ways:
+
+1. **The AmsNetId is not the IP.** For Z35474 the preflight dialled **`5.118.56.93:851`**. That is a
+   public internet address unrelated to the press; on this Mac it routes out of the LAN gateway, not the
+   VPN. It black-holes, so the probe spends its whole 10 s budget and returns "Probe timed out after
+   10s." The press is at 10.201.23.227. The driver config has no IP field, so the probe cannot know it.
+   Only the AMS router knows it.
+2. **851 is an AMS port, not a TCP port.** ADS/TCP is 48898. On a press where the NetId happens to equal
+   the IP (for example Z33856 `10.201.45.236.1.1`), the preflight would dial `10.201.45.236:801/851`.
+   That should be refused and return "Connect failed: ConnectionRefused", also a RED. I inferred this
+   from the code and did not run it against that press.
+
+Consequence: Test Connect cannot go green against a real TwinCAT runtime. It stops at Phase 1 before
+the ADS handshake, which works. The unit tests (`TwinCATDriverProbeTests`) pin the current behaviour:
+they open a loopback `TcpListener` on the "AMS port" and use `ads://127.0.0.1.1.1:{port}`, so the
+mismatch never shows up there. The probe also sends a TCP SYN to whatever internet host the NetId's
+first four octets name.
+
+The fix is not made here: tracked source was left unedited, as the task asked. The fix direction is
+to drop the remote preflight, or point it at the local AMS router (`127.0.0.1:48898`), which is the
+only TCP endpoint the ADS client actually uses. Phase 2's `ReadStateAsync`, bounded by the timeout,
+then becomes the real reachability check. The unit tests would need to change with it.
+
+## #755 status
+
+- "Test Connect happy path on a real ADS target": **not tickable.** The ADS path works (router, route,
+  ReadState = Run, device info, browse, reads), but the probe as AdminUI runs it returns a false RED.
+  Re-run this gate after the preflight fix.
+- The harness lives in session scratch, not in the repo:
+  `scratchpad/twincat/probeharness/` (`Program.cs`, modes `probe` | `read`).