task-galaxy-e2e branch — non-FOCAS work-in-progress snapshot
Catch-all commit for pending work on the task-galaxy-e2e branch that
wasn't part of the FOCAS migration. Grouping by topic so future per-topic
commits can be cherry-picked if needed.
TwinCAT
- src/.../Driver.TwinCAT/AdsTwinCATClient.cs + TwinCATDriverFactoryExtensions.cs:
factory-registration extensions + ADS client refinements.
- src/.../Driver.TwinCAT.Cli/Commands/BrowseCommand.cs: new browse command
for the TwinCAT test-client CLI.
- tests/.../Driver.TwinCAT.IntegrationTests/TwinCAT3SmokeTests.cs + TwinCatProject/:
fixture scaffold with a minimal POU + README pointing at the TCBSD/ESXi
VM for e2e.
- docs/Driver.TwinCAT.Cli.md + docs/drivers/TwinCAT-Test-Fixture.md:
documentation for the above.
- docs/v3/twincat-backlog.md: forward-looking backlog seed.
Admin UI + fleet status
- src/.../Admin/Components/Pages/Clusters/DriversTab.razor + Hosts.razor:
UI refresh for fleet-status rendering.
- src/.../Admin/Hubs/FleetStatusHub.cs + FleetStatusPoller.cs +
Admin/Program.cs: SignalR hub + poller plumbing for live fleet data.
- tests/.../Admin.Tests/FleetStatusPollerTests.cs: poller coverage.
Server + redundancy runtime (Phase 6.3 follow-ups)
- src/.../Server/Hosting/RedundancyPublisherHostedService.cs: HostedService
that owns the RedundancyStatePublisher lifecycle + wires peer reachability.
- src/.../Server/Redundancy/ServerRedundancyNodeWriter.cs: OPC UA
variable-node writer binding ServiceLevel + ServerUriArray to the
publisher's events.
- src/.../Server/Program.cs + Server.csproj: hosted-service registration.
- tests/.../Server.Tests/ServerRedundancyNodeWriterTests.cs +
Server.Tests.csproj: coverage for the above.
Configuration
- src/.../Configuration/Validation/DraftValidator.cs +
tests/.../Configuration.Tests/DraftValidatorTests.cs: draft-validation
refinements.
E2E scripts (shared infrastructure)
- scripts/e2e/README.md + _common.ps1 + test-all.ps1: shared helpers + the
all-drivers test-all runner.
- scripts/e2e/test-opcuaclient.ps1: OPC UA Client e2e runner.
Docs
- docs/v2/implementation/phase-6-{1,2,3,4}*.md + exit-gate-phase-{3,7}.md:
phase-gate + implementation doc updates.
- docs/v2/plan.md: top-level plan refresh.
- docs/v2/redundancy-interop-playbook.md: client interop playbook for the
Phase 6.3 redundancy-runtime work.
Two orphan FOCAS docs remain on disk but deliberately unstaged —
docs/v2/focas-deployment.md and docs/v2/implementation/focas-simulator-plan.md
describe the now-retired Tier-C topology and should either be rewritten
or deleted in a follow-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Joseph Doherty
@@ -2,60 +2,85 @@
|
||||
|
||||
Coverage map + gap inventory for the Beckhoff TwinCAT ADS driver.
|
||||
|
||||
**TL;DR:** Integration-test scaffolding lives at
|
||||
`tests/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT.IntegrationTests/` (task #221).
|
||||
`TwinCATXarFixture` probes TCP 48898 on an operator-supplied VM; three
|
||||
smoke tests (read / write / native notification) run end-to-end through
|
||||
the real ADS stack when the VM is reachable, skip cleanly otherwise.
|
||||
**Remaining operational work**: stand up a TwinCAT 3 XAR runtime in a
|
||||
Hyper-V VM, author the `.tsproj` project documented at
|
||||
`TwinCatProject/README.md`, rotate the 7-day trial license (or buy a
|
||||
paid runtime). Unit tests via `FakeTwinCATClient` still carry the
|
||||
exhaustive contract coverage.
|
||||
**TL;DR:** Integration-test suite lives at
|
||||
`tests/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT.IntegrationTests/`. `TwinCATXarFixture`
|
||||
probes TCP 48898 on an operator-supplied runtime; the suite runs **14
|
||||
`[TwinCATFact]` methods + one 16-case `[TwinCATTheory]` = 30 test cases** end-to-end
|
||||
through the real ADS stack when the runtime is reachable, skips cleanly
|
||||
otherwise. The runtime can be a Hyper-V XAR VM or a TCBSD VM
|
||||
(`TwinCatProject/README.md` covers both). Unit tests via `FakeTwinCATClient`
|
||||
still carry the exhaustive contract coverage alongside.
|
||||
|
||||
TwinCAT is the only driver outside Galaxy that uses **native
|
||||
notifications** (no polling) for `ISubscribable`, and the fake exposes a
|
||||
fire-event harness so notification routing is contract-tested rigorously
|
||||
at the unit layer.
|
||||
TwinCAT is the only driver outside Galaxy that uses **native notifications**
|
||||
(no polling) for `ISubscribable`. The integration suite verifies that path on
|
||||
the wire; the fake exposes a fire-event harness so notification routing is
|
||||
also contract-tested rigorously at the unit layer.
|
||||
|
||||
## What the fixture is
|
||||
|
||||
**Integration layer** (task #221, scaffolded):
|
||||
`tests/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT.IntegrationTests/` —
|
||||
`TwinCATXarFixture` TCP-probes ADS port 48898 on the host specified by
|
||||
`TWINCAT_TARGET_HOST` + requires `TWINCAT_TARGET_NETID` (AmsNetId of the
|
||||
VM). No fixture-owned lifecycle — XAR can't run in Docker because it
|
||||
bypasses the Windows kernel scheduler, so the VM stays
|
||||
operator-managed. `TwinCatProject/README.md` documents the required
|
||||
`.tsproj` project state; the file itself lands once the XAR VM is up +
|
||||
the project is authored. Three smoke tests:
|
||||
`Driver_reads_seeded_DINT_through_real_ADS`,
|
||||
`Driver_write_then_read_round_trip_on_scratch_REAL`, and
|
||||
`Driver_subscribe_receives_native_ADS_notifications_on_counter_changes`
|
||||
— all skip cleanly via `[TwinCATFact]` when the runtime isn't
|
||||
reachable.
|
||||
**Integration layer**: `tests/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT.IntegrationTests/`
|
||||
— `TwinCATXarFixture` TCP-probes ADS port 48898 on the host supplied by
|
||||
`TWINCAT_TARGET_HOST` (defaults to `localhost`) + requires
|
||||
`TWINCAT_TARGET_NETID` (AmsNetId of the runtime). Optionally takes
|
||||
`TWINCAT_TARGET_PORT` (default `851` = TC3 PLC runtime 1). No fixture-owned
|
||||
lifecycle — XAR / TCBSD can't run in Docker because they bypass the host
|
||||
kernel scheduler, so the runtime stays operator-managed.
|
||||
`TwinCatProject/README.md` documents the required project state; the tests
|
||||
gate on `[TwinCATFact]` / `[TwinCATTheory]` and skip cleanly when
|
||||
`TWINCAT_TARGET_NETID` is unset or the probe fails.
|
||||
|
||||
**Unit layer**: `tests/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT.Tests/` is
|
||||
still the primary coverage. `FakeTwinCATClient` also fakes the
|
||||
`AddDeviceNotification` flow so tests can trigger callbacks without a
|
||||
running runtime.
|
||||
**Unit layer**: `tests/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT.Tests/` remains the
|
||||
primary contract coverage. `FakeTwinCATClient` fakes the
|
||||
`AddDeviceNotification` flow so tests can trigger callbacks without a running
|
||||
runtime.
|
||||
|
||||
## What it actually covers
|
||||
|
||||
### Integration (XAR VM, task #221 — code scaffolded, needs VM + project)
|
||||
### Integration (live runtime)
|
||||
|
||||
- `TwinCAT3SmokeTests.Driver_reads_seeded_DINT_through_real_ADS` — real AMS
|
||||
handshake + ADS read of `GVL_Fixture.nCounter` (seeded at 1234, MAIN
|
||||
increments each cycle)
|
||||
- `TwinCAT3SmokeTests.Driver_write_then_read_round_trip_on_scratch_REAL` —
|
||||
real ADS write + read on `GVL_Fixture.rSetpoint`
|
||||
- `TwinCAT3SmokeTests.Driver_subscribe_receives_native_ADS_notifications_on_counter_changes`
|
||||
— real `AddDeviceNotification` against the cycle-incrementing counter;
|
||||
observes `OnDataChange` firing within 3 s of subscribe
|
||||
Every capability the driver implements is exercised on the wire:
|
||||
|
||||
All three gated on `TWINCAT_TARGET_HOST` + `TWINCAT_TARGET_NETID` env
|
||||
vars; skip cleanly via `[TwinCATFact]` when the VM isn't reachable or
|
||||
vars are unset.
|
||||
- **Read** — `Driver_reads_seeded_DINT_through_real_ADS` (AMS handshake +
|
||||
symbolic read of `GVL_Fixture.nCounter`)
|
||||
- **Write + read round-trip** — `Driver_write_then_read_round_trip_on_scratch_REAL`
|
||||
on `GVL_Fixture.rSetpoint`
|
||||
- **Array element round-trip** — `Driver_round_trips_array_element_write_and_read`
|
||||
on `GVL_Arrays.aReal1D[5]` (exercises `TwinCATSymbolPath` subscript
|
||||
rendering)
|
||||
- **Subscribe (native ADS notifications)** —
|
||||
`Driver_subscribe_receives_native_ADS_notifications_on_counter_changes`;
|
||||
observes `OnDataChange` firing within 10 s of subscribe
|
||||
- **Symbol browse (direct client path)** —
|
||||
`Driver_browses_committed_symbol_hierarchy_via_real_ADS` via
|
||||
`ITwinCATClient.BrowseSymbolsAsync`
|
||||
- **Symbol browse (through DiscoverAsync + `IAddressSpaceBuilder` pipeline)** —
|
||||
`DiscoverAsync_renders_declared_tags_and_controller_browse_hits_address_space_builder`
|
||||
verifies the real `TwinCAT/ → device/ → Discovered/` folder tree
|
||||
- **Auto-reconnect** — `Driver_auto_reconnects_after_underlying_client_is_disposed`
|
||||
disposes the `AdsClient` mid-flight; next read must re-establish
|
||||
- **Primitive type coverage** — `Driver_reads_every_primitive_type_with_correct_mapping`
|
||||
runs as a `[Theory]` against the 16 primitives in `GVL_Primitives`
|
||||
(Bool, SInt, USInt, Int, UInt, DInt, UDInt, LInt, ULInt, Real, LReal,
|
||||
String, Time, TimeOfDay, Date, DateTime) — asserts status + CLR type +
|
||||
seed value where ergonomic
|
||||
- **Bit-indexed BOOL** — `Driver_reads_bit_indexed_BOOL_from_word` against
|
||||
`GVL_Primitives.vWord.3` + `.4` (bits of `0xBEEF`)
|
||||
- **Nested UDT navigation** — `Driver_reads_deeply_nested_UDT_path` reads
|
||||
`GVL_Plant.Line1.Stations[1].Axes[1].Motor.Temperature` (LREAL) + `.Running` (BOOL)
|
||||
- **Multi-device routing + isolation** —
|
||||
`Driver_routes_reads_per_device_and_isolates_unreachable_peers` pairs the
|
||||
real runtime with a bogus AmsNetId; healthy device reads still succeed
|
||||
- **Probe loop + `IHostConnectivityProbe`** —
|
||||
`Probe_loop_raises_host_status_transition_to_Running_on_reachable_target`
|
||||
asserts `OnHostStatusChanged → Running` and snapshot parity
|
||||
- **Negative error mappings** —
|
||||
`Driver_reports_errors_for_unknown_tag_and_nonexistent_symbol_and_readonly_write`
|
||||
covers `BadNodeIdUnknown`, ghost-symbol communication errors, and the
|
||||
`BadNotWritable` short-circuit
|
||||
|
||||
All tests gate on `TWINCAT_TARGET_NETID` (required) via `[TwinCATFact]` /
|
||||
`[TwinCATTheory]`; `TWINCAT_TARGET_HOST` (default `localhost`) and
|
||||
`TWINCAT_TARGET_PORT` (default `851`) are optional overrides.
|
||||
|
||||
### Unit
|
||||
|
||||
@@ -65,54 +90,69 @@ vars are unset.
|
||||
- `TwinCATReadWriteTests` — read + write through the fake, status mapping
|
||||
- `TwinCATSymbolPathTests` — symbol-path routing for nested struct members
|
||||
- `TwinCATSymbolBrowserTests` — `ITagDiscovery.DiscoverAsync` via
|
||||
`ReadSymbolsAsync` (#188) + system-symbol filtering
|
||||
- `TwinCATNativeNotificationTests` — `AddDeviceNotification` (#189)
|
||||
registration, callback-delivery-to-`OnDataChange` wiring, unregister on
|
||||
unsubscribe
|
||||
`BrowseSymbolsAsync` + system-symbol filtering
|
||||
- `TwinCATNativeNotificationTests` — `AddDeviceNotification` registration,
|
||||
callback-delivery-to-`OnDataChange` wiring, unregister on unsubscribe
|
||||
- `TwinCATDriverTests` — `IDriver` lifecycle
|
||||
|
||||
Capability surfaces whose contract is verified: `IDriver`, `IReadable`,
|
||||
`IWritable`, `ITagDiscovery`, `ISubscribable`, `IHostConnectivityProbe`,
|
||||
`IPerCallHostResolver`.
|
||||
Capability surfaces whose contract is verified at the unit layer: `IDriver`,
|
||||
`IReadable`, `IWritable`, `ITagDiscovery`, `ISubscribable`,
|
||||
`IHostConnectivityProbe`, `IPerCallHostResolver`. The integration suite now
|
||||
verifies `ITagDiscovery` + `IHostConnectivityProbe` on the wire as well.
|
||||
|
||||
## Bugs caught by live runs
|
||||
|
||||
The integration suite surfaced three driver defects that `FakeTwinCATClient`
|
||||
couldn't, since each lived below the abstraction boundary:
|
||||
|
||||
1. **Notification cycle time unit** — `NotificationSettings(cycleTime, maxDelay)`
|
||||
takes **milliseconds** per Beckhoff InfoSys
|
||||
(`tcadsnetref/7313319051`), but the driver was multiplying by `10_000`
|
||||
under a "100 ns units" assumption. A requested 250 ms cycle was being set
|
||||
to ~41 minutes — subscribe never fired. Fix in `AdsTwinCATClient.AddNotificationAsync`.
|
||||
2. **`STRING(N)` / `WSTRING(N)` type mapper** — `MapSymbolTypeName` only
|
||||
matched bare `"STRING"` / `"WSTRING"`, so sized strings (the common case)
|
||||
fell off `BrowseSymbolsAsync` entirely. Fix: strip the `(…)` bound before
|
||||
the switch.
|
||||
3. **Bit-indexed BOOL path** — driver was sending `"GVL.vWord.3"` to ADS as
|
||||
a BOOL read. TwinCAT's symbol table doesn't expose bit-access paths; the
|
||||
read returned `DeviceSymbolNotFound`. Fix: strip the `.N` suffix, read
|
||||
the parent word as `uint`, extract the bit locally via `ExtractBit`.
|
||||
|
||||
All three paths are now pinned by live-wire tests.
|
||||
|
||||
## What it does NOT cover
|
||||
|
||||
### 1. AMS / ADS wire traffic
|
||||
### 1. AMS / ADS wire framing
|
||||
|
||||
No real AMS router frame is exchanged. Beckhoff's `TwinCAT.Ads` NuGet (their
|
||||
own .NET SDK, not libplctag-style OSS) has no in-process fake; tests stub
|
||||
the `ITwinCATClient` abstraction above it.
|
||||
No raw AMS packet is inspected. Beckhoff's `TwinCAT.Ads` NuGet (their own
|
||||
.NET SDK, not libplctag-style OSS) has no in-process fake at the frame
|
||||
level; tests run against a real router.
|
||||
|
||||
### 2. Multi-route AMS
|
||||
|
||||
ADS supports chained routes (`<localNetId> → <routerNetId> → <targetNetId>`)
|
||||
for PLCs behind an EC master / IPC gateway. Parse coverage exists; wire-path
|
||||
coverage doesn't.
|
||||
coverage is single-hop only.
|
||||
|
||||
### 3. Notification reliability under jitter
|
||||
### 3. Notification coalescing under jitter
|
||||
|
||||
`AddDeviceNotification` delivers at the runtime's cycle boundary; under high
|
||||
CPU load or network jitter real notifications can coalesce. The fake fires
|
||||
one callback per test invocation — real callback-coalescing behavior is
|
||||
untested.
|
||||
`AddDeviceNotification` delivers at the runtime's cycle boundary; under
|
||||
sustained CPU load or network jitter real notifications can coalesce. The
|
||||
live test only asserts at-least-one delivery within a generous window —
|
||||
coalescing behavior under stress isn't verified.
|
||||
|
||||
### 4. TC2 vs TC3 variant handling
|
||||
|
||||
TwinCAT 2 (ADS v1) and TwinCAT 3 (ADS v2) have subtly different
|
||||
`GetSymbolInfoByName` semantics + symbol-table layouts. Driver targets TC3;
|
||||
TC2 compatibility is not exercised.
|
||||
`GetSymbolInfoByName` semantics + symbol-table layouts. Driver + tests target
|
||||
TC3; TC2 compatibility is not exercised.
|
||||
|
||||
### 5. Cycle-time alignment for `ISubscribable`
|
||||
### 5. Alarms / history
|
||||
|
||||
Native ADS notifications fire on the PLC cycle boundary. The fake test
|
||||
harness assumes notifications fire on a timer the test controls;
|
||||
cycle-aligned firing under real PLC control is not verified.
|
||||
|
||||
### 6. Alarms / history
|
||||
|
||||
Driver doesn't implement `IAlarmSource` or `IHistoryProvider` — not in
|
||||
scope for this driver family. TwinCAT 3's TcEventLogger could theoretically
|
||||
back an `IAlarmSource`, but shipping that is a separate feature.
|
||||
Driver doesn't implement `IAlarmSource` or `IHistoryProvider` — not in scope
|
||||
for this driver family. TwinCAT 3's TcEventLogger could theoretically back
|
||||
an `IAlarmSource`, but shipping that is a separate feature.
|
||||
|
||||
## When to trust TwinCAT tests, when to reach for a rig
|
||||
|
||||
@@ -122,37 +162,25 @@ back an `IAlarmSource`, but shipping that is a separate feature.
|
||||
| "Does notification → `OnDataChange` wire correctly?" | yes (contract) | yes |
|
||||
| "Does symbol browsing filter TwinCAT internals?" | yes | yes |
|
||||
| "Does a real ADS read return correct bytes?" | no | yes (required) |
|
||||
| "Do notifications coalesce under load?" | no | yes (required) |
|
||||
| "Does auto-reconnect work on router restart?" | no (contract only) | yes (required) |
|
||||
| "Do notifications coalesce under sustained load?" | no | yes (required) |
|
||||
| "Does a TC2 PLC work the same as TC3?" | no | yes (required) |
|
||||
|
||||
## Follow-up candidates
|
||||
|
||||
1. **XAR VM live-population** — scaffolding is in place (this PR); the
|
||||
remaining work is operational: stand up the Hyper-V VM, install XAR,
|
||||
author the `.tsproj` per `TwinCatProject/README.md`, configure the
|
||||
bilateral ADS route, set `TWINCAT_TARGET_HOST` + `TWINCAT_TARGET_NETID`
|
||||
on the dev box. Then the three smoke tests transition skip → pass.
|
||||
Tracked as #221.
|
||||
2. **License-rotation automation** — XAR's 7-day trial expires on
|
||||
schedule. Either automate `TcActivate.exe /reactivate` via a
|
||||
scheduled task on the VM (not officially supported; reportedly works
|
||||
for some TC3 builds), or buy a paid runtime license (~$1k one-time
|
||||
per runtime per CPU) to kill the rotation. The doc at
|
||||
`TwinCatProject/README.md` §License rotation walks through both.
|
||||
3. **Lab rig** — cheapest IPC (CX7000 / CX9020) on a dedicated network;
|
||||
the only route that covers TC2 + real EtherCAT I/O timing + cycle
|
||||
jitter under CPU load.
|
||||
Deferred to v3 — see [`docs/v3/twincat-backlog.md`](../v3/twincat-backlog.md).
|
||||
Covers TC2 coverage, notification-coalescing-under-load, multi-hop AMS,
|
||||
license-rotation automation, and a dedicated lab IPC.
|
||||
|
||||
## Key fixture / config files
|
||||
|
||||
- `tests/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT.IntegrationTests/TwinCATXarFixture.cs`
|
||||
— TCP probe + skip-attributes + env-var parsing
|
||||
- `tests/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT.IntegrationTests/TwinCAT3SmokeTests.cs`
|
||||
— three wire-level smoke tests
|
||||
— wire-level test suite (14 `[TwinCATFact]` + 16-case `[TwinCATTheory]`)
|
||||
- `tests/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT.IntegrationTests/TwinCatProject/README.md`
|
||||
— project spec + VM setup + license-rotation notes
|
||||
- `tests/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT.Tests/FakeTwinCATClient.cs` —
|
||||
in-process fake with the notification-fire harness used by
|
||||
`TwinCATNativeNotificationTests`
|
||||
- `src/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT/TwinCATDriver.cs` — ctor takes
|
||||
`ITwinCATClientFactory`
|
||||
in-process fake with the notification-fire harness
|
||||
- `src/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT/TwinCATDriver.cs` — ctor is
|
||||
`(TwinCATDriverOptions, string driverInstanceId, ITwinCATClientFactory? = null)`
|
||||
|
||||
Reference in New Issue
Block a user