e299dc090a
v2-ci / unit-tests (push) Has been cancelled
The phase-6-*.ps1 scripts checked v2 Phase 6 code that later commits deliberately
deleted (76310b88 v1 Server/Admin, #522, #524, LiteDB cache, v3 migration squash):
56/56 failures were retired files, so they could not pass. Owner retired G1; G2
plus the per-area suites cover the invariants. phase-7-compliance.ps1 is untouched.
323 lines
13 KiB
Markdown
323 lines
13 KiB
Markdown
# v2 GA Lab Gates Plan
|
||
|
||
> **Canonical tracker**: `docs/v2/v2-release-readiness.md` — all code-path
|
||
> release blockers are closed as of 2026-04-24. This document maps the
|
||
> remaining exit-criteria from that tracker to concrete commands, automation
|
||
> boundaries, operator procedures, and pass criteria.
|
||
>
|
||
> **Status**: RELEASE-READY (code-path). Manual/lab gates remain open.
|
||
|
||
## The gate list
|
||
|
||
From `docs/v2/v2-release-readiness.md` §"Release-readiness exit criteria":
|
||
|
||
| # | Gate | Kind | Automatable here |
|
||
|---|------|------|-----------------|
|
||
| G1 | All four Phase 6.N compliance scripts exit 0 | Script | Yes — run on this box |
|
||
| G2 | `dotnet test ZB.MOM.WW.OtOpcUa.slnx` passes with <= 1 known-flake failure | Script | Yes — run on this box |
|
||
| G3 | Release blockers closed | Audit | Already closed (code-path) |
|
||
| G4 | Phase 5 driver complement shipped | Audit | Already closed |
|
||
| G5 | Production deployment checklist signed off by Fleet Admin | Operator | No — separate doc, human signoff |
|
||
| G6 | At least one end-to-end integration run against live Galaxy succeeds | Dev rig | ✅ **PASS 2026-11-12** (#615) — windev test Galaxy via mxaccessgw `447070a5`; see `docs/drivers/evidence/galaxy-g6-e2e-2026-11-12.md` |
|
||
| G7 | FOCAS live-CNC wire-level smoke (#54) passes against a real FANUC control | Lab hardware | ✅ **RESOLVED 2026-06-25** *(truth-sweep note 2026-08-07)* — live-validated on a real 31i-B (`10.201.31.5`): version gate, timer, PMC range, servo-meter, alarms, probe all verified live; caveat: `cnc_rdparam` found unsupported on that control. See `2026-06-25-focas-pdu-v3-30i-b-support.md` |
|
||
| G8 | OPC UA CTT / UA Compliance Test Tool passes against the live endpoint | Operator tool | No — requires CTT binary + live endpoint |
|
||
| G9 | Non-transparent redundancy cutover validated with >= 1 production client | Lab | No — see `docs/plans/phase-6-3-redundancy-interop-plan.md` |
|
||
|
||
---
|
||
|
||
## G1 — Phase 6 compliance scripts
|
||
|
||
> **RETIRED 2026-11-13 by the owner.** The scripts checked the v2 Phase 6 (2026-04) codebase. All 56 of their
|
||
> failures pointed at files a later commit deliberately deleted (the v1 `Server`/`Admin` projects in `76310b88`, #522,
|
||
> #524, the LiteDB cache, the v3 migration squash), so they could not pass. G2 (the full test suite) and the per-area
|
||
> test suites cover the invariants. The `phase-6-*.ps1` scripts were deleted. The last run and its classification:
|
||
> `docs/plans/evidence/2026-11-13-g1-compliance/RESULTS.md`. The section below is kept as history.
|
||
|
||
### Command
|
||
|
||
```powershell
|
||
pwsh ./scripts/compliance/phase-6-all.ps1
|
||
```
|
||
|
||
This meta-runner at `scripts/compliance/phase-6-all.ps1` invokes each
|
||
sub-script in a separate `powershell.exe` process to isolate exit codes:
|
||
|
||
| Sub-script | Phase | What it checks |
|
||
|-----------|-------|---------------|
|
||
| `phase-6-1-compliance.ps1` | 6.1 Resilience & Observability | Polly resilience classes, health endpoints, LiteDB sealed cache, observability sinks |
|
||
| `phase-6-2-compliance.ps1` | 6.2 Authorization runtime | `AuthorizationGate`, `TriePermissionEvaluator`, `NodeScopeResolver`, dispatch wiring in `DriverNodeManager` |
|
||
| `phase-6-3-compliance.ps1` | 6.3 Redundancy runtime | `ServiceLevelCalculator` 8-state band values, `RecoveryStateManager`, `ApplyLeaseRegistry`, `ServerRedundancyNodeWriter`; also invokes `dotnet test` with a baseline of 1097 |
|
||
| `phase-6-4-compliance.ps1` | 6.4 Admin UI completion | Data-layer types, Identification folder, deferred Blazor items marked `[DEFERRED]` |
|
||
|
||
### Pass criterion
|
||
|
||
```
|
||
Phase 6 aggregate: PASS
|
||
```
|
||
|
||
Exit code 0. Any `[FAIL]` line is a blocker. `[DEFERRED]` lines are expected
|
||
for the known-deferred surfaces listed in the implementation docs; they do not
|
||
fail the run.
|
||
|
||
### Prerequisites
|
||
|
||
- SQL Server `10.100.0.35,14330` reachable (Config DB tests use it).
|
||
- `dotnet` SDK on PATH (`.NET 10`).
|
||
- Run from repo root.
|
||
|
||
---
|
||
|
||
## G2 — Full solution test suite
|
||
|
||
### Command
|
||
|
||
```powershell
|
||
dotnet test ZB.MOM.WW.OtOpcUa.slnx --logger "console;verbosity=minimal"
|
||
```
|
||
|
||
For a more targeted run of integration suites that need their fixtures up:
|
||
|
||
```powershell
|
||
# bring modbus fixture up first
|
||
lmxopcua-fix up modbus standard
|
||
|
||
dotnet test ZB.MOM.WW.OtOpcUa.slnx --logger "console;verbosity=minimal"
|
||
```
|
||
|
||
### Pass criterion
|
||
|
||
- Passed count >= 1159 (2026-04-19 baseline after Phase 5 driver complement).
|
||
- Failed count <= 1 (the pre-existing
|
||
`SubscribeCommandTests.Execute_PrintsSubscriptionMessage` flake in
|
||
`Client.CLI` is the only tolerated failure).
|
||
- No new `[FAILED]` tests relative to the baseline.
|
||
|
||
### Known flake
|
||
|
||
`ZB.MOM.WW.OtOpcUa.Client.CLI.Tests::SubscribeCommandTests.Execute_PrintsSubscriptionMessage`
|
||
is a timing-sensitive subscribe-then-cancel test. Rerun the specific project
|
||
if it appears:
|
||
|
||
```powershell
|
||
dotnet test tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests `
|
||
--filter "FullyQualifiedName~SubscribeCommandTests.Execute_PrintsSubscriptionMessage" `
|
||
--count 3
|
||
```
|
||
|
||
If it fails all three runs, investigate; otherwise treat as flake.
|
||
|
||
### Docker fixtures needed for integration suites
|
||
|
||
| Driver | Command | Endpoint used |
|
||
|--------|---------|---------------|
|
||
| Modbus | `lmxopcua-fix up modbus standard` | `10.100.0.35:5020` |
|
||
| AB CIP | `lmxopcua-fix up abcip controllogix` | `10.100.0.35:44818` |
|
||
| S7 | `lmxopcua-fix up s7 s7_1500` | `10.100.0.35:1102` |
|
||
| OPC UA Client | `lmxopcua-fix up opcuaclient` | `opc.tcp://10.100.0.35:50000` |
|
||
| FOCAS | `lmxopcua-fix up focas` (mock server) | `10.100.0.35:8193` |
|
||
|
||
TwinCAT integration tests require the TCBSD ESXi VM at `10.100.0.128`
|
||
(AmsNetId `41.169.163.43.1.1`). Set env var before running:
|
||
|
||
```powershell
|
||
$env:TWINCAT_TARGET_HOST = "10.100.0.128"
|
||
$env:TWINCAT_TARGET_NETID = "41.169.163.43.1.1"
|
||
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT.IntegrationTests
|
||
```
|
||
|
||
Galaxy integration tests run against the live mxaccessgw on the dev box
|
||
(gate G6).
|
||
|
||
---
|
||
|
||
## G3 — Release blockers closed (audit, already satisfied)
|
||
|
||
All three code-path release blockers are closed per `v2-release-readiness.md`:
|
||
|
||
- Authorization dispatch wiring (task #143, PR #94) — CLOSED.
|
||
- Config fallback Phase 6.1 Stream D (task #136, PR #96) — CLOSED.
|
||
- Redundancy Phase 6.3 Streams A/C core (tasks #145/#147, PRs #98-99) — CLOSED.
|
||
|
||
No action required. Record the PR numbers in the release notes.
|
||
|
||
---
|
||
|
||
## G4 — Driver complement (audit, already satisfied)
|
||
|
||
All eight drivers shipped:
|
||
|
||
Galaxy, Modbus (+ DL205/S7/MELSEC profiles), S7 native, OPC UA Client, AB CIP,
|
||
AB Legacy, TwinCAT ADS, FOCAS (managed wire client — Tier-C isolation retired,
|
||
FOCAS is now Tier A in-process via `WireFocasClient`).
|
||
|
||
No action required.
|
||
|
||
---
|
||
|
||
## G5 — Production deployment checklist (operator action)
|
||
|
||
The deployment checklist is a separate document covering:
|
||
|
||
- Windows service install (`scripts/install/Install-Services.ps1`)
|
||
- Config DB migration (`scripts/db/Apply-Migrations.ps1`)
|
||
- Certificate provisioning and trust
|
||
- LDAP / GLAuth configuration for production AD target
|
||
- mxaccessgw API key provisioning (`apikey create-key` in the sibling repo)
|
||
- Service account permissions
|
||
- Prometheus / OpenTelemetry export configuration
|
||
- Firewall rules (port 4840 OPC UA, port 5120 gRPC to mxaccessgw,
|
||
Admin port 5000/5001)
|
||
|
||
**Sign-off party**: Fleet Admin (operator). Not automatable.
|
||
|
||
> **SIGNED OFF 2026-11-13** by the owner as Fleet Admin (#615).
|
||
|
||
Record sign-off as a comment on the v2 release issue.
|
||
|
||
---
|
||
|
||
## G6 — Live Galaxy end-to-end integration run
|
||
|
||
**Requires**: AVEVA System Platform installed on dev box (confirmed available
|
||
per project memory `project_aveva_platform_installed.md`); mxaccessgw running
|
||
with a provisioned API key; at least one Galaxy object deployed.
|
||
|
||
### Procedure
|
||
|
||
1. Start mxaccessgw:
|
||
|
||
```powershell
|
||
# in sibling repo C:\Users\dohertj2\Desktop\mxaccessgw\
|
||
dotnet run --project src/MxGateway.Server -- --apikey-path .local/api-key.txt
|
||
```
|
||
|
||
2. Start OtOpcUa server with Galaxy driver instance configured:
|
||
|
||
```powershell
|
||
sc start OtOpcUa
|
||
```
|
||
|
||
3. Browse via Client CLI:
|
||
|
||
```powershell
|
||
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- `
|
||
browse -u opc.tcp://localhost:4840 -r -d 3
|
||
```
|
||
|
||
4. Read a known Galaxy tag (e.g. a deployed `$UserDefined` object attribute):
|
||
|
||
```powershell
|
||
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- `
|
||
read -u opc.tcp://localhost:4840 -n "ns=2;s=<tag_name.AttributeName>"
|
||
```
|
||
|
||
5. Subscribe and verify live updates:
|
||
|
||
```powershell
|
||
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- `
|
||
subscribe -u opc.tcp://localhost:4840 -n "ns=2;s=<tag_name.AttributeName>" -i 1000
|
||
```
|
||
|
||
### Pass criterion
|
||
|
||
- Browse returns a non-empty node tree mirroring the Galaxy hierarchy.
|
||
- Read returns `Good` quality with a non-null value.
|
||
- Subscribe receives at least one data-change notification within 5 s
|
||
(or within the configured publishing interval).
|
||
- No `BadNoCommunication` or `BadTimeout` errors in the server log.
|
||
|
||
Record: Galaxy version, deployed object count, OtOpcUa git SHA.
|
||
|
||
---
|
||
|
||
## G7 — FOCAS live-CNC smoke (task #54)
|
||
|
||
**Requires**: real FANUC CNC with Ethernet option, accessible on TCP port 8193
|
||
from the dev box; CNC series known (e.g. 0i-F, 30i-B).
|
||
|
||
See `docs/plans/live-hardware-validation-runbooks.md` §FOCAS for the full
|
||
runbook.
|
||
|
||
### Pass criterion
|
||
|
||
- `WireFocasClient` opens a FOCAS2 session (`cnc_allclibhndl3` succeeds).
|
||
- Identity nodes (`Identity/SeriesNumber`, `Identity/MaxAxes`) return non-null
|
||
values matching the physical control panel display.
|
||
- At least one axis position (`Axes/X/AbsolutePosition` or similar) returns
|
||
`Good` quality with a plausible double value.
|
||
- Subscribe on a polled tag delivers at least three updates within 5 s.
|
||
- No `EW_SOCKET` (-1) or `EW_HANDLE` (-7) errors in the server log during a
|
||
2-minute soak.
|
||
|
||
Record: CNC series, firmware version, test date, OtOpcUa git SHA.
|
||
|
||
---
|
||
|
||
## G8 — OPC UA Conformance Test Tool (CTT) pass
|
||
|
||
> **WAIVED 2026-11-13 by the owner (#615):** no CTT run is required for v2 GA. The procedure below is kept for a
|
||
> later conformance effort.
|
||
|
||
**Requires**: OPC Foundation OPC UA Compliance Test Tool (CTT) or the
|
||
open-source UA Compliance Test Tool installed on the client machine;
|
||
live OtOpcUa server endpoint.
|
||
|
||
### Recommended minimum profile set
|
||
|
||
- `Attribute Read`
|
||
- `Attribute Write`
|
||
- `Browse`
|
||
- `Subscription` (DataChange)
|
||
- `Server-side monitoring`
|
||
- `Security — None profile` (if server configured with `Security:Profiles=[None]`)
|
||
|
||
### Procedure
|
||
|
||
1. Launch CTT. Add server endpoint: `opc.tcp://localhost:4840`.
|
||
2. Run the profile set above.
|
||
3. Capture the CTT report HTML/XML.
|
||
|
||
### Pass criterion
|
||
|
||
All mandatory test cases in each profile: **PASS** or **NOT APPLICABLE**.
|
||
|
||
Zero mandatory failures. Advisory failures may be documented with rationale
|
||
(e.g. optional capability not implemented).
|
||
|
||
Record: CTT version, profile set, OtOpcUa git SHA, report artifact.
|
||
|
||
---
|
||
|
||
## G9 — Non-transparent redundancy cutover with production client
|
||
|
||
> **PASS 2026-11-13 (#615)** with `otopcua-cli` as the client (owner-approved in place of UaExpert/Ignition), on an
|
||
> isolated 2-node pair in the deployed `Unset` mode: A1–A3, A5, A7, A8, B2 and B3 pass; A4/A5b are unmet under `Unset`
|
||
> and met under `Warm` (#717). Evidence: `docs/plans/evidence/2026-11-13-615-g9/RESULTS.md`.
|
||
|
||
See `docs/plans/phase-6-3-redundancy-interop-plan.md` for the full runbook.
|
||
|
||
**Minimum acceptable result**: one complete pass of the A-block (UaExpert
|
||
OPC UA signal verification) plus scenario B2 (UaExpert failover on Primary
|
||
kill).
|
||
|
||
Ignition 8.3 is the recommended production client per decision #85. If
|
||
Ignition is not available on the lab machine, UaExpert is accepted for v2 GA.
|
||
|
||
Record: client name + version, OtOpcUa git SHA, test date.
|
||
|
||
---
|
||
|
||
## Gate summary table
|
||
|
||
| Gate | Command / Procedure | Pass criterion | Owner |
|
||
|------|---------------------|----------------|-------|
|
||
| G1 | ~~`pwsh ./scripts/compliance/phase-6-all.ps1`~~ | **Retired 2026-11-13 (owner)**: superseded by G2 | — |
|
||
| G2 | `dotnet test ZB.MOM.WW.OtOpcUa.slnx` | >= 1159 passing, <= 1 failure — **PASS 2026-11-13** (evidence `2026-11-13-g2-full-suite/`) | Dev |
|
||
| G3 | Audit PR list in release-readiness.md | All blockers show CLOSED | Dev |
|
||
| G4 | Audit driver table | All 8 drivers listed as shipped | Dev |
|
||
| G5 | Run deployment checklist doc | All items checked; Fleet Admin signs off — **SIGNED OFF 2026-11-13** | Fleet Admin |
|
||
| G6 | Browse/read/subscribe against live Galaxy | Good quality, non-empty tree | Dev (dev box) |
|
||
| G7 | FOCAS CNC smoke — see live-hardware runbook | Session open, Good quality reads — **PASS 2026-11-13** (vd03, Makino 31i-B) | Dev + lab hardware |
|
||
| G8 | CTT profile run against live endpoint | **Waived 2026-11-13 (owner)** | — |
|
||
| G9 | Redundancy cutover runbook | A-block + B2 pass with >= 1 client — **PASS 2026-11-13** (`otopcua-cli`) | Dev + two instances |
|