Files
lmxopcua/docs/plans/v2-ga-lab-gates-plan.md
T
Joseph Doherty e299dc090a
v2-ci / unit-tests (push) Has been cancelled
chore(ga): retire G1 - delete the stale Phase 6 compliance scripts; v2 GA exit criteria all met/waived/retired
The phase-6-*.ps1 scripts checked v2 Phase 6 code that later commits deliberately
deleted (76310b88 v1 Server/Admin, #522, #524, LiteDB cache, v3 migration squash):
56/56 failures were retired files, so they could not pass. Owner retired G1; G2
plus the per-area suites cover the invariants. phase-7-compliance.ps1 is untouched.
2026-11-13 10:33:43 -05:00

323 lines
13 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# v2 GA Lab Gates Plan
> **Canonical tracker**: `docs/v2/v2-release-readiness.md` — all code-path
> release blockers are closed as of 2026-04-24. This document maps the
> remaining exit-criteria from that tracker to concrete commands, automation
> boundaries, operator procedures, and pass criteria.
>
> **Status**: RELEASE-READY (code-path). Manual/lab gates remain open.
## The gate list
From `docs/v2/v2-release-readiness.md` §"Release-readiness exit criteria":
| # | Gate | Kind | Automatable here |
|---|------|------|-----------------|
| G1 | All four Phase 6.N compliance scripts exit 0 | Script | Yes — run on this box |
| G2 | `dotnet test ZB.MOM.WW.OtOpcUa.slnx` passes with <= 1 known-flake failure | Script | Yes — run on this box |
| G3 | Release blockers closed | Audit | Already closed (code-path) |
| G4 | Phase 5 driver complement shipped | Audit | Already closed |
| G5 | Production deployment checklist signed off by Fleet Admin | Operator | No — separate doc, human signoff |
| G6 | At least one end-to-end integration run against live Galaxy succeeds | Dev rig | ✅ **PASS 2026-11-12** (#615) — windev test Galaxy via mxaccessgw `447070a5`; see `docs/drivers/evidence/galaxy-g6-e2e-2026-11-12.md` |
| G7 | FOCAS live-CNC wire-level smoke (#54) passes against a real FANUC control | Lab hardware | ✅ **RESOLVED 2026-06-25** *(truth-sweep note 2026-08-07)* — live-validated on a real 31i-B (`10.201.31.5`): version gate, timer, PMC range, servo-meter, alarms, probe all verified live; caveat: `cnc_rdparam` found unsupported on that control. See `2026-06-25-focas-pdu-v3-30i-b-support.md` |
| G8 | OPC UA CTT / UA Compliance Test Tool passes against the live endpoint | Operator tool | No — requires CTT binary + live endpoint |
| G9 | Non-transparent redundancy cutover validated with >= 1 production client | Lab | No — see `docs/plans/phase-6-3-redundancy-interop-plan.md` |
---
## G1 — Phase 6 compliance scripts
> **RETIRED 2026-11-13 by the owner.** The scripts checked the v2 Phase 6 (2026-04) codebase. All 56 of their
> failures pointed at files a later commit deliberately deleted (the v1 `Server`/`Admin` projects in `76310b88`, #522,
> #524, the LiteDB cache, the v3 migration squash), so they could not pass. G2 (the full test suite) and the per-area
> test suites cover the invariants. The `phase-6-*.ps1` scripts were deleted. The last run and its classification:
> `docs/plans/evidence/2026-11-13-g1-compliance/RESULTS.md`. The section below is kept as history.
### Command
```powershell
pwsh ./scripts/compliance/phase-6-all.ps1
```
This meta-runner at `scripts/compliance/phase-6-all.ps1` invokes each
sub-script in a separate `powershell.exe` process to isolate exit codes:
| Sub-script | Phase | What it checks |
|-----------|-------|---------------|
| `phase-6-1-compliance.ps1` | 6.1 Resilience & Observability | Polly resilience classes, health endpoints, LiteDB sealed cache, observability sinks |
| `phase-6-2-compliance.ps1` | 6.2 Authorization runtime | `AuthorizationGate`, `TriePermissionEvaluator`, `NodeScopeResolver`, dispatch wiring in `DriverNodeManager` |
| `phase-6-3-compliance.ps1` | 6.3 Redundancy runtime | `ServiceLevelCalculator` 8-state band values, `RecoveryStateManager`, `ApplyLeaseRegistry`, `ServerRedundancyNodeWriter`; also invokes `dotnet test` with a baseline of 1097 |
| `phase-6-4-compliance.ps1` | 6.4 Admin UI completion | Data-layer types, Identification folder, deferred Blazor items marked `[DEFERRED]` |
### Pass criterion
```
Phase 6 aggregate: PASS
```
Exit code 0. Any `[FAIL]` line is a blocker. `[DEFERRED]` lines are expected
for the known-deferred surfaces listed in the implementation docs; they do not
fail the run.
### Prerequisites
- SQL Server `10.100.0.35,14330` reachable (Config DB tests use it).
- `dotnet` SDK on PATH (`.NET 10`).
- Run from repo root.
---
## G2 — Full solution test suite
### Command
```powershell
dotnet test ZB.MOM.WW.OtOpcUa.slnx --logger "console;verbosity=minimal"
```
For a more targeted run of integration suites that need their fixtures up:
```powershell
# bring modbus fixture up first
lmxopcua-fix up modbus standard
dotnet test ZB.MOM.WW.OtOpcUa.slnx --logger "console;verbosity=minimal"
```
### Pass criterion
- Passed count >= 1159 (2026-04-19 baseline after Phase 5 driver complement).
- Failed count <= 1 (the pre-existing
`SubscribeCommandTests.Execute_PrintsSubscriptionMessage` flake in
`Client.CLI` is the only tolerated failure).
- No new `[FAILED]` tests relative to the baseline.
### Known flake
`ZB.MOM.WW.OtOpcUa.Client.CLI.Tests::SubscribeCommandTests.Execute_PrintsSubscriptionMessage`
is a timing-sensitive subscribe-then-cancel test. Rerun the specific project
if it appears:
```powershell
dotnet test tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests `
--filter "FullyQualifiedName~SubscribeCommandTests.Execute_PrintsSubscriptionMessage" `
--count 3
```
If it fails all three runs, investigate; otherwise treat as flake.
### Docker fixtures needed for integration suites
| Driver | Command | Endpoint used |
|--------|---------|---------------|
| Modbus | `lmxopcua-fix up modbus standard` | `10.100.0.35:5020` |
| AB CIP | `lmxopcua-fix up abcip controllogix` | `10.100.0.35:44818` |
| S7 | `lmxopcua-fix up s7 s7_1500` | `10.100.0.35:1102` |
| OPC UA Client | `lmxopcua-fix up opcuaclient` | `opc.tcp://10.100.0.35:50000` |
| FOCAS | `lmxopcua-fix up focas` (mock server) | `10.100.0.35:8193` |
TwinCAT integration tests require the TCBSD ESXi VM at `10.100.0.128`
(AmsNetId `41.169.163.43.1.1`). Set env var before running:
```powershell
$env:TWINCAT_TARGET_HOST = "10.100.0.128"
$env:TWINCAT_TARGET_NETID = "41.169.163.43.1.1"
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT.IntegrationTests
```
Galaxy integration tests run against the live mxaccessgw on the dev box
(gate G6).
---
## G3 — Release blockers closed (audit, already satisfied)
All three code-path release blockers are closed per `v2-release-readiness.md`:
- Authorization dispatch wiring (task #143, PR #94) — CLOSED.
- Config fallback Phase 6.1 Stream D (task #136, PR #96) — CLOSED.
- Redundancy Phase 6.3 Streams A/C core (tasks #145/#147, PRs #98-99) — CLOSED.
No action required. Record the PR numbers in the release notes.
---
## G4 — Driver complement (audit, already satisfied)
All eight drivers shipped:
Galaxy, Modbus (+ DL205/S7/MELSEC profiles), S7 native, OPC UA Client, AB CIP,
AB Legacy, TwinCAT ADS, FOCAS (managed wire client — Tier-C isolation retired,
FOCAS is now Tier A in-process via `WireFocasClient`).
No action required.
---
## G5 — Production deployment checklist (operator action)
The deployment checklist is a separate document covering:
- Windows service install (`scripts/install/Install-Services.ps1`)
- Config DB migration (`scripts/db/Apply-Migrations.ps1`)
- Certificate provisioning and trust
- LDAP / GLAuth configuration for production AD target
- mxaccessgw API key provisioning (`apikey create-key` in the sibling repo)
- Service account permissions
- Prometheus / OpenTelemetry export configuration
- Firewall rules (port 4840 OPC UA, port 5120 gRPC to mxaccessgw,
Admin port 5000/5001)
**Sign-off party**: Fleet Admin (operator). Not automatable.
> **SIGNED OFF 2026-11-13** by the owner as Fleet Admin (#615).
Record sign-off as a comment on the v2 release issue.
---
## G6 — Live Galaxy end-to-end integration run
**Requires**: AVEVA System Platform installed on dev box (confirmed available
per project memory `project_aveva_platform_installed.md`); mxaccessgw running
with a provisioned API key; at least one Galaxy object deployed.
### Procedure
1. Start mxaccessgw:
```powershell
# in sibling repo C:\Users\dohertj2\Desktop\mxaccessgw\
dotnet run --project src/MxGateway.Server -- --apikey-path .local/api-key.txt
```
2. Start OtOpcUa server with Galaxy driver instance configured:
```powershell
sc start OtOpcUa
```
3. Browse via Client CLI:
```powershell
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- `
browse -u opc.tcp://localhost:4840 -r -d 3
```
4. Read a known Galaxy tag (e.g. a deployed `$UserDefined` object attribute):
```powershell
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- `
read -u opc.tcp://localhost:4840 -n "ns=2;s=<tag_name.AttributeName>"
```
5. Subscribe and verify live updates:
```powershell
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- `
subscribe -u opc.tcp://localhost:4840 -n "ns=2;s=<tag_name.AttributeName>" -i 1000
```
### Pass criterion
- Browse returns a non-empty node tree mirroring the Galaxy hierarchy.
- Read returns `Good` quality with a non-null value.
- Subscribe receives at least one data-change notification within 5 s
(or within the configured publishing interval).
- No `BadNoCommunication` or `BadTimeout` errors in the server log.
Record: Galaxy version, deployed object count, OtOpcUa git SHA.
---
## G7 — FOCAS live-CNC smoke (task #54)
**Requires**: real FANUC CNC with Ethernet option, accessible on TCP port 8193
from the dev box; CNC series known (e.g. 0i-F, 30i-B).
See `docs/plans/live-hardware-validation-runbooks.md` §FOCAS for the full
runbook.
### Pass criterion
- `WireFocasClient` opens a FOCAS2 session (`cnc_allclibhndl3` succeeds).
- Identity nodes (`Identity/SeriesNumber`, `Identity/MaxAxes`) return non-null
values matching the physical control panel display.
- At least one axis position (`Axes/X/AbsolutePosition` or similar) returns
`Good` quality with a plausible double value.
- Subscribe on a polled tag delivers at least three updates within 5 s.
- No `EW_SOCKET` (-1) or `EW_HANDLE` (-7) errors in the server log during a
2-minute soak.
Record: CNC series, firmware version, test date, OtOpcUa git SHA.
---
## G8 — OPC UA Conformance Test Tool (CTT) pass
> **WAIVED 2026-11-13 by the owner (#615):** no CTT run is required for v2 GA. The procedure below is kept for a
> later conformance effort.
**Requires**: OPC Foundation OPC UA Compliance Test Tool (CTT) or the
open-source UA Compliance Test Tool installed on the client machine;
live OtOpcUa server endpoint.
### Recommended minimum profile set
- `Attribute Read`
- `Attribute Write`
- `Browse`
- `Subscription` (DataChange)
- `Server-side monitoring`
- `Security — None profile` (if server configured with `Security:Profiles=[None]`)
### Procedure
1. Launch CTT. Add server endpoint: `opc.tcp://localhost:4840`.
2. Run the profile set above.
3. Capture the CTT report HTML/XML.
### Pass criterion
All mandatory test cases in each profile: **PASS** or **NOT APPLICABLE**.
Zero mandatory failures. Advisory failures may be documented with rationale
(e.g. optional capability not implemented).
Record: CTT version, profile set, OtOpcUa git SHA, report artifact.
---
## G9 — Non-transparent redundancy cutover with production client
> **PASS 2026-11-13 (#615)** with `otopcua-cli` as the client (owner-approved in place of UaExpert/Ignition), on an
> isolated 2-node pair in the deployed `Unset` mode: A1–A3, A5, A7, A8, B2 and B3 pass; A4/A5b are unmet under `Unset`
> and met under `Warm` (#717). Evidence: `docs/plans/evidence/2026-11-13-615-g9/RESULTS.md`.
See `docs/plans/phase-6-3-redundancy-interop-plan.md` for the full runbook.
**Minimum acceptable result**: one complete pass of the A-block (UaExpert
OPC UA signal verification) plus scenario B2 (UaExpert failover on Primary
kill).
Ignition 8.3 is the recommended production client per decision #85. If
Ignition is not available on the lab machine, UaExpert is accepted for v2 GA.
Record: client name + version, OtOpcUa git SHA, test date.
---
## Gate summary table
| Gate | Command / Procedure | Pass criterion | Owner |
|------|---------------------|----------------|-------|
| G1 | ~~`pwsh ./scripts/compliance/phase-6-all.ps1`~~ | **Retired 2026-11-13 (owner)**: superseded by G2 | — |
| G2 | `dotnet test ZB.MOM.WW.OtOpcUa.slnx` | >= 1159 passing, <= 1 failure — **PASS 2026-11-13** (evidence `2026-11-13-g2-full-suite/`) | Dev |
| G3 | Audit PR list in release-readiness.md | All blockers show CLOSED | Dev |
| G4 | Audit driver table | All 8 drivers listed as shipped | Dev |
| G5 | Run deployment checklist doc | All items checked; Fleet Admin signs off — **SIGNED OFF 2026-11-13** | Fleet Admin |
| G6 | Browse/read/subscribe against live Galaxy | Good quality, non-empty tree | Dev (dev box) |
| G7 | FOCAS CNC smoke — see live-hardware runbook | Session open, Good quality reads — **PASS 2026-11-13** (vd03, Makino 31i-B) | Dev + lab hardware |
| G8 | CTT profile run against live endpoint | **Waived 2026-11-13 (owner)** | — |
| G9 | Redundancy cutover runbook | A-block + B2 pass with >= 1 client — **PASS 2026-11-13** (`otopcua-cli`) | Dev + two instances |