Files
lmxopcua/docs/v2/v2-release-readiness.md
T
Joseph Doherty e299dc090a
v2-ci / unit-tests (push) Has been cancelled
chore(ga): retire G1 - delete the stale Phase 6 compliance scripts; v2 GA exit criteria all met/waived/retired
The phase-6-*.ps1 scripts checked v2 Phase 6 code that later commits deliberately
deleted (76310b88 v1 Server/Admin, #522, #524, LiteDB cache, v3 migration squash):
56/56 failures were retired files, so they could not pass. Owner retired G1; G2
plus the per-area suites cover the invariants. phase-7-compliance.ps1 is untouched.
2026-11-13 10:33:43 -05:00

161 lines
24 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# v2 Release Readiness
> ⚠️ **Historical (audited 2026-07-27) — a v2-era snapshot; several "Closed" rows name types that do
> not exist in `src/`.** In particular `:41` claims ACL enforcement was "Closed 2026-04-24" via
> `AuthorizationBootstrap` / `NodeScopeResolver` / `OpcUaServerService` — **all three have zero
> occurrences in `src/`, and there is no per-node ACL gate on any operation** (see `deferment.md`
> §3.1 and the banner in `docs/ReadWriteOperations.md`). `RedundancyCoordinator`,
> `RedundancyStatePublisher`, `PeerReachabilityTracker`, `GenerationRefreshHostedService`,
> `ClusterTopologyLoader` and `SealedBootstrap` are likewise gone. The **unchecked GA exit criteria**
> near the end of the file are still live — see `deferment.md` §5.
> **Last updated**: 2026-04-24 (Phase 5 driver complement closed — AB CIP, AB Legacy, TwinCAT, FOCAS all shipped; FOCAS Tier-C retired for a pure-managed in-process client)
> **Status**: **RELEASE-READY** for v2 GA. Every exit criterion is met, waived or retired as of 2026-11-13 (G1 retired, G8 waived; G2, G5, G6, G7, G9 done). All three original code-path release blockers remain closed. Phase 5 is now complete. The #615 lab/operator gates are complete: G5 signed off by the Fleet Admin and G6/G9 passed (2026-11-12/13), and G8 (CTT) was waived by the owner (2026-11-13); hardening follow-ups remain; see exit-criteria checklist below.
This doc is the single view of where v2 stands against its release criteria. Update it whenever a deferred follow-up closes or a new release blocker is discovered.
## Release-readiness dashboard
| Phase | Shipped | Status |
|---|---|---|
| Phase 0 — Rename + entry gate | ✓ | Shipped |
| Phase 1 — Configuration + Admin scaffold | ✓ | Shipped (some UI items deferred to 6.4) |
| Phase 2 — Galaxy driver split (Proxy/Host/Shared) | ✓ | Shipped |
| Phase 3 — OPC UA server + LDAP + security profiles | ✓ | Shipped |
| Phase 4 — Redundancy scaffold (entities + endpoints) | ✓ | Shipped (runtime closes in 6.3) |
| Phase 5 — Drivers | ✓ | **Shipped** — Galaxy, Modbus (+ DL205/S7/MELSEC profiles), S7 native, OPC UA Client, AB CIP, AB Legacy, TwinCAT ADS, FOCAS (managed wire client) |
| Phase 6.1 — Resilience & Observability | ✓ | Shipped (PRs #78–83) |
| Phase 6.2 — Authorization runtime | ◐ core | Core shipped (PRs #84–88, #94 dispatch wiring); finer-grained Browse/Subscribe/Alarm/Call gating + 3-user interop matrix deferred |
| Phase 6.3 — Redundancy runtime | ◐ core | Core shipped (PRs #89–90, #98–99); peer-probe HostedServices, OPC UA variable-node binding, `sp_PublishGeneration` lease wrap, client interop matrix deferred |
| Phase 6.4 — Admin UI completion | ◐ data layer + Identification | Data layer + OPC 40010 Identification folder shipped (PRs #91–92, Identification audit close-out 2026-04-23); Blazor UI pieces deferred |
**Driver integration-test counts** (end-to-end against live or simulated targets): Modbus 26, FOCAS 9, AbCip 7, OpcUaClient 3, S7 3, AbLegacy 2, TwinCAT 2. Plus Galaxy's separate cross-FX parity/stability suite.
**Aggregate test counts** (2026-04-19 baseline): 1159 passing across the solution. One pre-existing Client.CLI `SubscribeCommandTests.Execute_PrintsSubscriptionMessage` flake tracked separately. Rerun `dotnet test ZB.MOM.WW.OtOpcUa.slnx` after the FOCAS migration commits land to refresh the number.
## Release blockers (must close before v2 GA)
All code-path release blockers are closed. The remaining items are live-hardware / manual validations listed under exit criteria.
### ~~Security — Phase 6.2 dispatch wiring~~ (task #143 — marked CLOSED 2026-04-19, PR #94)
> ⚠️ **THIS CLOSURE DOES NOT HOLD (re-verified against source 2026-07-27).** Every type named in this
> subsection — `AuthorizationGate`, `NodeScopeResolver`, `AuthorizationBootstrap`, `WriteAuthzPolicy`,
> `DriverNodeManager`, `FilterBrowseReferences`, `GateCallMethodRequests`, `MapCallOperation`,
> `EquipmentNamespaceContent`, and the `Node:Authorization:*` config keys — has **zero occurrences in
> `src/`**. There is no `OnReadValue` hook, no `Browse` override, no `CreateMonitoredItems` override and
> no `Call` override in the live node manager, and no HistoryRead path consults any gate.
>
> Whatever landed on the v2 branch did not survive into the shipped tree. The **only** data-plane
> authorization today is two server-wide role checks (`WriteOperate` for writes, `AlarmAck` for alarm
> methods); reads, browses, subscriptions and history reads are ungated, and `NodeAcl` rows are
> authored and deployed but never evaluated. See [`../security.md`](../security.md) §
> Data-Plane Authorization and `deferment.md` §3.1.
>
> The text below is retained as the historical record of what was *believed* closed. **Read none of it
> as current behaviour.**
**Closed**. `AuthorizationGate` + `NodeScopeResolver` thread through `OpcUaApplicationHost → OtOpcUaServer → DriverNodeManager`. `OnReadValue` + `OnWriteValue` + all four HistoryRead paths call `gate.IsAllowed(identity, operation, scope)` before the invoker. Production deployments activate enforcement by constructing `OpcUaApplicationHost` with an `AuthorizationGate(StrictMode: true)` + populating the `NodeAcl` table.
Remaining Stream C surfaces (hardening, not release-blocking) — **none of these shipped either**:
- ~~Browse + TranslateBrowsePathsToNodeIds gating with ancestor-visibility logic per `acl-design.md` §Browse.~~ **Partial, 2026-04-24.** `DriverNodeManager.Browse` override post-filters the `ReferenceDescription` list via a new `FilterBrowseReferences` helper — denied nodes disappear silently per OPC UA convention. Ancestor-visibility implication (Read-grant at `Line/Tag` implying Browse on `Line`) still to ship; needs a subtree-has-any-grant query on the trie evaluator. `TranslateBrowsePathsToNodeIds` surface not yet wired.
- ~~CreateMonitoredItems + TransferSubscriptions gating with per-item `(AuthGenerationId, MembershipVersion)` stamp so revoked grants surface `BadUserAccessDenied` within one publish cycle (decision #153).~~ **Partial, 2026-04-24.** `DriverNodeManager.CreateMonitoredItems` override pre-gates each request and pre-populates `BadUserAccessDenied` into the errors slot for denied items (the base stack honours pre-set errors and skips those items). Decision #153's per-item `(AuthGenerationId, MembershipVersion)` stamp for detecting mid-subscription revocation is still to ship — needs subscription-layer plumbing. TransferSubscriptions not yet wired (same pattern).
- ~~Alarm Acknowledge / Confirm / Shelve gating.~~ **Partial, 2026-04-24.** Acknowledge + Confirm map to dedicated `OpcUaOperation.AlarmAcknowledge` / `AlarmConfirm` via `MapCallOperation`; Shelve falls through to generic `OpcUaOperation.Call` (needs per-instance method NodeId resolution to distinguish — follow-up).
- ~~Call (method invocation) gating.~~ **Closed 2026-04-24.** `DriverNodeManager.Call` override pre-gates each `CallMethodRequest` via `GateCallMethodRequests`. Denied calls return `BadUserAccessDenied` without running the method. Alarm methods map to alarm-specific operation kinds; everything else gates as generic `Call`.
- ~~Finer-grained scope resolution — current `NodeScopeResolver` returns a flat cluster-level scope. Joining against the live Configuration DB to populate UnsArea / UnsLine / Equipment path is tracked as Stream C.12.~~ **Closed 2026-04-24.** `AuthorizationBootstrap` now loads `NodeAcl` rows for the current generation into a `PermissionTrieCache`, builds the gate, and merges every registered driver's `EquipmentNamespaceContent` into a full-path `NodeScopeResolver` index. `OpcUaServerService` calls the bootstrap after the equipment registry is populated, before `OpcUaApplicationHost.StartAsync`. Disabled by default — operators flip `Node:Authorization:Enabled=true` to enforce, `StrictMode=true` to reject anonymous/no-groups identities.
- 3-user integration matrix covering every operation × allow/deny.
### ~~Config fallback — Phase 6.1 Stream D wiring~~ (task #136 — **CLOSED** 2026-04-19, PR #96)
**Closed**. `SealedBootstrap` consumes `ResilientConfigReader` + `GenerationSealedCache` + `StaleConfigFlag` end-to-end; `/healthz` surfaces the stale flag.
Remaining follow-ups (hardening):
- A `HostedService` that polls `sp_GetCurrentGenerationForCluster` periodically so peer-published generations land in this node's cache without a restart.
- Richer snapshot payload via `sp_GetGenerationContent` so fallback can serve full generation content (DriverInstance enumeration, ACL rows, etc.) from the sealed cache alone.
### ~~Redundancy — Phase 6.3 Streams A/C core~~ (tasks #145 + #147 — **CLOSED** 2026-04-19, PRs #98–99)
**Closed**. `RedundancyCoordinator` + `RedundancyStatePublisher` + `PeerReachabilityTracker` orchestrate topology + apply lease + recovery state + peer reachability through `ServiceLevelCalculator` + emit `OnStateChanged` / `OnServerUriArrayChanged` edge-triggered events.
Remaining Phase 6.3 surfaces (hardening, not release-blocking):
- ~~`PeerHttpProbeLoop` + `PeerUaProbeLoop` HostedServices populating `PeerReachabilityTracker` on each tick.~~ **Closed 2026-04-24.** Two-layer probe model shipped: HTTP probe at 2 s / 1 s timeout against `/healthz`; OPC UA probe at 10 s / 5 s timeout via `DiscoveryClient.GetEndpoints`, short-circuiting when HTTP reports the peer unhealthy. Registered on the Server as `AddHostedService<PeerHttpProbeLoop>` + `AddHostedService<PeerUaProbeLoop>`. Publisher now sees accurate `PeerReachability` per peer instead of degrading to `Unknown` → Isolated-Primary band (230).
- ~~OPC UA variable-node wiring: bind `ServiceLevel` Byte + `ServerUriArray` String[] to the publisher's events via `BaseDataVariable.OnReadValue` / direct value push.~~ **Closed 2026-05-26.** `ServiceLevel` byte binding closed earlier under Path D. Peer-URI half closed via `OpcUaApplicationHost.PopulateServerArray` — populates self + each `PeerApplicationUris` entry into the SDK `IServerInternal.ServerUris` `StringTable`; clients read `Server.ServerArray` (NodeId `i=2254`). Validated by `DualEndpointTests` in `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/`. `ServerUriArray` proper (the redundancy-object-type child) remains deferred pending object-type upgrade.
- ~~`sp_PublishGeneration` wraps its apply in `await using var lease = coordinator.BeginApplyLease(...)` so the `PrimaryMidApply` band (200) fires during actual publishes (task #148 part 2).~~ **Closed 2026-04-24.** The apply loop now lives in `GenerationRefreshHostedService` — polls `sp_GetCurrentGenerationForCluster` every 5s, opens a lease when a new generation is detected, calls `RedundancyCoordinator.RefreshAsync` inside the `await using`, releases the lease on all exit paths. Replaces the previous "topology never refreshes without a process restart" behaviour.
- Client interop matrix — Ignition / Kepware / Aveva OI Gateway (Stream F, task #150). Manual + doc-only.
### ~~Phase 5 driver complement~~ (task #120 — **CLOSED** 2026-04-24)
**Closed**. All four deferred drivers shipped:
- **AB CIP** (PRs #202–222) — `Driver.AbCip`, `Driver.AbCip.IntegrationTests` (7 tests), AB CIP Cli. Live-boot verified against a ControlLogix rig.
- **AB Legacy** (PRs #202, #223) — `Driver.AbLegacy`, `Driver.AbLegacy.IntegrationTests` (2 tests), AB Legacy Cli. PCCC cip-path workaround for SLC/MicroLogix.
- **TwinCAT ADS** (PR #205) — `Driver.TwinCAT`, `Driver.TwinCAT.IntegrationTests` (2 tests), TwinCAT Cli. TCBSD/ESXi fixture for e2e since local Hyper-V / TwinCAT RTIME are mutually exclusive on the dev box.
- **FOCAS** (PRs #173, #199 + this session's migration) — `Driver.FOCAS` with an **in-process managed `FocasWireClient`** that speaks FOCAS/2 over TCP directly. Tier-C isolation retired — `Driver.FOCAS.Host` + `Driver.FOCAS.Shared` + `FwlibNative` P/Invoke + shim DLL + NSSM service all deleted. `Driver.FOCAS.IntegrationTests` covers 9 scenarios (fixed tree identity/axes/program/timers/spindle + user-authored PARAM/MACRO/PMC reads, Browse, Subscribe, IAlarmSource raise/clear, Probe transitions).
Decision recorded: FOCAS is **read-only** against the CNC by design — writes return `BadNotWritable`. See `docs/drivers/FOCAS.md` + `docs/drivers/FOCAS-Test-Fixture.md` for the deployment + coverage map.
## Nice-to-haves (not release-blocking)
- **Admin UI** — Phase 6.1 Stream E.2/E.3 (`/hosts` column refresh), Phase 6.2 Stream D (`RoleGrantsTab` + `AclsTab` Probe), Phase 6.3 Stream E (`RedundancyTab`), Phase 6.4 Streams A/B UI pieces, Stream C DiffViewer, Stream D `IdentificationFields.razor`. Tasks #134, #144, #149, #153, #155, #156, #157.
- **Background services** — Phase 6.1 Stream B.4 `ScheduledRecycleScheduler` HostedService (task #137), Phase 6.1 Stream A analyzer (task #135 — Roslyn analyzer asserting every capability surface routes through `CapabilityInvoker`).
- **Multi-host dispatch** — Phase 6.1 Stream A follow-up (task #135). Every driver currently gets a single pipeline keyed on `driver.DriverInstanceId`; multi-host drivers (Modbus with N PLCs) need per-PLC host resolution so failing PLCs trip per-PLC breakers without poisoning siblings. Decision #144 requires this but not wired.
- **Phase 7** — scripting + alarming + historian sink (plan drafted 2026-04-20 in `docs/v2/implementation/phase-7-*.md`). Out of scope for v2 GA.
## Live-hardware validations (task #54 + task family)
The code ships; these tasks remain open as lab/field verification:
- **#54** — FOCAS live-CNC wire-level smoke against a real FANUC control. ~~The mock's wire responder is PDU-verified against `fwlibe64.dll` upstream but OtOpcUa's managed client has not been pointed at a production CNC.~~ **Update 2026-08-07 (truth sweep): RESOLVED — live-validated on a real 31i-B (`10.201.31.5`) on 2026-06-25** (version gate, timer, PMC range, servo-meter, alarms, probe all verified live; `cnc_rdparam` found unsupported on that control). See `docs/plans/2026-06-25-focas-pdu-v3-30i-b-support.md`.
- **AB CIP live-boot** — already passed on a ControlLogix rig (PR #222). Continue to run ahead of each release.
- **TwinCAT wire-live** — TCBSD/ESXi fixture covers the common path; production PLC verification remains lab-gated.
## Running the release-readiness check
The Phase 6 compliance scripts (`scripts/compliance/phase-6-*.ps1`) were retired on 2026-11-13 (G1). They checked
code that has since been deliberately deleted. The automated check is now the full suite (G2):
```bash
dotnet test ZB.MOM.WW.OtOpcUa.slnx --logger "console;verbosity=minimal"
```
Run it with the shared fixtures on `10.100.0.35` up (`infra/README.md`). Restart `otopcua-pymodbus-standard` before
the run until #909 is fixed. The disposition rules and the 2026-11-13 result are in
`docs/plans/evidence/2026-11-13-g2-full-suite/RESULTS.md`.
## Release-readiness exit criteria
v2 GA requires all of the following:
- [x] ~~All four Phase 6.N compliance scripts exit 0.~~ **G1 RETIRED 2026-11-13 by the owner.** The scripts checked deliberately deleted v2 Phase 6 code (56/56 failures were retired files) and were deleted; G2 and the per-area tests cover the invariants. Evidence: `docs/plans/evidence/2026-11-13-g1-compliance/RESULTS.md`.
- [x] `dotnet test ZB.MOM.WW.OtOpcUa.slnx` passes with ≤ 1 known-flake failure. **G2 PASS 2026-11-13** on the product code: 2 full runs (20,589 / 20,588 passed). The one real defect found is fixed (`f416b883`); every other failure is a shared-fixture or load-timing issue (#909, #910, #911) whose test passes in isolation. Evidence: `docs/plans/evidence/2026-11-13-g2-full-suite/RESULTS.md`.
- [x] Release blockers listed above all closed.
- [x] Phase 5 driver complement shipped (Galaxy, Modbus, S7, OpcUaClient, AbCip, AbLegacy, TwinCAT, FOCAS).
- [x] Production deployment checklist (separate doc) signed off by Fleet Admin. **G5 SIGNED OFF 2026-11-13** by the owner as Fleet Admin (#615).
- [x] At least one end-to-end integration run against the live Galaxy on the dev box succeeds. **G6 PASS 2026-11-12** (#615): windev test Galaxy `DEV` through mxaccessgw `447070a5`; browse (raw + UNS), read, subscribe, write and readback, and native alarm → Part 9 → ack (confirmed in Galaxy) → clear. Evidence: `docs/drivers/evidence/galaxy-g6-e2e-2026-11-12.md`. Found #895 (fixed, `e113faf5`) and #896 (open). DeployWatcher not exercised (no headless Galaxy redeploy).
- [x] FOCAS live-CNC wire-level smoke (#54) runs clean against a real FANUC control. **G7 PASS 2026-11-13** on the deployed DEV server (vd03, `76e73e7d`) against the Makino Z-34184 FANUC 31i-B (series G431, version 22.0). Identity, axis `X/AbsolutePosition` (added by owner-approved deploy `e9fbe63e`, a surgical PureAdd), 6 updates in 5 s, and a 2-minute soak with 0 `EW_SOCKET`/`EW_HANDLE`. The identity values were not compared against the control panel. Evidence: `docs/drivers/evidence/focas-g7-2026-11-13/RESULTS.md`.
- [x] ~~OPC UA conformance test (CTT or UA Compliance Test Tool) passes against the live endpoint.~~ **G8 WAIVED by the owner 2026-11-13** (#615): no CTT run is required for v2 GA.
- [x] Non-transparent redundancy cutover validated with at least one production client (Ignition 8.3 recommended — see decision #85). **G9 PASS 2026-11-13** (#615), with `otopcua-cli` as the client (owner-approved), on an isolated 2-node pair in the deployed `Unset` mode (`003c7f89`): A1–A3, A5, A7, A8, B2 (crash) and B3 pass; A4/A5b are recorded as unmet under `Unset` and met under `Warm` (#717). Evidence: `docs/plans/evidence/2026-11-13-615-g9/RESULTS.md`.
## Change log
- **2026-11-13** — **G1 retired** by the owner and the `scripts/compliance/phase-6-*.ps1` scripts deleted: they checked deliberately deleted v2 Phase 6 code. Every v2 GA exit criterion is now met, waived (G8) or retired (G1).
- **2026-11-13** — **G7 PASS:** a read-only FOCAS live-CNC smoke on vd03 against the Makino Z-34184 31i-B. The owner approved adding one axis tag (`Axes/X/AbsolutePosition`, deploy `e9fbe63e`, surgical PureAdd); it reads 280.1574 Good. The panel comparison was not done (it needs a person at the machine). G1 awaits the owner's decision.
- **2026-11-13** — **G2 PASS.** Two full `dotnet test ZB.MOM.WW.OtOpcUa.slnx` runs (60 projects, ~21k tests). One real defect: a bare `"driver"` role literal from #717's `c569f806`, fixed in `f416b883`. Every other failure is environmental and filed: the Modbus fixture is swamped by load (#909), an intermittent opc-plc `BadRequestInterrupted` (#910), and a scripted-alarm run-guard timing test (#911). G1 (stale Phase 6 compliance scripts) and G7 (FOCAS live CNC) evidence is recorded and awaits the owner's decision.
- **2026-11-13** — **G5 (#615) signed off** by the owner as Fleet Admin. With G6/G9 passed and G8 waived, every #615 gate is complete and #615 is closed.
- **2026-11-13** — **G9 (#615) PASS and G8 waived.** The owner approved `otopcua-cli` as the G9 client and waived the CTT (G8) for GA. G9 ran on an isolated 2-node pair (`otopcua-g9`, `master` `003c7f89`, redundancy mode `Unset` as deployed). ServiceLevel 250/240, a follower stop left the Primary at 250, and ServerArray lists both URIs. On a graceful Primary stop the CLI's `-F` sessions failed over in about 11 s. On a `docker kill` of the Primary they failed over within 1 s, with a ~2 s data gap and no Bad values; the survivor reached 250 at +44 s. The returning node settled at 240 with no failback. A4 (`RedundancySupport`) and A5b (`ServerUriArray`) are unmet under `Unset` and met under `Warm` (#717, closed; enabling Warm is #907). The run found a CLI defect: array values printed as `System.String[]`, fixed by `ValueFormatter`. Evidence: `docs/plans/evidence/2026-11-13-615-g9/RESULTS.md`. G5 (Fleet Admin sign-off) is the only open exit criterion.
- **2026-11-12** — **G6 (#615) PASS: end to end against a live Galaxy.** A local fused admin+driver Host (OtOpcUa `00a5be26`) ran the `GalaxyMxGateway` driver against the windev test Galaxy `DEV` (System Platform 2020, 129 deployed objects) through mxaccessgw `447070a5`. Every plan §G6 criterion was met through both the raw (`ns=2;s=<RawPath>`) and UNS (`ns=3;s=Area/Line/Equipment/Signal`) NodeIds: non-empty browse mirroring area → object, `Good` reads, subscribe notifications inside 1 s, no `BadTimeout`. The only `BadNoCommunication` was a planned driver respawn on redeploy. Extra steps also passed: write and readback (also verified through an independent gateway session; anonymous negative control `BadUserAccessDenied`), and a Galaxy alarm raised by an OPC UA write surfacing as a Part 9 condition. That condition was acked over OPC UA and the ack was confirmed in Galaxy (`.Acked=True`, `.AckMsg` = the comment), then cleared. Severity 750 for AVEVA priority 500 (#887 band; data point for #894). The live run found **#895**: a native alarm authored the AdminUI-picker way (`attributeRef tag.Attr`) never went active, because the feed reports `Galaxy!Area.tag.Attr`. Fixed in `e113faf5` and re-verified live. It also found **#896** (open): the UNS mirror of an alarm-bearing tag is a dead `BadWaitingForInitialData` variable. DeployWatcher on a Galaxy redeploy was not run: no IDE/GRAccess on windev, and the Galaxy was shared with another agent. Evidence: `docs/drivers/evidence/galaxy-g6-e2e-2026-11-12.md`.
- **2026-05-26** — Gap-closeout pass. `OpcUaApplicationHost.PopulateServerArray` populates `Server.ServerArray` (NodeId `i=2254`) with self + `OpcUaApplicationHostOptions.PeerApplicationUris`, giving non-transparent peer URI visibility through the standard discovery surface. New `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/` IT project (`DualEndpointTests`) validates with two real `OpcUaApplicationHost` instances on loopback + a live OPCFoundation client `Session` read. CI `v2-ci.yml` `integration:` job converted to a matrix across `Host.IntegrationTests` + `OpcUaServer.IntegrationTests`. Per-role appsettings overlays shipped (`appsettings.admin.json` / `appsettings.driver.json` / `appsettings.admin-driver.json`) — `Program.cs:33-35` loads by alphabetical-joined role suffix. `FailoverScenarioTests` → `FailoverDuringDeployTests` rename. Stale empty `src/Server/{Server,Admin}` + `tests/Server/{Server.Tests,Admin.Tests,Admin.E2ETests}` directories deleted (no source, absent from `.slnx`).
- **2026-04-24** — Phase 5 driver complement closed (task #120 CLOSED). AB CIP, AB Legacy, TwinCAT, FOCAS all shipped. FOCAS migration: retired the Tier-C split (`Driver.FOCAS.Host` + `Driver.FOCAS.Shared` + `FwlibNative` + shim DLL deleted) in favour of a pure-managed in-process `FocasWireClient` inlined into `Driver.FOCAS`; driver is now read-only against the CNC by design. Integration test matrix grew to cover Browse / Subscribe / IAlarmSource / Probe end-to-end.
- **2026-04-23** — Phase 6.4 audit close-out. IdentificationFolderBuilder + OPC 40010 Identification folder verified against the shipped code.
- **2026-04-20** — Phase 7 plan drafted (`phase-7-scripting-and-alarming.md`, `phase-7-e2e-smoke.md`). Out of scope for v2 GA.
- **2026-04-19** — Release blocker #3 closed (PRs #98–99). Phase 6.3 Streams A + C core shipped: `ClusterTopologyLoader` + `RedundancyCoordinator` + `RedundancyStatePublisher` + `PeerReachabilityTracker`. Code-path release blockers all closed; remaining Phase 6.3 surfaces (peer-probe HostedServices, OPC UA variable-node binding, `sp_PublishGeneration` lease wrap, client interop matrix) are hardening follow-ups.
- **2026-04-19** — Release blocker #2 closed (PR #96). `SealedBootstrap` consumes `ResilientConfigReader` + `GenerationSealedCache` + `StaleConfigFlag`; `/healthz` surfaces the stale flag. Remaining follow-ups (periodic poller + richer snapshot payload) downgraded to hardening.
- **2026-04-19** — Release blocker #1 closed (PR #94). `AuthorizationGate` wired into `DriverNodeManager` Read / Write / HistoryRead dispatch. Remaining Stream C surfaces (Browse / Subscribe / Alarm / Call + finer-grained scope resolution) downgraded to hardening follow-ups — no longer release-blocking.
- **2026-04-19** — Phase 6.4 data layer merged (PRs #91–92). Phase 6 core complete.
- **2026-04-19** — Phase 6.3 core merged (PRs #89–90). `ServiceLevelCalculator` + `RecoveryStateManager` + `ApplyLeaseRegistry` land as pure logic; coordinator / UA-node wiring / Admin UI / interop deferred.
- **2026-04-19** — Phase 6.2 core merged (PRs #84–88). `AuthorizationGate` + `TriePermissionEvaluator` + `LdapGroupRoleMapping` land; dispatch wiring + Admin UI deferred.
- **2026-04-19** — Phase 6.1 shipped (PRs #78–83). Polly resilience + Tier A/B/C stability + health endpoints + LiteDB generation-sealed cache + Admin `/hosts` data layer all live.