docs(claude): refresh CLAUDE.md + record historian-gateway integration/dev-rig state
v2-ci / build (push) Failing after 42s
v2-ci / unit-tests (tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests) (push) Has been skipped
v2-ci / integration (tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests) (push) Has been skipped
v2-ci / integration (tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests) (push) Has been skipped

CLAUDE.md: add scadaproj sister-project index, fix mxaccessgw path to
~/Desktop/MxAccessGateway, note the temporary CVE-2025-6965 NuGetAuditSuppress,
and correct the driver-typed tag-editor list (OpcUaClient is now mapped; only
Galaxy falls back). Also captures docker-dev data-plane GroupToRole + ServerHistorian
live-verify config and the HistorianGateway integration/backlog notes.
This commit is contained in:
Joseph Doherty
2026-07-07 11:46:02 -04:00
parent 4dd47ce707
commit 384dbd7d36
6 changed files with 559 additions and 8 deletions
+45 -2
View File
@@ -17,6 +17,44 @@ service.
See `docs/v2/Galaxy.Performance.md` for the runtime perf surface
(tracing, metrics, soak harness).
## Related Projects (scadaproj umbrella)
This repo is one of a family of related SCADA / OT / Wonderware / OPC UA
**sister projects** that live as sibling directories under `~/Desktop/`. They are
**separate repos and separate processes**, coupled at runtime over wire protocols
(gRPC + OPC UA) — not by project/compile references — and share the
`ZB.MOM.WW.*` product namespace. The common subject is AVEVA System Platform
(Wonderware) "Galaxy" data, and `mxaccessgw` is the linchpin the others connect
through.
- **`~/Desktop/scadaproj`** ([`../scadaproj/CLAUDE.md`](../scadaproj/CLAUDE.md)) —
the umbrella/index workspace that aggregates the whole family (a high-level scan
of each sister project: purpose, location, stack, primary commands). It also hosts
the shared `ZB.MOM.WW.*` libraries — `ZB.MOM.WW.Auth`, `ZB.MOM.WW.Theme`,
`ZB.MOM.WW.Health`, `ZB.MOM.WW.Telemetry`, `ZB.MOM.WW.Configuration`, and
`ZB.MOM.WW.GalaxyRepository`. **This repo is indexed there** — see the OtOpcUa
entry in `../scadaproj/CLAUDE.md`.
- **`~/Desktop/MxAccessGateway`** (`mxaccessgw`,
[`../MxAccessGateway/CLAUDE.md`](../MxAccessGateway/CLAUDE.md)) — the gRPC gateway
that owns the MXAccess 32-bit COM bitness + STA pump (gateway x64 .NET 10 + per-session
x86 net48 worker). **OtOpcUa's `GalaxyDriver` depends on this** for live Galaxy
read/write/subscribe and Galaxy Repository browse.
- **`~/Desktop/HistorianGateway`**
([`../HistorianGateway/CLAUDE.md`](../HistorianGateway/CLAUDE.md)) — single-process
gRPC sidecar exposing the AVEVA Historian read/write API + read-only Galaxy browse
(via the shared `ZB.MOM.WW.GalaxyRepository` lib). Patterned on `mxaccessgw` but with
no COM / no x86 worker. **OtOpcUa's historian backend** (`Driver.Historian.Gateway`)
consumes it as the sole historian path.
- **`~/Desktop/ScadaBridge`**
([`../ScadaBridge/CLAUDE.md`](../ScadaBridge/CLAUDE.md)) — the distributed SCADA
platform (Akka.NET hub-and-spoke) whose Data Connection Layer consumes this server's
OPC UA address space.
**Propagate cross-repo changes to the umbrella index.** When a fact the index records
about OtOpcUa changes here — remote/push status, the driver set, the Galaxy data flow,
shared-lib consumption, or per-project commands — update the **OtOpcUa entry in
`../scadaproj/CLAUDE.md`** in the same change so the index never drifts from this repo.
## Architecture Overview
### Data Flow
@@ -61,7 +99,7 @@ Galaxy `mx_data_type` values map to OPC UA types (Boolean, Int32, Float, Double,
## mxaccessgw
The gateway lives in a sibling repo at `c:\Users\dohertj2\Desktop\mxaccessgw\`. See `docs/v2/Galaxy.ParityRig.md` for the gw setup recipe (build, API key provisioning via `apikey create-key`, env-var overrides for HTTP/2 cleartext + worker path). The gw's MXAccess Toolkit reference (its `gateway.md`) is the canonical MxAccess API doc; the standalone `mxaccess_documentation.md` previously kept in this repo retired in PR 7.3.
The gateway lives in a sibling repo at `~/Desktop/MxAccessGateway/` (`mxaccessgw` — see [Related Projects](#related-projects-scadaproj-umbrella)). See `docs/v2/Galaxy.ParityRig.md` for the gw setup recipe (build, API key provisioning via `apikey create-key`, env-var overrides for HTTP/2 cleartext + worker path). The gw's MXAccess Toolkit reference (its `gateway.md`) is the canonical MxAccess API doc; the standalone `mxaccess_documentation.md` previously kept in this repo retired in PR 7.3.
## Build Commands
@@ -73,6 +111,11 @@ dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests # a single tes
dotnet test --filter "FullyQualifiedName~MyTestClass.MyMethod" # a single test
```
`Directory.Build.props` carries a temporary `NuGetAuditSuppress` for
CVE-2025-6965 / GHSA-2m69-gcr7-jv3q (transitive native `SQLitePCLRaw.lib.e_sqlite3`,
no upstream fix yet) — so restore/build not flagging it is expected; remove the
suppression once a patched bundle ships.
Test projects live under `tests/<module>/` (Core, Server, Drivers,
Drivers/Cli, Client, Tooling) — there is no single unit-test project.
Unit suites are named `*.Tests`; integration suites are `*.IntegrationTests`
@@ -165,7 +208,7 @@ Address pickers in AdminUI support live browse for OpcUaClient and Galaxy driver
The AdminUI's global **UNS** page (`/uns`) is the single surface for managing the unified namespace fleet-wide (Area → Line → Equipment → Tag/VirtualTag), replacing the old per-cluster UNS/Equipment/Tags tabs. See `docs/Uns.md`.
The `/uns` **TagModal** uses **driver-typed tag-config editors**: it dispatches by the bound driver's `DriverType` to a per-driver editor (Modbus/S7/AbCip/AbLegacy/TwinCAT/Focas) via `TagConfigEditorMap`, with client-side validation via `TagConfigValidator`; unmapped drivers (OpcUaClient/Galaxy) fall back to the generic raw-`TagConfig`-JSON textarea. Each editor is a thin razor shell over a pure `<Driver>TagConfigModel` (`FromJson`/`ToJson`/`Validate`, preserves unknown keys). To add a driver's editor, copy the Modbus template under `Components/Shared/Uns/TagEditors/` + `Uns/TagEditors/`, reusing the driver's enums + camelCase JSON property names, and register it in `TagConfigEditorMap` + `TagConfigValidator`. See `docs/plans/2026-06-09-driver-typed-tag-editors-design.md`.
The `/uns` **TagModal** uses **driver-typed tag-config editors**: it dispatches by the bound driver's `DriverType` to a per-driver editor (Modbus/S7/AbCip/AbLegacy/TwinCAT/Focas/OpcUaClient) via `TagConfigEditorMap`, with client-side validation via `TagConfigValidator`; unmapped drivers (only Galaxy) fall back to the generic raw-`TagConfig`-JSON textarea. Each editor is a thin razor shell over a pure `<Driver>TagConfigModel` (`FromJson`/`ToJson`/`Validate`, preserves unknown keys). To add a driver's editor, copy the Modbus template under `Components/Shared/Uns/TagEditors/` + `Uns/TagEditors/`, reusing the driver's enums + camelCase JSON property names, and register it in `TagConfigEditorMap` + `TagConfigValidator`. See `docs/plans/2026-06-09-driver-typed-tag-editors-design.md`.
## Scripting / Script Editor
+183
View File
@@ -0,0 +1,183 @@
# HistorianGateway backend — issues identified
**Date:** 2026-06-27
**Context:** Installed the `ZB.MOM.WW.HistorianGateway` sidecar on **wonder-app-vd03** and updated OtOpcUa to
`origin/master` (`245316d8`, PR #423 "HistorianGateway as the OtOpcUa historian backend"), then smoke-tested
the OtOpcUa→gateway link live. These are the issues found during that work.
> **File/line references are against `origin/master` @ `245316d8`** (the build that contains the gateway
> backend — `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/`). They will not resolve on older
> local checkouts that predate PR #423.
> **Live evidence** came from the gateway's Prometheus `/metrics` on the box: an OPC UA `historyread` on a
> historized tag produced `historian_gateway_grpc_requests_total{method="ReadRaw",service="HistorianRead"}=1`
> and `historian_session_pool_warm_sessions=1`, while a historized-tag **deploy** produced **zero**
> `EnsureTags` calls.
---
## Issue 1 — `GatewayTagProvisioner` is never wired; EnsureTags auto-provisioning is dormant (HIGH, confirmed)
**Summary.** PR #423 ships `GatewayTagProvisioner` (the `IHistorianProvisioning` implementation that calls the
gateway's `EnsureTags`) and unit tests for it, but it is **never registered in DI and never passed into the
`AddressSpaceApplier`**. At runtime the applier uses the no-op `NullHistorianProvisioning`, so **deploying
historized tags does NOT provision them in the historian**. The documented "EnsureTags auto-provisions
historian tags on deploy" behaviour does not occur.
**Root cause (code).**
- `AddressSpaceApplier` constructor coalesces a missing provisioner to the no-op:
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceApplier.cs:63`
`_provisioning = provisioning ?? NullHistorianProvisioning.Instance;`
- The **only** construction site omits the `provisioning:` argument:
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs:258`
```csharp
var applier = new AddressSpaceApplier(
addressSpaceSink,
loggerFactory.CreateLogger<AddressSpaceApplier>(),
historizedSubscriptions: historizedSubscriptions); // <-- no provisioning:
```
- **No `IHistorianProvisioning` / `GatewayTagProvisioner` registration exists anywhere** in the repo
(`grep` over `origin/master` finds only the interface definition, `NullHistorianProvisioning`, and the
`GatewayTagProvisioner` constructor definition itself — no `AddSingleton<IHistorianProvisioning, …>` and
no `new GatewayTagProvisioner(…)`).
- Consequently `ProvisionHistorizedTags` (called unconditionally at
`AddressSpaceApplier.cs:208`, body at `:226`, dispatch at `:261`) builds the request list and invokes
`_provisioning.EnsureTagsAsync(...)` on the **no-op**, which returns without contacting the gateway.
**Live evidence.** On wonder-app-vd03: flagged the `parts-count` equipment tag `isHistorized:true`, deployed,
and restarted the host. The gateway `/metrics` showed **no** gRPC request series for `HistorianTags`/`EnsureTags`
(zero gateway calls). By contrast, an explicit OPC UA `historyread` on the same tag *did* reach the gateway
(`ReadRaw`), confirming the read path is wired but the provisioning path is not.
**Impact.**
- Historized tags are never created in the historian on deploy.
- When **continuous historization** (`ContinuousHistorization:Enabled=true`) is later turned on, the recorder
drains live values to historian tags that were never provisioned — those writes will fail / land nowhere
unless the tags are created by some other means.
- Docs (`docs/Historian.md`, repo `CLAUDE.md`, and the umbrella scadaproj `CLAUDE.md`) describe provisioning
as active; that claim is currently false. See Issue 4.
**Recommended fix (two parts).**
1. **Register the provisioner** in the Host alongside the existing gateway wiring — in
`src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`, inside the `if (serverHistorianOptions.Enabled)` block that
already calls `AddServerHistorian` / `AddAlarmHistorian` — e.g.
```csharp
builder.Services.AddSingleton<IHistorianProvisioning>(sp =>
new GatewayTagProvisioner(
HistorianGatewayClientAdapter.Create(serverHistorianOptions, sp.GetRequiredService<ILoggerFactory>()),
sp.GetRequiredService<ILogger<GatewayTagProvisioner>>()));
```
(mirror the `GatewayHistorianValueWriter` registration pattern already in that block).
2. **Pass it into the applier** at `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs:258`
by resolving it the same way sibling deps are resolved there (`resolver.GetService<…>()`):
```csharp
var provisioning = resolver.GetService<IHistorianProvisioning>();
var applier = new AddressSpaceApplier(
addressSpaceSink,
loggerFactory.CreateLogger<AddressSpaceApplier>(),
provisioning: provisioning,
historizedSubscriptions: historizedSubscriptions);
```
3. **Add a wiring/integration test** asserting `EnsureTags` is invoked on a historized-tag deploy. The existing
`GatewayTagProvisioner` unit tests pass but do not cover the DI wiring, which is exactly the gap that let
this ship dormant.
---
## Issue 2 — Provisioning success is logged nothing; the dormant path is invisible (LOW, observability)
**Summary.** The provisioning continuation in `AddressSpaceApplier` only emits a log line when
`result.Failed > 0 || result.Skipped > 0`. A fully-successful provisioning (`Ensured>0, Failed=0, Skipped=0`)
**and** the no-op `NullHistorianProvisioning` path both log **nothing**. This is what made Issue 1 invisible:
there was no log line either way, so "no provisioning log" could not be distinguished from "provisioning ran
and fully succeeded."
**Location.** `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceApplier.cs` — the
`EnsureTagsAsync(...).ContinueWith(...)` block (~lines 261283); the Information log is gated behind
`if (result.Failed > 0 || result.Skipped > 0)`.
**Recommended fix.** Emit an Information tally on every dispatch (e.g. `requested/ensured/skipped/failed`),
or at minimum whenever `Ensured > 0`, so a successful provisioning is observable and a no-op is detectable.
---
## Issue 3 — HistoryRead on a historized-but-unprovisioned tag blocks for the full CallTimeout (~30s) (MEDIUM, needs investigation)
**Summary.** An OPC UA `historyread` on a tag that is historized but has no corresponding historian
tag/data returned only after ~30 s — the gateway's `ReadRaw` recorded
`historian_gateway_grpc_request_duration_seconds_sum ≈ 30.5` for the single call, matching the
`ServerHistorian:CallTimeout` default of `00:00:30`. The read did not fail fast on an unknown tag; it ran to
the timeout and then errored (`outcome="error"`).
**Notes / attribution.**
- The slow leg is the gateway→historian read of a non-existent tag; the OtOpcUa-facing symptom is a 30 s OPC UA
HistoryRead latency governed by `ServerHistorianOptions.CallTimeout` (default 30 s, `…Runtime/Historian/ServerHistorianOptions.cs`).
- This is **coupled to Issue 1**: because provisioning never runs, freshly-historized tags don't exist in the
historian, so every HistoryRead on them hits this slow path until tags/data exist.
- Single data point (one unprovisioned tag); confirm whether the slow path is the historian, the gateway, or the
client-side timeout before changing behaviour.
**Recommended action.** Primarily fix Issue 1 so historian tags exist. Separately, consider a shorter default
HistoryRead timeout and/or fast-fail when the historian reports an unknown tag, so a missing tag doesn't hang an
OPC UA HistoryRead for 30 s.
---
## Issue 4 — Docs describe provisioning as active (LOW, documentation)
`docs/Historian.md`, the repo `CLAUDE.md`, and the umbrella `scadaproj/CLAUDE.md` state that the gateway backend
"auto-provisions historian tags on deploy (EnsureTags)". Given Issue 1, that is inaccurate in the shipped build.
Update the docs once Issue 1 is fixed, or flag the provisioning behaviour as not-yet-wired in the interim.
---
## Issue 5 — Provisioning covers only *newly-added* tags; flagging an existing tag for historization never provisions it (MEDIUM, follow-up to the Issue 1 fix)
**Status:** found while live-verifying the Issue 1 fix (commit `257214f7`) on wonder-app-vd03.
**Summary.** The Issue 1 fix correctly wires `GatewayTagProvisioner` and is confirmed deployed (the new
`"...provisioning completed (dispatched…"` log literal is present in the deployed `OpcUaServer.dll`). However,
`ProvisionHistorizedTags` still iterates **only** `plan.AddedEquipmentTags`:
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceApplier.cs` (`ProvisionHistorizedTags`):
`foreach (var tag in plan.AddedEquipmentTags)` — `ChangedEquipmentTags` is **not** considered.
So when an operator flags an **existing** tag `isHistorized:true` and deploys, that tag is classified
**Changed** (its config changed; the tag already existed) and is **never provisioned**. This is compounded by
the bootstrap "restore served state" behaviour: a host restart re-applies the served address space
**incrementally**, so an existing tag comes back as *unchanged/changed*, never *Added* — i.e. a restart does
not provision it either.
**Live evidence.** Flagging `parts-count` (an existing tag) historized + deploy + restart produced **zero**
`EnsureTags` calls at the gateway (`/metrics` had no `HistorianTags/EnsureTags` series). The read path on the
same tag *did* reach the gateway (`HistorianRead/ReadRaw`), confirming the tag was historized and the gateway
link works — only provisioning was skipped. (A raw `dbo.Tag` INSERT of a brand-new historized tag did not
materialise either, because a raw insert bypasses the normal compose/validate path — so the clean way to
exercise the *added* path is to author a new historized tag via the AdminUI.)
**Impact.** The most common operator action — take an existing tag and turn on historization — does **not**
auto-provision the historian tag. Continuous historization for such a tag would then write to a tag that does
not exist. Only genuinely **new** historized tags get provisioned.
**Recommended fix.** Provision the union of added **and** newly-historized changed tags: also iterate
`plan.ChangedEquipmentTags` (or compute "tags whose `IsHistorized` transitioned false→true") and include them
in the `EnsureTags` request. `EnsureTags` is idempotent, so re-ensuring an already-present tag is safe. Add a
test that flips an existing tag to historized and asserts `EnsureTags` is invoked.
---
## Pre-existing observation (not introduced by this work)
- **Serilog log path is service-CWD-relative.** On wonder-app-vd03 the host writes to
`C:\Windows\System32\logs\otopcua-<date>.log` (because the service CWD is `System32`), not under the install
directory. This was noted in prior sessions; re-observed here. Not a gateway-integration issue, but it makes
on-box diagnosis awkward and is worth fixing (anchor the Serilog file sink to an absolute/content-root path).
---
## What is working (for context)
The OtOpcUa↔gateway **read and credential path is confirmed working end-to-end** on wonder-app-vd03: an OPC UA
HistoryRead routed through `GatewayHistorianDataSource` to the gateway over h2c, authenticated with a
peppered-HMAC API key (`historian:read` scope), and leased a real historian session (the gateway reached
wonder-sql-vd03). Only the **provisioning** half (Issue 1) is unwired.
+26
View File
@@ -160,6 +160,20 @@ services:
Security__Ldap__ServiceAccountPassword: "serviceaccount123"
Security__DeployApiKey: "docker-dev-deploy-key"
Security__Auth__DisableLogin: "true"
# Map the shared-GLAuth data-plane groups (ReadOnly/WriteOperate/...) to data-plane role
# strings the OPC UA session ACL gates read. The DB LdapGroupRoleMapping is AdminRole-typed
# (AdminUI roles); this free-form GroupToRole baseline is the path for data-plane roles.
Security__Ldap__GroupToRole__ReadOnly: "ReadOnly"
Security__Ldap__GroupToRole__WriteOperate: "WriteOperate"
Security__Ldap__GroupToRole__WriteTune: "WriteTune"
Security__Ldap__GroupToRole__WriteConfigure: "WriteConfigure"
Security__Ldap__GroupToRole__AlarmAck: "AlarmAck"
# A2 live-verify: server-side HistoryRead against the Wonderware sidecar on the WW Historian VM.
# SharedSecret injected via env at `docker compose up` (never committed). Port = the sidecar's TCP listen port.
ServerHistorian__Enabled: "true"
ServerHistorian__Host: "10.100.0.48"
ServerHistorian__Port: "32569"
ServerHistorian__SharedSecret: "${OTOPCUA_HISTORIAN_SECRET:-}"
# Pin EF Core + ASP.NET Core to Warning so the per-poll Deployment SELECT /
# "Executed DbCommand" Information|Debug lines stop flooding the Serilog
# pipeline and starving the Akka cluster heartbeat thread. The host logs via
@@ -201,6 +215,18 @@ services:
Security__Ldap__ServiceAccountPassword: "serviceaccount123"
Security__DeployApiKey: "docker-dev-deploy-key"
Security__Auth__DisableLogin: "true"
# Data-plane group→role baseline (see central-1).
Security__Ldap__GroupToRole__ReadOnly: "ReadOnly"
Security__Ldap__GroupToRole__WriteOperate: "WriteOperate"
Security__Ldap__GroupToRole__WriteTune: "WriteTune"
Security__Ldap__GroupToRole__WriteConfigure: "WriteConfigure"
Security__Ldap__GroupToRole__AlarmAck: "AlarmAck"
# A2 live-verify: server-side HistoryRead against the Wonderware sidecar on the WW Historian VM.
# SharedSecret injected via env at `docker compose up` (never committed). Port = the sidecar's TCP listen port.
ServerHistorian__Enabled: "true"
ServerHistorian__Host: "10.100.0.48"
ServerHistorian__Port: "32569"
ServerHistorian__SharedSecret: "${OTOPCUA_HISTORIAN_SECRET:-}"
# Quiet EF/AspNetCore SQL flood — see central-1 (Serilog override). mem_limit/
# mem_reservation are inherited from the *otopcua-host anchor.
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
@@ -9,7 +9,7 @@
"codeTasks": "T1-T6 all complete; 11 commits; full-solution build 0 errors; all touched suites green; per-task + final integration review READY TO MERGE",
"liveVerify": "T5 fully verified live (inline New script -> dropdown shows new script label). T4 substantially verified (picker opens, live gateway browse works, edited alarm+scale+isHistorized config commits); final Use-this-address pick-and-assert blocked by the Galaxy object browser's href=# name-links navigating away under browser automation (pre-existing picker quirk, unrelated to the SetFullName change; preservation covered by 6 unit tests).",
"findings": "T6 (B1) production already shipped bb59fd4e 2026-06-15 (like #400); delivered missing behavioural test coverage instead. Stray throwaway script SC-ff9c2a3a7044 'vtag-livecheck script' (blank, unbound) left in dev DB during T5 verify; Scripts page has no delete affordance.",
"reconsiderGates": "T7/T8 NOT built (awaiting explicit go-ahead). T9/T10 operator/rig verify. T11 hardware-blocked."
"reconsiderGates": "T7 BUILT+SHIPPED (FB-7, master 30c9821f). T8 NOT built (awaiting explicit go-ahead). T9/T10 DONE (FB-9/FB-10, live-verified). T11 PARTIAL: Phase B native-alarm pipeline PROVEN live 2026-06-19 (Galaxy! ConditionId-prefix finding); Phase C HistoryRead + H6 ack->AVEVA + AbLegacy/FOCAS/TwinCAT probes remain AVEVA/hardware/no-local-target deferred."
},
"tasks": [
{"id": 1, "ref": "A1", "subject": "OpcUaClient history session-capture-before-gate race fix", "classification": "standard", "status": "completed", "commits": ["eb328e57","eede0c92"], "parallelizableWith": [2,3,4,5,6], "group": "A-actionable"},
@@ -22,7 +22,7 @@
{"id": 8, "ref": "B5", "subject": "Alarm-severity SetSeverity surgical update", "classification": "small", "status": "pending", "gate": "RECONSIDER — operationally invisible (alarm engine overwrites); do NOT build without explicit go-ahead", "group": "B-reconsider"},
{"id": 9, "ref": "C1", "subject": "Modbus-Int64 full live authoring on docker-dev", "classification": "verify", "status": "pending", "note": "operator/rig verify-only; seed a Modbus driver", "group": "C-verify"},
{"id": 10, "ref": "C2", "subject": "S7 + AbCip Test-Connect probe happy-path live-verify", "classification": "verify", "status": "pending", "note": "needs lmxopcua-fix fixtures from the Windows VM", "group": "C-verify"},
{"id": 11, "ref": "C3", "subject": "Device-gated proofs (H6 ack→AVEVA, Phase C T7, Phase B T9, AbLegacy/TwinCAT/FOCAS probe)", "classification": "blocked", "status": "pending", "note": "BLOCKED on real hardware; captured only", "group": "C-blocked"}
{"id": 11, "ref": "C3", "subject": "Device-gated proofs (H6 ack→AVEVA, Phase C T7, Phase B T9, AbLegacy/TwinCAT/FOCAS probe)", "classification": "blocked", "status": "partial", "note": "Live-assessed 2026-06-19 (gateway 10.100.0.48:5120 reachable from this Mac, so the 'all blocked' label was partly stale). OUTCOMES: (Phase B T9) DONE — native-alarm PIPELINE proven end-to-end live on docker-dev: authored a Galaxy alarm tag (FullName=Galaxy!TestArea.TestMachine_001.TestAlarm001, alarm{alarmType:OffNormalAlarm,severity:750}) -> Galaxy driver opened the native-alarm subscription (previously-gated path) -> gateway transition materialised Part 9 condition GalaxyNativeAlarm (ns=2;s=EQ-55297329838d/GalaxyNativeAlarm) carrying the EXACT gateway raise time 2026-05-24T16:04:10.856Z, surfaced via ConditionRefresh. KEY FINDING: ConditionId match REQUIRES the 'Galaxy!' source prefix on the authored FullName (without it every transition is silently dropped). Severity 700 = MapSeverity(750) (MediumHigh bucket), not a bug. Net state Inactive/Unacked because the gateway replays weeks-old historical raises -> a fresh Active-state + live /alerts row needs an alarm triggered on the AVEVA side. (Phase C T7) DEFERRED config+data-gated — confirmed the historized Galaxy node materialises + server HistoryRead path returns GoodNoData/0-values with ServerHistorian.Enabled=false (correct NullHistorianDataSource degrade); live data needs enabling ServerHistorian + Wonderware sidecar on 10.100.0.48 + actual stored history (synthetic ref has none). (H6 ack->AVEVA) DEFERRED — ack SEND reachable but Galaxy AckMsg is write-only/fire-and-forget; confirming it LANDED needs AVEVA-side observation on 10.100.0.48. (AbLegacy/FOCAS/TwinCAT probe) DEFERRED — no local sims running + no live-target probe harness; TwinCAT(ADS) genuinely has no rig target. Rig restored (alarm tag removed, Galaxy back to value-only).", "group": "C-blocked"}
],
"lastUpdated": "2026-06-19"
}
+34 -4
View File
@@ -1,6 +1,36 @@
# Pending — open follow-ups & deferrals
As of 2026-06-14. master HEAD `c24abc8a` (synced with origin; feature branch `feat/galaxy-phase-c-historian` ff-merged + deleted). Working tree is clean except the expected DISK-ONLY files: `docker-dev/docker-compose.yml` (M — uncommitted rig config, never staged) and `pending.md` (M — these notes, never staged), plus two untracked pre-existing `docs/plans/2026-06-14-write-outcome-self-correction-plan.md*` docs. HARD RULE: never `git add .`; never stage `pending.md` / `current.md` / `docker-dev/docker-compose.yml` / `sql_login.txt` / `src/Server/ZB.MOM.WW.OtOpcUa.Host/pki/`; never commit secrets.
As of 2026-06-16. master HEAD `1164d423` (synced with origin; feature branch `feat/stillpending-phase-5-probes` ff-merged + deleted). Working tree is clean except the expected DISK-ONLY files: `docker-dev/docker-compose.yml` (M — uncommitted rig config, never staged) and `pending.md` (M — these notes, never staged), plus two untracked pre-existing `docs/plans/2026-06-14-write-outcome-self-correction-plan.md*` docs. HARD RULE: never `git add .`; never stage `pending.md` / `current.md` / `docker-dev/docker-compose.yml` / `sql_login.txt` / `src/Server/ZB.MOM.WW.OtOpcUa.Host/pki/`; never commit secrets.
**CERT-ACTION AUDIT LOGGING + OpcUaProbeResult DEAD-LETTER TIDY MERGED+PUSHED master `3eb370d4` (2026-06-19, ff, 7 commits incl. 2 review-polish).** Branch `feat/cert-audit-and-probe-tidy` ff-merged + deleted. Design `docs/plans/2026-06-19-cert-audit-and-probe-deadletter-tidy-design.md` + plan/`.tasks.json`. Two independent items, subagent-driven (T1 cert-core ∥ T3 probe-drops dispatched concurrently — disjoint projects; T2 wire-in after T1; T1 spec+code review, T3 code review, final integration review = SHIP). **(1) Cert-action audit logging (was stillpending §A.7 deferred):** the FIRST live structured `AuditEvent` emit site. New `ActorAuditWriter : IAuditWriter` (`AdminUI/Audit/`, resolves the `AuditWriterActor` cluster singleton from `ActorRegistry` + fire-and-forget `Tell`, mirrors `AdminOperationsClient`) + pure `CertAuditEvents.Build` factory (`Category:"Certificate"`, `Action` Trust/Untrust/Delete, success rows OMIT the `error` key); `CertificateStoreManager` audits at the choke point on EVERY return path (success + all failures, via a `DeleteCore` refactor); `Certificates.razor` sources the actor from `AuthenticationState`. `AuditEvent` is a `ZB.MOM.WW.Audit` record (init-setters; `AuditOutcome.Success/Failure/Denied`; `CorrelationId` `Guid?`); package ships `NoOpAuditWriter.Instance` (test default). NO EF migration (`ConfigAuditLog` table + `AuditWriterActor` pre-existed), NO Commons/proto change. **(2) OpcUaProbeResult dead-letter tidy (Phase-4 follow-up (i)):** added `Receive<PeerOpcUaProbeActor.OpcUaProbeResult>(_=>{})` drop to `DriverHostActor` (all 3 behaviours: Steady/Applying/Stale), `HistorianAdapterActor`, `ScriptedAlarmHostActor` (mirrors `PeerProbeSupervisor`); guard test fails-without/passes-with. **Build clean; AdminUI.Tests 510/510, Runtime.Tests 273/273.** **LIVE `/run` PASS (agent-driven, login disabled):** rebuilt central-1/central-2 → both boot healthy (DI resolves the new `CertificateStoreManager(IConfiguration, IAuditWriter)` + `ActorAuditWriter`→singleton — the integration review's #1 risk), `AddressSpaceApplier: applied plan (added=91)`, ZERO `OpcUaProbeResult` dead-letters on both nodes (before AND after the run). **CERT-AUDIT END-TO-END PROVEN:** seeded a throwaway rejected cert, clicked Trust on the Certificates page (:9200) → `ConfigAuditLog` row `multi-role-test | Certificate:Trust | Success | <thumbprint> | {"store":"rejected→trusted","thumbprint":…}` (no error key — the success-omit polish), through the full live cluster seam (razor→manager→ActorAuditWriter→AuditWriterActor singleton→shared SQL); seed cert cleaned up. **Session also reconciled #400 (within-timestamp tie-cluster paging — already SHIPPED `2e6c6d3a`/`0f929ae6`, stale pending banners corrected) + RECLASSIFIED H2 (HistoryUpdate service) as INFRA-GATED/DEFERRED** (no historian backend can durably insert/replace/delete — same boundary as H5b; see stillpending §A.1). DEFERRED follow-up (unchanged): cert actions still also want to be reachable via an audit-query UI, but the DB rows now exist. **MINOR review nits (non-arch, from the final integration review — recorded so they're not lost):** (a) two UNREACHABLE `Certificates.razor` fallthrough arms (`"cannot delete from {Kind}"` / `"unknown action"`) build a `CertActionResult.Fail` INSIDE the razor switch, bypassing `CertificateStoreManager` → those (unreachable) failures aren't audited; add a comment or route through the manager; (b) `OpcUa:PkiStoreRoot` is read in BOTH `Certificates.razor:130` and `CertificateStoreManager` ctor — centralize on the manager. Both Minor, no rush.
**STILLPENDING BACKLOG — PHASE 5 (Test-Connect protocol handshakes) MERGED+PUSHED master `1164d423` (2026-06-16, ff, 15 commits) + LIVE `/run` 9/11 PASS.** Branch `feat/stillpending-phase-5-probes` ff-merged + deleted. Design `…-phase-5-probes-design.md` (`1f2d32ac`) + plan `…-probes.md`/`.tasks.json` (`2f7c6bf9`). **Scope locked = ALL 8 TCP-only probes get real handshakes (best-effort, AskUserQuestion).** Replaced bare `socket.ConnectAsync` with a protocol handshake after a TCP preflight, so a live-but-rejecting device reads RED not false-green; **Historian.Wonderware probe was ALREADY a real Hello/HelloAck handshake → untouched** (the §2 "task 9" note was stale). New 3-way contract: TCP-fail→`"Connect failed: {err}"` (unchanged) · handshake-ok→`Ok=true`+latency+descriptive msg · TCP-ok-but-rejected→`"Reachable at {host}:{port} but {proto} handshake failed: …"` (the new behavior) · timeout→`"Probe timed out"` (unchanged). `IDriverProbe`/`DriverProbeResult`/DI/csproj UNCHANGED; NO migration. Per-probe (no shared scaffold — keeps the 8 driver projects disjoint/parallelizable; subagent-driven 2 waves of 4 concurrent, Wave-1 bundle review + Wave-2 bundle review + final integration review = **SHIP**, all 5 invariants pass). **Handshakes:** Modbus FC03 via `ModbusTcpTransport` (exception PDU still=real device) `9b909002`; OpcUaClient `DiscoveryClient.GetEndpointsAsync` (no session/cert) `957a63cf`; S7 `Plc.OpenAsync` COTP+S7-setup `9a8336ff`; AbCip/AbLegacy `libplctag` Tag init (CIP-level error=reachable) `0c08b152`/`21f3e8fe`; TwinCAT `AdsClient.ReadStateAsync` DEGRADE-guarded `b663ae6e`; FOCAS `cnc_allclibhndl3` via a NEW direct `[DllImport("fwlib32")]` DEGRADE-guarded `5ed0276f`; Galaxy gRPC `GalaxyRepository.TestConnection` `2d688c2a`. **DEGRADE principle (TwinCAT/FOCAS): an env that can't ATTEMPT the handshake → `Ok=true` "TCP reachability only" (never worse than TCP); only a real device rejection is RED** — both empirically proven on this headless box (AdsServerException / DllNotFoundException → degrade). Code-review fixes: timeout-CTS bound on OpcUaClient/AbCip/AbLegacy + IPv4 preflight + sw.Stop (`9bfbbb0f`/`5df3c732`). **LIVE `/run` (agent-driven, sandbox-off so the rig is reachable; brought up local central-1 @ :4840 as a real OPC UA server): 9 PASS / 2 SKIP / 0 FAIL** — Modbus green vs sim `10.100.0.35:5020` + red vs non-Modbus; OpcUaClient green vs opc-plc `:50000` AND vs the LOCAL OtOpcUa server (3 endpoints) + red vs SQL/non-OPC-UA; Galaxy green vs gateway `10.100.0.48:5120` + red vs non-gRPC. **S7/AbCip GREEN SKIPPED — fixtures down** (no `lmxopcua-fix` on the Mac; unit-proven + reviewed). **LIVE `/run` CAUGHT + FIXED 2 REAL GALAXY BUGS unit-tests+2-reviews MISSED (`1164d423`): (1) `Retry={MaxAttempts=1}` → invalid Polly RetryStrategyOptions → the probe FAILED on EVERY live gateway (removed it; TCP-preflight+deadline already fail-fast); (2) a rejected key surfaces as a typed `MxGatewayAuthenticationException`, NOT a raw `RpcException`, so "auth-rejection=reachable" was bypassed → catch the typed auth/authorization exceptions → `Ok=true`.** Galaxy UseTls must match the endpoint scheme (`UseTls:false` for `http://` h2c — same constraint the driver enforces). New `DriverProbeHandshakeE2eTests` (skip-gated cross-protocol green/red discrimination). Driver tests all green (Modbus 277, OpcUaClient 122, S7 121, AbCip 276, AbLegacy 168, TwinCAT 145, FOCAS 196, Galaxy 272+3skip). Docs `docs/drivers/TestConnectProbes.md`. **NEW FOLLOW-UPS:** (i) S7+AbCip happy-path live-verify (bring up `lmxopcua-fix up s7 s7_1500`/`up abcip controllogix` from the Windows VM); (ii) AbLegacy/TwinCAT/FOCAS happy-path live-verify (no PLC5/SLC sim, no ADS target, no CNC+FWLIB). **Backlog Phases 68 + Phase 4b remain.**
**STILLPENDING BACKLOG — PHASE 4 (data-type tier: driver data-type & robustness coverage) MERGED+PUSHED master `050f5c4b` (2026-06-16, ff, 10 commits) + LIVE deploy-regression-clean.** Branch `feat/stillpending-phase-4-driver-datatypes` ff-merged + deleted. Design `…-phase-4-driver-datatypes-design.md` (`57d9f1b3`) + plan `…-driver-datatypes.md`/`.tasks.json` (`97c087f2`). **SCOPE NARROWED FROM ROADMAP after grounding** (AskUserQuestion locked both): **S7 wide types DEFERRED to Phase 4b** (driver reads via S7.Net address-strings → wide types need an `S7.Net.Types` byte-buffer read-path refactor; python-snap7 dev sim's wide-type support unconfirmed) and **ALL array work DEFERRED to a dedicated slice** (the live equipment-node materialiser `EnsureVariable` hard-wires `ValueRank=Scalar` with NO array params + `EquipmentTagPlan` carries no array metadata + S7/AbLegacy have no array READ support → surfacing arrays end-to-end is a coherent cross-cutting feature: plan-metadata + `EnsureVariable` contract change + per-driver read + Phase-6 UI, NOT a per-driver flag flip). Shipped 5 changes across 3 disjoint projects (subagent-driven, T1/T2/T4/T5 implementers dispatched CONCURRENTLY — disjoint projects, each commits by path — T3 after T2 same-file; every task spec/code reviewed + final integration review = **SHIP**): **T1 Modbus Int64/UInt64 node type** (`bd8fee61`): `ModbusDriver.MapDataType:1517` split `Int64⇒DriverDataType.Int64`/`UInt64⇒UInt64` (both ALREADY existed in the enum + `ResolveBuiltInDataType` ALREADY mapped `"Int64"`/`"UInt64"` → the gap was purely the driver lying Int32; codec round-trips long/ulong UNCHANGED; removed stale `Driver.Modbus-007` doc) — 271/271. **T2 FOCAS fail-fast factory** (`5c5aaef6`): new `IFocasClientFactory.EnsureUsable()` (no-op Wire/Fake, throws Unimplemented) called early in `InitializeAsync``unimplemented`/`none`/`stub` backend now FAULTS at init not first-read (operator footgun) — 191/191. **T3 FOCAS position scaling** (`49730752`+`88d9e3b1` stale-comment review fix): per-device `PositionDecimalPlaces` config (default 0 = unchanged) divides the 4 position fields by `10^dp` at `PublishAxisSnapshot`; widened `LastFixedSnapshots` `int→double` (the 6 refs through it are all already `Float64`; Int32 fixed-tree vars resolve from separate struct caches → no node-type change); `cnc_getfigure` auto-fetch stays deferred. **T4 Historian Total aggregate** (`5e27b5f7`): `ReadProcessedAsync` Total → query Average + post-multiply each bucket by `interval.TotalSeconds` (time-integral identity; Wonderware AnalogSummary has no Total column); null/bad buckets carried through → BadNoData — 35/35. **T5 Historian poison dead-letter cap** (`fcb38014`): `SqliteStoreAndForwardSink` `maxAttempts` ctor knob (default 10, guard >0) + `QueueRow`/`ReadBatch` carry `AttemptCount` (column pre-existed, NO migration) → RetryPlease dead-letters at `AttemptCount+1>=maxAttempts` (counts row leaving queue, no drift); threaded `AlarmHistorianOptions.MaxAttempts` + `AddAlarmHistorian` DI + `Validate()` warning — 27/27 + Runtime 267/267. **NO contract / `DriverDataType` / `IOpcUaAddressSpaceSink` / EF-migration change.** **Build clean (0 err).** **LIVE `/run`:** rebuilt central-1/central-2 to the branch → both healthy + `Phase7Applier: applied plan (added=90, rebuild=True)` + all materialisation + OPC UA serving, NO fatal/unhandled exception (the new driver DLLs + `AddAlarmHistorian maxAttempts` DI load+deploy clean). **Modbus-Int64 end-to-end node advertisement: UNIT-PROVEN** (271 tests + the equipment-node `"Int64"→DataTypeIds.Int64` resolver) — full live authoring DEFERRED (docker-dev seed has NO Modbus driver; sim `10.100.0.35:5020` reachable from the Mac but not seeded). **NEW FOLLOW-UPS:** (i) Phase-2 `OpcUaProbeResult` INFO dead-letters at `driver-host`/`historian-adapter`/`scripted-alarm-host` (those 3 actors subscribe the redundancy topic but don't drop the probe result like `OpcUaPublishActor` does — benign, capped at 10 then 5-min-suppressed, NOT a Phase-4 change — tidy by adding the same `Receive<OpcUaProbeResult>(_=>{})` drop); (ii) full Modbus-Int64 live authoring; (iii) **Phase 4b** = S7 wide types + Timer/Counter + the dedicated array slice + remaining §2 structural (AbCip/TwinCAT UDT, AbLegacy/TwinCAT bit-index RMW, Galaxy nesting, OpcUaClient event-history, FOCAS cnc_getfigure). **Backlog Phases 58 remain.**
**STILLPENDING BACKLOG — PHASE 3 (OPC UA standards completeness: H4 alarm Enable/Disable + H2 HistoryUpdate permission-bit + H6 native-ack→AVEVA) MERGED+PUSHED master `7eeb9fb0` (2026-06-15, ff, 16 commits) + LIVE deploy-regression-clean.** Branch `feat/stillpending-phase-3-opcua-standards` ff-merged + deleted. Design `…-phase-3-opcua-standards-design.md` (`40b883ef`) + plan `…-standards.md`/`.tasks.json` (`6ab3d863`). Subagent-driven, every task spec+code reviewed (serial for high-risk node-manager/driver/contract tasks) + final integration review = **SHIP** (all 7 cross-cutting checks PASS, incl. the load-bearing native-ack round-trip + node-id seam). **H2-bit** (`c236263e`): `NodePermissions.HistoryUpdate = 1<<12` (int-backed flags enum, NO migration; excluded from all composites — service stays deferred) + `TriePermissionEvaluator:86` maps HistoryUpdate→its own bit (was HistoryRead — closed the read⇒update hole). **H4** (`328bd1b9`+`be6858ba`): wired the missing node-manager `OnEnableDisable` (SDK `ConditionEnableEventHandler(ctx,cond,bool enabling)`, verified via reflection) → scripted = AlarmAck-gated `HandleAlarmCommand("Enable"/"Disable")`→engine (already handled it); native→`BadNotSupported`; native-check via lock-guarded `IsNativeAlarmNode`. **H6**: `418663b3` thread `bool isNative` through the SINGLE shared `MaterialiseAlarmCondition` (Phase7Applier true@:204/false@:295`_nativeAlarmNodeIds`); `ed941c51` `AlarmAcknowledgeRequest.OperatorUser` (additive) + GalaxyDriver forwards + ScriptedAlarmSource honors-else-`opcua-client`; `a6d9de09`+`87dd65b9` native `OnAcknowledge`→NEW `NativeAlarmAckRouter` (AlarmAck-gated `HandleNativeAlarmAck`, principal=DisplayName; scripted ack unchanged→engine); `93d9160d` `DriverHostActor` inverse map `conditionNodeId→(driver,FullName)` (lock-step w/ `_alarmNodeIdByDriverRef`, both keyed by `EquipmentNodeIds.Variable(EquipmentId,FolderPath,Name)`) + primary-gated `RouteNativeAlarmAck``DriverInstanceActor``driver.AcknowledgeAsync` (fire-and-forget, Connected-only); `30315185` host DI maps `NativeAlarmAck→RouteNativeAlarmAck` (Runtime has NO OpcUaServer dep). GOTCHAS: native-check MUST use lock-guarded `IsNativeAlarmNode` (raw set read races a concurrent RebuildAddressSpace — OnTimedUnshelve bypasses the Call-service lock); node-id seam airtight because BOTH sides derive from the same `EquipmentNodeIds.Variable` over the same composition (null/empty FolderPath normalizes identically). NO EF migration; only additive contracts. Also fixed a PRE-EXISTING Phase-0 miss (`8899d6e0`): 2 Galaxy guard tests asserted "PR 4.4"/"PR 4.W" text that Phase 0 `b4af9e7f` removed → updated to current guard text. **Build clean; Core 237/237, OpcUaServer 201/201, Runtime 267/267, Galaxy 264/264(+3 live-gw skip).** **LIVE `/run`:** rebuilt central-1/central-2 to the branch → healthy startup + `Phase7Applier: applied plan (added=90, rebuild=True)` + `scripted alarms materialised (alarms=1)` + OPC UA serving (ServiceLevel 250) — substantial node-manager/DriverHostActor/host changes deploy+materialise+serve with NO regression. **H4 client-Disable BLOCKED: Client.CLI has no enable/disable command** (plan assumed one) → H4 fully unit-proven; CLI enable/disable = a FOLLOW-UP. **H6 full native-ack→AVEVA OPERATOR-GATED**: needs a Galaxy NATIVE alarm (dev rig's 1 alarm is scripted) + the real gateway/AVEVA on `10.100.0.48`; round-trip wiring unit+integration-proven. Phases 48 remain.
**STILLPENDING BACKLOG — PHASE 2 (H3 health-aware redundancy ServiceLevel) MERGED+PUSHED master `4af8e65a` (2026-06-15, ff, 14 commits) + LIVE-PROVEN on the docker-dev 2-node rig.** Branch `feat/stillpending-phase-2-servicelevel` ff-merged + deleted. Design `docs/plans/2026-06-15-stillpending-phase-2-servicelevel-design.md` (`05283533`) + plan `…-servicelevel.md`/`.tasks.json` (`7605f4d8`). **LIVE `/run` ACCEPTANCE PASSED (agent-driven, login disabled):** rebuilt central-1+central-2 to the branch (`docker compose -f docker-dev/docker-compose.yml up -d --build central-1 central-2`); read `Server.ServiceLevel` (NodeId `i=2267`) via Client.CLI on :4840 (leader) + :4841 (follower). **BEFORE (old code): 240 / 100** (legacy role-only). **AFTER (branch, healthy): 250 / 240** (calculator path LIVE — the follower 100→240 + leader 240→250 is the discriminator vs legacy; proves dbHealthProbe Ask delivery + calculator invocation + NodeId self-match + the +10 leader bonus all live). **DB-down (stopped shared `otopcua-dev-sql-1`): 110 / 100** (`DbHealthProbe: probe failed``Stale=!DbReachable` → 100-tier, leader +10=110 — health-aware demotion). **Recovered (SQL restarted): 250 / 240.** Rig left healthy. (NOTE: stopping the SHARED SQL demotes BOTH nodes; single-node asymmetric isolation would need a per-node DB network block the rig doesn't cleanly support, but the both-demote + recover proof + the steady-state 250/240-vs-legacy-240/100 discriminator fully validate the wiring.) Wires the existing-but-never-invoked `ServiceLevelCalculator` into the live per-node publish path so a DB-unreachable / OPC-UA-unreachable node drops below its role level. **Subagent-driven, every task spec+code reviewed (serial for the high-risk actor tasks) + a final integration review = SHIP.** Commits: `ff0f62db` move `ServiceLevelCalculator`+`NodeHealthInputs``Core.Cluster` (`ZB.MOM.WW.OtOpcUa.Cluster.Redundancy`, cycle-free — Runtime already refs Core.Cluster, AVOIDS the deliberate no-Runtime→ControlPlane boundary); `3e609a2b`+`a9ff1a64` `OpcUaPublishActor.RecomputeServiceLevel` calls the calculator (DB+Stale+leader) with a **back-compat legacy seam** (no `dbHealthProbe` wired → old role-only 240/200/100/0, keeps the 2 pre-existing tests green) + Detached/no-entry guard → 0 + always-publish-first-even-0; `cf278035`+`5382eea9` `OpcUaProbeOk` from **peer-probes-me** (consume `OpcUaProbeResult` where NodeId==local; absent/stale verdict→true, only a RECENT observed false demotes); `37b32a56` periodic `HealthTick` Ask→PipeTo the local `DbHealthProbeActor` (Ask-timeout→Reachable=false fail-safe-demote); `f41e957e`+`5a064e08` NEW `PeerProbeSupervisor` spawns one `PeerOpcUaProbeActor` per non-self non-Detached driver peer (monotonic gen-suffix child names + Watch/Terminated-by-ref prune — stale Terminated can't evict a fresh same-peer child); `70e6d3d2`+`393b746d` docs/Redundancy.md WIRED; `4c78dcd3` DI wiring (`dbHealthProbe: dbHealth` into `OpcUaPublishActor.Props` + spawn `PeerProbeSupervisor` + `PeerProbeSupervisorKey`); `4af8e65a` supervisor explicitly drops co-mingled `OpcUaProbeResult` (no unhandled-msg noise). **Tiers (documented, now reachable): 0 down/Detached · 100 DB-down⇒stale · 200 DB-ok-stale · 240 healthy follower · 250 healthy leader. Behavior change: healthy Secondary 100→240** (both healthy nodes 240/250, leader preferred by +10 — clients still pick primary). `Stale = !DbReachable || (now-dbAsOf)>30s || (now-snapshotAsOf)>30s`. **Integration review's #1 risk (NodeId canonical form across the probe→publish seam) = PASS**: both `RedundancyStateActor.ToNodeId(member.Address)` + `ClusterRoleInfo.LocalNode` derive `host:port` from the same `PublicHostname`/`Port` that configures Akka remoting (same class as [[project_redundancy_state_delivery]], consistent here). NO EF migration, NO Commons contract change. **Build clean; ControlPlane.Tests 59/59, Runtime.Tests 264/264.** **LIVE `/run` GATE (Task 8, high-risk acceptance) PENDING:** on docker-dev MAIN (central-1/central-2 2-node warm), read `Server.ServiceLevel` on :4840/:4841 → leader 250 / follower 240; isolate ONE central node from SQL (docker network disconnect — do NOT stop the shared SQL container) → its ServiceLevel drops to 100 (below the healthy peer's 240); reconnect → climbs back. Agent drives docker-dev directly (login disabled). Phases 38 of the backlog remain (each gets its own plan when reached).
**STILLPENDING BACKLOG — PHASE 0 + PHASE 1 (H1+H5) MERGED+PUSHED master `c6a543d1` (ff, 15 commits) 2026-06-15; H1 LIVE-PROVEN on docker-dev.** Then `907005d2` (CLAUDE.md: local rig login-disabled → run /run yourself) + `4bd7180e` (fix docker-dev seed: stop seeding retired SystemPlatform namespace). **LIVE VERIFY:** rebuilt local rig to master, drove AdminUI `localhost:9200` (login disabled), toggled `layer0-logcheck` (EQ-55297329838d) Historize → Deploy → `Phase7Applier: applied plan (added=0, removed=0, changed=1, rebuild=True)` + `sealed (acks=6)` — a PURE changed-only deploy now rebuilds (pre-H1 = `rebuild=False`); reverted = 2nd proof; config pristine. **Blocker found+fixed:** docker-dev seed still inserted `Kind='SystemPlatform'` (retired) → 500s /deployments + fails publish; ran sanctioned CleanupSystemPlatformNamespaces SQL on dev DB + fixed seed. H5 threading flows through real deploy; durable IHistoryWriter.Record silent (NullHistoryWriter infra-gated). (Historical IN-PROGRESS detail below.) Driven by the approved roadmap `docs/plans/2026-06-15-stillpending-backlog-design.md` + plan `…-phase-0-1.md` (full backlog audit = `stillpending.md`). DONE on the branch (subagent-driven, each task spec+code reviewed, all green): **Phase 0 hygiene** (`b4af9e7f` 7 stale comments; `a9d267c9` security.md write-outcome + benign DraftSnapshot/LeaderChanged notes; `50a2fdf3` marked 18 shipped `.tasks.json` completed). **H1 — changed-only deploys now take effect** (`1e95856b` Phase7Applier `needsRebuild` includes `Changed*` counts, ChangedDrivers excluded by design; `1dc71369` count removed tags/vtags in RemovedNodes [review follow-up]; `ada01e1a` VirtualTagHostActor stop+respawns a child whose plan changed in place; `ebf2f1dd` prune `_planByVtag` on termination + crash-then-change test [review follow-up]). **H5 — vtag Historize honored at runtime** (`fc8121cb` `EquipmentVirtualTagPlan.Historize` + composer reads `v.Historize`, in Equals/GetHashCode; `9c5a0913` artifact decode reads `Historize` byte-parity; `83d3b9f7` planner Historize-toggle test + doc nit; `0c6d4c54` VirtualTagHostActor invokes injected `IHistoryWriter` on historized results [+ Runtime→Core.VirtualTags ProjectReference]; `2f30c54d` thread-safe CapturingHistoryWriter [review follow-up]; `4501f12` thread IHistoryWriter through DriverHostActor + DI `TryAddSingleton` Null default). **DURABLE AVEVA vtag-history sink = INFRA-GATED** (no sidecar live-data WriteDataValues RPC) → seam wired w/ NullHistoryWriter default, flagged on the design defer list. NO EF migration (Historize column + ConfigComposer whole-entity serialization already carry it). REMAINING on branch: Task 10 docs (this commit) + Task 11 final build/test/integration-review + finishing (merge is the user's call). LIVE `/run` gate (user-driven): edit equipment/alarm/vtag → confirm it takes effect without restart; toggle a vtag Historize → host respawns + writer invoked.
**#401 — AbCip + FOCAS ALARM-PROJECTION POLL-LOOP LEAK FIXED + MERGED + PUSHED master `151b7165` (2026-06-15, ff, 2 commits `6ba59f9d`+`151b7165`).** SIBLING of #399 found by auditing all `IAlarmSource` drivers, but WORSE: `AbCipAlarmProjection` + `FocasAlarmProjection` each `Task.Run` a poll loop per `SubscribeAsync` and survive in-place reconnect, so the actor's per-Connected re-subscribe leaked a LIVE never-cancelled poll loop each reconnect → CPU growth + DUPLICATE OnAlarmEvent transitions (functional, gated behind opt-in alarm projection). Fix (both projections, symmetric): `SubscribeAsync` snapshots existing subs under `_subsLock`, `_subs.Clear()`, adds the new sub, starts its loop, then `_ = RetireAsync(old)` fire-and-forget (Cancel→await loop→Dispose CTS). Snapshot+clear under the same lock `DisposeAsync` uses ⇒ no double-own/double-dispose (reviewer-confirmed airtight for all interleavings). Implemented via opus subagent (TDD fail→pass: subscribe twice → 1 device raise → exactly ONE event, was 2). AbCip 271/271, FOCAS 186/186. Code-reviewed (sonnet) — correct+safe; applied its 2 nice-to-haves (documented the at-most-ONE-duplicate-per-reconnect one-tick overlap residual in RetireAsync XML doc + guarded Cts.Dispose). **AUDIT RESULT: all 4 IAlarmSource drivers now correct — Galaxy (#399), AbCip+FOCAS (#401), OpcUaClient already safe (session-bound + clears `_alarmSubscriptions` on disconnect).**
**#399 — GALAXY ALARM-SUBSCRIPTION HANDLE LEAK FIXED + MERGED + PUSHED master `01388226` (2026-06-15, ff, 1 commit).** The session-less Galaxy `StreamAlarms` feed survives an in-place reconnect, so `DriverInstanceActor` re-subscribed on every Connected re-entry (dropping its cached handle without an Unsubscribe — sync teardown) and `GalaxyDriver.SubscribeAlarmsAsync` grew `_alarmSubscriptions` by one untracked handle per reconnect (slow unbounded leak; functionally harmless — gate is Count>0, fan-out reads [0] once). Fix: `SubscribeAlarmsAsync` `Clear()`s the set before `Add` (under the existing `_alarmHandlersLock`, atomic w.r.t. the fan-out reader), collapsing to ONE live handle — faithful because there's exactly one consumer per driver instance (factory-per-actor lifecycle). Chosen over async/fire-and-forget actor-side Unsubscribe (disproportionate). NEW regression test `Re_subscribe_collapses_to_a_single_handle_no_accumulation` (TDD-VERIFIED fail-without-fix: releasing the latest handle leaves the feed open if stale handles remain). Galaxy 263 pass/3 skip, Runtime native-alarm 24 pass. Code-reviewed (sonnet) — correct+complete, both nice-to-haves applied (structural-invariant comment + distinct-handle assert).
**#400 — within-timestamp cursor paging for oversized tie clusters: SHIPPED to master `2e6c6d3a` + review fix `0f929ae6` (2026-06-17).** No longer a documented follow-up. `HistoryPaging.SliceTieCluster` over-fetches the whole tie cluster (a bounded `start==end` read), carves out the next `cap` ties at `skip`, advances the cursor WITHIN the timestamp, and steps off it (`T + 1 tick`) once drained — wired into `OtOpcUaNodeManager.ServeRawPaged` (stall detected when a resume page's last sample is still at the boundary timestamp). New config `ServerHistorian:MaxTieClusterOverfetch` (default 65536) bounds the in-memory buffer with a `Validate()` warning; 6 `SliceTieCluster` unit tests + integration tests `Raw_oversized_tie_cluster_pages_within_the_timestamp` / `Raw_tie_cluster_beyond_overfetch_bound_fails_loudly`; `docs/Historian.md` "within-timestamp paging" section + config-table row updated. DELIBERATE residual backstop: a cluster strictly larger than `MaxTieClusterOverfetch` ties at one identical SourceTimestamp still loud-fails `BadHistoryOperationUnsupported` (a severe data anomaly — bounded on purpose to cap memory; raise the knob or `NumValuesPerNode` to cover it).
**#398 — RESTART-DRIVER "actor name is not unique" BUG FIXED + MERGED + PUSHED master `c9643f68` (2026-06-15, ff, 1 commit).** `DriverHostActor.HandleRestartDriver` stopped + respawned the child in ONE synchronous handler reusing the base actor name `drv-<id>`; `Context.Stop` is async so the old child was ALWAYS still registered → `InvalidActorNameException` deterministically on every AdminUI Restart press (crashed+restarted the host). Fix: a monotonic `_childSpawnGeneration` counter feeds a `-g<gen>` suffix on every spawned child name (children tracked by the `_children` dict via IActorRef, never by path, so the suffix is invisible). Also closes the same-shaped LATENT race in the reconcile path (removed-then-readded instance + a driver-type-change `ToStop+ToSpawn` in one plan — confirmed by review). NEW regression test `RestartDriver_respawns_the_child_without_an_actor_name_collision` (TDD-VERIFIED: fails on the old code with the exact `InvalidActorNameException` via `EventFilter.Exception<…>().Expect(0,…)`, passes with the fix). Runtime.Tests 238/238. Code-reviewed (sonnet) — correct+complete, no Critical/Important. `HandleReconnectDriver` confirmed bug-free (only `Tell`s ForceReconnect).
**⚠️ OUT-OF-BAND WORKING-TREE CHANGE I DID NOT MAKE (surfaced to user 2026-06-15, NOT touched):** `Directory.Packages.props` is modified (`ZB.MOM.WW.MxGateway.Client` + `…Contracts` bumped `0.1.0``0.1.1`) and two new `nuget-packages/ZB.MOM.WW.MxGateway.{Client,Contracts}.0.1.1.nupkg` (dated 2026-06-15 05:39) appeared during this session. These are EXTERNAL (likely a sibling-repo publish / parallel action) — NOT part of my B2 or #398 commits (both clean, verified). Left untouched per the "didn't-create-it → surface, don't act" rule; awaiting user decision whether to commit, revert, or ignore. `nuget-packages/` is NOT gitignored (tracked dir).
**B4 — docker-dev rig cleanup RESOLVED 2026-06-15 as "LEAVE-AS-IS" (user decision).** The rig keeps this session's verification artifacts on purpose: the historized dev `GalaxyTestTag` (= `TestMachine_002.TestHistoryValue`) + the local-only `docker-dev/docker-compose.yml` env block (`ServerHistorian__*` with `SharedSecret` via `${OTOPCUA_HISTORIAN_SECRET:-}`, + the data-plane `Security__Ldap__GroupToRole__*` map) so A2 (historian) + A3 (native alarms) + B1 (write-outcome) stay re-verifiable. `docker-compose.yml` stays an uncommitted DISK-ONLY file (never staged). No teardown, no commit — matches the earlier "leave the Galaxy verification artifacts as-is" decision. **All buckets (A2/A3 live, B1/B2/B3 code, B4 operational) are now CLOSED.** Remaining are logged follow-ups only: #398 (pre-existing Restart actor-name-not-unique), #399 (minor session-less alarm handle leak), #400 (within-timestamp cursor paging — now SHIPPED `2e6c6d3a`/`0f929ae6` 2026-06-17).
**B2 — HISTORIAN HISTORYREAD CONTINUATION-POINT PAGING (Raw) MERGED + PUSHED to master `aa1e21f5` (2026-06-15, fast-forward, 3 commits: `94c3ca60` feature, `bea0b482` review fixes, `aa1e21f5` doc).** Server-side time-based paging for the only count-capped arm (Raw): full page (count==NumValuesPerNode>0) ⇒ opaque 16-byte CP + tie-safe resume cursor (`HistoryPaging.ComputeResumeCursor`/`TrimBoundaryDuplicates` — resume from boundary timestamp INCLUSIVE, drop already-emitted head ties → no dup/skip); short page / `==0` (all values) ⇒ no CP. CPs bound to the OPC UA session via the SDK store (`SessionHistoryContinuationStore`, capped by `ServerConfiguration.MaxHistoryContinuationPoints`=100 oldest-evict, disposed on session close); tests inject `InMemoryHistoryContinuationStore` (in-process harness is session-less). `HistoryReleaseContinuationPoints` override frees cursors; unknown/evicted/released ⇒ `BadContinuationPointInvalid`. Processed/AtTime stay single-shot (no client count cap → no full-page signal; spec-conformant). NEW: `HistoryPaging.cs`, `HistoryContinuationStore.cs` + `HistoryPagingTests.cs` (15) + `NodeManagerHistoryReadPagingTests.cs` (7, boot a real SDK server). **182/182 OpcUaServer tests pass, full solution builds clean.** Code-reviewed (sonnet subagent): **C1 (critical) FIXED** — an oversized boundary tie cluster (more ties at one SourceTimestamp than NumValuesPerNode) could silently truncate a resume to GoodNoData and DROP the un-emitted ties (the (timestamp,skip) cursor can't advance past a single timestamp the fixed-(start,end,cap) backend keeps re-returning); now fails LOUDLY per node with `BadHistoryOperationUnsupported` + a log + `docs/Historian.md` "Paging limitation" note + regression test `Raw_tie_cluster_larger_than_page_fails_loudly_not_silently`. Also I3 (Save after HistoryData build → no orphaned cursor), N1 (dropped never-produced `HistoryReadKind` enum + Processed-only state fields — only Raw pages), N3 (ComputeResumeCursor empty-page guard), I1 (documented test-store eventual-consistency). Full within-timestamp cursor paging = task #400, SHIPPED `2e6c6d3a`/`0f929ae6` (2026-06-17) — `HistoryPaging.SliceTieCluster` + `MaxTieClusterOverfetch`-bounded over-fetch; only clusters beyond the bound still loud-fail by design. **Modified-value history (`HistoryReadModified`) is INFRA-GATED, not a code gap**: the AVEVA Wonderware backend (`IHistorianDataSource` TCP sidecar) has no modified-read path; server cleanly rejects per node + `IsReadModified` honoured; doc note added. **B2 COMPLETE.** (B1 write-outcome residuals + B3 GroupToRole doc already on master `a5c0c826`/`063d004f`.)
**GALAXY LIVE-GW INTEGRATION SMOKES (reopen + write-outcome) MERGED + PUSHED to master `d19deb9b` (2026-06-14, fast-forward, 2 commits).** Closed the two items the user re-opened with a live gateway: the harden-1b **Galaxy-reopen** integration test + the **write-outcome** Galaxy write-commit/persist proof. NEW skip-gated `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests/Runtime/GatewayGalaxyLiveReopenAndWriteTests.cs` (gated on `MXGW_ENDPOINT` + `GALAXY_MXGW_API_KEY`, like `GatewayGalaxyAlarmFeedLiveTests`; **skips clean in CI** — verified). **LIVE-PROVEN 2/2 against `http://10.100.0.48:5120` (2026-06-14):** (1) `Live_reopen_recreates_session_and_re_establishes_write_handles` — a live write seeds the writer's item-handle + supervisory-advise caches, then the EXACT `GalaxyDriver.ReopenAsync` primitive pair runs (`GalaxyMxSession.RecreateAsync``GatewayGalaxyDataWriter.InvalidateHandleCaches`), both caches drop to 0, and a post-reopen write re-establishes them + returns Good — proving `RecreateAsync` yields a working live session (the un-fakeable part the unit suite can't reach); (2) `Live_supervisory_write_commits_and_persists_to_galaxy` — writes a value distinct from current, then reads it back through a BRAND-NEW session — proving the no-login (`WriteUserId=0`) write COMMITS + PERSISTS only because of `AdviseSupervisory` (the `f05b5d79` fix). Code-review APPROVED (refinement applied: read-back completes an explicit `TaskCompletionSource<double?>` from the stream pump; `bufferedUpdateIntervalMs:0`). **DURABLE RUN RECIPE:** the gw `10.100.0.48:5120` IS reachable from this Mac; source the key WITHOUT echoing it — `KEY=$(docker exec otopcua-dev-central-1-1 printenv GALAXY_MXGW_API_KEY); MXGW_ENDPOINT=http://10.100.0.48:5120 GALAXY_MXGW_API_KEY="$KEY" dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests --filter FullyQualifiedName~GatewayGalaxyLiveReopenAndWrite` (central-1's container env carries a valid key while its galaxy session is connected; check with `docker exec … sh -c 'test -n "$GALAXY_MXGW_API_KEY" && echo SET'`). Writes only `TestMachine_002.TestFloat` (the established write-demo attr).
**RESIDUAL FOLLOW-UPS CLEANUP MERGED + PUSHED to master `fa2388ea` (2026-06-14, fast-forward, 7 commits = 1 plan doc + 6 test/doc).** Closed the offline-doable, non-blocking residuals from open-item #3 (write-pipeline / Phase B / Galaxy-nits sections) — **TESTS + DOCS only, ZERO production change.** Plan `docs/plans/2026-06-14-residual-followups-cleanup-plan.md`. **KEY RECONCILIATION FINDING:** a grounding sweep (3 Explore agents + direct verify) proved MOST of the listed residuals were ALREADY implemented in the harden-1b / write-outcome / `f05b5d79` work — pending.md's residual list was STALE. **Verified already-done (no work needed, file:line evidence):** WP(a) stale/reconnecting writes fast-fail (`DriverHostActor.cs:661` `Receive<RouteNodeWrite>`, `DriverInstanceActor.cs:229,316` `Receive<WriteAttribute>`); WP(b) `ExecuteSynchronously` already dropped (`DriverHostActor.cs:630` = `TaskContinuationOptions.None`) + `driverIds` forward-map already a `HashSet`; WP(c) FOCAS address cached via `GetOrAdd` (`FocasDriver.cs:247`); WP(d) raw-protocol-blob write test exists (`Primary_routes_write_for_raw_protocol_blob_tag`); WP(e) `[InlineData(2,false)]` future-enum trap already in the parity test; Galaxy(1) `_itemHandles`+`_supervisedHandles` cleared on reconnect (`GatewayGalaxyDataWriter.InvalidateHandleCaches()``GalaxyDriver.ReopenAsync`); Galaxy(2) `SubscriptionEstablished` self-Tell handled (debug-swallow) in all 3 `DriverInstanceActor` states (`:253,:301,:336`). **ACTUALLY DONE this pass (4 items):** Phase B(a) regression test that a `NativeAlarmRaised` during `Reconnecting` is DROPPED not dead-lettered (uses an `AllDeadLetters` EventStream probe so it genuinely guards removal of the `DriverInstanceActor.cs:345` drop handler — the first cut used `parent.ExpectNoMsg` only, which a code review caught as a false-pass gap); Phase B(b) test that `OperatorComment``AlarmTransitionEvent.Comment` (and `User=="device"`) through `DriverHostActor.ForwardNativeAlarm`; Phase B(c) severity-bucket doc note in `docs/ScriptedAlarms.md` (each transition snaps to 200/500/700/900 via `NativeAlarmProjector.MapSeverity`, overriding the authored seed, then SDK `EventSeverity` brackets via `OtOpcUaNodeManager.MapSeverity`); Galaxy(3) extracted shared `tests/.../Runtime.Tests/Drivers/StubDrivers.cs` harness (de-dup of the superset `StubDriver`/`WritableStubDriver`/`SubscribableStubDriver`/`StubHandle` from `DriverInstanceActorTests` + `DriverInstanceActorWriteAndSubscribeTests`; `BlockingInitDriver` left local). Built via subagent-driven dev (1 implementer bundle → code-review APPROVED w/ 1 Important = the dead-letter-probe gap, fixed). **Build clean; Runtime.Tests 236/0.** (The harden-1b Galaxy-reopen integration test noted here as deferred is **NOW DONE** — see the Galaxy live-gw smokes banner above, `d19deb9b`.)
**GALAXY PHASE C — SERVER-SIDE OPC UA HistoryRead MERGED + PUSHED to master `c24abc8a` (2026-06-14, fast-forward, 14 commits = 2 design/plan docs + 12 feature/test/doc).** The server now answers OPC UA **HistoryRead** (Raw / Processed / AtTime over historized variable nodes; **Events** over alarm-owning equipment-folder event-notifier nodes) for any equipment tag flagged historized, driver-agnostically, by dispatching to the registered `IHistorianDataSource` (the Wonderware historian TCP client, which already implemented that interface). **NO EF migration** — the flag rides in the existing `TagConfig` JSON blob (`{"FullName":"…","isHistorized":true,"historianTagname":"…"?}`, the Phase-B `alarm`-object carrier), `historianTagname` defaults to the tag's driver `FullName`. Design/plan `docs/plans/2026-06-14-galaxy-phase-c-historian-*.md`; guide `docs/Historian.md`. Pipeline: `Phase7Composer.ExtractTagHistorize` + byte-parity `DeploymentArtifact.ExtractTagHistorize``EquipmentTagPlan.{IsHistorized,HistorianTagname}``Phase7Applier` resolves `IsHistorized ? (HistorianTagname ?? FullName) : null` → sink seam (`IOpcUaAddressSpaceSink.EnsureVariable` +`string? historianTagname`) → `OtOpcUaNodeManager.EnsureVariable` sets `Historizing`+`AccessLevels.HistoryRead`+registers `_historizedTagnames`; the node manager overrides the four `CustomNodeManager2` HistoryRead virtuals, block-bridging to the `HistorianDataSource` property (volatile, default `NullHistorianDataSource.Instance`). DI mirrors `AlarmHistorian`: `AddServerHistorian` (config-gated, `ServerHistorian` appsettings section, Null default via `TryAddSingleton`) + `OtOpcUaSdkServer.SetHistorianDataSource` + Host `Program.cs`/`OtOpcUaServerHostedService` Start/Stop wiring. **Graceful degrade:** historized node + Null/unconfigured source → `Good_NoData` (empty), non-historized node → `BadHistoryOperationUnsupported`. **KEY FACTS/GOTCHAS:** the SDK base filters event reads by the `EventNotifier.HistoryRead` bit (variable nodes never reach the events arm); the SDK master propagates `errors[i].Code → results[i].StatusCode` (confirmed by decompiling `MasterNodeManager.HistoryReadAsync`), so the override signals per-node status via `errors[handle.Index]`; the two `HistoryReadResult` types (SDK `Opc.Ua.HistoryReadResult` vs Core.Abstractions DTO) are aliased `SdkHistoryReadResult`/`HistorianRead`; `ReadRawModifiedDetails.IsReadModified` defaults TRUE (Initialize() sets it) so a plain raw read must clear it — modified-history is unsupported; the events arm registers folder→sourceName (= equipment id) only when a non-Null historian is wired at promotion time (Host wires the source at StartAsync before any deployment materialises, so normal boot ordering is correct). Built via subagent-driven dev (T1/T2/T5 standard parallel review, T3/T4 high-risk serial spec→code, + a final integration review READY-TO-MERGE). **Build clean (0 errors); OpcUaServer.Tests 152/0, Runtime.Tests 234/0, Core.Abstractions.Tests 88/0.** **LIVE `/run` GATE (T7) DEFERRED — operator-driven: it needs the Wonderware sidecar + AVEVA Historian on the WW Historian VM `10.100.0.48`, which is NOT on the local docker-dev rig.** When run: author a historized Galaxy tag (`TagConfig` `"isHistorized":true`), set `ServerHistorian:Enabled=true` → sidecar (Host/Port/SharedSecret/TLS), deploy on `MAIN-galaxy-eq`, then `Client.CLI historyread -n "ns=2;s=<equip>/<tag>" --start … --end …` → samples; a non-historized tag → `BadHistoryOperationUnsupported`. KNOWN follow-ups (non-blocking, documented): single-shot reads only (no server-managed continuation-point paging yet); no modified-value history; no explicit timeout at the block-bridge call site (bounded by the `WonderwareHistorianClient` 30s `CallTimeout`); the StopAsync→DisposeAsync warm-shutdown window returns Good_NoData (mirrors the `SetNodeWriteGateway` pattern).
@@ -10,11 +40,11 @@ As of 2026-06-14. master HEAD `c24abc8a` (synced with origin; feature branch `fe
**WRITE-OUTCOME SELF-CORRECTION (#5) MERGED + PUSHED to master `1d797c1c` (2026-06-14, fast-forward, 6 commits).** A failed inbound device write now reverts the node to its real pre-write value (compare-and-revert) instead of leaving the optimistic-`Good` phantom. Design `docs/plans/2026-06-14-write-outcome-self-correction-design.md`; plan `…-plan.md`. NEW `IOpcUaNodeWriteGateway`/`NodeWriteOutcome` (Commons) + `ActorNodeWriteGateway` (Runtime, Asks `RouteNodeWrite`, returns the outcome) replace the fire-and-forget `Action` router; `OnEquipmentTagWrite` captures the prior value + fires an off-Lock (`RunContinuationsAsynchronously`) continuation that reverts on a failed outcome IF the node still holds the optimistic value (`ShouldRevert`). Build clean; Commons 39/0, Runtime 201/0, OpcUaServer 111/0. High-risk review verified the prior-value capture against the actual UA-.NETStandard source (`OnWriteValue` fires before `m_value=value`). **LIVE-PROVEN end-to-end** via a local Modbus exception-injector (FC06 reject on HR[20]): authorized failing write → gateway-logged `0x808B0000` reject → node reverts 99→20; authorized success (HR200=7777) stays; anon → BadUserAccessDenied. KEY FINDING: the **Galaxy gateway worker's `ExecuteWrite` is fire-and-forget** (returns OK without awaiting the MXAccess commit), so Galaxy writes ALWAYS return Success at the OPC UA layer and can NEVER surface a device-write failure to this revert — only protocol drivers (which await + return real status) can; this is the same gateway-side limitation noted under "optimistic-write phantom" (out of our scope). Two MINOR deferred follow-ups remain: a Bad-quality blip / OPC UA AuditWriteUpdateEvent on failure, and synchronous structural fail-fast (both explicitly out of scope per the chosen mechanism).
**HARDEN MILESTONE 1b cluster MERGED + PUSHED to master `945c2380` (2026-06-14, fast-forward, 9 commits).** Follow-ups #3 (data-plane role docs), #4 (write-pipeline review nits), and #6 (Galaxy driver nits) below are CLOSED. Plan: `docs/plans/2026-06-14-harden-milestone-1b-plan.md`. Build clean; Runtime 197/0, FOCAS 185/0, Galaxy 257/0(+1 pre-existing skip); final integration review READY-TO-MERGE. Two MINOR residual follow-ups surfaced by review (both deferred, non-blocking): (a) a *driver-level* regression test that `GalaxyDriver.ReopenAsync` actually calls `InvalidateHandleCaches` — needs a live gw (`RecreateAsync` can't be faked), so it's an integration test; (b) stub-driver test-class duplication between `DriverInstanceActorTests` + `DriverInstanceActorWriteAndSubscribeTests` (hygiene — extract a shared harness).
**HARDEN MILESTONE 1b cluster MERGED + PUSHED to master `945c2380` (2026-06-14, fast-forward, 9 commits).** Follow-ups #3 (data-plane role docs), #4 (write-pipeline review nits), and #6 (Galaxy driver nits) below are CLOSED. Plan: `docs/plans/2026-06-14-harden-milestone-1b-plan.md`. Build clean; Runtime 197/0, FOCAS 185/0, Galaxy 257/0(+1 pre-existing skip); final integration review READY-TO-MERGE. Two MINOR residual follow-ups surfaced by review (non-blocking): (a) a *driver-level* regression test that `GalaxyDriver.ReopenAsync` actually calls `InvalidateHandleCaches` STILL DEFERRED, needs a live gw (`RecreateAsync` can't be faked), so it's an integration test; (b) stub-driver test-class duplication between `DriverInstanceActorTests` + `DriverInstanceActorWriteAndSubscribeTests` **DONE `fa2388ea`** (shared `StubDrivers.cs`).
## STATE SUMMARY (post-compaction pickup)
**ALL feature work is SHIPPED + PUSHED to master `c24abc8a` (synced with origin). Nothing is blocking.** Milestone 1b (equipment-tag live values: live READ + authorized inbound WRITE across OpcUaClient / the 6 protocol drivers / Galaxy, via the `FullName→NodeId` router) is COMPLETE, and all three Galaxy phases shipped: **A** standard Equipment driver `c3c56172`, **B** native alarms `f9be3843`, **C** server-side HistoryRead `c24abc8a`. The session's cluster-harden / write-outcome-self-correction / driver-reconfigure-while-faulted follow-ups merged too (`945c2380` / `1d797c1c` / `56f73e49`). The six banners above carry each feature's mechanism + gotchas + deferred live gate; the closed open-follow-ups #1#7 (Phase B, Phase C, data-plane role docs, write-pipeline nits, write-outcome, Galaxy driver nits, reconfigure-while-faulted) are all DONE at those SHAs.
**ALL feature work is SHIPPED + PUSHED to master `d19deb9b` (synced with origin). Nothing is blocking.** Milestone 1b (equipment-tag live values: live READ + authorized inbound WRITE across OpcUaClient / the 6 protocol drivers / Galaxy, via the `FullName→NodeId` router) is COMPLETE, and all three Galaxy phases shipped: **A** standard Equipment driver `c3c56172`, **B** native alarms `f9be3843`, **C** server-side HistoryRead `c24abc8a`. The session's cluster-harden / write-outcome-self-correction / driver-reconfigure-while-faulted follow-ups merged too (`945c2380` / `1d797c1c` / `56f73e49`). The six banners above carry each feature's mechanism + gotchas + deferred live gate; the closed open-follow-ups #1#7 (Phase B, Phase C, data-plane role docs, write-pipeline nits, write-outcome, Galaxy driver nits, reconfigure-while-faulted) are all DONE at those SHAs.
**The ONLY genuinely open items (all user-driven / deferred — pick up here):**
@@ -23,7 +53,7 @@ As of 2026-06-14. master HEAD `c24abc8a` (synced with origin; feature branch `fe
- **Phase B native alarms** (T9) — author a Galaxy alarm tag whose `FullName` == the gateway `AlarmFullReference`, deploy on `MAIN-galaxy-eq`, trip → Part 9 condition + `/alerts` row. Recipe: Phase B banner.
- **Driver-reconfigure-while-faulted** — fault `MAIN-opcua-eq` with a bad config, deploy a corrected one (`POST http://localhost:9200/api/deployments`, `X-Api-Key: docker-dev-deploy-key`), confirm it adopts WITHOUT a node restart. Recipe: that banner.
2. **Rig cleanups** (operational, user-deferred) — see "Operational deferral" at the bottom.
3. **Minor non-blocking residual follow-ups** (review-surfaced, all explicitly deferred, none gate anything): Phase B residuals (a)(d) in its banner; write-outcome residuals (Bad-quality blip / AuditWriteUpdateEvent / synchronous fail-fast); harden-1b two residuals (`945c2380` banner: Galaxy-reopen integration test, stub-driver test-class de-dup); Phase C documented follow-ups (no continuation-point paging, no modified-value history, block-bridge timeout bounded only by the client's 30s `CallTimeout`); the data-plane `GroupToRole` production-default note; Galaxy `_itemHandles`/`_supervisedHandles` not cleared on reconnect + the cosmetic `SubscriptionEstablished` self-dead-letter.
3. **Minor non-blocking residual follow-ups** — MOSTLY CLOSED by the residual-followups-cleanup merge (`fa2388ea`; see the top banner for full reconciliation). DONE this pass: Phase B residuals (a)/(b)/(c) + the stub-driver de-dup. VERIFIED already-implemented (were stale in this list): all of write-pipeline (a)(e), Galaxy `_itemHandles`/`_supervisedHandles` clear-on-reconnect, and the `SubscriptionEstablished` self-dead-letter handler. **STILL OPEN (deferred, none gate anything):** write-outcome residuals (Bad-quality blip / AuditWriteUpdateEvent / synchronous fail-fast — design-deferred OPC UA surface, NOT the Galaxy write-commit proof which is now live-tested `d19deb9b`); Phase C documented follow-ups (no continuation-point paging, no modified-value history, block-bridge timeout bounded only by the client's 30s `CallTimeout`); Phase B residual (d) (inbound device-ack etc., DEFERRED by design); the data-plane `GroupToRole` production-default note.
**No queued feature remains** — Milestone 1b + Galaxy A/B/C were the headline deliverables and are all done. Future directions (NOT requested): the Phase C HistoryRead follow-ups above, or new driver/UNS work.
+269
View File
@@ -0,0 +1,269 @@
# Still Pending — RECONCILED 2026-06-18
> **★ UPDATE 2026-06-19 (master `a325ec54`) — NON-ARCH FOLLOW-UPS BATCH (A/B) SHIPPED.** Six code
> tasks merged via `feat/followups-batch`: OpcUaClient stale-session race (re-resolve session inside
> `_gate` in 5 history/subscribe/ack paths); Client.CLI `enable`/`disable` condition commands (H4
> client path); cert-audit review nits; Galaxy re-pick preserves edited alarm/historize/scale via
> pure `TagConfigJson.SetFullName`; inline-script dropdown label reorder fix. **The "write-outcome
> residuals" item (Bad blip + AuditWriteUpdateEvent + sync fail-fast) was found ALREADY-SHIPPED in
> `bb59fd4e` (2026-06-15) — delivered the missing behavioural test coverage instead.** Live-`/run`:
> T5 PASS; T4 substantial (picker + live gateway browse confirmed; final pick-and-assert blocked by
> the Galaxy object-browser `href=#` nav quirk under automation — preservation unit-tested).
> **RECONSIDER GATES still OPEN-by-choice (NOT built): F10b surgical DataType/IsArray writes;
> alarm-severity `SetSeverity`.** See `docs/plans/2026-06-19-followups-batch*` + memory.
>
> **Reconciled 2026-06-18** against master HEAD `651018f9` by a 7-way parallel per-subsystem
> code audit of the original 2026-06-15 backlog (which was generated at the older HEAD `151b7165`).
> Three days and a large run of "Phase 4 / H1H6 / driver" commits landed in between, so **most of
> the original backlog is now SHIPPED.** Every item below was re-verified in code (commit cited).
>
> **Legend**`OPEN` (genuine, actionable code gap) · `DEFERRED` (intentional scope boundary —
> needs infra, a future version, an SDK/gateway/sidecar change, or a by-design decision) ·
> `SHIPPED` (the cited gap no longer exists; recorded for the audit trail so it is not re-flagged).
---
## A. Genuinely OPEN — actionable code work (ranked)
1. **H2 — OPC UA `HistoryUpdate` service** — **RECLASSIFIED INFRA-GATED / DEFERRED (user decision 2026-06-19).**
The server-side override (mirror the 5 HistoryRead overrides: `HistoryUpdateData`/`HistoryUpdateEvents`/
`HistoryDeleteRawModified`/`HistoryDeleteAtTime`/`HistoryDeleteEvent`, resolve nodes via the same
`TryGetHistorizedTagname`/event-notifier maps, gate on the already-shipped `NodePermissions.HistoryUpdate =
1<<12` / commit `c236263e`) is buildable + offline-testable, BUT **no backend can durably write history**:
`IHistorianDataSource` is read-only (Raw/Processed/AtTime/Events + health — no insert/replace/delete) and
the AVEVA Wonderware TCP sidecar exposes no history-update RPC (the same permanent SDK/sidecar boundary as
H5b durable-write + no-modified-reads). So any `IHistoryUpdateSink` wired today could only reject — a custom
override would mostly restate the SDK's own default `HistoryUpdate` rejection. **Deferred** until a writable
historian backend exists; if/when one does, build the full service surface + the `IHistoryUpdateSink` seam
(Phase-C shape) at that time. _Moved out of §A "Genuinely OPEN" — see the Infra-gated OPEN note below + §B._
2. ~~**OpcUaClient — `IHistoryProvider.ReadEventsAsync`** event-history passthrough~~ — **SHIPPED master
`7a3d2712` 2026-06-18** (`feat/opcuaclient-read-events`). NO interface change (the "param extension" framing
was overcautious): the driver builds its OWN canonical 6-field BaseEventType `EventFilter` and maps the
upstream `HistoryEvent``HistoricalEvent` (the server re-projects only those 6 fields). Driver-internal;
live-proven vs opc-plc (wire path; non-empty events infra-gated — opc-plc isn't a historian).
3. ~~**FOCAS — `cnc_getfigure` wire command**~~ **SHIPPED master (branch
`feat/focas-figure-ww-poison-abcip-gate`) 2026-06-18.** `FocasWireClient.ReadPositionFiguresAsync`
(wire command `0x00D3`, self-delimiting per-axis `short dec` payload) + the `WireFocasClient`
stub wired to it; matching `focas_mock` `_wire_payload` handler + `position_figures` default-0 state.
The managed wire backend (production path to real CNCs) now auto-fetches per-axis figures; the
`PositionDecimalPlaces` knob is the graceful per-axis fallback (`FocasDriver.SafeProbe` → empty on
failure → knob, so it cannot regress). **LIVE-PROVEN** against the local focas-mock
(`Position_figures_scale_axis_via_wire_backend`: 12345 ÷ 10^3 = 12.345, auto figure beat the knob).
Real-Fanuc wire-fidelity stays bench-CNC-gated (whole-backend caveat, not unique to this command).
4. ~~**Galaxy — writer item-handle cache not shared with the subscription registry**~~
**SHIPPED master `8480e301` 2026-06-18.** `SubscriptionRegistry.TryResolveItemHandle` (authoritative
forward lookup) + writer `subscribedHandleSource` delegate → first write to an already-subscribed tag
borrows the live handle, skipping the redundant `AddItem`. Live-proven vs `10.100.0.48:5120` (3/3).
5. ~~**Historian.Wonderware — poison-event sidecar wire protocol**~~ **SHIPPED master (branch
`feat/focas-figure-ww-poison-abcip-gate`) 2026-06-18.** Additive `[Key(4)] byte[] PerEventStatus`
(0=Ack/1=Retry/2=Permanent) on `WriteAlarmEventsReply` (BOTH in-repo IPC ends — MessagePack/TCP, NOT a
Commons proto; `PerEventOk [Key(3)]` kept for rolling-deploy back-compat). The sidecar
`HistorianFrameHandler` now classifies **structurally-malformed** events (empty SourceName/AlarmType,
`EventTimeUtcTicks ≤ 0`) as **Permanent** and excludes them from the writer batch; the client
`WriteBatchAsync` prefers `PerEventStatus` (→ `PermanentFail` → the sink's existing immediate
dead-letter) and falls back to `PerEventOk`. **NO `IAlarmEventWriter` / Commons / Core.Abstractions
change** (`HistorianWriteOutcome.PermanentFail` already existed). Offline-proven (sidecar classifier
tests + client round-trip + sink). _Documented boundary (follow-up):_ an SDK-**semantic** permanent
rejection of a structurally-valid event still maps to Retry→cap until the infra-gated
`AahClientManagedAlarmEventWriter` surfaces richer per-event status.
6. ~~**AbCip — nested-struct member expansion in production**~~ **SHIPPED master (branch
`feat/hosts-rows-abcip-nested-hygiene`) 2026-06-18** (`3d8ce4e8`+`d203f31c`). FINDING: simpler than this
item assumed — the CIP Template Object member-info `u16` ALREADY carries the nested template id for a
struct member (bit 15 = struct flag, lower 12 bits = template instance id, same encoding as the Symbol
Object; the decoder's own remarks documented it). The prior fix DISCARDED it (`templateInstanceId: null`).
The close: carry `uint? NestedTemplateId` on `AbCipUdtMember` (decoded via the FULL 12-bit mask, not the
byte-cast typeCode), thread it into the existing `FetchUdtShapeAsync`/`@udt/{id}` path — NO new CIP query.
Offline-proven (decoder + no-seam threading test, AbCip 301/301). **Live-gate FORMALIZED 2026-06-18
(branch `feat/focas-figure-ww-poison-abcip-gate`):** a skip-gated Emulate-tier integration smoke
`AbCipEmulateNestedUdtTests` (mirrors `AbCipEmulateUdtReadTests` + `AbServerProfileGate.SkipUnless(Emulate)`)
asserts nested leaves addressable; `docs/drivers/AbCip.md` documents nested-struct as Emulate-tier
verified. A LOCAL gate is architecturally impossible — `ab_server` lacks CIP Template Object (class 0x6C);
the test compiles + skips locally and runs only against Logix Emulate / real ControlLogix.
7. ~~**AdminUI — Certificates page actions**~~ **SHIPPED master `bd791e79` 2026-06-18.** FleetAdmin-gated
Trust (rejected→trusted) / Untrust (trusted→rejected) / Delete via a new in-process `CertificateStoreManager`
(filesystem moves by thumbprint, honored live); inline Blazor confirm + server-side FleetAdmin re-check.
Live-proven on docker-dev. **CERT-ACTION AUDIT LOGGING SHIPPED master `3eb370d4` (2026-06-19):**
Trust/Untrust/Delete now write a `ConfigAuditLog` row (`Certificate:Trust|Untrust|Delete`, Outcome
Success/Failure on every path incl. failures) via the first live structured `AuditEvent` emit site —
new `ActorAuditWriter : IAuditWriter` (AdminUI adapter Telling the `AuditWriterActor` cluster singleton
resolved from `ActorRegistry`, mirrors `AdminOperationsClient`) + pure `CertAuditEvents.Build` factory,
audited at the `CertificateStoreManager` choke point (`DeleteCore` refactor); actor sourced from
`AuthenticationState`. NO EF migration (table + actor pre-existed), NO Commons/proto change.
LIVE-PROVEN end-to-end on docker-dev: a Trust on the Certificates page wrote
`multi-role-test | Certificate:Trust | Success | <thumbprint>` to `ConfigAuditLog`. (Was: only cert
*validation* went to Serilog.)
8. ~~**AdminUI — Hosts page per-driver-instance rows**~~ **SHIPPED master (branch
`feat/hosts-rows-abcip-nested-hygiene`) 2026-06-18** (`cb062fce`+`6457eba8`+`07bd63f8`). A new
cluster-grouped "Driver Instances" section on `/hosts` reusing the existing driver-health DPS pipeline
(added only an AdminUI-internal `IDriverStatusSnapshotStore.GetAll()` + a pure `HostsDriverView.Build`
grouping view-model + the Razor section) — NO Commons/interface/EF change. Grouped by `ClusterId` (the
reliable key; the health feed carries no per-Akka-member identity), nodes from ConfigDB `ClusterNode`,
driver rows enriched with Name/DriverType from `DriverInstance`, live status chips via the in-process
store's `SnapshotChanged` (dodges the Traefik self-hub trap) + the 5 s timer. LIVE `/run` PASS (both
central nodes rebuilt; MAIN cluster shows Galaxy/Modbus/OpcUaClient HEALTHY enriched + grouped; SITE-A/B
show nodes + "No drivers"). DEFERRED follow-up: TRUE per-Akka-member nesting (needs a hosting-node field
on `DriverHealthChanged` — a Commons change — or a per-node `DriverHostActor`-children Ask).
9. ~~**AdminUI driver-pages plan — Phase 6 + Phase 10**~~ **SHIPPED — RECONCILED 2026-06-18.** A
seam-by-seam audit confirmed the WHOLE driver-pages plan (Phases 010) is shipped; §A.9 was stale
(it over-counted from the stale `.tasks.json`). Phase 6 (live `DriverStatusPanel`) + Phase 8
(Reconnect/Restart) are wired end-to-end (DriverHealthChanged → AkkaDriverHealthPublisher invoked at
`DriverInstanceActor.PublishHealthSnapshot` → admin-gated `DriverStatusSignalRBridge` → snapshot store
→ panel in all 9 driver pages; page `DriverInstanceId` ⟷ actor `_driverInstanceId` keys match). Phase
10 E2E tests (DriverTestConnect/DriverReconnect/DriverStatusHub) shipped at `494da22c`. The one genuine
remnant — a **deployed-driver Healthy→Reconnecting→Healthy E2E** + the harness-fidelity fix
(`AddOtOpcUaRuntime` so harness drivers actually publish health) + a controllable fake driver factory —
SHIPPED on `feat/driver-pages-reconnect-e2e` (`docs/plans/2026-06-18-driver-pages-reconnect-e2e.md`),
driver E2E suite GREEN 18/0/4. The `.tasks.json` is now reconciled. DEFERRED follow-ups: full-stack
WebSocket+JWT DriverStatusHub test (no repo precedent; mock-hub test + manual runbook cover it); the
§8.3 manual browser smoke recording (10.4); the dotnet-test-testhost-can't-reach-remote-docker skip
quirk on this dev Mac.
10. ~~**AdminUI — Monaco `ctx.SetVirtualTag(...)` completions**~~ **SHIPPED master `70e1bde9` 2026-06-18**
(predates this audit — was stale here). The `ctx`-receiver guard is in
`ScriptAnalysisService.TryGetTagPathLiteral` (`:224`, mirrors the `ctx`-anchored
`EquipmentScriptPaths.GetTagRefRegex`) + a truthful `ctx.SetVirtualTag` hover note (it's a no-op in the
live single-tag evaluator).
11. ~~**F10b — in-place property writes vs full rebuild**~~ **FIRST SLICE SHIPPED master (branch
`feat/f10b-vtag-skip-rebuild-galaxy-comment`) 2026-06-18** (`6c6a2c42`+`b05c2811`). The provably-safe,
zero-node-mutation slice: `Phase7Applier.Apply` now SKIPS `RebuildAddressSpace` when a deploy's only
changes are VirtualTag deltas differing *solely* in `Expression`/`DependencyRefs`/`Historize` — a vtag's
materialised node depends ONLY on `{EquipmentId,FolderPath,Name,DataType}` (Historize is write-side
only), and the vtag ENGINE adopts the edit via `VirtualTagHostActor`'s INDEPENDENT respawn
(`DriverHostActor``ApplyVirtualTags`), so editing a script no longer tears down + rebuilds the whole
server address space (preserving every client's subscriptions + live values). Whitelist-of-may-differ
(`(prev with {those 3 = cur's}).Equals(cur)`) defaults to REBUILD for any other/structural change;
`EnsureVariable`'s existing-node early-return preserves the live node when no rebuild runs. RecordingSink
unit tests pin the boundary (skip cases + DataType/Name/FolderPath/EquipmentId/mixed ⇒ rebuild).
Code-review verified all 6 safety claims; no H1a stale-node regression.
**SECOND SLICE (surgical tag-attribute writes) SHIPPED master (branch `feat/f10b-surgical-tag-attribute-writes`)
2026-06-18** (`3fc258bd`+`f6618144`+`b472bba3`+fix `22b0611f`): equipment-tag **`Writable`** +
**`IsHistorized`/`HistorianTagname`** toggles now update the EXISTING node IN PLACE (AccessLevel + write
handler + Historizing + historian-map + `ClearChangeMasks`) via a new optional `ISurgicalAddressSpaceSink`
(`OtOpcUaNodeManager.UpdateTagAttributes`, mirrors `EnsureVariable` via a shared `ComposeAccessLevel`),
instead of a full rebuild — `Phase7Applier.TagDeltaIsSurgicalEligible` (no-alarm value variable, only those
3 differ) with rebuild fallback for anything else / missing node / non-surgical sink. **LIVE-PROVEN** on
docker-dev (toggled FastUInt1 isHistorized → `changed=1, surgicalTags=1, rebuild=False` on BOTH central
nodes, server healthy). **CRITICAL bug the live-run caught (all unit tests + 3 reviews missed it): the
optimization was INERT in prod** — actors inject `DeferredAddressSpaceSink` (the runtime swap-in wrapper),
which forwarded `IOpcUaAddressSpaceSink` but NOT `ISurgicalAddressSpaceSink`, so `_sink is ISurgical…` was
false → silent rebuild fallback (`rebuild=True, surgicalTags=0`). Fixed by forwarding the surgical
capability through the wrapper (`22b0611f`) + a regression test. _Still DEFERRED (dirty for rare edits /
no value): surgical `DataType`/`IsArray` writes (brief value-type mismatch; no ModelChangeEvents) +
alarm-severity `SetSeverity` (live-shadowed by the alarm engine, operationally invisible)._
12. ~~**Client.UI — Galaxy sub-attribute fallback collapses operator comments**~~ **CLOSED as a documented +
tested ACCEPTED-LIMITATION master (branch `feat/f10b-vtag-skip-rebuild-galaxy-comment`) 2026-06-18**
(`fac44187`). Investigation confirmed it is NOT code-fixable: Galaxy's `AckMsg` attribute is WRITE-ONLY
(no readable attribute to recover the last comment) and the OPC UA event SelectClause carries no comment
field — the operator comment is only available via the gateway's NATIVE alarm feed
(`GalaxyAlarmTransition.OperatorComment`, the Galaxy driver path), not this OPC UA client fallback. Closed
with a clarifying inline comment at the fallback `AlarmEvent?.Invoke` site + a tightened
`AlarmEventArgs.OperatorComment` doc naming both concrete reasons + a pinning test
(`OnAlarmEvent_GalaxyFallback_LeavesOperatorCommentNull`) so the null behavior is intentional + regression-guarded.
A behavioral fix would need a server-side change (out of scope).
13. ~~**§9 hygiene — Galaxy stale forward-ref comments**~~ **SHIPPED master (branch
`feat/hosts-rows-abcip-nested-hygiene`) 2026-06-18** (`9cf88e33`). All 5 sites
(`GalaxyDriver.cs:52,92,669`, `GalaxyDriverFactoryExtensions.cs:19`, `Health/HostStatusAggregator.cs:21`)
rewritten to the shipped standard-driver reality (each claim re-verified against the live code:
`_dataReader` reads via `ReadViaSubscribeOnceAsync`, `_ownedMxSession` built in
`BuildProductionRuntimeAsync`, `SyncPlatformsAsync` refreshes the probe watcher post-discovery, only
`GalaxyMxGateway` exists — no `Galaxy:Backend` switch, host-status re-raise wired in the ctor). The
unshipped "PR 4.W"/`Galaxy:Backend`/"legacy-host" forward-refs are gone. Comment-only; §9 now fully clean.
> **Infra-gated OPEN (code path exists or is buildable; only a no-op until a deployment supplies the backend):**
> **H5b — production `IHistoryWriter` for equipment-vtag historization.** The `Historize` flag is now
> threaded end-to-end (commits `fc8121cb`/`9c5a0913`/`4501f126`) but DI defaults to `NullHistoryWriter`
> no durable AVEVA sink is written/wired. Same infra-gated pattern as the alarm historian: writing a real
> sink is possible but only exercisable against the historian backend.
> **H2 — OPC UA `HistoryUpdate` service** (reclassified 2026-06-19, see §A.1). Server-side override is
> buildable but no historian backend can durably insert/replace/delete — `IHistorianDataSource` is read-only
> and the Wonderware sidecar has no update RPC; build the service + `IHistoryUpdateSink` seam when a writable
> backend exists.
---
## B. DEFERRED — intentional scope boundaries (not actionable without infra / future version)
**Driver write/array boundaries (reads shipped):**
- Array **writes** across all 5 drivers (OPC UA layer forces array nodes read-only — `Phase7Applier`
`writable = tag.Writable && !tag.IsArray`, commit `3bb2031d`).
- S7 wide-type **arrays** (Int64/UInt64/LReal/String/DateTime as arrays); S7 **Timer/Counter writes**
(read shipped); Modbus String/BitInRegister **array writes**.
- AbCip **whole-UDT bare-container** read (`BadNotSupported`) + write (throws) — operators address
individual member paths. `Structure→DriverDataType.String` placeholder retained.
- TwinCAT **pre-declared `Structure` tags** rejected at parse (discovery is the supported path);
**Flat-mode `SubSymbols` live caveat** — on some TC3 firmware Flat mode may not populate struct
SubSymbols → switch `SymbolsLoadMode.Flat``VirtualTree` (deliberately NOT changed; needs a real ADS
target to verify; would risk regressing the Phase 4c array discovery).
- AbLegacy I/O-image bit writes are NOT pre-rejected — they RMW and surface the real PCCC device status.
**Gateway-scope (mxaccessgw sibling repo, not this repo):**
- Galaxy `ExecuteWrite` fire-and-forget — the .NET writer fully propagates `WriteResult` status; the
invisible-failure concern is gateway-side.
**Reserved seams / future versions (no behavioral gap today):**
- `AuthorizationVerdict.Denied` (v2.1 explicit-deny), `NamespaceKind.Simulated` (replay driver),
`Script.Language` (multi-engine), `IDriverConfigEditor` (superseded by `TagConfigEditorMap`),
`DraftSnapshotFactory` placeholders (conservative fallbacks), Cluster `LeaderChanged` no-op
(ServiceLevel lives in `RedundancyStateActor`).
- AdminUI Monaco **InlayHints** endpoint — intentional no-op (`[]`).
- Historian **no modified-value history** (sidecar has no modified-read RPC); **Processed/AtTime
single-shot** (spec-conformant, no count cap).
- Redundancy **`ServerUriArray`** child of `Server.ServerRedundancy` — pending an SDK object-type upgrade
(read `Server.ServerArray` today).
- VirtualTags **serialized eval** under `_evalGate` (decision #19); **unbounded script memory / post-timeout
thread leak** — out-of-process runner is a v3 concern.
---
## C. Live `/run` verification gates — code done, user/operator-driven
- Galaxy Phase C historian T7 live HistoryRead (needs Wonderware sidecar + AVEVA on `10.100.0.48`).
- Galaxy Phase B native alarms T9 live `/run` (equivalent proof already achieved via `dcb0be65` + `/alerts`).
- Driver-reconfigure-while-faulted T3 live `/run`.
- AVEVA Historian rig smokes (skip-gated `rig-required` tests).
---
## D. SHIPPED since 2026-06-15 (audit trail — do NOT re-flag)
**§1:** H1a changed-only-deploy rebuild (`1e95856b`) · H1b vtag-respawn (`ada01e1a`) · H2 permission bit
(`c236263e`) · H3 health-aware ServiceLevel now live (`3e609a2b`/`70e6d3d2`) · H4 OPC UA Enable/Disable
(`328bd1b9`) · H5a Historize flag threaded (`fc8121cb`/`9c5a0913`) · H6 native-ack→AVEVA + authenticated
identity (`418663b3`/`ed941c51`/…).
**§2 drivers:** S7 wide scalars Int64/UInt64/LReal/String/DateTime r+w (`06b858eb`/`286be5df`/`5db08e9e`) ·
S7 Timer/Counter read (`8cfb8e92`) · Modbus Int64/UInt64 node type (`DriverDataType.Int64/UInt64`) ·
Wonderware `Total` aggregate (client-side `ScaleAverageToTotal`) · cross-driver 1-D array **reads** all 5
drivers (Phase 4c `a82c22c6`/`f4d5a5ee`/`95006939`/`3e742395`/`8d3dc321`) · Modbus String/Bit array read ·
AbCip controller-discovered UDT **member paths** (`4e141402`/`4a7b0fde`) · TwinCAT discovered struct
member expansion (`3699fc16`) · AbLegacy B/I/O bit writes via RMW (`5c6b7cd6`) · TwinCAT BOOL-within-word
RMW (`a73e20fb`) · FOCAS axis scaling logic + knob (`49730752`/`6855be28`) · FOCAS fail-fast on
unimplemented backend (`5c5aaef6`) · Galaxy parent-gobject nesting (`21c7645d`/`bec37848`) · all 4 (+more)
Test-Connect probes upgraded TCP→protocol handshake (`961b2b55` & per-driver).
**§4/§5 AdminUI:** typed TagConfig editors for OpcUaClient + Historian.Wonderware (`72d414ad`) ·
isHistorized/historianTagname first-class UI (`c00a5471`) · UNS-tree delete Enterprise+Cluster
(`526eebb3`) · create-new-script from inline vtag panel (`c98625fd`) · native-alarm `HistorizeToAveva`
opt-out + UI knob (`6a8020e7`) · Galaxy picker pre-fills native-alarm fields (`069a5f31`) · Client.UI
alarm Acknowledge + Shelve + Confirm surfaces (`4f55d894`/`60da9224`/`a774bb08`/`6a61f35a`).
**§6/§7:** Historian #400 within-timestamp tie-cluster paging (`2e6c6d3a`/`0f929ae6`) · per-cluster-scoping
plan COMPLETE (`.tasks.json` reconciled) · historian-tcp-transport plan COMPLETE (stale json) ·
adminui-followups plan COMPLETE — role-mapper (RoleGrants.razor + `LdapGroupRoleMappingService` +
`FleetAdmin` + `LdapGroupRoleMapping` entity/migration) + typed editors all shipped · adminui-driver-pages
Phases 7 (probes) + 9 (9 typed address pickers) shipped · Wonderware poison-event SQLite dead-letter half
(`fcb38014`).
**§9 stale comments corrected:** DriverHostActor, OpcUaPublishActor, ServiceCollectionExtensions,
IScriptLogPublisher/ScriptLogTopicSink, ScriptAnalysisService, docs/security.md (6 of 7 — only the Galaxy
PR-4.W forward-refs remain; see A.13).