docs(claude): refresh CLAUDE.md + record historian-gateway integration/dev-rig state
v2-ci / build (push) Failing after 42s
v2-ci / unit-tests (tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests) (push) Has been skipped
v2-ci / unit-tests (tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests) (push) Has been skipped
v2-ci / integration (tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests) (push) Has been skipped
v2-ci / integration (tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests) (push) Has been skipped

CLAUDE.md: add scadaproj sister-project index, fix mxaccessgw path to
~/Desktop/MxAccessGateway, note the temporary CVE-2025-6965 NuGetAuditSuppress,
and correct the driver-typed tag-editor list (OpcUaClient is now mapped; only
Galaxy falls back). Also captures docker-dev data-plane GroupToRole + ServerHistorian
live-verify config and the HistorianGateway integration/backlog notes.
This commit is contained in:
Joseph Doherty
2026-07-07 11:46:02 -04:00
parent 4dd47ce707
commit 384dbd7d36
6 changed files with 559 additions and 8 deletions
+34 -4
View File
@@ -1,6 +1,36 @@
# Pending — open follow-ups & deferrals
As of 2026-06-14. master HEAD `c24abc8a` (synced with origin; feature branch `feat/galaxy-phase-c-historian` ff-merged + deleted). Working tree is clean except the expected DISK-ONLY files: `docker-dev/docker-compose.yml` (M — uncommitted rig config, never staged) and `pending.md` (M — these notes, never staged), plus two untracked pre-existing `docs/plans/2026-06-14-write-outcome-self-correction-plan.md*` docs. HARD RULE: never `git add .`; never stage `pending.md` / `current.md` / `docker-dev/docker-compose.yml` / `sql_login.txt` / `src/Server/ZB.MOM.WW.OtOpcUa.Host/pki/`; never commit secrets.
As of 2026-06-16. master HEAD `1164d423` (synced with origin; feature branch `feat/stillpending-phase-5-probes` ff-merged + deleted). Working tree is clean except the expected DISK-ONLY files: `docker-dev/docker-compose.yml` (M — uncommitted rig config, never staged) and `pending.md` (M — these notes, never staged), plus two untracked pre-existing `docs/plans/2026-06-14-write-outcome-self-correction-plan.md*` docs. HARD RULE: never `git add .`; never stage `pending.md` / `current.md` / `docker-dev/docker-compose.yml` / `sql_login.txt` / `src/Server/ZB.MOM.WW.OtOpcUa.Host/pki/`; never commit secrets.
**CERT-ACTION AUDIT LOGGING + OpcUaProbeResult DEAD-LETTER TIDY MERGED+PUSHED master `3eb370d4` (2026-06-19, ff, 7 commits incl. 2 review-polish).** Branch `feat/cert-audit-and-probe-tidy` ff-merged + deleted. Design `docs/plans/2026-06-19-cert-audit-and-probe-deadletter-tidy-design.md` + plan/`.tasks.json`. Two independent items, subagent-driven (T1 cert-core ∥ T3 probe-drops dispatched concurrently — disjoint projects; T2 wire-in after T1; T1 spec+code review, T3 code review, final integration review = SHIP). **(1) Cert-action audit logging (was stillpending §A.7 deferred):** the FIRST live structured `AuditEvent` emit site. New `ActorAuditWriter : IAuditWriter` (`AdminUI/Audit/`, resolves the `AuditWriterActor` cluster singleton from `ActorRegistry` + fire-and-forget `Tell`, mirrors `AdminOperationsClient`) + pure `CertAuditEvents.Build` factory (`Category:"Certificate"`, `Action` Trust/Untrust/Delete, success rows OMIT the `error` key); `CertificateStoreManager` audits at the choke point on EVERY return path (success + all failures, via a `DeleteCore` refactor); `Certificates.razor` sources the actor from `AuthenticationState`. `AuditEvent` is a `ZB.MOM.WW.Audit` record (init-setters; `AuditOutcome.Success/Failure/Denied`; `CorrelationId` `Guid?`); package ships `NoOpAuditWriter.Instance` (test default). NO EF migration (`ConfigAuditLog` table + `AuditWriterActor` pre-existed), NO Commons/proto change. **(2) OpcUaProbeResult dead-letter tidy (Phase-4 follow-up (i)):** added `Receive<PeerOpcUaProbeActor.OpcUaProbeResult>(_=>{})` drop to `DriverHostActor` (all 3 behaviours: Steady/Applying/Stale), `HistorianAdapterActor`, `ScriptedAlarmHostActor` (mirrors `PeerProbeSupervisor`); guard test fails-without/passes-with. **Build clean; AdminUI.Tests 510/510, Runtime.Tests 273/273.** **LIVE `/run` PASS (agent-driven, login disabled):** rebuilt central-1/central-2 → both boot healthy (DI resolves the new `CertificateStoreManager(IConfiguration, IAuditWriter)` + `ActorAuditWriter`→singleton — the integration review's #1 risk), `AddressSpaceApplier: applied plan (added=91)`, ZERO `OpcUaProbeResult` dead-letters on both nodes (before AND after the run). **CERT-AUDIT END-TO-END PROVEN:** seeded a throwaway rejected cert, clicked Trust on the Certificates page (:9200) → `ConfigAuditLog` row `multi-role-test | Certificate:Trust | Success | <thumbprint> | {"store":"rejected→trusted","thumbprint":…}` (no error key — the success-omit polish), through the full live cluster seam (razor→manager→ActorAuditWriter→AuditWriterActor singleton→shared SQL); seed cert cleaned up. **Session also reconciled #400 (within-timestamp tie-cluster paging — already SHIPPED `2e6c6d3a`/`0f929ae6`, stale pending banners corrected) + RECLASSIFIED H2 (HistoryUpdate service) as INFRA-GATED/DEFERRED** (no historian backend can durably insert/replace/delete — same boundary as H5b; see stillpending §A.1). DEFERRED follow-up (unchanged): cert actions still also want to be reachable via an audit-query UI, but the DB rows now exist. **MINOR review nits (non-arch, from the final integration review — recorded so they're not lost):** (a) two UNREACHABLE `Certificates.razor` fallthrough arms (`"cannot delete from {Kind}"` / `"unknown action"`) build a `CertActionResult.Fail` INSIDE the razor switch, bypassing `CertificateStoreManager` → those (unreachable) failures aren't audited; add a comment or route through the manager; (b) `OpcUa:PkiStoreRoot` is read in BOTH `Certificates.razor:130` and `CertificateStoreManager` ctor — centralize on the manager. Both Minor, no rush.
**STILLPENDING BACKLOG — PHASE 5 (Test-Connect protocol handshakes) MERGED+PUSHED master `1164d423` (2026-06-16, ff, 15 commits) + LIVE `/run` 9/11 PASS.** Branch `feat/stillpending-phase-5-probes` ff-merged + deleted. Design `…-phase-5-probes-design.md` (`1f2d32ac`) + plan `…-probes.md`/`.tasks.json` (`2f7c6bf9`). **Scope locked = ALL 8 TCP-only probes get real handshakes (best-effort, AskUserQuestion).** Replaced bare `socket.ConnectAsync` with a protocol handshake after a TCP preflight, so a live-but-rejecting device reads RED not false-green; **Historian.Wonderware probe was ALREADY a real Hello/HelloAck handshake → untouched** (the §2 "task 9" note was stale). New 3-way contract: TCP-fail→`"Connect failed: {err}"` (unchanged) · handshake-ok→`Ok=true`+latency+descriptive msg · TCP-ok-but-rejected→`"Reachable at {host}:{port} but {proto} handshake failed: …"` (the new behavior) · timeout→`"Probe timed out"` (unchanged). `IDriverProbe`/`DriverProbeResult`/DI/csproj UNCHANGED; NO migration. Per-probe (no shared scaffold — keeps the 8 driver projects disjoint/parallelizable; subagent-driven 2 waves of 4 concurrent, Wave-1 bundle review + Wave-2 bundle review + final integration review = **SHIP**, all 5 invariants pass). **Handshakes:** Modbus FC03 via `ModbusTcpTransport` (exception PDU still=real device) `9b909002`; OpcUaClient `DiscoveryClient.GetEndpointsAsync` (no session/cert) `957a63cf`; S7 `Plc.OpenAsync` COTP+S7-setup `9a8336ff`; AbCip/AbLegacy `libplctag` Tag init (CIP-level error=reachable) `0c08b152`/`21f3e8fe`; TwinCAT `AdsClient.ReadStateAsync` DEGRADE-guarded `b663ae6e`; FOCAS `cnc_allclibhndl3` via a NEW direct `[DllImport("fwlib32")]` DEGRADE-guarded `5ed0276f`; Galaxy gRPC `GalaxyRepository.TestConnection` `2d688c2a`. **DEGRADE principle (TwinCAT/FOCAS): an env that can't ATTEMPT the handshake → `Ok=true` "TCP reachability only" (never worse than TCP); only a real device rejection is RED** — both empirically proven on this headless box (AdsServerException / DllNotFoundException → degrade). Code-review fixes: timeout-CTS bound on OpcUaClient/AbCip/AbLegacy + IPv4 preflight + sw.Stop (`9bfbbb0f`/`5df3c732`). **LIVE `/run` (agent-driven, sandbox-off so the rig is reachable; brought up local central-1 @ :4840 as a real OPC UA server): 9 PASS / 2 SKIP / 0 FAIL** — Modbus green vs sim `10.100.0.35:5020` + red vs non-Modbus; OpcUaClient green vs opc-plc `:50000` AND vs the LOCAL OtOpcUa server (3 endpoints) + red vs SQL/non-OPC-UA; Galaxy green vs gateway `10.100.0.48:5120` + red vs non-gRPC. **S7/AbCip GREEN SKIPPED — fixtures down** (no `lmxopcua-fix` on the Mac; unit-proven + reviewed). **LIVE `/run` CAUGHT + FIXED 2 REAL GALAXY BUGS unit-tests+2-reviews MISSED (`1164d423`): (1) `Retry={MaxAttempts=1}` → invalid Polly RetryStrategyOptions → the probe FAILED on EVERY live gateway (removed it; TCP-preflight+deadline already fail-fast); (2) a rejected key surfaces as a typed `MxGatewayAuthenticationException`, NOT a raw `RpcException`, so "auth-rejection=reachable" was bypassed → catch the typed auth/authorization exceptions → `Ok=true`.** Galaxy UseTls must match the endpoint scheme (`UseTls:false` for `http://` h2c — same constraint the driver enforces). New `DriverProbeHandshakeE2eTests` (skip-gated cross-protocol green/red discrimination). Driver tests all green (Modbus 277, OpcUaClient 122, S7 121, AbCip 276, AbLegacy 168, TwinCAT 145, FOCAS 196, Galaxy 272+3skip). Docs `docs/drivers/TestConnectProbes.md`. **NEW FOLLOW-UPS:** (i) S7+AbCip happy-path live-verify (bring up `lmxopcua-fix up s7 s7_1500`/`up abcip controllogix` from the Windows VM); (ii) AbLegacy/TwinCAT/FOCAS happy-path live-verify (no PLC5/SLC sim, no ADS target, no CNC+FWLIB). **Backlog Phases 68 + Phase 4b remain.**
**STILLPENDING BACKLOG — PHASE 4 (data-type tier: driver data-type & robustness coverage) MERGED+PUSHED master `050f5c4b` (2026-06-16, ff, 10 commits) + LIVE deploy-regression-clean.** Branch `feat/stillpending-phase-4-driver-datatypes` ff-merged + deleted. Design `…-phase-4-driver-datatypes-design.md` (`57d9f1b3`) + plan `…-driver-datatypes.md`/`.tasks.json` (`97c087f2`). **SCOPE NARROWED FROM ROADMAP after grounding** (AskUserQuestion locked both): **S7 wide types DEFERRED to Phase 4b** (driver reads via S7.Net address-strings → wide types need an `S7.Net.Types` byte-buffer read-path refactor; python-snap7 dev sim's wide-type support unconfirmed) and **ALL array work DEFERRED to a dedicated slice** (the live equipment-node materialiser `EnsureVariable` hard-wires `ValueRank=Scalar` with NO array params + `EquipmentTagPlan` carries no array metadata + S7/AbLegacy have no array READ support → surfacing arrays end-to-end is a coherent cross-cutting feature: plan-metadata + `EnsureVariable` contract change + per-driver read + Phase-6 UI, NOT a per-driver flag flip). Shipped 5 changes across 3 disjoint projects (subagent-driven, T1/T2/T4/T5 implementers dispatched CONCURRENTLY — disjoint projects, each commits by path — T3 after T2 same-file; every task spec/code reviewed + final integration review = **SHIP**): **T1 Modbus Int64/UInt64 node type** (`bd8fee61`): `ModbusDriver.MapDataType:1517` split `Int64⇒DriverDataType.Int64`/`UInt64⇒UInt64` (both ALREADY existed in the enum + `ResolveBuiltInDataType` ALREADY mapped `"Int64"`/`"UInt64"` → the gap was purely the driver lying Int32; codec round-trips long/ulong UNCHANGED; removed stale `Driver.Modbus-007` doc) — 271/271. **T2 FOCAS fail-fast factory** (`5c5aaef6`): new `IFocasClientFactory.EnsureUsable()` (no-op Wire/Fake, throws Unimplemented) called early in `InitializeAsync``unimplemented`/`none`/`stub` backend now FAULTS at init not first-read (operator footgun) — 191/191. **T3 FOCAS position scaling** (`49730752`+`88d9e3b1` stale-comment review fix): per-device `PositionDecimalPlaces` config (default 0 = unchanged) divides the 4 position fields by `10^dp` at `PublishAxisSnapshot`; widened `LastFixedSnapshots` `int→double` (the 6 refs through it are all already `Float64`; Int32 fixed-tree vars resolve from separate struct caches → no node-type change); `cnc_getfigure` auto-fetch stays deferred. **T4 Historian Total aggregate** (`5e27b5f7`): `ReadProcessedAsync` Total → query Average + post-multiply each bucket by `interval.TotalSeconds` (time-integral identity; Wonderware AnalogSummary has no Total column); null/bad buckets carried through → BadNoData — 35/35. **T5 Historian poison dead-letter cap** (`fcb38014`): `SqliteStoreAndForwardSink` `maxAttempts` ctor knob (default 10, guard >0) + `QueueRow`/`ReadBatch` carry `AttemptCount` (column pre-existed, NO migration) → RetryPlease dead-letters at `AttemptCount+1>=maxAttempts` (counts row leaving queue, no drift); threaded `AlarmHistorianOptions.MaxAttempts` + `AddAlarmHistorian` DI + `Validate()` warning — 27/27 + Runtime 267/267. **NO contract / `DriverDataType` / `IOpcUaAddressSpaceSink` / EF-migration change.** **Build clean (0 err).** **LIVE `/run`:** rebuilt central-1/central-2 to the branch → both healthy + `Phase7Applier: applied plan (added=90, rebuild=True)` + all materialisation + OPC UA serving, NO fatal/unhandled exception (the new driver DLLs + `AddAlarmHistorian maxAttempts` DI load+deploy clean). **Modbus-Int64 end-to-end node advertisement: UNIT-PROVEN** (271 tests + the equipment-node `"Int64"→DataTypeIds.Int64` resolver) — full live authoring DEFERRED (docker-dev seed has NO Modbus driver; sim `10.100.0.35:5020` reachable from the Mac but not seeded). **NEW FOLLOW-UPS:** (i) Phase-2 `OpcUaProbeResult` INFO dead-letters at `driver-host`/`historian-adapter`/`scripted-alarm-host` (those 3 actors subscribe the redundancy topic but don't drop the probe result like `OpcUaPublishActor` does — benign, capped at 10 then 5-min-suppressed, NOT a Phase-4 change — tidy by adding the same `Receive<OpcUaProbeResult>(_=>{})` drop); (ii) full Modbus-Int64 live authoring; (iii) **Phase 4b** = S7 wide types + Timer/Counter + the dedicated array slice + remaining §2 structural (AbCip/TwinCAT UDT, AbLegacy/TwinCAT bit-index RMW, Galaxy nesting, OpcUaClient event-history, FOCAS cnc_getfigure). **Backlog Phases 58 remain.**
**STILLPENDING BACKLOG — PHASE 3 (OPC UA standards completeness: H4 alarm Enable/Disable + H2 HistoryUpdate permission-bit + H6 native-ack→AVEVA) MERGED+PUSHED master `7eeb9fb0` (2026-06-15, ff, 16 commits) + LIVE deploy-regression-clean.** Branch `feat/stillpending-phase-3-opcua-standards` ff-merged + deleted. Design `…-phase-3-opcua-standards-design.md` (`40b883ef`) + plan `…-standards.md`/`.tasks.json` (`6ab3d863`). Subagent-driven, every task spec+code reviewed (serial for high-risk node-manager/driver/contract tasks) + final integration review = **SHIP** (all 7 cross-cutting checks PASS, incl. the load-bearing native-ack round-trip + node-id seam). **H2-bit** (`c236263e`): `NodePermissions.HistoryUpdate = 1<<12` (int-backed flags enum, NO migration; excluded from all composites — service stays deferred) + `TriePermissionEvaluator:86` maps HistoryUpdate→its own bit (was HistoryRead — closed the read⇒update hole). **H4** (`328bd1b9`+`be6858ba`): wired the missing node-manager `OnEnableDisable` (SDK `ConditionEnableEventHandler(ctx,cond,bool enabling)`, verified via reflection) → scripted = AlarmAck-gated `HandleAlarmCommand("Enable"/"Disable")`→engine (already handled it); native→`BadNotSupported`; native-check via lock-guarded `IsNativeAlarmNode`. **H6**: `418663b3` thread `bool isNative` through the SINGLE shared `MaterialiseAlarmCondition` (Phase7Applier true@:204/false@:295`_nativeAlarmNodeIds`); `ed941c51` `AlarmAcknowledgeRequest.OperatorUser` (additive) + GalaxyDriver forwards + ScriptedAlarmSource honors-else-`opcua-client`; `a6d9de09`+`87dd65b9` native `OnAcknowledge`→NEW `NativeAlarmAckRouter` (AlarmAck-gated `HandleNativeAlarmAck`, principal=DisplayName; scripted ack unchanged→engine); `93d9160d` `DriverHostActor` inverse map `conditionNodeId→(driver,FullName)` (lock-step w/ `_alarmNodeIdByDriverRef`, both keyed by `EquipmentNodeIds.Variable(EquipmentId,FolderPath,Name)`) + primary-gated `RouteNativeAlarmAck``DriverInstanceActor``driver.AcknowledgeAsync` (fire-and-forget, Connected-only); `30315185` host DI maps `NativeAlarmAck→RouteNativeAlarmAck` (Runtime has NO OpcUaServer dep). GOTCHAS: native-check MUST use lock-guarded `IsNativeAlarmNode` (raw set read races a concurrent RebuildAddressSpace — OnTimedUnshelve bypasses the Call-service lock); node-id seam airtight because BOTH sides derive from the same `EquipmentNodeIds.Variable` over the same composition (null/empty FolderPath normalizes identically). NO EF migration; only additive contracts. Also fixed a PRE-EXISTING Phase-0 miss (`8899d6e0`): 2 Galaxy guard tests asserted "PR 4.4"/"PR 4.W" text that Phase 0 `b4af9e7f` removed → updated to current guard text. **Build clean; Core 237/237, OpcUaServer 201/201, Runtime 267/267, Galaxy 264/264(+3 live-gw skip).** **LIVE `/run`:** rebuilt central-1/central-2 to the branch → healthy startup + `Phase7Applier: applied plan (added=90, rebuild=True)` + `scripted alarms materialised (alarms=1)` + OPC UA serving (ServiceLevel 250) — substantial node-manager/DriverHostActor/host changes deploy+materialise+serve with NO regression. **H4 client-Disable BLOCKED: Client.CLI has no enable/disable command** (plan assumed one) → H4 fully unit-proven; CLI enable/disable = a FOLLOW-UP. **H6 full native-ack→AVEVA OPERATOR-GATED**: needs a Galaxy NATIVE alarm (dev rig's 1 alarm is scripted) + the real gateway/AVEVA on `10.100.0.48`; round-trip wiring unit+integration-proven. Phases 48 remain.
**STILLPENDING BACKLOG — PHASE 2 (H3 health-aware redundancy ServiceLevel) MERGED+PUSHED master `4af8e65a` (2026-06-15, ff, 14 commits) + LIVE-PROVEN on the docker-dev 2-node rig.** Branch `feat/stillpending-phase-2-servicelevel` ff-merged + deleted. Design `docs/plans/2026-06-15-stillpending-phase-2-servicelevel-design.md` (`05283533`) + plan `…-servicelevel.md`/`.tasks.json` (`7605f4d8`). **LIVE `/run` ACCEPTANCE PASSED (agent-driven, login disabled):** rebuilt central-1+central-2 to the branch (`docker compose -f docker-dev/docker-compose.yml up -d --build central-1 central-2`); read `Server.ServiceLevel` (NodeId `i=2267`) via Client.CLI on :4840 (leader) + :4841 (follower). **BEFORE (old code): 240 / 100** (legacy role-only). **AFTER (branch, healthy): 250 / 240** (calculator path LIVE — the follower 100→240 + leader 240→250 is the discriminator vs legacy; proves dbHealthProbe Ask delivery + calculator invocation + NodeId self-match + the +10 leader bonus all live). **DB-down (stopped shared `otopcua-dev-sql-1`): 110 / 100** (`DbHealthProbe: probe failed``Stale=!DbReachable` → 100-tier, leader +10=110 — health-aware demotion). **Recovered (SQL restarted): 250 / 240.** Rig left healthy. (NOTE: stopping the SHARED SQL demotes BOTH nodes; single-node asymmetric isolation would need a per-node DB network block the rig doesn't cleanly support, but the both-demote + recover proof + the steady-state 250/240-vs-legacy-240/100 discriminator fully validate the wiring.) Wires the existing-but-never-invoked `ServiceLevelCalculator` into the live per-node publish path so a DB-unreachable / OPC-UA-unreachable node drops below its role level. **Subagent-driven, every task spec+code reviewed (serial for the high-risk actor tasks) + a final integration review = SHIP.** Commits: `ff0f62db` move `ServiceLevelCalculator`+`NodeHealthInputs``Core.Cluster` (`ZB.MOM.WW.OtOpcUa.Cluster.Redundancy`, cycle-free — Runtime already refs Core.Cluster, AVOIDS the deliberate no-Runtime→ControlPlane boundary); `3e609a2b`+`a9ff1a64` `OpcUaPublishActor.RecomputeServiceLevel` calls the calculator (DB+Stale+leader) with a **back-compat legacy seam** (no `dbHealthProbe` wired → old role-only 240/200/100/0, keeps the 2 pre-existing tests green) + Detached/no-entry guard → 0 + always-publish-first-even-0; `cf278035`+`5382eea9` `OpcUaProbeOk` from **peer-probes-me** (consume `OpcUaProbeResult` where NodeId==local; absent/stale verdict→true, only a RECENT observed false demotes); `37b32a56` periodic `HealthTick` Ask→PipeTo the local `DbHealthProbeActor` (Ask-timeout→Reachable=false fail-safe-demote); `f41e957e`+`5a064e08` NEW `PeerProbeSupervisor` spawns one `PeerOpcUaProbeActor` per non-self non-Detached driver peer (monotonic gen-suffix child names + Watch/Terminated-by-ref prune — stale Terminated can't evict a fresh same-peer child); `70e6d3d2`+`393b746d` docs/Redundancy.md WIRED; `4c78dcd3` DI wiring (`dbHealthProbe: dbHealth` into `OpcUaPublishActor.Props` + spawn `PeerProbeSupervisor` + `PeerProbeSupervisorKey`); `4af8e65a` supervisor explicitly drops co-mingled `OpcUaProbeResult` (no unhandled-msg noise). **Tiers (documented, now reachable): 0 down/Detached · 100 DB-down⇒stale · 200 DB-ok-stale · 240 healthy follower · 250 healthy leader. Behavior change: healthy Secondary 100→240** (both healthy nodes 240/250, leader preferred by +10 — clients still pick primary). `Stale = !DbReachable || (now-dbAsOf)>30s || (now-snapshotAsOf)>30s`. **Integration review's #1 risk (NodeId canonical form across the probe→publish seam) = PASS**: both `RedundancyStateActor.ToNodeId(member.Address)` + `ClusterRoleInfo.LocalNode` derive `host:port` from the same `PublicHostname`/`Port` that configures Akka remoting (same class as [[project_redundancy_state_delivery]], consistent here). NO EF migration, NO Commons contract change. **Build clean; ControlPlane.Tests 59/59, Runtime.Tests 264/264.** **LIVE `/run` GATE (Task 8, high-risk acceptance) PENDING:** on docker-dev MAIN (central-1/central-2 2-node warm), read `Server.ServiceLevel` on :4840/:4841 → leader 250 / follower 240; isolate ONE central node from SQL (docker network disconnect — do NOT stop the shared SQL container) → its ServiceLevel drops to 100 (below the healthy peer's 240); reconnect → climbs back. Agent drives docker-dev directly (login disabled). Phases 38 of the backlog remain (each gets its own plan when reached).
**STILLPENDING BACKLOG — PHASE 0 + PHASE 1 (H1+H5) MERGED+PUSHED master `c6a543d1` (ff, 15 commits) 2026-06-15; H1 LIVE-PROVEN on docker-dev.** Then `907005d2` (CLAUDE.md: local rig login-disabled → run /run yourself) + `4bd7180e` (fix docker-dev seed: stop seeding retired SystemPlatform namespace). **LIVE VERIFY:** rebuilt local rig to master, drove AdminUI `localhost:9200` (login disabled), toggled `layer0-logcheck` (EQ-55297329838d) Historize → Deploy → `Phase7Applier: applied plan (added=0, removed=0, changed=1, rebuild=True)` + `sealed (acks=6)` — a PURE changed-only deploy now rebuilds (pre-H1 = `rebuild=False`); reverted = 2nd proof; config pristine. **Blocker found+fixed:** docker-dev seed still inserted `Kind='SystemPlatform'` (retired) → 500s /deployments + fails publish; ran sanctioned CleanupSystemPlatformNamespaces SQL on dev DB + fixed seed. H5 threading flows through real deploy; durable IHistoryWriter.Record silent (NullHistoryWriter infra-gated). (Historical IN-PROGRESS detail below.) Driven by the approved roadmap `docs/plans/2026-06-15-stillpending-backlog-design.md` + plan `…-phase-0-1.md` (full backlog audit = `stillpending.md`). DONE on the branch (subagent-driven, each task spec+code reviewed, all green): **Phase 0 hygiene** (`b4af9e7f` 7 stale comments; `a9d267c9` security.md write-outcome + benign DraftSnapshot/LeaderChanged notes; `50a2fdf3` marked 18 shipped `.tasks.json` completed). **H1 — changed-only deploys now take effect** (`1e95856b` Phase7Applier `needsRebuild` includes `Changed*` counts, ChangedDrivers excluded by design; `1dc71369` count removed tags/vtags in RemovedNodes [review follow-up]; `ada01e1a` VirtualTagHostActor stop+respawns a child whose plan changed in place; `ebf2f1dd` prune `_planByVtag` on termination + crash-then-change test [review follow-up]). **H5 — vtag Historize honored at runtime** (`fc8121cb` `EquipmentVirtualTagPlan.Historize` + composer reads `v.Historize`, in Equals/GetHashCode; `9c5a0913` artifact decode reads `Historize` byte-parity; `83d3b9f7` planner Historize-toggle test + doc nit; `0c6d4c54` VirtualTagHostActor invokes injected `IHistoryWriter` on historized results [+ Runtime→Core.VirtualTags ProjectReference]; `2f30c54d` thread-safe CapturingHistoryWriter [review follow-up]; `4501f12` thread IHistoryWriter through DriverHostActor + DI `TryAddSingleton` Null default). **DURABLE AVEVA vtag-history sink = INFRA-GATED** (no sidecar live-data WriteDataValues RPC) → seam wired w/ NullHistoryWriter default, flagged on the design defer list. NO EF migration (Historize column + ConfigComposer whole-entity serialization already carry it). REMAINING on branch: Task 10 docs (this commit) + Task 11 final build/test/integration-review + finishing (merge is the user's call). LIVE `/run` gate (user-driven): edit equipment/alarm/vtag → confirm it takes effect without restart; toggle a vtag Historize → host respawns + writer invoked.
**#401 — AbCip + FOCAS ALARM-PROJECTION POLL-LOOP LEAK FIXED + MERGED + PUSHED master `151b7165` (2026-06-15, ff, 2 commits `6ba59f9d`+`151b7165`).** SIBLING of #399 found by auditing all `IAlarmSource` drivers, but WORSE: `AbCipAlarmProjection` + `FocasAlarmProjection` each `Task.Run` a poll loop per `SubscribeAsync` and survive in-place reconnect, so the actor's per-Connected re-subscribe leaked a LIVE never-cancelled poll loop each reconnect → CPU growth + DUPLICATE OnAlarmEvent transitions (functional, gated behind opt-in alarm projection). Fix (both projections, symmetric): `SubscribeAsync` snapshots existing subs under `_subsLock`, `_subs.Clear()`, adds the new sub, starts its loop, then `_ = RetireAsync(old)` fire-and-forget (Cancel→await loop→Dispose CTS). Snapshot+clear under the same lock `DisposeAsync` uses ⇒ no double-own/double-dispose (reviewer-confirmed airtight for all interleavings). Implemented via opus subagent (TDD fail→pass: subscribe twice → 1 device raise → exactly ONE event, was 2). AbCip 271/271, FOCAS 186/186. Code-reviewed (sonnet) — correct+safe; applied its 2 nice-to-haves (documented the at-most-ONE-duplicate-per-reconnect one-tick overlap residual in RetireAsync XML doc + guarded Cts.Dispose). **AUDIT RESULT: all 4 IAlarmSource drivers now correct — Galaxy (#399), AbCip+FOCAS (#401), OpcUaClient already safe (session-bound + clears `_alarmSubscriptions` on disconnect).**
**#399 — GALAXY ALARM-SUBSCRIPTION HANDLE LEAK FIXED + MERGED + PUSHED master `01388226` (2026-06-15, ff, 1 commit).** The session-less Galaxy `StreamAlarms` feed survives an in-place reconnect, so `DriverInstanceActor` re-subscribed on every Connected re-entry (dropping its cached handle without an Unsubscribe — sync teardown) and `GalaxyDriver.SubscribeAlarmsAsync` grew `_alarmSubscriptions` by one untracked handle per reconnect (slow unbounded leak; functionally harmless — gate is Count>0, fan-out reads [0] once). Fix: `SubscribeAlarmsAsync` `Clear()`s the set before `Add` (under the existing `_alarmHandlersLock`, atomic w.r.t. the fan-out reader), collapsing to ONE live handle — faithful because there's exactly one consumer per driver instance (factory-per-actor lifecycle). Chosen over async/fire-and-forget actor-side Unsubscribe (disproportionate). NEW regression test `Re_subscribe_collapses_to_a_single_handle_no_accumulation` (TDD-VERIFIED fail-without-fix: releasing the latest handle leaves the feed open if stale handles remain). Galaxy 263 pass/3 skip, Runtime native-alarm 24 pass. Code-reviewed (sonnet) — correct+complete, both nice-to-haves applied (structural-invariant comment + distinct-handle assert).
**#400 — within-timestamp cursor paging for oversized tie clusters: SHIPPED to master `2e6c6d3a` + review fix `0f929ae6` (2026-06-17).** No longer a documented follow-up. `HistoryPaging.SliceTieCluster` over-fetches the whole tie cluster (a bounded `start==end` read), carves out the next `cap` ties at `skip`, advances the cursor WITHIN the timestamp, and steps off it (`T + 1 tick`) once drained — wired into `OtOpcUaNodeManager.ServeRawPaged` (stall detected when a resume page's last sample is still at the boundary timestamp). New config `ServerHistorian:MaxTieClusterOverfetch` (default 65536) bounds the in-memory buffer with a `Validate()` warning; 6 `SliceTieCluster` unit tests + integration tests `Raw_oversized_tie_cluster_pages_within_the_timestamp` / `Raw_tie_cluster_beyond_overfetch_bound_fails_loudly`; `docs/Historian.md` "within-timestamp paging" section + config-table row updated. DELIBERATE residual backstop: a cluster strictly larger than `MaxTieClusterOverfetch` ties at one identical SourceTimestamp still loud-fails `BadHistoryOperationUnsupported` (a severe data anomaly — bounded on purpose to cap memory; raise the knob or `NumValuesPerNode` to cover it).
**#398 — RESTART-DRIVER "actor name is not unique" BUG FIXED + MERGED + PUSHED master `c9643f68` (2026-06-15, ff, 1 commit).** `DriverHostActor.HandleRestartDriver` stopped + respawned the child in ONE synchronous handler reusing the base actor name `drv-<id>`; `Context.Stop` is async so the old child was ALWAYS still registered → `InvalidActorNameException` deterministically on every AdminUI Restart press (crashed+restarted the host). Fix: a monotonic `_childSpawnGeneration` counter feeds a `-g<gen>` suffix on every spawned child name (children tracked by the `_children` dict via IActorRef, never by path, so the suffix is invisible). Also closes the same-shaped LATENT race in the reconcile path (removed-then-readded instance + a driver-type-change `ToStop+ToSpawn` in one plan — confirmed by review). NEW regression test `RestartDriver_respawns_the_child_without_an_actor_name_collision` (TDD-VERIFIED: fails on the old code with the exact `InvalidActorNameException` via `EventFilter.Exception<…>().Expect(0,…)`, passes with the fix). Runtime.Tests 238/238. Code-reviewed (sonnet) — correct+complete, no Critical/Important. `HandleReconnectDriver` confirmed bug-free (only `Tell`s ForceReconnect).
**⚠️ OUT-OF-BAND WORKING-TREE CHANGE I DID NOT MAKE (surfaced to user 2026-06-15, NOT touched):** `Directory.Packages.props` is modified (`ZB.MOM.WW.MxGateway.Client` + `…Contracts` bumped `0.1.0``0.1.1`) and two new `nuget-packages/ZB.MOM.WW.MxGateway.{Client,Contracts}.0.1.1.nupkg` (dated 2026-06-15 05:39) appeared during this session. These are EXTERNAL (likely a sibling-repo publish / parallel action) — NOT part of my B2 or #398 commits (both clean, verified). Left untouched per the "didn't-create-it → surface, don't act" rule; awaiting user decision whether to commit, revert, or ignore. `nuget-packages/` is NOT gitignored (tracked dir).
**B4 — docker-dev rig cleanup RESOLVED 2026-06-15 as "LEAVE-AS-IS" (user decision).** The rig keeps this session's verification artifacts on purpose: the historized dev `GalaxyTestTag` (= `TestMachine_002.TestHistoryValue`) + the local-only `docker-dev/docker-compose.yml` env block (`ServerHistorian__*` with `SharedSecret` via `${OTOPCUA_HISTORIAN_SECRET:-}`, + the data-plane `Security__Ldap__GroupToRole__*` map) so A2 (historian) + A3 (native alarms) + B1 (write-outcome) stay re-verifiable. `docker-compose.yml` stays an uncommitted DISK-ONLY file (never staged). No teardown, no commit — matches the earlier "leave the Galaxy verification artifacts as-is" decision. **All buckets (A2/A3 live, B1/B2/B3 code, B4 operational) are now CLOSED.** Remaining are logged follow-ups only: #398 (pre-existing Restart actor-name-not-unique), #399 (minor session-less alarm handle leak), #400 (within-timestamp cursor paging — now SHIPPED `2e6c6d3a`/`0f929ae6` 2026-06-17).
**B2 — HISTORIAN HISTORYREAD CONTINUATION-POINT PAGING (Raw) MERGED + PUSHED to master `aa1e21f5` (2026-06-15, fast-forward, 3 commits: `94c3ca60` feature, `bea0b482` review fixes, `aa1e21f5` doc).** Server-side time-based paging for the only count-capped arm (Raw): full page (count==NumValuesPerNode>0) ⇒ opaque 16-byte CP + tie-safe resume cursor (`HistoryPaging.ComputeResumeCursor`/`TrimBoundaryDuplicates` — resume from boundary timestamp INCLUSIVE, drop already-emitted head ties → no dup/skip); short page / `==0` (all values) ⇒ no CP. CPs bound to the OPC UA session via the SDK store (`SessionHistoryContinuationStore`, capped by `ServerConfiguration.MaxHistoryContinuationPoints`=100 oldest-evict, disposed on session close); tests inject `InMemoryHistoryContinuationStore` (in-process harness is session-less). `HistoryReleaseContinuationPoints` override frees cursors; unknown/evicted/released ⇒ `BadContinuationPointInvalid`. Processed/AtTime stay single-shot (no client count cap → no full-page signal; spec-conformant). NEW: `HistoryPaging.cs`, `HistoryContinuationStore.cs` + `HistoryPagingTests.cs` (15) + `NodeManagerHistoryReadPagingTests.cs` (7, boot a real SDK server). **182/182 OpcUaServer tests pass, full solution builds clean.** Code-reviewed (sonnet subagent): **C1 (critical) FIXED** — an oversized boundary tie cluster (more ties at one SourceTimestamp than NumValuesPerNode) could silently truncate a resume to GoodNoData and DROP the un-emitted ties (the (timestamp,skip) cursor can't advance past a single timestamp the fixed-(start,end,cap) backend keeps re-returning); now fails LOUDLY per node with `BadHistoryOperationUnsupported` + a log + `docs/Historian.md` "Paging limitation" note + regression test `Raw_tie_cluster_larger_than_page_fails_loudly_not_silently`. Also I3 (Save after HistoryData build → no orphaned cursor), N1 (dropped never-produced `HistoryReadKind` enum + Processed-only state fields — only Raw pages), N3 (ComputeResumeCursor empty-page guard), I1 (documented test-store eventual-consistency). Full within-timestamp cursor paging = task #400, SHIPPED `2e6c6d3a`/`0f929ae6` (2026-06-17) — `HistoryPaging.SliceTieCluster` + `MaxTieClusterOverfetch`-bounded over-fetch; only clusters beyond the bound still loud-fail by design. **Modified-value history (`HistoryReadModified`) is INFRA-GATED, not a code gap**: the AVEVA Wonderware backend (`IHistorianDataSource` TCP sidecar) has no modified-read path; server cleanly rejects per node + `IsReadModified` honoured; doc note added. **B2 COMPLETE.** (B1 write-outcome residuals + B3 GroupToRole doc already on master `a5c0c826`/`063d004f`.)
**GALAXY LIVE-GW INTEGRATION SMOKES (reopen + write-outcome) MERGED + PUSHED to master `d19deb9b` (2026-06-14, fast-forward, 2 commits).** Closed the two items the user re-opened with a live gateway: the harden-1b **Galaxy-reopen** integration test + the **write-outcome** Galaxy write-commit/persist proof. NEW skip-gated `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests/Runtime/GatewayGalaxyLiveReopenAndWriteTests.cs` (gated on `MXGW_ENDPOINT` + `GALAXY_MXGW_API_KEY`, like `GatewayGalaxyAlarmFeedLiveTests`; **skips clean in CI** — verified). **LIVE-PROVEN 2/2 against `http://10.100.0.48:5120` (2026-06-14):** (1) `Live_reopen_recreates_session_and_re_establishes_write_handles` — a live write seeds the writer's item-handle + supervisory-advise caches, then the EXACT `GalaxyDriver.ReopenAsync` primitive pair runs (`GalaxyMxSession.RecreateAsync``GatewayGalaxyDataWriter.InvalidateHandleCaches`), both caches drop to 0, and a post-reopen write re-establishes them + returns Good — proving `RecreateAsync` yields a working live session (the un-fakeable part the unit suite can't reach); (2) `Live_supervisory_write_commits_and_persists_to_galaxy` — writes a value distinct from current, then reads it back through a BRAND-NEW session — proving the no-login (`WriteUserId=0`) write COMMITS + PERSISTS only because of `AdviseSupervisory` (the `f05b5d79` fix). Code-review APPROVED (refinement applied: read-back completes an explicit `TaskCompletionSource<double?>` from the stream pump; `bufferedUpdateIntervalMs:0`). **DURABLE RUN RECIPE:** the gw `10.100.0.48:5120` IS reachable from this Mac; source the key WITHOUT echoing it — `KEY=$(docker exec otopcua-dev-central-1-1 printenv GALAXY_MXGW_API_KEY); MXGW_ENDPOINT=http://10.100.0.48:5120 GALAXY_MXGW_API_KEY="$KEY" dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests --filter FullyQualifiedName~GatewayGalaxyLiveReopenAndWrite` (central-1's container env carries a valid key while its galaxy session is connected; check with `docker exec … sh -c 'test -n "$GALAXY_MXGW_API_KEY" && echo SET'`). Writes only `TestMachine_002.TestFloat` (the established write-demo attr).
**RESIDUAL FOLLOW-UPS CLEANUP MERGED + PUSHED to master `fa2388ea` (2026-06-14, fast-forward, 7 commits = 1 plan doc + 6 test/doc).** Closed the offline-doable, non-blocking residuals from open-item #3 (write-pipeline / Phase B / Galaxy-nits sections) — **TESTS + DOCS only, ZERO production change.** Plan `docs/plans/2026-06-14-residual-followups-cleanup-plan.md`. **KEY RECONCILIATION FINDING:** a grounding sweep (3 Explore agents + direct verify) proved MOST of the listed residuals were ALREADY implemented in the harden-1b / write-outcome / `f05b5d79` work — pending.md's residual list was STALE. **Verified already-done (no work needed, file:line evidence):** WP(a) stale/reconnecting writes fast-fail (`DriverHostActor.cs:661` `Receive<RouteNodeWrite>`, `DriverInstanceActor.cs:229,316` `Receive<WriteAttribute>`); WP(b) `ExecuteSynchronously` already dropped (`DriverHostActor.cs:630` = `TaskContinuationOptions.None`) + `driverIds` forward-map already a `HashSet`; WP(c) FOCAS address cached via `GetOrAdd` (`FocasDriver.cs:247`); WP(d) raw-protocol-blob write test exists (`Primary_routes_write_for_raw_protocol_blob_tag`); WP(e) `[InlineData(2,false)]` future-enum trap already in the parity test; Galaxy(1) `_itemHandles`+`_supervisedHandles` cleared on reconnect (`GatewayGalaxyDataWriter.InvalidateHandleCaches()``GalaxyDriver.ReopenAsync`); Galaxy(2) `SubscriptionEstablished` self-Tell handled (debug-swallow) in all 3 `DriverInstanceActor` states (`:253,:301,:336`). **ACTUALLY DONE this pass (4 items):** Phase B(a) regression test that a `NativeAlarmRaised` during `Reconnecting` is DROPPED not dead-lettered (uses an `AllDeadLetters` EventStream probe so it genuinely guards removal of the `DriverInstanceActor.cs:345` drop handler — the first cut used `parent.ExpectNoMsg` only, which a code review caught as a false-pass gap); Phase B(b) test that `OperatorComment``AlarmTransitionEvent.Comment` (and `User=="device"`) through `DriverHostActor.ForwardNativeAlarm`; Phase B(c) severity-bucket doc note in `docs/ScriptedAlarms.md` (each transition snaps to 200/500/700/900 via `NativeAlarmProjector.MapSeverity`, overriding the authored seed, then SDK `EventSeverity` brackets via `OtOpcUaNodeManager.MapSeverity`); Galaxy(3) extracted shared `tests/.../Runtime.Tests/Drivers/StubDrivers.cs` harness (de-dup of the superset `StubDriver`/`WritableStubDriver`/`SubscribableStubDriver`/`StubHandle` from `DriverInstanceActorTests` + `DriverInstanceActorWriteAndSubscribeTests`; `BlockingInitDriver` left local). Built via subagent-driven dev (1 implementer bundle → code-review APPROVED w/ 1 Important = the dead-letter-probe gap, fixed). **Build clean; Runtime.Tests 236/0.** (The harden-1b Galaxy-reopen integration test noted here as deferred is **NOW DONE** — see the Galaxy live-gw smokes banner above, `d19deb9b`.)
**GALAXY PHASE C — SERVER-SIDE OPC UA HistoryRead MERGED + PUSHED to master `c24abc8a` (2026-06-14, fast-forward, 14 commits = 2 design/plan docs + 12 feature/test/doc).** The server now answers OPC UA **HistoryRead** (Raw / Processed / AtTime over historized variable nodes; **Events** over alarm-owning equipment-folder event-notifier nodes) for any equipment tag flagged historized, driver-agnostically, by dispatching to the registered `IHistorianDataSource` (the Wonderware historian TCP client, which already implemented that interface). **NO EF migration** — the flag rides in the existing `TagConfig` JSON blob (`{"FullName":"…","isHistorized":true,"historianTagname":"…"?}`, the Phase-B `alarm`-object carrier), `historianTagname` defaults to the tag's driver `FullName`. Design/plan `docs/plans/2026-06-14-galaxy-phase-c-historian-*.md`; guide `docs/Historian.md`. Pipeline: `Phase7Composer.ExtractTagHistorize` + byte-parity `DeploymentArtifact.ExtractTagHistorize``EquipmentTagPlan.{IsHistorized,HistorianTagname}``Phase7Applier` resolves `IsHistorized ? (HistorianTagname ?? FullName) : null` → sink seam (`IOpcUaAddressSpaceSink.EnsureVariable` +`string? historianTagname`) → `OtOpcUaNodeManager.EnsureVariable` sets `Historizing`+`AccessLevels.HistoryRead`+registers `_historizedTagnames`; the node manager overrides the four `CustomNodeManager2` HistoryRead virtuals, block-bridging to the `HistorianDataSource` property (volatile, default `NullHistorianDataSource.Instance`). DI mirrors `AlarmHistorian`: `AddServerHistorian` (config-gated, `ServerHistorian` appsettings section, Null default via `TryAddSingleton`) + `OtOpcUaSdkServer.SetHistorianDataSource` + Host `Program.cs`/`OtOpcUaServerHostedService` Start/Stop wiring. **Graceful degrade:** historized node + Null/unconfigured source → `Good_NoData` (empty), non-historized node → `BadHistoryOperationUnsupported`. **KEY FACTS/GOTCHAS:** the SDK base filters event reads by the `EventNotifier.HistoryRead` bit (variable nodes never reach the events arm); the SDK master propagates `errors[i].Code → results[i].StatusCode` (confirmed by decompiling `MasterNodeManager.HistoryReadAsync`), so the override signals per-node status via `errors[handle.Index]`; the two `HistoryReadResult` types (SDK `Opc.Ua.HistoryReadResult` vs Core.Abstractions DTO) are aliased `SdkHistoryReadResult`/`HistorianRead`; `ReadRawModifiedDetails.IsReadModified` defaults TRUE (Initialize() sets it) so a plain raw read must clear it — modified-history is unsupported; the events arm registers folder→sourceName (= equipment id) only when a non-Null historian is wired at promotion time (Host wires the source at StartAsync before any deployment materialises, so normal boot ordering is correct). Built via subagent-driven dev (T1/T2/T5 standard parallel review, T3/T4 high-risk serial spec→code, + a final integration review READY-TO-MERGE). **Build clean (0 errors); OpcUaServer.Tests 152/0, Runtime.Tests 234/0, Core.Abstractions.Tests 88/0.** **LIVE `/run` GATE (T7) DEFERRED — operator-driven: it needs the Wonderware sidecar + AVEVA Historian on the WW Historian VM `10.100.0.48`, which is NOT on the local docker-dev rig.** When run: author a historized Galaxy tag (`TagConfig` `"isHistorized":true`), set `ServerHistorian:Enabled=true` → sidecar (Host/Port/SharedSecret/TLS), deploy on `MAIN-galaxy-eq`, then `Client.CLI historyread -n "ns=2;s=<equip>/<tag>" --start … --end …` → samples; a non-historized tag → `BadHistoryOperationUnsupported`. KNOWN follow-ups (non-blocking, documented): single-shot reads only (no server-managed continuation-point paging yet); no modified-value history; no explicit timeout at the block-bridge call site (bounded by the `WonderwareHistorianClient` 30s `CallTimeout`); the StopAsync→DisposeAsync warm-shutdown window returns Good_NoData (mirrors the `SetNodeWriteGateway` pattern).
@@ -10,11 +40,11 @@ As of 2026-06-14. master HEAD `c24abc8a` (synced with origin; feature branch `fe
**WRITE-OUTCOME SELF-CORRECTION (#5) MERGED + PUSHED to master `1d797c1c` (2026-06-14, fast-forward, 6 commits).** A failed inbound device write now reverts the node to its real pre-write value (compare-and-revert) instead of leaving the optimistic-`Good` phantom. Design `docs/plans/2026-06-14-write-outcome-self-correction-design.md`; plan `…-plan.md`. NEW `IOpcUaNodeWriteGateway`/`NodeWriteOutcome` (Commons) + `ActorNodeWriteGateway` (Runtime, Asks `RouteNodeWrite`, returns the outcome) replace the fire-and-forget `Action` router; `OnEquipmentTagWrite` captures the prior value + fires an off-Lock (`RunContinuationsAsynchronously`) continuation that reverts on a failed outcome IF the node still holds the optimistic value (`ShouldRevert`). Build clean; Commons 39/0, Runtime 201/0, OpcUaServer 111/0. High-risk review verified the prior-value capture against the actual UA-.NETStandard source (`OnWriteValue` fires before `m_value=value`). **LIVE-PROVEN end-to-end** via a local Modbus exception-injector (FC06 reject on HR[20]): authorized failing write → gateway-logged `0x808B0000` reject → node reverts 99→20; authorized success (HR200=7777) stays; anon → BadUserAccessDenied. KEY FINDING: the **Galaxy gateway worker's `ExecuteWrite` is fire-and-forget** (returns OK without awaiting the MXAccess commit), so Galaxy writes ALWAYS return Success at the OPC UA layer and can NEVER surface a device-write failure to this revert — only protocol drivers (which await + return real status) can; this is the same gateway-side limitation noted under "optimistic-write phantom" (out of our scope). Two MINOR deferred follow-ups remain: a Bad-quality blip / OPC UA AuditWriteUpdateEvent on failure, and synchronous structural fail-fast (both explicitly out of scope per the chosen mechanism).
**HARDEN MILESTONE 1b cluster MERGED + PUSHED to master `945c2380` (2026-06-14, fast-forward, 9 commits).** Follow-ups #3 (data-plane role docs), #4 (write-pipeline review nits), and #6 (Galaxy driver nits) below are CLOSED. Plan: `docs/plans/2026-06-14-harden-milestone-1b-plan.md`. Build clean; Runtime 197/0, FOCAS 185/0, Galaxy 257/0(+1 pre-existing skip); final integration review READY-TO-MERGE. Two MINOR residual follow-ups surfaced by review (both deferred, non-blocking): (a) a *driver-level* regression test that `GalaxyDriver.ReopenAsync` actually calls `InvalidateHandleCaches` — needs a live gw (`RecreateAsync` can't be faked), so it's an integration test; (b) stub-driver test-class duplication between `DriverInstanceActorTests` + `DriverInstanceActorWriteAndSubscribeTests` (hygiene — extract a shared harness).
**HARDEN MILESTONE 1b cluster MERGED + PUSHED to master `945c2380` (2026-06-14, fast-forward, 9 commits).** Follow-ups #3 (data-plane role docs), #4 (write-pipeline review nits), and #6 (Galaxy driver nits) below are CLOSED. Plan: `docs/plans/2026-06-14-harden-milestone-1b-plan.md`. Build clean; Runtime 197/0, FOCAS 185/0, Galaxy 257/0(+1 pre-existing skip); final integration review READY-TO-MERGE. Two MINOR residual follow-ups surfaced by review (non-blocking): (a) a *driver-level* regression test that `GalaxyDriver.ReopenAsync` actually calls `InvalidateHandleCaches` STILL DEFERRED, needs a live gw (`RecreateAsync` can't be faked), so it's an integration test; (b) stub-driver test-class duplication between `DriverInstanceActorTests` + `DriverInstanceActorWriteAndSubscribeTests` **DONE `fa2388ea`** (shared `StubDrivers.cs`).
## STATE SUMMARY (post-compaction pickup)
**ALL feature work is SHIPPED + PUSHED to master `c24abc8a` (synced with origin). Nothing is blocking.** Milestone 1b (equipment-tag live values: live READ + authorized inbound WRITE across OpcUaClient / the 6 protocol drivers / Galaxy, via the `FullName→NodeId` router) is COMPLETE, and all three Galaxy phases shipped: **A** standard Equipment driver `c3c56172`, **B** native alarms `f9be3843`, **C** server-side HistoryRead `c24abc8a`. The session's cluster-harden / write-outcome-self-correction / driver-reconfigure-while-faulted follow-ups merged too (`945c2380` / `1d797c1c` / `56f73e49`). The six banners above carry each feature's mechanism + gotchas + deferred live gate; the closed open-follow-ups #1#7 (Phase B, Phase C, data-plane role docs, write-pipeline nits, write-outcome, Galaxy driver nits, reconfigure-while-faulted) are all DONE at those SHAs.
**ALL feature work is SHIPPED + PUSHED to master `d19deb9b` (synced with origin). Nothing is blocking.** Milestone 1b (equipment-tag live values: live READ + authorized inbound WRITE across OpcUaClient / the 6 protocol drivers / Galaxy, via the `FullName→NodeId` router) is COMPLETE, and all three Galaxy phases shipped: **A** standard Equipment driver `c3c56172`, **B** native alarms `f9be3843`, **C** server-side HistoryRead `c24abc8a`. The session's cluster-harden / write-outcome-self-correction / driver-reconfigure-while-faulted follow-ups merged too (`945c2380` / `1d797c1c` / `56f73e49`). The six banners above carry each feature's mechanism + gotchas + deferred live gate; the closed open-follow-ups #1#7 (Phase B, Phase C, data-plane role docs, write-pipeline nits, write-outcome, Galaxy driver nits, reconfigure-while-faulted) are all DONE at those SHAs.
**The ONLY genuinely open items (all user-driven / deferred — pick up here):**
@@ -23,7 +53,7 @@ As of 2026-06-14. master HEAD `c24abc8a` (synced with origin; feature branch `fe
- **Phase B native alarms** (T9) — author a Galaxy alarm tag whose `FullName` == the gateway `AlarmFullReference`, deploy on `MAIN-galaxy-eq`, trip → Part 9 condition + `/alerts` row. Recipe: Phase B banner.
- **Driver-reconfigure-while-faulted** — fault `MAIN-opcua-eq` with a bad config, deploy a corrected one (`POST http://localhost:9200/api/deployments`, `X-Api-Key: docker-dev-deploy-key`), confirm it adopts WITHOUT a node restart. Recipe: that banner.
2. **Rig cleanups** (operational, user-deferred) — see "Operational deferral" at the bottom.
3. **Minor non-blocking residual follow-ups** (review-surfaced, all explicitly deferred, none gate anything): Phase B residuals (a)(d) in its banner; write-outcome residuals (Bad-quality blip / AuditWriteUpdateEvent / synchronous fail-fast); harden-1b two residuals (`945c2380` banner: Galaxy-reopen integration test, stub-driver test-class de-dup); Phase C documented follow-ups (no continuation-point paging, no modified-value history, block-bridge timeout bounded only by the client's 30s `CallTimeout`); the data-plane `GroupToRole` production-default note; Galaxy `_itemHandles`/`_supervisedHandles` not cleared on reconnect + the cosmetic `SubscriptionEstablished` self-dead-letter.
3. **Minor non-blocking residual follow-ups** — MOSTLY CLOSED by the residual-followups-cleanup merge (`fa2388ea`; see the top banner for full reconciliation). DONE this pass: Phase B residuals (a)/(b)/(c) + the stub-driver de-dup. VERIFIED already-implemented (were stale in this list): all of write-pipeline (a)(e), Galaxy `_itemHandles`/`_supervisedHandles` clear-on-reconnect, and the `SubscriptionEstablished` self-dead-letter handler. **STILL OPEN (deferred, none gate anything):** write-outcome residuals (Bad-quality blip / AuditWriteUpdateEvent / synchronous fail-fast — design-deferred OPC UA surface, NOT the Galaxy write-commit proof which is now live-tested `d19deb9b`); Phase C documented follow-ups (no continuation-point paging, no modified-value history, block-bridge timeout bounded only by the client's 30s `CallTimeout`); Phase B residual (d) (inbound device-ack etc., DEFERRED by design); the data-plane `GroupToRole` production-default note.
**No queued feature remains** — Milestone 1b + Galaxy A/B/C were the headline deliverables and are all done. Future directions (NOT requested): the Phase C HistoryRead follow-ups above, or new driver/UNS work.