lmxopcua/pending.md

Pending — open follow-ups & deferrals

As of 2026-06-16. master HEAD 1164d423 (synced with origin; feature branch feat/stillpending-phase-5-probes ff-merged + deleted). Working tree is clean except the expected DISK-ONLY files: docker-dev/docker-compose.yml (M — uncommitted rig config, never staged) and pending.md (M — these notes, never staged), plus two untracked pre-existing docs/plans/2026-06-14-write-outcome-self-correction-plan.md* docs. HARD RULE: never git add .; never stage pending.md / current.md / docker-dev/docker-compose.yml / sql_login.txt / src/Server/ZB.MOM.WW.OtOpcUa.Host/pki/; never commit secrets.

CERT-ACTION AUDIT LOGGING + OpcUaProbeResult DEAD-LETTER TIDY MERGED+PUSHED master 3eb370d4 (2026-06-19, ff, 7 commits incl. 2 review-polish). Branch feat/cert-audit-and-probe-tidy ff-merged + deleted. Design docs/plans/2026-06-19-cert-audit-and-probe-deadletter-tidy-design.md + plan/.tasks.json. Two independent items, subagent-driven (T1 cert-core ∥ T3 probe-drops dispatched concurrently — disjoint projects; T2 wire-in after T1; T1 spec+code review, T3 code review, final integration review = SHIP). (1) Cert-action audit logging (was stillpending §A.7 deferred): the FIRST live structured AuditEvent emit site. New ActorAuditWriter : IAuditWriter (AdminUI/Audit/, resolves the AuditWriterActor cluster singleton from ActorRegistry + fire-and-forget Tell, mirrors AdminOperationsClient) + pure CertAuditEvents.Build factory (Category:"Certificate", Action Trust/Untrust/Delete, success rows OMIT the error key); CertificateStoreManager audits at the choke point on EVERY return path (success + all failures, via a DeleteCore refactor); Certificates.razor sources the actor from AuthenticationState. AuditEvent is a ZB.MOM.WW.Audit record (init-setters; AuditOutcome.Success/Failure/Denied; CorrelationId Guid?); package ships NoOpAuditWriter.Instance (test default). NO EF migration (ConfigAuditLog table + AuditWriterActor pre-existed), NO Commons/proto change. (2) OpcUaProbeResult dead-letter tidy (Phase-4 follow-up (i)): added Receive<PeerOpcUaProbeActor.OpcUaProbeResult>(_=>{}) drop to DriverHostActor (all 3 behaviours: Steady/Applying/Stale), HistorianAdapterActor, ScriptedAlarmHostActor (mirrors PeerProbeSupervisor); guard test fails-without/passes-with. Build clean; AdminUI.Tests 510/510, Runtime.Tests 273/273. LIVE /run PASS (agent-driven, login disabled): rebuilt central-1/central-2 → both boot healthy (DI resolves the new CertificateStoreManager(IConfiguration, IAuditWriter) + ActorAuditWriter→singleton — the integration review's #1 risk), AddressSpaceApplier: applied plan (added=91), ZERO OpcUaProbeResult dead-letters on both nodes (before AND after the run). CERT-AUDIT END-TO-END PROVEN: seeded a throwaway rejected cert, clicked Trust on the Certificates page (:9200) → ConfigAuditLog row multi-role-test | Certificate:Trust | Success | <thumbprint> | {"store":"rejected→trusted","thumbprint":…} (no error key — the success-omit polish), through the full live cluster seam (razor→manager→ActorAuditWriter→AuditWriterActor singleton→shared SQL); seed cert cleaned up. Session also reconciled #400 (within-timestamp tie-cluster paging — already SHIPPED 2e6c6d3a/0f929ae6, stale pending banners corrected) + RECLASSIFIED H2 (HistoryUpdate service) as INFRA-GATED/DEFERRED (no historian backend can durably insert/replace/delete — same boundary as H5b; see stillpending §A.1). DEFERRED follow-up (unchanged): cert actions still also want to be reachable via an audit-query UI, but the DB rows now exist. MINOR review nits (non-arch, from the final integration review — recorded so they're not lost): (a) two UNREACHABLE Certificates.razor fallthrough arms ("cannot delete from {Kind}" / "unknown action") build a CertActionResult.Fail INSIDE the razor switch, bypassing CertificateStoreManager → those (unreachable) failures aren't audited; add a comment or route through the manager; (b) OpcUa:PkiStoreRoot is read in BOTH Certificates.razor:130 and CertificateStoreManager ctor — centralize on the manager. Both Minor, no rush.

STILLPENDING BACKLOG — PHASE 5 (Test-Connect protocol handshakes) MERGED+PUSHED master 1164d423 (2026-06-16, ff, 15 commits) + LIVE /run 9/11 PASS. Branch feat/stillpending-phase-5-probes ff-merged + deleted. Design …-phase-5-probes-design.md (1f2d32ac) + plan …-probes.md/.tasks.json (2f7c6bf9). Scope locked = ALL 8 TCP-only probes get real handshakes (best-effort, AskUserQuestion). Replaced bare socket.ConnectAsync with a protocol handshake after a TCP preflight, so a live-but-rejecting device reads RED not false-green; Historian.Wonderware probe was ALREADY a real Hello/HelloAck handshake → untouched (the §2 "task 9" note was stale). New 3-way contract: TCP-fail→"Connect failed: {err}" (unchanged) · handshake-ok→Ok=true+latency+descriptive msg · TCP-ok-but-rejected→"Reachable at {host}:{port} but {proto} handshake failed: …" (the new behavior) · timeout→"Probe timed out" (unchanged). IDriverProbe/DriverProbeResult/DI/csproj UNCHANGED; NO migration. Per-probe (no shared scaffold — keeps the 8 driver projects disjoint/parallelizable; subagent-driven 2 waves of 4 concurrent, Wave-1 bundle review + Wave-2 bundle review + final integration review = SHIP, all 5 invariants pass). Handshakes: Modbus FC03 via ModbusTcpTransport (exception PDU still=real device) 9b909002; OpcUaClient DiscoveryClient.GetEndpointsAsync (no session/cert) 957a63cf; S7 Plc.OpenAsync COTP+S7-setup 9a8336ff; AbCip/AbLegacy libplctag Tag init (CIP-level error=reachable) 0c08b152/21f3e8fe; TwinCAT AdsClient.ReadStateAsync DEGRADE-guarded b663ae6e; FOCAS cnc_allclibhndl3 via a NEW direct [DllImport("fwlib32")] DEGRADE-guarded 5ed0276f; Galaxy gRPC GalaxyRepository.TestConnection 2d688c2a. DEGRADE principle (TwinCAT/FOCAS): an env that can't ATTEMPT the handshake → Ok=true "TCP reachability only" (never worse than TCP); only a real device rejection is RED — both empirically proven on this headless box (AdsServerException / DllNotFoundException → degrade). Code-review fixes: timeout-CTS bound on OpcUaClient/AbCip/AbLegacy + IPv4 preflight + sw.Stop (9bfbbb0f/5df3c732). LIVE /run (agent-driven, sandbox-off so the rig is reachable; brought up local central-1 @ :4840 as a real OPC UA server): 9 PASS / 2 SKIP / 0 FAIL — Modbus green vs sim 10.100.0.35:5020 + red vs non-Modbus; OpcUaClient green vs opc-plc :50000 AND vs the LOCAL OtOpcUa server (3 endpoints) + red vs SQL/non-OPC-UA; Galaxy green vs gateway 10.100.0.48:5120 + red vs non-gRPC. S7/AbCip GREEN SKIPPED — fixtures down (no lmxopcua-fix on the Mac; unit-proven + reviewed). LIVE /run CAUGHT + FIXED 2 REAL GALAXY BUGS unit-tests+2-reviews MISSED (1164d423): (1) Retry={MaxAttempts=1} → invalid Polly RetryStrategyOptions → the probe FAILED on EVERY live gateway (removed it; TCP-preflight+deadline already fail-fast); (2) a rejected key surfaces as a typed MxGatewayAuthenticationException, NOT a raw RpcException, so "auth-rejection=reachable" was bypassed → catch the typed auth/authorization exceptions → Ok=true. Galaxy UseTls must match the endpoint scheme (UseTls:false for http:// h2c — same constraint the driver enforces). New DriverProbeHandshakeE2eTests (skip-gated cross-protocol green/red discrimination). Driver tests all green (Modbus 277, OpcUaClient 122, S7 121, AbCip 276, AbLegacy 168, TwinCAT 145, FOCAS 196, Galaxy 272+3skip). Docs docs/drivers/TestConnectProbes.md. NEW FOLLOW-UPS: (i) S7+AbCip happy-path live-verify (bring up lmxopcua-fix up s7 s7_1500/up abcip controllogix from the Windows VM); (ii) AbLegacy/TwinCAT/FOCAS happy-path live-verify (no PLC5/SLC sim, no ADS target, no CNC+FWLIB). Backlog Phases 6–8 + Phase 4b remain.

STILLPENDING BACKLOG — PHASE 4 (data-type tier: driver data-type & robustness coverage) MERGED+PUSHED master 050f5c4b (2026-06-16, ff, 10 commits) + LIVE deploy-regression-clean. Branch feat/stillpending-phase-4-driver-datatypes ff-merged + deleted. Design …-phase-4-driver-datatypes-design.md (57d9f1b3) + plan …-driver-datatypes.md/.tasks.json (97c087f2). SCOPE NARROWED FROM ROADMAP after grounding (AskUserQuestion locked both): S7 wide types DEFERRED to Phase 4b (driver reads via S7.Net address-strings → wide types need an S7.Net.Types byte-buffer read-path refactor; python-snap7 dev sim's wide-type support unconfirmed) and ALL array work DEFERRED to a dedicated slice (the live equipment-node materialiser EnsureVariable hard-wires ValueRank=Scalar with NO array params + EquipmentTagPlan carries no array metadata + S7/AbLegacy have no array READ support → surfacing arrays end-to-end is a coherent cross-cutting feature: plan-metadata + EnsureVariable contract change + per-driver read + Phase-6 UI, NOT a per-driver flag flip). Shipped 5 changes across 3 disjoint projects (subagent-driven, T1/T2/T4/T5 implementers dispatched CONCURRENTLY — disjoint projects, each commits by path — T3 after T2 same-file; every task spec/code reviewed + final integration review = SHIP): T1 Modbus Int64/UInt64 node type (bd8fee61): ModbusDriver.MapDataType:1517 split Int64⇒DriverDataType.Int64/UInt64⇒UInt64 (both ALREADY existed in the enum + ResolveBuiltInDataType ALREADY mapped "Int64"/"UInt64" → the gap was purely the driver lying Int32; codec round-trips long/ulong UNCHANGED; removed stale Driver.Modbus-007 doc) — 271/271. T2 FOCAS fail-fast factory (5c5aaef6): new IFocasClientFactory.EnsureUsable() (no-op Wire/Fake, throws Unimplemented) called early in InitializeAsync → unimplemented/none/stub backend now FAULTS at init not first-read (operator footgun) — 191/191. T3 FOCAS position scaling (49730752+88d9e3b1 stale-comment review fix): per-device PositionDecimalPlaces config (default 0 = unchanged) divides the 4 position fields by 10^dp at PublishAxisSnapshot; widened LastFixedSnapshots int→double (the 6 refs through it are all already Float64; Int32 fixed-tree vars resolve from separate struct caches → no node-type change); cnc_getfigure auto-fetch stays deferred. T4 Historian Total aggregate (5e27b5f7): ReadProcessedAsync Total → query Average + post-multiply each bucket by interval.TotalSeconds (time-integral identity; Wonderware AnalogSummary has no Total column); null/bad buckets carried through → BadNoData — 35/35. T5 Historian poison dead-letter cap (fcb38014): SqliteStoreAndForwardSink maxAttempts ctor knob (default 10, guard >0) + QueueRow/ReadBatch carry AttemptCount (column pre-existed, NO migration) → RetryPlease dead-letters at AttemptCount+1>=maxAttempts (counts row leaving queue, no drift); threaded AlarmHistorianOptions.MaxAttempts + AddAlarmHistorian DI + Validate() warning — 27/27 + Runtime 267/267. NO contract / DriverDataType / IOpcUaAddressSpaceSink / EF-migration change. Build clean (0 err). LIVE /run: rebuilt central-1/central-2 to the branch → both healthy + Phase7Applier: applied plan (added=90, rebuild=True) + all materialisation + OPC UA serving, NO fatal/unhandled exception (the new driver DLLs + AddAlarmHistorian maxAttempts DI load+deploy clean). Modbus-Int64 end-to-end node advertisement: UNIT-PROVEN (271 tests + the equipment-node "Int64"→DataTypeIds.Int64 resolver) — full live authoring DEFERRED (docker-dev seed has NO Modbus driver; sim 10.100.0.35:5020 reachable from the Mac but not seeded). NEW FOLLOW-UPS: (i) Phase-2 OpcUaProbeResult INFO dead-letters at driver-host/historian-adapter/scripted-alarm-host (those 3 actors subscribe the redundancy topic but don't drop the probe result like OpcUaPublishActor does — benign, capped at 10 then 5-min-suppressed, NOT a Phase-4 change — tidy by adding the same Receive<OpcUaProbeResult>(_=>{}) drop); (ii) full Modbus-Int64 live authoring; (iii) Phase 4b = S7 wide types + Timer/Counter + the dedicated array slice + remaining §2 structural (AbCip/TwinCAT UDT, AbLegacy/TwinCAT bit-index RMW, Galaxy nesting, OpcUaClient event-history, FOCAS cnc_getfigure). Backlog Phases 5–8 remain.

STILLPENDING BACKLOG — PHASE 3 (OPC UA standards completeness: H4 alarm Enable/Disable + H2 HistoryUpdate permission-bit + H6 native-ack→AVEVA) MERGED+PUSHED master 7eeb9fb0 (2026-06-15, ff, 16 commits) + LIVE deploy-regression-clean. Branch feat/stillpending-phase-3-opcua-standards ff-merged + deleted. Design …-phase-3-opcua-standards-design.md (40b883ef) + plan …-standards.md/.tasks.json (6ab3d863). Subagent-driven, every task spec+code reviewed (serial for high-risk node-manager/driver/contract tasks) + final integration review = SHIP (all 7 cross-cutting checks PASS, incl. the load-bearing native-ack round-trip + node-id seam). H2-bit (c236263e): NodePermissions.HistoryUpdate = 1<<12 (int-backed flags enum, NO migration; excluded from all composites — service stays deferred) + TriePermissionEvaluator:86 maps HistoryUpdate→its own bit (was HistoryRead — closed the read⇒update hole). H4 (328bd1b9+be6858ba): wired the missing node-manager OnEnableDisable (SDK ConditionEnableEventHandler(ctx,cond,bool enabling), verified via reflection) → scripted = AlarmAck-gated HandleAlarmCommand("Enable"/"Disable")→engine (already handled it); native→BadNotSupported; native-check via lock-guarded IsNativeAlarmNode. H6: 418663b3 thread bool isNative through the SINGLE shared MaterialiseAlarmCondition (Phase7Applier true@:204/false@:295 → _nativeAlarmNodeIds); ed941c51 AlarmAcknowledgeRequest.OperatorUser (additive) + GalaxyDriver forwards + ScriptedAlarmSource honors-else-opcua-client; a6d9de09+87dd65b9 native OnAcknowledge→NEW NativeAlarmAckRouter (AlarmAck-gated HandleNativeAlarmAck, principal=DisplayName; scripted ack unchanged→engine); 93d9160d DriverHostActor inverse map conditionNodeId→(driver,FullName) (lock-step w/ _alarmNodeIdByDriverRef, both keyed by EquipmentNodeIds.Variable(EquipmentId,FolderPath,Name)) + primary-gated RouteNativeAlarmAck→DriverInstanceActor→driver.AcknowledgeAsync (fire-and-forget, Connected-only); 30315185 host DI maps NativeAlarmAck→RouteNativeAlarmAck (Runtime has NO OpcUaServer dep). GOTCHAS: native-check MUST use lock-guarded IsNativeAlarmNode (raw set read races a concurrent RebuildAddressSpace — OnTimedUnshelve bypasses the Call-service lock); node-id seam airtight because BOTH sides derive from the same EquipmentNodeIds.Variable over the same composition (null/empty FolderPath normalizes identically). NO EF migration; only additive contracts. Also fixed a PRE-EXISTING Phase-0 miss (8899d6e0): 2 Galaxy guard tests asserted "PR 4.4"/"PR 4.W" text that Phase 0 b4af9e7f removed → updated to current guard text. Build clean; Core 237/237, OpcUaServer 201/201, Runtime 267/267, Galaxy 264/264(+3 live-gw skip). LIVE /run: rebuilt central-1/central-2 to the branch → healthy startup + Phase7Applier: applied plan (added=90, rebuild=True) + scripted alarms materialised (alarms=1) + OPC UA serving (ServiceLevel 250) — substantial node-manager/DriverHostActor/host changes deploy+materialise+serve with NO regression. H4 client-Disable BLOCKED: Client.CLI has no enable/disable command (plan assumed one) → H4 fully unit-proven; CLI enable/disable = a FOLLOW-UP. H6 full native-ack→AVEVA OPERATOR-GATED: needs a Galaxy NATIVE alarm (dev rig's 1 alarm is scripted) + the real gateway/AVEVA on 10.100.0.48; round-trip wiring unit+integration-proven. Phases 4–8 remain.

STILLPENDING BACKLOG — PHASE 2 (H3 health-aware redundancy ServiceLevel) MERGED+PUSHED master 4af8e65a (2026-06-15, ff, 14 commits) + LIVE-PROVEN on the docker-dev 2-node rig. Branch feat/stillpending-phase-2-servicelevel ff-merged + deleted. Design docs/plans/2026-06-15-stillpending-phase-2-servicelevel-design.md (05283533) + plan …-servicelevel.md/.tasks.json (7605f4d8). LIVE /run ACCEPTANCE PASSED (agent-driven, login disabled): rebuilt central-1+central-2 to the branch (docker compose -f docker-dev/docker-compose.yml up -d --build central-1 central-2); read Server.ServiceLevel (NodeId i=2267) via Client.CLI on :4840 (leader) + :4841 (follower). BEFORE (old code): 240 / 100 (legacy role-only). AFTER (branch, healthy): 250 / 240 (calculator path LIVE — the follower 100→240 + leader 240→250 is the discriminator vs legacy; proves dbHealthProbe Ask delivery + calculator invocation + NodeId self-match + the +10 leader bonus all live). DB-down (stopped shared otopcua-dev-sql-1): 110 / 100 (DbHealthProbe: probe failed → Stale=!DbReachable → 100-tier, leader +10=110 — health-aware demotion). Recovered (SQL restarted): 250 / 240. Rig left healthy. (NOTE: stopping the SHARED SQL demotes BOTH nodes; single-node asymmetric isolation would need a per-node DB network block the rig doesn't cleanly support, but the both-demote + recover proof + the steady-state 250/240-vs-legacy-240/100 discriminator fully validate the wiring.) Wires the existing-but-never-invoked ServiceLevelCalculator into the live per-node publish path so a DB-unreachable / OPC-UA-unreachable node drops below its role level. Subagent-driven, every task spec+code reviewed (serial for the high-risk actor tasks) + a final integration review = SHIP. Commits: ff0f62db move ServiceLevelCalculator+NodeHealthInputs → Core.Cluster (ZB.MOM.WW.OtOpcUa.Cluster.Redundancy, cycle-free — Runtime already refs Core.Cluster, AVOIDS the deliberate no-Runtime→ControlPlane boundary); 3e609a2b+a9ff1a64 OpcUaPublishActor.RecomputeServiceLevel calls the calculator (DB+Stale+leader) with a back-compat legacy seam (no dbHealthProbe wired → old role-only 240/200/100/0, keeps the 2 pre-existing tests green) + Detached/no-entry guard → 0 + always-publish-first-even-0; cf278035+5382eea9 OpcUaProbeOk from peer-probes-me (consume OpcUaProbeResult where NodeId==local; absent/stale verdict→true, only a RECENT observed false demotes); 37b32a56 periodic HealthTick Ask→PipeTo the local DbHealthProbeActor (Ask-timeout→Reachable=false fail-safe-demote); f41e957e+5a064e08 NEW PeerProbeSupervisor spawns one PeerOpcUaProbeActor per non-self non-Detached driver peer (monotonic gen-suffix child names + Watch/Terminated-by-ref prune — stale Terminated can't evict a fresh same-peer child); 70e6d3d2+393b746d docs/Redundancy.md WIRED; 4c78dcd3 DI wiring (dbHealthProbe: dbHealth into OpcUaPublishActor.Props + spawn PeerProbeSupervisor + PeerProbeSupervisorKey); 4af8e65a supervisor explicitly drops co-mingled OpcUaProbeResult (no unhandled-msg noise). Tiers (documented, now reachable): 0 down/Detached · 100 DB-down⇒stale · 200 DB-ok-stale · 240 healthy follower · 250 healthy leader. Behavior change: healthy Secondary 100→240 (both healthy nodes 240/250, leader preferred by +10 — clients still pick primary). Stale = !DbReachable || (now-dbAsOf)>30s || (now-snapshotAsOf)>30s. Integration review's #1 risk (NodeId canonical form across the probe→publish seam) = PASS: both RedundancyStateActor.ToNodeId(member.Address) + ClusterRoleInfo.LocalNode derive host:port from the same PublicHostname/Port that configures Akka remoting (same class as project_redundancy_state_delivery, consistent here). NO EF migration, NO Commons contract change. Build clean; ControlPlane.Tests 59/59, Runtime.Tests 264/264. LIVE /run GATE (Task 8, high-risk acceptance) PENDING: on docker-dev MAIN (central-1/central-2 2-node warm), read Server.ServiceLevel on :4840/:4841 → leader 250 / follower 240; isolate ONE central node from SQL (docker network disconnect — do NOT stop the shared SQL container) → its ServiceLevel drops to 100 (below the healthy peer's 240); reconnect → climbs back. Agent drives docker-dev directly (login disabled). Phases 3–8 of the backlog remain (each gets its own plan when reached).

STILLPENDING BACKLOG — PHASE 0 + PHASE 1 (H1+H5) MERGED+PUSHED master c6a543d1 (ff, 15 commits) 2026-06-15; H1 LIVE-PROVEN on docker-dev. Then 907005d2 (CLAUDE.md: local rig login-disabled → run /run yourself) + 4bd7180e (fix docker-dev seed: stop seeding retired SystemPlatform namespace). LIVE VERIFY: rebuilt local rig to master, drove AdminUI localhost:9200 (login disabled), toggled layer0-logcheck (EQ-55297329838d) Historize → Deploy → Phase7Applier: applied plan (added=0, removed=0, changed=1, rebuild=True) + sealed (acks=6) — a PURE changed-only deploy now rebuilds (pre-H1 = rebuild=False); reverted = 2nd proof; config pristine. Blocker found+fixed: docker-dev seed still inserted Kind='SystemPlatform' (retired) → 500s /deployments + fails publish; ran sanctioned CleanupSystemPlatformNamespaces SQL on dev DB + fixed seed. H5 threading flows through real deploy; durable IHistoryWriter.Record silent (NullHistoryWriter infra-gated). (Historical IN-PROGRESS detail below.) Driven by the approved roadmap docs/plans/2026-06-15-stillpending-backlog-design.md + plan …-phase-0-1.md (full backlog audit = stillpending.md). DONE on the branch (subagent-driven, each task spec+code reviewed, all green): Phase 0 hygiene (b4af9e7f 7 stale comments; a9d267c9 security.md write-outcome + benign DraftSnapshot/LeaderChanged notes; 50a2fdf3 marked 18 shipped .tasks.json completed). H1 — changed-only deploys now take effect (1e95856b Phase7Applier needsRebuild includes Changed* counts, ChangedDrivers excluded by design; 1dc71369 count removed tags/vtags in RemovedNodes [review follow-up]; ada01e1a VirtualTagHostActor stop+respawns a child whose plan changed in place; ebf2f1dd prune _planByVtag on termination + crash-then-change test [review follow-up]). H5 — vtag Historize honored at runtime (fc8121cb EquipmentVirtualTagPlan.Historize + composer reads v.Historize, in Equals/GetHashCode; 9c5a0913 artifact decode reads Historize byte-parity; 83d3b9f7 planner Historize-toggle test + doc nit; 0c6d4c54 VirtualTagHostActor invokes injected IHistoryWriter on historized results [+ Runtime→Core.VirtualTags ProjectReference]; 2f30c54d thread-safe CapturingHistoryWriter [review follow-up]; 4501f12 thread IHistoryWriter through DriverHostActor + DI TryAddSingleton Null default). DURABLE AVEVA vtag-history sink = INFRA-GATED (no sidecar live-data WriteDataValues RPC) → seam wired w/ NullHistoryWriter default, flagged on the design defer list. NO EF migration (Historize column + ConfigComposer whole-entity serialization already carry it). REMAINING on branch: Task 10 docs (this commit) + Task 11 final build/test/integration-review + finishing (merge is the user's call). LIVE /run gate (user-driven): edit equipment/alarm/vtag → confirm it takes effect without restart; toggle a vtag Historize → host respawns + writer invoked.

#401 — AbCip + FOCAS ALARM-PROJECTION POLL-LOOP LEAK FIXED + MERGED + PUSHED master 151b7165 (2026-06-15, ff, 2 commits 6ba59f9d+151b7165). SIBLING of #399 found by auditing all IAlarmSource drivers, but WORSE: AbCipAlarmProjection + FocasAlarmProjection each Task.Run a poll loop per SubscribeAsync and survive in-place reconnect, so the actor's per-Connected re-subscribe leaked a LIVE never-cancelled poll loop each reconnect → CPU growth + DUPLICATE OnAlarmEvent transitions (functional, gated behind opt-in alarm projection). Fix (both projections, symmetric): SubscribeAsync snapshots existing subs under _subsLock, _subs.Clear(), adds the new sub, starts its loop, then _ = RetireAsync(old) fire-and-forget (Cancel→await loop→Dispose CTS). Snapshot+clear under the same lock DisposeAsync uses ⇒ no double-own/double-dispose (reviewer-confirmed airtight for all interleavings). Implemented via opus subagent (TDD fail→pass: subscribe twice → 1 device raise → exactly ONE event, was 2). AbCip 271/271, FOCAS 186/186. Code-reviewed (sonnet) — correct+safe; applied its 2 nice-to-haves (documented the at-most-ONE-duplicate-per-reconnect one-tick overlap residual in RetireAsync XML doc + guarded Cts.Dispose). AUDIT RESULT: all 4 IAlarmSource drivers now correct — Galaxy (#399), AbCip+FOCAS (#401), OpcUaClient already safe (session-bound + clears _alarmSubscriptions on disconnect).

#399 — GALAXY ALARM-SUBSCRIPTION HANDLE LEAK FIXED + MERGED + PUSHED master 01388226 (2026-06-15, ff, 1 commit). The session-less Galaxy StreamAlarms feed survives an in-place reconnect, so DriverInstanceActor re-subscribed on every Connected re-entry (dropping its cached handle without an Unsubscribe — sync teardown) and GalaxyDriver.SubscribeAlarmsAsync grew _alarmSubscriptions by one untracked handle per reconnect (slow unbounded leak; functionally harmless — gate is Count>0, fan-out reads [0] once). Fix: SubscribeAlarmsAsync Clear()s the set before Add (under the existing _alarmHandlersLock, atomic w.r.t. the fan-out reader), collapsing to ONE live handle — faithful because there's exactly one consumer per driver instance (factory-per-actor lifecycle). Chosen over async/fire-and-forget actor-side Unsubscribe (disproportionate). NEW regression test Re_subscribe_collapses_to_a_single_handle_no_accumulation (TDD-VERIFIED fail-without-fix: releasing the latest handle leaves the feed open if stale handles remain). Galaxy 263 pass/3 skip, Runtime native-alarm 24 pass. Code-reviewed (sonnet) — correct+complete, both nice-to-haves applied (structural-invariant comment + distinct-handle assert).

#400 — within-timestamp cursor paging for oversized tie clusters: SHIPPED to master 2e6c6d3a + review fix 0f929ae6 (2026-06-17). No longer a documented follow-up. HistoryPaging.SliceTieCluster over-fetches the whole tie cluster (a bounded start==end read), carves out the next cap ties at skip, advances the cursor WITHIN the timestamp, and steps off it (T + 1 tick) once drained — wired into OtOpcUaNodeManager.ServeRawPaged (stall detected when a resume page's last sample is still at the boundary timestamp). New config ServerHistorian:MaxTieClusterOverfetch (default 65536) bounds the in-memory buffer with a Validate() warning; 6 SliceTieCluster unit tests + integration tests Raw_oversized_tie_cluster_pages_within_the_timestamp / Raw_tie_cluster_beyond_overfetch_bound_fails_loudly; docs/Historian.md "within-timestamp paging" section + config-table row updated. DELIBERATE residual backstop: a cluster strictly larger than MaxTieClusterOverfetch ties at one identical SourceTimestamp still loud-fails BadHistoryOperationUnsupported (a severe data anomaly — bounded on purpose to cap memory; raise the knob or NumValuesPerNode to cover it).

#398 — RESTART-DRIVER "actor name is not unique" BUG FIXED + MERGED + PUSHED master c9643f68 (2026-06-15, ff, 1 commit). DriverHostActor.HandleRestartDriver stopped + respawned the child in ONE synchronous handler reusing the base actor name drv-<id>; Context.Stop is async so the old child was ALWAYS still registered → InvalidActorNameException deterministically on every AdminUI Restart press (crashed+restarted the host). Fix: a monotonic _childSpawnGeneration counter feeds a -g<gen> suffix on every spawned child name (children tracked by the _children dict via IActorRef, never by path, so the suffix is invisible). Also closes the same-shaped LATENT race in the reconcile path (removed-then-readded instance + a driver-type-change ToStop+ToSpawn in one plan — confirmed by review). NEW regression test RestartDriver_respawns_the_child_without_an_actor_name_collision (TDD-VERIFIED: fails on the old code with the exact InvalidActorNameException via EventFilter.Exception<…>().Expect(0,…), passes with the fix). Runtime.Tests 238/238. Code-reviewed (sonnet) — correct+complete, no Critical/Important. HandleReconnectDriver confirmed bug-free (only Tells ForceReconnect).

⚠️ OUT-OF-BAND WORKING-TREE CHANGE I DID NOT MAKE (surfaced to user 2026-06-15, NOT touched): Directory.Packages.props is modified (ZB.MOM.WW.MxGateway.Client + …Contracts bumped 0.1.0→0.1.1) and two new nuget-packages/ZB.MOM.WW.MxGateway.{Client,Contracts}.0.1.1.nupkg (dated 2026-06-15 05:39) appeared during this session. These are EXTERNAL (likely a sibling-repo publish / parallel action) — NOT part of my B2 or #398 commits (both clean, verified). Left untouched per the "didn't-create-it → surface, don't act" rule; awaiting user decision whether to commit, revert, or ignore. nuget-packages/ is NOT gitignored (tracked dir).

B4 — docker-dev rig cleanup RESOLVED 2026-06-15 as "LEAVE-AS-IS" (user decision). The rig keeps this session's verification artifacts on purpose: the historized dev GalaxyTestTag (= TestMachine_002.TestHistoryValue) + the local-only docker-dev/docker-compose.yml env block (ServerHistorian__* with SharedSecret via ${OTOPCUA_HISTORIAN_SECRET:-}, + the data-plane Security__Ldap__GroupToRole__* map) so A2 (historian) + A3 (native alarms) + B1 (write-outcome) stay re-verifiable. docker-compose.yml stays an uncommitted DISK-ONLY file (never staged). No teardown, no commit — matches the earlier "leave the Galaxy verification artifacts as-is" decision. All buckets (A2/A3 live, B1/B2/B3 code, B4 operational) are now CLOSED. Remaining are logged follow-ups only: #398 (pre-existing Restart actor-name-not-unique), #399 (minor session-less alarm handle leak), #400 (within-timestamp cursor paging — now SHIPPED 2e6c6d3a/0f929ae6 2026-06-17).

B2 — HISTORIAN HISTORYREAD CONTINUATION-POINT PAGING (Raw) MERGED + PUSHED to master aa1e21f5 (2026-06-15, fast-forward, 3 commits: 94c3ca60 feature, bea0b482 review fixes, aa1e21f5 doc). Server-side time-based paging for the only count-capped arm (Raw): full page (count==NumValuesPerNode>0) ⇒ opaque 16-byte CP + tie-safe resume cursor (HistoryPaging.ComputeResumeCursor/TrimBoundaryDuplicates — resume from boundary timestamp INCLUSIVE, drop already-emitted head ties → no dup/skip); short page / ==0 (all values) ⇒ no CP. CPs bound to the OPC UA session via the SDK store (SessionHistoryContinuationStore, capped by ServerConfiguration.MaxHistoryContinuationPoints=100 oldest-evict, disposed on session close); tests inject InMemoryHistoryContinuationStore (in-process harness is session-less). HistoryReleaseContinuationPoints override frees cursors; unknown/evicted/released ⇒ BadContinuationPointInvalid. Processed/AtTime stay single-shot (no client count cap → no full-page signal; spec-conformant). NEW: HistoryPaging.cs, HistoryContinuationStore.cs + HistoryPagingTests.cs (15) + NodeManagerHistoryReadPagingTests.cs (7, boot a real SDK server). 182/182 OpcUaServer tests pass, full solution builds clean. Code-reviewed (sonnet subagent): C1 (critical) FIXED — an oversized boundary tie cluster (more ties at one SourceTimestamp than NumValuesPerNode) could silently truncate a resume to GoodNoData and DROP the un-emitted ties (the (timestamp,skip) cursor can't advance past a single timestamp the fixed-(start,end,cap) backend keeps re-returning); now fails LOUDLY per node with BadHistoryOperationUnsupported + a log + docs/Historian.md "Paging limitation" note + regression test Raw_tie_cluster_larger_than_page_fails_loudly_not_silently. Also I3 (Save after HistoryData build → no orphaned cursor), N1 (dropped never-produced HistoryReadKind enum + Processed-only state fields — only Raw pages), N3 (ComputeResumeCursor empty-page guard), I1 (documented test-store eventual-consistency). Full within-timestamp cursor paging = task #400, SHIPPED 2e6c6d3a/0f929ae6 (2026-06-17) — HistoryPaging.SliceTieCluster + MaxTieClusterOverfetch-bounded over-fetch; only clusters beyond the bound still loud-fail by design. Modified-value history (HistoryReadModified) is INFRA-GATED, not a code gap: the AVEVA Wonderware backend (IHistorianDataSource TCP sidecar) has no modified-read path; server cleanly rejects per node + IsReadModified honoured; doc note added. B2 COMPLETE. (B1 write-outcome residuals + B3 GroupToRole doc already on master a5c0c826/063d004f.)

GALAXY LIVE-GW INTEGRATION SMOKES (reopen + write-outcome) MERGED + PUSHED to master d19deb9b (2026-06-14, fast-forward, 2 commits). Closed the two items the user re-opened with a live gateway: the harden-1b Galaxy-reopen integration test + the write-outcome Galaxy write-commit/persist proof. NEW skip-gated tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests/Runtime/GatewayGalaxyLiveReopenAndWriteTests.cs (gated on MXGW_ENDPOINT + GALAXY_MXGW_API_KEY, like GatewayGalaxyAlarmFeedLiveTests; skips clean in CI — verified). LIVE-PROVEN 2/2 against http://10.100.0.48:5120 (2026-06-14): (1) Live_reopen_recreates_session_and_re_establishes_write_handles — a live write seeds the writer's item-handle + supervisory-advise caches, then the EXACT GalaxyDriver.ReopenAsync primitive pair runs (GalaxyMxSession.RecreateAsync → GatewayGalaxyDataWriter.InvalidateHandleCaches), both caches drop to 0, and a post-reopen write re-establishes them + returns Good — proving RecreateAsync yields a working live session (the un-fakeable part the unit suite can't reach); (2) Live_supervisory_write_commits_and_persists_to_galaxy — writes a value distinct from current, then reads it back through a BRAND-NEW session — proving the no-login (WriteUserId=0) write COMMITS + PERSISTS only because of AdviseSupervisory (the f05b5d79 fix). Code-review APPROVED (refinement applied: read-back completes an explicit TaskCompletionSource<double?> from the stream pump; bufferedUpdateIntervalMs:0). DURABLE RUN RECIPE: the gw 10.100.0.48:5120 IS reachable from this Mac; source the key WITHOUT echoing it — KEY=$(docker exec otopcua-dev-central-1-1 printenv GALAXY_MXGW_API_KEY); MXGW_ENDPOINT=http://10.100.0.48:5120 GALAXY_MXGW_API_KEY="$KEY" dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests --filter FullyQualifiedName~GatewayGalaxyLiveReopenAndWrite (central-1's container env carries a valid key while its galaxy session is connected; check with docker exec … sh -c 'test -n "$GALAXY_MXGW_API_KEY" && echo SET'). Writes only TestMachine_002.TestFloat (the established write-demo attr).

RESIDUAL FOLLOW-UPS CLEANUP MERGED + PUSHED to master fa2388ea (2026-06-14, fast-forward, 7 commits = 1 plan doc + 6 test/doc). Closed the offline-doable, non-blocking residuals from open-item #3 (write-pipeline / Phase B / Galaxy-nits sections) — TESTS + DOCS only, ZERO production change. Plan docs/plans/2026-06-14-residual-followups-cleanup-plan.md. KEY RECONCILIATION FINDING: a grounding sweep (3 Explore agents + direct verify) proved MOST of the listed residuals were ALREADY implemented in the harden-1b / write-outcome / f05b5d79 work — pending.md's residual list was STALE. Verified already-done (no work needed, file:line evidence): WP(a) stale/reconnecting writes fast-fail (DriverHostActor.cs:661 Receive<RouteNodeWrite>, DriverInstanceActor.cs:229,316 Receive<WriteAttribute>); WP(b) ExecuteSynchronously already dropped (DriverHostActor.cs:630 = TaskContinuationOptions.None) + driverIds forward-map already a HashSet; WP(c) FOCAS address cached via GetOrAdd (FocasDriver.cs:247); WP(d) raw-protocol-blob write test exists (Primary_routes_write_for_raw_protocol_blob_tag); WP(e) [InlineData(2,false)] future-enum trap already in the parity test; Galaxy(1) _itemHandles+_supervisedHandles cleared on reconnect (GatewayGalaxyDataWriter.InvalidateHandleCaches() ← GalaxyDriver.ReopenAsync); Galaxy(2) SubscriptionEstablished self-Tell handled (debug-swallow) in all 3 DriverInstanceActor states (:253,:301,:336). ACTUALLY DONE this pass (4 items): Phase B(a) regression test that a NativeAlarmRaised during Reconnecting is DROPPED not dead-lettered (uses an AllDeadLetters EventStream probe so it genuinely guards removal of the DriverInstanceActor.cs:345 drop handler — the first cut used parent.ExpectNoMsg only, which a code review caught as a false-pass gap); Phase B(b) test that OperatorComment→AlarmTransitionEvent.Comment (and User=="device") through DriverHostActor.ForwardNativeAlarm; Phase B(c) severity-bucket doc note in docs/ScriptedAlarms.md (each transition snaps to 200/500/700/900 via NativeAlarmProjector.MapSeverity, overriding the authored seed, then SDK EventSeverity brackets via OtOpcUaNodeManager.MapSeverity); Galaxy(3) extracted shared tests/.../Runtime.Tests/Drivers/StubDrivers.cs harness (de-dup of the superset StubDriver/WritableStubDriver/SubscribableStubDriver/StubHandle from DriverInstanceActorTests + DriverInstanceActorWriteAndSubscribeTests; BlockingInitDriver left local). Built via subagent-driven dev (1 implementer bundle → code-review APPROVED w/ 1 Important = the dead-letter-probe gap, fixed). Build clean; Runtime.Tests 236/0. (The harden-1b Galaxy-reopen integration test noted here as deferred is NOW DONE — see the Galaxy live-gw smokes banner above, d19deb9b.)

GALAXY PHASE C — SERVER-SIDE OPC UA HistoryRead MERGED + PUSHED to master c24abc8a (2026-06-14, fast-forward, 14 commits = 2 design/plan docs + 12 feature/test/doc). The server now answers OPC UA HistoryRead (Raw / Processed / AtTime over historized variable nodes; Events over alarm-owning equipment-folder event-notifier nodes) for any equipment tag flagged historized, driver-agnostically, by dispatching to the registered IHistorianDataSource (the Wonderware historian TCP client, which already implemented that interface). NO EF migration — the flag rides in the existing TagConfig JSON blob ({"FullName":"…","isHistorized":true,"historianTagname":"…"?}, the Phase-B alarm-object carrier), historianTagname defaults to the tag's driver FullName. Design/plan docs/plans/2026-06-14-galaxy-phase-c-historian-*.md; guide docs/Historian.md. Pipeline: Phase7Composer.ExtractTagHistorize + byte-parity DeploymentArtifact.ExtractTagHistorize → EquipmentTagPlan.{IsHistorized,HistorianTagname} → Phase7Applier resolves IsHistorized ? (HistorianTagname ?? FullName) : null → sink seam (IOpcUaAddressSpaceSink.EnsureVariable +string? historianTagname) → OtOpcUaNodeManager.EnsureVariable sets Historizing+AccessLevels.HistoryRead+registers _historizedTagnames; the node manager overrides the four CustomNodeManager2 HistoryRead virtuals, block-bridging to the HistorianDataSource property (volatile, default NullHistorianDataSource.Instance). DI mirrors AlarmHistorian: AddServerHistorian (config-gated, ServerHistorian appsettings section, Null default via TryAddSingleton) + OtOpcUaSdkServer.SetHistorianDataSource + Host Program.cs/OtOpcUaServerHostedService Start/Stop wiring. Graceful degrade: historized node + Null/unconfigured source → Good_NoData (empty), non-historized node → BadHistoryOperationUnsupported. KEY FACTS/GOTCHAS: the SDK base filters event reads by the EventNotifier.HistoryRead bit (variable nodes never reach the events arm); the SDK master propagates errors[i].Code → results[i].StatusCode (confirmed by decompiling MasterNodeManager.HistoryReadAsync), so the override signals per-node status via errors[handle.Index]; the two HistoryReadResult types (SDK Opc.Ua.HistoryReadResult vs Core.Abstractions DTO) are aliased SdkHistoryReadResult/HistorianRead; ReadRawModifiedDetails.IsReadModified defaults TRUE (Initialize() sets it) so a plain raw read must clear it — modified-history is unsupported; the events arm registers folder→sourceName (= equipment id) only when a non-Null historian is wired at promotion time (Host wires the source at StartAsync before any deployment materialises, so normal boot ordering is correct). Built via subagent-driven dev (T1/T2/T5 standard parallel review, T3/T4 high-risk serial spec→code, + a final integration review READY-TO-MERGE). Build clean (0 errors); OpcUaServer.Tests 152/0, Runtime.Tests 234/0, Core.Abstractions.Tests 88/0. LIVE /run GATE (T7) DEFERRED — operator-driven: it needs the Wonderware sidecar + AVEVA Historian on the WW Historian VM 10.100.0.48, which is NOT on the local docker-dev rig. When run: author a historized Galaxy tag (TagConfig "isHistorized":true), set ServerHistorian:Enabled=true → sidecar (Host/Port/SharedSecret/TLS), deploy on MAIN-galaxy-eq, then Client.CLI historyread -n "ns=2;s=<equip>/<tag>" --start … --end … → samples; a non-historized tag → BadHistoryOperationUnsupported. KNOWN follow-ups (non-blocking, documented): single-shot reads only (no server-managed continuation-point paging yet); no modified-value history; no explicit timeout at the block-bridge call site (bounded by the WonderwareHistorianClient 30s CallTimeout); the StopAsync→DisposeAsync warm-shutdown window returns Good_NoData (mirrors the SetNodeWriteGateway pattern).

DRIVER-RECONFIGURE-WHILE-FAULTED (#7) MERGED + PUSHED to master 56f73e49 (2026-06-14, fast-forward, 5 commits = 2 docs + fix + review-nit + task-status). A DriverInstanceActor stuck Connecting/Reconnecting now adopts a corrected config delivered via ApplyDelta and re-initialises with it, instead of dead-lettering the message and retrying the stale config forever (old workaround = restart node). Design/plan docs/plans/2026-06-14-driver-reconfigure-while-faulted-*.md. Mechanism (approach B): a monotonic _initGeneration tags each InitializeAsync; InitializeSucceeded(int Generation)/InitializeFailed(string Reason, int Generation) carry it; the Connecting/Reconnecting result handlers drop superseded (stale-generation) results so a corrected config always wins against an old init still in flight; a new AdoptConfigDuringInit (wired into both not-connected states) calls InitializeAsync(newConfig) (swaps _currentConfigJson, bumps the generation, retries immediately) + replies ApplyResult(true,…). Contained to ONE file (src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs) + its test — NO host/contract/EF change; Connected/Stubbed ApplyDelta paths untouched; the two result records have ZERO external consumers (grep-verified) so the shape change is fully contained. Built via subagent-driven dev (high-risk full chain: spec ✅ · code ✅ · final integration ✅ — the integration pass traced the no-strand lifecycle in both states, host↔child contract, subscription/desired-refs/alarm safety, health/redundancy, double-adopt, and test fidelity, all clean). Build clean (0 errors); Runtime 224/224. LIVE /run GATE DEFERRED per user ("skip the live test part"). When run: put MAIN-opcua-eq into a faulted/Reconnecting state via a bad DriverConfig, deploy a corrected config (POST http://localhost:9200/api/deployments, X-Api-Key: docker-dev-deploy-key), and confirm from central-1 logs that the driver adopts the new config + connects WITHOUT a node restart.

GALAXY PHASE B — NATIVE ALARMS ON THE EQUIPMENT-TAG PATH MERGED + PUSHED to master f9be3843 (2026-06-14, fast-forward, 12 feature + 2 doc commits). A Galaxy equipment Tag marked as a native alarm via its TagConfig.alarm object — {"FullName":"tag.attr","alarm":{"alarmType":"OffNormalAlarm","severity":700}}, NO EF migration — now materialises a real OPC UA Part 9 AlarmConditionState under its equipment folder, driven live by the driver's IAlarmSource.OnAlarmEvent; transitions fan out to /alerts + the historian (Primary-gated). Design/plan docs/plans/2026-06-14-galaxy-phase-b-native-alarms-*.md. NEW seam (mirrors the scripted-alarm seam, reuses the condition sink UNCHANGED): AlarmEventArgs.Kind (additive contract; Galaxy populates it) → DriverInstanceActor subscribes OnAlarmEvent → AttributeAlarmPublished → DriverHostActor._alarmNodeIdByDriverRef + NativeAlarmProjector (transition→AlarmConditionSnapshot) → OpcUaPublishActor.AlarmStateUpdate → reused OtOpcUaNodeManager.WriteAlarmCondition. Built via subagent-driven dev (full per-task review chain). Build clean; Core.Abstractions 81/0, OpcUaServer 118/0, Runtime 222/0, Galaxy 262/0(+1 live-gw skip). THE FINAL INTEGRATION REVIEW CAUGHT A CRITICAL SEAM BUG every unit test missed (fixed in f9be3843): the alarm map is keyed by the dotted FullName, but GalaxyDriver puts the BARE owning object in AlarmEventArgs.SourceNodeId and the DOTTED alarm ref in ConditionId (GalaxyDriver.cs:1148-1149; AlarmFullReference) — so ForwardNativeAlarm MUST resolve on msg.Args.ConditionId (= AlarmFullReference = the authored FullName), NOT SourceNodeId. The unit test had masked it by setting SourceNodeId==FullName (never true in prod); it is now production-shaped (SourceNodeId="Temp" ≠ ConditionId="Temp.HiHi"=FullName) so it genuinely guards the seam. LIVE /run GATE (T9) NOT YET DONE — user-driven, deferred at merge (user choice "merge now"). When run: author a Galaxy alarm equipment tag whose FullName EXACTLY matches the gateway's AlarmFullReference (discoverable via the Galaxy picker/a probe), deploy on MAIN-galaxy-eq, trip the alarm → Part 9 condition goes active under the equipment + the /alerts row appears; clear → inactive. RESIDUAL non-blocking follow-ups (review-surfaced): (a) DetachSubscription alarm-coupling doc note + a dead-letter-during-reconnect regression test (WS-4b); (b) ack/comment-path test + assert evt.Comment (WS-5); (c) a docs/ScriptedAlarms.md note that authored severity (1..1000, seeds the condition at materialise) snaps to the projector's 4-bucket value (200/500/700/900) on the first transition; (d) DEFERRED by design: inbound device-ack (client Ack → IAlarmSource.AcknowledgeAsync → AVEVA), driving SubscribeAlarmsAsync from the materialised alarm-ref set (Galaxy doesn't need it), AdminUI Galaxy-picker alarm pre-fill, carrying raw OPC UA severity end-to-end. Phase C (server HistoryRead) DONE — merged c24abc8a (2026-06-14); see the Phase C banner at top.

WRITE-OUTCOME SELF-CORRECTION (#5) MERGED + PUSHED to master 1d797c1c (2026-06-14, fast-forward, 6 commits). A failed inbound device write now reverts the node to its real pre-write value (compare-and-revert) instead of leaving the optimistic-Good phantom. Design docs/plans/2026-06-14-write-outcome-self-correction-design.md; plan …-plan.md. NEW IOpcUaNodeWriteGateway/NodeWriteOutcome (Commons) + ActorNodeWriteGateway (Runtime, Asks RouteNodeWrite, returns the outcome) replace the fire-and-forget Action router; OnEquipmentTagWrite captures the prior value + fires an off-Lock (RunContinuationsAsynchronously) continuation that reverts on a failed outcome IF the node still holds the optimistic value (ShouldRevert). Build clean; Commons 39/0, Runtime 201/0, OpcUaServer 111/0. High-risk review verified the prior-value capture against the actual UA-.NETStandard source (OnWriteValue fires before m_value=value). LIVE-PROVEN end-to-end via a local Modbus exception-injector (FC06 reject on HR[20]): authorized failing write → gateway-logged 0x808B0000 reject → node reverts 99→20; authorized success (HR200=7777) stays; anon → BadUserAccessDenied. KEY FINDING: the Galaxy gateway worker's ExecuteWrite is fire-and-forget (returns OK without awaiting the MXAccess commit), so Galaxy writes ALWAYS return Success at the OPC UA layer and can NEVER surface a device-write failure to this revert — only protocol drivers (which await + return real status) can; this is the same gateway-side limitation noted under "optimistic-write phantom" (out of our scope). Two MINOR deferred follow-ups remain: a Bad-quality blip / OPC UA AuditWriteUpdateEvent on failure, and synchronous structural fail-fast (both explicitly out of scope per the chosen mechanism).

HARDEN MILESTONE 1b cluster MERGED + PUSHED to master 945c2380 (2026-06-14, fast-forward, 9 commits). Follow-ups #3 (data-plane role docs), #4 (write-pipeline review nits), and #6 (Galaxy driver nits) below are CLOSED. Plan: docs/plans/2026-06-14-harden-milestone-1b-plan.md. Build clean; Runtime 197/0, FOCAS 185/0, Galaxy 257/0(+1 pre-existing skip); final integration review READY-TO-MERGE. Two MINOR residual follow-ups surfaced by review (non-blocking): (a) a driver-level regression test that GalaxyDriver.ReopenAsync actually calls InvalidateHandleCaches — STILL DEFERRED, needs a live gw (RecreateAsync can't be faked), so it's an integration test; (b) stub-driver test-class duplication between DriverInstanceActorTests + DriverInstanceActorWriteAndSubscribeTests — DONE fa2388ea (shared StubDrivers.cs).

STATE SUMMARY (post-compaction pickup)

ALL feature work is SHIPPED + PUSHED to master d19deb9b (synced with origin). Nothing is blocking. Milestone 1b (equipment-tag live values: live READ + authorized inbound WRITE across OpcUaClient / the 6 protocol drivers / Galaxy, via the FullName→NodeId router) is COMPLETE, and all three Galaxy phases shipped: A standard Equipment driver c3c56172, B native alarms f9be3843, C server-side HistoryRead c24abc8a. The session's cluster-harden / write-outcome-self-correction / driver-reconfigure-while-faulted follow-ups merged too (945c2380 / 1d797c1c / 56f73e49). The six banners above carry each feature's mechanism + gotchas + deferred live gate; the closed open-follow-ups #1–#7 (Phase B, Phase C, data-plane role docs, write-pipeline nits, write-outcome, Galaxy driver nits, reconfigure-while-faulted) are all DONE at those SHAs.

The ONLY genuinely open items (all user-driven / deferred — pick up here):

1. User-driven live /run gates — the agent does NOT sign in; all code is merged + unit-verified; these are the operator's end-to-end confirmations:
   • Phase C HistoryRead (T7) — needs the Wonderware sidecar + AVEVA on the WW Historian VM 10.100.0.48 (NOT on the local docker-dev rig). Recipe: Phase C banner + docs/Historian.md.
   • Phase B native alarms (T9) — author a Galaxy alarm tag whose FullName == the gateway AlarmFullReference, deploy on MAIN-galaxy-eq, trip → Part 9 condition + /alerts row. Recipe: Phase B banner.
   • Driver-reconfigure-while-faulted — fault MAIN-opcua-eq with a bad config, deploy a corrected one (POST http://localhost:9200/api/deployments, X-Api-Key: docker-dev-deploy-key), confirm it adopts WITHOUT a node restart. Recipe: that banner.
2. Rig cleanups (operational, user-deferred) — see "Operational deferral" at the bottom.
3. Minor non-blocking residual follow-ups — MOSTLY CLOSED by the residual-followups-cleanup merge (fa2388ea; see the top banner for full reconciliation). DONE this pass: Phase B residuals (a)/(b)/(c) + the stub-driver de-dup. VERIFIED already-implemented (were stale in this list): all of write-pipeline (a)–(e), Galaxy _itemHandles/_supervisedHandles clear-on-reconnect, and the SubscriptionEstablished self-dead-letter handler. STILL OPEN (deferred, none gate anything): write-outcome residuals (Bad-quality blip / AuditWriteUpdateEvent / synchronous fail-fast — design-deferred OPC UA surface, NOT the Galaxy write-commit proof which is now live-tested d19deb9b); Phase C documented follow-ups (no continuation-point paging, no modified-value history, block-bridge timeout bounded only by the client's 30s CallTimeout); Phase B residual (d) (inbound device-ack etc., DEFERRED by design); the data-plane GroupToRole production-default note.

No queued feature remains — Milestone 1b + Galaxy A/B/C were the headline deliverables and are all done. Future directions (NOT requested): the Phase C HistoryRead follow-ups above, or new driver/UNS work.

---

The six historian code follow-ups (HistorizeToAveva opt-out, drain/capacity/retention config knobs, SharedSecret/DatabasePath/non-positive-knob startup validation, operator-recording for shelve/enable/disable, and the SqliteStoreAndForwardSink thread-safety nits) were all resolved on branch feat/alarm-historian-followups (plan: docs/plans/2026-06-11-alarm-historian-followups.md). They are no longer listed here.

Equipment-tag live values — MILESTONE 1b COMPLETE (2026-06-13)

The Galaxy standard-driver effort shipped Phase A (c3c56172) + the FullName→NodeId live-value ROUTER (c4435e4f, both pushed). The router is done + verified (322 tests + integration review READY-TO-MERGE). All three driver-publish gaps are now CLOSED — an equipment tag bound to OpcUaClient, any protocol driver, OR Galaxy publishes a live value delivered by the router (full detail in current.md "Milestone 1b" + docs/plans/2026-06-13-equipment-tag-live-values-design.md):

1. OpcUaClient has NO factory (real bug — always stubbed). DONE — SHIPPED+PUSHED master 22d553af 2026-06-13. Added OpcUaClientDriverFactoryExtensions (mirror Modbus) + registered it in DriverFactoryBootstrap. First live equipment-tag value PROVEN end-to-end: OpcUaClient driver MAIN-opcua-eq spawns stub=False, connects to opc-plc, subscribes to ns=3;s=FastUInt1; the FullName→NodeId router (c4435e4f) delivers it to the materialised variable ns=2;s=EQ-55297329838d/FastUInt1, which reads a live changing value (10135→10141, Good) via Client.CLI. Design/plan docs/plans/2026-06-13-opcuaclient-factory-*.md. Two incidental findings while live-verifying (see below).
2. Protocol drivers (Modbus/S7/AbCip/…) — equipment-tag↔driver tag-table linkage unbuilt. DONE — SHIPPED+PUSHED master 8d8c05f5 2026-06-13 (+ full inbound operator WRITE pipeline). Approach B (driver-side direct-ref): a shared EquipmentTagRefResolver<TDef> (Core.Abstractions) resolves an equipment-tag ref (the raw TagConfig JSON blob the router already keys on) into a transient driver tag-def on a _tagsByName miss — wired into READ + WRITE for all six drivers (Modbus/S7/AbCip/AbLegacy/TwinCAT/FOCAS), each with a hardened never-throw <Driver>EquipmentTagParser. Part B (write-through): writable nodes (Tag.AccessLevel==ReadWrite→CurrentReadWrite, byte-parity in Phase7Composer+DeploymentArtifact), an OnWriteValue gate on the WriteOperate data-plane role (mirrors the alarm-ack bridge; fire-and-forget dispatch since the SDK holds the node-manager Lock during OnWriteValue), a NodeWriteRouter on the node manager, and DriverHostActor.RouteNodeWrite (NodeId→driver reverse map, primary-gated). LIVE-PROVEN end-to-end: Modbus equipment tag (HR[100]) reads a live changing value; an authorized write (opc-writeop/WriteOperate) to HR[200] changes the register + persists; an anonymous write → BadUserAccessDenied. Design/plan docs/plans/2026-06-13-protocol-equipment-tag-linkage-*.md. Findings + rig artifacts below.
3. Galaxy — needs a reachable mxaccessgw. DONE — LIVE-PROVEN 2026-06-13 (no code change; config-only). The code-investigation confirmed Galaxy was already fully wired: GalaxyDriverFactoryExtensions IS registered in DriverFactoryBootstrap.cs:103 (not the missing-factory bug OpcUaClient had), and the Galaxy driver keys subscriptions on the FullReference (tag_name.AttributeName) DIRECTLY (no _tagsByName miss). gap (c) was purely a misconfigured dev driver-instance + placeholder tag ref + unset key — ALL data in existing columns, NO EF/schema change. Fixes applied to the dev rig (otopcua-dev-sql-1/OtOpcUa): MAIN-galaxy-eq DriverConfig gateway.endpoint https://10.100.0.35:5001→http://10.100.0.48:5120, useTls true→false, apiKeySecretRef env:MX_API_KEY(unset)→env:GALAXY_MXGW_API_KEY (the var the compose already wires on every node); GalaxyTestTag TagConfig.FullName TestMachine_002.SomeAttr(placeholder)→TestMachine_002.TestDuration (a real galaxy Float attr). The gateway API key was injected via ephemeral shell env at docker compose up -d --no-deps --force-recreate central-1 central-2 time (NEVER written to a tracked file; the compose's ${GALAXY_MXGW_API_KEY:-stale-default} substitution picks it up — the running containers carry the real key only until the next recreate-without-the-env-var). Live (central-1 logs): spawned GalaxyMxGateway driver MAIN-galaxy-eq (stub=False) → GalaxyMxSession connected — clientName=OtOpcUa (auth OK) → initialized — endpoint=http://10.100.0.48:5120 → subscribed to 1 refs (galaxy-sub-1) (TestMachine_002.TestDuration accepted, no BadNodeIdUnknown). Value: Client.CLI read ns=2;s=EQ-55297329838d/GalaxyTestTag → Value 0, Status 0x00000000 (Good), Source Time 2026-05-07T07:14:26Z (a real galaxy timestamp — a genuine attribute snapshot, NOT BadWaitingForInitialData; static because that attr isn't actively moving). Restore-the-rig SQL saved at /tmp/galaxy-gapc-snapshot.sql. Milestone 1b is now COMPLETE — all three gaps closed. Findings/follow-ups below.

Then: Phase B = native IAlarmSource alarms on the equipment-tag path (DONE — f9be3843); Phase C = server-side HistoryRead backend over the Wonderware reader (DONE — c24abc8a, 2026-06-14; design sections in docs/plans/2026-06-12-galaxy-standard-driver-design.md + the dedicated docs/plans/2026-06-14-galaxy-phase-c-historian-*.md).

Findings + follow-ups from the Galaxy gap-(c) live-verify (2026-06-13)

• Benign dead-letter (minor, pre-existing in the Galaxy driver — NOT introduced here). On subscribe the driver logs: Message [SubscriptionEstablished] from drv-MAIN-galaxy-eq to drv-MAIN-galaxy-eq was unhandled. [N] dead letters. The Galaxy DriverInstanceActor/driver sends itself a SubscriptionEstablished message that has no Receive<> handler. Harmless (the subscription IS established + delivering values), but noisy — add a handler (or stop self-Telling it). Cosmetic.
• CHANGING-value read PROVEN (2026-06-13). Repointed GalaxyTestTag.FullName→TestMachine_002.TestChangingInt (a script-driven Integer, sec=Operate): three Client.CLI reads returned 810 → 787 → 764 with real galaxy source timestamps advancing ~7s each (02:28:41/:48/:55) — a genuine live moving Galaxy value through the router (not optimistic/phantom). The dev rig is now left with GalaxyTestTag pointing here (DataType Int32, AccessLevel Read). Discovery was done with a throwaway probe (now deleted) using GalaxyDriverBrowser.OpenAsync→session.AttributesAsync("TestMachine_002"), which lists every attribute's SecurityClass (ViewOnly=read-only; FreeAccess/Operate/Tune/Configure=writable). Useful attrs on TestMachine_002: TestFloat(Float,Operate), TestDouble(Double,Operate), TestChangingInt(Integer,Operate,moves), TestDuration(ElapsedTime,Operate), AlarmInhibit(Boolean,FreeAccess).
• GALAXY WRITE-THROUGH — FIXED + MERGED to master f05b5d79 (AdviseSupervisory before raw Write). Symptom was every Galaxy operator write returning MxaccessFailure "ArgumentException: HRESULT 0x80070057" (E_INVALIDARG). TWO-LAYER root cause (debugged by SSH-reading the gateway source on 10.100.0.48 C:\Users\dohertj2\Desktop\mxaccessgw — there is NO live gateway file log: console-only/uncaptured, NSSM stdout.log stale, dashboard :5130 is Blazor/no-REST): (1) the writer AddItem'd an UN-advised handle → MXAccess Write threw E_INVALIDARG (worker chain ExecuteWrite→MxAccessSession.Write→MxAccessComServer.Write→AsProxyServer().Write(...)). (2) DEEPER — a plain Write runs with no user login (WriteUserId=0), and MXAccess only COMMITS such a write when the item is advised in SUPERVISORY mode; a regular Advise removed the E_INVALIDARG but never committed (proven by a persistence check: read-back showed the value, but a --force-recreate+fresh-resubscribe reverted to the original 0 @ 2026-05-07; the worker's ExecuteWrite is fire-and-forget, returns OK without awaiting OnWriteComplete). Confirmed against the sister ScadaBridge driver (~/Desktop/ScadaBridge/.../RealMxGatewayClient.cs): it commits the OTHER way — a configured non-zero WriteUserId + regular Advise + WriteBulk. We have no galaxy login → supervisory context. FIX: GatewayGalaxyDataWriter calls AdviseSupervisory (raw MxCommand{Kind=AdviseSupervisory, AdviseSupervisory=new AdviseSupervisoryCommand{ServerHandle,ItemHandle}} via session.InvokeAsync, mirroring InvokeWriteSecuredAsync; idempotent per handle via _supervisedHandles) before each raw WriteRawAsync; SecuredWrite/VerifiedWrite tags keep their own user-identity path (NeedsSecuredWrite unchanged — WriteSecured is ONLY for those special-security tags). The dead-end "reuse the subscription's advised handle" resolver attempt was reverted. LIVE-PROVEN: authorized write (opc-writeop/WriteOperate) of TestMachine_002.TestFloat=1234.5 then 8888.25 COMMITS + PERSISTS across recreate/re-subscribe (galaxy-sourced timestamp); anonymous → BadUserAccessDenied. 254 Galaxy tests green; central --build clean. OPEN follow-ups from this: (a) the worker's fire-and-forget ExecuteWrite can't surface an async write failure — with supervisory advise the write commits, but only a read-back confirms a specific write (gateway-side; out of our scope). (b) _itemHandles/_supervisedHandles caches aren't cleared on reconnect (pre-existing for _itemHandles) — a write right after a reconnect could use a stale handle; minor.
• OPTIMISTIC-WRITE PHANTOM (open follow-up — surface real write status to the client). The inbound write dispatch is fire-and-forget: it returns optimistic Good before the driver result (required — OnWriteValue runs under the node-manager Lock), and the SDK applies the written value to the node locally. So a write whose DEVICE write FAILS still returns Good, and for a STATIC attribute that never re-pushes, the wrong value LINGERS (a phantom the device never accepted). The pipeline already computes the real status in NodeWriteResult.Success/Reason but only LOGS it — consider surfacing it to the client. (How it was caught live: a failed Galaxy write showed the written value on read-back with a SERVER-clock source timestamp + a rejected driver log; a committed write shows a GALAXY-clock timestamp + no rejection, and persists across a re-subscribe.)
• Dev-rig Galaxy config is CORRECT + WORKING (left in place). The MAIN-galaxy-eq driver-instance is deployed and connecting to the live gateway http://10.100.0.48:5120. GalaxyTestTag (on EQ-55297329838d/filler-02, nw-uns namespace) is currently {"FullName":"TestMachine_002.TestFloat"}, DataType Float, AccessLevel ReadWrite (the write demo; galaxy now holds the last written value 8888.25). Other useful TestMachine_002 attrs (from the discovery probe): TestChangingInt(Integer,Operate,moves on its own — the live-changing READ demo), TestDouble(Double,Operate), TestDuration(ElapsedTime,Operate — reads as Float but a Float write is a type-mismatch), AlarmInhibit(Boolean,FreeAccess). To restore the original placeholder tag: /tmp/galaxy-gapc-snapshot.sql. The base seed docker-dev/seed/seed-clusters.sql still seeds the legacy SystemPlatform-namespace Galaxy driver (MAIN-galaxy-mxgw, tags TestMachine_001.TestAlarm001..003) — pre-Phase-A model, untouched/separate. The injected gateway key is EPHEMERAL — key=mxgw_otopcuakey2_so0… is supplied via shell env GALAXY_MXGW_API_KEY='…' at docker compose up --no-deps --force-recreate central-1 central-2; a recreate WITHOUT it re-exported falls back to the compose's stale default and Galaxy auth fails. ORDER on a redeploy: POST deploy FIRST, THEN recreate (a faulted driver ignores ApplyDelta).

Findings + follow-ups from the protocol-linkage + write-through work (2026-06-13)

• DATA-PLANE ROLE CONFIG REQUIREMENT (important, deployment-facing). The OPC UA session's roles come from two sources unioned: the DB LdapGroupRoleMapping (its Role column is the AdminRole enum — Administrator/Designer/Viewer only, for the AdminUI) AND the appsettings Security:Ldap:GroupToRole baseline (free-form string→string). The OPC UA data-plane gates (WriteOperate, AlarmAck, …) read literal role STRINGS that the AdminRole-typed DB mapping cannot produce — so a deployment MUST map its LDAP data-plane groups → data-plane role strings via GroupToRole, or write-through (and scripted-alarm OPC UA ack) is inert (every write → BadUserAccessDenied). The shared dev GLAuth already has dedicated groups+users (group WriteOperate, user opc-writeop, multi-role in all; opc-readonly); the dev rig just never seeded the GroupToRole. Consider a docs note (and/or a documented default) so production deployments wire this. (Same latent requirement applies to the pre-existing alarm-ack gate.)
• Write-pipeline review follow-ups (non-blocking, from the final integration review): (a) DriverHostActor.Stale (and DriverInstanceActor.Connecting/Reconnecting) have no RouteNodeWrite/WriteAttribute handler → an operator write while stale/reconnecting dead-letters and the 10s Ask times out with a generic log (client got optimistic Good). Add fast-fail handlers returning a clear status. (b) Drop TaskContinuationOptions.ExecuteSynchronously on the router ContinueWith; List.Contains→HashSet in the forward-map build (micro). (c) FOCAS re-parses the address on every equipment-tag write (_parsedAddressesByTagName miss; perf only, rare). (d) DriverHostActorWriteRoutingTests seeds a Galaxy-style {"FullName":...} artifact, not a raw protocol-driver TagConfig blob — add a raw-blob case for belt-and-suspenders (runtime path is identical + live-verified). (e) Task-9 parity test is a faithful simulation of ConfigComposer (ToSnapshot casts AccessLevel to int) not a through-the-real-serializer proof; add an InlineData(2,false) future-enum trap.

Dev-rig artifacts created for the protocol-linkage live-verify (left in place, NOT committed)

• docker-dev/docker-compose.yml gained Security__Ldap__GroupToRole__{ReadOnly,WriteOperate,WriteTune,WriteConfigure,AlarmAck} identity entries on both central nodes (needed for data-plane roles — see above). Uncommitted (rig config; the file was already modified at session start).
• DB seeds on otopcua-dev-sql-1/OtOpcUa: driver MAIN-modbus-eq (DriverType=Modbus, {"Host":"10.100.0.35","Port":5020,"UnitId":1,"Tags":[]}, namespace nw-uns, cluster MAIN) + tags tag-modbus-hr100 (HR[100] auto-increment, Read — read demo) and tag-modbus-hr200 (HR[200] scratch, ReadWrite — write demo), both on equipment EQ-55297329838d (filler-02). The pymodbus standard sim (10.100.0.35:5020) serves HR[0..31]=addr-as-value, HR[100]=auto-increment, HR[200..209]=writable scratch.

Incidental findings from the OpcUaClient live-verify (2026-06-13)

• Driver-reconfigure-while-faulted gap — FIXED + MERGED 56f73e49 (2026-06-14). (Was: a DriverInstanceActor stuck in Reconnecting/Connecting had no ApplyDelta handler, so a corrected config dead-lettered and the actor retried the OLD _currentConfigJson forever; workaround = restart the node.) Now Connecting/Reconnecting handle ApplyDelta via AdoptConfigDuringInit, re-initialising with the new config; a monotonic _initGeneration guard supersedes the in-flight old init so the corrected config always wins. See the top banner + docs/plans/2026-06-14-driver-reconfigure-while-faulted-*.md. Live /run gate deferred (user choice).
• Dev-rig config edit applied directly in DB. The MAIN-opcua-eq DriverConfig.targetNamespaceKind was 0 (Equipment, which requires a UnsMappingTable → InitializeAsync rejected it). Set to 1 (SystemPlatform — the direct-ref mode the equipment-tag model wants; empty unsMappingTable:{} passes validation) via a direct JSON_MODIFY UPDATE on otopcua-dev-sql-1 (DB OtOpcUa, SET QUOTED_IDENTIFIER ON required for JSON fns; sqlcmd -h and -y 0 are mutually exclusive — pick one). The AdminUI driver-edit combobox for "Target namespace kind" did not persist the change (suspected live-only Blazor binding bug — unverified; the DB edit sidestepped it). Deploy snapshots the live config DB directly (AdminOperationsActor → DraftSnapshotFactory.FromConfigDbAsync + ConfigComposer.SnapshotAndFlattenAsync), so a DB edit flows through on the next POST /api/deployments (new revisionHash).

Operational deferral (user choice)

1. docker-dev rig cleanup (round-1 T9) deferred. The local docker-dev rig still has the live-verify seed artifacts deployed: the t12-overheat scripted alarm, the SC-ba675b168a85 predicate script, the layer0-logcheck vtag/script, and filler-02's modified cycle-time-s line. Left as-is to inspect the working double-emit fix. To clean up: delete those artifacts in the AdminUI (or DB), revert filler-02's cycle-time-s to return ctx.GetTag("TestMachine_002.TestDuration").Value;, then redeploy (POST http://localhost:9200/api/deployments, header X-Api-Key: docker-dev-deploy-key).

2. Equipment-tag live-value verify artifacts (left in place — all now FUNCTIONAL). The docker-dev rig carries verify artifacts under the nw-uns Equipment namespace on EQ-55297329838d (filler-02), all three now working: MAIN-galaxy-eq (GalaxyMxGateway → live gateway, GalaxyTestTag = TestMachine_002.TestFloat RW, write-proven; needs the ephemeral GALAXY_MXGW_API_KEY re-exported on recreate — see Galaxy dev-rig note above); MAIN-opcua-eq (OpcUaClient, factory shipped 22d553af)

   • FastUInt1 tag ({"FullName":"ns=3;s=FastUInt1"}); MAIN-modbus-eq (Modbus) + tag-modbus-hr100 (read) / tag-modbus-hr200 (RW write demo). docker-dev is LOCAL on this Mac (OrbStack); central-1 @ localhost:4840/AdminUI+deploy @ localhost:9200, sql @ localhost:14330 (sa/OtOpcUa!Dev123), login disabled. Sims for the protocol/opcua verifies run on the docker host 10.100.0.35 (otopcua-pymodbus-standard :5020, otopcua-opc-plc :50000) — leave up or docker compose down per /opt/otopcua-modbus + /opt/otopcua-opcuaclient. Phase A backup OtOpcUa-prePhaseA-20260612-224908.bak is in the SQL volume /var/opt/mssql/backup/.