chore(r2-12): untrack/archive root planning files; honor pending.md's never-stage rule (07/U-4)
- pending.md, current.md: git rm --cached (untrack, keep on disk) + root-anchored .gitignore entries — making the files' own declared 'never stage' rule true. - looseends.md: deleted (2026-05-18 snapshot; sole open item + all follow-ups target the retired Wonderware historian sidecar rig, moot since the gateway cutover; recoverable from history). - stillpending.md -> docs/plans/2026-06-18-stillpending-backlog-audit.md (git mv; referenced by docs/Historian.md + memory index, preserved under dated-plan convention). - HISTORIAN-GATEWAY-INTEGRATION-ISSUES.md -> docs/plans/2026-06-27-historian-gateway-integration-issues.md (git mv). - docs/Historian.md §'Closed backlog §2 items' repointed at the archived path. Historical docs/plans/* references to the bare stillpending.md filename are frozen point-in-time records, left untouched per the documentation-audit convention.
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,269 @@
|
||||
# Still Pending — RECONCILED 2026-06-18
|
||||
|
||||
> **★ UPDATE 2026-06-19 (master `a325ec54`) — NON-ARCH FOLLOW-UPS BATCH (A/B) SHIPPED.** Six code
|
||||
> tasks merged via `feat/followups-batch`: OpcUaClient stale-session race (re-resolve session inside
|
||||
> `_gate` in 5 history/subscribe/ack paths); Client.CLI `enable`/`disable` condition commands (H4
|
||||
> client path); cert-audit review nits; Galaxy re-pick preserves edited alarm/historize/scale via
|
||||
> pure `TagConfigJson.SetFullName`; inline-script dropdown label reorder fix. **The "write-outcome
|
||||
> residuals" item (Bad blip + AuditWriteUpdateEvent + sync fail-fast) was found ALREADY-SHIPPED in
|
||||
> `bb59fd4e` (2026-06-15) — delivered the missing behavioural test coverage instead.** Live-`/run`:
|
||||
> T5 PASS; T4 substantial (picker + live gateway browse confirmed; final pick-and-assert blocked by
|
||||
> the Galaxy object-browser `href=#` nav quirk under automation — preservation unit-tested).
|
||||
> **RECONSIDER GATES still OPEN-by-choice (NOT built): F10b surgical DataType/IsArray writes;
|
||||
> alarm-severity `SetSeverity`.** See `docs/plans/2026-06-19-followups-batch*` + memory.
|
||||
>
|
||||
> **Reconciled 2026-06-18** against master HEAD `651018f9` by a 7-way parallel per-subsystem
|
||||
> code audit of the original 2026-06-15 backlog (which was generated at the older HEAD `151b7165`).
|
||||
> Three days and a large run of "Phase 4 / H1–H6 / driver" commits landed in between, so **most of
|
||||
> the original backlog is now SHIPPED.** Every item below was re-verified in code (commit cited).
|
||||
>
|
||||
> **Legend** — `OPEN` (genuine, actionable code gap) · `DEFERRED` (intentional scope boundary —
|
||||
> needs infra, a future version, an SDK/gateway/sidecar change, or a by-design decision) ·
|
||||
> `SHIPPED` (the cited gap no longer exists; recorded for the audit trail so it is not re-flagged).
|
||||
|
||||
---
|
||||
|
||||
## A. Genuinely OPEN — actionable code work (ranked)
|
||||
|
||||
1. **H2 — OPC UA `HistoryUpdate` service** — **RECLASSIFIED INFRA-GATED / DEFERRED (user decision 2026-06-19).**
|
||||
The server-side override (mirror the 5 HistoryRead overrides: `HistoryUpdateData`/`HistoryUpdateEvents`/
|
||||
`HistoryDeleteRawModified`/`HistoryDeleteAtTime`/`HistoryDeleteEvent`, resolve nodes via the same
|
||||
`TryGetHistorizedTagname`/event-notifier maps, gate on the already-shipped `NodePermissions.HistoryUpdate =
|
||||
1<<12` / commit `c236263e`) is buildable + offline-testable, BUT **no backend can durably write history**:
|
||||
`IHistorianDataSource` is read-only (Raw/Processed/AtTime/Events + health — no insert/replace/delete) and
|
||||
the AVEVA Wonderware TCP sidecar exposes no history-update RPC (the same permanent SDK/sidecar boundary as
|
||||
H5b durable-write + no-modified-reads). So any `IHistoryUpdateSink` wired today could only reject — a custom
|
||||
override would mostly restate the SDK's own default `HistoryUpdate` rejection. **Deferred** until a writable
|
||||
historian backend exists; if/when one does, build the full service surface + the `IHistoryUpdateSink` seam
|
||||
(Phase-C shape) at that time. _Moved out of §A "Genuinely OPEN" — see the Infra-gated OPEN note below + §B._
|
||||
|
||||
2. ~~**OpcUaClient — `IHistoryProvider.ReadEventsAsync`** event-history passthrough~~ — **SHIPPED master
|
||||
`7a3d2712` 2026-06-18** (`feat/opcuaclient-read-events`). NO interface change (the "param extension" framing
|
||||
was overcautious): the driver builds its OWN canonical 6-field BaseEventType `EventFilter` and maps the
|
||||
upstream `HistoryEvent`→`HistoricalEvent` (the server re-projects only those 6 fields). Driver-internal;
|
||||
live-proven vs opc-plc (wire path; non-empty events infra-gated — opc-plc isn't a historian).
|
||||
|
||||
3. ~~**FOCAS — `cnc_getfigure` wire command**~~ **SHIPPED master (branch
|
||||
`feat/focas-figure-ww-poison-abcip-gate`) 2026-06-18.** `FocasWireClient.ReadPositionFiguresAsync`
|
||||
(wire command `0x00D3`, self-delimiting per-axis `short dec` payload) + the `WireFocasClient`
|
||||
stub wired to it; matching `focas_mock` `_wire_payload` handler + `position_figures` default-0 state.
|
||||
The managed wire backend (production path to real CNCs) now auto-fetches per-axis figures; the
|
||||
`PositionDecimalPlaces` knob is the graceful per-axis fallback (`FocasDriver.SafeProbe` → empty on
|
||||
failure → knob, so it cannot regress). **LIVE-PROVEN** against the local focas-mock
|
||||
(`Position_figures_scale_axis_via_wire_backend`: 12345 ÷ 10^3 = 12.345, auto figure beat the knob).
|
||||
Real-Fanuc wire-fidelity stays bench-CNC-gated (whole-backend caveat, not unique to this command).
|
||||
|
||||
4. ~~**Galaxy — writer item-handle cache not shared with the subscription registry**~~
|
||||
**SHIPPED master `8480e301` 2026-06-18.** `SubscriptionRegistry.TryResolveItemHandle` (authoritative
|
||||
forward lookup) + writer `subscribedHandleSource` delegate → first write to an already-subscribed tag
|
||||
borrows the live handle, skipping the redundant `AddItem`. Live-proven vs `10.100.0.48:5120` (3/3).
|
||||
|
||||
5. ~~**Historian.Wonderware — poison-event sidecar wire protocol**~~ **SHIPPED master (branch
|
||||
`feat/focas-figure-ww-poison-abcip-gate`) 2026-06-18.** Additive `[Key(4)] byte[] PerEventStatus`
|
||||
(0=Ack/1=Retry/2=Permanent) on `WriteAlarmEventsReply` (BOTH in-repo IPC ends — MessagePack/TCP, NOT a
|
||||
Commons proto; `PerEventOk [Key(3)]` kept for rolling-deploy back-compat). The sidecar
|
||||
`HistorianFrameHandler` now classifies **structurally-malformed** events (empty SourceName/AlarmType,
|
||||
`EventTimeUtcTicks ≤ 0`) as **Permanent** and excludes them from the writer batch; the client
|
||||
`WriteBatchAsync` prefers `PerEventStatus` (→ `PermanentFail` → the sink's existing immediate
|
||||
dead-letter) and falls back to `PerEventOk`. **NO `IAlarmEventWriter` / Commons / Core.Abstractions
|
||||
change** (`HistorianWriteOutcome.PermanentFail` already existed). Offline-proven (sidecar classifier
|
||||
tests + client round-trip + sink). _Documented boundary (follow-up):_ an SDK-**semantic** permanent
|
||||
rejection of a structurally-valid event still maps to Retry→cap until the infra-gated
|
||||
`AahClientManagedAlarmEventWriter` surfaces richer per-event status.
|
||||
|
||||
6. ~~**AbCip — nested-struct member expansion in production**~~ **SHIPPED master (branch
|
||||
`feat/hosts-rows-abcip-nested-hygiene`) 2026-06-18** (`3d8ce4e8`+`d203f31c`). FINDING: simpler than this
|
||||
item assumed — the CIP Template Object member-info `u16` ALREADY carries the nested template id for a
|
||||
struct member (bit 15 = struct flag, lower 12 bits = template instance id, same encoding as the Symbol
|
||||
Object; the decoder's own remarks documented it). The prior fix DISCARDED it (`templateInstanceId: null`).
|
||||
The close: carry `uint? NestedTemplateId` on `AbCipUdtMember` (decoded via the FULL 12-bit mask, not the
|
||||
byte-cast typeCode), thread it into the existing `FetchUdtShapeAsync`/`@udt/{id}` path — NO new CIP query.
|
||||
Offline-proven (decoder + no-seam threading test, AbCip 301/301). **Live-gate FORMALIZED 2026-06-18
|
||||
(branch `feat/focas-figure-ww-poison-abcip-gate`):** a skip-gated Emulate-tier integration smoke
|
||||
`AbCipEmulateNestedUdtTests` (mirrors `AbCipEmulateUdtReadTests` + `AbServerProfileGate.SkipUnless(Emulate)`)
|
||||
asserts nested leaves addressable; `docs/drivers/AbCip.md` documents nested-struct as Emulate-tier
|
||||
verified. A LOCAL gate is architecturally impossible — `ab_server` lacks CIP Template Object (class 0x6C);
|
||||
the test compiles + skips locally and runs only against Logix Emulate / real ControlLogix.
|
||||
|
||||
7. ~~**AdminUI — Certificates page actions**~~ **SHIPPED master `bd791e79` 2026-06-18.** FleetAdmin-gated
|
||||
Trust (rejected→trusted) / Untrust (trusted→rejected) / Delete via a new in-process `CertificateStoreManager`
|
||||
(filesystem moves by thumbprint, honored live); inline Blazor confirm + server-side FleetAdmin re-check.
|
||||
Live-proven on docker-dev. **CERT-ACTION AUDIT LOGGING SHIPPED master `3eb370d4` (2026-06-19):**
|
||||
Trust/Untrust/Delete now write a `ConfigAuditLog` row (`Certificate:Trust|Untrust|Delete`, Outcome
|
||||
Success/Failure on every path incl. failures) via the first live structured `AuditEvent` emit site —
|
||||
new `ActorAuditWriter : IAuditWriter` (AdminUI adapter Telling the `AuditWriterActor` cluster singleton
|
||||
resolved from `ActorRegistry`, mirrors `AdminOperationsClient`) + pure `CertAuditEvents.Build` factory,
|
||||
audited at the `CertificateStoreManager` choke point (`DeleteCore` refactor); actor sourced from
|
||||
`AuthenticationState`. NO EF migration (table + actor pre-existed), NO Commons/proto change.
|
||||
LIVE-PROVEN end-to-end on docker-dev: a Trust on the Certificates page wrote
|
||||
`multi-role-test | Certificate:Trust | Success | <thumbprint>` to `ConfigAuditLog`. (Was: only cert
|
||||
*validation* went to Serilog.)
|
||||
|
||||
8. ~~**AdminUI — Hosts page per-driver-instance rows**~~ **SHIPPED master (branch
|
||||
`feat/hosts-rows-abcip-nested-hygiene`) 2026-06-18** (`cb062fce`+`6457eba8`+`07bd63f8`). A new
|
||||
cluster-grouped "Driver Instances" section on `/hosts` reusing the existing driver-health DPS pipeline
|
||||
(added only an AdminUI-internal `IDriverStatusSnapshotStore.GetAll()` + a pure `HostsDriverView.Build`
|
||||
grouping view-model + the Razor section) — NO Commons/interface/EF change. Grouped by `ClusterId` (the
|
||||
reliable key; the health feed carries no per-Akka-member identity), nodes from ConfigDB `ClusterNode`,
|
||||
driver rows enriched with Name/DriverType from `DriverInstance`, live status chips via the in-process
|
||||
store's `SnapshotChanged` (dodges the Traefik self-hub trap) + the 5 s timer. LIVE `/run` PASS (both
|
||||
central nodes rebuilt; MAIN cluster shows Galaxy/Modbus/OpcUaClient HEALTHY enriched + grouped; SITE-A/B
|
||||
show nodes + "No drivers"). DEFERRED follow-up: TRUE per-Akka-member nesting (needs a hosting-node field
|
||||
on `DriverHealthChanged` — a Commons change — or a per-node `DriverHostActor`-children Ask).
|
||||
|
||||
9. ~~**AdminUI driver-pages plan — Phase 6 + Phase 10**~~ **SHIPPED — RECONCILED 2026-06-18.** A
|
||||
seam-by-seam audit confirmed the WHOLE driver-pages plan (Phases 0–10) is shipped; §A.9 was stale
|
||||
(it over-counted from the stale `.tasks.json`). Phase 6 (live `DriverStatusPanel`) + Phase 8
|
||||
(Reconnect/Restart) are wired end-to-end (DriverHealthChanged → AkkaDriverHealthPublisher invoked at
|
||||
`DriverInstanceActor.PublishHealthSnapshot` → admin-gated `DriverStatusSignalRBridge` → snapshot store
|
||||
→ panel in all 9 driver pages; page `DriverInstanceId` ⟷ actor `_driverInstanceId` keys match). Phase
|
||||
10 E2E tests (DriverTestConnect/DriverReconnect/DriverStatusHub) shipped at `494da22c`. The one genuine
|
||||
remnant — a **deployed-driver Healthy→Reconnecting→Healthy E2E** + the harness-fidelity fix
|
||||
(`AddOtOpcUaRuntime` so harness drivers actually publish health) + a controllable fake driver factory —
|
||||
SHIPPED on `feat/driver-pages-reconnect-e2e` (`docs/plans/2026-06-18-driver-pages-reconnect-e2e.md`),
|
||||
driver E2E suite GREEN 18/0/4. The `.tasks.json` is now reconciled. DEFERRED follow-ups: full-stack
|
||||
WebSocket+JWT DriverStatusHub test (no repo precedent; mock-hub test + manual runbook cover it); the
|
||||
§8.3 manual browser smoke recording (10.4); the dotnet-test-testhost-can't-reach-remote-docker skip
|
||||
quirk on this dev Mac.
|
||||
|
||||
10. ~~**AdminUI — Monaco `ctx.SetVirtualTag(...)` completions**~~ **SHIPPED master `70e1bde9` 2026-06-18**
|
||||
(predates this audit — was stale here). The `ctx`-receiver guard is in
|
||||
`ScriptAnalysisService.TryGetTagPathLiteral` (`:224`, mirrors the `ctx`-anchored
|
||||
`EquipmentScriptPaths.GetTagRefRegex`) + a truthful `ctx.SetVirtualTag` hover note (it's a no-op in the
|
||||
live single-tag evaluator).
|
||||
|
||||
11. ~~**F10b — in-place property writes vs full rebuild**~~ **FIRST SLICE SHIPPED master (branch
|
||||
`feat/f10b-vtag-skip-rebuild-galaxy-comment`) 2026-06-18** (`6c6a2c42`+`b05c2811`). The provably-safe,
|
||||
zero-node-mutation slice: `Phase7Applier.Apply` now SKIPS `RebuildAddressSpace` when a deploy's only
|
||||
changes are VirtualTag deltas differing *solely* in `Expression`/`DependencyRefs`/`Historize` — a vtag's
|
||||
materialised node depends ONLY on `{EquipmentId,FolderPath,Name,DataType}` (Historize is write-side
|
||||
only), and the vtag ENGINE adopts the edit via `VirtualTagHostActor`'s INDEPENDENT respawn
|
||||
(`DriverHostActor`→`ApplyVirtualTags`), so editing a script no longer tears down + rebuilds the whole
|
||||
server address space (preserving every client's subscriptions + live values). Whitelist-of-may-differ
|
||||
(`(prev with {those 3 = cur's}).Equals(cur)`) defaults to REBUILD for any other/structural change;
|
||||
`EnsureVariable`'s existing-node early-return preserves the live node when no rebuild runs. RecordingSink
|
||||
unit tests pin the boundary (skip cases + DataType/Name/FolderPath/EquipmentId/mixed ⇒ rebuild).
|
||||
Code-review verified all 6 safety claims; no H1a stale-node regression.
|
||||
**SECOND SLICE (surgical tag-attribute writes) SHIPPED master (branch `feat/f10b-surgical-tag-attribute-writes`)
|
||||
2026-06-18** (`3fc258bd`+`f6618144`+`b472bba3`+fix `22b0611f`): equipment-tag **`Writable`** +
|
||||
**`IsHistorized`/`HistorianTagname`** toggles now update the EXISTING node IN PLACE (AccessLevel + write
|
||||
handler + Historizing + historian-map + `ClearChangeMasks`) via a new optional `ISurgicalAddressSpaceSink`
|
||||
(`OtOpcUaNodeManager.UpdateTagAttributes`, mirrors `EnsureVariable` via a shared `ComposeAccessLevel`),
|
||||
instead of a full rebuild — `Phase7Applier.TagDeltaIsSurgicalEligible` (no-alarm value variable, only those
|
||||
3 differ) with rebuild fallback for anything else / missing node / non-surgical sink. **LIVE-PROVEN** on
|
||||
docker-dev (toggled FastUInt1 isHistorized → `changed=1, surgicalTags=1, rebuild=False` on BOTH central
|
||||
nodes, server healthy). **CRITICAL bug the live-run caught (all unit tests + 3 reviews missed it): the
|
||||
optimization was INERT in prod** — actors inject `DeferredAddressSpaceSink` (the runtime swap-in wrapper),
|
||||
which forwarded `IOpcUaAddressSpaceSink` but NOT `ISurgicalAddressSpaceSink`, so `_sink is ISurgical…` was
|
||||
false → silent rebuild fallback (`rebuild=True, surgicalTags=0`). Fixed by forwarding the surgical
|
||||
capability through the wrapper (`22b0611f`) + a regression test. _Still DEFERRED (dirty for rare edits /
|
||||
no value): surgical `DataType`/`IsArray` writes (brief value-type mismatch; no ModelChangeEvents) +
|
||||
alarm-severity `SetSeverity` (live-shadowed by the alarm engine, operationally invisible)._
|
||||
|
||||
12. ~~**Client.UI — Galaxy sub-attribute fallback collapses operator comments**~~ **CLOSED as a documented +
|
||||
tested ACCEPTED-LIMITATION master (branch `feat/f10b-vtag-skip-rebuild-galaxy-comment`) 2026-06-18**
|
||||
(`fac44187`). Investigation confirmed it is NOT code-fixable: Galaxy's `AckMsg` attribute is WRITE-ONLY
|
||||
(no readable attribute to recover the last comment) and the OPC UA event SelectClause carries no comment
|
||||
field — the operator comment is only available via the gateway's NATIVE alarm feed
|
||||
(`GalaxyAlarmTransition.OperatorComment`, the Galaxy driver path), not this OPC UA client fallback. Closed
|
||||
with a clarifying inline comment at the fallback `AlarmEvent?.Invoke` site + a tightened
|
||||
`AlarmEventArgs.OperatorComment` doc naming both concrete reasons + a pinning test
|
||||
(`OnAlarmEvent_GalaxyFallback_LeavesOperatorCommentNull`) so the null behavior is intentional + regression-guarded.
|
||||
A behavioral fix would need a server-side change (out of scope).
|
||||
|
||||
13. ~~**§9 hygiene — Galaxy stale forward-ref comments**~~ **SHIPPED master (branch
|
||||
`feat/hosts-rows-abcip-nested-hygiene`) 2026-06-18** (`9cf88e33`). All 5 sites
|
||||
(`GalaxyDriver.cs:52,92,669`, `GalaxyDriverFactoryExtensions.cs:19`, `Health/HostStatusAggregator.cs:21`)
|
||||
rewritten to the shipped standard-driver reality (each claim re-verified against the live code:
|
||||
`_dataReader` reads via `ReadViaSubscribeOnceAsync`, `_ownedMxSession` built in
|
||||
`BuildProductionRuntimeAsync`, `SyncPlatformsAsync` refreshes the probe watcher post-discovery, only
|
||||
`GalaxyMxGateway` exists — no `Galaxy:Backend` switch, host-status re-raise wired in the ctor). The
|
||||
unshipped "PR 4.W"/`Galaxy:Backend`/"legacy-host" forward-refs are gone. Comment-only; §9 now fully clean.
|
||||
|
||||
> **Infra-gated OPEN (code path exists or is buildable; only a no-op until a deployment supplies the backend):**
|
||||
> **H5b — production `IHistoryWriter` for equipment-vtag historization.** The `Historize` flag is now
|
||||
> threaded end-to-end (commits `fc8121cb`/`9c5a0913`/`4501f126`) but DI defaults to `NullHistoryWriter` —
|
||||
> no durable AVEVA sink is written/wired. Same infra-gated pattern as the alarm historian: writing a real
|
||||
> sink is possible but only exercisable against the historian backend.
|
||||
> **H2 — OPC UA `HistoryUpdate` service** (reclassified 2026-06-19, see §A.1). Server-side override is
|
||||
> buildable but no historian backend can durably insert/replace/delete — `IHistorianDataSource` is read-only
|
||||
> and the Wonderware sidecar has no update RPC; build the service + `IHistoryUpdateSink` seam when a writable
|
||||
> backend exists.
|
||||
|
||||
---
|
||||
|
||||
## B. DEFERRED — intentional scope boundaries (not actionable without infra / future version)
|
||||
|
||||
**Driver write/array boundaries (reads shipped):**
|
||||
- Array **writes** across all 5 drivers (OPC UA layer forces array nodes read-only — `Phase7Applier`
|
||||
`writable = tag.Writable && !tag.IsArray`, commit `3bb2031d`).
|
||||
- S7 wide-type **arrays** (Int64/UInt64/LReal/String/DateTime as arrays); S7 **Timer/Counter writes**
|
||||
(read shipped); Modbus String/BitInRegister **array writes**.
|
||||
- AbCip **whole-UDT bare-container** read (`BadNotSupported`) + write (throws) — operators address
|
||||
individual member paths. `Structure→DriverDataType.String` placeholder retained.
|
||||
- TwinCAT **pre-declared `Structure` tags** rejected at parse (discovery is the supported path);
|
||||
**Flat-mode `SubSymbols` live caveat** — on some TC3 firmware Flat mode may not populate struct
|
||||
SubSymbols → switch `SymbolsLoadMode.Flat`→`VirtualTree` (deliberately NOT changed; needs a real ADS
|
||||
target to verify; would risk regressing the Phase 4c array discovery).
|
||||
- AbLegacy I/O-image bit writes are NOT pre-rejected — they RMW and surface the real PCCC device status.
|
||||
|
||||
**Gateway-scope (mxaccessgw sibling repo, not this repo):**
|
||||
- Galaxy `ExecuteWrite` fire-and-forget — the .NET writer fully propagates `WriteResult` status; the
|
||||
invisible-failure concern is gateway-side.
|
||||
|
||||
**Reserved seams / future versions (no behavioral gap today):**
|
||||
- `AuthorizationVerdict.Denied` (v2.1 explicit-deny), `NamespaceKind.Simulated` (replay driver),
|
||||
`Script.Language` (multi-engine), `IDriverConfigEditor` (superseded by `TagConfigEditorMap`),
|
||||
`DraftSnapshotFactory` placeholders (conservative fallbacks), Cluster `LeaderChanged` no-op
|
||||
(ServiceLevel lives in `RedundancyStateActor`).
|
||||
- AdminUI Monaco **InlayHints** endpoint — intentional no-op (`[]`).
|
||||
- Historian **no modified-value history** (sidecar has no modified-read RPC); **Processed/AtTime
|
||||
single-shot** (spec-conformant, no count cap).
|
||||
- Redundancy **`ServerUriArray`** child of `Server.ServerRedundancy` — pending an SDK object-type upgrade
|
||||
(read `Server.ServerArray` today).
|
||||
- VirtualTags **serialized eval** under `_evalGate` (decision #19); **unbounded script memory / post-timeout
|
||||
thread leak** — out-of-process runner is a v3 concern.
|
||||
|
||||
---
|
||||
|
||||
## C. Live `/run` verification gates — code done, user/operator-driven
|
||||
- Galaxy Phase C historian T7 live HistoryRead (needs Wonderware sidecar + AVEVA on `10.100.0.48`).
|
||||
- Galaxy Phase B native alarms T9 live `/run` (equivalent proof already achieved via `dcb0be65` + `/alerts`).
|
||||
- Driver-reconfigure-while-faulted T3 live `/run`.
|
||||
- AVEVA Historian rig smokes (skip-gated `rig-required` tests).
|
||||
|
||||
---
|
||||
|
||||
## D. SHIPPED since 2026-06-15 (audit trail — do NOT re-flag)
|
||||
|
||||
**§1:** H1a changed-only-deploy rebuild (`1e95856b`) · H1b vtag-respawn (`ada01e1a`) · H2 permission bit
|
||||
(`c236263e`) · H3 health-aware ServiceLevel now live (`3e609a2b`/`70e6d3d2`) · H4 OPC UA Enable/Disable
|
||||
(`328bd1b9`) · H5a Historize flag threaded (`fc8121cb`/`9c5a0913`) · H6 native-ack→AVEVA + authenticated
|
||||
identity (`418663b3`/`ed941c51`/…).
|
||||
|
||||
**§2 drivers:** S7 wide scalars Int64/UInt64/LReal/String/DateTime r+w (`06b858eb`/`286be5df`/`5db08e9e`) ·
|
||||
S7 Timer/Counter read (`8cfb8e92`) · Modbus Int64/UInt64 node type (`DriverDataType.Int64/UInt64`) ·
|
||||
Wonderware `Total` aggregate (client-side `ScaleAverageToTotal`) · cross-driver 1-D array **reads** all 5
|
||||
drivers (Phase 4c `a82c22c6`/`f4d5a5ee`/`95006939`/`3e742395`/`8d3dc321`) · Modbus String/Bit array read ·
|
||||
AbCip controller-discovered UDT **member paths** (`4e141402`/`4a7b0fde`) · TwinCAT discovered struct
|
||||
member expansion (`3699fc16`) · AbLegacy B/I/O bit writes via RMW (`5c6b7cd6`) · TwinCAT BOOL-within-word
|
||||
RMW (`a73e20fb`) · FOCAS axis scaling logic + knob (`49730752`/`6855be28`) · FOCAS fail-fast on
|
||||
unimplemented backend (`5c5aaef6`) · Galaxy parent-gobject nesting (`21c7645d`/`bec37848`) · all 4 (+more)
|
||||
Test-Connect probes upgraded TCP→protocol handshake (`961b2b55` & per-driver).
|
||||
|
||||
**§4/§5 AdminUI:** typed TagConfig editors for OpcUaClient + Historian.Wonderware (`72d414ad`) ·
|
||||
isHistorized/historianTagname first-class UI (`c00a5471`) · UNS-tree delete Enterprise+Cluster
|
||||
(`526eebb3`) · create-new-script from inline vtag panel (`c98625fd`) · native-alarm `HistorizeToAveva`
|
||||
opt-out + UI knob (`6a8020e7`) · Galaxy picker pre-fills native-alarm fields (`069a5f31`) · Client.UI
|
||||
alarm Acknowledge + Shelve + Confirm surfaces (`4f55d894`/`60da9224`/`a774bb08`/`6a61f35a`).
|
||||
|
||||
**§6/§7:** Historian #400 within-timestamp tie-cluster paging (`2e6c6d3a`/`0f929ae6`) · per-cluster-scoping
|
||||
plan COMPLETE (`.tasks.json` reconciled) · historian-tcp-transport plan COMPLETE (stale json) ·
|
||||
adminui-followups plan COMPLETE — role-mapper (RoleGrants.razor + `LdapGroupRoleMappingService` +
|
||||
`FleetAdmin` + `LdapGroupRoleMapping` entity/migration) + typed editors all shipped · adminui-driver-pages
|
||||
Phases 7 (probes) + 9 (9 typed address pickers) shipped · Wonderware poison-event SQLite dead-letter half
|
||||
(`fcb38014`).
|
||||
|
||||
**§9 stale comments corrected:** DriverHostActor, OpcUaPublishActor, ServiceCollectionExtensions,
|
||||
IScriptLogPublisher/ScriptLogTopicSink, ScriptAnalysisService, docs/security.md (6 of 7 — only the Galaxy
|
||||
PR-4.W forward-refs remain; see A.13).
|
||||
@@ -0,0 +1,183 @@
|
||||
# HistorianGateway backend — issues identified
|
||||
|
||||
**Date:** 2026-06-27
|
||||
**Context:** Installed the `ZB.MOM.WW.HistorianGateway` sidecar on **wonder-app-vd03** and updated OtOpcUa to
|
||||
`origin/master` (`245316d8`, PR #423 "HistorianGateway as the OtOpcUa historian backend"), then smoke-tested
|
||||
the OtOpcUa→gateway link live. These are the issues found during that work.
|
||||
|
||||
> **File/line references are against `origin/master` @ `245316d8`** (the build that contains the gateway
|
||||
> backend — `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/`). They will not resolve on older
|
||||
> local checkouts that predate PR #423.
|
||||
|
||||
> **Live evidence** came from the gateway's Prometheus `/metrics` on the box: an OPC UA `historyread` on a
|
||||
> historized tag produced `historian_gateway_grpc_requests_total{method="ReadRaw",service="HistorianRead"}=1`
|
||||
> and `historian_session_pool_warm_sessions=1`, while a historized-tag **deploy** produced **zero**
|
||||
> `EnsureTags` calls.
|
||||
|
||||
---
|
||||
|
||||
## Issue 1 — `GatewayTagProvisioner` is never wired; EnsureTags auto-provisioning is dormant (HIGH, confirmed)
|
||||
|
||||
**Summary.** PR #423 ships `GatewayTagProvisioner` (the `IHistorianProvisioning` implementation that calls the
|
||||
gateway's `EnsureTags`) and unit tests for it, but it is **never registered in DI and never passed into the
|
||||
`AddressSpaceApplier`**. At runtime the applier uses the no-op `NullHistorianProvisioning`, so **deploying
|
||||
historized tags does NOT provision them in the historian**. The documented "EnsureTags auto-provisions
|
||||
historian tags on deploy" behaviour does not occur.
|
||||
|
||||
**Root cause (code).**
|
||||
- `AddressSpaceApplier` constructor coalesces a missing provisioner to the no-op:
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceApplier.cs:63`
|
||||
`_provisioning = provisioning ?? NullHistorianProvisioning.Instance;`
|
||||
- The **only** construction site omits the `provisioning:` argument:
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs:258`
|
||||
```csharp
|
||||
var applier = new AddressSpaceApplier(
|
||||
addressSpaceSink,
|
||||
loggerFactory.CreateLogger<AddressSpaceApplier>(),
|
||||
historizedSubscriptions: historizedSubscriptions); // <-- no provisioning:
|
||||
```
|
||||
- **No `IHistorianProvisioning` / `GatewayTagProvisioner` registration exists anywhere** in the repo
|
||||
(`grep` over `origin/master` finds only the interface definition, `NullHistorianProvisioning`, and the
|
||||
`GatewayTagProvisioner` constructor definition itself — no `AddSingleton<IHistorianProvisioning, …>` and
|
||||
no `new GatewayTagProvisioner(…)`).
|
||||
- Consequently `ProvisionHistorizedTags` (called unconditionally at
|
||||
`AddressSpaceApplier.cs:208`, body at `:226`, dispatch at `:261`) builds the request list and invokes
|
||||
`_provisioning.EnsureTagsAsync(...)` on the **no-op**, which returns without contacting the gateway.
|
||||
|
||||
**Live evidence.** On wonder-app-vd03: flagged the `parts-count` equipment tag `isHistorized:true`, deployed,
|
||||
and restarted the host. The gateway `/metrics` showed **no** gRPC request series for `HistorianTags`/`EnsureTags`
|
||||
(zero gateway calls). By contrast, an explicit OPC UA `historyread` on the same tag *did* reach the gateway
|
||||
(`ReadRaw`), confirming the read path is wired but the provisioning path is not.
|
||||
|
||||
**Impact.**
|
||||
- Historized tags are never created in the historian on deploy.
|
||||
- When **continuous historization** (`ContinuousHistorization:Enabled=true`) is later turned on, the recorder
|
||||
drains live values to historian tags that were never provisioned — those writes will fail / land nowhere
|
||||
unless the tags are created by some other means.
|
||||
- Docs (`docs/Historian.md`, repo `CLAUDE.md`, and the umbrella scadaproj `CLAUDE.md`) describe provisioning
|
||||
as active; that claim is currently false. See Issue 4.
|
||||
|
||||
**Recommended fix (two parts).**
|
||||
1. **Register the provisioner** in the Host alongside the existing gateway wiring — in
|
||||
`src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`, inside the `if (serverHistorianOptions.Enabled)` block that
|
||||
already calls `AddServerHistorian` / `AddAlarmHistorian` — e.g.
|
||||
```csharp
|
||||
builder.Services.AddSingleton<IHistorianProvisioning>(sp =>
|
||||
new GatewayTagProvisioner(
|
||||
HistorianGatewayClientAdapter.Create(serverHistorianOptions, sp.GetRequiredService<ILoggerFactory>()),
|
||||
sp.GetRequiredService<ILogger<GatewayTagProvisioner>>()));
|
||||
```
|
||||
(mirror the `GatewayHistorianValueWriter` registration pattern already in that block).
|
||||
2. **Pass it into the applier** at `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs:258`
|
||||
by resolving it the same way sibling deps are resolved there (`resolver.GetService<…>()`):
|
||||
```csharp
|
||||
var provisioning = resolver.GetService<IHistorianProvisioning>();
|
||||
var applier = new AddressSpaceApplier(
|
||||
addressSpaceSink,
|
||||
loggerFactory.CreateLogger<AddressSpaceApplier>(),
|
||||
provisioning: provisioning,
|
||||
historizedSubscriptions: historizedSubscriptions);
|
||||
```
|
||||
3. **Add a wiring/integration test** asserting `EnsureTags` is invoked on a historized-tag deploy. The existing
|
||||
`GatewayTagProvisioner` unit tests pass but do not cover the DI wiring, which is exactly the gap that let
|
||||
this ship dormant.
|
||||
|
||||
---
|
||||
|
||||
## Issue 2 — Provisioning success is logged nothing; the dormant path is invisible (LOW, observability)
|
||||
|
||||
**Summary.** The provisioning continuation in `AddressSpaceApplier` only emits a log line when
|
||||
`result.Failed > 0 || result.Skipped > 0`. A fully-successful provisioning (`Ensured>0, Failed=0, Skipped=0`)
|
||||
**and** the no-op `NullHistorianProvisioning` path both log **nothing**. This is what made Issue 1 invisible:
|
||||
there was no log line either way, so "no provisioning log" could not be distinguished from "provisioning ran
|
||||
and fully succeeded."
|
||||
|
||||
**Location.** `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceApplier.cs` — the
|
||||
`EnsureTagsAsync(...).ContinueWith(...)` block (~lines 261–283); the Information log is gated behind
|
||||
`if (result.Failed > 0 || result.Skipped > 0)`.
|
||||
|
||||
**Recommended fix.** Emit an Information tally on every dispatch (e.g. `requested/ensured/skipped/failed`),
|
||||
or at minimum whenever `Ensured > 0`, so a successful provisioning is observable and a no-op is detectable.
|
||||
|
||||
---
|
||||
|
||||
## Issue 3 — HistoryRead on a historized-but-unprovisioned tag blocks for the full CallTimeout (~30s) (MEDIUM, needs investigation)
|
||||
|
||||
**Summary.** An OPC UA `historyread` on a tag that is historized but has no corresponding historian
|
||||
tag/data returned only after ~30 s — the gateway's `ReadRaw` recorded
|
||||
`historian_gateway_grpc_request_duration_seconds_sum ≈ 30.5` for the single call, matching the
|
||||
`ServerHistorian:CallTimeout` default of `00:00:30`. The read did not fail fast on an unknown tag; it ran to
|
||||
the timeout and then errored (`outcome="error"`).
|
||||
|
||||
**Notes / attribution.**
|
||||
- The slow leg is the gateway→historian read of a non-existent tag; the OtOpcUa-facing symptom is a 30 s OPC UA
|
||||
HistoryRead latency governed by `ServerHistorianOptions.CallTimeout` (default 30 s, `…Runtime/Historian/ServerHistorianOptions.cs`).
|
||||
- This is **coupled to Issue 1**: because provisioning never runs, freshly-historized tags don't exist in the
|
||||
historian, so every HistoryRead on them hits this slow path until tags/data exist.
|
||||
- Single data point (one unprovisioned tag); confirm whether the slow path is the historian, the gateway, or the
|
||||
client-side timeout before changing behaviour.
|
||||
|
||||
**Recommended action.** Primarily fix Issue 1 so historian tags exist. Separately, consider a shorter default
|
||||
HistoryRead timeout and/or fast-fail when the historian reports an unknown tag, so a missing tag doesn't hang an
|
||||
OPC UA HistoryRead for 30 s.
|
||||
|
||||
---
|
||||
|
||||
## Issue 4 — Docs describe provisioning as active (LOW, documentation)
|
||||
|
||||
`docs/Historian.md`, the repo `CLAUDE.md`, and the umbrella `scadaproj/CLAUDE.md` state that the gateway backend
|
||||
"auto-provisions historian tags on deploy (EnsureTags)". Given Issue 1, that is inaccurate in the shipped build.
|
||||
Update the docs once Issue 1 is fixed, or flag the provisioning behaviour as not-yet-wired in the interim.
|
||||
|
||||
---
|
||||
|
||||
## Issue 5 — Provisioning covers only *newly-added* tags; flagging an existing tag for historization never provisions it (MEDIUM, follow-up to the Issue 1 fix)
|
||||
|
||||
**Status:** found while live-verifying the Issue 1 fix (commit `257214f7`) on wonder-app-vd03.
|
||||
|
||||
**Summary.** The Issue 1 fix correctly wires `GatewayTagProvisioner` and is confirmed deployed (the new
|
||||
`"...provisioning completed (dispatched…"` log literal is present in the deployed `OpcUaServer.dll`). However,
|
||||
`ProvisionHistorizedTags` still iterates **only** `plan.AddedEquipmentTags`:
|
||||
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceApplier.cs` (`ProvisionHistorizedTags`):
|
||||
`foreach (var tag in plan.AddedEquipmentTags)` — `ChangedEquipmentTags` is **not** considered.
|
||||
|
||||
So when an operator flags an **existing** tag `isHistorized:true` and deploys, that tag is classified
|
||||
**Changed** (its config changed; the tag already existed) and is **never provisioned**. This is compounded by
|
||||
the bootstrap "restore served state" behaviour: a host restart re-applies the served address space
|
||||
**incrementally**, so an existing tag comes back as *unchanged/changed*, never *Added* — i.e. a restart does
|
||||
not provision it either.
|
||||
|
||||
**Live evidence.** Flagging `parts-count` (an existing tag) historized + deploy + restart produced **zero**
|
||||
`EnsureTags` calls at the gateway (`/metrics` had no `HistorianTags/EnsureTags` series). The read path on the
|
||||
same tag *did* reach the gateway (`HistorianRead/ReadRaw`), confirming the tag was historized and the gateway
|
||||
link works — only provisioning was skipped. (A raw `dbo.Tag` INSERT of a brand-new historized tag did not
|
||||
materialise either, because a raw insert bypasses the normal compose/validate path — so the clean way to
|
||||
exercise the *added* path is to author a new historized tag via the AdminUI.)
|
||||
|
||||
**Impact.** The most common operator action — take an existing tag and turn on historization — does **not**
|
||||
auto-provision the historian tag. Continuous historization for such a tag would then write to a tag that does
|
||||
not exist. Only genuinely **new** historized tags get provisioned.
|
||||
|
||||
**Recommended fix.** Provision the union of added **and** newly-historized changed tags: also iterate
|
||||
`plan.ChangedEquipmentTags` (or compute "tags whose `IsHistorized` transitioned false→true") and include them
|
||||
in the `EnsureTags` request. `EnsureTags` is idempotent, so re-ensuring an already-present tag is safe. Add a
|
||||
test that flips an existing tag to historized and asserts `EnsureTags` is invoked.
|
||||
|
||||
---
|
||||
|
||||
## Pre-existing observation (not introduced by this work)
|
||||
|
||||
- **Serilog log path is service-CWD-relative.** On wonder-app-vd03 the host writes to
|
||||
`C:\Windows\System32\logs\otopcua-<date>.log` (because the service CWD is `System32`), not under the install
|
||||
directory. This was noted in prior sessions; re-observed here. Not a gateway-integration issue, but it makes
|
||||
on-box diagnosis awkward and is worth fixing (anchor the Serilog file sink to an absolute/content-root path).
|
||||
|
||||
---
|
||||
|
||||
## What is working (for context)
|
||||
|
||||
The OtOpcUa↔gateway **read and credential path is confirmed working end-to-end** on wonder-app-vd03: an OPC UA
|
||||
HistoryRead routed through `GatewayHistorianDataSource` to the gateway over h2c, authenticated with a
|
||||
peppered-HMAC API key (`historian:read` scope), and leased a real historian session (the gateway reached
|
||||
wonder-sql-vd03). Only the **provisioning** half (Issue 1) is unwired.
|
||||
Reference in New Issue
Block a user