docs: truth sweep — reconcile stale deferment/archreview/plan records with shipped state
v2-ci / build (push) Successful in 6m26s
v2-ci / unit-tests (push) Failing after 18m0s

- deferment.md: node ACLs no longer authored-but-inert (#520 shipped through stage 7, opt-in Security:Acl:Enabled); failure-visibility trio mostly closed (01/S-1 + 03/S4 fixed, 06/S-1 partial); bulkhead + WriteIdempotentAttribute closed by R2-02; Sql Postgres+Oracle dialects shipped (d54f3c83); akka-hosting partials verified superseded
- archreview/00-OVERALL.md: staleness banner on the carried-open block (04/C-1→R2-05, 03/S2+S3→R2-08, 03/P1→R2-07, driver fleet→R2-09/R2-11/#457, hygiene→R2-12, trio per above; what genuinely remains from theme 4)
- archreview/plans/STATUS.md: carved out the true exceptions to "every Round-2 live gate GREEN" (R2-03 T11/T12 deferred-live, R2-10 T10 breaker-OPEN never forced); #459 closed by mesh Phase 7 drill D4 (c50ebcf7)
- archreview/plans/INTEGRATION-SWEEP-STATUS.md: "follow-ups unstarted" → all 7 shipped as PRs #446–#452
- archreview/06-gateway-integrations.md: 06/S-1 STILL OPEN → PARTIAL (fail-closed advise + unconfirmed meter; residual cross-repo)
- 2026-07-31-apikey-acl-enforcement-design.md: status → implemented through stage 7, #520 CLOSED 2026-08-01
- 2026-07-21-per-cluster-mesh-design.md: "Phases 1–7 not started" → PROGRAM COMPLETE 2026-07-24 (Phase 7 c50ebcf7)
- 2026-07-20-localdb-phase2-live-gate.md: merged as 06b5d81b; open design question shipped by mesh Phase 6 (checks 3/4 still never re-run on the three-mesh topology)
- 2026-07-20-localdb-phase1-progress.md: Tasks 10–16 REMAIN → Phase 1 complete, live-gated 8/8, merged c957db52
- 2026-08-06-clusterclient-to-grpc-tracking.md: defect 4 OPEN → fixed by c0475f2d + e54883a8, merged @ b360239e
- 2026-08-06-clusterclient-to-grpc.md.tasks.json: Task 15 pending → completed (Phase 4 gate PASS 8/8 @ 0b194003, record d0ac6fc5)
- 2026-07-22-mesh-phase2-live-gate.md: DriverStatusPanel orphan → wired into /hosts by #546 (9420f8ef)
- 2026-07-21-issue-485-live-gate.md: pointer from the "NOT fixed here" finding to the #486 fix (DriverHostActor.cs:1696-1717 + FetchAndCache mirror)
- 2026-08-05-sql-postgres-oracle.md: chrome/AdminUI gate not deferred — PASSED 2026-08-05, found + fixed the Test-Connect provider gate (4e461951)
- v2-ga-lab-gates-plan.md / v2-release-readiness.md / live-hardware-validation-runbooks.md: FOCAS live-CNC smoke #54 RESOLVED on a real 31i-B 2026-06-25 (cnc_rdparam caveat preserved)
- Mqtt-Test-Fixture.md: STATE/DCMD "not implemented" → shipped + live-gated (#508 2a7d1167, #511 857a10b4)
- 2026-08-01-continuous-historization-live-gate.md: poison-tag FIFO pin "filed, not fixed" → #538 fixed same day (e7fbe977, gate 4bdce910; Capacity default now 500k)
- 2026-05-26-akka-hosting-alignment-plan.md.tasks.json: F8/F9/F10/F13/F14 partial → completed, superseded by the v3 address-space rework (Phase7Composer/ScriptedAlarmActor deleted; evaluators/SDK sinks/security policies production-wired)

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
This commit is contained in:
Joseph Doherty
2026-08-07 02:02:54 -04:00
parent 8b5c478c36
commit 290117660d
20 changed files with 128 additions and 29 deletions
+19
View File
@@ -59,6 +59,25 @@
**Carried open from 2026-07-08 (re-verified, no remediation branch targeted them):**
> **⚠️ Staleness banner — update 2026-08-07 (truth sweep): most of this block has since closed in
> Round 2** (see `archreview/plans/STATUS.md` for the per-plan record):
> - **04/C-1** closed by **R2-05** — named policies on all 38 routable AdminUI pages + the
> `PageAuthorizationGuardTests` anti-drift guard.
> - **03/S2 + 03/S3** closed by **R2-08** (LDAP async/timeout/outage circuit; HistoryRead bounded
> async fan-out) — both live-verified 2026-07-15.
> - **03/P1** closed by **R2-07** (surgical pure-adds/removes/mixed; marked REMEDIATED in
> `03-server-runtime.md`).
> - The **driver-fleet batch** closed by **R2-09 / R2-11 / #457** (PollGroupEngine v2, TagConfig
> parse consolidation, strict enum parsing).
> - **Repo hygiene** closed by **R2-12** (AVEVA DLLs removed, husks archived, CLI docs, CI dedupe).
> - The **failure-visibility trio**: 01/S-1 **fixed** (`AddressSpaceApplyOutcome.FailedNodes`),
> 03/S4 **fixed** (`PrimaryGatePolicy` default-deny on unknown role, live-proven R2-04 T15),
> 06/S-1 **partial** (fail-closed advise path + `galaxy.writes.unconfirmed` meter; residual is the
> cross-repo gateway-side WriteComplete correlation).
>
> What genuinely remains from theme 4 below: ~100 fixed sleeps in driver suites, no bUnit (04/U-1),
> and 03/U6 (behavioral wiring tests cover only 2 of 6 dispatch sites).
- **Failure-visibility group** — applier swallows sink failures / no failure field in `AddressSpaceApplyOutcome` (01/S-1); Galaxy write success optimistic (06/S-1); primary gate default-allow on unknown role (03/S4). STATUS.md queues these as "next"; still the top carried batch.
- **04/C-1 (High)** — AdminUI's largest mutating surface still bare `[Authorize]` (GlobalUns, EquipmentPage, cluster/driver editors, Reservations); three-idiom authz mix. Stakes **rose** this round: the ungated surface now includes live resilience config.
- **Availability** — LDAP block-bridging (03/S2), HistoryRead block-bridging (03/S3), full-rebuild-per-tag-add killing all subscriptions (03/P1 — now unblocked by the U2 forwarding guard, work not started).
+1 -1
View File
@@ -16,7 +16,7 @@
| ID | One-liner | Status @ f6eaa267 |
|---|---|---|
| S-1 (High) | Galaxy write success is optimistic; committed-write failure can never surface | **STILL OPEN** — `GatewayGalaxyDataWriter.cs:281-302` unchanged (empty-statuses ⇒ `Good`; advise-failure path `:234-243` still proceeds). STATUS.md lists "06/S-1 (Galaxy fail-closed)" under *suggested next*, not done. |
| S-1 (High) | Galaxy write success is optimistic; committed-write failure can never surface | ~~**STILL OPEN** — `GatewayGalaxyDataWriter.cs:281-302` unchanged (empty-statuses ⇒ `Good`; advise-failure path `:234-243` still proceeds). STATUS.md lists "06/S-1 (Galaxy fail-closed)" under *suggested next*, not done.~~ **Update 2026-08-07 (truth sweep): PARTIAL** — the fail-closed advise path and the `galaxy.writes.unconfirmed` meter shipped in `GatewayGalaxyDataWriter.cs` (failure-visibility meters, advise failure no longer proceeds; empty statuses still ⇒ provisional Good but metered). Residual = the gateway-side WriteComplete correlation (cross-repo, mxaccessgw). |
| S-2 (Med) | EventPump saturation drops the newest events (inverted staleness bias) | **STILL OPEN** — `EventPump.cs:128-140` unchanged. |
| S-3 (Med) | `ReadViaSubscribeOnceAsync` can wait forever on a non-cancellable token | **STILL OPEN** — `GalaxyDriver.cs:738-755` unchanged. |
| S-4 (Med) | Transport-failure detection is EventPump-only; subscription-less driver never degrades | **STILL OPEN** — `ReportTransportFailure` still has the single pump-fault caller (`GalaxyDriver.cs:929-949`). |
+4 -1
View File
@@ -223,4 +223,7 @@ TCP opens before the register map fully loads.
- **All fixtures torn down** — modbus/abcip/s7/opcuaclient/focas on the host removed (ports 5020/44818/1102/
50000/8193 all closed); the local Host SQL fixture `down -v`'d. Shared host is clean.
- **Repo:** `master @ 30672888`, clean. No sweep changes committed (this file is the only artifact).
- **Nothing merged for the sweep** — it's a read-only test run; the follow-ups above are unstarted.
- **Nothing merged for the sweep** — it's a read-only test run. ~~The follow-ups above are unstarted.~~
**Update 2026-08-07 (truth sweep):** stale on arrival — all 7 follow-ups shipped 2026-07-15 as PRs
**#446–#452** (see the "Follow-up progress: ALL 7 CLOSED" note at the top of this file, which this
line contradicted).
+8 -4
View File
@@ -224,7 +224,7 @@ shared docker-dev rig / VPN infra isn't contended. To run one-at-a-time:
- **Full integration-suite sweep** (10 `*.IntegrationTests` projects, one at a time): discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24. **RAN 2026-07-14 — see [`INTEGRATION-SWEEP-STATUS.md`](INTEGRATION-SWEEP-STATUS.md).** Outcome: OpcUaClient(+Browser)/S7/Modbus-standard green, Host 105/117, TwinCAT clean-skip; the rest fail on **pre-existing test/fixture bit-rot** (AbCip static-init bug, Modbus pymodbus-4.x sim drift, FOCAS 127.0.0.1-mock topology, AbLegacy no fixture, OpcUaServer cert-store config, Host openldap:2.6 image gone). **No OtOpcUa regression.** 7 follow-ups in that file.
- **docker-dev `:9200` live-`/run`** (rebuild BOTH centrals): R2-02 T15 (ResilienceConfig warning banner / input lockout / unknown-key survival), R2-03 T12 (VT fail→Bad→recover + P7 no-recompile), R2-10 T10 (breaker-open on both centrals + recovery + FOLLOWUP-10 log line), R2-11 T22 (typed-editor Writable checkbox), **R2-07 T12 (surgical add/remove/mixed — F10b inertness risk, MUST run)**, R2-05 T15 (over-gating regression only — deny unobservable under `DisableLogin`).
- **2-node redundancy rig:** ~~R2-04 T13/T15~~ **BOTH ✅ PASSED 2026-07-14** — T15 live docker-dev gate + T13 in-process delivery-path test (see below).
- **Genuinely external / needs-user:** ~~ALL CLOSED as of 2026-07-15.~~ ~~R2-06 T12~~ (6/6 on the VPN, 2026-07-13). ~~R2-08 S3 `Category=LiveIntegration` gateway soak~~ **CLOSED 2026-07-15** (6/6 vs. the running local 0.2.0 gateway + real historian over VPN). ~~R2-01 T11~~ **CLOSED 2026-07-15** (S7 blackhole via `docker pause`; found + fixed the read-leg wall-clock gap, PR #453). ~~R2-08 GLAuth outage cycle~~ **CLOSED 2026-07-15** (`LdapGlauthOutageLiveTests` vs. the harness GLAuth paused via `docker pause`, PR #455). **Every Round-2 live gate is now GREEN.**
- **Genuinely external / needs-user:** ~~ALL CLOSED as of 2026-07-15.~~ ~~R2-06 T12~~ (6/6 on the VPN, 2026-07-13). ~~R2-08 S3 `Category=LiveIntegration` gateway soak~~ **CLOSED 2026-07-15** (6/6 vs. the running local 0.2.0 gateway + real historian over VPN). ~~R2-01 T11~~ **CLOSED 2026-07-15** (S7 blackhole via `docker pause`; found + fixed the read-leg wall-clock gap, PR #453). ~~R2-08 GLAuth outage cycle~~ **CLOSED 2026-07-15** (`LdapGlauthOutageLiveTests` vs. the harness GLAuth paused via `docker pause`, PR #455). **Every Round-2 live gate is now GREEN** — *except the two true exceptions carved out below (truth sweep 2026-08-07): R2-03 T11/T12 remain deferred-live (needs a data-fed driver fixture) and R2-10 T10's breaker-OPEN state was never forced live (pipeline verified, breaker-open chip not); see the "docker-dev live-`/run` pass" section's own R2-03/R2-10 entries, which this line contradicted.*
### Deferred behavior-change follow-up — **SHIPPED 2026-07-15 (Gitea #457)**
**R2-11 Phase C** — flip the runtime driver parsers to strict (typo'd enum ⇒ `BadNodeIdUnknown` instead of a
@@ -247,7 +247,11 @@ data). R2-01's S7 blackhole gate is **CLOSED (2026-07-15)** — run via `docker
it found + fixed the read-leg wall-clock gap (PR #453). R2-08's GLAuth outage-cycle leg is **CLOSED (2026-07-15, PR #455)** —
`LdapGlauthOutageLiveTests` vs. the harness GLAuth on `10.100.0.35`. R2-08 S3 `Category=LiveIntegration` gateway soak is
**CLOSED (2026-07-15, 6/6)** and R2-06 (6/6, 2026-07-13) — both vs. the local 0.2.0 gateway + real historian over VPN.
**Every Round-2 live gate is now GREEN — no Round-2 items remain unexercised.**
**Every Round-2 live gate is now GREEN — no Round-2 items remain unexercised.** *(Correction, truth
sweep 2026-08-07: overstated — two items DO remain unexercised, per this file's own live-`/run` pass
section: R2-03 T11/T12 is deferred-live pending a data-fed driver fixture, and R2-10 T10's
breaker-OPEN state was never forced — the pipeline is live-proven but the explicit `breaker OPENED`
line + resilience chip were not triggered.)*
### Residual follow-ups — tracked in Gitea (2026-07-15)
@@ -258,7 +262,7 @@ The remaining non-blocking items are filed as issues (none gate the remediation,
| **#456** | #10 — observe retry/breaker firing in live dispatch; thread `ResilienceConfig` into the deploy artifact (tier-defaults only today) | niche / sub-gap |
| **#457** | R2-11 Phase C — flip runtime parsers to strict, after fleets run `TagConfigValidationMode=Error` clean | deliberately deferred (operational rollout) |
| **#458** | Remove the `NuGetAuditSuppress` for CVE-2025-6965 once a patched SQLitePCLRaw bundle ships | upstream-gated |
| **#459** | SBR topology — crashing the OLDEST Akka node does not fail over | pre-registered user-deferred design gap |
| **#459** | SBR topology — crashing the OLDEST Akka node does not fail over. **Update 2026-08-07 (truth sweep): CLOSED** — the downing strategy moved to Akka `AutoDowning` (`master @ 0de1352e`, 2026-07-21) and mesh Phase 7 drill D4 (`c50ebcf7`, 2026-07-24) ran the 1-vs-1 crash-the-oldest drill live: every survivor stayed Up without self-downing | pre-registered user-deferred design gap — since closed |
Brought up the 2-node central cluster (`central-1`+`central-2`, SQL, traefik) from source at merged master `6aeb53a9`
(single-tenant rig; drove the `:9200` AdminUI via browser + Client.CLI on `:4840`/`:4841`).
@@ -298,7 +302,7 @@ Brought up the 2-node central cluster (`central-1`+`central-2`, SQL, traefik) fr
**Rig note:** the pass mutated the persisted ConfigDb (added a `drv-modbus-line3-01` Modbus driver on MAIN; removed
`capper-03`'s `production-count` virtual tag for the PureRemove test). Reset with
`docker compose -f docker-dev/docker-compose.yml down -v`. R2-01's S7 blackhole gate is now CLOSED (2026-07-15, PR #453);
**all Round-2 live gates are now GREEN** (R2-06 + R2-08 S3 verified on the VPN 2026-07-15; R2-01 + R2-08 GLAuth self-serviced on `10.100.0.35`).
**all Round-2 live gates are now GREEN** (R2-06 + R2-08 S3 verified on the VPN 2026-07-15; R2-01 + R2-08 GLAuth self-serviced on `10.100.0.35`) — *with the two carve-outs recorded above (truth sweep 2026-08-07): R2-03 T12 remains live-blocked on a data-fed rig, and R2-10 T10's breaker-OPEN was never forced.*
---
+29 -8
View File
@@ -32,7 +32,7 @@ how much they should worry you:
| # | Theme | Why it matters | Now |
|---|---|---|---|
| **1** | **Three subsystems are fully authored, persisted, and shipped — and never executed.** Node ACLs, `IRediscoverable`/`IHostConnectivityProbe`, and `DriverTypeRegistry`. | An operator can author a rule, save it, deploy it green, and have it do **nothing**. §3 | ◐ **Two of three now settled**: `IRediscoverable` consumed (#518), `IHostConnectivityProbe` re-transported to telemetry (#521), `DriverTypeRegistry` + the tier machinery **deleted** (#522). Only **node ACLs** remain authored-but-inert, tracked as #520 |
| **1** | **Three subsystems are fully authored, persisted, and shipped — and never executed.** Node ACLs, `IRediscoverable`/`IHostConnectivityProbe`, and `DriverTypeRegistry`. | An operator can author a rule, save it, deploy it green, and have it do **nothing**. §3 | ✅ **All three settled**: `IRediscoverable` consumed (#518), `IHostConnectivityProbe` re-transported to telemetry (#521), `DriverTypeRegistry` + the tier machinery **deleted** (#522). Node ACLs are **now enforced** — #520 shipped through stage 7 (merges `fbe7403c` stages 1-2/4, `3a6b2a1d` stage 3, `1ccc7400`/`f692f6af` stage 5 write/alarms/HistoryRead, `d4389f57`/`00aa4866` stage 6 read/browse/subscribe, `3796c0bf` stage 7 authoring; wired at `Host/Program.cs:394-398` → `DataPlaneAuthorizer`). Enforcement is **opt-in**: `Security:Acl:Enabled` defaults `false` |
| **2** | **AdminUI authoring-dispatch gaps.** Calculation is offered in the driver picker but has no config form; three drivers have no device form; CSV + browse-commit dispatch maps miss the new drivers. | The "registered but unauthorable" class that has now bitten twice. §1.2 | ✅ **Closed** — G-1…G-6 fixed and guarded; G-7 remains |
| **3** | **17 open issues, all confirmed still open**, several broader than filed. | §2 | ◐ 6 resolved + closed, 6 new filed (one since closed) → **17 open**, none stale. Two of the six closures (#521, #522) resolved *differently from how they were filed* — see §3.2/§3.3 |
| **4** | **Bookkeeping drift.** 8 `.tasks.json` files show ~140 "pending" tasks for work that shipped; 5 CLAUDE.md status claims are stale. | Someone follows the tracker and rebuilds a shipped driver. §6, §7 | ✅ **Closed** — 13 plan files + 11 archreview gates written back |
@@ -56,9 +56,9 @@ So everything below is **untracked work**. Nothing in it is a data-loss or outag
| **4 live gates blocked on hardware/licence** — archreview R2-03 (VT Good→Bad on a data-fed rig), R2-10 (force breaker-OPEN), Galaxy `ScanStateProbeParityTests` (licence-gated), v2 GA exit criteria (FOCAS live-CNC smoke #54, OPC UA CTT pass, Ignition 8.3 cutover) | Cannot be closed from this box. Not decisions — inputs. | §5 |
| **1 deferred live-verify task** — `2026-06-14-driver-reconfigure-while-faulted` Task 3, subject literally "DEFERRED (skipped)" | Now runnable: docker-dev login is disabled, so `/run` verification needs no operator. | §5 |
| **G-7 residual** — Sql / MTConnect / Calculation / OpcUaClient have no Contracts-side `Inspect` | **The deploy gate is weaker than the authoring gate.** `TagConfigValidator` covers 11 of 12, so config arriving by **CSV import or API** — the paths that skip the UI — meets only the weaker gate. | §1.2 |
| **3 carried-open archreview findings** — the failure-visibility trio: 01/S-1 (`AddressSpaceApplier` swallows every sink failure; a deploy reports **Applied** while broken), 03/S4 (primary gate default-allows on unknown role), 06/S-1 (Galaxy optimistic write success) | No remediation branch ever targeted them. 01/S-1 is the sharpest: a green deploy that isn't. | §6.3 |
| ~~**3 carried-open archreview findings** — the failure-visibility trio~~ **Update 2026-08-07 (truth sweep): mostly CLOSED.** 01/S-1 **FIXED** — `AddressSpaceApplier.cs:95-97` tallies swallowed removal-pass failures into `AddressSpaceApplyOutcome.FailedNodes` (comment cites 01/S-1). 03/S4 **FIXED** — `PrimaryGatePolicy.cs` default-denies on unknown role when a driver peer exists (doc-comment "Fixes archreview 03/S4"), live-proven R2-04 T15. 06/S-1 **PARTIAL** — fail-closed advise path + `galaxy.writes.unconfirmed` meter shipped in `GatewayGalaxyDataWriter.cs`; residual = gateway-side WriteComplete correlation (cross-repo, mxaccessgw). | Residual is only the 06/S-1 cross-repo half. | §6.3 |
| **1 genuinely unexecuted plan** — `2026-05-29-adminui-followups`, **19 pending / 0 completed** | The only substantial engineering left. ⚠️ Its first step is an **overlap audit**, not code: several items look superseded by v3's `TagConfigEditorMap`. | §6.2 |
| **11 other open plan tasks** — 5 akka-hosting partials (likely v3-superseded), 4+1 followups-batch (open *by choice*), fixedtree Task 11, 1 manual smoke checklist | Small, and two of the groups are decisions already taken. | §6.2 |
| **11 other open plan tasks** — ~~5 akka-hosting partials (likely v3-superseded)~~ *(verified superseded 2026-08-07 — F8/F9/F10/F13/F14 marked completed; `Phase7Composer`/`ScriptedAlarmActor` deleted, evaluators/SDK sinks/security policies all production-wired)*, 4+1 followups-batch (open *by choice*), fixedtree Task 11, 1 manual smoke checklist | Small, and two of the groups are decisions already taken. | §6.2 |
| **Review-coverage gap** — 10 shipped projects never reviewed (Calculation, Historian.Gateway, Mqtt +Browser/Contracts, MTConnect +Contracts, Sql +Browser/Contracts) while **3 modules still review the retired Wonderware backend** | Re-verified 2026-08-04 against `code-reviews/`. Unchanged since the audit. | §6.3 |
| **`RawBrowseModal` stale-open race** (added 2026-08-05) — `OnParametersSetAsync` never re-checks staleness after its awaits, so a superseded open's late resolve can install the wrong device's browse config. The identical shape was found by review and **fixed** in `RawTagModal`/`RawManualTagEntryModal` (post-await staleness re-check; see `docs/plans/2026-08-05-raw-modal-picker-live-config-design.md` §6); the browse modal was deliberately left out of that branch's scope. | Lower stakes than the fixed pair (the modal resolves the device it was opened for and shows its own error banner), but the fix pattern is proven and cheap to port. | design doc §6 |
@@ -66,7 +66,10 @@ So everything below is **untracked work**. Nothing in it is a data-loss or outag
unknown role" — is the *same shape* as a decision deliberately taken on 2026-08-04 in
`PrimaryGatePolicy.ShouldRecordHistorizedValues`, where an unknown role **captures**. That is right for
historization (a duplicate sample beats a hole in the trend, and the duplicate is bounded by promotion)
and wrong for a write gate. Closing 03/S4 must not sweep both under one rule.
and wrong for a write gate. Closing 03/S4 must not sweep both under one rule. **Update 2026-08-07
(truth sweep): it didn't** — 03/S4 is closed with exactly this split preserved: `PrimaryGatePolicy.
ShouldServiceAsPrimary` default-denies on unknown role when a driver peer exists, while
`ShouldRecordHistorizedValues` still captures on unknown.
---
@@ -77,7 +80,7 @@ themes into tracked decisions. What remains:
| Item | Nature | Where |
|---|---|---|
| **#520** node ACLs authored + deployed but **never enforced** | Decision: wire it or retire the surface. Non-enforcement is now stated in docs and warned in the UI, so nobody can author a deny rule believing it works — but reads are still ungated. | §3.1 |
| **#520** node ACLs authored + deployed but **never enforced** ✅ **CLOSED 2026-08-01** *(truth-sweep note 2026-08-07: shipped through stage 7 — enforcement merges `fbe7403c`/`3a6b2a1d`/`1ccc7400`/`d4389f57`/`3796c0bf`; opt-in via `Security:Acl:Enabled`, default `false`)* | Decision: wire it or retire the surface. ~~Non-enforcement is now stated in docs and warned in the UI, so nobody can author a deny rule believing it works — but reads are still ungated.~~ It was wired: with the flag on, scoped grants decide every data-plane operation incl. read/browse/subscribe. | §3.1 |
| **#521** `IHostConnectivityProbe` / `DriverHostStatus` dead surface ✅ **CLOSED** `abda6f9e` | Resolved by changing the *transport*, not by either option as filed — the publisher as designed was unbuildable after Phase 4. Host status now rides `DriverHealthChanged` to `/hosts`; table dropped. | §3.2 |
| **#522** driver stability tiers inert ✅ **CLOSED** `abda6f9e` | Deleted. The inertness was broader than filed (nothing constructed `MemoryTracking` either), and the model's premise — an out-of-process Host to recycle — no longer exists. `DriverTier` kept. | §3.3 |
| ~~**#523** remove the periodic 5-min device re-browse~~ | ✅ **Done + closed on the tracker.** **AbCip and FOCAS now have no tag-change signal at all**; that is the accepted consequence, not an oversight. | §9 (2026-07-30, #523) |
@@ -242,6 +245,16 @@ subsystems are fully built and reachable by operators, but no production code pa
### 3.1 Node-level ACLs are authorable, deployed — and unenforced
> ✅ **Update 2026-08-07 (truth sweep): NO LONGER TRUE — #520 shipped through stage 7 and is CLOSED
> (2026-08-01).** Enforcement merges: `fbe7403c` (stages 1-2, 4: API-key identity + keys/ACLs in the
> artifact), `3a6b2a1d` (stage 3: variable-depth scope model), `1ccc7400`/`f692f6af` (stage 5: write,
> alarms, HistoryRead), `d4389f57`/`00aa4866` (stage 6: read, browse, subscribe), `3796c0bf` (stage 7:
> ACL authoring/scope picker). `TriePermissionEvaluator` is now wired at `Host/Program.cs:394-398` into
> `DataPlaneAuthorizer`, registered on every node. Enforcement is **opt-in** — `Security:Acl:Enabled`
> defaults `false`; with it off the two coarse server-wide gates remain the whole of data-plane
> authorization, with it on scoped `NodeAcl` grants decide every data-plane operation. The section
> below is preserved as the pre-#520 record.
> ◐ **Still true; now impossible to miss** (`53ede679`, Gitea **#520**). `docs/security.md`,
> `docs/ReadWriteOperations.md`, `docs/v2/acl-design.md` and `v2-release-readiness.md` corrected, and
> `ClusterAcls.razor` / `AclEdit.razor` carry a warning alert. The audit named
@@ -332,8 +345,8 @@ assigns Galaxy and FOCAS to Tier C.
| `IDriverConfigEditor` — interface, zero implementors, zero production consumers | `Core.Abstractions/IDriverConfigEditor.cs:9`; superseded by `TagConfigEditorMap` / `DriverConfigModal` |
| `GenericDriverNodeManager` — explicitly **not** a production dispatch path | `Core/OpcUa/GenericDriverNodeManager.cs:71` ("verified 07/#10: zero production references") |
| `IDriverSupervisor` — Tier-C recycle machinery has **zero implementations**, yet the parser still validates `RecycleIntervalSeconds` | archreview 01/U-2 |
| Resilience **bulkhead** — documented, defaulted, parsed, AdminUI-authorable; `Build` never adds a concurrency limiter | archreview 01/U-6 |
| `WriteIdempotentAttribute` — zero readers; write dispatch hardcodes `isIdempotent: true` | archreview 01/S-8 = 03/S12 |
| ~~Resilience **bulkhead** — documented, defaulted, parsed, AdminUI-authorable; `Build` never adds a concurrency limiter~~ **Update 2026-08-07 (truth sweep): closed by archreview R2-02** — the bulkhead knob was **deleted end-to-end** (migration-free; stored keys ignored) with an inertness guard test; see `archreview/plans/STATUS.md` R2-02 row | archreview 01/U-6 |
| ~~`WriteIdempotentAttribute` — zero readers; write dispatch hardcodes `isIdempotent: true`~~ **Update 2026-08-07 (truth sweep): closed by archreview R2-02** — dispatch flipped to `isIdempotent: false` (RED-first on the Runtime wiring test) and the attribute's false claim corrected; see `archreview/plans/STATUS.md` R2-02 row | archreview 01/S-8 = 03/S12 |
| ~60 lines of unreachable code retained behind #507's guard | `DriverHostActor.cs:1002`, `#pragma warning disable CS0162` |
| Three empty `Driver.Historian.Wonderware*` directories (0 `.cs`, no `.csproj`, absent from `.slnx`) | verified for this report — only stale `bin/`+`obj/` remain |
@@ -359,7 +372,7 @@ commented-out DI registrations.** Real deferred work lives in prose doc-comments
- **S7** — array **writes** unsupported entirely (`S7Driver.cs:1139`); several array *read* types unsupported (`:386,404,413,419,428,442,465,807,959`) — all fail fast at config-validation; legacy C-area counter reinterpretation live-hardware-gated (`:759`); Counter BCD on S7-300/400 undecoded.
- **AbCip** — ALMD-only alarm projection, ALMA deferred (`AbCipAlarmProjection.cs:16,324`); no CIP Template Object reader, UDT layouts declaration-driven (`AbCipDriverOptions.cs:172`); whole-UDT writer deferred (`LibplctagTagRuntime.cs:190`); `AckCmd` pulse must be wired client-side (`:30`).
- **Sql** — `SqlTagModel.Query` deferred to P3, **enforced end-to-end** across parser, validator, planner and editor (`SqlProvider.cs:22,37`; `SqlEquipmentTagParser.cs:24,103`; `SqlGroupPlanner.cs:44,249`; `SqlTagConfigModel.cs:114`) — exemplary deferral discipline. Postgres/ODBC dialects deferred behind `ISqlDialect`.
- **Sql** — `SqlTagModel.Query` deferred to P3, **enforced end-to-end** across parser, validator, planner and editor (`SqlProvider.cs:22,37`; `SqlEquipmentTagParser.cs:24,103`; `SqlGroupPlanner.cs:44,249`; `SqlTagConfigModel.cs:114`) — exemplary deferral discipline. ~~Postgres/ODBC dialects deferred behind `ISqlDialect`~~ *(update 2026-08-07, truth sweep: **PostgreSQL and Oracle dialects shipped 2026-08-05**, merge `d54f3c83` — managed Npgsql / Oracle.ManagedDataAccess.Core, live-gated; `MySql`/`Odbc` remain reserved enum names that fail at construction)*.
- **MTConnect** — `TIME_SERIES` vectors deferred to P1.5; `DATA_SET`/`TABLE` coded `BadNotSupported` with a null value rather than approximated (`MTConnectObservationIndex.cs:53,272`) — deliberately chosen over a Good-coded lie. `RawTagEntry.DeviceName` ignored for routing (two `/raw` Devices under one driver share an Agent connection) — "a design change, not a bug fix". No DataType-vs-blob validation at authoring.
- **MQTT** — `MaxPayloadBytes` is a constructor knob, not operator-facing (`MqttSubscriptionManager.cs:170`).
- **TwinCAT** — Flat-mode struct expansion carries a **LIVE RISK** note, unverified against a real TC3 target (`AdsTwinCATClient.cs:270`); VirtualTree-mode browse unverified; TC2 coverage, multi-hop AMS route and lab rig all hardware-blocked (`docs/v3/twincat-backlog.md`).
@@ -475,6 +488,11 @@ bookkeeping, not backlog.
targeted them" block. The largest coherent batch is the **failure-visibility trio**: 01/S-1
(`AddressSpaceApplier` swallows every sink failure, deploy reports Applied even when broken), 03/S4
(primary gate default-allow on unknown role), 06/S-1 (Galaxy optimistic write success).
**Update 2026-08-07 (truth sweep): the trio is mostly closed.** 01/S-1 fixed
(`AddressSpaceApplier.cs:95-97` → `AddressSpaceApplyOutcome.FailedNodes`); 03/S4 fixed
(`PrimaryGatePolicy.cs` default-deny on unknown role with a driver peer, live-proven R2-04 T15);
06/S-1 partial (fail-closed advise path + `galaxy.writes.unconfirmed` meter in
`GatewayGalaxyDataWriter.cs`; residual = gateway-side WriteComplete correlation, cross-repo).
- **`stillpending.md` does not exist** — the backlog is `docs/plans/2026-06-18-stillpending-backlog-audit.md`.
---
@@ -544,6 +562,9 @@ appear in §7.1 as well.
`docs/ReadWriteOperations.md` first; it is the one that could mislead a security review.~~
✅ **Done** — decided *make non-enforcement explicit*; wire-up tracked as Gitea **#520**. See §9.
(`docs/ReadWriteOperations.md` was **not** the sharpest one — `docs/security.md` was.)
**Update 2026-08-07 (truth sweep): the wire-up itself has since shipped** — #520 CLOSED 2026-08-01
through stage 7 (merges `fbe7403c`, `3a6b2a1d`, `1ccc7400`, `d4389f57`, `3796c0bf`); enforcement
opt-in via `Security:Acl:Enabled` (default `false`). See §3.1's banner.
2. ~~**#518 + #507 together** — neither fix is observable alone.~~ ✅ **Done.** The framing was right
that they had to be handled together, but wrong about the resolution: **#507 was not revivable**,
so it was deleted rather than fixed. See §9.
+1 -1
View File
@@ -116,7 +116,7 @@ test. No automated coverage asserts that two concurrent drivers coexist.
| Does a Sparkplug birth → alias → DATA → death cycle work? | ✅ yes (live, vs. the simulator) |
| Does an NCMD rebirth request actually get answered? | ✅ yes (the simulator subscribes + re-births) |
| Sparkplug seq-gap → rebirth | ❌ unit-only (the sim never skips a seq) |
| Sparkplug primary-host STATE / DCMD writes | ❌ not implemented |
| Sparkplug primary-host STATE / DCMD writes | ✅ both shipped + live-gated *(corrected 2026-08-07, truth sweep)* — NCMD/DCMD write-through as Gitea #508 (merge `2a7d1167`, 2026-07-31, 21/21 live) and primary-host STATE + Last Will as Gitea #511 (merge `857a10b4`, 2026-07-31, 5/5 live) |
## Follow-up candidates
@@ -82,13 +82,13 @@
{"id": "F5", "subject": "Follow-up: ConfigPublishCoordinator multi-node happy-path test", "status": "completed", "classification": "standard", "estMinutes": 30, "parallelizableWith": [], "blockedBy": [], "commit": "5cfbe8b", "deviation": "Delivered by Task 59 — DeployHappyPathTests.StartDeployment_seals_after_both_nodes_apply exercises the exact 'dispatch to N driver nodes, all ack, seals' flow via the real 2-node TwoNodeClusterHarness rather than a multi-system TestKit. Cleaner because it tests the production code path end-to-end.", "origin": "Self-review of Task 30 — single-ActorSystem TestKit can't simulate the plan's 'dispatch to N driver nodes, all ack, seals' happy path because DiscoverDriverNodes() needs real cluster membership. Add a multi-system test (two ActorSystems joined into one cluster, driver-role on the second)."},
{"id": "F6", "subject": "Follow-up: RedundancyStateActor publisher abstraction so tests don't need DPS bootstrap", "status": "completed", "classification": "small", "estMinutes": 10, "parallelizableWith": [], "blockedBy": [], "commit": "dfc143c", "origin": "Self-review of Task 35 — RedundancyStateActorTests are skipped because single-node DistributedPubSub bootstrap is unreliable in TestKit. Inject an Action<object> broadcast so tests can replace it with a probe; un-skip both tests."},
{"id": "F7", "subject": "Follow-up: DriverInstanceActor full engine wiring (subscriptions, writes, ApplyDelta diff)", "status": "completed", "classification": "standard", "estMinutes": 45, "parallelizableWith": [], "blockedBy": [44], "origin": "Self-review of Task 41 — subscription publishing, ApplyDelta diffing, bad-quality-on-disconnect, write path, and supervisor backoff are stubbed. Wire after OpcUaPublishActor lands.", "shipped": "All three pieces landed: (1) spawn lifecycle in DriverHostActor (DriverSpawnPlanner + IDriverFactory seam) — da14149, (2) ISubscribable wiring + OPC UA status-code → OpcUaQuality severity-bit mapping + DetachSubscription on disconnect/PostStop, (3) IWritable.WriteAsync write path with 5s timeout, status-code bubble-up, and AttributeValuePublished published to parent on every OnDataChange — both shipped in the F7-residual batch. Host DI binding (DriverFactoryBootstrap registers AbCip/AbLegacy/FOCAS/Galaxy/Modbus/S7/TwinCAT factories) lives in src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/."},
{"id": "F8", "subject": "Follow-up: VirtualTagActor engine wiring (compile expression, subscribe deps, publish result)", "status": "partial", "classification": "standard", "estMinutes": 30, "parallelizableWith": [], "blockedBy": [], "origin": "Self-review of Task 42 — VirtualTagEngine.Evaluate not called; DependencyValueChanged just buffers.", "shipped": "(1) IVirtualTagEvaluator seam + NullVirtualTagEvaluator default. VirtualTagActor calls evaluator on DependencyValueChanged, dedupes unchanged results, emits EvaluationResult to parent, publishes Warning ScriptLogEntry on failure. (2) DependencyMuxActor in Runtime fans out DriverInstanceActor.AttributeValuePublished from DriverHostActor through to interested VirtualTagActor subscribers. VirtualTagActor takes dependencyRefs + mux ActorRef in Props, registers interest in PreStart, unregisters in PostStop. WithOtOpcUaRuntimeActors spawns the mux + threads it into DriverHostActor. Production binding to Core.VirtualTags.VirtualTagEngine (expression compile + dep extraction) still TODO — split as F8b."},
{"id": "F9", "subject": "Follow-up: ScriptedAlarmActor engine wiring + state persistence", "status": "partial", "classification": "standard", "estMinutes": 30, "parallelizableWith": [], "blockedBy": [], "origin": "Self-review of Task 43 — AlarmConditionService not called; PreRestart persistence to ScriptedAlarmState DB not wired; HistorianAdapter rows not emitted.", "shipped": "(1) IScriptedAlarmEvaluator seam + NullScriptedAlarmEvaluator default. ScriptedAlarmActor takes AlarmConfig (id/name/path/severity/predicate), evaluates on DependencyValueChanged, publishes AlarmTransitionEvent + ScriptLogEntry on every transition. (2) IAlarmActorStateStore seam in Commons.Engines + NullAlarmActorStateStore default + EfAlarmActorStateStore production adapter over the ScriptedAlarmState entity. ScriptedAlarmActor PreStart loads + restores; every Transition fires a fire-and-forget save with lastAckUser. Predicate binding to Core.ScriptedAlarms.ScriptedAlarmEngine still TODO — split as F9b."},
{"id": "F10", "subject": "Follow-up: OpcUaPublishActor SDK integration (address-space writes + ServiceLevel + RebuildAddressSpace)", "status": "partial", "classification": "high-risk", "estMinutes": 60, "parallelizableWith": [], "blockedBy": [47], "origin": "Self-review of Task 44 — SDK calls stubbed; counters only. Wire after Phase 7 OpcUaServer extraction.", "shipped": "(1) IOpcUaAddressSpaceSink + IServiceLevelPublisher seams in Commons.OpcUa with Null* defaults. OpcUaPublishActor routes through the sink, dedupes ServiceLevelChanged, subscribes to redundancy-state DPS topic, maps redundancy snapshot to a coarse ServiceLevel (Primary+leader=240, Primary=200, Secondary=100, Detached=0). (2) OtOpcUaNodeManager (CustomNodeManager2) + OtOpcUaSdkServer (StandardServer subclass) + SdkAddressSpaceSink in OpcUaServer — lazy variable creation on first WriteValue, WriteAlarmState shape, RebuildAddressSpace tear-down. Variable updates propagate via ClearChangeMasks so subscribed OPC UA clients see them. Tests boot a real StandardServer + verify sink writes show up in the manager. Production wiring through OpcUaApplicationHost.StartAsync (default server = OtOpcUaSdkServer) + IServiceLevelPublisher SDK binding + #109 OpcUaPublishActor→Phase7Applier integration are the remaining pieces."},
{"id": "F8", "subject": "Follow-up: VirtualTagActor engine wiring (compile expression, subscribe deps, publish result)", "status": "completed", "supersededNote": "Marked completed 2026-08-07 (truth sweep): superseded by the v3 address-space rework — the production IVirtualTagEvaluator (Host/Engines/RoslynVirtualTagEvaluator.cs) is registered on driver nodes and hardened (archreview R2-03), so real expression evaluation is wired; the F8b residual no longer applies.", "classification": "standard", "estMinutes": 30, "parallelizableWith": [], "blockedBy": [], "origin": "Self-review of Task 42 — VirtualTagEngine.Evaluate not called; DependencyValueChanged just buffers.", "shipped": "(1) IVirtualTagEvaluator seam + NullVirtualTagEvaluator default. VirtualTagActor calls evaluator on DependencyValueChanged, dedupes unchanged results, emits EvaluationResult to parent, publishes Warning ScriptLogEntry on failure. (2) DependencyMuxActor in Runtime fans out DriverInstanceActor.AttributeValuePublished from DriverHostActor through to interested VirtualTagActor subscribers. VirtualTagActor takes dependencyRefs + mux ActorRef in Props, registers interest in PreStart, unregisters in PostStop. WithOtOpcUaRuntimeActors spawns the mux + threads it into DriverHostActor. Production binding to Core.VirtualTags.VirtualTagEngine (expression compile + dep extraction) still TODO — split as F8b."},
{"id": "F9", "subject": "Follow-up: ScriptedAlarmActor engine wiring + state persistence", "status": "completed", "supersededNote": "Marked completed 2026-08-07 (truth sweep): superseded by the v3 address-space rework — ScriptedAlarmActor is retired (see ScriptedAlarmHostActor's own doc-comment); ScriptedAlarmHostActor carries the engine wiring and Part 9 condition state persists via LocalDbAlarmConditionStateStore (mesh Phase 4).", "classification": "standard", "estMinutes": 30, "parallelizableWith": [], "blockedBy": [], "origin": "Self-review of Task 43 — AlarmConditionService not called; PreRestart persistence to ScriptedAlarmState DB not wired; HistorianAdapter rows not emitted.", "shipped": "(1) IScriptedAlarmEvaluator seam + NullScriptedAlarmEvaluator default. ScriptedAlarmActor takes AlarmConfig (id/name/path/severity/predicate), evaluates on DependencyValueChanged, publishes AlarmTransitionEvent + ScriptLogEntry on every transition. (2) IAlarmActorStateStore seam in Commons.Engines + NullAlarmActorStateStore default + EfAlarmActorStateStore production adapter over the ScriptedAlarmState entity. ScriptedAlarmActor PreStart loads + restores; every Transition fires a fire-and-forget save with lastAckUser. Predicate binding to Core.ScriptedAlarms.ScriptedAlarmEngine still TODO — split as F9b."},
{"id": "F10", "subject": "Follow-up: OpcUaPublishActor SDK integration (address-space writes + ServiceLevel + RebuildAddressSpace)", "status": "completed", "supersededNote": "Marked completed 2026-08-07 (truth sweep): superseded by the v3 address-space rework — OpcUaPublishActor routes through the production SdkAddressSpaceSink + SdkServiceLevelPublisher in ZB.MOM.WW.OtOpcUa.OpcUaServer, with surgical apply via AddressSpaceApplier/AddressSpaceChangeClassifier (archreview R2-07); OpcUaApplicationHost is production-instantiated.", "classification": "high-risk", "estMinutes": 60, "parallelizableWith": [], "blockedBy": [47], "origin": "Self-review of Task 44 — SDK calls stubbed; counters only. Wire after Phase 7 OpcUaServer extraction.", "shipped": "(1) IOpcUaAddressSpaceSink + IServiceLevelPublisher seams in Commons.OpcUa with Null* defaults. OpcUaPublishActor routes through the sink, dedupes ServiceLevelChanged, subscribes to redundancy-state DPS topic, maps redundancy snapshot to a coarse ServiceLevel (Primary+leader=240, Primary=200, Secondary=100, Detached=0). (2) OtOpcUaNodeManager (CustomNodeManager2) + OtOpcUaSdkServer (StandardServer subclass) + SdkAddressSpaceSink in OpcUaServer — lazy variable creation on first WriteValue, WriteAlarmState shape, RebuildAddressSpace tear-down. Variable updates propagate via ClearChangeMasks so subscribed OPC UA clients see them. Tests boot a real StandardServer + verify sink writes show up in the manager. Production wiring through OpcUaApplicationHost.StartAsync (default server = OtOpcUaSdkServer) + IServiceLevelPublisher SDK binding + #109 OpcUaPublishActor→Phase7Applier integration are the remaining pieces."},
{"id": "F11", "subject": "Follow-up: HistorianAdapterActor named-pipe IPC + SqliteStoreAndForwardSink wiring", "status": "completed", "classification": "standard", "estMinutes": 30, "parallelizableWith": [], "blockedBy": [], "commit": "6861381", "deviationNotes": "Reshaped HistorianAdapterActor around the existing IAlarmHistorianSink abstraction (alarm-event shape, not the original tag-history-row stub). Defaults to NullAlarmHistorianSink; production deployments wire SqliteStoreAndForwardSink + WonderwareHistorianClient via AddOtOpcUaRuntime overrides. Actor now exposes GetStatus returning HistorianSinkStatus for diagnostics. Named-pipe transport implementation lives unchanged in src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware.Client/WonderwareHistorianClient.cs — the actor is intentionally just a fire-and-forget bridge.", "origin": "Self-review of Task 45 — stub buffers in-memory; named-pipe + SQLite store-and-forward not wired."},
{"id": "F12", "subject": "Follow-up: PeerOpcUaProbeActor real opc.tcp ping (replace Ok=true stub)", "status": "completed", "classification": "small", "estMinutes": 20, "parallelizableWith": [], "blockedBy": [], "commit": "b06e3ae", "deviation": "TCP-connect probe rather than full OPC UA Hello/Acknowledge handshake. Enough for the redundancy calc; deeper liveness signals can layer on later without changing the actor's contract.", "origin": "Self-review of Task 45 — RunProbe always returns Ok=true; replace with OPC UA Client connect."},
{"id": "F13", "subject": "Follow-up: Full OpcUaApplicationHost extraction (security/alarms/history/observability)", "status": "partial", "classification": "high-risk", "estMinutes": 120, "parallelizableWith": [], "blockedBy": [], "commit": "36c4751-partial", "deviationNotes": "F13a (cert auto-creation) shipped in 36c4751. Remaining: endpoint-security wiring (SecurityProfileResolver into ServerConfiguration.SecurityPolicies), LDAP user-token validator (the OPC UA UserNameToken path; HTTP-layer LDAP auth is separate and already in OtOpcUa.Security), scripted-alarm node manager creation, history backend wiring, observability hooks (OpenTelemetry metrics + traces). These are gated by F10's OpcUaPublishActor SDK integration — until F10 lands, nothing instantiates OpcUaApplicationHost so the missing wiring is dead weight.", "origin": "Self-review of Task 46 — facade only boots ApplicationInstance + StandardServer. Legacy 391-line file pulls Server.Security/Alarms/History/Observability. Pull those into thin OpcUaServer interfaces."},
{"id": "F14", "subject": "Follow-up: Migrate side-effecting Phase7Composer (EquipmentNodeWalker, trace logs, node cache)", "status": "partial", "classification": "standard", "estMinutes": 60, "parallelizableWith": [], "blockedBy": [], "origin": "Self-review of Task 47 — pure version covers the projection. Walker + alarm sink registration + cache mutation stay in legacy until split into Phase7Applier.", "shipped": "Phase7Plan + Phase7Planner.Compute (pure diff over EquipmentNodes/DriverInstancePlans/ScriptedAlarmPlans by stable id, with Added/Removed/Changed lists). Phase7Applier consumes plan + IOpcUaAddressSpaceSink: drives RebuildAddressSpace on Equipment/Alarm topology change, writes inactive AlarmState for removed nodes, catches + logs sink faults. Driver-only changes correctly skip the rebuild (DriverHostActor's spawn-plan in Runtime handles those). Walker integration with the real SDK NodeManager is the remaining piece — split as F14b (consumes the existing EquipmentNodeWalker once F10b lands an SDK builder)."},
{"id": "F13", "subject": "Follow-up: Full OpcUaApplicationHost extraction (security/alarms/history/observability)", "status": "completed", "supersededNote": "Marked completed 2026-08-07 (truth sweep): superseded by the v3 address-space rework + #520 — OpcUaApplicationHost wires security profiles into ServerConfiguration.SecurityPolicies (BuildSecurityPolicies/BuildUserTokenPolicies; the SecurityProfileResolver type no longer exists in src/); the OPC UA user-token path shipped and was then replaced by API-key UserName auth (#520; LdapOpcUaUserAuthenticator deleted); scripted alarms, history backend (HistorianGateway) and OTel observability all production-wired.", "classification": "high-risk", "estMinutes": 120, "parallelizableWith": [], "blockedBy": [], "commit": "36c4751-partial", "deviationNotes": "F13a (cert auto-creation) shipped in 36c4751. Remaining: endpoint-security wiring (SecurityProfileResolver into ServerConfiguration.SecurityPolicies), LDAP user-token validator (the OPC UA UserNameToken path; HTTP-layer LDAP auth is separate and already in OtOpcUa.Security), scripted-alarm node manager creation, history backend wiring, observability hooks (OpenTelemetry metrics + traces). These are gated by F10's OpcUaPublishActor SDK integration — until F10 lands, nothing instantiates OpcUaApplicationHost so the missing wiring is dead weight.", "origin": "Self-review of Task 46 — facade only boots ApplicationInstance + StandardServer. Legacy 391-line file pulls Server.Security/Alarms/History/Observability. Pull those into thin OpcUaServer interfaces."},
{"id": "F14", "subject": "Follow-up: Migrate side-effecting Phase7Composer (EquipmentNodeWalker, trace logs, node cache)", "status": "completed", "supersededNote": "Marked completed 2026-08-07 (truth sweep): superseded by the v3 address-space rework — Phase7Composer/Phase7Planner/Phase7Applier no longer exist in src/; replaced by AddressSpaceComposer + AddressSpaceApplier in ZB.MOM.WW.OtOpcUa.OpcUaServer.", "classification": "standard", "estMinutes": 60, "parallelizableWith": [], "blockedBy": [], "origin": "Self-review of Task 47 — pure version covers the projection. Walker + alarm sink registration + cache mutation stay in legacy until split into Phase7Applier.", "shipped": "Phase7Plan + Phase7Planner.Compute (pure diff over EquipmentNodes/DriverInstancePlans/ScriptedAlarmPlans by stable id, with Added/Removed/Changed lists). Phase7Applier consumes plan + IOpcUaAddressSpaceSink: drives RebuildAddressSpace on Equipment/Alarm topology change, writes inactive AlarmState for removed nodes, catches + logs sink faults. Driver-only changes correctly skip the rebuild (DriverHostActor's spawn-plan in Runtime handles those). Walker integration with the real SDK NodeManager is the remaining piece — split as F14b (consumes the existing EquipmentNodeWalker once F10b lands an SDK builder)."},
{"id": "F15", "subject": "Follow-up: Migrate 47 legacy Admin Blazor components into AdminUI library", "status": "completed", "classification": "high-risk", "estMinutes": 180, "commit": "Phase A-D (read views) + F15.2 batches 1-4 (live-edit CRUD) + F15.3 (live alerts/script-log/CSV import/Monaco)", "deviationNotes": "All 4 phases of read-only views shipped: Phase A (shell/auth/fleet/hosts), B (cluster CRUD + Overview/Redundancy), C (Equipment/UNS/Namespaces/Drivers/Tags/ACLs), D (Audit/VirtualTags/ScriptedAlarms/Scripts/RoleGrants/Certificates/Reservations/AlarmsHistorian). Per Q1–Q5 of docs/v2/AdminUI-rebuild-plan.md: typed driver editors deferred, top-level VirtualTags/ScriptedAlarms kept (Q2 reversed for cross-cluster discoverability), routes-not-tabs adopted, fleet-wide LDAP→role map only, generic login errors. Live-edit forms (F15.2) and ScriptLog page (depends on F16 ScriptLogHub) are explicit follow-ups.", "parallelizableWith": [], "blockedBy": [], "origin": "Self-review of Task 48 — only MapAdminUI scaffold + 1 new page (Deployments). 47 pages stay in legacy Admin (accepted-broken) until Task 56."},
{"id": "F16", "subject": "Follow-up: Bridge FleetStatusBroadcaster → SignalR hubs (FleetStatusHub / AlertHub / ScriptLogHub)", "status": "completed", "classification": "standard", "estMinutes": 30, "parallelizableWith": [], "blockedBy": [], "commit": "f18c285", "deviation": "FleetStatusHub bridge landed. AlertHub + ScriptLogHub deferred — they need upstream message contracts that aren't defined yet (alerts emerge from F9 ScriptedAlarmActor, script logs from F8 VirtualTagActor).", "origin": "Self-review of Task 49 — hubs are passive Hub subclasses; the bridge from FleetStatusBroadcaster.broadcast → IHubContext is not wired."},
{"id": "F17", "subject": "Follow-up: FleetDiagnosticsClient real Akka ActorSelection round-trip (GetDiagnosticsRequest)", "status": "completed", "classification": "standard", "estMinutes": 30, "parallelizableWith": [], "blockedBy": [], "commit": "8f32b89", "origin": "Self-review of Task 51 — client returns an empty snapshot stub. Add GetDiagnosticsRequest contract + DriverHostActor handler + real Ask/Reply."},
@@ -6,7 +6,11 @@
**Skill in use:** `superpowers-extended-cc:executing-plans` — batch, verify, commit per task, report between batches.
**Do NOT merge to master** — plan stops at Task 15 (DoD) and reports; Task 16 is a rig live-gate needing explicit user go-ahead.
## Status: Tasks 0–9 DONE (10 commits). Tasks 10–16 REMAIN.
## Status: ~~Tasks 0–9 DONE (10 commits). Tasks 10–16 REMAIN.~~ SUPERSEDED — Phase 1 COMPLETED
**Update 2026-08-07 (truth sweep):** Phase 1 completed in full, live-gated **8/8** on the docker-dev
rig, and merged to `master` as `c957db52` (`Merge feat/localdb-phase1`). The resume state below is
historical.
| # | Task | State | Commit |
|---|---|---|---|
@@ -123,6 +123,13 @@ state. Re-run under the control: red, for exactly the cases that matter. This is
## The open design question
> **Update 2026-08-07 (truth sweep): resolved — recommendation (C) shipped as mesh Phase 6**
> (merged 2026-07-24, tip `2839deb5`): `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped
> singleton, so the election is per application `Cluster` exactly as recommended below. What honestly
> remains open: gate checks 3 and 4 (only-the-primary-drains; failover double-delivery count) were
> **never re-run** on the post-Phase-6 three-mesh topology, so the drain gate's intent is still
> unproven live.
The gate leaves one question that is a design fork, not a bug to patch:
**A pair cannot currently identify its own Primary.** Redundancy roles are elected cluster-wide, but
@@ -178,3 +185,6 @@ reclaiming the de-duplication benefit, not an outstanding hazard.
## Not merged
Per the plan, nothing on this branch is merged.
**Update 2026-08-07 (truth sweep):** since merged — `06b5d81b` (`Merge feat/localdb-phase2: alarm
store-and-forward on LocalDb + live-gate fixes`).
@@ -40,6 +40,11 @@ The pause measured **<1s** on both runs, well inside `acceptable-heartbeat-pause
## Finding — the node records `Applied` for a deployment it skipped (pre-existing, NOT fixed here)
> **Update 2026-08-07 (truth sweep): fixed as issue #486 — see the Addendum below.** In today's
> source the fix lives at `Runtime/Drivers/DriverHostActor.cs:1696-1717` (null blob ⇒ status
> `Failed`, ack `Failed`, revision NOT advanced; the comment cites #486), mirrored on the
> FetchAndCache path (~`:1609-1653`, "artifact could not be read (#486) and keeps last-known-good").
Both guards leave the node running its previous configuration, which is the intent. But `ApplyAndAck`
advances `_currentRevision = revision` and writes `NodeDeploymentState = Applied` **before** it knows
whether anything was applied — that assignment sits immediately after `ReconcileDrivers`, which returns
@@ -1,7 +1,9 @@
# Per-Cluster Akka Mesh — design
> **Status:** design for review. **Phases 0a and 0b are DONE and merged** (2026-07-21); Phases 1–7
> are not started and each needs its own plan.
> **Status:** **PROGRAM COMPLETE 2026-07-24** — all phases (0a/0b/1/2/3/4/5/6/7) done, live-gated
> and merged to `master`; Phase 7 failover drills 6/6 PASSED @ `c50ebcf7`. See the tracking table in
> `2026-07-22-per-cluster-mesh-program.md`. *(Was: "Phases 1–7 are not started" — corrected
> 2026-08-07, truth sweep.)*
> **Decisions settled 2026-07-21** — see §6: one central DB with driver nodes disconnected from it,
> the two-node keep-oldest gap closed upstream and now closed here too (§6.2), and their auth posture
> matched for now.
@@ -197,6 +197,11 @@ hosts it. It was orphaned when v3 Batch 2 retired the routed `/clusters/{id}/dri
So the `driver-control` channel has no operator surface **on either transport** — this is a
pre-existing v3 regression that Phase 2 merely surfaced. Filed as a follow-up; not fixed here.
**Update 2026-08-07 (truth sweep): since fixed as Gitea #546** (merge `9420f8ef`) —
`DriverStatusPanel` is now rendered by the `/hosts` page's expandable driver rows
(`AdminUI/Components/Pages/Hosts.razor` ~line 297), so the Reconnect/Restart operator surface exists
again.
**Residual risk for the channel:** central-side dispatch is sabotage-verified by
`MeshRouterDispatchTests` (all four `AdminOperationsActor` call sites go through one `Dispatch`
helper); node-side `EventStream` receipt by `DriverHostActor` is proven **live** by this gate, since
@@ -1,6 +1,12 @@
# API-key identity + node ACL enforcement (Gitea #520)
**Status:** design agreed 2026-07-31, not yet implemented.
**Status:** implemented through stage 7 — **#520 CLOSED 2026-08-01**. Stage merges: `fbe7403c`
(stages 1-2, 4: API-key identity + keys/ACLs via the artifact), `3a6b2a1d` (stage 3: variable-depth
scope model + scopePaths producer), `1ccc7400`/`f692f6af` (stage 5: enforcement on write, alarms,
HistoryRead), `d4389f57`/`00aa4866` (stage 6: enforcement on read, browse, subscribe), `3796c0bf`
(stage 7: ACL scope picker + authoring-time resolution). Enforcement is opt-in via
`Security:Acl:Enabled` (default `false`). *(Was: "design agreed 2026-07-31, not yet implemented" —
corrected 2026-08-07, truth sweep.)*
**Supersedes:** `docs/v2/acl-design.md` (LDAP-group-keyed model, never wired).
## Why this exists
@@ -208,6 +208,16 @@ an id, so partial acking could drop older unwritten entries) — but the consequ
is a real operational hazard, since `Capacity` defaults to `0` (unbounded). Filed, not fixed: the fix
is a per-tag or non-prefix ack, which changes the outbox contract.
> **Update 2026-08-07 (truth sweep): filed as Gitea #538 and FIXED the same day** (`e7fbe977`, gate
> record `4bdce910` — PASSED live). The drain now tracks consecutive failures per historian tag and
> abandons a tag's entries once past `MaxWriteAttempts` (default 5) so the FIFO head advances; the
> counter only advances on a pass where some *other* tag wrote successfully, which is what separates
> "this tag is poison" from "the backend is down" (a plain count would discard the backlog during an
> ordinary gateway restart). Abandonments log a Warning and ride `RecorderStatus.AbandonedEntries`.
> The same fix also gave `Capacity` a non-zero default (500,000 entries) as the disk backstop for the
> one case the quarantine deliberately cannot resolve (a historian refusing every write); `0`
> restores unbounded retention.
## Environment notes from the 2026-08-01 run (superseded for read-back, still accurate)
**The read-back hop.** Neither historian available today can serve it:
+4 -1
View File
@@ -9,7 +9,10 @@ ADO.NET providers (Npgsql, Oracle.ManagedDataAccess.Core).
**Architecture:** See `docs/plans/2026-08-05-sql-postgres-oracle-design.md` (this branch). Two new
dialects in the existing `.Dialect` project; dialect-owned parameter markers/bind-names +
`ConfigureCommand` (Oracle BindByName); provider gates widened; AdminUI dropdown; env-gated docker
integration suites. Chrome/AdminUI live gate is DEFERRED to the operator.
integration suites. ~~Chrome/AdminUI live gate is DEFERRED to the operator.~~ **Update 2026-08-07
(truth sweep): the chrome/AdminUI gate RAN and PASSED 2026-08-05** — and found a real defect (Test
Connect was SqlServer-gated for both new providers), fixed in merge `4e461951`; see
`2026-08-05-sql-postgres-oracle-design.md` §4.3 record.
**Tech Stack:** .NET 10, central package management (`Directory.Packages.props`), xUnit v3 +
Shouldly, SQLite falsifiability control.
@@ -135,7 +135,7 @@ Defects found during execution (live-gate or review), with root cause + pinned t
| 1 | 1B.2 | **Failover silently defeated on a hanging central** (CRITICAL, caught by Opus code review): the ack deadline was one absolute `DateTime` shared across the endpoint walk — a hung (not refused) central-1 burned the whole budget and central-2 was dialed with an expired deadline | deadline hoisted outside the `foreach`; a kill-style drill can't see it (refused = fast-fail), only a hang discriminates | `e54711a2`; pinned by `Each_endpoint_gets_a_FRESH_deadline` (mutation-verified); 1B.3 gains a `docker pause` hang drill |
| 2 | 1A.2 | **Dark-switch rollback deactivated the known-cluster gate** (Opus review, I-2): gate published only under `Mode=Grpc` while the interceptor + service are port-mapped in every mode — a rollback to ClusterClient/Dps would silently restore the timing oracle + pre-auth flood | gate activation coupled to transport mode instead of the endpoint it protects | `48dc2120` — gate follows the endpoint; pinned by the every-mode Theory + Dps guard pin `86633f14` |
| 5 | 1A.3 (live gate, run 2) | **The `PreStart` resolve loses its race on real boots** — 2 of 3 observed admin boots logged `Failed to resolve IMeshCommandDialer` / `IMeshPskProvider`, same `AuditWriterActorKey` cause. NOT a defect: it is exactly the case the review-prescribed hardening was written for | `PreStart` runs on a dispatcher thread concurrently with the builder's remaining starters, so it can still precede starter #4 | **Closed by `e54883a8`, and the heal is PROVEN LIVE** — the coordinator singleton was moved onto the node that lost the race (stop central-2) and it fanned a dispatch to all five live nodes with no dialer error. Residual is cosmetic (Error text reads permanent; will trip a soak error-grep on a healthy node) → Phase 2 |
| 4 | 1A.3 (live gate, run 1) | **An admin node cannot boot** (BLOCKED 1A): both central nodes die at startup with `MissingActorRegistryEntryException: ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey`, taking `AkkaHostedService` and the whole host with them. Mode-INDEPENDENT — reproduced on the same image with `MeshTransport__Mode=ClusterClient`, so it is not the gate's flip. Driver-only nodes are unaffected (they register neither `IMeshPskProvider` nor `ActorAuditWriter`) | `WithOtOpcUaControlPlaneSingletons`'s first `WithActors` starter eagerly resolves `IMeshCommandDialer` + `IMeshPskProvider` to pass into `CentralCommunicationActor.Props`. That graph is dialer → `IMeshPskProvider` → `ISecretResolver` → (`AddZbSecrets`) → `IAuditWriter` → `ActorAuditWriter`, whose ctor calls `registry.Get<AuditWriterActorKey>()` — and `WithSingleton<AuditWriterActorKey>` is registered ~60 lines later in the same builder. Same class as scadaproj#1 and mesh Phase 6's `IClusterRoleInfo` crash: only visible with the DI graph built in a real host, which is why the container-activation test (no `AddZbSecrets` in its collection) and `TwoNodeClusterHarness` (no provider at all) both miss it. Aggravating: the failed process does not exit, so `docker ps` reports `Up restarts=0` while the node serves no HTTP | **OPEN.** Fix + red-first pin belong to 1A (the gate made no product change). Suggested shape: hand the actor lazy accessors, mirroring the `Func<>` the same block already uses for the coordinator proxy. Pin must build the graph in starter order (a Host.IntegrationTests boot of an admin-role host), not merely activate the dialer |
| 4 | 1A.3 (live gate, run 1) | **An admin node cannot boot** (BLOCKED 1A): both central nodes die at startup with `MissingActorRegistryEntryException: ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey`, taking `AkkaHostedService` and the whole host with them. Mode-INDEPENDENT — reproduced on the same image with `MeshTransport__Mode=ClusterClient`, so it is not the gate's flip. Driver-only nodes are unaffected (they register neither `IMeshPskProvider` nor `ActorAuditWriter`) | `WithOtOpcUaControlPlaneSingletons`'s first `WithActors` starter eagerly resolves `IMeshCommandDialer` + `IMeshPskProvider` to pass into `CentralCommunicationActor.Props`. That graph is dialer → `IMeshPskProvider` → `ISecretResolver` → (`AddZbSecrets`) → `IAuditWriter` → `ActorAuditWriter`, whose ctor calls `registry.Get<AuditWriterActorKey>()` — and `WithSingleton<AuditWriterActorKey>` is registered ~60 lines later in the same builder. Same class as scadaproj#1 and mesh Phase 6's `IClusterRoleInfo` crash: only visible with the DI graph built in a real host, which is why the container-activation test (no `AddZbSecrets` in its collection) and `TwoNodeClusterHarness` (no provider at all) both miss it. Aggravating: the failed process does not exit, so `docker ps` reports `Up restarts=0` while the node serves no HTTP | **FIXED** — closed in 1A by `c0475f2d` (lazy resolve of dialer + PSK provider — central boots again) + `e54883a8` (memoize on first use; a lost `PreStart` race self-heals at the first refresh, review-prescribed), merged in Phase 1A @ `b360239e`; gate re-run PASSED 8/8 (matches the Phase 1A board row above). Original suggested shape: hand the actor lazy accessors, mirroring the `Func<>` the same block already uses for the coordinator proxy |
| 3 | 1A.2 | **Fan parked the actor mailbox** (self-caught by the implementer): dialing from the receive ran `TryGetPsk`'s cold-cache 3 s block pre-await (cold after EVERY refresh due to `InvalidateAll`), stalling deployment acks behind the fan | async methods run synchronously to the first await | `Task.Run` hop (token None — a canceled scheduling would read as a DB failure via PipeTo); pinned red-first by `A_slow_dial_does_not_park_the_actors_mailbox` |
## Carry-forward register
@@ -240,7 +240,8 @@
"branch": "(master)",
"classification": "high-risk",
"parallelizableWith": [],
"status": "pending",
"status": "completed",
"completionNotes": "Gate PASSED 8/8 on master @ 0b194003 (provenance-guarded post-deletion image); record d0ac6fc5 — ClusterClient-to-gRPC program CLOSED. Status corrected from pending 2026-08-07 (truth sweep).",
"blockedBy": [
14
]
@@ -13,6 +13,12 @@
## 1. FANUC FOCAS — Live CNC Smoke (task #54)
> **Update 2026-08-07 (truth sweep): this smoke has RUN — RESOLVED + live-validated on a real 31i-B
> (`10.201.31.5`) on 2026-06-25.** Version gate, timer, PMC range, servo-meter, alarms and probe all
> fixed and verified live; **caveat preserved: `cnc_rdparam` was found unsupported on that control.**
> Full record: `docs/plans/2026-06-25-focas-pdu-v3-30i-b-support.md` (+ its implementation plan).
> The runbook below remains valid for re-runs on other series.
### Background
The FOCAS driver (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.FOCAS/`) uses the
+1 -1
View File
@@ -19,7 +19,7 @@ From `docs/v2/v2-release-readiness.md` §"Release-readiness exit criteria":
| G4 | Phase 5 driver complement shipped | Audit | Already closed |
| G5 | Production deployment checklist signed off by Fleet Admin | Operator | No — separate doc, human signoff |
| G6 | At least one end-to-end integration run against live Galaxy succeeds | Dev rig | No — requires AVEVA platform |
| G7 | FOCAS live-CNC wire-level smoke (#54) passes against a real FANUC control | Lab hardware | No — requires FANUC CNC |
| G7 | FOCAS live-CNC wire-level smoke (#54) passes against a real FANUC control | Lab hardware | ✅ **RESOLVED 2026-06-25** *(truth-sweep note 2026-08-07)* — live-validated on a real 31i-B (`10.201.31.5`): version gate, timer, PMC range, servo-meter, alarms, probe all verified live; caveat: `cnc_rdparam` found unsupported on that control. See `2026-06-25-focas-pdu-v3-30i-b-support.md` |
| G8 | OPC UA CTT / UA Compliance Test Tool passes against the live endpoint | Operator tool | No — requires CTT binary + live endpoint |
| G9 | Non-transparent redundancy cutover validated with >= 1 production client | Lab | No — see `docs/plans/phase-6-3-redundancy-interop-plan.md` |
+1 -1
View File
@@ -108,7 +108,7 @@ Decision recorded: FOCAS is **read-only** against the CNC by design — writes r
The code ships; these tasks remain open as lab/field verification:
- **#54** — FOCAS live-CNC wire-level smoke against a real FANUC control. The mock's wire responder is PDU-verified against `fwlibe64.dll` upstream but OtOpcUa's managed client has not been pointed at a production CNC.
- **#54** — FOCAS live-CNC wire-level smoke against a real FANUC control. ~~The mock's wire responder is PDU-verified against `fwlibe64.dll` upstream but OtOpcUa's managed client has not been pointed at a production CNC.~~ **Update 2026-08-07 (truth sweep): RESOLVED — live-validated on a real 31i-B (`10.201.31.5`) on 2026-06-25** (version gate, timer, PMC range, servo-meter, alarms, probe all verified live; `cnc_rdparam` found unsupported on that control). See `docs/plans/2026-06-25-focas-pdu-v3-30i-b-support.md`.
- **AB CIP live-boot** — already passed on a ControlLogix rig (PR #222). Continue to run ahead of each release.
- **TwinCAT wire-live** — TCBSD/ESXi fixture covers the common path; production PLC verification remains lab-gated.