docs: perf roadmap remainder — outcomes + follow-up register

docs/plans/2026-08-15-perf-roadmap-remainder.md

@@ -1264,3 +1264,152 @@ Wave 4: 28 (←all)
 apply-path changes (run the existing live-gate recipes afterwards, as items 1–2 did);
 the HistorianGateway contract change; any change to `redundancy-state`, the deploy
 classifier, or the ACL trie itself.
+
+---
+
+## Outcomes
+
+All 28 tasks landed on `perf/roadmap-remainder`. Close-out sweep (2026-08-16):
+`dotnet build ZB.MOM.WW.OtOpcUa.slnx` — **0 errors**, 896 warnings (all pre-existing:
+`OTOPCUA0001` on deliberate unwrapped driver calls in tests, `xUnit1051`, two `CS8602`);
+full unit sweep across all 43 `*.Tests` projects — **10 310 passed, 0 failed, 19 skipped**.
+`*.IntegrationTests` are build-only by repo convention, and the DB-backed suites
+(`*.Configuration.Tests`, `*.Admin.Tests`, `*.Server.Tests`) ran for real against the
+reachable central SQL Server at `10.100.0.35,14330` — nothing was skipped for want of a
+database. The three known load-related flakes
+(`DriverInstanceActorCommsLossWatchdogTests`, `RawTreeServiceRenameWarningTests`,
+`RouteNativeAlarmAck_on_non_primary_is_dropped`) all passed on the first run; no project
+needed an isolated re-run.
+
+Note for anyone re-running the sweep by globbing `tests/*/*.Tests`: two directories under
+`tests/Drivers/` — `…Driver.Historian.Wonderware.Tests` and
+`…Driver.Historian.Wonderware.Client.Tests` — are **empty leftovers** of the retired
+Wonderware backend, holding only stale `bin`/`obj` and no `.csproj`. They are not in the
+solution. A glob-driven sweep reports `MSB1003` for them; that is the glob, not a failure.
+
+Every task's spec review and code review was run and closed before the task was marked
+done; what follows records only the outcome and any deviation from the plan as written.
+
+| # | Task | Commit(s) | Outcome / deviation |
+|---|---|---|---|
+| 1 | Outbox batch append with group commit | `6ff64ca0` + `73205865` | Done. The follow-up commit made the batch bookkeeping incremental and replaced the durability test with a genuine group-commit pin (a second reader). Deviation: the second-reader durability test is fragile on macOS — see the register. |
+| 2 | Batched mux→subscriber delivery, single-eval virtual tags | **inside `140bf0c1`** | Done. ⚠️ Its 7 Runtime files were swept into Task 22's commit — see the mixed-commit note below. Both code review Importants (a yielding batch-test fake, mid-batch exception policy) were folded into Task 4's brief and closed there. |
+| 3 | Parallel drain with bounded concurrency | `a76b4e07` | Done, approved as specified (`DrainConcurrency`, default 8). One inherited pre-existing defect recorded in the register (shutdown cancellation counted as a per-tag failure). |
+| 4 | Recorder batched ingest — one fsync per burst | `384568b4` + `451273c8` | Done. The follow-up pinned the append-failure branch via three independent mutations and corrected the loss-accounting docs; the recorder's `64` literals were aligned to `1024` there. |
+| 5 | Historian defaults, config drift, docs | `0298a0b9` | Done. Added the `Historian.md` key table and recorded the cross-repo batch-RPC ceiling. Minor: the pinning test's doc comment overclaims appsettings-drift coverage. |
+| 6 | Per-session decision-cache shards | `debd44c2` | Done. Replaces the flat cache + wholesale `Clear` with per-session shards. Register: `_globalCount` drift under races; no concurrency stress test. |
+| 7 | Authorizer hot path — lookup first, lazy deny reasons | `d8a17f2e` | Done. Cache lookup moved ahead of scope construction; deny reasons are built only on the deny path; revocation/expiry stay ahead of the cache. |
+| 8 | Browse and TranslateBrowsePath compaction | `541a223b` + `aad7583b` | Done — O(n) in-place compaction. The follow-up test pins survivor **order** through the compaction (falsified red first). Minor: the `FanOutTargetIdList` fake couples to the SDK's single-`Add` contract. |
+| 9 | RevalidateMonitoredItems dedup | `fbe8aaad` | Done. Deviation from spec, deliberately: the dedup key is a **realm-qualified 3-tuple** `(session, node, realm)` rather than the spec's 2-tuple — the realm is carried explicitly at every sink seam and dropping it could collapse two distinct nodes. Register: the empty-identifier pattern is worth tightening. |
+| 10 | Modbus `InitializeAsync` transport leak | `c3970a2e` + `409a58b6` | Done. The follow-up added the lifecycle gate and replaced a vacuous reprobe test with a non-vacuous one. |
+| 11 | PollGroupEngine fixed-rate pacing, overrun counter, async unsubscribe | `9cf400f1` + `485f4ea1` | Done. The follow-up widened the fast-reader control's tolerance to one scheduler blip (200 ms interval, ≤1). Register: `NoteOverrun` conflates slow-success with slow-failure; `backoffCap` needs a constructor doc note. |
+| 12 | S7 narrow-scalar span codec | `8d91833f` | Done. **Plan error corrected:** the plan's spec mentioned `SByte`; S7 has no such scalar in this codec's domain, so it was not implemented. The omission is intentional, not a gap. |
+| 13 | S7 span planner + grouped `ReadAsync` | `39f67a0f` | Done. Spec review accepted the demotion rule and the 6th (factory-binding) file beyond the plan's file list. ⚠️ The `16` / `220` span defaults are **not live-gated** — see the register. |
+| 14 | AbCip bounded-parallel reads | `d6a4fbd6` + `d5f573b0` | Done (`MaxConcurrentReads`, default 8). The follow-up added a per-device `ConnectGate` single-flight so cold-connect backoff survives the parallel read window. Minor: a stale `EvictRuntime`-race comment (~L1161); test timing margin is tight on slow CI. |
+| 15 | Dirty-set coalescing in `ScriptedAlarmEngine` | `5df793e7` | Done. Write-behind persistence was **considered and rejected** — see `ScriptedAlarms.md`. Register: `TrackBackgroundTask` registration-ordering gap (pre-existing), non-volatile `_disposed`. |
+| 16 | Deployments list projection | `dbc9a132` | Done — `/deployments` no longer materialises artifact blobs to render a list. |
+| 17 | Drift watermark probe | `684e59d8` | Done. **Design deviation, accepted:** the probe is a `GroupBy(_ => 1)` single-round-trip aggregate rather than the plan's separate count+max queries. Register: that aggregate is verified only against EF InMemory, so the perf claim is unverified against real SQL Server. |
+| 18 | ScriptTagCatalog TTL snapshot cache | `3795899d` | Done — 5 s snapshot cache. |
+| 19 | EffectiveNameGuard batch check + bulk add-references | `fa36d64e` | Done. Behaviour deviation recorded: for a **mixed-failure** batch the reported error is now the batch's first failure rather than the per-row one, a minor error-precedence shift. |
+| 20 | Raw tree tag-name projection | `ba977dbe` | Done — tag rows render from a projection that omits `TagConfig`. |
+| 21 | Rename script-scan filters in SQL | `fe8d132c` + `ca87329a` | Done. The follow-up added `Distinct` + 200-term chunking with `ScriptId`-keyed cross-batch dedup. Register: the `CHARINDEX`/collation widening needs a SQL-Server-backed test; the worst-case 250 sequential round-trips are undocumented. |
+| 22 | Pooled context factory + RoleGrants off the circuit-scoped context | `140bf0c1` | Done. ⚠️ This commit **also carries Task 2's files** — see the note below. |
+| 23 | Fleet/Hosts poll overlap guards | `4b6a93bb` + `63081d52` | Done. The follow-up gave the Fleet page the same async timer disposal the Hosts page has. |
+| 24 | MQTT unchanged-value suppression | `2c708f55` + `50035b36` | Done. The follow-up pinned the status-code half of the suppression (with falsification) and documented that Sparkplug bypasses it. |
+| 25 | `DriverHostActor` per-driver index + batched comms-loss fan | `b6275f43` | Done. Consequence for readers: after this, **no production code sends the per-value `AttributeValuePublished`** — the mux's per-value receive is test-only. Minor: cross-sender `RegisterInterest` ordering in one test is not structurally guaranteed; one empty-bucket guard is unreachable. |
+| 26 | Direct-apply artifact read off the mailbox | `5b92eb0f` | Done. A `BusLogging` comment dispute in review was resolved in the implementer's favour. Register: the Direct-mode SQL read carries no timeout/cancellation; `Bootstrap`'s orphan replay exits `PreStart` in `Applying`. |
+| 27 | MTConnect sample chunks parse from bytes | `34f836e0` | Done — no per-chunk string copies. |
+| 28 | Full sweep + plan close-out docs | this commit + the comment-only follow-up | Done. Sweep green; docs updated (`CLAUDE.md`, `security.md`, `DriverLifecycle.md`, `ScriptedAlarms.md`). |
+
+### The mixed commit `140bf0c1`
+
+Task 22's implementer ran a bare `git commit` (no pathspec) while Task 2's 7 Runtime files
+were already staged in the shared index. Both tasks' content therefore lives in
+**`140bf0c1`**, under Task 22's message
+(`perf(adminui): pooled DbContext factory; role-grants reads per-call, not per-circuit`),
+which names only Task 22's half.
+
+Both halves were fully spec- and code-reviewed at that commit — nothing is unreviewed, and
+nothing is missing from the branch. **Decision: the history stays as it is.** No empty
+marker commit was created; an empty commit would add a second misleading object rather than
+correct the first. This note is the record. If you are archaeologising Task 2 (batched
+mux→subscriber delivery, single-eval virtual tags), look in `140bf0c1`, not for a commit
+whose message mentions it.
+
+### Follow-up register
+
+Open items the work surfaced but deliberately did not close. None blocks the branch.
+
+**Cross-repo**
+
+- **HistorianGateway needs a multi-tag batch `WriteLiveValues` RPC.** Client `0.3.0` exposes
+  only a single-tag call, so bounded parallelism (Task 3's `DrainConcurrency`) is the
+  ceiling — the drain cannot become one round-trip per batch until the gateway contract
+  grows the RPC. Already recorded in [`docs/Historian.md`](../Historian.md).
+
+**Drivers**
+
+- **S7: multi-variable PDUs for span leftovers.** The Task 13 planner groups contiguous
+  reads into one PDU per span; tags that do not fall in a span are still one PDU each. A
+  multi-var PDU would absorb them.
+- ⚠️ **S7 span planner has no live gate.** The `16` (max gap) / `220` (max span bytes)
+  defaults were chosen analytically and pinned only by unit tests. Run the planner against
+  the `s7_1500` fixture (`lmxopcua-fix up s7 s7_1500`, `S7_SIM_ENDPOINT=10.100.0.35:1102`)
+  before trusting them on real hardware.
+- **`SpanReadMaxSpanBytes` has no validation guard.** A value above the negotiated PDU size
+  is accepted at config time and fails at read time.
+- **MTConnect `Build` could stream through `XmlReader`.** Task 27 removed the per-chunk
+  string copies; the document build still materialises.
+- **No driver-level meter for the PollGroupEngine overrun counter.** The counter exists and
+  is observable in-process but is not exported per driver.
+- **`NoteOverrun` conflates a slow success with a slow failure** — a poll that overran
+  because the device was slow and one that overran because it timed out increment the same
+  counter. Worth a doc caveat rather than a code change.
+- **`backoffCap` wants a constructor doc note** so a future caller does not have to read the
+  loop to learn its units and its interaction with the poll interval.
+
+**Runtime / deploy path**
+
+- **`UpsertNodeDeploymentState` and `TryRecoverFromStale` still write synchronously** on the
+  actor thread. Both are commented as such in `DriverHostActor`; neither is on the hot path
+  that Task 26 moved off the mailbox.
+- ⚠️ **Direct-mode `LoadArtifactBlob` carries no timeout or cancellation token** (Task 26
+  review, Important). The read now runs off the mailbox so it cannot wedge the actor, but a
+  hung SQL read still pins a thread-pool thread indefinitely unless the `ConfigDb`
+  connection string's command timeout bounds it. **Confirm that it does** — the code relies
+  on it and does not assert it.
+- **The historian drain's `catch` swallows a shutdown `OperationCanceledException` as a
+  per-tag failure** when other tags in the same pass succeeded, spuriously incrementing
+  `_tagWriteFailures` (Task 3 review; pre-existing, inherited). Rethrowing a genuine
+  `ct` cancellation is the fix.
+- **Host-side mux interest gating was skipped** — the batch envelope (Task 2) made the
+  saving it would have bought immaterial. Recorded so it is not re-proposed as an oversight.
+
+**ACL**
+
+- **Task 6: `_globalCount` can drift under concurrent shard eviction**, and there is no
+  concurrency stress test. Drift is bounded and only affects when trimming triggers, not
+  correctness of any decision.
+- **Task 9: the empty-identifier pattern should be tightened** — the dedup key tolerates an
+  empty node identifier rather than rejecting it.
+- **Task 17: the `GroupBy(_ => 1)` aggregate is unverified against real SQL Server.** It is
+  covered only by EF InMemory, which does not exercise the provider's translation, so the
+  single-round-trip claim is untested where it matters.
+- **Task 21: the `CHARINDEX`/collation widening needs a SQL-Server-backed test**, and the
+  worst-case **250 sequential round-trips** (a rename touching the chunking limit) should
+  carry a one-line comment at the call site.
+
+**Alarms**
+
+- **Task 15: `TrackBackgroundTask` has a registration-ordering gap** (pre-existing) — a task
+  can complete before it is registered. Also `_disposed` is non-volatile.
+
+**Test-suite reliability** (affects how reliably a regression is *caught*, not shipped behaviour)
+
+- **Task 1's second-reader durability test is fragile on macOS** — it depends on filesystem
+  flush visibility to a second reader, which the platform does not guarantee on the same
+  timescale as Windows/Linux.
+- **The #491 falsification control is nondeterministic under load.** It relies on a
+  `ThreadStatic` actor cell, so under a loaded sweep the intended-red mutation can
+  occasionally pass. The test is still worth keeping; do not read a single green run of it
+  as proof the regression is caught.