Files

T

Joseph Doherty 07fd17a34e docs(plan): perf follow-ups close-out — outcomes, live-gate evidence, register annotations

2026-08-17 04:42:51 -04:00

92 KiB

Raw Blame History

Perf Roadmap Remainder Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans (or superpowers-extended-cc:subagent-driven-development in-session) to implement this plan task-by-task.

Goal: Close the remaining items of the 2026-08-15 architectural performance review — roadmap items 3 (historian group commit + parallel drain), 4 (ACL cache re-shape), 5 (S7 span grouping + AbCip bounded parallelism), 6 (alarm-engine coalescing), 7 (AdminUI hot spots) — plus the residual finding-07 blocking-I/O sites and finding-08 per-value waste multipliers.

Architecture: Six independent work streams on one branch (perf/roadmap-remainder), each stream a small set of TDD tasks. The historian stream extends the item-1 batching pattern one hop further (mux → subscribers) so the outbox can group-commit; the ACL stream re-shapes the decision cache and hot path without touching the trie; the driver stream copies Modbus's coalescing shape into S7 and adds a bounded window to AbCip; the alarm stream adds dirty-set coalescing while preserving the pinned crash-consistency semantics; AdminUI gets projections + watermark/TTL caches; the residuals batch the comms-loss fan and move the last hot-mailbox blob read off-thread.

Tech Stack: .NET 10, Akka.NET, EF Core (SQL Server + InMemory tests), FasterLog, S7netplus, libplctag, xUnit + Shouldly, Akka.TestKit.

Verified baseline: master a8c93a00. All line numbers below were re-verified against that commit by six parallel exploration passes (2026-08-15); where the review's numbers had drifted, the corrected ones are used here.

Execution conventions (all tasks):

Work in this checkout on branch perf/roadmap-remainder. Stage explicit paths only (git add <paths>), never git add -A.
TDD: write the failing test first, watch it fail, implement, watch it pass, run the project's full unit suite, commit.
Test commands: dotnet test tests/<area>/<project> for a project, dotnet test --filter "FullyQualifiedName~<Class>.<Method>" for one test.
Do not touch *.IntegrationTests projects (Docker fixtures) — build-only.
If a task needs a file not in its Files: block, STOP and surface it as a plan defect.

Stream A — Historian group commit + parallel drain (review finding 03, Critical)

Current facts: IHistorizationOutbox has only single-entry AppendAsync (src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorizationOutbox.cs:11–44); FasterLogHistorizationOutbox.AppendAsync (src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/Recorder/FasterLogHistorizationOutbox.cs:97–137) does one EnqueueAsync + one CommitAsync (fsync) per entry in PerEntry mode, plus a second CommitAsync per capacity truncation. Values arrive at the recorder as one VirtualTagActor.DependencyValueChanged message per value because the item-1 batch envelope stops at DependencyMuxActor.OnAttributeValueBatchPublished (src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyMuxActor.cs:124–139), which fans one message per (item, subscriber). The drain (ContinuousHistorizationRecorder.DrainOnceAsync, .../Runtime/Historian/ContinuousHistorizationRecorder.cs:500–626) awaits one WriteLiveValuesAsync per tag group sequentially. The gateway client (0.3.0) has no multi-tag write RPC — a batch RPC on the HistorianGateway contract is a cross-repo follow-up, out of scope here; the client-side fix is bounded parallelism.

Task 1: Outbox batch append with group commit

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: Task 2, Task 3, Task 6, Task 8, Task 10, Task 11, Task 12, Task 14, Task 15, Task 16, Task 18, Task 19, Task 20, Task 22, Task 23, Task 24, Task 27

Files:

Modify: src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorizationOutbox.cs
Modify: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/Recorder/FasterLogHistorizationOutbox.cs
Test: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/Recorder/FasterLogHistorizationOutboxTests.cs
Modify (fakes only): tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Historian/ContinuousHistorizationRecorderTests.cs (the nested InMemoryOutbox), tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ServiceCollectionExtensionsTests.cs (the nested FakeOutbox)

Spec:

Add to IHistorizationOutbox:

/// <summary>
///     Appends a burst of entries with ONE durability commit for the whole batch (group
///     commit). Under PerEntry commit mode this is the per-batch fsync that replaces N
///     per-entry fsyncs; under Periodic it is indistinguishable from N AppendAsync calls.
///     An empty list is a no-op.
/// </summary>
ValueTask AppendBatchAsync(IReadOnlyList<HistorizationOutboxEntry> entries, CancellationToken ct);

FasterLogHistorizationOutbox.AppendBatchAsync: loop _log.EnqueueAsync(payload, ct) for every entry, then one await _log.CommitAsync(ct) when _commitMode == PerEntry (zero commits when Periodic). Then do the _state-locked bookkeeping for all entries in one lock acquisition (add all to _live/_index, run the drop-oldest loop once), and — if anything was dropped — one _log.TruncateUntil(...) + one CommitAsync. Net worst case: two commits per batch (vs 2×N today at capacity).
Re-implement single AppendAsync as AppendBatchAsync([entry], ct)-equivalent (may delegate; keep the public single-entry signature and its existing behaviour).
Update the two test fakes (InMemoryOutbox, FakeOutbox) to implement the new member (loop their existing AppendCore; InMemoryOutbox keeps its yielding-on-pool-thread variant applied to the batch method too — same await Task.Delay(1).ConfigureAwait(false) idiom, see the comment at ContinuousHistorizationRecorderTests.cs:695–748 for why Task.Yield() is vacuous).

Tests (write first, in FasterLogHistorizationOutboxTests.cs):

AppendBatch_then_peek_returns_fifo_in_batch_order — batch of 3, PeekBatchAsync(10) returns the 3 in order; restart-recovery (dispose + reopen, mirroring Remove_truncates_and_survives_restart at :41) still sees all 3 (proves the batch was committed durably).
AppendBatch_at_capacity_drops_oldest_across_the_whole_batch — capacity 2, batch of 4: only the last 2 survive, DroppedCount == 2.
AppendBatch_empty_is_noop — no throw, count unchanged.

Commit: perf(historian): group-commit batch append on the historization outbox

Task 2: Batched mux→subscriber delivery, single-eval virtual tags

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: Task 1, Task 3, Task 6, Task 8, Task 10, Task 11, Task 12, Task 14, Task 15, Task 16, Task 18, Task 19, Task 20, Task 22, Task 23, Task 24, Task 27

Files:

Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyMuxActor.cs
Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagActor.cs
Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyConsumerMuxAdapter.cs
Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ContinuousHistorizationRecorder.cs (receive registration only — the batch handler loops the existing per-value logic; the group-commit rewrite is Task 4)
Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmHostActor.cs (receive registration only — loop _upstream.Push per item)
Test: tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/VirtualTags/DependencyMuxActorTests.cs
Test: tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/VirtualTags/VirtualTagActorTests.cs

Spec:

New message, colocated with DependencyValueChanged in VirtualTagActor:

/// <summary>
///     One subscriber's slice of one upstream burst: every item in this batch matched
///     this subscriber's registered interest set. Never empty. List ownership transfers
///     to the receiver (the mux allocates a fresh list per subscriber per burst).
/// </summary>
public sealed record DependencyValueBatch(IReadOnlyList<DependencyValueChanged> Items);

DependencyMuxActor.OnAttributeValueBatchPublished (:124–139): instead of sub.Tell(dep) per (item, subscriber), accumulate per-subscriber lists (Dictionary<IActorRef, List<DependencyValueChanged>>, allocated lazily — nothing when no item matches), then one sub.Tell(new DependencyValueBatch(list)) per subscriber. The per-value comms-loss path (OnAttributeValuePublished, :102–115) also emits a single-item DependencyValueBatch — after this task the mux emits only batches.
Subscribers gain a Receive<VirtualTagActor.DependencyValueBatch> that loops their existing per-item handler — mechanical for DependencyConsumerMuxAdapter (whatever its current dispatch is, applied per item), ContinuousHistorizationRecorder (ReceiveAsync looping OnValueChangedAsync per item — Task 4 replaces this), and ScriptedAlarmHostActor (OnDependencyChanged per item). Keep each subscriber's existing DependencyValueChanged receive registration (tests and any direct senders still use it — e.g. ContinuousHistorizationRecorderTests Tells the per-value message directly).
VirtualTagActor is the one subscriber whose batch handler is NOT a loop: apply all items to _dependencies first, then run Evaluate once (the existing OnDependencyChanged body from :179–224 split into apply-phase and evaluate-phase). A K-input script hit by a K-item burst evaluates once instead of K times, and that one evaluation sees the complete input set.

Tests (write first):

DependencyMuxActorTests: A_burst_reaches_each_subscriber_as_one_batch_message — register subscriber for ["a","b"], send a 3-item AttributeValueBatchPublished (a, b, unmatched), ExpectMsg<VirtualTagActor.DependencyValueBatch> with exactly 2 items and no further message (ExpectNoMsg). Update the existing AttributeValueBatchPublished_is_filtered_per_item_to_interested_subscribers (:158) to the batch shape.
VirtualTagActorTests: A_batch_touching_K_inputs_evaluates_once — counting evaluator (a IVirtualTagEvaluator fake with an eval counter), 3-input tag, one 3-item batch ⇒ counter == 1, one EvaluationResult to parent carrying the value computed from all 3 inputs. Falsification control: send the same 3 values as 3 separate DependencyValueChanged messages ⇒ counter == 3 (proves the batch path, not value dedup, is what collapsed the count).

Commit: perf(runtime): mux delivers per-subscriber batches; virtual tags evaluate once per burst

Task 3: Parallel drain with bounded concurrency

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: Task 1, Task 2, Task 6, Task 8, Task 10, Task 11, Task 12, Task 14, Task 15, Task 16, Task 18, Task 19, Task 20, Task 22, Task 23, Task 24, Task 27

Files:

Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ContinuousHistorizationRecorder.cs (DrainOnceAsync :500–626, Props :178–191, ctor)
Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ContinuousHistorizationOptions.cs
Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs (spawn site :463–481 threads the new option)
Test: tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Historian/ContinuousHistorizationRecorderTests.cs

Spec:

New option DrainConcurrency (int, default 8, Validate() warns when < 1) on ContinuousHistorizationOptions; thread through Props/ctor and the spawn site.

DrainOnceAsync: replace the sequential foreach over tag groups (:517–545) with a bounded-parallel execution — materialise the groups, then:

using var window = new SemaphoreSlim(_drainConcurrency);
var tasks = groups.Select(async g =>
{
    await window.WaitAsync(ct).ConfigureAwait(false);
    try
    {
        bool ok;
        try { ok = await _writer.WriteLiveValuesAsync(g.Key, g.Values, ct).ConfigureAwait(false); }
        catch { ok = false; }
        return (g.Key, g.Values.Count, ok);
    }
    finally { window.Release(); }
}).ToList();
var outcomes = await Task.WhenAll(tasks).ConfigureAwait(false);

then compute succeeded / failedGroups from outcomes and run the unchanged #538 decision block (:554–611) against them. _tagWriteFailures mutation stays post-WhenAll on the single logical drain flow (never inside the parallel lambdas) — the dictionary is actor-state.

The all-or-nothing ack (RemoveAsync(batch[^1].Id)) and the #538 semantics are behaviour-pinned: A_backend_that_fails_every_tag_abandons_nothing (:455–486), A_permanently_failing_tag_is_abandoned_so_healthy_tags_stop_being_rewritten (:408), and A_tag_that_recovers_within_its_budget_is_not_abandoned (:490) must pass unmodified. Do not change what counts as "another tag wrote this pass" — it is now "any other group's outcome in this pass was ok".

Tests (write first):

Drain_writes_distinct_tags_concurrently_within_the_window — a gating FakeValueWriter variant that blocks the first call on a TCS and counts concurrently-in-flight calls: with 3 tags queued and concurrency 8, at least 2 calls are in flight before the gate releases. (Add MaxObservedConcurrency to the fake under its existing lock.)
Drain_concurrency_of_one_is_sequential — same fixture, concurrency 1 ⇒ MaxObservedConcurrency == 1 (falsification control for the window).
All existing recorder tests green unmodified.

Commit: perf(historian): bounded-parallel per-tag drain (DrainConcurrency, default 8)

Task 4: Recorder batched ingest — one fsync per burst

Classification: high-risk Estimated implement time: ~4 min Parallelizable with: Task 7, Task 9, Task 13, Task 17, Task 21, Task 25 (blocked by Tasks 1, 2, 3 — same file as Task 3)

Files:

Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ContinuousHistorizationRecorder.cs
Test: tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Historian/ContinuousHistorizationRecorderTests.cs

Spec:

Rewrite the DependencyValueBatch handler (registered in Task 2 as a loop) as the real batched path: run the existing per-value gates (ref filter :285–288, capture gate :297–315 — re-read per value, the pinned semantic of The_capture_gate_is_re_read_per_value_so_promotion_resumes_capture, numeric coercion :317–324) per item, accumulate the surviving HistorizationOutboxEntry list (one per historian name per item via _refMap), then one await _outbox.AppendBatchAsync(entries, _lifetime.Token).ConfigureAwait(false) and one trailing self.Tell(DrainTick.Instance). Capture IActorRef self = Self before the await — same ConfigureAwait(false)-destroys-Self hazard as the live-gate defect (:341 comment).
Keep the single-value DependencyValueChanged handler delegating to the same core (a 1-item batch) so existing tests and any direct senders behave identically.
On append failure: count _outboxAppendFailures once per failed batch append, log, drop the batch's values (same policy as the per-value path, applied batch-wide).

Tests (write first):

A_burst_appends_once_and_drains_all_values — instrument InMemoryOutbox with an AppendBatchCallCount; send one 5-item DependencyValueBatch (all refs historized) ⇒ AppendBatchCallCount == 1, all 5 values eventually reach the writer.
A_yielding_batch_append_neither_faults_the_actor_nor_loses_values — the pool-thread yielding outbox + a batch message; actor stays alive (send GetStatus after), values drain. (Mirrors A_yielding_append_neither_faults_the_actor_nor_loses_the_value :359.)
The_capture_gate_suppresses_items_within_a_batch — gate closed for the batch ⇒ nothing appended (extends A_closed_capture_gate_suppresses_the_value :543 to the batch path).

Commit: perf(historian): batch ingest — one group commit per value burst

Task 5: Historian defaults, config drift, docs

Classification: small Estimated implement time: ~3 min Parallelizable with: none (blocked by Tasks 3, 4)

Files:

Modify: src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json (:40–51)
Modify: tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Historian/ContinuousHistorizationOptionsTests.cs
Modify: docs/Historian.md, docs/Configuration.md (ContinuousHistorization key tables)
Modify: CLAUDE.md (the ContinuousHistorization key table: add DrainConcurrency, correct DrainBatchSize default)

Spec:

Raise DrainBatchSize default 64 → 1024 (options class + appsettings). Rationale to record in the XML doc: the drain is now bounded-parallel and the peek is an in-memory scan; 64 made the outbox fill during any gateway hiccup.
Fix the shipped-appsettings drift found in exploration: appsettings.json sets Capacity: 0 (unbounded) while the options default and docs/Historian.md:176 say 500_000 — set appsettings to 500000 and add the missing MaxWriteAttempts: 5 and new DrainConcurrency: 8 keys, so the shipped file matches the documented posture.
Add a defaults-pinning test to ContinuousHistorizationOptionsTests (none exists — exploration confirmed changing defaults is currently uncaught): assert CommitMode == "PerEntry", DrainBatchSize == 1024, DrainConcurrency == 8, Capacity == 500_000, MaxWriteAttempts == 5.
Docs: document AppendBatchAsync group commit ("PerEntry now means one fsync per ingest burst, not per value"), DrainConcurrency, the new DrainBatchSize, and record the cross-repo follow-up: a multi-tag batch WriteLiveValues RPC on the HistorianGateway contract (client 0.3.0 has none — bounded parallelism is the ceiling until then).

Commit: chore(historian): batch-era defaults + appsettings/doc alignment

Stream B — ACL cache re-shape (review finding 04, High)

Current facts: SessionAclDecisionCache (src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/Security/SessionAclDecisionCache.cs) is one flat ConcurrentDictionary with MaxEntries = 100_000 (:48) and a wholesale Clear() at cap (:92), reading ConcurrentDictionary.Count (all-locks snapshot) on every store. DataPlaneAuthorizer.Evaluate (Security/DataPlaneAuthorizer.cs:176–233) runs TryBuildScope — the Split('/') at :305 — before the cache lookup, allocates a 5-capture closure per call even on hits, and builds interpolated deny reasons eagerly on the Debug-logged hot deny path. Browse filtering uses a backwards RemoveAt loop at OtOpcUaNodeManager.cs:2947–2960 (twin at TranslateBrowsePath :2998–3005 the review missed). RevalidateMonitoredItems (:3173–3221) walks every live item with no (session, node) dedup, two lock (Lock) takes per changed item, on the deploy thread. No session-close eviction hook exists anywhere (verified) — the shard cap is the bound.

Task 6: Per-session decision-cache shards

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: Task 1, Task 2, Task 3, Task 8, Task 10, Task 11, Task 12, Task 14, Task 15, Task 16, Task 18, Task 19, Task 20, Task 22, Task 23, Task 24, Task 27

Files:

Modify: src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/Security/SessionAclDecisionCache.cs
Test: tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/SessionAclDecisionCacheTests.cs

Spec:

Re-shape storage to ConcurrentDictionary<string /*sessionKey*/, Shard> where

private sealed class Shard
{
    public readonly ConcurrentDictionary<ShardKey, bool> Entries = new();
    public int ApproxCount; // Interlocked; approximate by design — never call Entries.Count on the hot path
}
private readonly record struct ShardKey(OpcUaOperation Operation, AddressSpaceRealm Realm, string NodeIdentifier);

public const int MaxEntriesPerSession = 20_000; — at the per-shard cap, clear that shard only (shard.Entries.Clear(); Interlocked.Exchange(ref shard.ApproxCount, 0);). One machine consumer crawling a 50k-node tree now cycles only its own shard; other sessions' hits survive. Keep MaxEntries = 100_000 as the global approximate bound: track a global Interlocked counter; when it trips, clear all shards (the today behaviour, now a rare fallback). Both caps use the approximate counters — no ConcurrentDictionary.Count anywhere on the store path.
Preserve the public surface the tests and NodeManagerStage6EnforcementTests:191–197 pin: Count (now the exact sum over shards — off-hot-path property, may enumerate), Generation, Clear(), and GetOrAdd(long generation, string sessionKey, OpcUaOperation op, AddressSpaceRealm realm, string nodeIdentifier, Func<bool> compute). Keep the two pinned semantics verbatim: generation-flush clears everything (single CAS-winner clears, :100–108 comment logic), and a decision overtaken by a deploy mid-compute is not stored (A_decision_overtaken_by_a_deploy_is_not_stored).
XML-doc the design shift: the old "clearing beats evicting" argument (:37–47) now applies per shard.

Tests (write first):

A_full_session_shard_clears_only_that_session — fill session "A" past MaxEntriesPerSession, one entry for session "B" ⇒ B's entry still hits (compute not re-invoked), A's shard restarted.
The_global_bound_still_holds — adapt the existing The_cache_is_bounded (:117) to many sessions; Count stays ≤ MaxEntries + one shard's worth.
All 9 existing facts pass (adjust only constructions, never assertions).

Commit: perf(acl): per-session decision-cache shards — a crawl no longer evicts the fleet

Task 7: Authorizer hot path — lookup first, no closures, lazy deny reasons

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: Task 4, Task 9, Task 13, Task 17, Task 21, Task 25 (blocked by Task 6)

Files:

Modify: src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/Security/DataPlaneAuthorizer.cs
Modify: src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/Security/SessionAclDecisionCache.cs (add the TryGet/Set pair)
Test: tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/DataPlaneAuthorizerTests.cs

Spec:

Add an allocation-free fast path to the cache: bool TryGet(long generation, string sessionKey, OpcUaOperation op, AddressSpaceRealm realm, string nodeIdentifier, out bool decision) and void Set(long generation, ...same..., bool decision) (Set carries the overtaken-generation drop check). Keep GetOrAdd delegating to them.
Reorder Evaluate (:176–233): identity/principal/cluster checks → key-revocation re-check (must stay outside the cache — the :203–208 comment and NodeManagerStage6EnforcementTests clock test pin it) → extract nodeId.Identifier as string (null/empty ⇒ deny, absorbing TryBuildScope's guard) → Decisions.TryGet(...) — hit returns here, no scope build, no closure → on miss: TryBuildScope + Walk + Decisions.Set(...).
Deny-reason laziness: Deny currently interpolates at every call site before deciding the level (:310–325, cached denials log Debug). Change the hot-path call sites (:214, :232) to pass the format pieces and let Deny build the string only when _log.IsEnabled(...) for the chosen level. Verify whether the SDK's ServiceResult is immutable; if so return a cached static readonly ServiceResult DeniedResult = new(StatusCodes.BadUserAccessDenied); from the deny path — if anything mutates ServiceResult (check SDK source via DeepWiki OPCFoundation/UA-.NETStandard if unsure), keep the per-call allocation and note why in a comment.
The load-bearing ALLOW test (NodeManagerAclEnforcementTests.An_acl_grant_authorizes_a_write_from_a_session_carrying_no_roles) and all 17 DataPlaneAuthorizerTests facts must pass unmodified.

Tests (write first, in DataPlaneAuthorizerTests.cs):

A_cache_hit_does_not_rebuild_the_scope — a counting evaluator fake (existing test fakes count Authorize calls): two identical AuthorizeCached calls ⇒ evaluator called once; assert via the cache Count == 1 that the second call was a hit. (The scope-build itself is not directly observable — the evaluator count is the proxy: hit ⇒ no Walk ⇒ no scope consumption.)
Key_revocation_is_rechecked_on_a_cache_hit — prime an allow, revoke the key in the snapshot, same call ⇒ denied (pins the ordering of step 2).

Commit: perf(acl): authorizer hot path — cache lookup before scope build, lazy deny reasons

Task 8: Browse and TranslateBrowsePath compaction

Classification: standard Estimated implement time: ~3 min Parallelizable with: Task 1, Task 2, Task 3, Task 6, Task 10, Task 11, Task 12, Task 14, Task 15, Task 16, Task 18, Task 19, Task 20, Task 22, Task 23, Task 24, Task 27

Files:

Modify: src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs (Browse :2919–2963, TranslateBrowsePath :2998–3005)
Test: tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerStage6EnforcementTests.cs

Spec:

Replace both backwards RemoveAt(ii) loops with forward two-pointer compaction over the appended range [before, Count): walk forward, copy each permitted reference down to a write cursor, then remove the tail once (for (var ii = references.Count - 1; ii >= write; ii--) references.RemoveAt(ii); — or RemoveRange if the concrete IList supports it; it is typed IList<T>, so the tail loop of RemoveAt-at-end is O(removed) with no element moves). Semantics unchanged: same Permits gate per reference, same filtered count, same LogAclFiltering call.
Apply identically to the targetIds loop in TranslateBrowsePath.

Tests (write first): extend the existing browse-filtering coverage in NodeManagerStage6EnforcementTests with an order-preservation case: grant on nodes {1st, 3rd, 5th} of a 6-child folder ⇒ result is exactly those three in original order (the compaction bug this guards against is duplicated/reordered survivors). The over-the-wire suite (AclSubscriptionEnforcementTests.Browse_returns_only_the_granted_subtree) stays green untouched.

Commit: perf(acl): O(n) browse/translate filtering compaction

Task 9: RevalidateMonitoredItems dedup

Classification: standard Estimated implement time: ~4 min Parallelizable with: Task 4, Task 7, Task 13, Task 17, Task 21, Task 25 (blocked by Task 8 — same file)

Files:

Modify: src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs (RevalidateMonitoredItems :3173–3221)
Test: tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerStage6EnforcementTests.cs

Spec:

Inside the walk, memoize the verdict per distinct (sessionKey, nodeIdentifier) in a local Dictionary<(string, string), bool> — sessionKey = identity.ApiKeyId ?? identity.Principal (same derivation the authorizer uses), node identifier = (string)tracked.Handle.NodeId.Identifier. N monitored items on the same node from the same session now cost one IsAllowedCached evaluation instead of N. Items whose identity is not a RoleCarryingUserIdentity keep today's per-item path.
Leave the per-item lock (Lock) takes as-is (attach/detach must hold the SDK lock; batching them under one hold would extend the lock across QueueValue calls — out of scope, note as considered-and-rejected in a comment).

Tests (write first): a unit test in NodeManagerStage6EnforcementTests (the rig there constructs the node manager with a fake authorizer): create 3 monitored items on the same node for the same identity via OnMonitoredItemCreated, flip authority, call RevalidateMonitoredItems(), assert the counting authorizer evaluated that (session, node) once — falsification control: 3 items on 3 distinct nodes ⇒ 3 evaluations.

Commit: perf(acl): dedup subscription revalidation by (session, node)

Stream C — Driver round-trips (review finding 05, High)

Current facts (all line-verified): S7 reads one tag per wire request under a single gate (S7Driver.ReadAsync :503–599, per-tag ReadOneAsync :601–636), with every address pre-parsed at init into S7ParsedAddress(Area, DbNumber, Size, ByteOffset, BitOffset) (:260–267) and a contiguous bulk primitive already on the seam (IS7Plc.ReadBytesAsync, IS7Plc.cs:44; no multi-var read — multi-var PDUs are a recorded follow-up, not in scope). AbCip awaits each fallback tag strictly sequentially (AbCipDriver.ReadAsync :576–600); the libplctag handle is per-tag async-capable but NOT concurrency-safe per handle (per-tag GetRuntimeLock exists, :1398–1414), and results are index-addressed. PollGroupEngine paces delay-after-completion with zero instrumentation (PollGroupEngine.cs:142–195) and Unsubscribe blocks up to 5 s (:114–137); the engine's project (Core.Abstractions) has no references at all, so metrics are exposed as a counter + callback, not a Meter. The Modbus InitializeAsync transport overwrite-without-dispose is confirmed still present (ModbusDriver.cs:265), reachable via DriverInstanceActor.AdoptConfigDuringInit (:752–764); MQTT's teardown-first pattern (MqttDriver.cs:946–1033) is the template, and Modbus's own ReadCoalescedAsync (:787+) is the in-repo prior art for span planning.

Task 10: Modbus InitializeAsync transport leak

Classification: small Estimated implement time: ~3 min Parallelizable with: Task 1, Task 2, Task 3, Task 6, Task 8, Task 11, Task 12, Task 14, Task 15, Task 16, Task 18, Task 19, Task 20, Task 22, Task 23, Task 24, Task 27

Files:

Modify: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusDriver.cs (InitializeAsync :252–307)
Test: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests/ModbusLifecycleHygieneTests.cs

Spec:

At the top of InitializeAsync (before _transport = _transportFactory(_options) at :265), dispose what a prior initialize published: cancel+dispose _probeCts and _reprobeCts (null-tolerant, the TeardownAsync :1675–1702 idiom), and if (_transport is not null) await _transport.DisposeAsync().ConfigureAwait(false); _transport = null;. Do NOT call full TeardownAsync() — it disposes _poll (the ctor-created engine, never rebuilt) and clears the tag maps; scope the fix to the three leaked resources, with a comment naming AdoptConfigDuringInit as the reachable caller (mirror the MQTT comment at MqttDriver.cs:952–961).
First initialize sees three nulls — no behaviour change.

Tests (write first, in ModbusLifecycleHygieneTests.cs using the existing DisposeCount fake transport at :23):

A_second_InitializeAsync_disposes_the_first_transport — init twice with a factory returning distinct fakes ⇒ first fake DisposeCount == 1, second == 0, driver healthy on the second config.

Commit: fix(modbus): dispose the previous transport and probe loops on re-initialize

Task 11: PollGroupEngine fixed-rate pacing, overrun counter, async unsubscribe

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: Task 1, Task 2, Task 3, Task 6, Task 8, Task 10, Task 12, Task 14, Task 15, Task 16, Task 18, Task 19, Task 20, Task 22, Task 23, Task 24, Task 27

Files:

Modify: src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/PollGroupEngine.cs
Modify (mechanical call-site swap to the async unsubscribe): src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusDriver.cs:1250–1252, src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Focas/FocasDriver.cs:841–843, src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.TwinCat/TwinCatDriver.cs:579–588, src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbLegacy/AbLegacyDriver.cs:492–494, src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7/S7Driver.cs:1442–1444, plus the AbCip and Sql engine call sites (locate \.Unsubscribe\( in those two drivers — same one-line swap)
Test: tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/PollGroupEngineTests.cs

Spec:

Fixed-rate pacing: in PollLoopAsync (:142–195), stamp Stopwatch.GetTimestamp() before PollOnceAsync, compute elapsed = Stopwatch.GetElapsedTime(start) after it, and delay max(interval - elapsed, TimeSpan.Zero) instead of the full interval. When elapsed > interval increment a long _overrunCount (Interlocked; expose as public long OverrunCount) and invoke the new optional ctor callback Action<TimeSpan>? onOverrun = null (wrapped in try/catch like ReportError). The failure/backoff branch is untouched: when consecutiveFailures > 0 and a backoff cap is configured, ConnectionBackoff.ComputeDelay still wins (a failing device must not be hammered at fixed rate).
Async unsubscribe: add public async ValueTask<bool> UnsubscribeAsync(ISubscriptionHandle handle) — same TryRemove, then cancel + await task.WaitAsync(TimeSpan.FromSeconds(5)) under try/catch (the DisposeAsync :272–295 shape) + CTS dispose. Keep the sync Unsubscribe for compatibility (tests use it), sharing the teardown via a small private core. Swap every driver's Task UnsubscribeAsync(...) body from the blocking _poll.Unsubscribe(h) to await _poll.UnsubscribeAsync(h) — this unblocks the DriverInstanceActor mailbox turn that awaits it (DriverInstanceActor.cs:482, :894).
No Meter here — Core.Abstractions deliberately references nothing; the counter + callback is the instrumentation surface (record in the class doc; wiring a meter at driver level is a follow-up).

Tests (write first, in PollGroupEngineTests.cs, real-delay style like the existing suite):

A_slow_reader_overrun_is_counted_and_the_next_poll_starts_promptly — reader that delays 150 ms on interval 50 ms: after ~4 polls OverrunCount >= 2 and the observed poll count over 1 s is ≥ 5 (fixed-rate; the old delay-after-completion shape would give ≤ 5 in 1 s at 200 ms/cycle — assert with slack, this suite tolerates CI jitter with WaitForAsync).
A_fast_reader_never_overruns — instant reader, OverrunCount == 0 after several cycles (falsification control).
UnsubscribeAsync_stops_callbacks_like_the_sync_path — mirror the existing Unsubscribe assertions through the async API.

Commit: perf(drivers): fixed-rate poll pacing with overrun counter; non-blocking unsubscribe

Task 12: S7 narrow-scalar span codec

Classification: standard Estimated implement time: ~4 min Parallelizable with: Task 1, Task 2, Task 3, Task 6, Task 8, Task 10, Task 11, Task 14, Task 15, Task 16, Task 18, Task 19, Task 20, Task 22, Task 23, Task 24, Task 27

Files:

Modify: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7/S7Driver.cs (new internal static helpers next to DecodeScalarBlock :753+)
Test: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.Tests/S7SpanCodecTests.cs (new)

Spec:

internal static object DecodeNarrowScalar(S7TagDefinition tag, S7ParsedAddress addr, ReadOnlySpan<byte> slice) — decodes the narrow scalars that today go through plc.ReadAsync(string) + ReinterpretRawValue (:632, :1000) from a big-endian byte slice positioned at the tag's own offset: Bool from slice[0] bit addr.BitOffset; Byte/SByte; Int16/UInt16 from the BE word; Int32/UInt32/Float32 from the BE dword. Semantic parity with ReinterpretRawValue is the requirement — read that method first and mirror its sign/width handling exactly (it is the reference behaviour the integration parity test in Task 13 will compare against).
internal static int NarrowScalarWidth(S7ParsedAddress addr) — Bit/Byte ⇒ 1, Word ⇒ 2, DWord ⇒ 4 (align with ElementByteSize :868).

Tests (write first, table-driven [Theory] over the S7DataType × slice matrix):

known BE byte patterns decode to the exact expected CLR values, including sign edges (0x8000 ⇒ Int16 -32768 / UInt16 32768), float bit patterns, and bit extraction for BitOffset 0 and 7.

Commit: feat(s7): pure narrow-scalar decode from byte spans

Task 13: S7 span planner + grouped ReadAsync

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: Task 4, Task 7, Task 9, Task 17, Task 21, Task 25 (blocked by Task 12 — same file)

Files:

Create: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7/S7ReadSpanPlanner.cs
Modify: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7/S7Driver.cs (ReadAsync :503–599)
Modify: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.Contracts/S7DriverOptions.cs (or wherever S7DriverOptions lives — locate it; the options type the ctor takes)
Test: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.Tests/S7ReadSpanPlannerTests.cs (new)
Test: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.Tests/S7SpanReadParityTests.cs (new)

Spec:

Options: SpanReadMaxGapBytes (int, default 16; <= -1 disables span grouping) and SpanReadMaxSpanBytes (int, default 220 — under one 240-byte PDU's payload; ReadBytesAsync would transparently chunk bigger spans, the cap bounds wasted gap bytes, not correctness).
S7ReadSpanPlanner.Build(IReadOnlyList<(int OriginalIndex, S7TagDefinition Tag, S7ParsedAddress Addr)> resolved, int maxGapBytes, int maxSpanBytes) — pure function, Modbus-ReadCoalescedAsync-shaped (ModbusDriver.cs:787+ is the reference): eligible = everything except Area is Timer or Counter (their ByteOffset is a device number, not an address) and except array tags with ArrayCount >= 1 only if mixing them complicates width math — arrays ARE eligible (width = ArrayCount * ElementByteSize), strings are eligible (width = StringLength + 2 via ScalarByteWidth). Group by (Area, DbNumber), sort by ByteOffset, merge while nextStart - currentEnd <= maxGapBytes and mergedSpan <= maxSpanBytes. A span never extends past the last member's end — gap bytes between members are read (they exist inside the DB between two real tags), but nothing beyond the highest member end is requested, so a span can never fail where the individual reads would have succeeded. Output: spans (area, db, start, length, members with relative offsets) + leftovers (ineligible / disabled ⇒ per-tag path).
ReadAsync: after resolving references (keep the exact unresolved/BadNodeIdUnknown handling), when grouping is enabled build the plan and execute one ReadBytesAsync(ToS7NetArea(area), db, start, length, ct) per span under the existing _gate, then decode each member by slicing (DecodeNarrowScalar from Task 12 for narrow scalars, DecodeScalarBlock/DecodeArrayBlock for buffer types — both take a block starting at the tag's offset, which the slice provides). Per-span failure maps through the existing exception taxonomy (:552–595: OCE ⇒ _plcDead+throw, access-denied ⇒ BadNotSupported+Faulted, PlcException ⇒ BadDeviceFailure+Degraded, else ⇒ BadCommunicationError+Degraded) applied to every member of the failed span — extract that catch ladder into a private helper so the span path and the per-tag path share it verbatim. Leftover tags keep today's ReadOneAsync path. Health writes stay as today (whole method already runs under _gate, single-threaded — no new races).
minInterval/subscription flow untouched — this is inside the reader delegate.

Tests (write first):

S7ReadSpanPlannerTests: contiguous tags merge; gap > max splits; span cap splits; distinct DBs never merge; Timer/Counter excluded; span end == last member end (assert no span length overshoots).
S7SpanReadParityTests — the falsification gate: a fake IS7Plc holding a byte image per (area, db) serving both ReadBytesAsync (slice of the image) and ReadAsync (decode from the same image at the parsed address — implement in the fake); a mixed tag set (bits at both bit offsets, words, dwords, a float, a string, an array, with gaps) read through a grouping driver and a SpanReadMaxGapBytes = -1 driver ⇒ snapshot-for-snapshot identical values and statuses. Plus A_span_read_failure_degrades_every_member_like_the_per_tag_path (fake throws from ReadBytesAsync ⇒ all members BadDeviceFailure/BadCommunicationError per the exception type, health Degraded).

Commit: perf(s7): span-grouped reads — one PDU per contiguous block instead of one per tag

Task 14: AbCip bounded-parallel reads

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: Task 1, Task 2, Task 3, Task 6, Task 8, Task 10, Task 11, Task 12, Task 15, Task 16, Task 18, Task 19, Task 20, Task 22, Task 23, Task 24, Task 27

Files:

Modify: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbCip/AbCipDriver.cs (ReadAsync :576–600)
Modify: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbCip.Contracts/AbCipDriverOptions.cs (+ the factory DTO mapping in AbCipDriverFactoryExtensions.cs)
Test: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbCip.Tests/AbCipParallelReadTests.cs (new)

Spec:

Option MaxConcurrentReads (int, default 8, clamp ≥ 1) on AbCipDriverOptions + DTO + factory mapping (follow the EnableDeclarationOnlyUdtGrouping plumbing at AbCipDriverFactoryExtensions.cs:87, :148).
ReadAsync: replace the two sequential foreach-await loops with one bounded-window execution over plan.Groups ∪ plan.Fallbacks (a SemaphoreSlim(maxConcurrent) + Task.WhenAll, the Task 3 shape). Safety rails, all already in the code and to be preserved: the per-tag GetRuntimeLock serialises Read→GetStatus→Decode against the shared handle (05/STAB-4, :629–630 comment) — the window parallelises across tags, never within one; results are index-addressed into results[OriginalIndex] (:576–600), so completion order is irrelevant; EnsureTagRuntimeAsync's duplicate-create race is already handled (:1013–1018).
Health writes must not race: inspect how ReadSingleAsync/ReadGroupAsync write driver health today; hoist to a post-WhenAll aggregation — collect each task's outcome (Healthy/Degraded/Faulted + message) and write the health field once after the batch with worst-outcome-wins (Faulted > Degraded > Healthy), LastSuccessfulRead advanced if any tag succeeded. This preserves the observable contract of GetHealth() that the comms-loss watchdog depends on (CLAUDE.md §"Comms-loss detection") — LastSuccessfulRead still advances on any success, still freezes on total failure.
Caller cancellation: propagate OCE like today (a cancelled batch may leave some tasks unstarted — fine; Task.WhenAll rethrow after ensuring the window can't deadlock — release in finally).

Tests (write first, FakeAbCipTag-based like AbCipDriverReadTests.cs:12–35):

Reads_overlap_up_to_the_window — a FakeAbCipTag subclass whose ReadAsync blocks on a shared TCS and records max-concurrent (Interlocked): 4 tags, window 8 ⇒ observed concurrency ≥ 2; window 1 ⇒ exactly 1 (falsification control).
Same_tag_requested_twice_is_still_serialised — two references to the same tag name in one batch ⇒ the fake's per-instance concurrent-read count never exceeds 1 (the runtime lock holds).
Health_after_a_mixed_batch_is_worst_outcome_with_advanced_LastSuccessfulRead — one succeeding + one throwing tag ⇒ Degraded and LastSuccessfulRead != null.

Commit: perf(abcip): bounded-parallel tag reads (MaxConcurrentReads, default 8)

Stream D — Alarm-engine coalescing (review finding 06, High)

Current facts (zero drift from the review): every upstream push spawns an unthrottled fire-and-forget ReevaluateAsync that queues on the single _evalGate (src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/ScriptedAlarmEngine.cs:507–515, gate :146), with _store.SaveAsync inside the gate (:539) and a 250 ms Task.Run+WaitAsync per predicate (TimedScriptEvaluator.cs:78–81) — N pushes during one slow evaluation ⇒ N serialized re-evaluations. Crucially RefillReadCache (:651–657) always reads the latest _valueCache values, so coalescing loses nothing by design.

Design decision (differs from the review's write-behind suggestion, deliberately): SaveAsync stays inside the gate, before the in-memory update. The persist-before-update ordering is pinned by Store_save_failure_leaves_in_memory_state_unchanged (:602–628) and is a real crash-consistency property (a store failure must leave both sides at the prior state). A write-behind queue would trade that away for latency the dirty-set coalescing already removes: post-coalescing, saves happen only on genuine transitions (rare), not per push. Record this rationale in the engine's class doc.

Task 15: Dirty-set coalescing in ScriptedAlarmEngine

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: Task 1, Task 2, Task 3, Task 6, Task 8, Task 10, Task 11, Task 12, Task 14, Task 16, Task 18, Task 19, Task 20, Task 22, Task 23, Task 24, Task 27

Files:

Modify: src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/ScriptedAlarmEngine.cs
Test: tests/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms.Tests/ScriptedAlarmEngineTests.cs

Spec:

Add engine state:

private readonly HashSet<string> _dirtyAlarmIds = new(StringComparer.Ordinal);
private readonly object _dirtyLock = new();
private bool _drainScheduled; // guarded by _dirtyLock

OnUpstreamChange (:507–515): keep the _valueCache write; replace the per-push TrackBackgroundTask(ReevaluateAsync(alarmIds.ToArray(), ...)) with: under _dirtyLock, union the referencing alarm ids into _dirtyAlarmIds (no ToArray allocation); if _drainScheduled is false, set it and TrackBackgroundTask(DrainDirtyAsync(CancellationToken.None)). A push landing while a drain is running just grows the set — the running drain's loop picks it up.
DrainDirtyAsync: loop — under _dirtyLock swap the set out (var batch = _dirtyAlarmIds.ToArray(); _dirtyAlarmIds.Clear(); — when empty, clear _drainScheduled and exit); then run the existing ReevaluateAsync(batch, ct) body per swap (gate → per-alarm evaluate-latest → persist-before-update → release → fire emissions). Reuse ReevaluateAsync as-is by calling it — its shape is already correct for a batch; only the spawn-per-push is being replaced. Keep the field name _evalGate and its acquire/release shape — OnEvent_emission_happens_outside_evalGate (:754–807) reflects on that private field by name.
RunShelvingCheck (:831–836) can keep its direct spawn (5 s cadence, not a storm path) — but route it through the dirty set if trivially possible (union all ids + schedule) for one code path.
Dispose drain semantics unchanged: DrainDirtyAsync is tracked via TrackBackgroundTask, so Dispose_drains_in_flight_reevaluation_tasks (:816–852) holds. The _disposed re-check inside the gate (:528) stays.

Tests (write first):

A_storm_of_pushes_coalesces_to_one_evaluation_per_alarm — a save-and-eval-counting setup: block the gate via BlockingSaveAlarmStateStore (:1231–1271) mid-first-eval, push the same tag 10 more times, release ⇒ the alarm's predicate evaluated at most 2 more times (the in-flight one + one drain pass over the coalesced set), not 10. Count evaluations via a script that increments a static/captured counter through ctx — or simpler: a real script return (int)ctx.GetTag("Temp").Value > 100; and a save-counting store decorator (add one: CountingAlarmStateStore wrapping InMemoryAlarmStateStore) asserting transitions, plus assert via the counting store that no redundant saves happened. Include the falsification control: 10 pushes with alternating values that each cross the threshold ⇒ the final state reflects the latest value (coalescing is latest-wins, never lost-update).
A_push_during_a_drain_is_not_lost — push while the drain is inside the gate ⇒ a second drain pass runs and evaluates it.
All existing engine facts pass unmodified — pay attention to the Task.Delay(100/150) quality tests (:667–694 etc.): the drain must start immediately when the gate is free (no timer/delay in the scheduling path — the coalescing window is "whatever queued while the gate was busy", not a wall-clock batch window).

Commit: perf(alarms): dirty-set coalescing — one evaluation per alarm per drain, latest values win

Stream E — AdminUI hot spots (review finding 09, Medium)

Current facts: /deployments materialises 50 full Deployment entities including ArtifactBlob and runs a full 18-table ConfigComposer.SnapshotAndFlattenAsync compose per visit (Deployments.razor:80–99); the Monaco catalog rebuilds six fleet-wide table scans per completion/hover call with zero caching anywhere in the solution (ScriptTagCatalog.BuildEntriesAsync, IScriptTagCatalog.cs:159–219; /completions fires per keystroke inside string literals — monaco-init.js:33,198); bulk add-references runs a fresh DbContext + 4 queries per tag (EffectiveNameGuard.cs:21–88, loop at UnsTreeService.cs:214–242); the raw tree loads full Tag entities (with TagConfig nvarchar(max)) to render three fields (RawTreeService.cs:163–203); every raw rename loads every script body (:1415–1436); the context factory is not pooled despite comments claiming it is (Configuration/ServiceCollectionExtensions.cs:25); RoleGrants pins a scoped DbContext for the circuit via LdapGroupRoleMappingService; /fleet and /hosts poll on 10 s / 5 s timers with no overlap guard. AdminUI tests are pure EF InMemory (no SQL Server needed).

Task 16: Deployments list projection

Classification: small Estimated implement time: ~3 min Parallelizable with: Task 1, Task 2, Task 3, Task 6, Task 8, Task 10, Task 11, Task 12, Task 14, Task 15, Task 18, Task 19, Task 20, Task 22, Task 23, Task 24, Task 27

Files:

Modify: src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Deployments.razor

Spec:

Replace the full-entity load (:83–87) with a blob-free projection:
```
private sealed record DeploymentRow(
    Guid DeploymentId, string RevisionHash, DeploymentStatus Status,
    string CreatedBy, DateTime CreatedAtUtc, DateTime? SealedAtUtc);
```
.Select(d => new DeploymentRow(...)) before ToListAsync(). The table (:54–64) renders exactly these six columns; the drift check needs only Status/RevisionHash — adjust latestSealed accordingly. No test project covers this page (verified); the compile + the Task 17 tests cover the drift seam.

Commit: perf(adminui): /deployments lists a projection, not 50 artifact blobs

Task 17: Drift watermark probe

Classification: standard Estimated implement time: ~5 min Parallelizable with: Task 4, Task 7, Task 9, Task 13, Task 21, Task 25 (blocked by Task 16 — same razor file)

Files:

Create: src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Deployments/DeploymentDriftProbe.cs
Modify: src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Deployments.razor
Modify: src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/EndpointRouteBuilderExtensions.cs (register singleton)
Test: tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/Deployments/DeploymentDriftProbeTests.cs (new)

Spec:

DeploymentDriftProbe — singleton over IDbContextFactory<OtOpcUaConfigDbContext>: Task<bool?> GetDriftAsync(string latestSealedRevisionHash, CancellationToken ct). Watermark = one cheap aggregate query over ConfigEdit (MAX(EditedAtUtc), COUNT(*) — a single GroupBy(_ => 1).Select(...) or two scalars; ConfigEdit is the append-only audit row every mutating live-edit writes, so any topology change moves it). Cache (watermark, sealedHash) → bool drift under a SemaphoreSlim(1,1); recompute ConfigComposer.SnapshotAndFlattenAsync only when the watermark or the sealed hash changed. Comment the accepted staleness: an edit path that bypasses ConfigEdit would not invalidate — acceptable for a UI chip, and today every AdminOperationsActor mutation writes one.
Deployments.razor.ReloadAsync drift branch (:89–98) delegates to the probe.

Tests (InMemory factory pattern from UnsTreeTestDb.cs:144–150):

An_unchanged_watermark_reuses_the_cached_verdict — counting decorator impossible here, so assert indirectly: seed, call twice, mutate nothing ⇒ second call returns without recompose. Make the compose seam injectable (Func<OtOpcUaConfigDbContext, Task<string>> composeRevisionHash ctor parameter defaulting to the real SnapshotAndFlattenAsync(...).RevisionHash) so the test injects a counting fake ⇒ compose count == 1 after two calls.
A_new_config_edit_recomposes — add a ConfigEdit row between calls ⇒ compose count == 2, and drift flips when the fake returns a different hash.

Commit: perf(adminui): drift chip recomposes only when the config-edit watermark moves

Task 18: ScriptTagCatalog TTL snapshot cache

Classification: standard Estimated implement time: ~4 min Parallelizable with: Task 1, Task 2, Task 3, Task 6, Task 8, Task 10, Task 11, Task 12, Task 14, Task 15, Task 16, Task 19, Task 20, Task 22, Task 23, Task 24, Task 27

Files:

Modify: src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/ScriptAnalysis/IScriptTagCatalog.cs
Modify: src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/EndpointRouteBuilderExtensions.cs (:98 area — register the cache singleton)
Test: tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/ScriptAnalysis/ScriptTagCatalogTests.cs

Spec:

New ScriptTagCatalogSnapshotCache (singleton, in the same file or adjacent): holds (IReadOnlyList<ScriptTagInfo> Entries, long StampTicks); TimeProvider ctor dependency; TTL const 5 s. GetOrBuildAsync(Func<CancellationToken, Task<IReadOnlyList<ScriptTagInfo>>> build, ct) — double-checked under a SemaphoreSlim(1,1) so concurrent circuits share one rebuild. Staleness bound is deliberate and documented: completions may lag a just-saved tag by ≤ 5 s.
ScriptTagCatalog (stays scoped) takes the singleton cache; GetPathsAsync and GetTagInfoAsync source entries from cache.GetOrBuildAsync(BuildEntriesAsync, ct). GetTagInfoAsync's linear scan can stay (list is in memory now). GetEquipmentReferenceNamesAsync (2 scoped queries) is untouched.

Tests (extend ScriptTagCatalogTests.cs, which already builds the catalog over an InMemory factory):

Two_calls_within_ttl_scan_the_db_once — a counting IDbContextFactory decorator (wrap TestFactory, count CreateDbContextAsync) ⇒ second GetPathsAsync creates no context.
An_expired_ttl_rebuilds — Microsoft.Extensions.Time.Testing.FakeTimeProvider advanced 6 s ⇒ context created again, and a tag added between calls appears.

Commit: perf(adminui): 5s snapshot cache in front of the script tag catalog

Task 19: EffectiveNameGuard batch check + bulk add-references

Files:

Modify: src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/IEffectiveNameGuard.cs
Modify: src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/EffectiveNameGuard.cs
Modify: src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/UnsTreeService.cs (AddReferencesAsync :171–257)
Test: tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/Uns/EffectiveNameGuardTests.cs, tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/Uns/UnsTreeServiceReferenceTests.cs

Spec:

Add to the interface (default-implemented in terms of CheckAsync so the NoOpEffectiveNameGuard and the test FakeGuard keep compiling — or implement on both, whichever is cleaner):

/// <summary>Checks many proposed names against ONE load of the equipment's name space.
/// Returns the first collision message, or null when all are free.</summary>
Task<string?> CheckManyAsync(string equipmentId, IReadOnlyList<string> proposedEffectiveNames,
    EffectiveNameSourceKind kind, CancellationToken ct = default);

EffectiveNameGuard.CheckManyAsync: one DbContext, the same 4 per-equipment queries (:34, :41, :59, :73) loaded once, then all names checked in memory with the same collision rules/messages as CheckAsync (extract the shared in-memory check).
UnsTreeService.AddReferencesAsync: hoist the guard out of the loop — collect the effective names during the (kept) per-tag validation loop, then one CheckManyAsync(equipmentId, names, Reference, ct) before building pending. N-tag bulk add: N+ contexts + 4N queries → 1 context + 4 queries.

Tests: batch-vs-loop parity in EffectiveNameGuardTests (same collisions detected, same messages, first-collision-wins ordering); update UnsTreeServiceReferenceTests' FakeGuard and assert a 3-tag add calls the guard once.

Commit: perf(adminui): one guard round-trip per bulk reference add

Task 20: Raw tree tag-name projection

Classification: small Estimated implement time: ~3 min Parallelizable with: Task 1, Task 2, Task 3, Task 6, Task 8, Task 10, Task 11, Task 12, Task 14, Task 15, Task 16, Task 18, Task 19, Task 22, Task 23, Task 24, Task 27

Files:

Modify: src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/RawTreeService.cs (LoadDeviceChildrenAsync :163–182, LoadTagGroupChildrenAsync :184–203, BuildTagNode :302–313)
Test: tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/Uns/RawTreeServiceTests.cs

Spec: project the two tag queries to exactly what BuildTagNode consumes — read BuildTagNode first and enumerate its Tag member accesses (exploration saw TagId, Name, RowVersion + whatever else the method touches); introduce a private record for the projection and a BuildTagNode overload taking it. TagConfig (nvarchar(max) JSON) must not be in the SELECT. Existing RawTreeServiceTests green.

Commit: perf(adminui): raw tree renders tag rows from a projection without TagConfig

Task 21: Rename script-scan filters in SQL

Classification: standard Estimated implement time: ~4 min Parallelizable with: Task 4, Task 7, Task 9, Task 13, Task 17, Task 25 (blocked by Task 20 — same file)

Files:

Modify: src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/RawTreeService.cs (BuildScriptReferenceWarningAsync :1415–1436)
Test: tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/Uns/RawTreeServiceRenameWarningTests.cs

Spec: push the substring match into the query so unmatched script bodies never leave the database: build a disjunction predicate over oldPaths by composing Expression.OrElse around s.SourceCode.Contains(path) (EF translates string.Contains to CHARINDEX on SQL Server and evaluates in-memory on InMemory — both fine; write a small private static Expression<Func<Script, bool>> AnyPathContained(IReadOnlyList<string> paths) helper with a shared parameter). Project { Name } only. Guard the degenerate case (oldPaths.Count == 0 returns null — already there). Keep the ordinal semantics note: SQL Server CHARINDEX under the column's collation may be case-insensitive where StringComparison.Ordinal was not — RawPaths are operator-authored identifiers; accept the (strictly wider) match and say so in a comment, since a false-positive warning is harmless and a false negative was the old in-memory scan's real risk profile anyway. Existing RawTreeServiceRenameWarningTests must pass unmodified (they pin match behaviour on InMemory, where Contains stays ordinal).

Commit: perf(adminui): rename script scan filters server-side, stops shipping every script body

Task 22: Pooled context factory + RoleGrants off the circuit-scoped context

Files:

Modify: src/Core/ZB.MOM.WW.OtOpcUa.Configuration/ServiceCollectionExtensions.cs (:19–37)
Modify: src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Services/LdapGroupRoleMappingService.cs
Test: tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests/ (locate the existing LdapGroupRoleMappingService tests, adjust construction)

Spec:

AddDbContextFactory → AddPooledDbContextFactory (:25). Precondition to verify before the swap: OtOpcUaConfigDbContext has a single public DbContextOptions-taking ctor and no injected scoped services (exploration saw the standard shape at UnsTreeTestDb.cs construction) — if anything else shows up, stop and surface it.
Refactor LdapGroupRoleMappingService from a ctor-injected scoped OtOpcUaConfigDbContext to IDbContextFactory<OtOpcUaConfigDbContext> — create + await-dispose a context per method (ListAllAsync/CreateAsync/DeleteAsync and any siblings). This unpins the RoleGrants circuit's long-lived context AND makes the service safe under Blazor's concurrent renders. Then delete the scoped CreateDbContext() bridge at :30 if LdapGroupRoleMappingService was its only consumer — grep for other ctor-injected OtOpcUaConfigDbContext takers first (AuthEndpoints.LoginAsync reaches the service, not the context, per the comment); if another consumer exists, keep the bridge and note it.
Fix the three stale "pooled factory" comments exploration found (IScriptTagCatalog.cs:82–84, EffectiveNameGuard.cs:15, UnsTreeTestDb.cs:142–143) — they become true; reword to state pooling is now real.

Commit: perf(adminui): pooled DbContext factory; role-grants reads per-call, not per-circuit

Task 23: Fleet/Hosts poll overlap guards

Files:

Modify: src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Fleet.razor (:102–171)
Modify: src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Hosts.razor (:316–400)

Spec: in both timer callbacks add a int _tickRunning overlap guard (if (Interlocked.Exchange(ref _tickRunning, 1) == 1) return; … finally Volatile.Write(ref _tickRunning, 0);) so a slow DB tick never stacks behind the next, and pass a page-lifetime CancellationTokenSource token into the EF calls, cancelled in Dispose. No behaviour change otherwise; no page tests exist (verified) — compile + manual /run sanity is the gate.

Commit: fix(adminui): fleet/hosts poll ticks no longer overlap; cancel on dispose

Stream F — Residual fan-out waste + blocking I/O (review findings 07/08)

Current facts: the mux Tell from DriverHostActor.ForwardBatchToMux (:1137–1170) is one envelope per burst — with Task 2 in place, host-side interest gating is near-zero value (deliberately skipped; the mux already early-outs and now emits one message per interested subscriber). MQTT raises per message with no unchanged-value suppression (MqttSubscriptionManager.Deliver :703–753 → Raise :762–784) while LastValueCache.Update runs immediately before the raise. The comms-loss fan (FanCommsLossToDependencyMux :1293–1312) scans the whole (driverId, rawPath) map and Tells one per-value message per ref; OnDriverConnectivityChanged (:1219–1250) scans the alarm map the same way. The one remaining hot-mailbox blob read is the Direct apply path (LoadArtifactBlob :2034–2059 called from ReconcileDriversForApply :2080–2084; second copy at PushDesiredSubscriptions :2380–2384 on the restore path); UpsertNodeDeploymentState's sync SaveChanges (≤3 small row writes per apply) and the PreStart-only bootstrap reads are recorded follow-ups, not in scope — they are bounded per-deploy costs, not per-value costs. MTConnect's /sample chunks go byte[] → string → trimmed string → XDocument (MultipartStreamReader.ReadNextPartAsync :655, MTConnectAgentClient.cs:254) while a Stream overload already exists (MTConnectStreamsParser.cs:127–142); serving on-demand reads from the live index was considered and rejected in-code (MTConnectDriver.cs:652–667 — a second writer racing the pump) — honor that.

Task 24: MQTT unchanged-value suppression

Files:

Modify: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt/MqttSubscriptionManager.cs (Deliver :703–753, Raise :762–784, DegradeTopic :679–683)
Test: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests/MqttSubscriptionManagerTests.cs

Spec: before Values.Update(name, snapshot) in Deliver, read the prior snapshot (Values.Read(name) — it returns BadWaitingForInitialData for unseen, which never equals a real value+status, so the first message always raises); when prior.Value equals snapshot.Value (use the same structural comparison PollGroupEngine.ValuesAreDifferent uses — replicate the array-aware compare locally or extract; MQTT values are scalars/strings/byte[] from JSON) and prior.StatusCode == snapshot.StatusCode, still Values.Update (freshen timestamps for reads) but skip Raise. DegradeTopic's Bad raise survives (status differs). Timer-republishing brokers stop re-triggering the whole downstream fan per identical message.

Tests (write first, NewManager harness :29):

An_identical_republish_raises_nothing — same topic/payload twice ⇒ one OnDataChange; read still returns the value with the newer timestamp.
A_changed_value_after_a_suppressed_republish_raises (falsification control).
A_status_transition_still_raises — degrade after a good value ⇒ Bad raised.
Sparkplug path: confirm SparkplugIngestor raises through its own seam (MqttDriver.cs:1198) — if it bypasses Deliver, it is out of scope; note it either way in the commit body.

Commit: perf(mqtt): suppress unchanged-value republish fan-out

Task 25: DriverHostActor per-driver index + batched comms-loss fan

Classification: high-risk Estimated implement time: ~4 min Parallelizable with: Task 4, Task 7, Task 9, Task 13, Task 17, Task 21 (blocked by Task 2 — the mux batch message must exist)

Files:

Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs (ApplyDesiredSubscriptions :2506–2549, OnDriverConnectivityChanged :1219–1250, FanCommsLossToDependencyMux :1293–1312)
Test: tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorLiveValueTests.cs (or a new DriverHostActorCommsLossTests.cs beside it, using the RuntimeActorTestBase + SeedV3Deployment harness)

Spec:

During ApplyDesiredSubscriptions (where _nodeIdByDriverRef / _alarmNodeIdByDriverRef are rebuilt), also build Dictionary<string /*driverId*/, List<...>> secondary indexes for both maps (same value tuples, grouped). Rebuilt whole per apply like the primaries — no incremental maintenance.
OnDriverConnectivityChanged and FanCommsLossToDependencyMux iterate the indexed driver's list instead of scanning every entry.
Batch the comms-loss mux fan: instead of one AttributeValuePublished Tell per ref, send one DriverInstanceActor.AttributeValueBatchPublished carrying every ref as a AttributeValueItem(rawPath, null, BadNoCommunication, ts) — the mux batch path (Task 2) fans it per interest. One driver dying with 5k tags: 5k Tells → 1.

Tests (write first): Comms_loss_reaches_the_mux_as_one_batch_of_bad_items — seeded deployment with ≥2 tags on the driver, trigger the connectivity-lost path, probe registered for those refs expects one DependencyValueBatch whose items are all BadNoCommunication; a second driver's tags are untouched (index correctness — falsification control with two drivers).

Commit: perf(runtime): per-driver index for connectivity fans; comms-loss is one batch

Task 26: Direct-apply artifact read off the mailbox

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: none (blocked by Task 25 — same file)

Files:

Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs
Test: tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorApplyingStashTests.cs (or a sibling file, same harness)

Spec:

The Direct-mode dispatch path currently calls the sync LoadArtifactBlob (:2034–2059, EF read of a potentially multi-MB gzip blob) inline on the mailbox inside ReconcileDriversForApply/ApplyAndAck. Restructure Direct dispatch to the FetchAndCache shape that already exists in this file: on dispatch, pipe Task.Run(() => LoadArtifactBlob(deploymentId)) via .PipeTo(Self, success: bytes => new <existing-or-new fetched-for-apply message>, failure: ...), enter the Applying state (which already stashes concurrent dispatches — Task history: item-2's stash + wedge watchdog :1063–1089, :1868–1895), arm the same wedge watchdog, and run the existing HandleFetchedForApply-style continuation when the bytes arrive. Reuse the existing messages/continuations wherever they fit (HandleFetchedForApply :1789–1862 already handles in-hand bytes + parse + reconcile + acks); the goal is ONE apply continuation for both modes, with only the byte-source differing. A failed load fails the apply exactly like a failed fetch (#485 semantics: no bytes is no answer — never tear down).
RestoreApplied's boot-path read (:2330) and PushDesiredSubscriptions' restore read (:2380–2384) may stay sync — they run at PreStart/restore, not in live traffic; add a comment saying so. TryRecoverFromStale/UpsertNodeDeploymentState sync writes are recorded follow-ups (bounded per-apply row writes), also commented.
Behavioural pins that must stay green: the whole DriverHostActorApplyingStashTests suite (stash/dedup/wedge), DriverHostActorReconcileTests, DriverHostActorBatchFanOutTests, and the Fleet ack flow tests.

Tests (write first): A_slow_direct_artifact_read_does_not_block_the_host_mailbox — gate the blob read (inject a gated IDbContextFactory or an artifact-load seam — follow the GatedArtifactCache pattern from DriverHostActorAsyncCacheWriteTests.cs:44), dispatch a Direct apply, and while the read is gated assert the host still answers an unrelated message (e.g. forwards a value batch to the publish probe); release ⇒ apply completes and acks Applied.

Commit: perf(runtime): direct-mode artifact read piped off the host mailbox

Task 27: MTConnect sample chunks parse from bytes

Classification: standard Estimated implement time: ~3 min Parallelizable with: Task 1, Task 2, Task 3, Task 6, Task 8, Task 10, Task 11, Task 12, Task 14, Task 15, Task 16, Task 18, Task 19, Task 20, Task 22, Task 23, Task 24

Files:

Modify: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect/MTConnectAgentClient.cs (:254, MultipartStreamReader.ReadNextPartAsync :616–655)
Modify: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect/MTConnectStreamsParser.cs (byte-span entry point)
Test: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests/MTConnectStreamsParseTests.cs

Spec: stop materialising each streamed chunk as a string: ReadNextPartAsync returns the raw byte[] (rename/overload; the string conversion at :655 moves to the one caller that still needs it, if any); add MTConnectStreamsParser.Parse(ReadOnlyMemory<byte> xml) that wraps the bytes in a MemoryStream and reuses the existing Stream overload (:127–142), handling the BOM the string path trimmed (XDocument.Load handles a UTF-8 BOM natively — verify with a BOM fixture). The /sample loop at :254 uses the byte path. The DOM walk itself stays (a full XmlReader streaming rewrite of Build is a recorded follow-up — this task removes the double string copy, which is the allocation bulk). The on-demand /current read design (:652–667 throwaway index) is deliberately untouched.

Tests: extend MTConnectStreamsParseTests with byte-input theories over the same fixtures (File.ReadAllBytes), incl. a BOM-prefixed and a trailing-newline-suffixed variant (the exact framing residue the string path trimmed at :110).

Commit: perf(mtconnect): sample chunks parse from bytes, no per-chunk string copies

Task 28: Full sweep + plan close-out docs

Classification: standard Estimated implement time: ~5 min Parallelizable with: none (blocked by all prior tasks)

Files:

Modify: docs/plans/2026-08-15-perf-roadmap-remainder.md (this file — outcomes section)
Modify: CLAUDE.md (only where behaviour documented there changed: the ContinuousHistorization table already updated in Task 5; check the drivers/GetHealth and ACL sections for statements the work invalidated)
Modify: docs/security.md (decision-cache sharding note), docs/DriverLifecycle.md (fixed-rate pacing + overrun counter, async unsubscribe), docs/ScriptedAlarms.md (dirty-set coalescing + the write-behind-rejected rationale)

Spec:

dotnet build ZB.MOM.WW.OtOpcUa.slnx — zero errors/warnings-as-errors.
dotnet test ZB.MOM.WW.OtOpcUa.slnx — full unit sweep green (integration projects build-only per repo convention; DB-backed suites need the central SQL Server at 10.100.0.35,14330 reachable — if it is not, run every non-DB project individually and record which were skipped).
Append an "## Outcomes" section here: per task, one line — done/deviation. Record the follow-up register explicitly: HistorianGateway multi-tag batch RPC (cross-repo); S7 multi-var PDUs for span leftovers; UpsertNodeDeploymentState/TryRecoverFromStale sync writes; MTConnect XmlReader streaming Build; driver-level meter for the engine overrun counter; host-side mux interest gating (skipped — batch envelope made it moot).
Docs edits per Files list.

Commit: docs: perf roadmap remainder — outcomes + follow-up register

Dependency graph (execution order)

Wave 1 (parallel, disjoint files): 1, 2, 3, 6, 8, 10, 11, 12, 14, 15, 16, 18, 19, 20, 22, 23, 24, 27 Wave 2 (parallel, after their blockers): 4 (←1,2,3), 7 (←6), 9 (←8), 13 (←12), 17 (←16), 21 (←20), 25 (←2) Wave 3: 5 (←3,4), 26 (←25) Wave 4: 28 (←all)

Not covered by this plan (explicit): live docker-dev gating of the historian and apply-path changes (run the existing live-gate recipes afterwards, as items 1–2 did); the HistorianGateway contract change; any change to redundancy-state, the deploy classifier, or the ACL trie itself.

Outcomes

All 28 tasks landed on perf/roadmap-remainder. Close-out sweep (2026-08-16): dotnet build ZB.MOM.WW.OtOpcUa.slnx — 0 errors, 896 warnings (all pre-existing: OTOPCUA0001 on deliberate unwrapped driver calls in tests, xUnit1051, two CS8602); full unit sweep across all 43 *.Tests projects — 10 310 passed, 0 failed, 19 skipped. *.IntegrationTests are build-only by repo convention, and the DB-backed suites (*.Configuration.Tests, *.Admin.Tests, *.Server.Tests) ran for real against the reachable central SQL Server at 10.100.0.35,14330 — nothing was skipped for want of a database. The three known load-related flakes (DriverInstanceActorCommsLossWatchdogTests, RawTreeServiceRenameWarningTests, RouteNativeAlarmAck_on_non_primary_is_dropped) all passed on the first run; no project needed an isolated re-run.

Note for anyone re-running the sweep by globbing tests/*/*.Tests: two directories under tests/Drivers/ — …Driver.Historian.Wonderware.Tests and …Driver.Historian.Wonderware.Client.Tests — are empty leftovers of the retired Wonderware backend, holding only stale bin/obj and no .csproj. They are not in the solution. A glob-driven sweep reports MSB1003 for them; that is the glob, not a failure.

Every task's spec review and code review was run and closed before the task was marked done; what follows records only the outcome and any deviation from the plan as written.

#	Task	Commit(s)	Outcome / deviation
1	Outbox batch append with group commit	`6ff64ca0` + `73205865`	Done. The follow-up commit made the batch bookkeeping incremental and replaced the durability test with a genuine group-commit pin (a second reader). Deviation: the second-reader durability test is fragile on macOS — see the register.
2	Batched mux→subscriber delivery, single-eval virtual tags	inside `140bf0c1`	Done. ⚠️ Its 7 Runtime files were swept into Task 22's commit — see the mixed-commit note below. Both code review Importants (a yielding batch-test fake, mid-batch exception policy) were folded into Task 4's brief and closed there.
3	Parallel drain with bounded concurrency	`a76b4e07`	Done, approved as specified (`DrainConcurrency`, default 8). One inherited pre-existing defect recorded in the register (shutdown cancellation counted as a per-tag failure).
4	Recorder batched ingest — one fsync per burst	`384568b4` + `451273c8`	Done. The follow-up pinned the append-failure branch via three independent mutations and corrected the loss-accounting docs; the recorder's `64` literals were aligned to `1024` there.
5	Historian defaults, config drift, docs	`0298a0b9`	Done. Added the `Historian.md` key table and recorded the cross-repo batch-RPC ceiling. Minor: the pinning test's doc comment overclaims appsettings-drift coverage.
6	Per-session decision-cache shards	`debd44c2`	Done. Replaces the flat cache + wholesale `Clear` with per-session shards. Register: `_globalCount` drift under races; no concurrency stress test.
7	Authorizer hot path — lookup first, lazy deny reasons	`d8a17f2e`	Done. Cache lookup moved ahead of scope construction; deny reasons are built only on the deny path; revocation/expiry stay ahead of the cache.
8	Browse and TranslateBrowsePath compaction	`541a223b` + `aad7583b`	Done — O(n) in-place compaction. The follow-up test pins survivor order through the compaction (falsified red first). Minor: the `FanOutTargetIdList` fake couples to the SDK's single-`Add` contract.
9	RevalidateMonitoredItems dedup	`fbe8aaad`	Done. Deviation from spec, deliberately: the dedup key is a realm-qualified 3-tuple `(session, node, realm)` rather than the spec's 2-tuple — the realm is carried explicitly at every sink seam and dropping it could collapse two distinct nodes. Register: the empty-identifier pattern is worth tightening.
10	Modbus `InitializeAsync` transport leak	`c3970a2e` + `409a58b6`	Done. The follow-up added the lifecycle gate and replaced a vacuous reprobe test with a non-vacuous one.
11	PollGroupEngine fixed-rate pacing, overrun counter, async unsubscribe	`9cf400f1` + `485f4ea1`	Done. The follow-up widened the fast-reader control's tolerance to one scheduler blip (200 ms interval, ≤1). Register: `NoteOverrun` conflates slow-success with slow-failure; `backoffCap` needs a constructor doc note.
12	S7 narrow-scalar span codec	`8d91833f`	Done. Plan error corrected: the plan's spec mentioned `SByte`; S7 has no such scalar in this codec's domain, so it was not implemented. The omission is intentional, not a gap.
13	S7 span planner + grouped `ReadAsync`	`39f67a0f`	Done. Spec review accepted the demotion rule and the 6th (factory-binding) file beyond the plan's file list. ⚠️ The `16` / `220` span defaults are not live-gated — see the register.
14	AbCip bounded-parallel reads	`d6a4fbd6` + `d5f573b0`	Done (`MaxConcurrentReads`, default 8). The follow-up added a per-device `ConnectGate` single-flight so cold-connect backoff survives the parallel read window. Minor: a stale `EvictRuntime`-race comment (~L1161); test timing margin is tight on slow CI.
15	Dirty-set coalescing in `ScriptedAlarmEngine`	`5df793e7`	Done. Write-behind persistence was considered and rejected — see `ScriptedAlarms.md`. Register: `TrackBackgroundTask` registration-ordering gap (pre-existing), non-volatile `_disposed`.
16	Deployments list projection	`dbc9a132`	Done — `/deployments` no longer materialises artifact blobs to render a list.
17	Drift watermark probe	`684e59d8`	Done. Design deviation, accepted: the probe is a `GroupBy(_ => 1)` single-round-trip aggregate rather than the plan's separate count+max queries. Register: that aggregate is verified only against EF InMemory, so the perf claim is unverified against real SQL Server.
18	ScriptTagCatalog TTL snapshot cache	`3795899d`	Done — 5 s snapshot cache.
19	EffectiveNameGuard batch check + bulk add-references	`fa36d64e`	Done. Behaviour deviation recorded: for a mixed-failure batch the reported error is now the batch's first failure rather than the per-row one, a minor error-precedence shift.
20	Raw tree tag-name projection	`ba977dbe`	Done — tag rows render from a projection that omits `TagConfig`.
21	Rename script-scan filters in SQL	`fe8d132c` + `ca87329a`	Done. The follow-up added `Distinct` + 200-term chunking with `ScriptId`-keyed cross-batch dedup. Register: the `CHARINDEX`/collation widening needs a SQL-Server-backed test; the worst-case 250 sequential round-trips are undocumented.
22	Pooled context factory + RoleGrants off the circuit-scoped context	`140bf0c1`	Done. ⚠️ This commit also carries Task 2's files — see the note below.
23	Fleet/Hosts poll overlap guards	`4b6a93bb` + `63081d52`	Done. The follow-up gave the Fleet page the same async timer disposal the Hosts page has.
24	MQTT unchanged-value suppression	`2c708f55` + `50035b36`	Done. The follow-up pinned the status-code half of the suppression (with falsification) and documented that Sparkplug bypasses it.
25	`DriverHostActor` per-driver index + batched comms-loss fan	`b6275f43`	Done. Consequence for readers: after this, no production code sends the per-value `AttributeValuePublished` — the mux's per-value receive is test-only. Minor: cross-sender `RegisterInterest` ordering in one test is not structurally guaranteed; one empty-bucket guard is unreachable.
26	Direct-apply artifact read off the mailbox	`5b92eb0f`	Done. A `BusLogging` comment dispute in review was resolved in the implementer's favour. Register: the Direct-mode SQL read carries no timeout/cancellation; `Bootstrap`'s orphan replay exits `PreStart` in `Applying`.
27	MTConnect sample chunks parse from bytes	`34f836e0`	Done — no per-chunk string copies.
28	Full sweep + plan close-out docs	this commit + the comment-only follow-up	Done. Sweep green; docs updated (`CLAUDE.md`, `security.md`, `DriverLifecycle.md`, `ScriptedAlarms.md`).

The mixed commit `140bf0c1`

Task 22's implementer ran a bare git commit (no pathspec) while Task 2's 7 Runtime files were already staged in the shared index. Both tasks' content therefore lives in 140bf0c1, under Task 22's message (perf(adminui): pooled DbContext factory; role-grants reads per-call, not per-circuit), which names only Task 22's half.

Both halves were fully spec- and code-reviewed at that commit — nothing is unreviewed, and nothing is missing from the branch. Decision: the history stays as it is. No empty marker commit was created; an empty commit would add a second misleading object rather than correct the first. This note is the record. If you are archaeologising Task 2 (batched mux→subscriber delivery, single-eval virtual tags), look in 140bf0c1, not for a commit whose message mentions it.

Follow-up register

Open items the work surfaced but deliberately did not close. None blocks the branch.

Annotated 2026-08-17 by the perf/followups branch (plan: 2026-08-17-perf-followups.md). Items it closed carry a — closed by <sha>, 2026-08-17 suffix; items with no suffix remain open. Annotations are append-only — the original text of each item is untouched, including where the follow-up work reached a different conclusion than the item predicted.

Cross-repo

HistorianGateway needs a multi-tag batch WriteLiveValues RPC. Client 0.3.0 exposes only a single-tag call, so bounded parallelism (Task 3's DrainConcurrency) is the ceiling — the drain cannot become one round-trip per batch until the gateway contract grows the RPC. Already recorded in docs/Historian.md.

Drivers

S7: multi-variable PDUs for span leftovers. The Task 13 planner groups contiguous reads into one PDU per span; tags that do not fall in a span are still one PDU each. A multi-var PDU would absorb them. — closed by 20022083, 2026-08-17 (per-chunk fallback: S7.Net throws for the whole multi-var request, so a bad item fails its chunk, not the pass; Timer/Counter areas are excluded on an int-vs-ushort box-parity difference).
⚠️ S7 span planner has no live gate. The 16 (max gap) / 220 (max span bytes) defaults were chosen analytically and pinned only by unit tests. Run the planner against the s7_1500 fixture (lmxopcua-fix up s7 s7_1500, S7_SIM_ENDPOINT=10.100.0.35:1102) before trusting them on real hardware. — closed by 57c66416, 2026-08-17 (LIVE GATE PASSED vs 10.100.0.35:1102; 3 wire requests for 10 tags with spans on vs 10 with spans off).
SpanReadMaxSpanBytes has no validation guard. A value above the negotiated PDU size is accepted at config time and fails at read time. — closed by acb37724, 2026-08-17.
MTConnect Build could stream through XmlReader. Task 27 removed the per-chunk string copies; the document build still materialises. — closed by cda82c07, 2026-08-17 (566/566 parity, ~19% allocation win; one disclosed error-ordering change on a double-broken document).
No driver-level meter for the PollGroupEngine overrun counter. The counter exists and is observable in-process but is not exported per driver. — closed by e22ecea0, 2026-08-17 (tag keys moved to snake_case consts in 97902d68 after review).
NoteOverrun conflates a slow success with a slow failure — a poll that overran because the device was slow and one that overran because it timed out increment the same counter. Worth a doc caveat rather than a code change. — closed by e22ecea0, 2026-08-17 (doc caveat, as proposed).
backoffCap wants a constructor doc note so a future caller does not have to read the loop to learn its units and its interaction with the poll interval. — closed by e22ecea0, 2026-08-17.

Runtime / deploy path

UpsertNodeDeploymentState and TryRecoverFromStale still write synchronously on the actor thread. Both are commented as such in DriverHostActor; neither is on the hot path that Task 26 moved off the mailbox.
⚠️ Direct-mode LoadArtifactBlob carries no timeout or cancellation token (Task 26 review, Important). The read now runs off the mailbox so it cannot wedge the actor, but a hung SQL read still pins a thread-pool thread indefinitely unless the ConfigDb connection string's command timeout bounds it. Confirm that it does — the code relies on it and does not assert it. — closed by dba5f806, 2026-08-17 (explicit 30 s SetCommandTimeout behind an IsRelational() guard; EF 10.0.7 verified empirically to reset it on return to the pool, so there is no pool pollution).
The historian drain's catch swallows a shutdown OperationCanceledException as a per-tag failure when other tags in the same pass succeeded, spuriously incrementing _tagWriteFailures (Task 3 review; pre-existing, inherited). Rethrowing a genuine ct cancellation is the fix. — closed by 7870c91f, 2026-08-17 (the red run proved the bug also discarded healthy tags' samples at shutdown, not just the counter).
Host-side mux interest gating was skipped — the batch envelope (Task 2) made the saving it would have bought immaterial. Recorded so it is not re-proposed as an oversight.

ACL

Task 6: _globalCount can drift under concurrent shard eviction, and there is no concurrency stress test. Drift is bounded and only affects when trimming triggers, not correctness of any decision. — closed by d8849345, 2026-08-17 (the planned counter swap alone was insufficient: a seal sentinel plus a TryBookInsert compare-exchange were both needed, each proven necessary by a 6/6-red falsification run).
Task 9: the empty-identifier pattern should be tightened — the dedup key tolerates an empty node identifier rather than rejecting it. — closed by c4d8e4a2, 2026-08-17 (pattern-form guard; the bypass is pinned by a clock-call count).
Task 17: the GroupBy(_ => 1) aggregate is unverified against real SQL Server. It is covered only by EF InMemory, which does not exercise the provider's translation, so the single-round-trip claim is untested where it matters. — closed by c149410b, 2026-08-17 (aggregate confirmed live against SQL Server; the suite skips cleanly offline via an xunit-v2 SqlServerFactAttribute).
Task 21: the CHARINDEX/collation widening needs a SQL-Server-backed test, and the worst-case 250 sequential round-trips (a rename touching the chunking limit) should carry a one-line comment at the call site. — closed by c149410b, 2026-08-17 (collation widening confirmed live; round-trips comment added).

Alarms

Task 15: TrackBackgroundTask has a registration-ordering gap (pre-existing) — a task can complete before it is registered. Also _disposed is non-volatile. — closed by 5ae43023, 2026-08-17 (a TCS-placeholder register-first factory closes the ordering gap; _disposed made volatile).

Test-suite reliability (affects how reliably a regression is caught, not shipped behaviour)

Task 1's second-reader durability test is fragile on macOS — it depends on filesystem flush visibility to a second reader, which the platform does not guarantee on the same timescale as Windows/Linux. — closed by 6b46c485, 2026-08-17 (a robust bounded wait replaced the fixed sleep; the anticipated macOS skip turned out to be unnecessary — verified empirically, the reader does become visible).
The #491 falsification control is nondeterministic under load. It relies on a ThreadStatic actor cell, so under a loaded sweep the intended-red mutation can occasionally pass. The test is still worth keeping; do not read a single green run of it as proof the regression is caught. — closed by 6b46c485, 2026-08-17 (documented in the test file as an honest nondeterminism note; the nondeterminism itself is inherent and remains).

Accepted-not-fixed at the final integration review (2026-08-15) — carried knowingly, none blocking: the poll pacing behaviour change ships with no production consumer of the overrun counter (the per-driver meter follow-up above is the mitigation); no live gate was run for this branch (docker-dev recipe: 2026-08-01-continuous-historization-live-gate.md); _globalCount drift is negative-only in direction, so it can only trim early, never late; ScriptTagCatalogSnapshotCache expires on wall-clock TTL; the S7 span reader stamps a partial span when the PDU comes back short; MTConnect's IsFramingOnly fast path is ASCII-only; and a minor Razor exposure where StateHasChanged runs inside a finally.

Annotation, 2026-08-17 (append-only). Two of the accepted-not-fixed items above have since been retired by perf/followups: the overrun counter now has a production consumer (the per-driver meter, e22ecea0 + 97902d68), and this branch's code state has been live-gated — S7 span reads against the s7_1500 fixture (57c66416) and a full docker-dev regression on the merged remainder (2026-08-17, no commit). The _globalCount drift item is likewise closed (d8849345). The rest of the paragraph stands as written.

92 KiB Raw Blame History Unescape Escape

Perf Roadmap Remainder Implementation Plan

Stream A — Historian group commit + parallel drain (review finding 03, Critical)

Task 1: Outbox batch append with group commit

Task 2: Batched mux→subscriber delivery, single-eval virtual tags

Task 3: Parallel drain with bounded concurrency

Task 4: Recorder batched ingest — one fsync per burst

Task 5: Historian defaults, config drift, docs

Stream B — ACL cache re-shape (review finding 04, High)

Task 6: Per-session decision-cache shards

Task 7: Authorizer hot path — lookup first, no closures, lazy deny reasons

Task 8: Browse and TranslateBrowsePath compaction

Task 9: RevalidateMonitoredItems dedup

Stream C — Driver round-trips (review finding 05, High)

Task 10: Modbus InitializeAsync transport leak

Task 11: PollGroupEngine fixed-rate pacing, overrun counter, async unsubscribe

Task 12: S7 narrow-scalar span codec

Task 13: S7 span planner + grouped ReadAsync

Task 14: AbCip bounded-parallel reads

Stream D — Alarm-engine coalescing (review finding 06, High)

Task 15: Dirty-set coalescing in ScriptedAlarmEngine

Stream E — AdminUI hot spots (review finding 09, Medium)

Task 16: Deployments list projection

Task 17: Drift watermark probe

Task 18: ScriptTagCatalog TTL snapshot cache

Task 19: EffectiveNameGuard batch check + bulk add-references

Task 20: Raw tree tag-name projection

Task 21: Rename script-scan filters in SQL

Task 22: Pooled context factory + RoleGrants off the circuit-scoped context

Task 23: Fleet/Hosts poll overlap guards

Stream F — Residual fan-out waste + blocking I/O (review findings 07/08)

Task 24: MQTT unchanged-value suppression

Task 25: DriverHostActor per-driver index + batched comms-loss fan

Task 26: Direct-apply artifact read off the mailbox

Task 27: MTConnect sample chunks parse from bytes

Task 28: Full sweep + plan close-out docs

Dependency graph (execution order)

Outcomes

The mixed commit 140bf0c1

Follow-up register

92 KiB

Raw Blame History

The mixed commit `140bf0c1`