Files
lmxopcua/docs/plans/2026-08-15-value-pipeline-batching.md
T
Joseph Doherty aec5ce565a docs: record value-pipeline batching (plan 2026-08-15)
- docs/v2/Galaxy.Performance.md gains a "Value-path batching (2026-08-15)"
  section: the batch seams (IBatchSubscribable / PollGroupEngine onBatch ->
  AttributeValueBatchPublished -> AttributeValueBatchUpdate -> sink WriteValues
  with 512-chunked single-lock batches in the node manager), the invariants (#519
  status codes verbatim, per-ITEM metering, Modbus deadband honored via the
  driver-filtered buffer, the ValueWriteBatchException partial-failure contract),
  the RealmKey re-key, and the pinned dispatcher's throughput 1 -> 100.

- The plan doc gains a "Follow-ups (recorded, not implemented)" section: the
  PollGroupEngine unconditional-collection trap (a future per-tag filter in
  S7/AbCip/FOCAS would silently reintroduce the deadband-bypass bug), a two-level
  driverId -> (rawPath -> refs) routing map, null/empty guard consistency on the
  internal batch messages, a possible DriverHostActorLiveValueTests merge, and the
  pre-existing aborted-cycle tail-loss on the three unbuffered drivers.

- tasks.json: all 13 tasks completed.

CLAUDE.md needed no edit: its only statement in this area ("One driver publish for
a RawPath fans (in DriverHostActor) to the raw NodeId AND every referencing UNS
NodeId") is still true — batching changed the message granularity, not the
fan-out semantics.

Verification: dotnet build ZB.MOM.WW.OtOpcUa.slnx clean (0 errors, no new warning
classes, zero warnings in any production file). 9 unit projects green, 3470
passed / 5 skipped / 0 failed: Core.Abstractions 305, Core 351+1s, Commons 348,
Runtime 575+3s, OpcUaServer 424+1s, Modbus 399, S7 302, AbCip 410, FOCAS 356.
No DB-backed suite was run — tests/Server/ZB.MOM.WW.OtOpcUa.Server.Tests does not
exist, and nothing this branch touches is covered by the SQL-backed projects.
2026-08-15 13:10:55 -04:00

40 KiB
Raw Blame History

Value-Pipeline Batching Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.

Goal: Batch the hot value path end to end — one message per poll cycle instead of one per tag at every hop (driver → DriverInstanceActor → DriverHostActor → OpcUaPublishActor → node manager), one SDK-lock acquisition per batch instead of per value, and zero per-write key allocations in the node manager.

Architecture: A new optional IBatchSubscribable capability lets a driver raise one OnDataChangeBatch per production burst (poll cycle); PollGroupEngine gains a batch callback so the four engine-based poll drivers get it for free. DriverInstanceActor forwards one AttributeValueBatchPublished per burst (wrapping per-tag events from non-batch drivers as single-item batches); DriverHostActor resolves the whole batch's NodeRealmRefs and sends ONE AttributeValueBatchUpdate to OpcUaPublishActor, plus ONE batch Tell to the dependency mux. The sink gains WriteValues(IReadOnlyList<ValueWrite>) (default-interface-method fallback loops WriteValue, so fakes/Null sink are untouched); OtOpcUaNodeManager overrides it taking Lock once per batch, and its node maps are re-keyed by a (realm, id) struct so the per-write new NodeId(...).ToString() allocations disappear. The pinned dispatcher's throughput rises from 1 to 100 (same single thread, fewer mailbox re-schedules).

Tech Stack: C# / .NET 10, Akka.NET (classic actors), OPC Foundation UA .NET Standard, xUnit + Shouldly, Akka.TestKit.

Branch: perf/value-pipeline-batching (off master).


Context for implementers (read before any task)

Verified facts about the current code — do not re-derive, and do not violate the invariants:

  • The per-tag path today: PollGroupEngine.PollOnceAsync (src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/PollGroupEngine.cs:202-226) diffs per tag and invokes _onChange per changed tag. Drivers forward that to ISubscribable.OnDataChange (per-tag event, src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/ISubscribable.cs:36). DriverInstanceActor.HandleSubscribeAsync attaches _dataChangeHandler at DriverInstanceActor.cs:880 (one DataChangeForward Tell per tag), OnDataChangeForward (:1138-1143) Tells parent one AttributeValuePublished per tag. DriverHostActor.ForwardToMux (DriverHostActor.cs:976-1006) Tells the mux per value and Tells OpcUaPublishActor one AttributeValueUpdate per NodeRealmRef (UNS fan-out at :997-999). OpcUaPublishActor.HandleAttributeUpdate (OpcUaPublishActor.cs:319-331) calls _sink.WriteValue per message. OtOpcUaNodeManager.WriteValue (OtOpcUaNodeManager.cs:508-533) takes the manager-wide lock (Lock) per value and allocates via MapKey(realm, id) = new NodeId(id, ns).ToString() (:133).
  • Invariants to preserve:
    1. #519 status codes ride verbatim — the full uint status code, sub-code included, must reach the SDK unchanged at every hop.
    2. Timestamps are carried from the driver — never stamp DateTime.UtcNow on the value path.
    3. PollGroupEngine's no-overlap guarantee (delay-after-completion, one loop task per subscription) must not change in this plan.
    4. Per-tag OnDataChange keeps working — Galaxy's cold-read path and driver CLIs subscribe it. A batch-capable driver raises BOTH events (batch once per cycle + per-tag per item); a consumer subscribes to exactly ONE of the two. DriverInstanceActor prefers batch when the driver implements IBatchSubscribable.
    5. DriverInstanceActor.DetachSubscription (:934-943) must detach the batch handler too — the detach is load-bearing (the IDriver outlives the actor across respawns; see CLAUDE.md §DeployWatcher).
    6. The mux's per-item receive stays — DriverHostActor.FanCommsLossToDependencyMux (:1135-1142) fabricates per-item AttributeValuePublished messages; DependencyMuxActor keeps that receive and gains a batch receive alongside.
    7. OpcUaPublishActor keeps its per-item AttributeValueUpdate receive — VirtualTagHostActor and the alarm paths still send it. The batch message is additive.
    8. The pinned dispatcher stays pinned — the OPC UA SDK requires a single thread. Only throughput changes.
    9. _writes increments (OpcUaPublishActor.cs:324) and the OtOpcUaTelemetry.OpcUaSinkWrite meter must count per-VALUE (per item), not per batch, so dashboards don't drop by 100×.
  • Message-record homes: AttributeValuePublished is declared in DriverInstanceActor (:109); AttributeValueUpdate in OpcUaPublishActor (:52); NodeRealmRef is a private struct of DriverHostActor (:183).
  • Sink implementors that must keep compiling: NullOpcUaAddressSpaceSink (same file as the interface), DeferredAddressSpaceSink (src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/DeferredAddressSpaceSink.cs), SdkAddressSpaceSink (src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/SdkAddressSpaceSink.cs), plus test fakes. Using a default interface method for WriteValues means only the ones that want batching override it.
  • Commands: build dotnet build ZB.MOM.WW.OtOpcUa.slnx; single project dotnet test tests/<module>/<project>; single test dotnet test --filter "FullyQualifiedName~<Class>.<Method>". DB-backed suites need the central SQL Server — on this Mac, prefer running the specific unit projects named per task, not the whole solution, until the final task.
  • Commit style: follow recent history (feat(...), fix(...), chore(...) with a scope). One commit per task.

Task 1: Batch contract types in Core.Abstractions

Classification: small Estimated implement time: ~3 min Parallelizable with: Task 10

Files:

  • Modify: src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/ISubscribable.cs
  • Test: tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests/DataChangeBatchContractTests.cs (create)

Step 1: Write the failing test

using Shouldly;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using Xunit;

namespace ZB.MOM.WW.OtOpcUa.Core.Tests;

public class DataChangeBatchContractTests
{
    private sealed record FakeHandle(string DiagnosticId) : ISubscriptionHandle;

    [Fact]
    public void Batch_args_carry_handle_and_items()
    {
        var handle = new FakeHandle("h1");
        var snap = new DataValueSnapshot(42, 0u, DateTime.UtcNow, DateTime.UtcNow);
        var args = new DataChangeBatchEventArgs(handle, new[] { new DataChangeItem("plc/tag1", snap) });
        args.SubscriptionHandle.ShouldBe(handle);
        args.Items.Count.ShouldBe(1);
        args.Items[0].FullReference.ShouldBe("plc/tag1");
        args.Items[0].Snapshot.ShouldBe(snap);
    }
}

Note: check DataValueSnapshot's actual constructor shape in src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/ first and adjust the test's construction to it (positional record with Value/StatusCode/timestamps). Do not change DataValueSnapshot.

Step 2: Run test to verify it fails

Run: dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests --filter "FullyQualifiedName~DataChangeBatchContractTests" Expected: FAIL — DataChangeBatchEventArgs / DataChangeItem not defined.

Step 3: Implement

Append to src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/ISubscribable.cs:

/// <summary>One changed attribute within a <see cref="DataChangeBatchEventArgs"/>.</summary>
/// <param name="FullReference">Driver-side full reference of the changed attribute.</param>
/// <param name="Snapshot">New value + quality + timestamps.</param>
public sealed record DataChangeItem(string FullReference, DataValueSnapshot Snapshot);

/// <summary>Event payload for <see cref="IBatchSubscribable.OnDataChangeBatch"/> — every attribute
/// that changed within ONE production burst (a poll cycle, an inbound message, a stream chunk).</summary>
/// <param name="SubscriptionHandle">The handle returned by the original <see cref="ISubscribable.SubscribeAsync"/> call.</param>
/// <param name="Items">The changed attributes, one entry per changed tag. Never empty.</param>
public sealed record DataChangeBatchEventArgs(
    ISubscriptionHandle SubscriptionHandle,
    IReadOnlyList<DataChangeItem> Items);

/// <summary>
///     Optional driver capability: data changes delivered one event per production burst instead of
///     one event per tag. A driver implementing this raises <see cref="OnDataChangeBatch"/> once per
///     burst AND continues raising the per-tag <see cref="ISubscribable.OnDataChange"/> for each item
///     (legacy consumers: Galaxy cold reads, driver CLIs). A consumer subscribes to EXACTLY ONE of the
///     two events — subscribing both double-delivers. DriverInstanceActor prefers the batch event.
/// </summary>
public interface IBatchSubscribable : ISubscribable
{
    /// <summary>Batched data-change notification — one raise per production burst.</summary>
    event EventHandler<DataChangeBatchEventArgs>? OnDataChangeBatch;
}

Step 4: Run test to verify it passes

Run: same filter. Expected: PASS.

Step 5: Commit

git add src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/ISubscribable.cs tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests/DataChangeBatchContractTests.cs
git commit -m "feat(core): add IBatchSubscribable batch data-change contract"

Task 2: PollGroupEngine batch callback

Classification: standard Estimated implement time: ~5 min Parallelizable with: none (Task 1 first)

Files:

  • Modify: src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/PollGroupEngine.cs
  • Test: tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests/PollGroupEngineBatchTests.cs (create)

Step 1: Write the failing tests (find the existing PollGroupEngine test file first — if one exists in tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests, put these in a new class beside it and copy its reader-fake idiom):

Test cases (write all four):

  1. Batch_fires_once_per_cycle_with_only_changed_items — reader returns 3 refs; first poll (initial push) batch has all 3; then reader changes only 1 value; next batch has exactly that 1 item; per-tag onChange also fired for the same items (both callbacks see identical content).
  2. No_batch_when_nothing_changed — second poll with identical snapshots ⇒ no batch invocation (and no per-tag invocation).
  3. Batch_is_not_invoked_when_callback_absent — construct without onBatch; engine behaves exactly as today (per-tag only). Assert per-tag count.
  4. Batch_items_are_in_reader_order.

Use a short interval (e.g. 30 ms) + TaskCompletionSource/polling-with-timeout to await callbacks, mirroring the existing engine tests' style. Never Thread.Sleep long constants.

Step 2: Run to verify failure

Run: dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests --filter "FullyQualifiedName~PollGroupEngineBatchTests" Expected: FAIL — constructor has no onBatch parameter.

Step 3: Implement

In PollGroupEngine:

  • Add field private readonly Action<ISubscriptionHandle, IReadOnlyList<DataChangeItem>>? _onBatch;
  • Add optional ctor param Action<ISubscriptionHandle, IReadOnlyList<DataChangeItem>>? onBatch = null (AFTER the existing optional params, so every existing call site compiles unchanged) and assign it.
  • In PollOnceAsync (currently :202-226): collect changed items lazily and raise the batch after the loop:
List<DataChangeItem>? changed = null;
for (var i = 0; i < state.TagReferences.Count; i++)
{
    var tagRef = state.TagReferences[i];
    var current = snapshots[i];
    var lastSeen = state.LastValues.TryGetValue(tagRef, out var prev) ? prev : default;

    if (forceRaise || ValuesAreDifferent(lastSeen?.Value, current.Value) || lastSeen?.StatusCode != current.StatusCode)
    {
        state.LastValues[tagRef] = current;
        _onChange(state.Handle, tagRef, current);
        if (_onBatch is not null) (changed ??= new List<DataChangeItem>()).Add(new DataChangeItem(tagRef, current));
    }
}
if (changed is { Count: > 0 }) _onBatch!(state.Handle, changed);

Keep the surrounding contract-violation check and everything else byte-identical. Update the class remarks to document the batch callback (one raise per cycle, only when non-empty, after all per-tag callbacks of that cycle).

Step 4: Run tests

Run: dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests (whole project — the existing engine tests must stay green). Expected: PASS.

Step 5: Commit

git add -A && git commit -m "feat(core): PollGroupEngine optional per-cycle batch callback"

Task 3a: Modbus + S7 raise the batch event

Classification: standard Estimated implement time: ~5 min Parallelizable with: Task 3b, Task 5

Files:

  • Modify: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusDriver.cs
  • Modify: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7/S7Driver.cs
  • Test: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests/ModbusBatchSubscriptionTests.cs (create; mirror the fixture idiom of ModbusSubscriptionTests.cs)

Step 1: Write the failing test — subscribe via the driver against the existing in-test transport fake (copy ModbusSubscriptionTests' setup); assert:

  1. ModbusDriver is IBatchSubscribable.
  2. After one poll cycle with N changed registers, OnDataChangeBatch fired once with N items AND OnDataChange fired N times (both events, identical content — contract from Task 1).

Step 2: Run: dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests --filter "FullyQualifiedName~ModbusBatchSubscriptionTests" — FAIL (interface not implemented).

Step 3: Implement — in each driver:

  • Change the class declaration to implement IBatchSubscribable (it already implements ISubscribable).
  • Add public event EventHandler<DataChangeBatchEventArgs>? OnDataChangeBatch;
  • Find the new PollGroupEngine(...) construction; pass onBatch: (handle, items) => OnDataChangeBatch?.Invoke(this, new DataChangeBatchEventArgs(handle, items)).

Per-tag raising is already wired via the engine's onChange — leave it.

Step 4: Run the Modbus + S7 test projects: dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests && dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.Tests — PASS (all existing tests too).

Step 5: git add -A && git commit -m "feat(drivers): Modbus + S7 implement IBatchSubscribable via PollGroupEngine batch callback"

Deviation (landed): Modbus does not forward the engine's items argument — it ignores it and raises the batch from its own per-cycle buffer of the items that passed ShouldPublish. The engine collects its batch list unconditionally and has no knowledge of the driver's deadband filter, so a straight pass-through would carry changes the per-tag event suppressed, and a consumer that preferred the batch event would see deadband silently stop working (review finding). onBatch is therefore used only as the end-of-cycle signal, draining the buffer; a cycle whose every change was suppressed raises nothing. Because that buffer is the only piece of cross-cycle state any of the four poll drivers holds, the driver's onChange bridge also wraps OnDataChange?.Invoke(...) in a swallowing try/catch (mirroring PollGroupEngine.ReportError): the engine invokes the bridge unguarded, so a throwing external subscriber would unwind the cycle before onBatch ran and the undrained items would merge into a later cycle's batch. The buffer append stays outside the catch — the item was published as far as the batch contract is concerned; only the per-tag subscriber faulted. S7/AbCip/FOCAS forward the engine's list as this task describes and hold no such state.


Task 3b: AbCip + FOCAS raise the batch event

Classification: standard Estimated implement time: ~4 min Parallelizable with: Task 3a, Task 5

Files:

  • Modify: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbCip/AbCipDriver.cs
  • Modify: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.FOCAS/FocasDriver.cs

Same mechanical change as Task 3a (interface + event + onBatch at the new PollGroupEngine(...) site). If either driver constructs the engine differently (check first), adapt but keep the identical contract. No new per-driver test required (the engine behavior is pinned by Task 2; the wiring pattern is pinned by Task 3a's test) — but the projects' existing suites must stay green.

Run: dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbCip.Tests && dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Tests

Commit: feat(drivers): AbCip + FOCAS implement IBatchSubscribable


Task 4: DriverInstanceActor batch forwarding

Classification: high-risk (actor model, subscription lifecycle) Estimated implement time: ~5 min Parallelizable with: Task 5

Files:

  • Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs
  • Test: tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverInstanceActorBatchForwardTests.cs (create; copy the TestKit + stub-driver idiom from the existing DriverInstanceActor*Tests in that directory)

Step 1: Write the failing tests

  1. Batch_driver_forwards_one_batch_message — stub driver implementing IBatchSubscribable; raise OnDataChangeBatch with 3 items after subscribe; parent probe receives exactly ONE AttributeValueBatchPublished with 3 items (driver id + per-item FullReference/Value/StatusCode/timestamp mapped as OnDataChangeForward maps them today: ts = Snapshot.SourceTimestampUtc ?? Snapshot.ServerTimestampUtc).
  2. Batch_driver_per_tag_event_not_double_forwarded — the stub raises BOTH events (per contract); parent receives only the batch message, no legacy AttributeValuePublished.
  3. Per_tag_driver_forwards_single_item_batches — stub implementing only ISubscribable; one OnDataChange ⇒ parent receives one AttributeValueBatchPublished with 1 item.
  4. Detach_removes_batch_handler — after Unsubscribe (or Connected→Reconnecting, whichever the existing detach tests exercise), raising the batch event produces no message.

Step 2: Run: dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~DriverInstanceActorBatchForwardTests" — FAIL.

Step 3: Implement

  • Add records next to AttributeValuePublished (:109):
/// <summary>One value within an <see cref="AttributeValueBatchPublished"/>.</summary>
public sealed record AttributeValueItem(string FullReference, object? Value, uint StatusCode, DateTime TimestampUtc);
/// <summary>Batched form of <see cref="AttributeValuePublished"/> — every changed attribute of ONE
/// driver production burst in one message. The parent fans the whole batch to the mux and the OPC UA
/// publish path in one hop each. Status codes ride verbatim (#519).</summary>
public sealed record AttributeValueBatchPublished(string DriverInstanceId, IReadOnlyList<AttributeValueItem> Items);
  • Private message private sealed record DataChangeBatchForward(IReadOnlyList<DataChangeItem> Items); + field private EventHandler<DataChangeBatchEventArgs>? _batchChangeHandler;
  • In HandleSubscribeAsync (:878-882): when subscribable is IBatchSubscribable batchable, attach ONLY the batch handler (_batchChangeHandler = (_, args) => self.Tell(new DataChangeBatchForward(args.Items)); batchable.OnDataChangeBatch += _batchChangeHandler;) and do NOT attach _dataChangeHandler; else attach _dataChangeHandler exactly as today.
  • Receive<DataChangeBatchForward>(OnDataChangeBatchForward); beside the existing receive (:480); handler builds the items list once and Tells parent one AttributeValueBatchPublished.
  • OnDataChangeForward (:1138) becomes a single-item batch: Context.Parent.Tell(new AttributeValueBatchPublished(_driverInstanceId, new[] { new AttributeValueItem(msg.FullReference, msg.Snapshot.Value, msg.Snapshot.StatusCode, ts) })); — keep the AttributeValuePublished record declared (the mux comms-loss path and existing tests use the type), but the actor no longer sends it.
  • DetachSubscription (:934-943): also detach _batchChangeHandler from IBatchSubscribable and null it.

IMPORTANT — downstream compile/behavior: DriverHostActor currently receives AttributeValuePublished (DriverHostActor.cs:960). Until Task 8 lands, add a minimal bridge in THIS task so the system stays green: in DriverHostActor, add Receive<DriverInstanceActor.AttributeValueBatchPublished>(batch => { foreach (var i in batch.Items) ForwardToMux(new DriverInstanceActor.AttributeValuePublished(batch.DriverInstanceId, i.FullReference, i.Value, i.StatusCode, i.TimestampUtc)); }); — Task 8 replaces this bridge with the real batched fan-out. Existing DriverHostActor tests that send AttributeValuePublished directly stay green (the receive stays).

Step 4: Run: dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests — PASS (existing DriverInstanceActor forward tests may need updating IF they assert the parent receives AttributeValuePublished from a live driver event — update those to expect the single-item batch; do NOT weaken assertions on content).

Step 5: git add -A && git commit -m "feat(runtime): DriverInstanceActor forwards batched data changes"


Task 5: ValueWrite + sink WriteValues default method

Classification: small Estimated implement time: ~4 min Parallelizable with: Task 3a, 3b, 4

Files:

  • Modify: src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/IOpcUaAddressSpaceSink.cs
  • Modify: src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/DeferredAddressSpaceSink.cs
  • Test: tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/AddressSpaceSinkBatchTests.cs (create)

Step 1: Failing tests

  1. Default_WriteValues_loops_WriteValue — a minimal fake implementing only the existing members; call WriteValues with 3 items; the fake's recorded WriteValue calls match in order/content.
  2. Deferred_sink_forwards_WriteValues_to_inner_once — a fake inner that overrides WriteValues (records one batch call); DeferredAddressSpaceSink.WriteValues reaches the inner's override (NOT 3 WriteValue calls).

Step 3: Implement

In IOpcUaAddressSpaceSink.cs:

/// <summary>One value write within a <see cref="IOpcUaAddressSpaceSink.WriteValues"/> batch.
/// Same semantics per field as <see cref="IOpcUaAddressSpaceSink.WriteValue"/> (#519: status verbatim).</summary>
public readonly record struct ValueWrite(
    string NodeId, object? Value, uint StatusCode, DateTime SourceTimestampUtc, AddressSpaceRealm Realm);

Add to the interface (after WriteValue):

/// <summary>Write a batch of Variable values. Semantically identical to calling
/// <see cref="WriteValue"/> per item; an SDK-backed sink overrides this to take its internal
/// lock ONCE per batch instead of once per value (the hot-path win). The default implementation
/// loops <see cref="WriteValue"/> so existing sinks and fakes behave unchanged.</summary>
void WriteValues(IReadOnlyList<ValueWrite> values)
{
    ArgumentNullException.ThrowIfNull(values);
    for (var i = 0; i < values.Count; i++)
    {
        var v = values[i];
        WriteValue(v.NodeId, v.Value, v.StatusCode, v.SourceTimestampUtc, v.Realm);
    }
}

In DeferredAddressSpaceSink, add the forwarding override: public void WriteValues(IReadOnlyList<ValueWrite> values) => _inner.WriteValues(values);

Step 4: dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests — PASS.

Step 5: git add -A && git commit -m "feat(commons): batched WriteValues on the address-space sink"


Task 6: OtOpcUaNodeManager batched write under one lock

Classification: high-risk (SDK locking) Estimated implement time: ~4 min Parallelizable with: Task 7

Files:

  • Modify: src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs (WriteValue region, :508-533)
  • Modify: src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/SdkAddressSpaceSink.cs (forward WriteValues to the manager)
  • Test: tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerBatchWriteTests.cs (create; copy the manager-construction idiom from an existing OpcUaServer.Tests file that instantiates OtOpcUaNodeManager)

Step 1: Failing tests

  1. WriteValues_lands_every_value — materialise 3 variables (or rely on create-on-first-write), WriteValues a 3-item batch, read each variable's Value/StatusCode/Timestamp back via the manager's public surface.
  2. WriteValues_creates_missing_variables — same lazy-create semantics as WriteValue.

Step 3: Implement

  • Extract the body of WriteValue's lock (Lock) block into private void WriteValueLocked(string nodeId, ushort ns, string key, object? value, uint statusCode, DateTime ts) (verbatim move).
  • WriteValue computes ns/key then lock (Lock) { WriteValueLocked(...); } — behavior identical.
  • Add:
/// <summary>Batched WriteValue: one Lock acquisition for the whole batch. Same per-item semantics.</summary>
public void WriteValues(IReadOnlyList<ValueWrite> values)
{
    ArgumentNullException.ThrowIfNull(values);
    if (values.Count == 0) return;
    EnsureAddressSpaceCreated();
    lock (Lock)
    {
        for (var i = 0; i < values.Count; i++)
        {
            var v = values[i];
            WriteValueLocked(v.NodeId, NamespaceIndexForRealm(v.Realm), MapKey(v.Realm, v.NodeId), v.Value, v.StatusCode, v.SourceTimestampUtc);
        }
    }
}
  • In SdkAddressSpaceSink, override WriteValues forwarding to the manager's method (it currently forwards WriteValue 1:1 — mirror that).

Step 4: dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests — PASS.

Step 5: git add -A && git commit -m "feat(opcuaserver): node manager batched WriteValues under one lock"


Task 7: OpcUaPublishActor batch message

Classification: standard Estimated implement time: ~4 min Parallelizable with: Task 6

Files:

  • Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs
  • Test: tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/OpcUa/OpcUaPublishActorBatchTests.cs (create; copy the actor-construction idiom from OpcUaPublishActorApplyFailureTests.cs)

Step 1: Failing tests

  1. Batch_message_reaches_sink_as_one_WriteValues_call — fake sink records WriteValues batches; send AttributeValueBatchUpdate with 3 items; sink saw one batch of 3.
  2. Batch_increments_write_counter_per_item — whatever surface exposes _writes (check for an existing status/metrics accessor in the actor's tests; if none, assert via the telemetry meter or skip the counter assertion and note it) — at minimum assert behavior parity: 3 items produce the same observable effect as 3 single AttributeValueUpdate messages.

Step 3: Implement

  • Add next to AttributeValueUpdate (:52):
/// <summary>Batched form of <see cref="AttributeValueUpdate"/>: every fan-out write of one driver
/// production burst in one message — the sink takes its lock once for the whole batch.</summary>
public sealed record AttributeValueBatchUpdate(IReadOnlyList<ValueWrite> Items);

(ValueWrite comes from ZB.MOM.WW.OtOpcUa.Commons.OpcUa — already imported.)

  • Register Receive<AttributeValueBatchUpdate>(HandleAttributeBatchUpdate); beside :288.
  • Handler mirrors HandleAttributeUpdate (:319-331): try/catch WARN on throw, _sink.WriteValues(msg.Items), then _writes += msg.Items.Count via Interlocked.Add(ref _writes, msg.Items.Count) and OtOpcUaTelemetry.OpcUaSinkWrite.Add(msg.Items.Count, new KeyValuePair<string, object?>("kind", "value")) — per-item counting (invariant 9).

Step 4: dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~OpcUaPublishActorBatch" then the whole project — PASS.

Step 5: git add -A && git commit -m "feat(runtime): OpcUaPublishActor batched value updates"


Task 8: DriverHostActor batched fan-out (replaces the Task 4 bridge)

Classification: high-risk (the hottest routing seam) Estimated implement time: ~5 min Parallelizable with: none (needs 4 + 7; Task 9 lands with it)

Files:

  • Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs
  • Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyMuxActor.cs (Task 9 content — same commit is fine if splitting is awkward, but prefer two commits)
  • Test: tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorBatchFanOutTests.cs (create; copy the host-actor fixture idiom from DriverHostActorNativeAlarmTests.cs)

Step 1: Failing tests

  1. Batch_fans_out_to_all_node_realm_refs_in_one_publish_message — arrange the host with an applied composition where tag A maps to 1 raw + 1 UNS NodeId and tag B maps to 1 raw; send AttributeValueBatchPublished with A+B; the publish-actor probe receives exactly ONE AttributeValueBatchUpdate containing 3 ValueWrites with correct NodeId/Realm/value/status/timestamp.
  2. Unmapped_items_are_dropped_not_faulted — batch with one unmapped ref: other items still land; no crash.
  3. Batch_reaches_mux_once — mux probe receives ONE batch message carrying all items (not N).

Step 3: Implement

Replace the Task 4 bridge with a real handler:

private void ForwardBatchToMux(DriverInstanceActor.AttributeValueBatchPublished msg)
{
    _dependencyMux?.Tell(msg);   // one envelope per burst; the mux filters per item (Task 9)

    if (_opcUaPublishActor is null) return;

    List<ValueWrite>? writes = null;
    foreach (var item in msg.Items)
    {
        if (_nodeIdByDriverRef.TryGetValue((msg.DriverInstanceId, item.FullReference), out var nodeIds))
        {
            writes ??= new List<ValueWrite>(msg.Items.Count * 2);
            foreach (var n in nodeIds)
                writes.Add(new ValueWrite(n.NodeId, item.Value, item.StatusCode, item.TimestampUtc, n.Realm));
        }
        else if (_log.IsDebugEnabled)
        {
            _log.Debug("DriverHost {Node}: no bound NodeId for ({Driver},{Ref}) — value dropped",
                _localNode, msg.DriverInstanceId, item.FullReference);
        }
    }
    if (writes is not null)
        _opcUaPublishActor.Tell(new ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.AttributeValueBatchUpdate(writes));
}

Keep ForwardToMux and its Receive<AttributeValuePublished> registration untouched (comms-loss fan + legacy tests). Register Receive<DriverInstanceActor.AttributeValueBatchPublished>(ForwardBatchToMux); in place of the Task 4 bridge.

Step 4: dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests — PASS.

Step 5: git add -A && git commit -m "feat(runtime): DriverHostActor batched value fan-out"


Task 9: DependencyMuxActor batch receive

Classification: small Estimated implement time: ~3 min Parallelizable with: Task 8 (same PR-let; do after 8's records exist)

Files:

  • Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyMuxActor.cs
  • Test: tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ — add a batch case beside the existing mux tests (find them via CalculationDependencyFlowTests.cs / grep DependencyMuxActor in that project)

Step 1: Failing test — register a subscriber for ref X; send a batch containing X and an uninteresting Y; subscriber receives exactly one DependencyValueChanged for X, nothing for Y.

Step 3: Implement — add:

Receive<DriverInstanceActor.AttributeValueBatchPublished>(OnAttributeValueBatchPublished);
private void OnAttributeValueBatchPublished(DriverInstanceActor.AttributeValueBatchPublished msg)
{
    foreach (var item in msg.Items)
    {
        if (!_byRef.TryGetValue(item.FullReference, out var subscribers) || subscribers.Count == 0) continue;
        var dep = new VirtualTagActor.DependencyValueChanged(item.FullReference, item.Value, item.TimestampUtc, item.StatusCode);
        foreach (var sub in subscribers) sub.Tell(dep);
    }
}

Keep the per-item receive (comms-loss path).

Step 4/5: project tests green; git commit -m "feat(runtime): dependency mux accepts batched publishes"


Task 10: Raise pinned-dispatcher throughput

Classification: trivial Estimated implement time: ~2 min Parallelizable with: Task 1

Files:

  • Modify: src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:112-116
  • Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs:301-312 (the inline HOCON copy)

Change throughput = 1 → throughput = 100 in BOTH places (they must match — the inline HOCON exists for consumers without akka.conf; see the comment at ServiceCollectionExtensions.cs:301-303). Update the comments: pinning alone guarantees the SDK's single-thread requirement; throughput only sets how many queued messages drain per scheduling run, so 100 removes a mailbox re-schedule round-trip per message with zero concurrency change.

Run: dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~ServiceCollectionExtensionsTests"

Commit: perf(runtime): raise opcua pinned dispatcher throughput 1 -> 100


Task 11: Re-key node-manager maps (kill per-write key allocations)

Classification: high-risk (38 call sites, mechanical but wide) Estimated implement time: ~5 min (compiler-driven) Parallelizable with: none (after Task 6 — same file)

Files:

  • Modify: src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs

Step 1: Baseline — run dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests green BEFORE starting (this task relies on the existing suite as its net; it should need no new tests, but the whole suite must pass unchanged after).

Step 2: Implement

  • Add near MapKey (:126-139):
/// <summary>Zero-allocation realm-qualified map key. Replaces the formatted "ns=N;s=id" string key —
/// the (realm, id) pair is the same identity without the per-write NodeId + string allocations.</summary>
private readonly record struct RealmKey(AddressSpaceRealm Realm, string Id);
  • Change MapKey(AddressSpaceRealm realm, string id) to return new RealmKey(realm, id); change MapKey(NodeId nodeId) to new RealmKey(RealmOf(nodeId.NamespaceIndex), (string)nodeId.Identifier) — RealmOf already exists (~:123); confirm its exact name/signature and that every MapKey(NodeId) call site only ever passes nodes from the two owned namespaces (they do — SDK-facing seams on owned handles). If nodeId.Identifier can be non-string on any call path (numeric-id nodes), guard with as string ?? nodeId.Identifier?.ToString() ?? string.Empty.
  • Change every map/set keyed by the old string key to RealmKey: _variables, _folders, _alarmConditions, _historizedTagnames, _eventNotifierSources, _nativeAlarmNodeIds, and the notifier-wiring bookkeeping — let the compiler find all of them (38 MapKey call sites).
  • Prefix scans (RemoveEquipmentSubtree ~:2539-2596, InScope usages): match on key.Realm == realm && key.Id.StartsWith(prefix, ...) — semantics identical (the old string key embedded the realm as ns=N;).
  • ResolveNodeState(string key) (~:2240-2261): change its parameter to RealmKey.
  • Do NOT change public API surface or any behavior — this is a pure key-representation change.

Step 3: dotnet build ZB.MOM.WW.OtOpcUa.slnx then dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests and dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests — ALL green, zero test edits expected (tests drive the public surface). If a test pokes the key format directly, stop and reassess rather than bending the test.

Step 4: git add -A && git commit -m "perf(opcuaserver): struct realm keys replace formatted NodeId string keys"


Task 12: Full verification + docs

Classification: standard Estimated implement time: ~5 min Parallelizable with: none (last)

Files:

  • Modify: CLAUDE.md (only if a statement it makes is now false — check the pinned-dispatcher/throughput mentions; keep edits surgical)
  • Modify: docs/v2/Galaxy.Performance.md — add a short "value-path batching" note (one paragraph: the batch seams, the invariants, the dispatcher change)

Steps:

  1. dotnet build ZB.MOM.WW.OtOpcUa.slnx — zero warnings introduced.
  2. dotnet test ZB.MOM.WW.OtOpcUa.slnx — run what the environment can (DB-backed suites need 10.100.0.35; if unreachable, run every non-DB project under tests/Core, tests/Drivers (unit), tests/Server/Runtime.Tests, tests/Server/OpcUaServer.Tests and record which suites were skipped and why in the commit message).
  3. Docs + commit: docs: record value-pipeline batching (plan 2026-08-15).

Explicit non-goals (do NOT let scope creep into these)

  • No conflation / bounded mailboxes — batching per burst is this plan's backpressure story; keyed conflation is a follow-up.
  • No fan-out-map push-down to DriverInstanceActor — the host stays the router.
  • No MQTT / MTConnect / Sparkplug / Galaxy / OpcUaClient batch implementations — they flow through the single-item-batch path unchanged.
  • No change to alarm paths (AttributeAlarmPublished, AlarmStateUpdate) — value path only.
  • No PollGroupEngine cadence change (fixed-rate scheduling is a separate item).

Follow-ups (recorded, not implemented)

Raised during Tasks 1–12 review and deliberately left out of this plan's scope. None is a defect in the delivered work; each is a place where the next change in the area could go wrong quietly.

  1. PollGroupEngine collects batch items unconditionally — the batch event is only as trustworthy as the driver's onChange. The engine has no knowledge of a driver's publish filter, so its items list is pre-filter by construction. That is safe today only because S7, AbCip and FOCAS forward every onChange unconditionally, and Modbus — the one driver with a deadband — ignores the engine's list and raises from its own filtered buffer. The trap: a future per-tag filter added to S7/AbCip/FOCAS's onChange would silently reintroduce the exact deadband-bypass class of bug fixed for Modbus in cd4ce305, with a green test suite (the two events would simply disagree, and only a consumer preferring the batch would notice). Consider changing the engine's onChange contract to Func<..., bool> so the engine collects an item only when the driver reports it published — making the filter structurally impossible to skip rather than a convention each driver must remember.
  2. Two-level driverId → (rawPath → refs) routing map in DriverHostActor. The fan-out map is keyed by a composite of driver id + raw path, so resolving a batch re-hashes the driver id once per item even though a batch is by definition single-driver. Nesting the map would hash the driver id once per batch and the raw path once per item. Measure first — batches are small enough that this may be noise relative to the message-count win already banked.
  3. Null/empty guard consistency on the internal batch messages. The guards on AttributeValueBatchPublished / AttributeValueBatchUpdate grew per task and are not uniform — some seams drop an empty batch early, others tolerate it downstream. These are internal messages with trusted senders so nothing is unsafe today; worth one pass to make the contract identical at every hop and stated once, rather than re-derived per reader.
  4. Possible merge of DriverHostActorLiveValueTests into the batch suite. After Task 8 both files drive the same fan-out through the same message; the split is now historical rather than meaningful. Merging would remove a genuine ambiguity about where a new fan-out test belongs — but check first that the older suite's per-item comms-loss cases really are covered, since those enter via a different seam.
  5. Aborted-cycle tail-loss when a per-tag handler throws, for the three unbuffered drivers. If a consumer's onChange throws mid-cycle in S7/AbCip/FOCAS, the cycle aborts and the items already collected for that cycle never reach a batch raise. Pre-existing and narrower than the Modbus case (Modbus's own buffer is per-handle and is dropped on unsubscribe/dispose, so it cannot leak), and it predates this plan — the per-tag path lost the same tail before batching existed. Recorded so it is not mistaken for a regression introduced here.