Files
lmxopcua/docs/plans/2026-08-15-value-pipeline-batching.md
T
Joseph Doherty aec5ce565a docs: record value-pipeline batching (plan 2026-08-15)
- docs/v2/Galaxy.Performance.md gains a "Value-path batching (2026-08-15)"
  section: the batch seams (IBatchSubscribable / PollGroupEngine onBatch ->
  AttributeValueBatchPublished -> AttributeValueBatchUpdate -> sink WriteValues
  with 512-chunked single-lock batches in the node manager), the invariants (#519
  status codes verbatim, per-ITEM metering, Modbus deadband honored via the
  driver-filtered buffer, the ValueWriteBatchException partial-failure contract),
  the RealmKey re-key, and the pinned dispatcher's throughput 1 -> 100.

- The plan doc gains a "Follow-ups (recorded, not implemented)" section: the
  PollGroupEngine unconditional-collection trap (a future per-tag filter in
  S7/AbCip/FOCAS would silently reintroduce the deadband-bypass bug), a two-level
  driverId -> (rawPath -> refs) routing map, null/empty guard consistency on the
  internal batch messages, a possible DriverHostActorLiveValueTests merge, and the
  pre-existing aborted-cycle tail-loss on the three unbuffered drivers.

- tasks.json: all 13 tasks completed.

CLAUDE.md needed no edit: its only statement in this area ("One driver publish for
a RawPath fans (in DriverHostActor) to the raw NodeId AND every referencing UNS
NodeId") is still true — batching changed the message granularity, not the
fan-out semantics.

Verification: dotnet build ZB.MOM.WW.OtOpcUa.slnx clean (0 errors, no new warning
classes, zero warnings in any production file). 9 unit projects green, 3470
passed / 5 skipped / 0 failed: Core.Abstractions 305, Core 351+1s, Commons 348,
Runtime 575+3s, OpcUaServer 424+1s, Modbus 399, S7 302, AbCip 410, FOCAS 356.
No DB-backed suite was run — tests/Server/ZB.MOM.WW.OtOpcUa.Server.Tests does not
exist, and nothing this branch touches is covered by the SQL-backed projects.
2026-08-15 13:10:55 -04:00

632 lines
40 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Value-Pipeline Batching Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
**Goal:** Batch the hot value path end to end — one message per poll cycle instead of one per tag at every hop (driver → DriverInstanceActor → DriverHostActor → OpcUaPublishActor → node manager), one SDK-lock acquisition per batch instead of per value, and zero per-write key allocations in the node manager.
**Architecture:** A new optional `IBatchSubscribable` capability lets a driver raise one `OnDataChangeBatch` per production burst (poll cycle); `PollGroupEngine` gains a batch callback so the four engine-based poll drivers get it for free. `DriverInstanceActor` forwards one `AttributeValueBatchPublished` per burst (wrapping per-tag events from non-batch drivers as single-item batches); `DriverHostActor` resolves the whole batch's NodeRealmRefs and sends ONE `AttributeValueBatchUpdate` to `OpcUaPublishActor`, plus ONE batch Tell to the dependency mux. The sink gains `WriteValues(IReadOnlyList<ValueWrite>)` (default-interface-method fallback loops `WriteValue`, so fakes/Null sink are untouched); `OtOpcUaNodeManager` overrides it taking `Lock` once per batch, and its node maps are re-keyed by a `(realm, id)` struct so the per-write `new NodeId(...).ToString()` allocations disappear. The pinned dispatcher's `throughput` rises from 1 to 100 (same single thread, fewer mailbox re-schedules).
**Tech Stack:** C# / .NET 10, Akka.NET (classic actors), OPC Foundation UA .NET Standard, xUnit + Shouldly, Akka.TestKit.
**Branch:** `perf/value-pipeline-batching` (off `master`).
---
## Context for implementers (read before any task)
Verified facts about the current code — do not re-derive, and do not violate the invariants:
- **The per-tag path today:** `PollGroupEngine.PollOnceAsync` (`src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/PollGroupEngine.cs:202-226`) diffs per tag and invokes `_onChange` per changed tag. Drivers forward that to `ISubscribable.OnDataChange` (per-tag event, `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/ISubscribable.cs:36`). `DriverInstanceActor.HandleSubscribeAsync` attaches `_dataChangeHandler` at `DriverInstanceActor.cs:880` (one `DataChangeForward` Tell per tag), `OnDataChangeForward` (`:1138-1143`) Tells parent one `AttributeValuePublished` per tag. `DriverHostActor.ForwardToMux` (`DriverHostActor.cs:976-1006`) Tells the mux per value and Tells `OpcUaPublishActor` one `AttributeValueUpdate` **per NodeRealmRef** (UNS fan-out at `:997-999`). `OpcUaPublishActor.HandleAttributeUpdate` (`OpcUaPublishActor.cs:319-331`) calls `_sink.WriteValue` per message. `OtOpcUaNodeManager.WriteValue` (`OtOpcUaNodeManager.cs:508-533`) takes the manager-wide `lock (Lock)` per value and allocates via `MapKey(realm, id)` = `new NodeId(id, ns).ToString()` (`:133`).
- **Invariants to preserve:**
1. **#519 status codes ride verbatim** — the full `uint` status code, sub-code included, must reach the SDK unchanged at every hop.
2. **Timestamps are carried from the driver** — never stamp `DateTime.UtcNow` on the value path.
3. **PollGroupEngine's no-overlap guarantee** (delay-after-completion, one loop task per subscription) must not change in this plan.
4. **Per-tag `OnDataChange` keeps working** — Galaxy's cold-read path and driver CLIs subscribe it. A batch-capable driver raises BOTH events (batch once per cycle + per-tag per item); a consumer subscribes to exactly ONE of the two. `DriverInstanceActor` prefers batch when the driver implements `IBatchSubscribable`.
5. **`DriverInstanceActor.DetachSubscription`** (`:934-943`) must detach the batch handler too — the detach is load-bearing (the `IDriver` outlives the actor across respawns; see CLAUDE.md §DeployWatcher).
6. **The mux's per-item receive stays** — `DriverHostActor.FanCommsLossToDependencyMux` (`:1135-1142`) fabricates per-item `AttributeValuePublished` messages; `DependencyMuxActor` keeps that receive and gains a batch receive alongside.
7. **`OpcUaPublishActor` keeps its per-item `AttributeValueUpdate` receive** — `VirtualTagHostActor` and the alarm paths still send it. The batch message is additive.
8. **The pinned dispatcher stays pinned** — the OPC UA SDK requires a single thread. Only `throughput` changes.
9. **`_writes` increments** (`OpcUaPublishActor.cs:324`) and the `OtOpcUaTelemetry.OpcUaSinkWrite` meter must count per-VALUE (per item), not per batch, so dashboards don't drop by 100×.
- **Message-record homes:** `AttributeValuePublished` is declared in `DriverInstanceActor` (`:109`); `AttributeValueUpdate` in `OpcUaPublishActor` (`:52`); `NodeRealmRef` is a private struct of `DriverHostActor` (`:183`).
- **Sink implementors that must keep compiling:** `NullOpcUaAddressSpaceSink` (same file as the interface), `DeferredAddressSpaceSink` (`src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/DeferredAddressSpaceSink.cs`), `SdkAddressSpaceSink` (`src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/SdkAddressSpaceSink.cs`), plus test fakes. Using a **default interface method** for `WriteValues` means only the ones that want batching override it.
- **Commands:** build `dotnet build ZB.MOM.WW.OtOpcUa.slnx`; single project `dotnet test tests/<module>/<project>`; single test `dotnet test --filter "FullyQualifiedName~<Class>.<Method>"`. DB-backed suites need the central SQL Server — on this Mac, prefer running the specific unit projects named per task, not the whole solution, until the final task.
- **Commit style:** follow recent history (`feat(...)`, `fix(...)`, `chore(...)` with a scope). One commit per task.
---
### Task 1: Batch contract types in Core.Abstractions
**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 10
**Files:**
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/ISubscribable.cs`
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests/DataChangeBatchContractTests.cs` (create)
**Step 1: Write the failing test**
```csharp
using Shouldly;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Core.Tests;
public class DataChangeBatchContractTests
{
private sealed record FakeHandle(string DiagnosticId) : ISubscriptionHandle;
[Fact]
public void Batch_args_carry_handle_and_items()
{
var handle = new FakeHandle("h1");
var snap = new DataValueSnapshot(42, 0u, DateTime.UtcNow, DateTime.UtcNow);
var args = new DataChangeBatchEventArgs(handle, new[] { new DataChangeItem("plc/tag1", snap) });
args.SubscriptionHandle.ShouldBe(handle);
args.Items.Count.ShouldBe(1);
args.Items[0].FullReference.ShouldBe("plc/tag1");
args.Items[0].Snapshot.ShouldBe(snap);
}
}
```
Note: check `DataValueSnapshot`'s actual constructor shape in `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/` first and adjust the test's construction to it (positional record with Value/StatusCode/timestamps). Do not change `DataValueSnapshot`.
**Step 2: Run test to verify it fails**
Run: `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests --filter "FullyQualifiedName~DataChangeBatchContractTests"`
Expected: FAIL — `DataChangeBatchEventArgs` / `DataChangeItem` not defined.
**Step 3: Implement**
Append to `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/ISubscribable.cs`:
```csharp
/// <summary>One changed attribute within a <see cref="DataChangeBatchEventArgs"/>.</summary>
/// <param name="FullReference">Driver-side full reference of the changed attribute.</param>
/// <param name="Snapshot">New value + quality + timestamps.</param>
public sealed record DataChangeItem(string FullReference, DataValueSnapshot Snapshot);
/// <summary>Event payload for <see cref="IBatchSubscribable.OnDataChangeBatch"/> — every attribute
/// that changed within ONE production burst (a poll cycle, an inbound message, a stream chunk).</summary>
/// <param name="SubscriptionHandle">The handle returned by the original <see cref="ISubscribable.SubscribeAsync"/> call.</param>
/// <param name="Items">The changed attributes, one entry per changed tag. Never empty.</param>
public sealed record DataChangeBatchEventArgs(
ISubscriptionHandle SubscriptionHandle,
IReadOnlyList<DataChangeItem> Items);
/// <summary>
/// Optional driver capability: data changes delivered one event per production burst instead of
/// one event per tag. A driver implementing this raises <see cref="OnDataChangeBatch"/> once per
/// burst AND continues raising the per-tag <see cref="ISubscribable.OnDataChange"/> for each item
/// (legacy consumers: Galaxy cold reads, driver CLIs). A consumer subscribes to EXACTLY ONE of the
/// two events — subscribing both double-delivers. DriverInstanceActor prefers the batch event.
/// </summary>
public interface IBatchSubscribable : ISubscribable
{
/// <summary>Batched data-change notification — one raise per production burst.</summary>
event EventHandler<DataChangeBatchEventArgs>? OnDataChangeBatch;
}
```
**Step 4: Run test to verify it passes**
Run: same filter. Expected: PASS.
**Step 5: Commit**
```bash
git add src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/ISubscribable.cs tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests/DataChangeBatchContractTests.cs
git commit -m "feat(core): add IBatchSubscribable batch data-change contract"
```
---
### Task 2: PollGroupEngine batch callback
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none (Task 1 first)
**Files:**
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/PollGroupEngine.cs`
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests/PollGroupEngineBatchTests.cs` (create)
**Step 1: Write the failing tests** (find the existing PollGroupEngine test file first — if one exists in `tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests`, put these in a new class beside it and copy its reader-fake idiom):
Test cases (write all four):
1. `Batch_fires_once_per_cycle_with_only_changed_items` — reader returns 3 refs; first poll (initial push) batch has all 3; then reader changes only 1 value; next batch has exactly that 1 item; per-tag `onChange` also fired for the same items (both callbacks see identical content).
2. `No_batch_when_nothing_changed` — second poll with identical snapshots ⇒ no batch invocation (and no per-tag invocation).
3. `Batch_is_not_invoked_when_callback_absent` — construct without `onBatch`; engine behaves exactly as today (per-tag only). Assert per-tag count.
4. `Batch_items_are_in_reader_order`.
Use a short interval (e.g. 30 ms) + `TaskCompletionSource`/polling-with-timeout to await callbacks, mirroring the existing engine tests' style. Never `Thread.Sleep` long constants.
**Step 2: Run to verify failure**
Run: `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests --filter "FullyQualifiedName~PollGroupEngineBatchTests"`
Expected: FAIL — constructor has no `onBatch` parameter.
**Step 3: Implement**
In `PollGroupEngine`:
- Add field `private readonly Action<ISubscriptionHandle, IReadOnlyList<DataChangeItem>>? _onBatch;`
- Add optional ctor param `Action<ISubscriptionHandle, IReadOnlyList<DataChangeItem>>? onBatch = null` (AFTER the existing optional params, so every existing call site compiles unchanged) and assign it.
- In `PollOnceAsync` (currently `:202-226`): collect changed items lazily and raise the batch after the loop:
```csharp
List<DataChangeItem>? changed = null;
for (var i = 0; i < state.TagReferences.Count; i++)
{
var tagRef = state.TagReferences[i];
var current = snapshots[i];
var lastSeen = state.LastValues.TryGetValue(tagRef, out var prev) ? prev : default;
if (forceRaise || ValuesAreDifferent(lastSeen?.Value, current.Value) || lastSeen?.StatusCode != current.StatusCode)
{
state.LastValues[tagRef] = current;
_onChange(state.Handle, tagRef, current);
if (_onBatch is not null) (changed ??= new List<DataChangeItem>()).Add(new DataChangeItem(tagRef, current));
}
}
if (changed is { Count: > 0 }) _onBatch!(state.Handle, changed);
```
Keep the surrounding contract-violation check and everything else byte-identical. Update the class remarks to document the batch callback (one raise per cycle, only when non-empty, after all per-tag callbacks of that cycle).
**Step 4: Run tests**
Run: `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests` (whole project — the existing engine tests must stay green).
Expected: PASS.
**Step 5: Commit**
```bash
git add -A && git commit -m "feat(core): PollGroupEngine optional per-cycle batch callback"
```
---
### Task 3a: Modbus + S7 raise the batch event
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 3b, Task 5
**Files:**
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusDriver.cs`
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7/S7Driver.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests/ModbusBatchSubscriptionTests.cs` (create; mirror the fixture idiom of `ModbusSubscriptionTests.cs`)
**Step 1: Write the failing test** — subscribe via the driver against the existing in-test transport fake (copy `ModbusSubscriptionTests`' setup); assert:
1. `ModbusDriver` is `IBatchSubscribable`.
2. After one poll cycle with N changed registers, `OnDataChangeBatch` fired once with N items AND `OnDataChange` fired N times (both events, identical content — contract from Task 1).
**Step 2:** Run: `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests --filter "FullyQualifiedName~ModbusBatchSubscriptionTests"` — FAIL (interface not implemented).
**Step 3: Implement** — in each driver:
- Change the class declaration to implement `IBatchSubscribable` (it already implements `ISubscribable`).
- Add `public event EventHandler<DataChangeBatchEventArgs>? OnDataChangeBatch;`
- Find the `new PollGroupEngine(...)` construction; pass `onBatch: (handle, items) => OnDataChangeBatch?.Invoke(this, new DataChangeBatchEventArgs(handle, items))`.
Per-tag raising is already wired via the engine's `onChange` — leave it.
**Step 4:** Run the Modbus + S7 test projects:
`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests && dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.Tests` — PASS (all existing tests too).
**Step 5:** `git add -A && git commit -m "feat(drivers): Modbus + S7 implement IBatchSubscribable via PollGroupEngine batch callback"`
**Deviation (landed):** Modbus does **not** forward the engine's `items` argument — it ignores it and
raises the batch from its own per-cycle buffer of the items that passed `ShouldPublish`. The engine
collects its batch list unconditionally and has no knowledge of the driver's deadband filter, so a
straight pass-through would carry changes the per-tag event suppressed, and a consumer that preferred
the batch event would see deadband silently stop working (review finding). `onBatch` is therefore used
only as the end-of-cycle signal, draining the buffer; a cycle whose every change was suppressed raises
nothing. Because that buffer is the only piece of cross-cycle state any of the four poll drivers holds,
the driver's `onChange` bridge also wraps `OnDataChange?.Invoke(...)` in a swallowing try/catch
(mirroring `PollGroupEngine.ReportError`): the engine invokes the bridge unguarded, so a throwing
external subscriber would unwind the cycle before `onBatch` ran and the undrained items would merge
into a later cycle's batch. The buffer append stays outside the catch — the item was published as far
as the batch contract is concerned; only the per-tag subscriber faulted. S7/AbCip/FOCAS forward the
engine's list as this task describes and hold no such state.
---
### Task 3b: AbCip + FOCAS raise the batch event
**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 3a, Task 5
**Files:**
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbCip/AbCipDriver.cs`
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.FOCAS/FocasDriver.cs`
Same mechanical change as Task 3a (interface + event + `onBatch` at the `new PollGroupEngine(...)` site). If either driver constructs the engine differently (check first), adapt but keep the identical contract. No new per-driver test required (the engine behavior is pinned by Task 2; the wiring pattern is pinned by Task 3a's test) — but the projects' existing suites must stay green.
Run: `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbCip.Tests && dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Tests`
Commit: `feat(drivers): AbCip + FOCAS implement IBatchSubscribable`
---
### Task 4: DriverInstanceActor batch forwarding
**Classification:** high-risk (actor model, subscription lifecycle)
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 5
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverInstanceActorBatchForwardTests.cs` (create; copy the TestKit + stub-driver idiom from the existing `DriverInstanceActor*Tests` in that directory)
**Step 1: Write the failing tests**
1. `Batch_driver_forwards_one_batch_message` — stub driver implementing `IBatchSubscribable`; raise `OnDataChangeBatch` with 3 items after subscribe; parent probe receives exactly ONE `AttributeValueBatchPublished` with 3 items (driver id + per-item FullReference/Value/StatusCode/timestamp mapped as `OnDataChangeForward` maps them today: `ts = Snapshot.SourceTimestampUtc ?? Snapshot.ServerTimestampUtc`).
2. `Batch_driver_per_tag_event_not_double_forwarded` — the stub raises BOTH events (per contract); parent receives only the batch message, no legacy `AttributeValuePublished`.
3. `Per_tag_driver_forwards_single_item_batches` — stub implementing only `ISubscribable`; one `OnDataChange` ⇒ parent receives one `AttributeValueBatchPublished` with 1 item.
4. `Detach_removes_batch_handler` — after Unsubscribe (or Connected→Reconnecting, whichever the existing detach tests exercise), raising the batch event produces no message.
**Step 2:** Run: `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~DriverInstanceActorBatchForwardTests"` — FAIL.
**Step 3: Implement**
- Add records next to `AttributeValuePublished` (`:109`):
```csharp
/// <summary>One value within an <see cref="AttributeValueBatchPublished"/>.</summary>
public sealed record AttributeValueItem(string FullReference, object? Value, uint StatusCode, DateTime TimestampUtc);
/// <summary>Batched form of <see cref="AttributeValuePublished"/> — every changed attribute of ONE
/// driver production burst in one message. The parent fans the whole batch to the mux and the OPC UA
/// publish path in one hop each. Status codes ride verbatim (#519).</summary>
public sealed record AttributeValueBatchPublished(string DriverInstanceId, IReadOnlyList<AttributeValueItem> Items);
```
- Private message `private sealed record DataChangeBatchForward(IReadOnlyList<DataChangeItem> Items);` + field `private EventHandler<DataChangeBatchEventArgs>? _batchChangeHandler;`
- In `HandleSubscribeAsync` (`:878-882`): when `subscribable is IBatchSubscribable batchable`, attach ONLY the batch handler (`_batchChangeHandler = (_, args) => self.Tell(new DataChangeBatchForward(args.Items)); batchable.OnDataChangeBatch += _batchChangeHandler;`) and do NOT attach `_dataChangeHandler`; else attach `_dataChangeHandler` exactly as today.
- `Receive<DataChangeBatchForward>(OnDataChangeBatchForward);` beside the existing receive (`:480`); handler builds the items list once and Tells parent one `AttributeValueBatchPublished`.
- `OnDataChangeForward` (`:1138`) becomes a single-item batch: `Context.Parent.Tell(new AttributeValueBatchPublished(_driverInstanceId, new[] { new AttributeValueItem(msg.FullReference, msg.Snapshot.Value, msg.Snapshot.StatusCode, ts) }));` — keep the `AttributeValuePublished` record declared (the mux comms-loss path and existing tests use the type), but the actor no longer sends it.
- `DetachSubscription` (`:934-943`): also detach `_batchChangeHandler` from `IBatchSubscribable` and null it.
**IMPORTANT — downstream compile/behavior:** `DriverHostActor` currently receives `AttributeValuePublished` (`DriverHostActor.cs:960`). Until Task 8 lands, add a minimal bridge in THIS task so the system stays green: in `DriverHostActor`, add `Receive<DriverInstanceActor.AttributeValueBatchPublished>(batch => { foreach (var i in batch.Items) ForwardToMux(new DriverInstanceActor.AttributeValuePublished(batch.DriverInstanceId, i.FullReference, i.Value, i.StatusCode, i.TimestampUtc)); });` — Task 8 replaces this bridge with the real batched fan-out. Existing `DriverHostActor` tests that send `AttributeValuePublished` directly stay green (the receive stays).
**Step 4:** Run: `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests` — PASS (existing DriverInstanceActor forward tests may need updating IF they assert the parent receives `AttributeValuePublished` from a live driver event — update those to expect the single-item batch; do NOT weaken assertions on content).
**Step 5:** `git add -A && git commit -m "feat(runtime): DriverInstanceActor forwards batched data changes"`
---
### Task 5: `ValueWrite` + sink `WriteValues` default method
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 3a, 3b, 4
**Files:**
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/IOpcUaAddressSpaceSink.cs`
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/DeferredAddressSpaceSink.cs`
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/AddressSpaceSinkBatchTests.cs` (create)
**Step 1: Failing tests**
1. `Default_WriteValues_loops_WriteValue` — a minimal fake implementing only the existing members; call `WriteValues` with 3 items; the fake's recorded `WriteValue` calls match in order/content.
2. `Deferred_sink_forwards_WriteValues_to_inner_once` — a fake inner that overrides `WriteValues` (records one batch call); `DeferredAddressSpaceSink.WriteValues` reaches the inner's override (NOT 3 `WriteValue` calls).
**Step 3: Implement**
In `IOpcUaAddressSpaceSink.cs`:
```csharp
/// <summary>One value write within a <see cref="IOpcUaAddressSpaceSink.WriteValues"/> batch.
/// Same semantics per field as <see cref="IOpcUaAddressSpaceSink.WriteValue"/> (#519: status verbatim).</summary>
public readonly record struct ValueWrite(
string NodeId, object? Value, uint StatusCode, DateTime SourceTimestampUtc, AddressSpaceRealm Realm);
```
Add to the interface (after `WriteValue`):
```csharp
/// <summary>Write a batch of Variable values. Semantically identical to calling
/// <see cref="WriteValue"/> per item; an SDK-backed sink overrides this to take its internal
/// lock ONCE per batch instead of once per value (the hot-path win). The default implementation
/// loops <see cref="WriteValue"/> so existing sinks and fakes behave unchanged.</summary>
void WriteValues(IReadOnlyList<ValueWrite> values)
{
ArgumentNullException.ThrowIfNull(values);
for (var i = 0; i < values.Count; i++)
{
var v = values[i];
WriteValue(v.NodeId, v.Value, v.StatusCode, v.SourceTimestampUtc, v.Realm);
}
}
```
In `DeferredAddressSpaceSink`, add the forwarding override: `public void WriteValues(IReadOnlyList<ValueWrite> values) => _inner.WriteValues(values);`
**Step 4:** `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests` — PASS.
**Step 5:** `git add -A && git commit -m "feat(commons): batched WriteValues on the address-space sink"`
---
### Task 6: OtOpcUaNodeManager batched write under one lock
**Classification:** high-risk (SDK locking)
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 7
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs` (WriteValue region, `:508-533`)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/SdkAddressSpaceSink.cs` (forward `WriteValues` to the manager)
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerBatchWriteTests.cs` (create; copy the manager-construction idiom from an existing OpcUaServer.Tests file that instantiates `OtOpcUaNodeManager`)
**Step 1: Failing tests**
1. `WriteValues_lands_every_value` — materialise 3 variables (or rely on create-on-first-write), `WriteValues` a 3-item batch, read each variable's Value/StatusCode/Timestamp back via the manager's public surface.
2. `WriteValues_creates_missing_variables` — same lazy-create semantics as `WriteValue`.
**Step 3: Implement**
- Extract the body of `WriteValue`'s `lock (Lock)` block into `private void WriteValueLocked(string nodeId, ushort ns, string key, object? value, uint statusCode, DateTime ts)` (verbatim move).
- `WriteValue` computes `ns`/`key` then `lock (Lock) { WriteValueLocked(...); }` — behavior identical.
- Add:
```csharp
/// <summary>Batched WriteValue: one Lock acquisition for the whole batch. Same per-item semantics.</summary>
public void WriteValues(IReadOnlyList<ValueWrite> values)
{
ArgumentNullException.ThrowIfNull(values);
if (values.Count == 0) return;
EnsureAddressSpaceCreated();
lock (Lock)
{
for (var i = 0; i < values.Count; i++)
{
var v = values[i];
WriteValueLocked(v.NodeId, NamespaceIndexForRealm(v.Realm), MapKey(v.Realm, v.NodeId), v.Value, v.StatusCode, v.SourceTimestampUtc);
}
}
}
```
- In `SdkAddressSpaceSink`, override `WriteValues` forwarding to the manager's method (it currently forwards `WriteValue` 1:1 — mirror that).
**Step 4:** `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests` — PASS.
**Step 5:** `git add -A && git commit -m "feat(opcuaserver): node manager batched WriteValues under one lock"`
---
### Task 7: OpcUaPublishActor batch message
**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 6
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/OpcUa/OpcUaPublishActorBatchTests.cs` (create; copy the actor-construction idiom from `OpcUaPublishActorApplyFailureTests.cs`)
**Step 1: Failing tests**
1. `Batch_message_reaches_sink_as_one_WriteValues_call` — fake sink records `WriteValues` batches; send `AttributeValueBatchUpdate` with 3 items; sink saw one batch of 3.
2. `Batch_increments_write_counter_per_item` — whatever surface exposes `_writes` (check for an existing status/metrics accessor in the actor's tests; if none, assert via the telemetry meter or skip the counter assertion and note it) — at minimum assert behavior parity: 3 items produce the same observable effect as 3 single `AttributeValueUpdate` messages.
**Step 3: Implement**
- Add next to `AttributeValueUpdate` (`:52`):
```csharp
/// <summary>Batched form of <see cref="AttributeValueUpdate"/>: every fan-out write of one driver
/// production burst in one message — the sink takes its lock once for the whole batch.</summary>
public sealed record AttributeValueBatchUpdate(IReadOnlyList<ValueWrite> Items);
```
(`ValueWrite` comes from `ZB.MOM.WW.OtOpcUa.Commons.OpcUa` — already imported.)
- Register `Receive<AttributeValueBatchUpdate>(HandleAttributeBatchUpdate);` beside `:288`.
- Handler mirrors `HandleAttributeUpdate` (`:319-331`): try/catch WARN on throw, `_sink.WriteValues(msg.Items)`, then `_writes += msg.Items.Count` via `Interlocked.Add(ref _writes, msg.Items.Count)` and `OtOpcUaTelemetry.OpcUaSinkWrite.Add(msg.Items.Count, new KeyValuePair<string, object?>("kind", "value"))` — per-item counting (invariant 9).
**Step 4:** `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~OpcUaPublishActorBatch"` then the whole project — PASS.
**Step 5:** `git add -A && git commit -m "feat(runtime): OpcUaPublishActor batched value updates"`
---
### Task 8: DriverHostActor batched fan-out (replaces the Task 4 bridge)
**Classification:** high-risk (the hottest routing seam)
**Estimated implement time:** ~5 min
**Parallelizable with:** none (needs 4 + 7; Task 9 lands with it)
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyMuxActor.cs` (Task 9 content — same commit is fine if splitting is awkward, but prefer two commits)
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorBatchFanOutTests.cs` (create; copy the host-actor fixture idiom from `DriverHostActorNativeAlarmTests.cs`)
**Step 1: Failing tests**
1. `Batch_fans_out_to_all_node_realm_refs_in_one_publish_message` — arrange the host with an applied composition where tag A maps to 1 raw + 1 UNS NodeId and tag B maps to 1 raw; send `AttributeValueBatchPublished` with A+B; the publish-actor probe receives exactly ONE `AttributeValueBatchUpdate` containing 3 `ValueWrite`s with correct NodeId/Realm/value/status/timestamp.
2. `Unmapped_items_are_dropped_not_faulted` — batch with one unmapped ref: other items still land; no crash.
3. `Batch_reaches_mux_once` — mux probe receives ONE batch message carrying all items (not N).
**Step 3: Implement**
Replace the Task 4 bridge with a real handler:
```csharp
private void ForwardBatchToMux(DriverInstanceActor.AttributeValueBatchPublished msg)
{
_dependencyMux?.Tell(msg); // one envelope per burst; the mux filters per item (Task 9)
if (_opcUaPublishActor is null) return;
List<ValueWrite>? writes = null;
foreach (var item in msg.Items)
{
if (_nodeIdByDriverRef.TryGetValue((msg.DriverInstanceId, item.FullReference), out var nodeIds))
{
writes ??= new List<ValueWrite>(msg.Items.Count * 2);
foreach (var n in nodeIds)
writes.Add(new ValueWrite(n.NodeId, item.Value, item.StatusCode, item.TimestampUtc, n.Realm));
}
else if (_log.IsDebugEnabled)
{
_log.Debug("DriverHost {Node}: no bound NodeId for ({Driver},{Ref}) — value dropped",
_localNode, msg.DriverInstanceId, item.FullReference);
}
}
if (writes is not null)
_opcUaPublishActor.Tell(new ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.AttributeValueBatchUpdate(writes));
}
```
Keep `ForwardToMux` and its `Receive<AttributeValuePublished>` registration untouched (comms-loss fan + legacy tests). Register `Receive<DriverInstanceActor.AttributeValueBatchPublished>(ForwardBatchToMux);` in place of the Task 4 bridge.
**Step 4:** `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests` — PASS.
**Step 5:** `git add -A && git commit -m "feat(runtime): DriverHostActor batched value fan-out"`
---
### Task 9: DependencyMuxActor batch receive
**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 8 (same PR-let; do after 8's records exist)
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyMuxActor.cs`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/` — add a batch case beside the existing mux tests (find them via `CalculationDependencyFlowTests.cs` / grep `DependencyMuxActor` in that project)
**Step 1: Failing test** — register a subscriber for ref X; send a batch containing X and an uninteresting Y; subscriber receives exactly one `DependencyValueChanged` for X, nothing for Y.
**Step 3: Implement** — add:
```csharp
Receive<DriverInstanceActor.AttributeValueBatchPublished>(OnAttributeValueBatchPublished);
```
```csharp
private void OnAttributeValueBatchPublished(DriverInstanceActor.AttributeValueBatchPublished msg)
{
foreach (var item in msg.Items)
{
if (!_byRef.TryGetValue(item.FullReference, out var subscribers) || subscribers.Count == 0) continue;
var dep = new VirtualTagActor.DependencyValueChanged(item.FullReference, item.Value, item.TimestampUtc, item.StatusCode);
foreach (var sub in subscribers) sub.Tell(dep);
}
}
```
Keep the per-item receive (comms-loss path).
**Step 4/5:** project tests green; `git commit -m "feat(runtime): dependency mux accepts batched publishes"`
---
### Task 10: Raise pinned-dispatcher throughput
**Classification:** trivial
**Estimated implement time:** ~2 min
**Parallelizable with:** Task 1
**Files:**
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:112-116`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs:301-312` (the inline HOCON copy)
Change `throughput = 1` → `throughput = 100` in BOTH places (they must match — the inline HOCON exists for consumers without akka.conf; see the comment at `ServiceCollectionExtensions.cs:301-303`). Update the comments: pinning alone guarantees the SDK's single-thread requirement; throughput only sets how many queued messages drain per scheduling run, so 100 removes a mailbox re-schedule round-trip per message with zero concurrency change.
Run: `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~ServiceCollectionExtensionsTests"`
Commit: `perf(runtime): raise opcua pinned dispatcher throughput 1 -> 100`
---
### Task 11: Re-key node-manager maps (kill per-write key allocations)
**Classification:** high-risk (38 call sites, mechanical but wide)
**Estimated implement time:** ~5 min (compiler-driven)
**Parallelizable with:** none (after Task 6 — same file)
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs`
**Step 1: Baseline** — run `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests` green BEFORE starting (this task relies on the existing suite as its net; it should need no new tests, but the whole suite must pass unchanged after).
**Step 2: Implement**
- Add near `MapKey` (`:126-139`):
```csharp
/// <summary>Zero-allocation realm-qualified map key. Replaces the formatted "ns=N;s=id" string key —
/// the (realm, id) pair is the same identity without the per-write NodeId + string allocations.</summary>
private readonly record struct RealmKey(AddressSpaceRealm Realm, string Id);
```
- Change `MapKey(AddressSpaceRealm realm, string id)` to return `new RealmKey(realm, id)`; change `MapKey(NodeId nodeId)` to `new RealmKey(RealmOf(nodeId.NamespaceIndex), (string)nodeId.Identifier)` — `RealmOf` already exists (~`:123`); confirm its exact name/signature and that every `MapKey(NodeId)` call site only ever passes nodes from the two owned namespaces (they do — SDK-facing seams on owned handles). If `nodeId.Identifier` can be non-string on any call path (numeric-id nodes), guard with `as string ?? nodeId.Identifier?.ToString() ?? string.Empty`.
- Change every map/set keyed by the old string key to `RealmKey`: `_variables`, `_folders`, `_alarmConditions`, `_historizedTagnames`, `_eventNotifierSources`, `_nativeAlarmNodeIds`, and the notifier-wiring bookkeeping — let the compiler find all of them (38 `MapKey` call sites).
- Prefix scans (`RemoveEquipmentSubtree` ~`:2539-2596`, `InScope` usages): match on `key.Realm == realm && key.Id.StartsWith(prefix, ...)` — semantics identical (the old string key embedded the realm as `ns=N;`).
- `ResolveNodeState(string key)` (~`:2240-2261`): change its parameter to `RealmKey`.
- Do NOT change public API surface or any behavior — this is a pure key-representation change.
**Step 3:** `dotnet build ZB.MOM.WW.OtOpcUa.slnx` then `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests` and `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests` — ALL green, zero test edits expected (tests drive the public surface). If a test pokes the key format directly, stop and reassess rather than bending the test.
**Step 4:** `git add -A && git commit -m "perf(opcuaserver): struct realm keys replace formatted NodeId string keys"`
---
### Task 12: Full verification + docs
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none (last)
**Files:**
- Modify: `CLAUDE.md` (only if a statement it makes is now false — check the pinned-dispatcher/`throughput` mentions; keep edits surgical)
- Modify: `docs/v2/Galaxy.Performance.md` — add a short "value-path batching" note (one paragraph: the batch seams, the invariants, the dispatcher change)
Steps:
1. `dotnet build ZB.MOM.WW.OtOpcUa.slnx` — zero warnings introduced.
2. `dotnet test ZB.MOM.WW.OtOpcUa.slnx` — run what the environment can (DB-backed suites need `10.100.0.35`; if unreachable, run every non-DB project under tests/Core, tests/Drivers (unit), tests/Server/Runtime.Tests, tests/Server/OpcUaServer.Tests and record which suites were skipped and why in the commit message).
3. Docs + commit: `docs: record value-pipeline batching (plan 2026-08-15)`.
---
## Explicit non-goals (do NOT let scope creep into these)
- **No conflation / bounded mailboxes** — batching per burst is this plan's backpressure story; keyed conflation is a follow-up.
- **No fan-out-map push-down to DriverInstanceActor** — the host stays the router.
- **No MQTT / MTConnect / Sparkplug / Galaxy / OpcUaClient batch implementations** — they flow through the single-item-batch path unchanged.
- **No change to alarm paths** (`AttributeAlarmPublished`, `AlarmStateUpdate`) — value path only.
- **No PollGroupEngine cadence change** (fixed-rate scheduling is a separate item).
---
## Follow-ups (recorded, not implemented)
Raised during Tasks 1–12 review and deliberately left out of this plan's scope. None is a defect in the
delivered work; each is a place where the next change in the area could go wrong quietly.
1. **`PollGroupEngine` collects batch items unconditionally — the batch event is only as trustworthy as
the driver's `onChange`.** The engine has no knowledge of a driver's publish filter, so its items list
is pre-filter by construction. That is safe *today* only because S7, AbCip and FOCAS forward every
`onChange` unconditionally, and Modbus — the one driver with a deadband — ignores the engine's list and
raises from its own filtered buffer. The trap: a future per-tag filter added to S7/AbCip/FOCAS's
`onChange` would silently reintroduce the exact deadband-bypass class of bug fixed for Modbus in
`cd4ce305`, with a green test suite (the two events would simply disagree, and only a consumer
preferring the batch would notice). Consider changing the engine's `onChange` contract to
`Func<..., bool>` so the engine collects an item only when the driver reports it published — making the
filter structurally impossible to skip rather than a convention each driver must remember.
2. **Two-level `driverId → (rawPath → refs)` routing map in `DriverHostActor`.** The fan-out map is keyed
by a composite of driver id + raw path, so resolving a batch re-hashes the driver id once per item even
though a batch is by definition single-driver. Nesting the map would hash the driver id once per batch
and the raw path once per item. Measure first — batches are small enough that this may be noise
relative to the message-count win already banked.
3. **Null/empty guard consistency on the internal batch messages.** The guards on
`AttributeValueBatchPublished` / `AttributeValueBatchUpdate` grew per task and are not uniform — some
seams drop an empty batch early, others tolerate it downstream. These are internal messages with
trusted senders so nothing is unsafe today; worth one pass to make the contract identical at every hop
and stated once, rather than re-derived per reader.
4. **Possible merge of `DriverHostActorLiveValueTests` into the batch suite.** After Task 8 both files
drive the same fan-out through the same message; the split is now historical rather than meaningful.
Merging would remove a genuine ambiguity about where a new fan-out test belongs — but check first that
the older suite's per-item comms-loss cases really are covered, since those enter via a different seam.
5. **Aborted-cycle tail-loss when a per-tag handler throws, for the three unbuffered drivers.** If a
consumer's `onChange` throws mid-cycle in S7/AbCip/FOCAS, the cycle aborts and the items already
collected for that cycle never reach a batch raise. **Pre-existing and narrower than the Modbus case**
(Modbus's own buffer is per-handle and is dropped on unsubscribe/dispose, so it cannot leak), and it
predates this plan — the per-tag path lost the same tail before batching existed. Recorded so it is not
mistaken for a regression introduced here.