From b9e9df4081b67c2a45d0d94f16cc8a74ea43215d Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 16 Aug 2026 05:33:33 -0400 Subject: [PATCH] =?UTF-8?q?docs(code):=20honest=20rationales=20=E2=80=94?= =?UTF-8?q?=20Akka=20restart=20semantics,=20per-key=20shards,=20EF=20liter?= =?UTF-8?q?al=20inlining;=20retire=20or=20annotate=20orphaned=20members?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CLAUDE.md | 15 +++++-- .../2026-08-15-perf-roadmap-remainder.md | 10 +++++ docs/security.md | 28 +++++++----- .../Historian/IHistorizationOutbox.cs | 11 ++++- .../IHistorizedTagSubscriptionSink.cs | 8 ++-- .../Uns/RawTreeService.cs | 11 ++++- .../OtOpcUaNodeManager.cs | 8 ++++ .../Security/SessionAclDecisionCache.cs | 43 ++++++++++++------- .../ContinuousHistorizationRecorder.cs | 12 ++++-- .../ScriptedAlarms/ScriptedAlarmHostActor.cs | 25 ++++++----- .../DependencyConsumerMuxAdapter.cs | 20 +++++---- 11 files changed, 133 insertions(+), 58 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index d4194a01..ea643afd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -281,10 +281,17 @@ non-`Healthy` state **and** its `LastSuccessfulRead` has been frozen for 90 s. T all 12: nine poll drivers via `Degraded`, MQTT via live connection state, Galaxy via its reconnect supervisor. `CalculationDriver` reports `Healthy` while running and so can never trip — correct, as it has no transport. Re-check this if a new driver lands. -- ⚠️ **The reconnect retry MUST tear down before re-initialising.** `ModbusDriver` (`_transport`) and - `MqttDriver` (`_connection`) overwrite an `IAsyncDisposable` field inside `InitializeAsync` without - disposing the old one, so re-initialising without `ShutdownAsync` leaks a socket per attempt — ~360 for an - hour-long outage at the 10 s retry interval. Latent while only a manual Force Reconnect reached this path. +- **The reconnect retry MUST tear down before re-initialising — FIXED in both drivers, keep it that way.** + `ModbusDriver` (`_transport`) and `MqttDriver` (`_connection`) used to overwrite an `IAsyncDisposable` + field inside `InitializeAsync` without disposing the old one, so re-initialising without `ShutdownAsync` + leaked a socket per attempt — ~360 for an hour-long outage at the 10 s retry interval. Both now dispose + first: `ModbusDriver.InitializeCoreAsync` releases `_transport` **and** the `_probeCts`/`_reprobeCts` + probe loops before rebuilding them (`c3970a2e` + the lifecycle gate in `409a58b6`), deliberately *not* + via the shared `TeardownAsync()` because that would also drop `_poll` and every active subscription with + nothing to re-drive them; `MqttDriver.InitializeCoreAsync` calls its (idempotent) `TeardownAsync()` first, + which reaches `_connection` and the primary-host connection (`2b4b3fcb`). The path is no longer latent — + the comms-loss watchdog reaches it, not just a manual Force Reconnect — so a new driver, or a new + `IAsyncDisposable` field on these two, must repeat the dispose-before-overwrite. - The threshold is hardcoded (`DefaultCommsLossTimeout`), matching `HealthPollInterval` / `DefaultReconnectInterval`. Unlike the #523 drift detector this generates **no** traffic on a healthy system — it is a local `GetHealth()` read, and only reaches the network once a driver has already stopped diff --git a/docs/plans/2026-08-15-perf-roadmap-remainder.md b/docs/plans/2026-08-15-perf-roadmap-remainder.md index be410bb7..3ab138df 100644 --- a/docs/plans/2026-08-15-perf-roadmap-remainder.md +++ b/docs/plans/2026-08-15-perf-roadmap-remainder.md @@ -1413,3 +1413,13 @@ Open items the work surfaced but deliberately did not close. None blocks the bra `ThreadStatic` actor cell, so under a loaded sweep the intended-red mutation can occasionally pass. The test is still worth keeping; do not read a single green run of it as proof the regression is caught. + +**Accepted-not-fixed at the final integration review (2026-08-15)** — carried knowingly, none +blocking: the poll **pacing** behaviour change ships with no production consumer of the overrun +counter (the per-driver meter follow-up above is the mitigation); **no live gate was run** for this +branch (docker-dev recipe: +[`2026-08-01-continuous-historization-live-gate.md`](2026-08-01-continuous-historization-live-gate.md)); +`_globalCount` drift is **negative-only** in direction, so it can only trim early, never late; +`ScriptTagCatalogSnapshotCache` expires on **wall-clock TTL**; the S7 span reader **stamps a partial +span** when the PDU comes back short; MTConnect's `IsFramingOnly` fast path is **ASCII-only**; and a +minor Razor exposure where `StateHasChanged` runs inside a `finally`. diff --git a/docs/security.md b/docs/security.md index d873997f..1eb6bf42 100644 --- a/docs/security.md +++ b/docs/security.md @@ -446,27 +446,35 @@ for the life of a generation, so memoizing the walk is sound. A key's `ExpiresUt *clock*, not with a deployment — so revocation and expiry are re-checked on every call, outside the cache. Caching them would have let an expired key keep working until the next deploy. -**The cache is sharded per session, because a flat one let any client evict every other client's -decisions.** It used to be a single dictionary with the session identity folded *into* the composite key +**The cache is sharded per calling KEY, because a flat one let any client evict every other client's +decisions.** It used to be a single dictionary with the caller identity folded *into* the composite key and one process-wide bound of `MaxEntries = 100_000`, whose only eviction was a wholesale `Clear()` at the cap. So one machine consumer crawling a large address space past 100k entries wiped the memoized -walks of every other session on the server, and each store paid a `ConcurrentDictionary.Count` — an -all-locks snapshot — to test the cap. It is now a dictionary of per-session shards keyed by the session's -`ApiKeyId ?? Principal`, each holding its own `(operation, realm, nodeIdentifier)` entries, with two +walks of every other caller on the server, and each store paid a `ConcurrentDictionary.Count` — an +all-locks snapshot — to test the cap. It is now a dictionary of shards keyed by +**`ApiKeyId ?? Principal`**, each holding its own `(operation, realm, nodeIdentifier)` entries, with two bounds: | Bound | Value | What it clears | |---|---|---| -| `MaxEntriesPerSession` | 20 000 | that one session's shard | +| `MaxEntriesPerSession` | 20 000 | that one **key's** shard | | `MaxEntries` | 100 000 | every shard (the shard *objects* are dropped, not emptied) | -Both are compile-time constants, not appsettings keys. Five sessions simultaneously at their per-session -ceiling reach the global cap; **the number of shards is not itself bounded** — the bound is entries. +⚠️ **The shard key is the API key, not the session — the constant's name predates it.** Every session +authenticating with the same `otua__…` credential shares one shard and one 20 000-entry budget, +so a crawl by one of them *can* clear the memoized decisions of its sibling sessions. What the sharding +guarantees is isolation **across distinct keys**: one key's crawl can never evict another key's entries. +(An Anonymous session shards under the reserved `anonymous` principal, so all of them likewise share +one shard.) + +Both bounds are compile-time constants, not appsettings keys. Five keys simultaneously at their +per-key ceiling reach the global cap; **the number of shards is not itself bounded** — the bound is +entries. Things to know before changing it: -- **There is no session-close eviction hook**, deliberately. A closed session's shard is shed the next - time a wholesale clear happens — a generation bump, or the global cap tripping. Dropping the shard +- **There is no session-close eviction hook**, deliberately. The shard of a key with no live sessions + left is shed the next time a wholesale clear happens — a generation bump, or the global cap tripping. Dropping the shard objects rather than emptying them is what makes that sweep work. - **The counters are approximate under concurrency, by design.** The cap test is a check-then-act over `Volatile.Read` of `Interlocked` counters rather than a lock, so concurrent stores can overshoot a cap diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorizationOutbox.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorizationOutbox.cs index 626bb5cd..9b48653b 100644 --- a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorizationOutbox.cs +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorizationOutbox.cs @@ -13,7 +13,16 @@ public interface IHistorizationOutbox : IDisposable /// Lifetime count of entries dropped because an append would have exceeded capacity. long DroppedCount { get; } - /// Appends to the tail of the durable buffer. + /// + /// Appends to the tail of the durable buffer. + /// + /// Retained for tests and direct callers; production appends only through + /// . The recorder coalesces a driver burst into one group + /// commit, so this single-entry form has no production call site — it is kept as the + /// one-entry expression of the same contract (and is what 's + /// durability window is described against), not as a deprecated path. + /// + /// /// The value record to buffer. /// Cancellation token. /// A task that represents the asynchronous operation. diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorizedTagSubscriptionSink.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorizedTagSubscriptionSink.cs index 984c8395..ee1cfab5 100644 --- a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorizedTagSubscriptionSink.cs +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorizedTagSubscriptionSink.cs @@ -5,9 +5,11 @@ namespace ZB.MOM.WW.OtOpcUa.Core.Abstractions; /// because a HistorianTagname override makes them diverge: /// /// -/// — the driver-published reference the per-node dependency mux fans its -/// DependencyValueChanged by (the tag's driver-side FullName). The recorder -/// registers mux interest and matches incoming values by THIS. +/// — the driver-published reference the per-node dependency mux fans values +/// by (the tag's driver-side FullName); in production those arrive as a +/// DependencyValueBatch per driver burst, with the per-value DependencyValueChanged +/// retained for direct senders and tests. The recorder registers mux interest and matches +/// incoming values by THIS. /// /// /// — the resolved historian tag name the value is written under (a diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/RawTreeService.cs b/src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/RawTreeService.cs index 25e77479..2c0f4c29 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/RawTreeService.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/RawTreeService.cs @@ -1491,8 +1491,15 @@ public sealed class RawTreeService(IDbContextFactory dbF var sourceCode = Expression.Property(parameter, nameof(Script.SourceCode)); var contains = typeof(string).GetMethod(nameof(string.Contains), [typeof(string)])!; - // Indexing a captured array (rather than Expression.Constant(path)) keeps EF parameterising each - // path instead of inlining it as a SQL literal, so the paths never reach the generated SQL text. + // Indexing an array (rather than Expression.Constant(path) directly) keeps the expression tree + // cache-friendly: every rename of the same path-count produces the same tree shape. + // ⚠️ It does NOT guarantee parameterisation. The array here is a constant with no closure over it, + // so EF may well inline each path into the SQL as an escaped literal — which is safe (EF escapes + // the Contains argument either way, so there is no injection surface) but means a rename can emit up + // to ScriptScanBatchSize (200) literals per batch and churn the plan cache. Whether the provider + // actually parameterises this shape is unverified: proving it needs a SQL-Server-backed test, not + // EF InMemory. Recorded in the follow-up register of + // docs/plans/2026-08-15-perf-roadmap-remainder.md. var values = paths.ToArray(); Expression? body = null; for (var i = 0; i < values.Length; i++) diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs index 3410abeb..14c61341 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs @@ -316,6 +316,14 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 /// /// Re-authorizes live subscriptions when a deployment installs a new authorization generation. /// + /// + /// ⚠️ This is the only trigger, and the pass's memo depends on that. + /// memoizes key revocation and expiry for the span of one + /// pass — safe only while revalidation is deploy-triggered, because a pass is then a momentary + /// event whose verdicts cannot go stale within it. A future periodic trigger must give each tick a + /// fresh memo (it already does — the dictionary is pass-local) and must not be tempted to reuse one + /// across ticks, or an expiry landing between ticks would be served the previous tick's allow. + /// /// The newly published snapshot; unused — the current state is re-read per item. private void OnAuthorityChanged(DataPlaneAuthoritySnapshot snapshot) => RevalidateMonitoredItems(); diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/Security/SessionAclDecisionCache.cs b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/Security/SessionAclDecisionCache.cs index 15d004c6..f9e1fbc4 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/Security/SessionAclDecisionCache.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/Security/SessionAclDecisionCache.cs @@ -32,13 +32,16 @@ namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.Security; /// deploy costs one repopulation and cannot be wrong. Deployments are rare relative to reads. /// /// -/// Storage is sharded per session, under a two-level bound. Each session key owns its own -/// dictionary, capped at ; overflowing it clears that shard -/// only. A second, global bound clears every shard, and exists purely -/// so many sessions each staying under their own cap cannot add up to unbounded growth. The flat -/// cache this replaced had only the global bound, which made one machine consumer crawling a large -/// tree an eviction event for every other session on the server — the sessions doing ordinary work -/// paid for the crawl. Both caps are enforced against counters, never +/// Storage is sharded per calling KEY, under a two-level bound. The shard key is +/// ApiKeyId ?? Principal, so sessions sharing one API key share one shard and one +/// cap — the isolation is across distinct keys, not across +/// sessions. Overflowing a shard clears that shard only, so a crawl by one key cannot evict +/// another key's entries. A second, global bound clears every shard, and +/// exists purely so many keys each staying under their own cap cannot add up to unbounded growth. +/// The flat cache this replaced had only the global bound, which made one machine consumer +/// crawling a large tree an eviction event for every other caller on the server — the callers +/// doing ordinary work paid for the crawl. Both caps are enforced against +/// counters, never /// , which locks the whole table. /// /// @@ -46,7 +49,7 @@ public sealed class SessionAclDecisionCache { /// /// Process-wide entry ceiling, past which every shard clears. The outer of the two bounds, - /// and a rare fallback: it exists so a large fleet of sessions each staying well under + /// and a rare fallback: it exists so a large fleet of distinct keys each staying well under /// still cannot grow the cache without limit. /// /// @@ -59,16 +62,18 @@ public sealed class SessionAclDecisionCache public const int MaxEntries = 100_000; /// - /// Per-session entry ceiling, past which that session's shard alone clears. The inner and normally - /// the only bound reached: it is what keeps one machine consumer crawling a 50k-node tree from - /// evicting the decisions of every other session on the server. + /// Per-key entry ceiling (the shard key is ApiKeyId ?? Principal, so every session + /// using the same API key shares this one budget), past which that shard alone clears. The inner + /// and normally the only bound reached: it is what keeps one machine consumer crawling a 50k-node + /// tree from evicting the decisions taken for every OTHER key on the server. The name predates the + /// shard key and is kept for source compatibility. /// /// /// Clearing beats evicting within a shard: an LRU needs per-entry bookkeeping on the hot path /// to save a trie walk that is itself cheap, so the eviction policy would cost more than the miss it /// prevents. The failure mode of clearing is a burst of recomputation, which is exactly the steady - /// state of not having a cache at all — and sharding confines that burst to the session that caused - /// it, which is the part the old flat cache got wrong. + /// state of not having a cache at all — and sharding confines that burst to the key that caused it, + /// which is the part the old flat cache got wrong. /// public const int MaxEntriesPerSession = 20_000; @@ -105,7 +110,7 @@ public sealed class SessionAclDecisionCache /// answer is already memoized, so the allocation the cache exists to avoid is paid on the hit /// path too. Read and Browse ask this question once per node, which is precisely where paying /// for a walk that never runs is worth removing. remains, delegating - /// here, for callers that genuinely have nothing cheaper to offer. + /// here, but has no production caller — it is exercised by the cache's own tests. /// /// /// The generation is checked (and a stale cache dropped) here rather than in @@ -187,8 +192,14 @@ public sealed class SessionAclDecisionCache /// /// Returns the memoized decision for this key, computing and storing it when absent or when the - /// generation has moved on. A convenience over + for callers - /// that have no cheaper way to express the walk than a closure. + /// generation has moved on. A convenience over + . + /// + /// Retained for tests; production has no caller. uses + /// the split / pair precisely so a hit pays for no closure, + /// and no other caller has appeared since. Kept because it is the readable single-expression + /// form of the same protocol and is how the cache's tests drive it — not because a production + /// caller is expected back. + /// /// /// The authorization generation the decision is being taken under. /// Stable identity of the calling session — its key id, else its principal. diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ContinuousHistorizationRecorder.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ContinuousHistorizationRecorder.cs index 799d9606..e66d7a8d 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ContinuousHistorizationRecorder.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ContinuousHistorizationRecorder.cs @@ -8,16 +8,20 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.Historian; /// /// Continuous-historization engine for non-Galaxy (driver) tags. Registers interest with the /// per-node for the configured historized tag refs, then taps the -/// values the mux fans for those refs: +/// values the mux fans for those refs — in production a +/// per driver burst, with the per-value +/// receive retained for direct senders and tests: /// value → coerce to numeric → append to the durable (the crash /// boundary) → a self-paced drain writes the outbox through 's /// live-value path and acks (truncates) on success. /// /// /// -/// Message path (grounded against the real actors). The mux fans -/// — which carries only -/// (TagId, Value, TimestampUtc) and no quality — to every registered subscriber. +/// Message path (grounded against the real actors). The mux fans each value — carrying +/// only (TagId, Value, TimestampUtc) and no quality — to every registered +/// subscriber, batched as a on the production +/// path and as a bare from a direct sender +/// or a test. /// The recorder reuses that existing fan-out (no change to DriverHostActor), so values /// arrive Good-quality by the same convention the scripted-alarm host uses (the mux only /// forwards driver-published values); historized samples are recorded with diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmHostActor.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmHostActor.cs index 621a0ea4..3ec6e08c 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmHostActor.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmHostActor.cs @@ -29,9 +29,11 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.ScriptedAlarms; /// /// /// -/// Data flow. delivers a -/// for every tag the loaded alarms -/// depend on; the host pushes each into the upstream. The engine self-evaluates: its own +/// Data flow. delivers the values of every tag the +/// loaded alarms depend on — in production as a batched +/// per driver burst, with the per-value +/// receive retained for direct senders and +/// tests; the host pushes each into the upstream. The engine self-evaluates: its own /// observer re-runs the predicates and raises /// . The host does NOT call evaluate — it only /// feeds values in and fans emissions out. @@ -134,9 +136,10 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor /// Factory method to create Props for a . /// The OPC UA publish actor that consumes /// bridged from engine emissions. - /// Optional dependency multiplexer the host registers interest with so it - /// receives a per dependency tag. Null on the - /// dev/Mac path (no live values). + /// Optional dependency multiplexer the host registers interest with so it receives + /// its dependency tags' values — a per burst in + /// production, or a per-value from a direct + /// sender or a test. Null on the dev/Mac path (no live values). /// The mux-fed upstream the engine reads + subscribes from. MUST be the /// same instance the was constructed around. /// The scripted-alarm engine this host owns + disposes. @@ -403,10 +406,12 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor /// Loops the per-value handler, preserving in-burst order. /// /// Mid-batch exception policy: unguarded, deliberately. An exception escaping - /// drops the remaining items of this burst and restarts the actor - /// — exactly what it already did when the same values arrived as N separate messages (the tail is - /// then lost to the mailbox flush instead of to the loop), so batching changes the blast radius by - /// nothing. Catching per item is deliberately NOT done: the handler is a single + /// drops the remaining items of this burst and restarts the actor. + /// That IS wider than the per-message path — Akka restarts do not flush the mailbox, only the + /// failing message is dropped, so as N separate messages the same fault would cost one value rather + /// than the burst's tail. Accepted: the engine tolerates a missed intermediate value (the next + /// delivery for that tag re-converges its state). Catching per item is deliberately NOT done: the + /// handler is a single /// IUpstreamValueSource.Push into an in-memory cache, so a throw means the upstream itself is /// broken and swallowing it would feed the engine a silently truncated burst. Contrast /// ContinuousHistorizationRecorder.IngestAsync, which DOES catch per item because its diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyConsumerMuxAdapter.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyConsumerMuxAdapter.cs index 74f82867..a33bc254 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyConsumerMuxAdapter.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyConsumerMuxAdapter.cs @@ -8,9 +8,11 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.VirtualTags; /// Bridges a driver that implements onto the per-node /// . Spawned as a child of DriverHostActor for each such /// driver: on start (and on every apply via ) it registers the driver's -/// declared with the mux, and forwards each -/// the mux fans out into -/// . +/// declared with the mux, and forwards each value +/// the mux fans out into . The production +/// delivery is the batched — one message per +/// driver burst; the per-value receive remains +/// for direct senders and tests, and the batch handler simply loops it. /// /// /// This is the exact seam a VirtualTagActor uses to receive its inputs, generalised so a @@ -76,11 +78,13 @@ public sealed class DependencyConsumerMuxAdapter : ReceiveActor /// /// Mid-batch exception policy: unguarded, matching the single-value path. An exception /// escaping drops the remaining items of this - /// burst and restarts the adapter — the same outcome the single-value handler already has (there the - /// tail is lost to the mailbox flush rather than to the loop), so batching does not widen it. - /// Catching per item HERE only would make the batch path quietly more forgiving than the single-value - /// one for the same consumer fault, so the two are deliberately kept identical: a consumer that needs - /// per-value fault tolerance owns it inside its own OnDependencyValue. + /// burst and restarts the adapter. This DOES widen the per-fault loss: Akka restarts do not flush + /// the mailbox (only the failing message is dropped), so as N separate messages the same fault + /// would have cost one value, while batched it costs the rest of the burst's tail. Accepted, + /// because an tolerates missed intermediate values — the next + /// delivery re-converges its state — and catching per item HERE only would make the batch path + /// quietly more forgiving than the single-value one for the same consumer fault. A consumer that + /// needs per-value fault tolerance owns it inside its own OnDependencyValue. /// /// The batch of matched dependency values. private void OnDependencyValueBatch(VirtualTagActor.DependencyValueBatch msg)