cde5cdfd9fde81863a677f45a1fbca63b18d06ad
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joseph Doherty
|
aec5ce565a |
docs: record value-pipeline batching (plan 2026-08-15)
- docs/v2/Galaxy.Performance.md gains a "Value-path batching (2026-08-15)" section: the batch seams (IBatchSubscribable / PollGroupEngine onBatch -> AttributeValueBatchPublished -> AttributeValueBatchUpdate -> sink WriteValues with 512-chunked single-lock batches in the node manager), the invariants (#519 status codes verbatim, per-ITEM metering, Modbus deadband honored via the driver-filtered buffer, the ValueWriteBatchException partial-failure contract), the RealmKey re-key, and the pinned dispatcher's throughput 1 -> 100. - The plan doc gains a "Follow-ups (recorded, not implemented)" section: the PollGroupEngine unconditional-collection trap (a future per-tag filter in S7/AbCip/FOCAS would silently reintroduce the deadband-bypass bug), a two-level driverId -> (rawPath -> refs) routing map, null/empty guard consistency on the internal batch messages, a possible DriverHostActorLiveValueTests merge, and the pre-existing aborted-cycle tail-loss on the three unbuffered drivers. - tasks.json: all 13 tasks completed. CLAUDE.md needed no edit: its only statement in this area ("One driver publish for a RawPath fans (in DriverHostActor) to the raw NodeId AND every referencing UNS NodeId") is still true — batching changed the message granularity, not the fan-out semantics. Verification: dotnet build ZB.MOM.WW.OtOpcUa.slnx clean (0 errors, no new warning classes, zero warnings in any production file). 9 unit projects green, 3470 passed / 5 skipped / 0 failed: Core.Abstractions 305, Core 351+1s, Commons 348, Runtime 575+3s, OpcUaServer 424+1s, Modbus 399, S7 302, AbCip 410, FOCAS 356. No DB-backed suite was run — tests/Server/ZB.MOM.WW.OtOpcUa.Server.Tests does not exist, and nothing this branch touches is covered by the SQL-backed projects. |
||
|
Joseph Doherty
|
c51c59987b |
fix(galaxy): publish an unrepresentable value as BadTypeMismatch, not a good null
Reverses the conservative call from the previous commit at the user's direction, and they were right: "surface it on a counter first" left the wire lying. An MXAccess event can carry Good quality while its payload is a kind MxValueDecoder has no mapping for, so the old behaviour published null-under-Good — which a client cannot tell apart from a genuine null reading. A value the gateway DID send would be consumed as a value it did not, with the only evidence in a log nobody is tailing. Bad data is the honest wire representation of lost data. BadTypeMismatch is an existing StatusCodeMap constant, deliberately reused rather than written inline: StatusCodeParityTests finds status constants by reflecting over Good/Uncertain/Bad-prefixed uint fields and checking them against the pinned SDK, so an inline literal would be invisible to that guard (Gitea #497, where 16 wrong constants had accumulated across 6 drivers exactly this way). Logging is now once per DISTINCT value kind rather than once per pump. The set is bounded by the MxValue oneof enum so a broken producer still cannot flood, but a second unrepresentable kind appearing later gets its own line — warn-once-per-pump would have silently swallowed it, which is the same class of bug as the null this commit fixes. The test's fixture matters more than its assertion: the undecodable event carries Quality = 192 (OPC DA Good), so a decoder that gave up silently would publish Good and fail. Paired with a positive control — a decodable value at the same quality must still publish Good — so it cannot be satisfied by a pump that marks everything Bad. Falsified by reverting the status to Good: the test reds, and only that test. Verified: 331 Galaxy tests green, Core.Abstractions 298 (the parity guard), solution builds 0 errors. Docs corrected in the two places that recorded the old decision as deliberate. |
||
|
Joseph Doherty
|
4769044f88 |
feat(galaxy): handle the replay-gap sentinel and sparse-array values
Closes the two follow-ups left open by the MxGateway 0.2.0 bump. Reading the
gateway's own contract first changed what each of them is, and one of my earlier
claims was simply wrong -- both corrections are recorded below.
Replay gap
----------
A sentinel MxEvent carries family = UNSPECIFIED, so it fell through EventPump's
family switch into the default drop: we silently ignored a documented MUST ("you
missed events -- discard local state and re-snapshot"). It is now intercepted and
answered by re-issuing SubscribeBulk for every tracked subscription, since
MXAccess delivers an initial value per item on subscribe -- re-subscribing IS the
re-snapshot. It does not restart the event pump (the stream carrying the sentinel
is healthy, and the handler runs on that stream's own read loop, so disposing the
pump from there would deadlock on itself), it does not block the read loop, and
re-entry is gated so a burst collapses to one pass instead of a subscribe storm
per sentinel.
Handled on the READ loop, not in the dispatch loop, because everything past that
point is subject to newest-dropped channel pressure -- and "you missed events" is
the one signal whose loss is unrecoverable. Pinned by a test that fills the
channel and still expects the sentinel.
⚠️ This cannot fire in the shipped configuration, and I did not pretend otherwise
or paper over it by enabling resume. The gateway emits the sentinel only for a
stream opened with after_worker_sequence > 0; GatewayGalaxySubscriber passes 0,
which the gateway treats as "fresh stream, not a resume -- plain attach, no
replay, no sentinel". Enabling resume is a larger and genuinely delicate change:
the sequence is only meaningful within ONE gateway session, our reconnect
recreates the session, and replaying a remembered sequence against a fresh
session sets the server's live watermark ABOVE every event it is about to send --
silencing the stream instead of recovering it. That deserves its own change with
a live gate, which I cannot run from here.
Sparse arrays -- correcting the last commit message
---------------------------------------------------
I wrote that the bump "makes handling it possible" on the decode path. That is
wrong: MxSparseArray is documented write-only, the gateway expands it before
forwarding, and "the worker never receives or produces it". So the decode path
is unreachable by contract and the write path has no use for it either -- it is a
wire compaction for a WHOLE-array write, not a partial update (unmentioned
indices are RESET, deliberately, because the gateway cannot read-modify-write
without racing the provider). It is therefore the wrong primitive for an OPC UA
IndexRange write, which expects untouched elements preserved -- and WriteRequest
carries no IndexRange in the first place.
It is decoded anyway, as contract defence: the alternative if a future gateway
ever emits one is a silent null published under Good quality. Fill semantics
mirror the gateway's own SparseArrayExpander exactly (epoch for Time, empty
string, Integer widening to long[] when any element is int64) so a value that
round-trips through a write reads back as the same CLR array.
Also: the decoder is now total by contract, since it runs inside the dispatch
loop where an escaping exception kills dispatch for the life of the pump. A
malformed payload degrades instead of throwing.
A real defect found in the same code
------------------------------------
A stream the GATEWAY completes -- a closed or evicted session -- ends the RPC
without an error, so nothing threw, the supervisor was never notified, and the
pump exited silently. GetHealth() kept reporting Healthy, which also means the
#527 comms-loss watchdog could never trip, because it requires a non-Healthy
state. The driver went permanently deaf with no signal anywhere. This one is
live, unlike the two above. Now reported as a fault; the falsification control is
the existing CleanShutdown_DoesNotInvokeOnStreamFault, since our own disposal
also ends the stream and must NOT be reported.
Value kinds we cannot represent are now counted and warned once rather than
published as an indistinguishable null. The status is deliberately NOT downgraded
to Bad: nothing can reach that branch today, and changing what an OPC UA client
sees for a live shape is not a change to make without a gate. The counter exists
so the next person has evidence instead of a guess.
Verified: 330 Galaxy tests green (12 new), plus Galaxy.Browser 21, Runtime 558,
Core 351; full solution builds 0 errors with the warning census unchanged. Every
new test was falsified first -- disabling the gap interception reds 3 of 4 (the
throwing-callback test passes vacuously when the callback never runs, by
construction), disabling sparse decode plus the representability signal reds all
7 decoder tests, and both were restored and re-verified green.
Docs: the Galaxy.Performance invariant was stated as received = dispatched +
dropped + in-flight, which this breaks -- a sentinel is received but never enters
the channel -- so it now reads + replay_gaps. Galaxy.md gains the three
operational notes; DriverLifecycle.md records that "reaches a non-Healthy state"
is an obligation a driver can silently fail to meet, which is the general shape
of the clean-end bug.
|
||
|
Joseph Doherty
|
0001cdd579 |
fix(scripted-alarms): reuse per-alarm evaluation scratch on the hot path
Core.ScriptedAlarms-009 resolution: replace the per-call Dictionary +
AlarmPredicateContext allocation with a per-alarm reusable AlarmScratch
held in _scratchByAlarmId, refilled in place under _evalGate on each
evaluation. The hot path no longer allocates per upstream tag change.
Why this matters:
On a busy line where many tags feeding many alarms change frequently,
the old BuildReadCache allocated a fresh dictionary + context on every
predicate evaluation — a steady stream of short-lived allocations the
GC eventually has to reclaim. With the reuse, the dictionary and
context are allocated once per alarm (on first evaluation) and refilled
in place across every subsequent re-eval.
Implementation:
- New private AlarmScratch class holds the reusable
Dictionary<string, DataValueSnapshot> read cache (pre-sized to the
alarm's Inputs.Count) and the AlarmPredicateContext that wraps it by
reference. The context observes refilled values without being
re-created.
- ConcurrentDictionary<string, AlarmScratch> _scratchByAlarmId on the
engine, cleared in LoadAsync alongside _alarms so a config-publish
drops the prior generation's scratch (Inputs / Logger may change).
- EvaluatePredicateToStateAsync looks up scratch via GetOrAdd, calls
the new RefillReadCache(Dictionary, IReadOnlySet) helper to clear +
repopulate the dictionary in place, then runs the predicate against
the reused context.
- BuildReadCache removed.
Safety:
Reuse is serialised under _evalGate which guarantees no two threads
ever observe the same scratch in a half-refilled state. The
AlarmPredicateContext is bound to the scratch dictionary by reference,
so the predicate's ctx.GetTag(path) sees the freshly-refilled values
rather than a stale snapshot.
Verification:
- All 66 ScriptedAlarms tests pass (was 63 — three new regression tests
locking the reuse contract).
- All 56 VirtualTags tests still pass (unchanged).
- All 104 Core.Scripting tests still pass (unchanged).
New tests in ScriptedAlarmEngineTests:
- Reevaluation_reuses_the_same_read_cache_dictionary — asserts
ReferenceEquals(scratch_before, scratch_after) across two
evaluations of the same alarm.
- Reevaluation_reuses_the_same_predicate_context — same, for the
context.
- LoadAsync_drops_the_prior_generations_scratch — asserts a config
publish wipes the prior scratch (so a stale Logger / Inputs can't
leak into the new generation).
Internal test hooks TryGetScratchReadCacheForTest /
TryGetScratchContextForTest added via the existing
InternalsVisibleTo for the tests project. Kept internal — not part of
the public engine surface.
Docs:
- docs/v2/Galaxy.Performance.md "Scripted-alarm engine" section
rewritten as "hot-path allocation reuse" documenting the new
contract + reuse safety reasoning + the three regression tests.
- code-reviews/Core.ScriptedAlarms/findings.md -009 flipped
Won't Fix → Resolved.
- code-reviews/README.md regenerated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Joseph Doherty
|
99354bfaf2 |
fix(core-scripted-alarms): resolve Low code-review findings (Core.ScriptedAlarms-003,006,008,010,011; -009 documented)
- Core.ScriptedAlarms-003: emit OnEvent OUTSIDE _evalGate by collecting
pending emissions during the gate-held section and flushing them after
release; eliminates re-entrancy deadlock the docs already promised.
- Core.ScriptedAlarms-006: track every fire-and-forget Reevaluate /
ShelvingCheck task in _inFlight; Dispose drains the set so the engine
no longer races store writes against teardown.
- Core.ScriptedAlarms-008: store comments as ImmutableList<AlarmComment>
so AppendComment is O(log n) instead of O(n).
- Core.ScriptedAlarms-010: document the deliberate input-quality
asymmetry (Uncertain drives the predicate, renders {?} in the message)
in docs/ScriptedAlarms.md and on MessageTemplate.Resolve remarks.
- Core.ScriptedAlarms-011: propagate the no-op reason through
TransitionResult.NoOp(state, reason) and log it from
ScriptedAlarmEngine.ApplyAsync.
- Core.ScriptedAlarms-009 (Won't Fix per recommendation): documented the
per-evaluation dictionary allocation in docs/v2/Galaxy.Performance.md
with a mitigation path if a future soak surfaces pressure.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Joseph Doherty
|
edee47d77f |
PR 6.W — Galaxy.Performance.md
Documents the four perf surfaces shipped in Phase 6: - Tracing surface (PR 6.1) — table of every span the driver emits + rationale for stream-level (not per-event) coverage. - Metrics surface (PR 6.2) — three EventPump counters, tagging scheme, the bounded-channel design, and the received = dispatched + dropped + in-flight invariant. - Buffered update interval (PR 6.3) — how MxAccess.PublishingIntervalMs flows through both subscribe paths and what's still pending on the gw side (typed SetBufferedUpdateInterval helper). - Soak scenario (PR 6.4) — env-var-gated 24h × 50k validation with the CI-compressed override recipe. - Tuned defaults (PR 6.5) — table of every default with source + notes; rows marked "unchanged" carry the explicit "no live data argues for changing this" caveat. Closes with a "where to look first when something's slow" runbook section so on-call doesn't have to re-derive the trace+metric correlation map from primary docs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |