Review follow-ups to 34829ea0.
IMPORTANT - the leak filter had its own flake vector. This test saturates the
thread pool by design, and TimedScriptEvaluator starts its 5s clock at WaitAsync
rather than when the inner Task.Run gets a thread, so a legitimately-enqueued
evaluation can blow the budget while still queued and surface "script timed out
after 5.0s". The reason != exhaustion filter classified that as a raw disposed
escape and hard-failed on it - handing the test back the load-fragility the
reshape removed. Timeouts are now their own bucket, tolerated and rate-bounded
exactly as exhaustions are; a raw disposed escape still fails on first occurrence.
Doc-comment names this second residual vector alongside the exhaustion calibration.
MINOR (scope extension authorised by review) - "evaluator disposed during
evaluation" existed as three unshared literals. Promoted to
ScriptEvalFailureReasons alongside VirtualTagEvalResult in Commons.Engines (the
assembly all three already reference) and referenced from both evaluators and the
test. The timeout prefix joins it, since the test now classifies on it too. String
values are byte-identical - no behaviour change.
Falsified: with a 12ms script budget the race produced real timeouts (3/2695).
Classified as leaks (the pre-review filter) they hard-failed the test 6/6 items;
bucketed as starvation they sit 0.11% under the 10% bound and the test is green.
Reverses the conservative call from the previous commit at the user's direction,
and they were right: "surface it on a counter first" left the wire lying. An
MXAccess event can carry Good quality while its payload is a kind MxValueDecoder
has no mapping for, so the old behaviour published null-under-Good — which a
client cannot tell apart from a genuine null reading. A value the gateway DID
send would be consumed as a value it did not, with the only evidence in a log
nobody is tailing. Bad data is the honest wire representation of lost data.
BadTypeMismatch is an existing StatusCodeMap constant, deliberately reused rather
than written inline: StatusCodeParityTests finds status constants by reflecting
over Good/Uncertain/Bad-prefixed uint fields and checking them against the pinned
SDK, so an inline literal would be invisible to that guard (Gitea #497, where 16
wrong constants had accumulated across 6 drivers exactly this way).
Logging is now once per DISTINCT value kind rather than once per pump. The set is
bounded by the MxValue oneof enum so a broken producer still cannot flood, but a
second unrepresentable kind appearing later gets its own line — warn-once-per-pump
would have silently swallowed it, which is the same class of bug as the null this
commit fixes.
The test's fixture matters more than its assertion: the undecodable event carries
Quality = 192 (OPC DA Good), so a decoder that gave up silently would publish Good
and fail. Paired with a positive control — a decodable value at the same quality
must still publish Good — so it cannot be satisfied by a pump that marks
everything Bad. Falsified by reverting the status to Good: the test reds, and only
that test.
Verified: 331 Galaxy tests green, Core.Abstractions 298 (the parity guard),
solution builds 0 errors. Docs corrected in the two places that recorded the old
decision as deliberate.
Closes the two follow-ups left open by the MxGateway 0.2.0 bump. Reading the
gateway's own contract first changed what each of them is, and one of my earlier
claims was simply wrong -- both corrections are recorded below.
Replay gap
----------
A sentinel MxEvent carries family = UNSPECIFIED, so it fell through EventPump's
family switch into the default drop: we silently ignored a documented MUST ("you
missed events -- discard local state and re-snapshot"). It is now intercepted and
answered by re-issuing SubscribeBulk for every tracked subscription, since
MXAccess delivers an initial value per item on subscribe -- re-subscribing IS the
re-snapshot. It does not restart the event pump (the stream carrying the sentinel
is healthy, and the handler runs on that stream's own read loop, so disposing the
pump from there would deadlock on itself), it does not block the read loop, and
re-entry is gated so a burst collapses to one pass instead of a subscribe storm
per sentinel.
Handled on the READ loop, not in the dispatch loop, because everything past that
point is subject to newest-dropped channel pressure -- and "you missed events" is
the one signal whose loss is unrecoverable. Pinned by a test that fills the
channel and still expects the sentinel.
⚠️ This cannot fire in the shipped configuration, and I did not pretend otherwise
or paper over it by enabling resume. The gateway emits the sentinel only for a
stream opened with after_worker_sequence > 0; GatewayGalaxySubscriber passes 0,
which the gateway treats as "fresh stream, not a resume -- plain attach, no
replay, no sentinel". Enabling resume is a larger and genuinely delicate change:
the sequence is only meaningful within ONE gateway session, our reconnect
recreates the session, and replaying a remembered sequence against a fresh
session sets the server's live watermark ABOVE every event it is about to send --
silencing the stream instead of recovering it. That deserves its own change with
a live gate, which I cannot run from here.
Sparse arrays -- correcting the last commit message
---------------------------------------------------
I wrote that the bump "makes handling it possible" on the decode path. That is
wrong: MxSparseArray is documented write-only, the gateway expands it before
forwarding, and "the worker never receives or produces it". So the decode path
is unreachable by contract and the write path has no use for it either -- it is a
wire compaction for a WHOLE-array write, not a partial update (unmentioned
indices are RESET, deliberately, because the gateway cannot read-modify-write
without racing the provider). It is therefore the wrong primitive for an OPC UA
IndexRange write, which expects untouched elements preserved -- and WriteRequest
carries no IndexRange in the first place.
It is decoded anyway, as contract defence: the alternative if a future gateway
ever emits one is a silent null published under Good quality. Fill semantics
mirror the gateway's own SparseArrayExpander exactly (epoch for Time, empty
string, Integer widening to long[] when any element is int64) so a value that
round-trips through a write reads back as the same CLR array.
Also: the decoder is now total by contract, since it runs inside the dispatch
loop where an escaping exception kills dispatch for the life of the pump. A
malformed payload degrades instead of throwing.
A real defect found in the same code
------------------------------------
A stream the GATEWAY completes -- a closed or evicted session -- ends the RPC
without an error, so nothing threw, the supervisor was never notified, and the
pump exited silently. GetHealth() kept reporting Healthy, which also means the
#527 comms-loss watchdog could never trip, because it requires a non-Healthy
state. The driver went permanently deaf with no signal anywhere. This one is
live, unlike the two above. Now reported as a fault; the falsification control is
the existing CleanShutdown_DoesNotInvokeOnStreamFault, since our own disposal
also ends the stream and must NOT be reported.
Value kinds we cannot represent are now counted and warned once rather than
published as an indistinguishable null. The status is deliberately NOT downgraded
to Bad: nothing can reach that branch today, and changing what an OPC UA client
sees for a live shape is not a change to make without a gate. The counter exists
so the next person has evidence instead of a guess.
Verified: 330 Galaxy tests green (12 new), plus Galaxy.Browser 21, Runtime 558,
Core 351; full solution builds 0 errors with the warning census unchanged. Every
new test was falsified first -- disabling the gap interception reds 3 of 4 (the
throwing-callback test passes vacuously when the callback never runs, by
construction), disabling sparse decode plus the representability signal reds all
7 decoder tests, and both were restored and re-verified green.
Docs: the Galaxy.Performance invariant was stated as received = dispatched +
dropped + in-flight, which this breaks -- a sentinel is received but never enters
the channel -- so it now reads + replay_gaps. Galaxy.md gains the three
operational notes; DriverLifecycle.md records that "reaches a non-Healthy state"
is an obligation a driver can silently fail to meet, which is the general shape
of the clean-end bug.
Driver.Mqtt.Contracts-004/-005/-006/-007.
-004: retainSeed was the last behavioural key read leniently, so "retainSeed":"false"
— what a CSV import produces — silently became the default TRUE, the INVERSE of the
authored intent. Both mappers now read it through TryReadBoolStrict (the qos
absent/valid/present-but-invalid split) and DescribeRejection names it. The authoring
side needed no hand-mirrored rule: MqttTagConfigModel.Validate already consults this
same parse through TagConfigParseGate, so the save is blocked with the inspector's
wording — pinned by a new AdminUI parity theory pair.
-005: the seven [Range] attributes on MqttDriverOptions/MqttPlainOptions never ran —
the options are bound by JsonSerializer, and nothing in the four MQTT projects calls
Validator.TryValidateObject. Two of them CONTRADICTED the real authority (KeepAlive's
[Range(1,…)] forbade the zero Normalize admits as legal MQTT; MaxPayloadBytes' forbade
the non-positive value its own remarks document as falling back to 1 MiB), so wiring
them up would have broken deliberate behaviour. Deleted, with each property naming
where its bound actually lives. The one escape they papered over is closed: an
out-of-range defaultQos rode into the SUBSCRIBE verbatim, and MqttSubscriptionManager's
constructor now clamps it exactly as the AdminUI driver form already persists.
-006: the DateTime write arm cast a negative long to ulong, so 1969-12-31 published a
timestamp ~584 million years in the future; it is now refused as ValueNotRepresentable.
An Unspecified kind was handed to ToUniversalTime(), which reads it as LOCAL — the same
authored instant encoded differently on the two nodes of a redundant pair; it is now
read as UTC.
-007: Format/FormatState enforced only null/empty, and nothing at all on deviceId, so
the type was not round-trip-closed. They now apply the parser's IsValidSegment (plus
'/', which only the format side can see). MqttWriteEncoder's own guards stay — TryEncode
returns an error rather than throwing, and that contract does not change.
The two value-bearing fields were read asymmetrically and nothing said why.
whereValue went through ReadScalar (string/number/boolean — the design doc's own
wide-row example authors it as the number 7); keyValue went through the string-only
ReadString, so a numerically-authored key refused the tag outright. Nothing downstream
justified the split: both are BOUND as DbParameters and both are matched back as text —
SqlPollReader.IndexByKey stringifies the key cell precisely so an authored key can face
an INTEGER key column.
The refusal was also mis-explained. Inspect branched on ReadString(root, "keyValue") is
null, so a numerically-authored key drew "Sql KeyValue tag has no 'keyValue'" — telling
the operator the field is missing when it is right there in the blob, the #549 shape of
an authoring surface disagreeing with the runtime about WHY.
keyValue now uses ReadScalar; Inspect mirrors it with HasScalar and says what a usable
key is; SqlTagConfigModel loads it with the shared TagConfigJson.GetScalarText (which
also replaces SqlRowSelectorModel's private copy), so the editor is not left stricter
than the parser. An object/array/null key is still refused. docs/drivers/Sql.md updated.
RED first: 4 failing in Driver.Sql.Tests.
-003 is a documented Won't Fix: the mitigating control is the ConfigEditor page
policy on the only two surfaces that reach a browse session (/raw and /uns),
there is no HTTP endpoint for browse, and PageAuthorizationGuardTests pins both
pages to that exact policy. Recorded in docs/security.md under Control-Plane
Authorization and pointed at from SqlDriverBrowser's class summary.
Driver.Mqtt.Browser-002/-003/-004/-005.
-002: ObservePlainTopic's pre-pass judged segment LENGTH only, so a plain topic
segment carrying a newline or an ANSI escape became the picker's displayed label —
a rule ObserveSparkplugBirth already applied via IsUsableLabel. The control-character
half is now single-sourced (HasControlCharacter) and the plain marker text converges
with the Sparkplug one. An EMPTY topic level stays accepted on purpose: it is legal
MQTT and its terminal leaf is a real committable address.
-003: GetOrCreate read _nodeCount and incremented it later, so N observers could each
see cap-1 and all insert. The slot is now reserved with one Interlocked.Increment and
given back on both non-consuming paths.
-004: Project ordered a level by Segment alone, but the level is keyed by Path
precisely because two Sparkplug siblings CAN share a Segment — leaving the tie to
ConcurrentDictionary enumeration order, which .NET's randomised string hashing makes
differ per process. .ThenBy(Path) makes the sort total.
-005: no behaviour change — DefaultNodeCap's doc now states the real per-session
ceiling and records that the aggregate across sessions is bounded by the reaper's
idle TTL, not by any session cap.
ListSchemasSql has two consumers and the remarks only justified the divergence
from the catalog gate's viewpoint. SqlBrowseSession.RootAsync renders every row
as an expandable schema, so an Oracle browse offers every database user. The
narrower ALL_TABLES/ALL_VIEWS query is named, along with why it is not taken
here: it narrows the gate too, and that is a poll-path behaviour change only a
real Oracle server can gate.
Inspect parsed the TagConfig blob once itself (root kind + the present-but-discarded
changeTriggered/timerIntervalMs checks) and then called TryReadTagConfig(reference, out _),
which parsed the same string from scratch again for the scriptId rule. The field-by-field
read is now TryReadRoot(JsonElement), which both entry points share, so Inspect consults
the one authority over the root it already has. The public string-based TryReadTagConfig
signature other modules consume is unchanged.
Pure refactor: added CalculationTagDefinitionFactoryInspectTests (17 cases pinning the
exact wording, the rejection-vs-advisory split and their order, and the
never-stricter-than-the-runtime-read invariant). They pass unchanged against the
pre-refactor code — verified by stashing the production change.
Joseph Doherty