The batched ingest's append-failure catch had zero coverage, and two doc
statements overstated the loss: the Warning said the burst's values were
"dropped", and RecorderStatus.OutboxAppendFailures said "the value was
dropped, not recorded". The outbox contract is prefix-never-orphan, so a
partial-prefix failure leaves entries durably queued that drain normally.
- New test pins the branch: one 3-item burst whose group commit commits a
1-entry prefix then throws => OutboxAppendFailures == 1 (per burst, not
per value), TotalRecorded == 0, actor alive, and the prefix still reaches
the writer via the post-failure DrainTick (drainInterval 30s, so the
steady cadence cannot account for it). Falsified twice: counting per item
and dropping the drain nudge each turn it red.
- Second test pins the OperationCanceledException filter from its
observable side - a cancel that is NOT shutdown is metered, not swallowed.
Removing the 'when (_lifetime.IsCancellationRequested)' turns it red. The
shutdown side has no cheap seam (the counter dies with the actor).
- Docs corrected: OutboxAppendFailures counts FAILED BATCH APPENDS, one per
burst whatever its size (a granularity change from the per-value era), and
is not a count of values lost; TotalRecorded under-counts when a commit
fails after a partial prefix. Log message says "failed to append" and
names the prefix. No functional change beyond the message string.
Also folds in the DrainBatchSize default alignment behind 0298a0b9: the
recorder's two ctor defaults and its non-positive fallback still said 64
while ContinuousHistorizationOptions.DrainBatchSize now defaults to 1024.
No test pinned the old 64.
C1 — a non-rebuild apply reporting failedNodes > 0 no longer advances _lastApplied.
Pre-delta-materialisation the full passes re-ensured every node on every apply, which
silently repaired any prior swallowed sink fault. The delta path removed that accidental
self-healing: a failed node sat inside _lastApplied, never appeared in a diff set again,
and EnsureVariable early-returns on the ones that landed — so it was lost until an
unrelated change forced a rebuild. The advance also moved to AFTER the materialise
passes, so a throwing pass leaves _lastApplied at the previous composition too. A
rebuild still advances even with failures (the ratchet is its resync path).
I2 — a UnsReferenceDelta whose identity is unchanged (same NodeId, EquipmentId and
BackingRawPath) is now applied in place via UpdateTagAttributes(Uns) instead of
remove+recreate, so the mirror's MonitoredItems survive an operator historizing or
reshaping the raw tag behind them. Remove+recreate stays for identity-changing deltas
(re-point, effective-name edit).
I3 — RemoveVariableNode severs the inverse half of every outgoing Organizes edge, the
variable-node counterpart of UnwireAlarmNotifiers. Rebuilds used to sweep these up;
surgical applies made rebuilds rare, so they accumulated on the raw node for the life of
the process.
I5 — added UNS reference variables are announced (their equipment folder is the parent,
already the Uns realm). Raw adds still are not; the gap comment now says so honestly
rather than calling it neutral and pre-existing.
I4 + M1 — corrected two comments that no longer described the code: the "re-point →
rebuild" claim in ApplyPureRemove, and the "N lock acquisitions" rationale for the delta
passes (Ensure* early-return on a dictionary probe before the lock).
Adversarial review of 9f68f272 (classifier: non-alarm raw-tag deltas became
surgical) found stale UNS inheritance after a surgical raw edit. Editing a raw
tag's IsHistorized / HistorianTagname / IsArray / ArrayLength left every
referencing UNS node STALE: none of those fields were on UnsReferenceVariable,
so no coincident UnsReferenceDelta fired; the applier updated the raw node in
place; and EnsureVariable early-returns on an existing node. Consequences:
HistoryRead through the UNS NodeId served a different series than through the
raw NodeId (breaking the documented dual-registration invariant), a
de-historized tag kept serving history via UNS, and a scalar->array flip left
the UNS mirror scalar AND writable (bypassing the array read-only write gate).
UnsReferenceVariable now carries Backing{IsHistorized,HistorianTagname,IsArray,
ArrayLength}, copied from the backing tag's TagConfigIntent at compose time, so
record equality makes the reference Changed in the SAME plan. Both deltas are
non-rebuild kinds, so the existing machinery repairs the mirror surgically:
remove-previous in Apply, recreate in MaterialiseUnsReferences. That pass now
reads the four values from the RECORD (the single truth the planner diffed)
instead of a rawByPath back-lookup, preserving historian resolution exactly
(blank override => BackingRawPath, which is the backing tag's NodeId).
The artifact-decode seam needs no change: DeploymentArtifact.BuildRawAndUnsSubtrees
drives AddressSpaceComposer.Compose verbatim, which is what guarantees byte-parity.
The classifier needs no change either (a UnsReferenceDelta never forces Rebuild);
its comment now records why.
Also from the same review: a three-place-contract comment on RawTagPlan's
hand-written Equals naming its two downstream consumers, plus classifier pins for
the .Any(!eligible) quantifier, an ArrayLength-only delta, and an
alarm-content-only delta.
ConfigComposer gzips the serialised snapshot (CompressionLevel.Fastest)
immediately after SerializeToUtf8Bytes and BEFORE taking the SHA-256, so
the compressed bytes ARE the artifact everywhere: Deployment.ArtifactBlob,
the LocalDb cache chunks, the gRPC fetch stream, and the RevisionHash a
FetchAndCache node verifies against. The invariant the deploy path rests on
— RevisionHash == SHA-256(stored/wire bytes) — is preserved, which is what
leaves the fetcher, the cache and the serve path untouched: nothing between
the composer and DeploymentArtifact.Open ever looks inside the blob.
Read side: Open() sniffs the gzip magic 1F 8B (valid JSON can never start
with 0x1F, so no framing of our own is needed) and inflates through a size
guard — MaxDecompressedBytes, 512 MiB, internal + mutable so the guard can
be exercised without inflating half a gigabyte per test run. Pre-upgrade
uncompressed blobs take the identical path they always did.
Every decompression failure — corrupt stream, truncated stream, over-limit
inflation — is reported as JsonException, the same exception family the
malformed-JSON path already owns. So the lenient static wrappers still
degrade to empty results and the apply paths still fail the apply and keep
last-known-good (#485); no caller's decision changes because the reason the
bytes are unusable changed.
gzip output is not guaranteed byte-stable across .NET releases. Harmless
here: the hash is computed FROM the emitted bytes at compose time and never
re-derived from the JSON, so the worst an implementation change costs is
one spurious "changed" revision after a runtime upgrade. Snapshot
determinism is unaffected — compression wraps the final bytes and reorders
nothing.
Tests: gzip/uncompressed parity through both the instance parses and the
statics, corrupt-gzip leniency through the wrappers, the size guard (with a
falsifiability control proving the same bytes inflate fine under the
production limit), and composer-side blob-starts-with-1F8B +
RevisionHash == SHA256(blob) with a control proving the hash is NOT over
the JSON.
Task 4 (9f68f272) taught AddressSpaceChangeClassifier that a non-alarm
RawTagDelta and any UnsReferenceDelta are surgical-eligible, so such plans
started classifying AttributeOnly/PureAdd instead of Rebuild. The applier's
surgical branch did not know about either delta set, so those deploys fell
through as SILENT NO-OPS (or, on a non-surgical sink, applied nothing at all
where a rebuild was the safe answer). This closes that gap:
- Raw-tag loop: an eligible ChangedRawTags delta is applied IN PLACE via
ISurgicalAddressSpaceSink.UpdateTagAttributes in the RAW realm, computing
writable (`Writable && !IsArray`) and the historian tagname (blank override
⇒ the RawPath) exactly as MaterialiseRawSubtree does. A delta whose ONLY
difference is ReferencingEquipmentPaths is node-inert for a value tag and
is SKIPPED entirely — that is the shape a single UnsTagReference add/drop
produces, and it used to rebuild the world.
- UNS-reference loop: a ChangedUnsReferenceVariables delta removes the
PREVIOUS node in place (terminal Bad, then RemoveVariableNode, Uns realm).
Remove+recreate rather than an in-place update because a re-point must move
the Organizes edge (the sink has no RemoveReference) and a display-name
edit changes the NodeId itself; the cost is that one node's monitored
items, versus every subscription server-wide today.
- The branch trigger now fires for both sets, so a non-surgical sink falls
back to a full rebuild instead of applying nothing.
Every new operation is inside the existing one-way rebuild ratchet: any false
or throw ⇒ SafeRebuild(). The eligibility filter is defense-in-depth — when
kind != Rebuild the classifier already guarantees every delta is eligible.
The recreate of a changed UNS reference is NOT done here. Until Task 6 the
publish actor still materialises the FULL composition after Apply returns, so
the existing MaterialiseUnsReferences pass recreates it and the system is
correct end to end after this commit; the remove-then-recreate ordering holds
because Apply returns before the materialise passes run. Task 6 narrows those
passes to the plan's delta and keeps the same ordering.
Tests: 7 new applier tests (in-place raw update incl. array/override parity,
reference-only skip, UNS-ref remove, both ratchet fallbacks, non-surgical-sink
rebuild). RecordingSink now captures the realm on surgical / remove / value-write
calls, so the existing UNS-realm assertions are explicit too.
Every public entry point on DeploymentArtifact took ReadOnlySpan<byte> and did
JsonDocument.Parse(blob.ToArray()) — a full copy plus a full document build per
call — so one apply paid ~8-11 full parses of the same bytes (the scoped
ParseComposition alone parsed three times: scope, composition, BuildClusterSets).
DeploymentArtifact.Open(byte[]) now parses once into ParsedDeploymentArtifact,
which owns the JsonDocument and exposes an instance method mirroring each static
entry point. The moved bodies operate on Root; BuildClusterSets takes JsonElement
instead of re-parsing the blob; ResolveClusterScope is memoized per node id, since
every consumer of one apply asks about the same node.
Every static is now a thin wrapper (Open + delegate + dispose) keeping its
ReadOnlySpan<byte> signature and its lenient empty/malformed contract — the
wrapper pays the one copy and owns the JsonException catch, so output shapes are
unchanged. Hot callers migrate to Open() separately.
ParsedDeploymentArtifactTests pins same-bytes-same-output through both doors for
all six entry points over a rich multi-cluster artifact (raw chain + UNS refs +
scripts + keys + ACLs), with falsifiability controls proving the fixture actually
scopes, plus memoization and after-dispose-use coverage.
Accumulated fast-follows from the Task 1-11 reviews. No behaviour change on the
value path.
- Strengthen OpcUaPublishActorBatchTests.Sink_throw_is_swallowed_and_logged. The
survival assertion alone did NOT prove the catch exists: Akka's default
supervision RESTARTS a throwing actor and the restart preserves the mailbox, so
the follow-up message landed either way and the test passed against a build with
no catch at all. It now asserts via EventFilter that the exception NEVER reaches
supervision (Expect(0) on the InvalidOperationException) and that the handler's
own Warning IS logged, keeping the survival assertion. Everything that must have
happened goes inside the Expect(0) action — it counts matches at the end of the
action, so an assertion left outside would race the actor and pass vacuously.
Verified falsifiable: deleting the catch turns the test red.
- MapKey(NodeId): the non-string identifier branch collapsed to a string.Empty key
silently. Such a key cannot match anything the manager stored, so the caller sees
an unexplained miss rather than an error. It now logs a Utils.LogWarning naming
the seam. Every current call site is provably string-only, so this should never
fire.
- The two WireAlarmNotifiers Utils.LogInfo sites interpolated the RealmKey struct,
rendering a raw struct dump. Interpolate .Id and .Realm as separate placeholders
so log-grep patterns stay consistent with the file's other sites.
- Stale comments: the _eventNotifierSources key is a RealmKey now, not a
namespace-qualified NodeId string; and three comments still named
DriverHostActor.ForwardToMux, which Task 8 replaced with ForwardBatchToMux.
DependencyMuxActor now receives DriverInstanceActor.AttributeValueBatchPublished — the
primary value ingress since DriverHostActor's batched fan-out — and filters it per item,
emitting one DependencyValueChanged per genuinely interested (item, subscriber) pair with
that item's own value/timestamp/status sub-code (#519).
The per-item AttributeValuePublished receive stays: DriverHostActor's comms-loss fan
fabricates it, and per-tag drivers still take that path. Indexed for-loop over Items to
avoid an interface-enumerator allocation per burst.
Replaces the temporary per-item unwrap bridge with a real batched handler: one
AttributeValueBatchPublished burst now costs ONE hop to the dependency mux and ONE
OpcUaPublishActor.AttributeValueBatchUpdate, instead of one message per value.
Routing per item is unchanged — the v3 Batch 4 single-source fan-out (raw NodeId +
every referencing UNS NodeId, identical value/status/timestamp), with an unmapped ref
Debug-logged and dropped without disturbing the rest of the burst. A burst in which
nothing resolves sends no publish message at all rather than an empty batch.
Indexed for-loop over Items is deliberate: no interface-enumerator allocation per
burst on the hottest routing seam. ForwardToMux and its registrations stay for the
comms-loss fan and per-tag drivers.
The /admin/secrets link was already in the side rail, but ungated — every
authenticated user saw it, and a Designer/Operator clicking it was bounced by
the page's own [Authorize(Policy = SecretsAuthorization.ManagePolicy)]. Hide
the advertised dead end.
Gated on that same policy constant rather than Roles="Administrator". The two
are equivalent today (the RCL's AdminRole is "Administrator", read off the same
ClaimTypes.Role claim FleetAdmin uses), so this is not about reach — it is that
the rail cannot drift if the RCL ever changes its policy.
Live-gated on docker-dev, both directions, because the positive half alone
proves nothing here: the link rendered for an admin BEFORE this change too, so
"admin still sees it" is satisfied equally by a working gate and by an inert
one. Negative control — gate swapped to Roles="NoSuchRoleFalsifyProbe", rebuilt,
item went to 0 while /role-grants and /apikeys stayed at 1 (the item is hidden,
not the rail broken). Restored build hashed to the same image ID as the
pre-falsification one, proving the restore was byte-exact. This also rules out
AuthorizeView being inert inside ThemeShell's <Nav> slot under this host's
static router, which would have silently hidden the link from everyone.
SecretsNavLinkTests covers it in the AdminUI's actual idiom — source scan +
metadata cross-check, NOT bUnit (the bunit PackageVersion pin has zero
PackageReferences repo-wide). The load-bearing assertion is index ordering
(AuthorizeView open < NavRailItem < close): a scan that merely finds both tags
in the file passes when they are not nested, which is the ungated state.
Scope: Secrets only. /role-grants and /apikeys are FleetAdmin (Administrator-
only) too and stay visible to everyone, so the rail is now inconsistent —
deliberate, as hiding those is a UX call for the whole rail rather than
something to smuggle into this change. Noted in the markup.
Family-wide task from the scadaproj index session (ScadaBridge ee193cd2,
ignitionoee 3627bb8, HistorianGateway 58b6418, mxgw 87d575d).
Gitea #561 ("production writes a 0-byte log") is closed as not-a-defect. The
production log on wonder-app-vd03 listed as 0 bytes with LastWriteTime ==
CreationTime; opening it showed 4,658,893 bytes of correct output, written
continuously since service start.
On Windows the directory entry's size and last-write time are not refreshed while
the writing process holds the file open, so Get-ChildItem, dir and FileInfo.Length
all report size-at-open for the life of the handle. A live, healthy log is
indistinguishable from a dead sink by metadata alone — which is what made the
false reading survive six weeks of investigation.
That same trap was in the 29a202f5 instrumentation: CheckFileSinkGrowth measured
with FileInfo.Length, so it would have reported "self-check FAILED: the sink did
not grow" at every boot on a perfectly healthy Windows service — and a
self-check's false positive reads as confirmation of the very bug it was added to
find. MeasureFamilyBytes now reads Length through an opened FileStream:
FileShare.ReadWrite to open it at all while the sink holds its write handle, and
FileShare.Delete so the probe never blocks a roll or retention delete.
Both instruments are kept. A sink that cannot write its file is still a real and
otherwise-invisible failure — a sibling repo reproduced it via handle contention,
where RollingFileSink logs the lock-out to SelfLog and falls through to _001. Only
the two doc claims that are now false are corrected: the class summary's
"production ran that way for six weeks" and the warning text's "#561 failure mode".
Tests: A_working_file_sink_passes already measured while Serilog held the file
open — precisely the broken case — and passed throughout, because it only ever ran
on macOS, where the length is honest. That platform blindness is now recorded in
the class summary, alongside two tests pinning the sharing contract the fix
depends on.
Four-minor jump 0.2.3 → 0.6.2. The release line exists because a deploy
renaming the app directory took the MxGateway credential store with it
(2026-08-09, two silent days): Secrets:SqlitePath must now be absolute AND
outside the content root, rejected at boot rather than relocated. 0.6.2 over
0.6.1 is message-only (the rejection text now teaches the mounted-volume
case and forecloses the override hunt) — the live rig evidence below was
gathered at 0.6.1 and the validator behavior is identical.
Adoption changes beyond the pins:
- appsettings.json drops the relative "otopcua-secrets.db" (now boot-fatal);
dev gets the library's absolute per-user default, deployments supply their
own (wonder already pins an absolute out-of-root path in the service env).
- Program.cs's pre-host ${secret:} expander now calls the FOUR-argument
AddZbSecrets with builder.Environment.ContentRootPath. This call site is
the actual fix, not the pin: the throwaway container has no
IHostEnvironment, and without the explicit root the under-content-root
rule is skipped at exactly the moment the migrator creates the store file
(the 0.6.0 defect). Do NOT simplify back to the 3-argument overload — it
compiles and silently reopens the gap. Pinned by a Program.cs source-scan
test AND a runtime falsification control that RUNS the migration and
demonstrably creates the store at the rejected path under the 3-arg form.
- docker-dev: Secrets__SqlitePath moves /app/data → /data via a SECOND mount
of the same per-node volume (same file at the volume root, so seeded
stores carry over) because /app is the container's content root and the
new rule correctly rejects it. LocalDb stays on /app/data (secrets-only
rule).
- SecretsStorePathValidationTests: positive-control probes, artifact-first
(relative and in-content-root paths must FAIL leaving no file; outside-
root passes; the no-explicit-root control proves the file assertions can
fail) — a clean boot is equally consistent with the validator never being
wired.
Live-verified on the six-node docker-dev rig (at 0.6.1): all nodes boot
under the new validation with stores at /data; an AdminUI-written secret
converged into BOTH central nodes' independent stores; site-a held only its
own pair's secrets (Phase 6 pair-local isolation intact). Replicator wire
behavior unchanged 0.3.0–0.6.x per CHANGELOG; registration-graph pins and
the full suite pass.
Production ran six weeks with a healthy process and a 0-byte log because
Serilog isolates per-sink failures: a file sink whose every write fails is
invisible from anywhere but SelfLog, which was not enabled. Two instruments:
- LogPipelineDiagnostics.EnableSelfLog (before the first logger builds):
SelfLog to logs/serilog-selflog.txt (rotated at 5 MB) + stderr — the only
channel that carries the exception the sink swallows.
- CheckFileSinkGrowth (after the startup banner): verifies each configured,
unbuffered file sink's rolled-file family physically grew across a marker
event; a silent sink is reported at boot via SelfLog/stderr and the
console sink instead of discovered months later. Growth-across-marker,
not existence — a previous run's bytes must not mask a silent sink (the
false pass an existence check would have given on wonder-app-vd03).
Never throws: both instruments swallow their own failures — a diagnostics
channel that can fail startup is worse than the defect it watches for.
The load-bearing test is the failure case (a sink that writes nothing IS
reported); a check that can only pass is indistinguishable from no check.
The page ran members, the driver-instances heading, and every cluster group
together inside one panel; driver name and id slammed inline; one long driver
exception blew the table out sideways (Controls scrolled out of reach); and the
inline #546 expander row dumped a full-width panel into the table.
- One panel per cluster group, node chips in the panel head.
- Driver id renders on its own muted mono line under the name.
- Last error abbreviated to 60 chars (DisplayText.Abbreviate — #504 rule),
full text on the tooltip and in the Controls dialog.
- DriverStatusPanel moved from an inline expander row to a modal (one at a
time); the dialog re-resolves its row from the current groups each render so
the 5 s rebuild can't strand it, and the Enabled-in-key recycle discipline
carries over unchanged. HostsPanelWiringTests re-pinned to the dialog markup.
Live-verified on docker-dev: separated cards, in-viewport table, dialog opens
with live breaker/status data and grouped Reconnect|Restart, closes cleanly.
Driver.Mqtt.Contracts-004/-005/-006/-007.
-004: retainSeed was the last behavioural key read leniently, so "retainSeed":"false"
— what a CSV import produces — silently became the default TRUE, the INVERSE of the
authored intent. Both mappers now read it through TryReadBoolStrict (the qos
absent/valid/present-but-invalid split) and DescribeRejection names it. The authoring
side needed no hand-mirrored rule: MqttTagConfigModel.Validate already consults this
same parse through TagConfigParseGate, so the save is blocked with the inspector's
wording — pinned by a new AdminUI parity theory pair.
-005: the seven [Range] attributes on MqttDriverOptions/MqttPlainOptions never ran —
the options are bound by JsonSerializer, and nothing in the four MQTT projects calls
Validator.TryValidateObject. Two of them CONTRADICTED the real authority (KeepAlive's
[Range(1,…)] forbade the zero Normalize admits as legal MQTT; MaxPayloadBytes' forbade
the non-positive value its own remarks document as falling back to 1 MiB), so wiring
them up would have broken deliberate behaviour. Deleted, with each property naming
where its bound actually lives. The one escape they papered over is closed: an
out-of-range defaultQos rode into the SUBSCRIBE verbatim, and MqttSubscriptionManager's
constructor now clamps it exactly as the AdminUI driver form already persists.
-006: the DateTime write arm cast a negative long to ulong, so 1969-12-31 published a
timestamp ~584 million years in the future; it is now refused as ValueNotRepresentable.
An Unspecified kind was handed to ToUniversalTime(), which reads it as LOCAL — the same
authored instant encoded differently on the two nodes of a redundant pair; it is now
read as UTC.
-007: Format/FormatState enforced only null/empty, and nothing at all on deviceId, so
the type was not round-trip-closed. They now apply the parser's IsValidSegment (plus
'/', which only the format side can see). MqttWriteEncoder's own guards stay — TryEncode
returns an error rather than throwing, and that contract does not change.
Joseph Doherty