- docs/v2/Galaxy.Performance.md gains a "Value-path batching (2026-08-15)"
section: the batch seams (IBatchSubscribable / PollGroupEngine onBatch ->
AttributeValueBatchPublished -> AttributeValueBatchUpdate -> sink WriteValues
with 512-chunked single-lock batches in the node manager), the invariants (#519
status codes verbatim, per-ITEM metering, Modbus deadband honored via the
driver-filtered buffer, the ValueWriteBatchException partial-failure contract),
the RealmKey re-key, and the pinned dispatcher's throughput 1 -> 100.
- The plan doc gains a "Follow-ups (recorded, not implemented)" section: the
PollGroupEngine unconditional-collection trap (a future per-tag filter in
S7/AbCip/FOCAS would silently reintroduce the deadband-bypass bug), a two-level
driverId -> (rawPath -> refs) routing map, null/empty guard consistency on the
internal batch messages, a possible DriverHostActorLiveValueTests merge, and the
pre-existing aborted-cycle tail-loss on the three unbuffered drivers.
- tasks.json: all 13 tasks completed.
CLAUDE.md needed no edit: its only statement in this area ("One driver publish for
a RawPath fans (in DriverHostActor) to the raw NodeId AND every referencing UNS
NodeId") is still true — batching changed the message granularity, not the
fan-out semantics.
Verification: dotnet build ZB.MOM.WW.OtOpcUa.slnx clean (0 errors, no new warning
classes, zero warnings in any production file). 9 unit projects green, 3470
passed / 5 skipped / 0 failed: Core.Abstractions 305, Core 351+1s, Commons 348,
Runtime 575+3s, OpcUaServer 424+1s, Modbus 399, S7 302, AbCip 410, FOCAS 356.
No DB-backed suite was run — tests/Server/ZB.MOM.WW.OtOpcUa.Server.Tests does not
exist, and nothing this branch touches is covered by the SQL-backed projects.
Accumulated fast-follows from the Task 1-11 reviews. No behaviour change on the
value path.
- Strengthen OpcUaPublishActorBatchTests.Sink_throw_is_swallowed_and_logged. The
survival assertion alone did NOT prove the catch exists: Akka's default
supervision RESTARTS a throwing actor and the restart preserves the mailbox, so
the follow-up message landed either way and the test passed against a build with
no catch at all. It now asserts via EventFilter that the exception NEVER reaches
supervision (Expect(0) on the InvalidOperationException) and that the handler's
own Warning IS logged, keeping the survival assertion. Everything that must have
happened goes inside the Expect(0) action — it counts matches at the end of the
action, so an assertion left outside would race the actor and pass vacuously.
Verified falsifiable: deleting the catch turns the test red.
- MapKey(NodeId): the non-string identifier branch collapsed to a string.Empty key
silently. Such a key cannot match anything the manager stored, so the caller sees
an unexplained miss rather than an error. It now logs a Utils.LogWarning naming
the seam. Every current call site is provably string-only, so this should never
fire.
- The two WireAlarmNotifiers Utils.LogInfo sites interpolated the RealmKey struct,
rendering a raw struct dump. Interpolate .Id and .Realm as separate placeholders
so log-grep patterns stay consistent with the file's other sites.
- Stale comments: the _eventNotifierSources key is a RealmKey now, not a
namespace-qualified NodeId string; and three comments still named
DriverHostActor.ForwardToMux, which Task 8 replaced with ForwardBatchToMux.
DependencyMuxActor now receives DriverInstanceActor.AttributeValueBatchPublished — the
primary value ingress since DriverHostActor's batched fan-out — and filters it per item,
emitting one DependencyValueChanged per genuinely interested (item, subscriber) pair with
that item's own value/timestamp/status sub-code (#519).
The per-item AttributeValuePublished receive stays: DriverHostActor's comms-loss fan
fabricates it, and per-tag drivers still take that path. Indexed for-loop over Items to
avoid an interface-enumerator allocation per burst.
Replaces the temporary per-item unwrap bridge with a real batched handler: one
AttributeValueBatchPublished burst now costs ONE hop to the dependency mux and ONE
OpcUaPublishActor.AttributeValueBatchUpdate, instead of one message per value.
Routing per item is unchanged — the v3 Batch 4 single-source fan-out (raw NodeId +
every referencing UNS NodeId, identical value/status/timestamp), with an unmapped ref
Debug-logged and dropped without disturbing the rest of the burst. A burst in which
nothing resolves sends no publish message at all rather than an empty batch.
Indexed for-loop over Items is deliberate: no interface-enumerator allocation per
burst on the hottest routing seam. ForwardToMux and its registrations stay for the
comms-loss fan and per-tag drivers.
Closes the pin skew against ScadaBridge (which already pins 0.2.1). Pure pin
bump — no code change was required:
- `RegisterReplicated(string)` still delegates to `baselineExistingRows: false`,
so `LocalDbSetup.OnReady` is byte-identical in behaviour.
- Everything 0.2.x added is additive (`DeregisterReplicated`, `MaxBatchBytes`,
the `baselineExistingRows` overload); none of it is called here.
- The wire protocol is unchanged (0.1.x peers still sync) and the on-disk
bookkeeping schema v1 -> v2 upgrades in place on open, so no rig, volume or
operator action is needed.
The one thing that DID need reconciling is documentation. 0.2.0 made batching
byte-budgeted: `MaxBatchBytes` (2 MB default, under the 4 MB gRPC cap) now
bounds the wire message, and `MaxBatchSize` is demoted to a secondary row cap
plus the DB read-page size. The rig's `MaxBatchSize = 16` was chosen as a
hand-computed byte proxy ("row-count-only against gRPC's 4 MB cap"), and that
rationale is now wrong everywhere it appeared — appsettings, Configuration.md,
the pair-replication runbook, and both docker-dev site-a services.
The VALUE stays 16, for a different and still-real reason: OplogStore /
SnapshotStreamer materialise a whole read page in memory *before* the
byte-budget split runs, so 16 x ~171 KB artifact chunk rows bounds that
transient allocation at ~2.7 MB. Documented as such, with `MaxBatchBytes`
added to both config tables.
CLAUDE.md records the pin and two deliberate non-adoptions: this repo does not
take ScadaBridge's conditional-registration + boot-time `DeregisterReplicated`
self-heal (OnReady registers all four tables unconditionally, and
LocalDbSetupTests pins that set), and consequently does not use
`baselineExistingRows` either — so enabling replication on a long-running
default-OFF node converges on subsequent writes, not on rows already in the
file. Both are behaviour changes, out of scope for a pin bump.
Verification: solution builds clean; full suite run (57 projects). All 66
LocalDb tests in Host.IntegrationTests green. One unrelated failure,
RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_
fails_with_disposed — a Roslyn evaluator disposal race with no LocalDb in its
path, whose own comment documents that its "at most one clear per evaluation"
assumption depends on thread-pool pressure. It passed 3/3 re-runs in isolation
and fails only under full-solution parallel load; pre-existing test-design
fragility, not a regression from this bump.
The /admin/secrets link was already in the side rail, but ungated — every
authenticated user saw it, and a Designer/Operator clicking it was bounced by
the page's own [Authorize(Policy = SecretsAuthorization.ManagePolicy)]. Hide
the advertised dead end.
Gated on that same policy constant rather than Roles="Administrator". The two
are equivalent today (the RCL's AdminRole is "Administrator", read off the same
ClaimTypes.Role claim FleetAdmin uses), so this is not about reach — it is that
the rail cannot drift if the RCL ever changes its policy.
Live-gated on docker-dev, both directions, because the positive half alone
proves nothing here: the link rendered for an admin BEFORE this change too, so
"admin still sees it" is satisfied equally by a working gate and by an inert
one. Negative control — gate swapped to Roles="NoSuchRoleFalsifyProbe", rebuilt,
item went to 0 while /role-grants and /apikeys stayed at 1 (the item is hidden,
not the rail broken). Restored build hashed to the same image ID as the
pre-falsification one, proving the restore was byte-exact. This also rules out
AuthorizeView being inert inside ThemeShell's <Nav> slot under this host's
static router, which would have silently hidden the link from everyone.
SecretsNavLinkTests covers it in the AdminUI's actual idiom — source scan +
metadata cross-check, NOT bUnit (the bunit PackageVersion pin has zero
PackageReferences repo-wide). The load-bearing assertion is index ordering
(AuthorizeView open < NavRailItem < close): a scan that merely finds both tags
in the file passes when they are not nested, which is the ungated state.
Scope: Secrets only. /role-grants and /apikeys are FleetAdmin (Administrator-
only) too and stay visible to everyone, so the rail is now inconsistent —
deliberate, as hiding those is a UX call for the whole rail rather than
something to smuggle into this change. Noted in the markup.
Family-wide task from the scadaproj index session (ScadaBridge ee193cd2,
ignitionoee 3627bb8, HistorianGateway 58b6418, mxgw 87d575d).
Reverses the conservative call from the previous commit at the user's direction,
and they were right: "surface it on a counter first" left the wire lying. An
MXAccess event can carry Good quality while its payload is a kind MxValueDecoder
has no mapping for, so the old behaviour published null-under-Good — which a
client cannot tell apart from a genuine null reading. A value the gateway DID
send would be consumed as a value it did not, with the only evidence in a log
nobody is tailing. Bad data is the honest wire representation of lost data.
BadTypeMismatch is an existing StatusCodeMap constant, deliberately reused rather
than written inline: StatusCodeParityTests finds status constants by reflecting
over Good/Uncertain/Bad-prefixed uint fields and checking them against the pinned
SDK, so an inline literal would be invisible to that guard (Gitea #497, where 16
wrong constants had accumulated across 6 drivers exactly this way).
Logging is now once per DISTINCT value kind rather than once per pump. The set is
bounded by the MxValue oneof enum so a broken producer still cannot flood, but a
second unrepresentable kind appearing later gets its own line — warn-once-per-pump
would have silently swallowed it, which is the same class of bug as the null this
commit fixes.
The test's fixture matters more than its assertion: the undecodable event carries
Quality = 192 (OPC DA Good), so a decoder that gave up silently would publish Good
and fail. Paired with a positive control — a decodable value at the same quality
must still publish Good — so it cannot be satisfied by a pump that marks
everything Bad. Falsified by reverting the status to Good: the test reds, and only
that test.
Verified: 331 Galaxy tests green, Core.Abstractions 298 (the parity guard),
solution builds 0 errors. Docs corrected in the two places that recorded the old
decision as deliberate.
Closes the two follow-ups left open by the MxGateway 0.2.0 bump. Reading the
gateway's own contract first changed what each of them is, and one of my earlier
claims was simply wrong -- both corrections are recorded below.
Replay gap
----------
A sentinel MxEvent carries family = UNSPECIFIED, so it fell through EventPump's
family switch into the default drop: we silently ignored a documented MUST ("you
missed events -- discard local state and re-snapshot"). It is now intercepted and
answered by re-issuing SubscribeBulk for every tracked subscription, since
MXAccess delivers an initial value per item on subscribe -- re-subscribing IS the
re-snapshot. It does not restart the event pump (the stream carrying the sentinel
is healthy, and the handler runs on that stream's own read loop, so disposing the
pump from there would deadlock on itself), it does not block the read loop, and
re-entry is gated so a burst collapses to one pass instead of a subscribe storm
per sentinel.
Handled on the READ loop, not in the dispatch loop, because everything past that
point is subject to newest-dropped channel pressure -- and "you missed events" is
the one signal whose loss is unrecoverable. Pinned by a test that fills the
channel and still expects the sentinel.
⚠️ This cannot fire in the shipped configuration, and I did not pretend otherwise
or paper over it by enabling resume. The gateway emits the sentinel only for a
stream opened with after_worker_sequence > 0; GatewayGalaxySubscriber passes 0,
which the gateway treats as "fresh stream, not a resume -- plain attach, no
replay, no sentinel". Enabling resume is a larger and genuinely delicate change:
the sequence is only meaningful within ONE gateway session, our reconnect
recreates the session, and replaying a remembered sequence against a fresh
session sets the server's live watermark ABOVE every event it is about to send --
silencing the stream instead of recovering it. That deserves its own change with
a live gate, which I cannot run from here.
Sparse arrays -- correcting the last commit message
---------------------------------------------------
I wrote that the bump "makes handling it possible" on the decode path. That is
wrong: MxSparseArray is documented write-only, the gateway expands it before
forwarding, and "the worker never receives or produces it". So the decode path
is unreachable by contract and the write path has no use for it either -- it is a
wire compaction for a WHOLE-array write, not a partial update (unmentioned
indices are RESET, deliberately, because the gateway cannot read-modify-write
without racing the provider). It is therefore the wrong primitive for an OPC UA
IndexRange write, which expects untouched elements preserved -- and WriteRequest
carries no IndexRange in the first place.
It is decoded anyway, as contract defence: the alternative if a future gateway
ever emits one is a silent null published under Good quality. Fill semantics
mirror the gateway's own SparseArrayExpander exactly (epoch for Time, empty
string, Integer widening to long[] when any element is int64) so a value that
round-trips through a write reads back as the same CLR array.
Also: the decoder is now total by contract, since it runs inside the dispatch
loop where an escaping exception kills dispatch for the life of the pump. A
malformed payload degrades instead of throwing.
A real defect found in the same code
------------------------------------
A stream the GATEWAY completes -- a closed or evicted session -- ends the RPC
without an error, so nothing threw, the supervisor was never notified, and the
pump exited silently. GetHealth() kept reporting Healthy, which also means the
#527 comms-loss watchdog could never trip, because it requires a non-Healthy
state. The driver went permanently deaf with no signal anywhere. This one is
live, unlike the two above. Now reported as a fault; the falsification control is
the existing CleanShutdown_DoesNotInvokeOnStreamFault, since our own disposal
also ends the stream and must NOT be reported.
Value kinds we cannot represent are now counted and warned once rather than
published as an indistinguishable null. The status is deliberately NOT downgraded
to Bad: nothing can reach that branch today, and changing what an OPC UA client
sees for a live shape is not a change to make without a gate. The counter exists
so the next person has evidence instead of a guess.
Verified: 330 Galaxy tests green (12 new), plus Galaxy.Browser 21, Runtime 558,
Core 351; full solution builds 0 errors with the warning census unchanged. Every
new test was falsified first -- disabling the gap interception reds 3 of 4 (the
throwing-callback test passes vacuously when the callback never runs, by
construction), disabling sparse decode plus the representability signal reds all
7 decoder tests, and both were restored and re-verified green.
Docs: the Galaxy.Performance invariant was stated as received = dispatched +
dropped + in-flight, which this breaks -- a sentinel is received but never enters
the channel -- so it now reads + replay_gaps. Galaxy.md gains the three
operational notes; DriverLifecycle.md records that "reaches a non-Healthy state"
is an obligation a driver can silently fail to meet, which is the general shape
of the clean-end bug.
Audited every ZB.MOM.WW.* pin against what is actually published. Fifteen of
seventeen were already current; two were behind, for different reasons:
- Auth 0.1.5 -> 0.2.0 was published on the Gitea feed and simply not adopted.
- MxGateway 0.1.1 -> 0.2.0 was never adopted because these two packages resolve
from the checked-in ./nuget-packages folder feed, not from Gitea, so a bump
needs the .nupkg copied in from ~/Desktop/MxAccessGateway/dist/. Nothing
reports the folder as stale; it just quietly keeps serving the old version.
0.1.2 (MxSparseArray) was skipped over entirely for the same reason.
Both are purely additive at the API surface (diffed the shipped XML docs member
by member: zero removals; Contracts +19, Client +45).
Auth 0.2.0 adds LDAP endpoint failover -- an ordered FallbackServers list with
sticky endpoint preference. Its one new validation rule rejects a malformed
FallbackServers entry at boot, which cannot bite us: we never set the key and it
defaults to empty. Net behaviour change here is zero; the failover is available
if AdminUI login ever wants a backup DC.
MxGateway 0.2.0 adds sparse-array values, replay-gap stream items, secret
redaction helpers and a batch of session RPCs. Two notes on the value path:
- MxValueDecoder's `_ => null` fallback already absorbed an unknown MxValue kind,
so a sparse array degrades to null on the old and new contracts alike -- the
bump does not change what we decode, it only makes handling it possible. Same
for MxEvent.ReplayGap, which EventPump ignores; we stream with
afterWorkerSequence: 0 and re-prime from the subscription on reconnect rather
than relying on replay. Both are open follow-ups, not regressions.
- Staying on 0.1.1 was never protection: the gateway decides what goes on the
wire, and prod already runs a server past this contract.
We pin no gateway contract version anywhere, so there is no handshake to update.
Verified: restore clean, build 0 errors, and the warning census is unchanged
(the 20 CS0618 obsoletes are all OPC UA SDK calls, none from these packages).
Every project that references either package is green -- Galaxy 317, Galaxy
Browser 21, Security 89, Host 106, AdminUI 1499 = 2032 tests, 0 failures.
Not updating the scadaproj index: its OtOpcUa entry names the shared libraries
but records no version pins, and which libraries we consume has not changed.
Gitea #561 ("production writes a 0-byte log") is closed as not-a-defect. The
production log on wonder-app-vd03 listed as 0 bytes with LastWriteTime ==
CreationTime; opening it showed 4,658,893 bytes of correct output, written
continuously since service start.
On Windows the directory entry's size and last-write time are not refreshed while
the writing process holds the file open, so Get-ChildItem, dir and FileInfo.Length
all report size-at-open for the life of the handle. A live, healthy log is
indistinguishable from a dead sink by metadata alone — which is what made the
false reading survive six weeks of investigation.
That same trap was in the 29a202f5 instrumentation: CheckFileSinkGrowth measured
with FileInfo.Length, so it would have reported "self-check FAILED: the sink did
not grow" at every boot on a perfectly healthy Windows service — and a
self-check's false positive reads as confirmation of the very bug it was added to
find. MeasureFamilyBytes now reads Length through an opened FileStream:
FileShare.ReadWrite to open it at all while the sink holds its write handle, and
FileShare.Delete so the probe never blocks a roll or retention delete.
Both instruments are kept. A sink that cannot write its file is still a real and
otherwise-invisible failure — a sibling repo reproduced it via handle contention,
where RollingFileSink logs the lock-out to SelfLog and falls through to _001. Only
the two doc claims that are now false are corrected: the class summary's
"production ran that way for six weeks" and the warning text's "#561 failure mode".
Tests: A_working_file_sink_passes already measured while Serilog held the file
open — precisely the broken case — and passed throughout, because it only ever ran
on macOS, where the length is honest. That platform blindness is now recorded in
the class summary, alongside two tests pinning the sharing contract the fix
depends on.
Four-minor jump 0.2.3 → 0.6.2. The release line exists because a deploy
renaming the app directory took the MxGateway credential store with it
(2026-08-09, two silent days): Secrets:SqlitePath must now be absolute AND
outside the content root, rejected at boot rather than relocated. 0.6.2 over
0.6.1 is message-only (the rejection text now teaches the mounted-volume
case and forecloses the override hunt) — the live rig evidence below was
gathered at 0.6.1 and the validator behavior is identical.
Adoption changes beyond the pins:
- appsettings.json drops the relative "otopcua-secrets.db" (now boot-fatal);
dev gets the library's absolute per-user default, deployments supply their
own (wonder already pins an absolute out-of-root path in the service env).
- Program.cs's pre-host ${secret:} expander now calls the FOUR-argument
AddZbSecrets with builder.Environment.ContentRootPath. This call site is
the actual fix, not the pin: the throwaway container has no
IHostEnvironment, and without the explicit root the under-content-root
rule is skipped at exactly the moment the migrator creates the store file
(the 0.6.0 defect). Do NOT simplify back to the 3-argument overload — it
compiles and silently reopens the gap. Pinned by a Program.cs source-scan
test AND a runtime falsification control that RUNS the migration and
demonstrably creates the store at the rejected path under the 3-arg form.
- docker-dev: Secrets__SqlitePath moves /app/data → /data via a SECOND mount
of the same per-node volume (same file at the volume root, so seeded
stores carry over) because /app is the container's content root and the
new rule correctly rejects it. LocalDb stays on /app/data (secrets-only
rule).
- SecretsStorePathValidationTests: positive-control probes, artifact-first
(relative and in-content-root paths must FAIL leaving no file; outside-
root passes; the no-explicit-root control proves the file assertions can
fail) — a clean boot is equally consistent with the validator never being
wired.
Live-verified on the six-node docker-dev rig (at 0.6.1): all nodes boot
under the new validation with stores at /data; an AdminUI-written secret
converged into BOTH central nodes' independent stores; site-a held only its
own pair's secrets (Phase 6 pair-local isolation intact). Replicator wire
behavior unchanged 0.3.0–0.6.x per CHANGELOG; registration-graph pins and
the full suite pass.
The modal's existence was a function of _rows (FindSourceScript() as the
visibility gate), so any render where the row is momentarily unresolvable —
a future reload nulling or replacing the list — unmounts the subtree and
disposes the Close handler's binding id mid-click (GetRequiredEventBindingEntry).
Latent here (_rows is written once, in OnInitializedAsync) but the identical
shape bit ScadaBridge live the moment a page refresh landed, and it presents
as an unattributable intermittent once one does.
ScadaBridge's shape, ported by the family sweep: intent (_sourceScriptId)
gates existence, the resolve drives only content; an unresolvable script
renders a notice with the frame and Close still mounted, and closing remains
the operator's action rather than a side effect of list state.
Production ran six weeks with a healthy process and a 0-byte log because
Serilog isolates per-sink failures: a file sink whose every write fails is
invisible from anywhere but SelfLog, which was not enabled. Two instruments:
- LogPipelineDiagnostics.EnableSelfLog (before the first logger builds):
SelfLog to logs/serilog-selflog.txt (rotated at 5 MB) + stderr — the only
channel that carries the exception the sink swallows.
- CheckFileSinkGrowth (after the startup banner): verifies each configured,
unbuffered file sink's rolled-file family physically grew across a marker
event; a silent sink is reported at boot via SelfLog/stderr and the
console sink instead of discovered months later. Growth-across-marker,
not existence — a previous run's bytes must not mask a silent sink (the
false pass an existence check would have given on wonder-app-vd03).
Never throws: both instruments swallow their own failures — a diagnostics
channel that can fail startup is worse than the defect it watches for.
The load-bearing test is the failure case (a sink that writes nothing IS
reported); a check that can only pass is indistinguishable from no check.
The page ran members, the driver-instances heading, and every cluster group
together inside one panel; driver name and id slammed inline; one long driver
exception blew the table out sideways (Controls scrolled out of reach); and the
inline #546 expander row dumped a full-width panel into the table.
- One panel per cluster group, node chips in the panel head.
- Driver id renders on its own muted mono line under the name.
- Last error abbreviated to 60 chars (DisplayText.Abbreviate — #504 rule),
full text on the tooltip and in the Controls dialog.
- DriverStatusPanel moved from an inline expander row to a modal (one at a
time); the dialog re-resolves its row from the current groups each render so
the 5 s rebuild can't strand it, and the Enabled-in-key recycle discipline
carries over unchanged. HostsPanelWiringTests re-pinned to the dialog markup.
Live-verified on docker-dev: separated cards, in-viewport table, dialog opens
with live breaker/status data and grouped Reconnect|Restart, closes cleanly.
Sweep task 5 (Shared/Drivers + Acls):
- btn-group the Save/Cancel pair in DriverFormShell (all 13 driver forms),
CollectionEditor foot + per-row Edit/Delete, DriverConfigModal /
DeviceModal / DriverTagPicker / AclScopePickerModal feet,
DriverStatusPanel Reconnect+Restart and its inline restart confirm,
and the Refresh+Close browse toolbars on the FOCAS/AbCip/TwinCAT pickers.
- Replace the no-op inline border-color:var(--alert) tint with .notice-alert
in DriverFormShell, DriverConfigModal, DeviceModal, MqttDriverForm,
MTConnectDriverForm, MqttDeviceForm.
- Replace inline style="width:18px" expander sizing with .tree-twist in
DriverBrowseTree and AclScopeTree.
Markup only — every @onclick, binding, disabled and arm->confirm flow is
unchanged.
Joseph Doherty