SnapshotStreamer.SendAsync stamped as_of_seq = MAX(seq), which reads 0 once ack-pruning has
emptied a converged sender's oplog. A receiver whose watermark genuinely is 0 (a rebuilt node —
ScadaBridge's site-a-a since its 2026-08-15 rebuild) was healed by the snapshot but never lifted
by it (the receiver's watermark update is correctly MAX-guarded), so the 0.1.2 empty-oplog
horizon rule saw 0 < last_acked_seq as a gap on every subsequent handshake and re-streamed the
whole database on every reconnect until the sender wrote again. Observed live: the same 127k-row
full snapshot on two consecutive restarts of the pair.
Fix: as_of_seq = max(MAX(seq), last_acked_seq). Every delta pruned below last_acked_seq is
already folded into the rows the snapshot carries, so that is the floor it may claim; the
receiver's MAX guard then lifts the watermark and the next handshake finds no gap. Sender-side
only; wire-compatible; no schema change.
Test WipedPeer_SnapshotLiftsItsWatermark_SoTheNextReconnectSendsNoSecondSnapshot fails on the
unfixed code exactly as the rig does (watermark 0, expected 3) and passes with the fix: the
second episode exchanges only a delta, and a capturing logger on the snapshot streamer proves
no snapshot was sent. Suite 216/216 on 8 of 9 runs; one unnamed single-test failure on the first
run did not recur in eight further runs and could not be attributed.
Three review follow-ups on 122940c. Still 0.3.3 — unpublished, so no further
version bump. No behaviour change to any succeeding path.
- The per-entry poison handler's recovery work (ROLLBACK TO + the dead-letter
INSERT) was unguarded: a wedged connection, an I/O error, or a dead-letter
INSERT that faults would propagate the SECOND exception and silently discard
the FIRST — the one that says why the entry was poison at all. The pair is now
wrapped; a recovery failure throws an AggregateException whose message names
the entry (table + seq) and the recovery fault, and whose InnerException is the
ORIGINAL entry fault, with the recovery fault carried alongside it at
InnerExceptions[1] with its own stack. The outcome is unchanged and still
fail-closed: nothing commits, and disposing the batch transaction unwinds it.
OperationCanceledException is deliberately excluded from the wrap so a
shutdown still surfaces as a cancellation rather than as a fault. No upstream
caller catches SqliteException from the apply path (checked), so widening the
escaping type is safe.
Injected cheaply after all: a RAISE(ABORT) trigger on __localdb_dead_letter,
stacked on the existing row-version poison trigger, produces exactly the
two-fault case with no production test seam. The test asserts both causes are
carried AND that the batch committed nothing — the entry that applied cleanly
before the poison one is gone and the peer watermark never moved.
- Register_SelfReferencingCascade_Throws pins the README's "or from itself"
claim: a table that is its own cascading parent (parent_id REFERENCES
nodes(id) ON DELETE CASCADE) is refused. The guard needed no special case —
it reads every foreign key in the database — but the promise is now tested.
- Comment on why EnterEntryAsync sits outside the try: a SAVEPOINT that failed
opened nothing, so a handler ROLLBACK TO would name a savepoint that does not
exist and fault in turn.
Both new tests were run against 122940c first and fail there: the self-reference
case throws nothing, and the stacked-fault case escapes as a bare
SqliteException("dead-letter write failed") with the original cause gone.
Suite: 215/215 green, build 0 warnings / 0 errors.
Two latent apply-path hazards from the ScadaBridge adversarial architecture
review. Neither has a live trigger in any current consumer — ScadaBridge's ten
replicated tables carry no foreign keys at all (verified against the running rig's
site-localdb.db) and OtOpcUa's LocalDb tables carry none either — but both are the
kind that surface as silent divergence rather than as an error, so they are closed
before a consumer's schema grows into them.
Wire-compatible: no proto change, no lib_schema_version bump, no on-disk schema
change. No consumer code change.
- RegisterReplicated now refuses a table that is the PARENT of a foreign key with
a row-mutating ON DELETE action (CASCADE / SET NULL / SET DEFAULT), naming the
offending child column and action. The apply path writes remote rows with
INSERT OR REPLACE, whose PK-conflict resolution DELETES the existing row first;
connections run PRAGMA foreign_keys=ON, so that implicit delete fires the child's
ON DELETE action while the capture guard is raised. The child rows would change
on the applying node only, produce no oplog entry, and diverge the pair with
nothing able to detect it — LWW compares only rows it was told about. RESTRICT /
NO ACTION stay allowed: they abort the statement instead of mutating rows behind
capture's back, which the apply path already handles as per-entry poison. The
check inspects every foreign key in the database and runs at registration;
SQLite cannot add a foreign key to an existing table, so a child table created
later is caught at the next startup's registration, not at the DDL. Documented
on ILocalDb.RegisterReplicated and in the README's limitations list.
- An entry's row write and its ledger write are now atomic with each other. They
are two statements and SQLite aborts only the failing one, so a statement-level
fault between them — the case the per-entry poison catch exists for — committed
a half-applied entry: a tombstone whose table row was gone while
__localdb_row_version still called it live (SnapshotStreamer can only warn and
skip that, so the peer never learns the row died), or an upsert whose new row
carried the OLD version and therefore lost the next comparison it should have
won. The pair now runs inside a per-entry SAVEPOINT, rolled back (and released,
so nothing nests) before dead-lettering, leaving both writes or neither while
the rest of the batch proceeds and commits.
Every new test was run against the unfixed code first and fails there with exactly
the divergence shape described: the tombstone case reads back null instead of the
live row, the upsert case reads back the remote row against an unchanged ledger,
and the registration cases throw nothing.
Suite: 213/213 green, build 0 warnings / 0 errors.
A session that faulted during the handshake cancelled its writer loop immediately,
which could kill its OWN Handshake while it still sat in the control lane: the
enqueue -> receive -> validate -> fault path has no yields, while the writer's
wakeup waits in the thread-pool queue, so with the peer's handshake already
buffered the cancel won the race almost every time (39/40 in the new pinning
test). Mismatch detection is symmetric — the peer computes its fail-closed
verdict FROM our Handshake — so the healthy peer never learned anything: in
tests it waited out the full 15 s receive cap (the register-row-41 flake on
SyncSessionTests.Handshake_DigestMismatch_FailsClosed), and in production it
would misclassify a permanent schema mismatch as a generic disconnect and
reconnect-loop at the fast backoff instead of taking the classified
FailedPrecondition -> LogError + max-backoff path.
RunAsync's fault teardown now completes both lanes so the writer drains what is
already enqueued and exits on its own, with a 2 s deferred cancel as the backstop
for a wedged transport. Caller-initiated cancellation is unchanged (the linked
token is already cancelled, so shutdown is not delayed), and nothing on the wire
changes — a 0.3.0 peer simply receives the Handshake it was always owed.
New SyncSessionTests.Handshake_FaultTeardown_StillTransmitsTheLocalHandshake pins
the race (40 concurrent preloaded-mismatch sessions; 39/40 red pre-fix, green
post-fix).
BatchBudgetTests hardened separately: the register's other flaky test
(Pump_RowLargerThanTheBudget_IsSentAloneRatherThanStallingTheStream) awaits the
scripted-peer channel without ever observing the session task, so ANY session
fault surfaced as an opaque 15 s timeout. A run-observing NextAsync overload now
races the session task at every read site in the class, so the next flake reports
the session's real exception instead. (The handshake-race root cause does not
structurally apply to this test — no handshake fault occurs there — so its
intermittent timeout is attributed to the fault-hiding wait, now removed.)
Suite 200/200 green. Closes ScadaBridge deferred-work register row 41 (cross-repo).
Secrets/push.sh, Theme/pack.sh+push.sh, Audit/pack.sh+push.sh and Auth/pack.sh+push.sh
all carried the CWD-pack bug LocalDb's push.sh was cured of in 0038d46: a bare
'dotnet pack -c Release -o ./artifacts' packs whatever solution sits in the CALLER's
working directory, and the push scripts then shove the caller's ./artifacts/*.nupkg
onto the live Gitea feed (the incident that briefly published 25 unrelated packages).
Each script now cds to its own repo root, names its .slnx explicitly, writes to an
absolute artifacts path, and (push.sh) pushes an anchored glob. Verified by bash -n on
all seven plus a positive control: Audit's pack.sh run from a foreign CWD packed only
ZB.MOM.WW.Audit into its own artifacts dir and left the caller's directory untouched.
Push scripts deliberately NOT executed (they publish); pack half verified only.
Closes ScadaBridge deferred-work register row 42 (cross-repo).
dotnet pack with no project argument packs whatever solution sits in the caller's
CWD, so invoking this script by absolute path from another repo's checkout packed
and published 25 unrelated packages to the live Gitea feed under its name. Now it
cds to its own repo root and names the solution and an absolute output path.
Baselining one table used to set a single per-DATABASE needs_snapshot bit, and a
snapshot streams every registered table in full — so opting one table into
replication late re-sent all of them, in both directions at once on a pair
enabling replication late. On a database holding a high-volume table (an event
log at its retention cap) that is the largest transfer this library performs.
0.2.1 documented the cost and deferred the fix as needing an on-disk change.
Schema v3 adds __localdb_snapshot_state (table_name PK, needs_snapshot), created
by the shared DDL with IF NOT EXISTS so an older file gains it on open; the v2->v3
transition is a version stamp only. RegisterReplicated(..., baselineExistingRows:
true) now flags that table alone. The database-wide flag survives unchanged for
the paths that genuinely reach every table — a cap prune or an impossible peer ack
loses delta continuity for the oplog as a whole. DeregisterReplicated drops a
table's pending flag with its ledger rows, since nothing would ever stream (and so
clear) it afterwards.
Wire compatibility is additive and negotiated by CAPABILITY, not version:
lib_schema_version stays 1 because ValidateHandshake compares it fail-closed for
equality, so bumping it would stop 0.1.x/0.2.x pairs syncing outright. Handshake
gains supports_partial_snapshot (field 5) and SnapshotBegin gains a tables scope
(field 2) where EMPTY means all tables — precisely what a pre-0.3.0 sender emits
and what a pre-0.3.0 receiver assumes. A peer that does not advertise the
capability is served a full snapshot: a scoped one would make it advance
last_applied_remote_seq to the snapshot's as-of and silently discard every delta
below it for the tables left out.
For the same reason a partial snapshot moves no watermark on either side. The
receiver keeps last_applied_remote_seq (carrying only the HLC across, which is
MAX-guarded), and the sender keeps _sentThruSeq, so the deltas the snapshot
overlaps are still pumped and merged by LWW.
Crash safety: snapshot debt now clears on the receiver's new SnapshotAck rather
than on enqueue, so an ungraceful exit part-way through re-streams the same scope
on reconnect. The ack rides the unbounded CONTROL lane like DeltaAck, so it
composes with 0.2.1's bounded 64-message inbox and its out-of-handshake snapshot.
Against a pre-0.3.0 peer the 0.2.x clear-on-enqueue behaviour is kept — there is
no ack to wait for — with the handshake's horizon gap check as the net.
Tests 199/199 (was 188): single-table baseline streams only that table, mixed-
version interop (peer without the capability gets a full snapshot), debt standing
until SnapshotAck, partial snapshot not advancing the receiver's watermark, no
SnapshotAck emitted to an old peer, end-to-end over two real databases, plus v2->v3
upgrade and per-table flag isolation.
README, ILocalDb remarks (which named this as the follow-up) and the umbrella
CLAUDE.md index updated.
The umbrella LocalDb row and the pack/test quick-reference now name 0.2.1 (188 tests,
3 nupkgs on the feed). Records both findings, the snapshot-out-of-handshake change that
bounding the inbox required, and the deferred per-table needs_snapshot. Consumer pins are
unchanged: ScadaBridge still pins 0.2.0, OtOpcUa still pins 0.1.3.
Two adversarial-review findings against 0.2.0. Wire-compatible with 0.1.x/0.2.0 peers:
no proto change, no lib_schema_version bump, no on-disk schema change.
F4 — DeregisterReplicated destroyed the HLC clock's only durable crash anchor. Clock
recovery on open is MAX(hlc_high_water, MAX(oplog.hlc), MAX(row_version.hlc),
last_seen_hlc); hlc_high_water is written only by Dispose, and deregistration deletes the
table's rows from BOTH ledgers. On a file whose registered tables have all been
deregistered, an ungraceful exit (crash, SIGKILL, power loss) therefore left nothing above
zero to recover from: the reopened clock restarts from wall-clock time and can re-issue
stamps below HLCs peers already hold, which LWW discards as older. The current clock is now
flushed inside the deregistration transaction, before the prune, as a separate command so
it cannot inflate the "did anything change?" answer. FlushHlcHighWater takes an optional
transaction (SQLite rejects a command with no transaction on a connection that has one) and
is now MAX-guarded — every writer of that column is flushing a monotonic clock, so a lower
value can only be stale.
F3 — the sync inbox was Channel.CreateUnbounded, making inbound memory a function of what
the PEER sends. The bad case is not hypothetical: a snapshot streams every row of every
registered table, needs_snapshot is per-database, and both sides of a pair that enables
replication late baseline unconditionally — so a mutual full snapshot of a large dataset
(an event log near its storage cap) was buffered whole, in memory, on both nodes at once.
The inbox is now bounded at 64 messages with wait-mode backpressure, so the ceiling is
~64 x MaxBatchBytes — a configured number rather than a peer-controlled one — and a full
inbox stops draining the gRPC stream, which closes the peer's flow-control window. The
pump writes with WriteAsync, not TryWrite: on a full bounded channel TryWrite drops the
message, and every inbound message is protocol state.
Bounding alone would have deadlocked the very case it was aimed at. The snapshot a node
owes was streamed from INSIDE the handshake, before the receive loop existed, so neither
side drained its inbox while sending; a mutual snapshot converged only because the inbox
was unbounded. The snapshot now runs as the first phase of the outbound task, started
alongside the receive loop, so the inbox is drained for the whole life of the session.
Interleaving needed no new safety: snapshot rows and deltas both merge through LWW and the
snapshot's watermark write is MAX-guarded. Ordering within the outbound lane is preserved
(the pump still starts only after the snapshot, from its as-of seq).
Per-table needs_snapshot (the finding's optional third part) is NOT done: the flag lives in
__localdb_peer_state and narrowing it needs an on-disk schema change, which this fix
deliberately avoids. Documented as a follow-up, and the cost is now written down where it
is decided — RegisterReplicated(baselineExistingRows) remarks and the README both say that
baselining one table re-streams every registered table in both directions, and that
high-volume tables should be trimmed before enabling replication late.
188/188 tests pass (was 184). New coverage: the deregistered clock anchor surviving an
ungraceful reopen (a second instance over the same file with no Dispose — fails without the
flush, recovering at 0), the flush not counting as work done, bounded-inbox backpressure
under a slow reader (the pump stalls at the bound and loses nothing, proven by contiguous
in-order delivery), and a mutual snapshot over a capacity-1 transport converging — that
last one deadlocks and times out against the old handshake-phase snapshot ordering.
Version 0.2.1.
WP3.3 of the ScadaBridge architectural-review remediation. Six changes, all
wire-compatible with 0.1.x peers — no proto change, no lib_schema_version bump. The
on-disk bookkeeping schema goes to v2 and is upgraded in place when a file is opened.
DeregisterReplicated(table). There was no way to stop capturing once a file had been
registered, so a node that does not replicate kept paying three triggers inside every
write transaction, appending to an oplog nothing would ever drain. Drops the triggers and
prunes the table's oplog + row-version rows in one transaction; idempotent, and
deliberately usable on a table this process never registered, since stale triggers left by
an EARLIER build are the primary case. Deregistration must be symmetric across a pair, and
already is enforced: the handshake compares registered-table digests fail-closed, so a node
that drops a table its peer still replicates stops syncing loudly rather than diverging.
RegisterReplicated(table, baselineExistingRows: true). Closes the late-opt-in hole: the
snapshot streamer pages from __localdb_row_version, which only triggers populate, so a node
that started replicating LATE never sent its pre-existing rows — silently, forever. Seeding
alone would not have fixed it, because a node whose oplog starts at seq 1 presents no gap at
the handshake and is told it needs no snapshot; the seed therefore also flags needs_snapshot,
and the existing resync machinery carries the rows. The floor is HLC 0 stamped with this
node's id: every genuine HLC is a UTC millisecond shifted left 16 bits and so is strictly
greater, which makes a baselined row lose every comparison against a real remote write and
win only where the peer has no version of that key at all. Two nodes baselining the same key
both hold 0 and resolve by the node-id tie-break — convergent, and nothing better is
available for two rows of unordered antiquity. Opt-in, so the documented CDC contract is
unchanged unless asked. Pruning the ledger on deregister is safe precisely because
re-registration can re-baseline; the two APIs are a pair.
O(1) backlog depth. GetOplogDepth ran COUNT(*) over the entire unacked backlog, once a
minute from maintenance AND on every metrics scrape — a full scan of a million rows exactly
when a node is already behind. Now MAX(seq) - MAX(last_acked_seq, MIN(seq) - 1), floored at
0: three b-tree seeks, measured at 58 VM steps against 100,020 for the count on a 50k
backlog. Exact, because seq is AUTOINCREMENT and every prune path removes a prefix (ack
pruning by watermark, the row cap by count, the age cap by an HLC cutoff that rises with
seq). The MIN clamp is what keeps it exact after a cap prune has moved the horizon above a
stalled watermark — without it the caps would re-flag on every tick forever.
Byte-budget batching. MaxBatchSize was the only cap on an outbound message, which made it a
proxy for a byte budget that consumers had to tune against their widest row: ScadaBridge
pinned it to 16 because 70 KB config_json times the 500 default is ~35 MB against a 4 MB
gRPC limit. MaxBatchBytes (default 2 MB) now bounds delta and snapshot messages by summed
serialized size, with the row count demoted to a secondary cap. A row larger than the budget
is sent alone rather than stalling the stream on it. Since more, smaller messages would
otherwise mean more apply transactions and more acks, inbound delta batches and acks ALREADY
queued are coalesced — one transaction and one ack per burst, bounded by the sender's own
caps, and a no-op under a trickle.
LwwApplier prepares each statement once. It went through the generic transaction façade,
which builds a fresh SqliteCommand per call: three SQL parses per applied row, on the path
that drains a reconnect backlog. Statements are now held for the batch and rebound.
Dropped __localdb_oplog_hlc. It indexed (table_name, pk_json, hlc), which no query has ever
used — every oplog access path is by seq or a full hlc-predicate delete — while costing a
b-tree insert inside every captured consumer write.
184/184 tests pass (was 149). New coverage: schema upgrade drops the index and preserves
identity/watermarks/data, deregistration idempotency and trigger absence and cross-process
cleanup, baseline seeding at the floor with LWW safety proved through the real applier,
depth asserted equal to the counting query across every prune scenario plus a constant-time
assertion over a 200k backlog, byte-budget chunking including the one-giant-row edge on both
the delta and snapshot paths, prepare-once statement counting, and coalescing that must not
swallow the message that ends its run.
Version 0.2.0.
Two local glauth containers (zb-ldap-primary :3893 / zb-ldap-backup :3894)
simulating the NAM two-DC topology, so the primary can be killed during a
drill without touching the shared 10.100.0.35 instance. Drill recipe +
verified-PASS record (HistorianGateway, 2026-08-13) in the README.
deployment.md:
- §1/§2.1: add wonder-app-vq03, wonder-app-vq04 and wonder-sql-vq03 with IPs and
per-host roles; new §7.1 QA port map + live active/standby state.
- QA is a real 2-node HA pair (ScadaBridge central+site pairs, OtOpcUa pair) where
DEV is a single all-in-one box, so it is the only environment here that exercises
failover and leader election. MxGateway/HistorianGateway/Overview run on vq03 only.
- Every QA fact is marked scan (port probe) or dash (read off the QA Overview
dashboard's live polling). No QA config file was read; connection strings and
service names stay unverified.
- Correct the tier: vd03 is DEV, not production (vd=dev, vq=qa, per
mes-delmia-integration-api.md §1.1). Its ASPNETCORE_ENVIRONMENT really is
"Production" -- a .NET overlay name, orthogonal to the tier, and the likely source
of the original error. No production box has been identified.
- Record that QA requires Basic auth on /management where DEV does not, so DEV-only
procedures relying on unauthenticated management calls will not work on QA.
- Record that inbound API keys never cross environments (per-stack peppered-HMAC,
excluded from config bundles) plus the 401-vs-400 probe, with the caveat that auth
runs before route resolution so a 401 needs a positive control.
- Withdraw the SSH-on-2222 claim made earlier in this same pass: all three wonder
boxes accept TCP on 2222 and send no SSH banner, so servecli is not serving SSH and
no shell access was available. Port scan cannot establish a service; banner grab can.
CLAUDE.md:
- Overview: record the second instance on wonder-app-vq03:5320, and that its real-FQDN
registry is correct rather than a divergence -- the 127.0.0.1 rule is specific to
vd03 co-locating every instance on one host.
- Add the DEV-vs-prod note pointing at deployment.md §2.1/§7.1.
CLAUDE.md: feed now carries 0.2.1; record the referral defect and that every
app is still exposed until it bumps off 0.1.5 (a functional fix, not
housekeeping), plus the new tools/LdapProbe.
ldap_setup.md: findings from probing the live NAM directory --
- LDAPS 636, StartTLS 389 and plain 389 all connect on both DCs. The only TLS
chain error is UntrustedRoot, so installing zimmer-rootca is sufficient; the
RSA 1024-bit key is rejected only by OpenSSL at SECLEVEL=2, i.e. Debian .NET
containers, not by macOS or Windows hosts.
- The UG_WW_* Wonderware group convention, 39 groups enumerated directory-wide.
This partly overturns this doc's earlier "no group->role convention exists"
claim: that was true of Ignition, which was the only source audited, and
false of the domain. Three shapes with a consistent role suffix (ADMIN, RO,
OPERATOR, ENGINEER, SUPERVISOR, TESTER) across environment and site axes.
- A proposed mapping onto CANONICAL-ROLES, with the three gaps left open
rather than guessed: SUPERVISOR/TESTER have no canonical equivalent, nothing
maps to Deployer, and the DEV/QA/PROD axis is orthogonal to role so it must
ride in the per-project scope payload or the role set triples.
- The referral defect and its fix.
Diagnostic for pinning down the LDAP settings the four ZB.MOM.WW apps need
against the Zimmer NAM domain, and for answering which AD groups can be mapped
to app roles. Found the referral defect fixed in 0.2.1.
Two independent halves in one run:
Part A drives the REAL ZB.MOM.WW.Auth.LdapAuthService -- same bind-then-search-
then-bind, same options validator, same failure enum -- so its verdict IS what
the apps produce rather than an approximation of it.
Part B asks the directory what the library cannot, which is where the group
answer comes from: full group DNs, nested groups via
LDAP_MATCHING_RULE_IN_CHAIN, the primary group (AD omits it from memberOf, so
it can never be a role group), security vs distribution type, decoded
userAccountControl, referral counts, and the TLS chain with key sizes.
--list-groups '<pat>' enumerates every matching group in the directory, which
a membership dump structurally cannot do -- it only ever shows one account's
memberships.
Passwords are prompted for and never echoed; they cannot be passed as
arguments, which would put them in shell history. --self-bind lets the user
under test stand in for the service account, so group discovery does not
require the svc_igntnapp secret.
Not packable, outside the Auth solution and its central package management,
referencing the Auth projects directly so it tracks current source.
Active Directory answers any search based at a domain root with continuation
references (CN=Configuration, DC=ForestDnsZones, ...). With referral following
disabled -- the default, and what we want, since chasing one would re-bind
somewhere we never authenticated -- Novell surfaces each by throwing
LdapReferralException from Next().
That escaped NovellLdapConnection.Search, and LdapAuthService maps a failed
search to ServiceAccountBindFailed. So every login against
SearchBase = DC=nam,DC=zimmer,DC=com failed system-side on BOTH DCs in ~546 ms
while the directory, the credentials and the service account were all healthy.
Identical on each endpoint, so failover could not help either.
A continuation reference is a pointer to another naming context, never a
statement that this search failed, so the loop now skips them and continues --
returning exactly the entries in the naming context we bound to. Extracted as
ReadEntries(ILdapSearchResults) so the handling is unit-testable through the
Novell seam without a live directory. A 1000-referral cap guards a client that
throws without advancing, which would otherwise spin forever inside a
fail-closed auth path.
Non-referral faults still propagate untouched: swallowing those would turn
"the directory is broken" into "no such user", which is the wrong answer to
give about someone's credentials.
7 tests, 6 verified red against the pre-fix code; the seventh is the
non-referral control that must stay green either way. Suite 257/0, 2 skipped.
Live-proven against the ZB NAM domain root before and after.
Asked whether the gateway's new session row also covered HistorianGateway.
It did not, and could not: HistorianGateway publishes its pool depths on a
different TIER with a different KEY SET.
mxaccessgw active tier total / ready / faulted one check
historiangw READY tier warmSessions / pool / … TWO checks
The session row is located by the total/ready/faulted triple, so it correctly
ignored these rather than half-matching them. This adds a sibling row instead
of unifying the two behind a shared shape, because there is no such shape -
inventing one would mean rendering a contract neither app publishes.
No poller change was needed: the ready tier's body is already retained, so
this is entirely a rendering change.
Labelled "Sessions pooled", deliberately not "connections". warmSessions is
pool DEPTH - warm pre-authenticated sessions held ready - and sessions are
leased on demand, so the event pool legitimately sits at 0 until something
reads events. Calling that a connection count would make a normal idle pool
read as an outage, so 0 is shown plainly and the row goes red only on
consecutiveFactoryFailures, which is an actual fault.
Pools render read-then-event rather than in dictionary order, so the line is
stable across polls and the pool an operator cares about comes first.
188 tests (183 + 5): ordering, that an idle pool is not painted red, that a
failing factory is, a positive control on the omission assertion, and one
pinning that neither app's row can render the other's data.
The gateway publishes its live MXAccess session counts, and the dashboard
could not reach them. Four things stood between the data and the card, and
only the last one is cosmetic:
The active tier was probed only for instances declared HasActiveRole. A
standalone gateway has no active/standby role, so its active tier was never
requested at all. The tempting fix is to declare mxgw HasActiveRole, and it
is wrong: that flag also paints an Active/Standby badge, so it would buy the
data by asserting a role that does not exist. The probe is now unconditional
for any reachable instance while ObserveActive stays role-gated, separating
"what does this tier report" from "am I the active half".
The active probe's parsed body was then discarded - only ready.Report was
kept - so InstanceSnapshot gains ActiveReport. The two tiers stay separate
rather than merged, because a check name is only unique within a tier.
The card's one data-rendering path, ClusterDataLine, is located by the Akka
leader/activeNode keys and would ignore session data even once present. The
new row follows that helper's own precedent and locates the check by the data
it carries - the total/ready/faulted triple - not by name, requiring all three
so a future check publishing a lone "total" cannot be mistaken for it.
Rendered on the card FACE, not inside the collapsed check list: a count behind
a disclosure triangle is one nobody reads. Zero is shown rather than hidden,
since a gateway holding no sessions is the interesting case and suppressing it
would make the row appear only when the news is good.
Cost, stated plainly: every reachable instance now makes one extra HTTP
request per poll. The unreachable-instance skip is untouched, so the
worst-case sweep duration - which is set by timeouts, not by healthy
round-trips - is unchanged. Four cadence tests were counting raw requests as a
proxy for "instances polled"; they now count ready-tier probes, which is what
they always meant and is robust to tier changes.
The Wonder registry comment claiming the active tier ran an empty check set
was true when written and stopped being true when the gateway added the
check; it now warns against the HasActiveRole shortcut instead.
183 tests pass (177 + 6), including a positive control on the absence
assertion so "row not rendered" cannot pass vacuously.
Suggested by the OtOpcUa session, the only real containerized deployment in this family
and the one that hit the false positive. A store on a volume mounted inside the content
root is durable across container replacement but is rejected anyway, because the rule
cannot tell a mount from an ordinary directory. The message now says so, gives the fix,
and forecloses the wrong one by stating that no override exists.
A false positive's entire cost is the confusion in the moment, so an error that names its
own fix costs nothing on the true-positive path and turns the false positive into a
two-minute edit. Message only, no behaviour change.
Their case for keeping the rule strict is recorded in the CHANGELOG because it is better
than the trade-off I had written: the workaround is an improvement rather than a
concession, an override flag has the lifecycle of a knob set once and never revisited,
and mount detection would make the verdict depend on runtime platform state so the same
path would pass on one host and fail on another.
Joseph Doherty