Raised by the mxaccessgw session, which went back to its own tests after I reported a
vacuous assertion in mine and found the same class one step further from the fuse.
The rejecting test asserts no store file is left at a rejected path. That claim is only
meaningful if something proves the write path is live at all: "no file was created" is
indistinguishable from "nothing in this container ever creates files". The positive
control was supposed to establish that and did not -- it asserted Assert.NotNull on a
resolved SecretsSqliteConnectionFactory, which is close to a tautology once no exception
escaped, and it never ran the migration at all. So if the pre-host container ever stopped
touching the store, the rejecting test would have gone quietly vacuous while the positive
control kept passing. Neither would have failed.
The positive control now runs SqliteSecretsStoreMigrator.MigrateAsync and asserts the
database IS created. The pair is mutually checking: the migration writes the file when
allowed and does not when rejected, and neither assertion stands on its own.
Verified the same way as the last one, by simulating the failure rather than reasoning
about it: replacing the migration with a no-op makes the positive control fail with "the
accepted path must actually get a store file". Reverted after measuring.
No library change; 0.6.1 as published is unaffected. Suite 502 passing.
Two defects in the 0.6.1 regression test, the first raised by the mxaccessgw session
from its own adoption, the second exposed by acting on it.
Assertion order. The test asserted the throw before the artifact. That hides the
stronger finding: a regression that stops the throw but still creates the store fails
with "no exception was thrown" and never mentions the database sitting at the rejected
path -- which IS the defect. Now captures with Record.ExceptionAsync, asserts no file
created first, and asserts the exception type last. Notable because the defect under
test was itself "a boot outcome was observed instead of the artifact being probed", so
closing it with an outcome-first assertion would have repeated the mistake.
Vacuous assertions. Simulating the regression -- replacing the 4-argument call with the
3-argument one, which still compiles, exactly as the call-site comments warn -- showed
the test failing on the exception assertion rather than the file assertion. No file had
been created in EITHER case: the test resolved SecretsSqliteConnectionFactory, whose
constructor never touches the filesystem. The File.Exists assertions could not fail. The
test now resolves SqliteSecretsStoreMigrator and runs MigrateAsync, which is what a
pre-host expander actually does and what actually creates the database.
Verified by simulating the regression again: the test now fails with "a rejected path
must not be left holding a store file" rather than "no exception was thrown". Reverted
after measuring.
No library change; 0.6.1 as published is unaffected. Suite 502 passing.
Found by the HistorianGateway adoption of 0.6.0, which probed the rejected paths
instead of observing a clean boot — a clean boot being equally consistent with the
validator not being wired at all.
An application that expands ${secret:} references before building its host composes
this subsystem in a throwaway ServiceCollection with no IHostEnvironment in it, then
migrates the store from that container. SecretsOptionsValidator resolves the content
root from IHostEnvironment and skips the under-content-root rule when it is absent, so
pre-host an absolute in-content-root path passed the only rule still applying, the
migrator CREATED the database, and the boot failed a second later once the real host
registered IHostEnvironment and the rule finally fired.
The observable result was a rejected boot that still left an empty store, with -wal and
-shm siblings, at the rejected path — the same artifact that made the MxGateway outage
read as "the database is there, it's just empty", which is precisely what this guard
exists to prevent. A relative path was unaffected because the absolute-path rule fires
with or without a content root; that asymmetry is what exposed it.
The mistake was conflating "no IHostEnvironment registered" with "no content root
exists". New AddZbSecrets overload takes the content root explicitly; the 3-argument
overload is unchanged and still falls back to IHostEnvironment, so this is additive.
SecretsOptionsValidator gains a string? constructor alongside the IHostEnvironment? one.
Reproduced first in an isolated harness before changing anything: a bare host rejects
both bad paths leaving no artifacts, and a bare ServiceCollection with no
IHostEnvironment migrates an in-content-root path to disk. Both are now regression tests,
the second paired with a positive control asserting the old behaviour still holds when no
content root is supplied — so the test is about the explicit content root rather than
about paths in general.
Also documents an upgrade note that applies to 0.6.0: a store seeded under the old
relative default is orphaned rather than migrated, and nothing signals it.
Secrets:SqlitePath is now validated at startup by SecretsOptionsValidator using two
rules in the new public SecretsStorePathRules. Both come from production incidents and
neither implies the other.
Must be absolute: a relative value resolves against the process working directory,
which for a Windows service is C:\Windows\System32, so the store lands outside the
installation; in a dev loop it lands in the source tree (a gateway was observed writing
its secrets database into src/, tripping that repo's own tree-hygiene test).
Must not be inside the content root: an absolute path can still point into the
directory every deployment procedure in this family renames on upgrade. On 2026-08-09 a
gateway's credential store sat inside its Server\ directory; the deploy renamed Server
to Server.bak.*, the database went with it, and the process created a fresh empty one
at the same path. Every consumer lost its credential, silently, for two days. The
existing "must be absolute" convention did not help — the path was absolute.
The rules reject rather than relocate. Silently moving a credential store presents later
as an authentication failure pointing nowhere near the configuration that caused it; a
boot error is seen immediately by whoever caused it. Validation is deliberately not
exempted in Development: a guard that never runs in a dev loop or in CI rots, and the
configuration behind the outage looked entirely reasonable.
BREAKING for configuration, not API. SecretsOptions.SqlitePath now defaults to an
absolute per-user path under LocalApplicationData; it was "secrets.db", which the new
rules reject. An application configuring a relative path will fail to boot — remove the
key or set an absolute one outside the content root. All five deployed instances of this
family were checked first and already set an absolute path outside their content root,
so the migration edge (a host relying on the OLD default now finding nothing and
creating an empty store — this bug, re-introduced by its own fix) does not apply to any
of them.
Per-user rather than machine-wide is deliberate: CommonApplicationData is /usr/share on
Linux and macOS, which a normal user cannot write, so a machine-wide library default
would fail every consuming application's boot on a developer machine. An application may
compute its own default; the configured value always wins.
Validator registration is idempotent. AddZbSecrets is legitimately called more than once
in real compositions, and TryAddEnumerable cannot de-duplicate a factory-built
descriptor — it de-duplicates on implementation type, which a factory does not carry, so
it throws on the second call rather than skipping. The existing suite caught this.
Tests cover both rules including the segment-boundary case (/srv/app-data must not read
as inside /srv/app — a false positive on a fail-closed startup rule is an application
that will not boot), that the shipped default satisfies the rules it ships with, that
the default is not machine-wide, idempotent registration, and — most importantly — that
a bad path actually throws when IOptions is resolved, since a correctly implemented
validator that is never invoked is the same "guard that does not fire" failure this
change exists to prevent.
Family bumps to 0.6.0. Adds a dependency on ZB.MOM.WW.Configuration 0.1.0 for the shared
OptionsValidatorBase/ValidationBuilder vocabulary, which every consumer already
references.
This repo hosts one application besides its shared libraries, and the index did
not mention it at all. Adds what it is, that its registry ships empty by design,
and the two constraints that bit during the first real deployment: publish
self-contained (wonder has no .NET 10 runtime), and do not put co-located
instances on a shared host name, because leader-to-instance resolution matches on
host only and the first match wins the chip for every group.
Live on the box, the ScadaBridge *site* card claimed "Leader · central". Central
and site are separate Akka clusters, so that name was simply wrong.
Cause is in the registry, not the app. ResolveToInstanceName matches a reported
Akka leader address to an instance on HOST ONLY - correct by design, because an
app's Akka remoting port and its HTTP port differ, so comparing ports would never
match. It assumes what holds on the docker rig and not here: distinct hosts per
instance. All five instances are on one box, and the scan is fleet-wide in
registry order, so with every BaseUrl on "localhost" the first matching entry won
the chip for every group.
No host value makes this resolve CORRECTLY, because the instances really are
co-located. So make it never match instead: ScadaBridge's Akka addresses use host
"localhost" and OtOpcUa's uses the FQDN, and "127.0.0.1" is string-unequal to
both. Every chip now falls back to the raw Akka address - unambiguous and always
true. A raw address beats a confident wrong name.
Using the FQDN for OtOpcUa's BaseUrl would have resolved its chip correctly and
was rejected: that name resolves to a link-local IPv6 on this box, and routing
the real health probe through it to improve a label trades the primary function
for a cosmetic one.
ManagementUrl values are unchanged - they are followed by the operator's browser
from another machine and must keep the box address.
Adds appsettings.Wonder.json for the first deployment of this dashboard to a
real box. Named "Wonder" rather than "Production", mirroring the existing
Docker overlay: this is ONE deployment's fleet, not a claim about what every
production deployment looks like.
Every port was established by probing the running box, not inferred from the
docker rig - and the two disagree, so inference would have been wrong:
OtOpcUa :9000 (AdminUI on the same listener)
ScadaBridge Central :8085 (rig serves this on :5000)
ScadaBridge Site :8084 (no UI - / is 404, so the card carries no link)
MxAccessGateway :5130 (:5120 is gRPC)
HistorianGateway :5220 (:5221 is gRPC h2c)
:8083 and :8086 answer 400 to an HTTP/1.1 GET - h2c gRPC - and are excluded.
Both gateways are HasActiveRole:false. Their /health/active does return 200,
but only because the active tier runs an EMPTY check set when no Active-tagged
check is registered, so an Active badge would assert a role neither has.
Central and site are separate groups because they are separate Akka clusters;
one shared group would manufacture a leader disagreement.
BaseUrl is localhost (probed by the service, on the box) while ManagementUrl
carries the box address (followed by the operator's browser, elsewhere).
Serilog's file path is absolute and outside the Server directory an upgrade
replaces - a relative path resolves against the service AppDirectory, so every
upgrade would silently eat the log history. Same class of defect that cost
HistorianGateway its secrets store on a swap.
Boot-verified locally under ASPNETCORE_ENVIRONMENT=Wonder: registry passes
OverviewOptionsValidator and the host starts.
Family alignment ahead of the first wonder-app-vd03 deploy. Both bumps are
inert for this app, established rather than assumed:
- Theme 0.3.1 -> 0.4.1 is two commits: the upstreamed Bootstrap .btn sizing
block and the rail-btn-block modifier. Overview answers NO to both of the
questions the kit README says to ask - it declares no .btn sizing of its own
and has ZERO call sites carrying a bare `btn` token (its single button is
class="rail-btn"), so the new .btn rule matches nothing here. The .rail-btn
refactor that introduced --rail-btn-inset resolves to the same 0.6rem the
literal margin used, so the one call site is unchanged. Same answer as
HistorianGateway. Note a naive \bbtn\b grep is a FALSE POSITIVE on
"rail-btn" - the hyphen is a word boundary; match a whitespace-delimited
token instead.
- Health 0.2.0 -> 0.3.0 is a pure version-number alignment: the core
ZB.MOM.WW.Health package is byte-identical between the two, with the entire
delta confined to ZB.MOM.WW.Health.Akka (ActiveNodeHealthCheck,
ClusterActiveNode, AkkaActiveNodeGate), which this app does not reference.
Verified with a positive control on the diff path. The 0.2.1 active-tier
503 change therefore cannot reach this app's own endpoints; it registers no
Active-tagged check, so /health/active runs an empty set and stays 200.
Overview consumes the 0.3.0 `activeNode` data key over the WIRE from other
apps' health JSON, which is independent of the package it compiles against -
LeaderResolver already ranked it (344bdc4) while pinned to 0.2.0.
Release build 0 warnings; 177/177 tests pass. All remaining outdated packages
are test-only (bunit, coverlet, test SDK); nothing the app ships is behind.
A commit that removed a modal's self-closing behaviour shipped two comments in the
same file still describing it as self-closing — review checks whether the code is
right, not whether the surrounding prose survived. And the property that predicts
which modals break on refresh is 'holds no state of its own' (pure projection of a
list), not the presence of a detail field.
Readers take 'what I could not establish' as evidence the rest was held to that
standard. Write the claim the evidence licenses, not the broader one; and when two
items in your own not-established list contradict each other, flag it — that
contradiction is the strongest lead you have, not another unknown.
A bare 32-hex key id matches none of the six credential keywords (verified). Added
an entropy scan and a PEM check as generic complements, marked the prefix list
repo-specific, and made the primary rule explicit: a gate that mints anything must
grep the commit for the literal values it produced.
A filename audit finds the wrong-file problem; it cannot find a credential pasted
into a legitimately-tracked doc. Both checks are cheap and catch disjoint failures.
Verified: .claude/ is gitignored in OtOpcUa and HistorianGateway, NOT in mxgw or
ScadaBridge — in those two, enumerate-first discipline is the only guard.
A sweep's landing commit is where unrelated local state gets swept in. One repo's
'78 modified' was 76 modified + 2 untracked, one of which was .claude/ holding
settings.local.json — a literal 'git add -A' would have put permission config into
a tracked commit feeding a production deploy, invisible in a 76-file diff.
Two deadlocked investigations both inherited 'file exists => sink constructed =>
no events flowed'; the pinned sink version creates no file until the first emitted
event, so the artifact proved the opposite. And 'a second log wrote in the same
directory' rules out directory-level failure, not per-handle failure — a proof
about the container is not a proof about the member.
HistorianGateway has no local block AND no Bootstrap button call sites (all 15
buttons use the kit's .rail-btn chassis class), so the kit's .btn rule matches
nothing and bumping it changes nothing visible. My previous correction claimed the
opposite. Checking this needs two greps — declaration AND call sites — which come
apart exactly when an app doesn't use Bootstrap buttons.
Three instances in one round (btn-block universality, sweep behavior-preservation,
the mxgw Bootstrap exemption), all the same shape, all caught by the asserter
re-checking rather than by challenge. Fix is a cheap per-member check that runs,
or an honestly-scoped claim when it hasn't.
HistorianGateway has no local .btn sizing in either its committed tree or its
sweep branch (only vendored bootstrap.min.css), so the shipped comment and README
naming all five consumers were wrong. Consequence worth stating: HG's bump to
0.4.0+ is a VISIBLE change (buttons shrink to .85rem), not the no-op cleanup it is
for consumers that already carried the block.
The prior wording claimed the two rules 'cannot drift apart', which is only true
for the documented composition. 0.6rem still appears twice; the fallback confines
a stale copy to the undocumented modifier-alone path. Records why hoisting the
var to :root would be worse rather than better.
Unreleased — behaviourally identical to the published 0.4.1 (browser-verified:
composed <a> and <button> both fill the rail foot exactly, 0.00px short). The
var() carries a 0.6rem fallback because --rail-btn-inset is declared on .rail-btn,
so rail-btn-block used alone would otherwise have an undefined var, invalidating
the whole width declaration at computed-value time and silently reverting to
width:auto — the exact <button> shrink-wrap bug 0.4.1 fixed.
Two lessons from the Theme 0.4.0 rollout across five apps.
Live gate: after a kit-CSS change a normal reload can serve the cached previous
layout.css while the server serves the new one. .btn then computes at
Bootstrap's 16px — the exact signature of a failed upstreaming — so the natural
response is to restore the local rule just deleted, entrenching the duplication
the change removed. Hard-reload before trusting any computed-style probe.
Upstreaming: the kit ships sizing only, so trim the local .btn rather than
deleting it. mxgw's block also carried border-radius/font-weight/white-space,
which a wholesale delete would have dropped silently; three of five apps' blocks
were pure sizing and safe to delete outright, and the difference is invisible
without diffing. Also records why a retained border-radius must stay a literal.
Marks the .btn block as upstreamed in 0.4.0 rather than a candidate.
0.4.0's rail-btn-block used display:block + width:auto, which fills for an <a>
but SHRINK-WRAPS a <button> — a form control does not take the fill-available
behaviour a non-replaced block box gets. Measured in one 197px rail-foot
container: <a> filled to 177.42px (0 short), <button> came up 114px short.
That matters because a rail foot normally holds both element types: Sign In is
an anchor, Sign Out is a form-submit button. So 0.4.0's modifier got one right
and one wrong, and the wrong one is what signed-in operators see — worse than
the width:100% overflow it was meant to replace (~10px of visible clipping).
Fixed with an explicit calc(100% - 1.2rem) + box-sizing:border-box, which
measures 0px short for both element types. Both rejected shorthands and their
measured failures are recorded in the CSS so neither is reintroduced as a
'simplification'.
Found by HistorianGateway while evaluating adoption; it held off adopting
rather than trading one defect for a larger one. No consumer had adopted the
modifier, so nothing shipped broken. The .btn half of 0.4.0 is unaffected and
is already live-verified in three apps.
Build 0 errors, 48/48 tests.
Bootstrap hard-codes --bs-btn-font-size: 1rem, oversized against the kit's
0.9rem body. All five consuming apps had independently written the identical
block into their own stylesheets (ignitionoee app.css, OtOpcUa site.css, mxgw
site.css, HistorianGateway site.css, ScadaBridge site.css), each carrying a
'delete this when the kit ships a .btn rule' comment. This is that release.
Sized by CSS-variable override only, never by redeclaring box properties:
Bootstrap's .btn-group seam/radius machinery reads the same variables, so a
literal font-size at equal specificity flattens .btn-sm into a padding-only
difference. That was a real defect in mxgw's local copy, so the rationale is
recorded in the CSS to stop it being 'simplified' back.
Lands in layout.css, which ThemeHead emits after theme.css and both after
Bootstrap, so the override wins on source order at equal specificity.
Ships in the same 0.4.0 release as rail-btn-block. Consumers get the sizing on
bump (not opt-in, unlike rail-btn-block) and should delete their local copies
in the same change. Build 0 errors, 48/48 tests, packed asset verified to carry
.btn, .btn-sm/.btn-group-sm and rail-btn-block with no literal font-size.
HistorianGateway hand-rolled a .rail-btn override purely to make the rail-foot
sign-out button span the rail; its own comment flagged folding it into the kit.
Adds a composable modifier instead: class="rail-btn rail-btn-block".
Uses display:block + width:auto, NOT width:100%. The hand-rolled overrides set
width:100%, which resolves against the containing block and then adds .rail-btn's
0 0.6rem side margins on top, overflowing the rail by 1.2rem. width:auto fills
the same inset correctly.
Scope note: the app overrides that motivated this do more than full-width (they
also restyle font-size, padding, border, background and hover). Adopting the
modifier removes the width hack; whether to also drop the restyle is each app's
call, since it changes their rail appearance.
Version 0.3.1 -> 0.4.0 (additive). Build 0 errors, 48/48 tests, nupkg verified
to carry the new rule in staticwebassets/css/layout.css. Not yet published.
CLAUDE.md (Secrets row):
- mxgw is NOT Bootstrap-free (libman.json pins bootstrap@5.3.3, App.razor:7
links it ahead of ThemeHead). The scadaproj#2 modal gate had skipped mxgw on
that false premise; gate since run and PASS 2026-08-11.
- scoped-CSS bundle claim scoped to its 2026-07-19 date; OtOpcUa links one now
too, so it is two of four rather than one.
admin_ui_cleanup.md — five rounds now (adds mxgw, HistorianGateway, ScadaBridge):
- 1a-1: test sources are a fifth definition site, checked BEFORE the ghost list.
A class matched only by a test is a selector, never a ghost. Extends to
substring and visible-text assertions, so button labels are load-bearing.
- 1a-4: two inverted button-sizing instances (literal font-size neutering
btn-sm app-wide; a chassis class standing in for the button system).
- 1a-5: determine dark scheme per app; where one exists the step inverts.
- 1a-6 (new): a token override emitted as an inline style on the shell root
defeats every scheme-level :root override. OtOpcUa carries the inert-but-
latent instance.
- 1c: reword nothing before checking test sources; the docs may be the wrong
document, not the page.
- 2: btn-sm fold breaks button.btn-*.btn-sm locators; cell-clip is wrong for
resizable columns; briefs carry the rubric, not just line numbers.
- verification: baseline pre-existing failures by measuring; a suspiciously
fast green is a vacuous pass, not a pass.
- live gate: mandatory bUnit or not; hit-testability over visibility; measure
behaviour where computed values mislead (-webkit-box serializes flow-root);
record what could not be exercised.
- new: exemptions from a family-wide sweep must re-derive their premise.
Program closeout: scadaproj#4 CLOSED (comment issuecomment-5751) — ScadaBridge's
central pair now runs ONE shared SQL-Server store in Grpc mode (0.5.0
FallbackEndpoints, 0.5.1 virgin-DB migrator race fix; ScadaBridge @ e697477c),
live-gated PASS 5/5 + re-drill. The pre-Serilog container-wedge defect the gate
reproduced twice is filed as ScadaBridge#34 (pre-existing, out of program scope).
- clustered-secrets.md: central pair = one shared SQL store (connstr literal/env,
never ${secret:}); rewrap = ONE central run + both nodes of each site pair;
FallbackEndpoints same-store-only rule; 0.5.1 minimum for central pairs
- components/secrets/GAPS.md: resolution section appended; stale wait-on-#4 struck
- CLAUDE.md Secrets row: #4 CLOSED, family 0.5.1, gate + residual recorded
- tasks.json: all 6 tasks completed
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
The ScadaBridge central shared-store live gate proved SqlServerSecretsStoreMigrator's
"safe to run concurrently" claim false on a virgin database: Serializable isolation does
not serialize the IF NOT EXISTS metadata probes against another session's in-flight
CREATE, so of two central nodes booting together the loser died with SqlException 2714
("There is already an object named 'zbsecrets'") — the retry filter covered only
deadlock 1205. Losing such a race means the object now exists (the desired end state),
so 2714 (schema/table), 1913 (index) and 2627 (schema_version MERGE-insert PK) are now
retryable alongside 1205; the retry re-runs the idempotent IF NOT EXISTS path.
Classification scans the whole SqlException.Errors collection (2714 arrives stacked with
2759), the retry cap keeps genuine conflicts failing closed, and 2705 is deliberately
excluded (unreachable — columns live in a single CREATE TABLE statement). Predicate
extracted as an internal static pinned by offline tests; live suite gains an env-gated
4-concurrent-migrators virgin-database test. Six packages pushed to the Gitea feed at
0.5.1.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Joseph Doherty