Live on the box, the ScadaBridge *site* card claimed "Leader · central". Central
and site are separate Akka clusters, so that name was simply wrong.
Cause is in the registry, not the app. ResolveToInstanceName matches a reported
Akka leader address to an instance on HOST ONLY - correct by design, because an
app's Akka remoting port and its HTTP port differ, so comparing ports would never
match. It assumes what holds on the docker rig and not here: distinct hosts per
instance. All five instances are on one box, and the scan is fleet-wide in
registry order, so with every BaseUrl on "localhost" the first matching entry won
the chip for every group.
No host value makes this resolve CORRECTLY, because the instances really are
co-located. So make it never match instead: ScadaBridge's Akka addresses use host
"localhost" and OtOpcUa's uses the FQDN, and "127.0.0.1" is string-unequal to
both. Every chip now falls back to the raw Akka address - unambiguous and always
true. A raw address beats a confident wrong name.
Using the FQDN for OtOpcUa's BaseUrl would have resolved its chip correctly and
was rejected: that name resolves to a link-local IPv6 on this box, and routing
the real health probe through it to improve a label trades the primary function
for a cosmetic one.
ManagementUrl values are unchanged - they are followed by the operator's browser
from another machine and must keep the box address.
Adds appsettings.Wonder.json for the first deployment of this dashboard to a
real box. Named "Wonder" rather than "Production", mirroring the existing
Docker overlay: this is ONE deployment's fleet, not a claim about what every
production deployment looks like.
Every port was established by probing the running box, not inferred from the
docker rig - and the two disagree, so inference would have been wrong:
OtOpcUa :9000 (AdminUI on the same listener)
ScadaBridge Central :8085 (rig serves this on :5000)
ScadaBridge Site :8084 (no UI - / is 404, so the card carries no link)
MxAccessGateway :5130 (:5120 is gRPC)
HistorianGateway :5220 (:5221 is gRPC h2c)
:8083 and :8086 answer 400 to an HTTP/1.1 GET - h2c gRPC - and are excluded.
Both gateways are HasActiveRole:false. Their /health/active does return 200,
but only because the active tier runs an EMPTY check set when no Active-tagged
check is registered, so an Active badge would assert a role neither has.
Central and site are separate groups because they are separate Akka clusters;
one shared group would manufacture a leader disagreement.
BaseUrl is localhost (probed by the service, on the box) while ManagementUrl
carries the box address (followed by the operator's browser, elsewhere).
Serilog's file path is absolute and outside the Server directory an upgrade
replaces - a relative path resolves against the service AppDirectory, so every
upgrade would silently eat the log history. Same class of defect that cost
HistorianGateway its secrets store on a swap.
Boot-verified locally under ASPNETCORE_ENVIRONMENT=Wonder: registry passes
OverviewOptionsValidator and the host starts.
Family alignment ahead of the first wonder-app-vd03 deploy. Both bumps are
inert for this app, established rather than assumed:
- Theme 0.3.1 -> 0.4.1 is two commits: the upstreamed Bootstrap .btn sizing
block and the rail-btn-block modifier. Overview answers NO to both of the
questions the kit README says to ask - it declares no .btn sizing of its own
and has ZERO call sites carrying a bare `btn` token (its single button is
class="rail-btn"), so the new .btn rule matches nothing here. The .rail-btn
refactor that introduced --rail-btn-inset resolves to the same 0.6rem the
literal margin used, so the one call site is unchanged. Same answer as
HistorianGateway. Note a naive \bbtn\b grep is a FALSE POSITIVE on
"rail-btn" - the hyphen is a word boundary; match a whitespace-delimited
token instead.
- Health 0.2.0 -> 0.3.0 is a pure version-number alignment: the core
ZB.MOM.WW.Health package is byte-identical between the two, with the entire
delta confined to ZB.MOM.WW.Health.Akka (ActiveNodeHealthCheck,
ClusterActiveNode, AkkaActiveNodeGate), which this app does not reference.
Verified with a positive control on the diff path. The 0.2.1 active-tier
503 change therefore cannot reach this app's own endpoints; it registers no
Active-tagged check, so /health/active runs an empty set and stays 200.
Overview consumes the 0.3.0 `activeNode` data key over the WIRE from other
apps' health JSON, which is independent of the package it compiles against -
LeaderResolver already ranked it (344bdc4) while pinned to 0.2.0.
Release build 0 warnings; 177/177 tests pass. All remaining outdated packages
are test-only (bunit, coverlet, test SDK); nothing the app ships is behind.
A commit that removed a modal's self-closing behaviour shipped two comments in the
same file still describing it as self-closing — review checks whether the code is
right, not whether the surrounding prose survived. And the property that predicts
which modals break on refresh is 'holds no state of its own' (pure projection of a
list), not the presence of a detail field.
Readers take 'what I could not establish' as evidence the rest was held to that
standard. Write the claim the evidence licenses, not the broader one; and when two
items in your own not-established list contradict each other, flag it — that
contradiction is the strongest lead you have, not another unknown.
A bare 32-hex key id matches none of the six credential keywords (verified). Added
an entropy scan and a PEM check as generic complements, marked the prefix list
repo-specific, and made the primary rule explicit: a gate that mints anything must
grep the commit for the literal values it produced.
A filename audit finds the wrong-file problem; it cannot find a credential pasted
into a legitimately-tracked doc. Both checks are cheap and catch disjoint failures.
Verified: .claude/ is gitignored in OtOpcUa and HistorianGateway, NOT in mxgw or
ScadaBridge — in those two, enumerate-first discipline is the only guard.
A sweep's landing commit is where unrelated local state gets swept in. One repo's
'78 modified' was 76 modified + 2 untracked, one of which was .claude/ holding
settings.local.json — a literal 'git add -A' would have put permission config into
a tracked commit feeding a production deploy, invisible in a 76-file diff.
Two deadlocked investigations both inherited 'file exists => sink constructed =>
no events flowed'; the pinned sink version creates no file until the first emitted
event, so the artifact proved the opposite. And 'a second log wrote in the same
directory' rules out directory-level failure, not per-handle failure — a proof
about the container is not a proof about the member.
HistorianGateway has no local block AND no Bootstrap button call sites (all 15
buttons use the kit's .rail-btn chassis class), so the kit's .btn rule matches
nothing and bumping it changes nothing visible. My previous correction claimed the
opposite. Checking this needs two greps — declaration AND call sites — which come
apart exactly when an app doesn't use Bootstrap buttons.
Three instances in one round (btn-block universality, sweep behavior-preservation,
the mxgw Bootstrap exemption), all the same shape, all caught by the asserter
re-checking rather than by challenge. Fix is a cheap per-member check that runs,
or an honestly-scoped claim when it hasn't.
HistorianGateway has no local .btn sizing in either its committed tree or its
sweep branch (only vendored bootstrap.min.css), so the shipped comment and README
naming all five consumers were wrong. Consequence worth stating: HG's bump to
0.4.0+ is a VISIBLE change (buttons shrink to .85rem), not the no-op cleanup it is
for consumers that already carried the block.
The prior wording claimed the two rules 'cannot drift apart', which is only true
for the documented composition. 0.6rem still appears twice; the fallback confines
a stale copy to the undocumented modifier-alone path. Records why hoisting the
var to :root would be worse rather than better.
Unreleased — behaviourally identical to the published 0.4.1 (browser-verified:
composed <a> and <button> both fill the rail foot exactly, 0.00px short). The
var() carries a 0.6rem fallback because --rail-btn-inset is declared on .rail-btn,
so rail-btn-block used alone would otherwise have an undefined var, invalidating
the whole width declaration at computed-value time and silently reverting to
width:auto — the exact <button> shrink-wrap bug 0.4.1 fixed.
Two lessons from the Theme 0.4.0 rollout across five apps.
Live gate: after a kit-CSS change a normal reload can serve the cached previous
layout.css while the server serves the new one. .btn then computes at
Bootstrap's 16px — the exact signature of a failed upstreaming — so the natural
response is to restore the local rule just deleted, entrenching the duplication
the change removed. Hard-reload before trusting any computed-style probe.
Upstreaming: the kit ships sizing only, so trim the local .btn rather than
deleting it. mxgw's block also carried border-radius/font-weight/white-space,
which a wholesale delete would have dropped silently; three of five apps' blocks
were pure sizing and safe to delete outright, and the difference is invisible
without diffing. Also records why a retained border-radius must stay a literal.
Marks the .btn block as upstreamed in 0.4.0 rather than a candidate.
0.4.0's rail-btn-block used display:block + width:auto, which fills for an <a>
but SHRINK-WRAPS a <button> — a form control does not take the fill-available
behaviour a non-replaced block box gets. Measured in one 197px rail-foot
container: <a> filled to 177.42px (0 short), <button> came up 114px short.
That matters because a rail foot normally holds both element types: Sign In is
an anchor, Sign Out is a form-submit button. So 0.4.0's modifier got one right
and one wrong, and the wrong one is what signed-in operators see — worse than
the width:100% overflow it was meant to replace (~10px of visible clipping).
Fixed with an explicit calc(100% - 1.2rem) + box-sizing:border-box, which
measures 0px short for both element types. Both rejected shorthands and their
measured failures are recorded in the CSS so neither is reintroduced as a
'simplification'.
Found by HistorianGateway while evaluating adoption; it held off adopting
rather than trading one defect for a larger one. No consumer had adopted the
modifier, so nothing shipped broken. The .btn half of 0.4.0 is unaffected and
is already live-verified in three apps.
Build 0 errors, 48/48 tests.
Bootstrap hard-codes --bs-btn-font-size: 1rem, oversized against the kit's
0.9rem body. All five consuming apps had independently written the identical
block into their own stylesheets (ignitionoee app.css, OtOpcUa site.css, mxgw
site.css, HistorianGateway site.css, ScadaBridge site.css), each carrying a
'delete this when the kit ships a .btn rule' comment. This is that release.
Sized by CSS-variable override only, never by redeclaring box properties:
Bootstrap's .btn-group seam/radius machinery reads the same variables, so a
literal font-size at equal specificity flattens .btn-sm into a padding-only
difference. That was a real defect in mxgw's local copy, so the rationale is
recorded in the CSS to stop it being 'simplified' back.
Lands in layout.css, which ThemeHead emits after theme.css and both after
Bootstrap, so the override wins on source order at equal specificity.
Ships in the same 0.4.0 release as rail-btn-block. Consumers get the sizing on
bump (not opt-in, unlike rail-btn-block) and should delete their local copies
in the same change. Build 0 errors, 48/48 tests, packed asset verified to carry
.btn, .btn-sm/.btn-group-sm and rail-btn-block with no literal font-size.
HistorianGateway hand-rolled a .rail-btn override purely to make the rail-foot
sign-out button span the rail; its own comment flagged folding it into the kit.
Adds a composable modifier instead: class="rail-btn rail-btn-block".
Uses display:block + width:auto, NOT width:100%. The hand-rolled overrides set
width:100%, which resolves against the containing block and then adds .rail-btn's
0 0.6rem side margins on top, overflowing the rail by 1.2rem. width:auto fills
the same inset correctly.
Scope note: the app overrides that motivated this do more than full-width (they
also restyle font-size, padding, border, background and hover). Adopting the
modifier removes the width hack; whether to also drop the restyle is each app's
call, since it changes their rail appearance.
Version 0.3.1 -> 0.4.0 (additive). Build 0 errors, 48/48 tests, nupkg verified
to carry the new rule in staticwebassets/css/layout.css. Not yet published.
CLAUDE.md (Secrets row):
- mxgw is NOT Bootstrap-free (libman.json pins bootstrap@5.3.3, App.razor:7
links it ahead of ThemeHead). The scadaproj#2 modal gate had skipped mxgw on
that false premise; gate since run and PASS 2026-08-11.
- scoped-CSS bundle claim scoped to its 2026-07-19 date; OtOpcUa links one now
too, so it is two of four rather than one.
admin_ui_cleanup.md — five rounds now (adds mxgw, HistorianGateway, ScadaBridge):
- 1a-1: test sources are a fifth definition site, checked BEFORE the ghost list.
A class matched only by a test is a selector, never a ghost. Extends to
substring and visible-text assertions, so button labels are load-bearing.
- 1a-4: two inverted button-sizing instances (literal font-size neutering
btn-sm app-wide; a chassis class standing in for the button system).
- 1a-5: determine dark scheme per app; where one exists the step inverts.
- 1a-6 (new): a token override emitted as an inline style on the shell root
defeats every scheme-level :root override. OtOpcUa carries the inert-but-
latent instance.
- 1c: reword nothing before checking test sources; the docs may be the wrong
document, not the page.
- 2: btn-sm fold breaks button.btn-*.btn-sm locators; cell-clip is wrong for
resizable columns; briefs carry the rubric, not just line numbers.
- verification: baseline pre-existing failures by measuring; a suspiciously
fast green is a vacuous pass, not a pass.
- live gate: mandatory bUnit or not; hit-testability over visibility; measure
behaviour where computed values mislead (-webkit-box serializes flow-root);
record what could not be exercised.
- new: exemptions from a family-wide sweep must re-derive their premise.
Program closeout: scadaproj#4 CLOSED (comment issuecomment-5751) — ScadaBridge's
central pair now runs ONE shared SQL-Server store in Grpc mode (0.5.0
FallbackEndpoints, 0.5.1 virgin-DB migrator race fix; ScadaBridge @ e697477c),
live-gated PASS 5/5 + re-drill. The pre-Serilog container-wedge defect the gate
reproduced twice is filed as ScadaBridge#34 (pre-existing, out of program scope).
- clustered-secrets.md: central pair = one shared SQL store (connstr literal/env,
never ${secret:}); rewrap = ONE central run + both nodes of each site pair;
FallbackEndpoints same-store-only rule; 0.5.1 minimum for central pairs
- components/secrets/GAPS.md: resolution section appended; stale wait-on-#4 struck
- CLAUDE.md Secrets row: #4 CLOSED, family 0.5.1, gate + residual recorded
- tasks.json: all 6 tasks completed
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
The ScadaBridge central shared-store live gate proved SqlServerSecretsStoreMigrator's
"safe to run concurrently" claim false on a virgin database: Serializable isolation does
not serialize the IF NOT EXISTS metadata probes against another session's in-flight
CREATE, so of two central nodes booting together the loser died with SqlException 2714
("There is already an object named 'zbsecrets'") — the retry filter covered only
deadlock 1205. Losing such a race means the object now exists (the desired end state),
so 2714 (schema/table), 1913 (index) and 2627 (schema_version MERGE-insert PK) are now
retryable alongside 1205; the retry re-runs the idempotent IF NOT EXISTS path.
Classification scans the whole SqlException.Errors collection (2714 arrives stacked with
2759), the retry cap keeps genuine conflicts failing closed, and 2705 is deliberately
excluded (unreachable — columns live in a single CREATE TABLE statement). Predicate
extracted as an internal static pinned by offline tests; live suite gains an env-gated
4-concurrent-migrators virgin-database test. Six packages pushed to the Gitea feed at
0.5.1.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Bump ZB.MOM.WW.Secrets family 0.4.1 -> 0.5.0 and date the CHANGELOG
entry. All 6 packages (Abstractions, Secrets, Replicator.SqlServer,
Replicator.AkkaDotNet, Replicator.Grpc, Ui) packed and pushed to the
dohertj2-gitea feed; restore-verified from a scratch consumer with a
fresh package cache. Suite 470 passed / 0 failed / 15 skipped.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Review follow-ups: the warning now carries only the two endpoint URIs and the
status code — attaching the exception object would let any real sink render
its ToString(), which for RpcException includes server-authored Status.Detail.
The leak pin test now also asserts no exception rides on the warning, via a
new WarningExceptions capture on RecordingLogger. Plus two cosmetic nits:
unified ThrowIfNull idiom, documented that FallbackEndpoints mutation after
Validate is unsupported.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
SecretsGrpcHubClientOptions gains a getter-only FallbackEndpoints list:
additional hub addresses tried in order when a call fails with a
transport-level RpcException. Validation fails startup on blank/relative
entries and on any duplicate of the primary or another fallback (a
duplicated endpoint silently halves the configured failover).
New FailoverSecretsHubReader composes one GrpcSecretsHubClient per
endpoint: the answering endpoint is sticky (a sweep is GetManifest + N
GetSecrets batches — without stickiness every call after a primary
outage re-pays a full CallDeadline before reaching the healthy
fallback), rotation wraps so a recovered primary is retried when the
stuck-to endpoint fails, only RpcException triggers failover
(cancellation and defects propagate untouched), and when every endpoint
fails the last RpcException is rethrown so the sweep degrades to
last-known-good exactly as today. Failover warnings name only the two
endpoint URIs and the status code — never any credential material.
DI: zero fallbacks keeps the registration graph byte-identical to 0.4.x
(single keyed channel under ChannelServiceKey, plain
GrpcSecretsHubClient); with fallbacks, one container-owned keyed channel
per endpoint (fallback i under "{ChannelServiceKey}:fallback:{i}") wired
into the failover reader. GrpcSecretsHubClient itself is unchanged.
Safe only when every listed endpoint serves the SAME store — for
ScadaBridge, the central pair over one shared SQL Server store; the
options doc and CHANGELOG carry the warning.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Task 9 (closeout) of the Secrets gRPC-hub program. Bookkeeping only; no code.
- clustered-secrets.md, Production enablement: the gRPC hub is no longer
pending. It is BUILT, SHIPPED (0.4.1) and LIVE-GATED PASS 4/4 (ScadaBridge
main @ f6c3f7c5, docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md), so the
"until that ships, ScadaBridge replication stays off" line is replaced with
the truth: the code path is merged and gated, and production enablement is a
separate operator action behind a still-default-off flag. OtOpcUa's
pair-local topology is recorded as live-gated (lmxopcua master @ 1bd79c8e)
rather than pending re-gate. Rotation fan-out corrected: on ScadaBridge it is
BOTH central nodes plus both nodes of every site pair, because each central
node has its own independent store.
- components/secrets/GAPS.md: new "gRPC hub delivered" section recording the
0.4.0/0.4.1 publish, both gates with SHAs and gate-doc paths, and the
residual. Existing narrative left intact and amended, not rewritten.
- CLAUDE.md Secrets row: lib version 0.3.0 -> 0.4.1 and the app-pin claim
corrected (ScadaBridge pins 0.4.1 for the Grpc replicator; the other three
still pin 0.2.3); the "production topologies DECIDED" region now records
delivery, both gates, default-off, and the residual.
- Plan ledger: task 9 completed.
Residual carried forward as scadaproj#4 — ScadaBridge's central pair does not
converge with itself (central-b served an authenticated but EMPTY manifest all
gate while central-a held every secret) and the site client dials a single
endpoint; these are one design question, since failover to an empty hub would
succeed and silently stop convergence. scadaproj#3 is closed.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
The ScadaBridge live gate (check 4) found SecretsHubAuthInterceptor denying
correctly but logging nothing: the only trace of a refusal was a framework
Grpc.AspNetCore.Server Information line. A follower whose bearer token was
mis-rotated therefore stopped converging silently, with the hub showing it
only at INF.
Every denial is now counted and summarised in a Warning emitted on the first
denial after startup or after a 60-second quiet window, carrying the denials
since the previous summary plus a breakdown by cause. A single probe is
visible immediately; a sustained flood is bounded to one line per window,
which is what makes warning safe at all — the endpoint is unauthenticated by
construction, so a per-denial warning would be an unbounded log write driven
by an unauthenticated caller. Counters are Interlocked and the emit is
decided by a compare-exchange on the last-emit timestamp, so exactly one
thread emits per window and a denial counted after the drain carries into
the next summary rather than being lost.
The cause breakdown is server-side diagnostics only. Denied() is untouched:
every rejection still returns the identical uniform Unauthenticated status,
the comparison is still constant-time, and no token material appears in any
log line. The existing warn-once-on-unconfigured-token signal is unchanged
and still emitted exactly once; denials in that state count toward the new
summary's "unconfigured" category.
TimeProvider is an optional default-valued parameter rather than a second
overload — the type must keep exactly one public constructor or the gRPC
interceptor activator silently stops invoking the auth gate.
Two existing tests scoped their assertions to the configuration warning,
since both kinds now share a category: the new line is deliberate behavior,
not drift.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Bumps the ZB.MOM.WW.Secrets family from 0.3.0 to 0.4.0 and adds the new
ZB.MOM.WW.Secrets.Replicator.Grpc package to the packable set. Publishes
all six 0.4.0 nupkgs to the dohertj2-gitea feed and restore-verifies the
new package resolves from it in a fresh --packages directory. Existing
packages are behaviorally unchanged — no breaking changes.
Adds ZB.MOM.WW.Secrets/CHANGELOG.md (none existed previously) to record
this and prior releases, and lists the new package in README.md's
Packages table. Also includes the controller's pending ledger updates
for docs/plans/2026-08-07-secrets-grpc-hub-implementation.md (Tasks
1-4 and 8 marked completed).
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
An internal validateEagerly seam on AddZbSecretsGrpcHub replaces the harness's
hand-copied registration fragment, so the deny-all second-layer tests exercise
the production wiring and cannot drift from it.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Joseph Doherty