Commit Graph

511 Commits

Author SHA1 Message Date
Joseph Doherty 522cfbfda6 fix(localdb): flush the HLC anchor on deregister, bound the sync inbox (0.2.1)
Two adversarial-review findings against 0.2.0. Wire-compatible with 0.1.x/0.2.0 peers:
no proto change, no lib_schema_version bump, no on-disk schema change.

F4 — DeregisterReplicated destroyed the HLC clock's only durable crash anchor. Clock
recovery on open is MAX(hlc_high_water, MAX(oplog.hlc), MAX(row_version.hlc),
last_seen_hlc); hlc_high_water is written only by Dispose, and deregistration deletes the
table's rows from BOTH ledgers. On a file whose registered tables have all been
deregistered, an ungraceful exit (crash, SIGKILL, power loss) therefore left nothing above
zero to recover from: the reopened clock restarts from wall-clock time and can re-issue
stamps below HLCs peers already hold, which LWW discards as older. The current clock is now
flushed inside the deregistration transaction, before the prune, as a separate command so
it cannot inflate the "did anything change?" answer. FlushHlcHighWater takes an optional
transaction (SQLite rejects a command with no transaction on a connection that has one) and
is now MAX-guarded — every writer of that column is flushing a monotonic clock, so a lower
value can only be stale.

F3 — the sync inbox was Channel.CreateUnbounded, making inbound memory a function of what
the PEER sends. The bad case is not hypothetical: a snapshot streams every row of every
registered table, needs_snapshot is per-database, and both sides of a pair that enables
replication late baseline unconditionally — so a mutual full snapshot of a large dataset
(an event log near its storage cap) was buffered whole, in memory, on both nodes at once.
The inbox is now bounded at 64 messages with wait-mode backpressure, so the ceiling is
~64 x MaxBatchBytes — a configured number rather than a peer-controlled one — and a full
inbox stops draining the gRPC stream, which closes the peer's flow-control window. The
pump writes with WriteAsync, not TryWrite: on a full bounded channel TryWrite drops the
message, and every inbound message is protocol state.

Bounding alone would have deadlocked the very case it was aimed at. The snapshot a node
owes was streamed from INSIDE the handshake, before the receive loop existed, so neither
side drained its inbox while sending; a mutual snapshot converged only because the inbox
was unbounded. The snapshot now runs as the first phase of the outbound task, started
alongside the receive loop, so the inbox is drained for the whole life of the session.
Interleaving needed no new safety: snapshot rows and deltas both merge through LWW and the
snapshot's watermark write is MAX-guarded. Ordering within the outbound lane is preserved
(the pump still starts only after the snapshot, from its as-of seq).

Per-table needs_snapshot (the finding's optional third part) is NOT done: the flag lives in
__localdb_peer_state and narrowing it needs an on-disk schema change, which this fix
deliberately avoids. Documented as a follow-up, and the cost is now written down where it
is decided — RegisterReplicated(baselineExistingRows) remarks and the README both say that
baselining one table re-streams every registered table in both directions, and that
high-volume tables should be trimmed before enabling replication late.

188/188 tests pass (was 184). New coverage: the deregistered clock anchor surviving an
ungraceful reopen (a second instance over the same file with no Dispose — fails without the
flush, recovering at 0), the flush not counting as work done, bounded-inbox backpressure
under a slow reader (the pump stalls at the bound and loses nothing, proven by contiguous
in-order delivery), and a mutual snapshot over a capacity-1 transport converging — that
last one deadlocks and times out against the old handshake-phase snapshot ordering.

Version 0.2.1.
2026-08-14 23:34:55 -04:00
Joseph Doherty 1ce0c5955f docs: umbrella index — LocalDb 0.2.0 + ScadaBridge consumption 2026-08-14 22:21:41 -04:00
Joseph Doherty 9377fa1678 perf(localdb): dereg API, O(1) backlog depth, byte-budget batching, prepare-once apply (0.2.0)
WP3.3 of the ScadaBridge architectural-review remediation. Six changes, all
wire-compatible with 0.1.x peers — no proto change, no lib_schema_version bump. The
on-disk bookkeeping schema goes to v2 and is upgraded in place when a file is opened.

DeregisterReplicated(table). There was no way to stop capturing once a file had been
registered, so a node that does not replicate kept paying three triggers inside every
write transaction, appending to an oplog nothing would ever drain. Drops the triggers and
prunes the table's oplog + row-version rows in one transaction; idempotent, and
deliberately usable on a table this process never registered, since stale triggers left by
an EARLIER build are the primary case. Deregistration must be symmetric across a pair, and
already is enforced: the handshake compares registered-table digests fail-closed, so a node
that drops a table its peer still replicates stops syncing loudly rather than diverging.

RegisterReplicated(table, baselineExistingRows: true). Closes the late-opt-in hole: the
snapshot streamer pages from __localdb_row_version, which only triggers populate, so a node
that started replicating LATE never sent its pre-existing rows — silently, forever. Seeding
alone would not have fixed it, because a node whose oplog starts at seq 1 presents no gap at
the handshake and is told it needs no snapshot; the seed therefore also flags needs_snapshot,
and the existing resync machinery carries the rows. The floor is HLC 0 stamped with this
node's id: every genuine HLC is a UTC millisecond shifted left 16 bits and so is strictly
greater, which makes a baselined row lose every comparison against a real remote write and
win only where the peer has no version of that key at all. Two nodes baselining the same key
both hold 0 and resolve by the node-id tie-break — convergent, and nothing better is
available for two rows of unordered antiquity. Opt-in, so the documented CDC contract is
unchanged unless asked. Pruning the ledger on deregister is safe precisely because
re-registration can re-baseline; the two APIs are a pair.

O(1) backlog depth. GetOplogDepth ran COUNT(*) over the entire unacked backlog, once a
minute from maintenance AND on every metrics scrape — a full scan of a million rows exactly
when a node is already behind. Now MAX(seq) - MAX(last_acked_seq, MIN(seq) - 1), floored at
0: three b-tree seeks, measured at 58 VM steps against 100,020 for the count on a 50k
backlog. Exact, because seq is AUTOINCREMENT and every prune path removes a prefix (ack
pruning by watermark, the row cap by count, the age cap by an HLC cutoff that rises with
seq). The MIN clamp is what keeps it exact after a cap prune has moved the horizon above a
stalled watermark — without it the caps would re-flag on every tick forever.

Byte-budget batching. MaxBatchSize was the only cap on an outbound message, which made it a
proxy for a byte budget that consumers had to tune against their widest row: ScadaBridge
pinned it to 16 because 70 KB config_json times the 500 default is ~35 MB against a 4 MB
gRPC limit. MaxBatchBytes (default 2 MB) now bounds delta and snapshot messages by summed
serialized size, with the row count demoted to a secondary cap. A row larger than the budget
is sent alone rather than stalling the stream on it. Since more, smaller messages would
otherwise mean more apply transactions and more acks, inbound delta batches and acks ALREADY
queued are coalesced — one transaction and one ack per burst, bounded by the sender's own
caps, and a no-op under a trickle.

LwwApplier prepares each statement once. It went through the generic transaction façade,
which builds a fresh SqliteCommand per call: three SQL parses per applied row, on the path
that drains a reconnect backlog. Statements are now held for the batch and rebound.

Dropped __localdb_oplog_hlc. It indexed (table_name, pk_json, hlc), which no query has ever
used — every oplog access path is by seq or a full hlc-predicate delete — while costing a
b-tree insert inside every captured consumer write.

184/184 tests pass (was 149). New coverage: schema upgrade drops the index and preserves
identity/watermarks/data, deregistration idempotency and trigger absence and cross-process
cleanup, baseline seeding at the floor with LWW safety proved through the real applier,
depth asserted equal to the counting query across every prune scenario plus a constant-time
assertion over a 200k backlog, byte-budget chunking including the one-giant-row edge on both
the delta and snapshot paths, prepare-once statement counting, and coalescing that must not
swallow the message that ends its run.

Version 0.2.0.
2026-08-14 22:08:43 -04:00
Joseph Doherty 92246c9734 docs(inventory): full QA §7.1 refresh for the 2026-08-14 VQ04 singleton clone — 10-instance fleet, psbridge :6150 + glauth :3893 port rows, VQ04→VQ03 Galaxy/alarm dependency; CLAUDE.md Overview note updated 2026-08-14 12:39:14 -04:00
Joseph Doherty 1fee9755d7 docs(deployment): VQ04 now runs its own MxGateway/HistorianGateway/Overview (cloned from VQ03 2026-08-14, shared API-key stores); QA access = psbridge :6150, servecli retired 2026-08-14 12:35:56 -04:00
Joseph Doherty 205bddb3b8 docs(plan): flag ignitionoee LDAP as ON HOLD -- needs new UG_WW_* groups minted first 2026-08-13 15:01:15 -04:00
Joseph Doherty 206c588608 docs(plans): VD03 NAM AD LDAP cutover plan (not executed) 2026-08-13 09:24:17 -04:00
Joseph Doherty bd929d98fc docs(ldap): e2e NAM AD login verified via HG docker -- svc_igntadmin, stage diagnostics 2026-08-13 09:17:42 -04:00
Joseph Doherty 990f4c018d docs: HG e42ff31 is on main, not a branch 2026-08-13 08:58:21 -04:00
Joseph Doherty 5da4ab3ebd docs(index): HistorianGateway UI-sweep branch merged to main (e42ff31) — Theme 0.4.1 + Health 0.3.0 + Secrets 0.6.2 + Auth 0.2.1 now on its default 2026-08-13 08:58:09 -04:00
Joseph Doherty 3d2f3f59b2 docs(index): Auth 0.2.1 adoption commits pushed on all four apps 2026-08-13 08:52:25 -04:00
Joseph Doherty 58c2e14c38 docs(index): Auth 0.2.1 adopted by all four apps; FallbackServers live-gated on HG + ScadaBridge rig 2026-08-13 08:49:25 -04:00
Joseph Doherty 58f605609d feat(infra): redundant GLAuth pair for FallbackServers failover drills
Two local glauth containers (zb-ldap-primary :3893 / zb-ldap-backup :3894)
simulating the NAM two-DC topology, so the primary can be killed during a
drill without touching the shared 10.100.0.35 instance. Drill recipe +
verified-PASS record (HistorianGateway, 2026-08-13) in the README.
2026-08-13 08:28:05 -04:00
Joseph Doherty f42aa14033 docs(inventory): add the QA stack (vq03/vq04/sql-vq03) and correct vd03 DEV-vs-prod
deployment.md:
- §1/§2.1: add wonder-app-vq03, wonder-app-vq04 and wonder-sql-vq03 with IPs and
  per-host roles; new §7.1 QA port map + live active/standby state.
- QA is a real 2-node HA pair (ScadaBridge central+site pairs, OtOpcUa pair) where
  DEV is a single all-in-one box, so it is the only environment here that exercises
  failover and leader election. MxGateway/HistorianGateway/Overview run on vq03 only.
- Every QA fact is marked scan (port probe) or dash (read off the QA Overview
  dashboard's live polling). No QA config file was read; connection strings and
  service names stay unverified.
- Correct the tier: vd03 is DEV, not production (vd=dev, vq=qa, per
  mes-delmia-integration-api.md §1.1). Its ASPNETCORE_ENVIRONMENT really is
  "Production" -- a .NET overlay name, orthogonal to the tier, and the likely source
  of the original error. No production box has been identified.
- Record that QA requires Basic auth on /management where DEV does not, so DEV-only
  procedures relying on unauthenticated management calls will not work on QA.
- Record that inbound API keys never cross environments (per-stack peppered-HMAC,
  excluded from config bundles) plus the 401-vs-400 probe, with the caveat that auth
  runs before route resolution so a 401 needs a positive control.
- Withdraw the SSH-on-2222 claim made earlier in this same pass: all three wonder
  boxes accept TCP on 2222 and send no SSH banner, so servecli is not serving SSH and
  no shell access was available. Port scan cannot establish a service; banner grab can.

CLAUDE.md:
- Overview: record the second instance on wonder-app-vq03:5320, and that its real-FQDN
  registry is correct rather than a divergence -- the 127.0.0.1 rule is specific to
  vd03 co-locating every instance on one host.
- Add the DEV-vs-prod note pointing at deployment.md §2.1/§7.1.
2026-08-13 08:08:23 -04:00
Joseph Doherty 818775b311 docs(index): Auth 0.2.1 published -- AD referral fix + NAM group findings
CLAUDE.md: feed now carries 0.2.1; record the referral defect and that every
app is still exposed until it bumps off 0.1.5 (a functional fix, not
housekeeping), plus the new tools/LdapProbe.

ldap_setup.md: findings from probing the live NAM directory --

- LDAPS 636, StartTLS 389 and plain 389 all connect on both DCs. The only TLS
  chain error is UntrustedRoot, so installing zimmer-rootca is sufficient; the
  RSA 1024-bit key is rejected only by OpenSSL at SECLEVEL=2, i.e. Debian .NET
  containers, not by macOS or Windows hosts.
- The UG_WW_* Wonderware group convention, 39 groups enumerated directory-wide.
  This partly overturns this doc's earlier "no group->role convention exists"
  claim: that was true of Ignition, which was the only source audited, and
  false of the domain. Three shapes with a consistent role suffix (ADMIN, RO,
  OPERATOR, ENGINEER, SUPERVISOR, TESTER) across environment and site axes.
- A proposed mapping onto CANONICAL-ROLES, with the three gaps left open
  rather than guessed: SUPERVISOR/TESTER have no canonical equivalent, nothing
  maps to Deployer, and the DEV/QA/PROD axis is orthogonal to role so it must
  ride in the per-project scope payload or the role set triples.
- The referral defect and its fix.
2026-08-12 21:27:23 -04:00
Joseph Doherty 4e01cbc2c3 feat(tools): ldapprobe -- NAM AD auth + group discovery probe
Diagnostic for pinning down the LDAP settings the four ZB.MOM.WW apps need
against the Zimmer NAM domain, and for answering which AD groups can be mapped
to app roles. Found the referral defect fixed in 0.2.1.

Two independent halves in one run:

Part A drives the REAL ZB.MOM.WW.Auth.LdapAuthService -- same bind-then-search-
then-bind, same options validator, same failure enum -- so its verdict IS what
the apps produce rather than an approximation of it.

Part B asks the directory what the library cannot, which is where the group
answer comes from: full group DNs, nested groups via
LDAP_MATCHING_RULE_IN_CHAIN, the primary group (AD omits it from memberOf, so
it can never be a role group), security vs distribution type, decoded
userAccountControl, referral counts, and the TLS chain with key sizes.
--list-groups '<pat>' enumerates every matching group in the directory, which
a membership dump structurally cannot do -- it only ever shows one account's
memberships.

Passwords are prompted for and never echoed; they cannot be passed as
arguments, which would put them in shell history. --self-bind lets the user
under test stand in for the service account, so group discovery does not
require the svc_igntnapp secret.

Not packable, outside the Auth solution and its central package management,
referencing the Auth projects directly so it tracks current source.
2026-08-12 21:27:14 -04:00
Joseph Doherty 1f8ab8e3b3 fix(auth): skip AD continuation referrals in LDAP search (0.2.1)
Active Directory answers any search based at a domain root with continuation
references (CN=Configuration, DC=ForestDnsZones, ...). With referral following
disabled -- the default, and what we want, since chasing one would re-bind
somewhere we never authenticated -- Novell surfaces each by throwing
LdapReferralException from Next().

That escaped NovellLdapConnection.Search, and LdapAuthService maps a failed
search to ServiceAccountBindFailed. So every login against
SearchBase = DC=nam,DC=zimmer,DC=com failed system-side on BOTH DCs in ~546 ms
while the directory, the credentials and the service account were all healthy.
Identical on each endpoint, so failover could not help either.

A continuation reference is a pointer to another naming context, never a
statement that this search failed, so the loop now skips them and continues --
returning exactly the entries in the naming context we bound to. Extracted as
ReadEntries(ILdapSearchResults) so the handling is unit-testable through the
Novell seam without a live directory. A 1000-referral cap guards a client that
throws without advancing, which would otherwise spin forever inside a
fail-closed auth path.

Non-referral faults still propagate untouched: swallowing those would turn
"the directory is broken" into "no such user", which is the wrong answer to
give about someone's credentials.

7 tests, 6 verified red against the pre-fix code; the seventh is the
non-referral control that must stay green either way. Suite 257/0, 2 skipped.

Live-proven against the ZB NAM domain root before and after.
2026-08-12 21:27:05 -04:00
Joseph Doherty 846b77f5b4 feat(overview): show historian session-pool depths on the HistorianGateway card
Asked whether the gateway's new session row also covered HistorianGateway.
It did not, and could not: HistorianGateway publishes its pool depths on a
different TIER with a different KEY SET.

  mxaccessgw       active tier   total / ready / faulted        one check
  historiangw      READY  tier   warmSessions / pool / …        TWO checks

The session row is located by the total/ready/faulted triple, so it correctly
ignored these rather than half-matching them. This adds a sibling row instead
of unifying the two behind a shared shape, because there is no such shape -
inventing one would mean rendering a contract neither app publishes.

No poller change was needed: the ready tier's body is already retained, so
this is entirely a rendering change.

Labelled "Sessions pooled", deliberately not "connections". warmSessions is
pool DEPTH - warm pre-authenticated sessions held ready - and sessions are
leased on demand, so the event pool legitimately sits at 0 until something
reads events. Calling that a connection count would make a normal idle pool
read as an outage, so 0 is shown plainly and the row goes red only on
consecutiveFactoryFailures, which is an actual fault.

Pools render read-then-event rather than in dictionary order, so the line is
stable across polls and the pool an operator cares about comes first.

188 tests (183 + 5): ordering, that an idle pool is not painted red, that a
failing factory is, a positive control on the omission assertion, and one
pinning that neither app's row can render the other's data.
2026-08-12 05:16:25 -04:00
Joseph Doherty da276ff15a feat(overview): show MXAccess session counts on the gateway card
The gateway publishes its live MXAccess session counts, and the dashboard
could not reach them. Four things stood between the data and the card, and
only the last one is cosmetic:

The active tier was probed only for instances declared HasActiveRole. A
standalone gateway has no active/standby role, so its active tier was never
requested at all. The tempting fix is to declare mxgw HasActiveRole, and it
is wrong: that flag also paints an Active/Standby badge, so it would buy the
data by asserting a role that does not exist. The probe is now unconditional
for any reachable instance while ObserveActive stays role-gated, separating
"what does this tier report" from "am I the active half".

The active probe's parsed body was then discarded - only ready.Report was
kept - so InstanceSnapshot gains ActiveReport. The two tiers stay separate
rather than merged, because a check name is only unique within a tier.

The card's one data-rendering path, ClusterDataLine, is located by the Akka
leader/activeNode keys and would ignore session data even once present. The
new row follows that helper's own precedent and locates the check by the data
it carries - the total/ready/faulted triple - not by name, requiring all three
so a future check publishing a lone "total" cannot be mistaken for it.

Rendered on the card FACE, not inside the collapsed check list: a count behind
a disclosure triangle is one nobody reads. Zero is shown rather than hidden,
since a gateway holding no sessions is the interesting case and suppressing it
would make the row appear only when the news is good.

Cost, stated plainly: every reachable instance now makes one extra HTTP
request per poll. The unreachable-instance skip is untouched, so the
worst-case sweep duration - which is set by timeouts, not by healthy
round-trips - is unchanged. Four cadence tests were counting raw requests as a
proxy for "instances polled"; they now count ready-tier probes, which is what
they always meant and is robust to tier changes.

The Wonder registry comment claiming the active tier ran an empty check set
was true when written and stopped being true when the gateway added the
check; it now warns against the HasActiveRole shortcut instead.

183 tests pass (177 + 6), including a positive control on the absence
assertion so "row not rendered" cannot pass vacuously.
2026-08-12 04:47:06 -04:00
Joseph Doherty 81e0005628 docs(index): OtOpcUa on 0.6.2; the NuGet partial cached-index trap recurs per publish 2026-08-11 09:09:05 -04:00
Joseph Doherty b850c3a47f docs(index): Secrets 0.6.2 — rejection message teaches the container case 2026-08-11 09:04:55 -04:00
Joseph Doherty c5e894d9b3 feat(secrets): 0.6.2 — the content-root rejection message teaches the container case
Suggested by the OtOpcUa session, the only real containerized deployment in this family
and the one that hit the false positive. A store on a volume mounted inside the content
root is durable across container replacement but is rejected anyway, because the rule
cannot tell a mount from an ordinary directory. The message now says so, gives the fix,
and forecloses the wrong one by stating that no override exists.

A false positive's entire cost is the confusion in the moment, so an error that names its
own fix costs nothing on the true-positive path and turns the false positive into a
two-minute edit. Message only, no behaviour change.

Their case for keeping the rule strict is recorded in the CHANGELOG because it is better
than the trade-off I had written: the workaround is an improvement rather than a
concession, an override flag has the lifecycle of a knob set once and never revisited,
and mount detection would make the verdict depend on runtime platform state so the same
path would pass on one host and fail on another.
2026-08-11 09:04:02 -04:00
Joseph Doherty 74066bbe27 docs(secrets): record the container over-rejection and the OtOpcUa 0.6.1 adoption 2026-08-11 09:01:04 -04:00
Joseph Doherty 6084465e23 test(secrets): the positive control was a tautology propping up the negative one
Raised by the mxaccessgw session, which went back to its own tests after I reported a
vacuous assertion in mine and found the same class one step further from the fuse.

The rejecting test asserts no store file is left at a rejected path. That claim is only
meaningful if something proves the write path is live at all: "no file was created" is
indistinguishable from "nothing in this container ever creates files". The positive
control was supposed to establish that and did not -- it asserted Assert.NotNull on a
resolved SecretsSqliteConnectionFactory, which is close to a tautology once no exception
escaped, and it never ran the migration at all. So if the pre-host container ever stopped
touching the store, the rejecting test would have gone quietly vacuous while the positive
control kept passing. Neither would have failed.

The positive control now runs SqliteSecretsStoreMigrator.MigrateAsync and asserts the
database IS created. The pair is mutually checking: the migration writes the file when
allowed and does not when rejected, and neither assertion stands on its own.

Verified the same way as the last one, by simulating the failure rather than reasoning
about it: replacing the migration with a no-op makes the positive control fail with "the
accepted path must actually get a store file". Reverted after measuring.

No library change; 0.6.1 as published is unaffected. Suite 502 passing.
2026-08-11 08:56:46 -04:00
Joseph Doherty d6fe3ae62a test(secrets): the pre-host regression test was asserting the weaker claim, and vacuously
Two defects in the 0.6.1 regression test, the first raised by the mxaccessgw session
from its own adoption, the second exposed by acting on it.

Assertion order. The test asserted the throw before the artifact. That hides the
stronger finding: a regression that stops the throw but still creates the store fails
with "no exception was thrown" and never mentions the database sitting at the rejected
path -- which IS the defect. Now captures with Record.ExceptionAsync, asserts no file
created first, and asserts the exception type last. Notable because the defect under
test was itself "a boot outcome was observed instead of the artifact being probed", so
closing it with an outcome-first assertion would have repeated the mistake.

Vacuous assertions. Simulating the regression -- replacing the 4-argument call with the
3-argument one, which still compiles, exactly as the call-site comments warn -- showed
the test failing on the exception assertion rather than the file assertion. No file had
been created in EITHER case: the test resolved SecretsSqliteConnectionFactory, whose
constructor never touches the filesystem. The File.Exists assertions could not fail. The
test now resolves SqliteSecretsStoreMigrator and runs MigrateAsync, which is what a
pre-host expander actually does and what actually creates the database.

Verified by simulating the regression again: the test now fails with "a rejected path
must not be left holding a store file" rather than "no exception was thrown". Reverted
after measuring.

No library change; 0.6.1 as published is unaffected. Suite 502 passing.
2026-08-11 08:52:53 -04:00
Joseph Doherty 90b7cc6c48 docs(index): Secrets 0.6.1 — the pin alone does not close the pre-host gap; mxgw 0.6.0 pushed and needs a follow-up 2026-08-11 08:46:13 -04:00
Joseph Doherty 9b6afe3efd docs(index): Secrets 0.6.1 — close the pre-host hole in the store-path guard 2026-08-11 08:39:21 -04:00
Joseph Doherty cbfe296279 fix(secrets): close the pre-host hole in the store-path guard (0.6.1)
Found by the HistorianGateway adoption of 0.6.0, which probed the rejected paths
instead of observing a clean boot — a clean boot being equally consistent with the
validator not being wired at all.

An application that expands ${secret:} references before building its host composes
this subsystem in a throwaway ServiceCollection with no IHostEnvironment in it, then
migrates the store from that container. SecretsOptionsValidator resolves the content
root from IHostEnvironment and skips the under-content-root rule when it is absent, so
pre-host an absolute in-content-root path passed the only rule still applying, the
migrator CREATED the database, and the boot failed a second later once the real host
registered IHostEnvironment and the rule finally fired.

The observable result was a rejected boot that still left an empty store, with -wal and
-shm siblings, at the rejected path — the same artifact that made the MxGateway outage
read as "the database is there, it's just empty", which is precisely what this guard
exists to prevent. A relative path was unaffected because the absolute-path rule fires
with or without a content root; that asymmetry is what exposed it.

The mistake was conflating "no IHostEnvironment registered" with "no content root
exists". New AddZbSecrets overload takes the content root explicitly; the 3-argument
overload is unchanged and still falls back to IHostEnvironment, so this is additive.
SecretsOptionsValidator gains a string? constructor alongside the IHostEnvironment? one.

Reproduced first in an isolated harness before changing anything: a bare host rejects
both bad paths leaving no artifacts, and a bare ServiceCollection with no
IHostEnvironment migrates an in-content-root path to disk. Both are now regression tests,
the second paired with a positive control asserting the old behaviour still holds when no
content root is supplied — so the test is about the explicit content root rather than
about paths in general.

Also documents an upgrade note that applies to 0.6.0: a store seeded under the old
relative default is orphaned rather than migrated, and nothing signals it.
2026-08-11 08:37:53 -04:00
Joseph Doherty 105c230cb1 docs(index): Secrets 0.6.0 — fail-closed store-path validation, default moves outside the content root 2026-08-11 08:26:05 -04:00
Joseph Doherty 944595ad32 feat(secrets): fail-closed store-path validation; default moves outside the content root
Secrets:SqlitePath is now validated at startup by SecretsOptionsValidator using two
rules in the new public SecretsStorePathRules. Both come from production incidents and
neither implies the other.

Must be absolute: a relative value resolves against the process working directory,
which for a Windows service is C:\Windows\System32, so the store lands outside the
installation; in a dev loop it lands in the source tree (a gateway was observed writing
its secrets database into src/, tripping that repo's own tree-hygiene test).

Must not be inside the content root: an absolute path can still point into the
directory every deployment procedure in this family renames on upgrade. On 2026-08-09 a
gateway's credential store sat inside its Server\ directory; the deploy renamed Server
to Server.bak.*, the database went with it, and the process created a fresh empty one
at the same path. Every consumer lost its credential, silently, for two days. The
existing "must be absolute" convention did not help — the path was absolute.

The rules reject rather than relocate. Silently moving a credential store presents later
as an authentication failure pointing nowhere near the configuration that caused it; a
boot error is seen immediately by whoever caused it. Validation is deliberately not
exempted in Development: a guard that never runs in a dev loop or in CI rots, and the
configuration behind the outage looked entirely reasonable.

BREAKING for configuration, not API. SecretsOptions.SqlitePath now defaults to an
absolute per-user path under LocalApplicationData; it was "secrets.db", which the new
rules reject. An application configuring a relative path will fail to boot — remove the
key or set an absolute one outside the content root. All five deployed instances of this
family were checked first and already set an absolute path outside their content root,
so the migration edge (a host relying on the OLD default now finding nothing and
creating an empty store — this bug, re-introduced by its own fix) does not apply to any
of them.

Per-user rather than machine-wide is deliberate: CommonApplicationData is /usr/share on
Linux and macOS, which a normal user cannot write, so a machine-wide library default
would fail every consuming application's boot on a developer machine. An application may
compute its own default; the configured value always wins.

Validator registration is idempotent. AddZbSecrets is legitimately called more than once
in real compositions, and TryAddEnumerable cannot de-duplicate a factory-built
descriptor — it de-duplicates on implementation type, which a factory does not carry, so
it throws on the second call rather than skipping. The existing suite caught this.

Tests cover both rules including the segment-boundary case (/srv/app-data must not read
as inside /srv/app — a false positive on a fail-closed startup rule is an application
that will not boot), that the shipped default satisfies the rules it ships with, that
the default is not machine-wide, idempotent registration, and — most importantly — that
a bad path actually throws when IOptions is resolved, since a correctly implemented
validator that is never invoked is the same "guard that does not fire" failure this
change exists to prevent.

Family bumps to 0.6.0. Adds a dependency on ZB.MOM.WW.Configuration 0.1.0 for the shared
OptionsValidatorBase/ValidationBuilder vocabulary, which every consumer already
references.
2026-08-11 08:18:37 -04:00
Joseph Doherty f088cb81ba docs(index): ScadaBridge Central UI sweep merged + deployed to wonder (Theme 0.4.1 live) 2026-08-11 07:31:54 -04:00
Joseph Doherty 003783a593 docs(index): record ZB.MOM.WW.Overview — the hosted family dashboard, now deployed to wonder
This repo hosts one application besides its shared libraries, and the index did
not mention it at all. Adds what it is, that its registry ships empty by design,
and the two constraints that bit during the first real deployment: publish
self-contained (wonder has no .NET 10 runtime), and do not put co-located
instances on a shared host name, because leader-to-instance resolution matches on
host only and the first match wins the chip for every group.
2026-08-11 06:54:08 -04:00
Joseph Doherty 4f97e72ac9 fix(overview): wonder registry — 127.0.0.1 BaseUrls so leader names cannot resolve wrongly
Live on the box, the ScadaBridge *site* card claimed "Leader · central". Central
and site are separate Akka clusters, so that name was simply wrong.

Cause is in the registry, not the app. ResolveToInstanceName matches a reported
Akka leader address to an instance on HOST ONLY - correct by design, because an
app's Akka remoting port and its HTTP port differ, so comparing ports would never
match. It assumes what holds on the docker rig and not here: distinct hosts per
instance. All five instances are on one box, and the scan is fleet-wide in
registry order, so with every BaseUrl on "localhost" the first matching entry won
the chip for every group.

No host value makes this resolve CORRECTLY, because the instances really are
co-located. So make it never match instead: ScadaBridge's Akka addresses use host
"localhost" and OtOpcUa's uses the FQDN, and "127.0.0.1" is string-unequal to
both. Every chip now falls back to the raw Akka address - unambiguous and always
true. A raw address beats a confident wrong name.

Using the FQDN for OtOpcUa's BaseUrl would have resolved its chip correctly and
was rejected: that name resolves to a link-local IPv6 on this box, and routing
the real health probe through it to improve a label trades the primary function
for a cosmetic one.

ManagementUrl values are unchanged - they are followed by the operator's browser
from another machine and must keep the box address.
2026-08-11 06:50:27 -04:00
Joseph Doherty c582508ae5 feat(overview): wonder-app-vd03 registry overlay
Adds appsettings.Wonder.json for the first deployment of this dashboard to a
real box. Named "Wonder" rather than "Production", mirroring the existing
Docker overlay: this is ONE deployment's fleet, not a claim about what every
production deployment looks like.

Every port was established by probing the running box, not inferred from the
docker rig - and the two disagree, so inference would have been wrong:

  OtOpcUa             :9000  (AdminUI on the same listener)
  ScadaBridge Central :8085  (rig serves this on :5000)
  ScadaBridge Site    :8084  (no UI - / is 404, so the card carries no link)
  MxAccessGateway     :5130  (:5120 is gRPC)
  HistorianGateway    :5220  (:5221 is gRPC h2c)

:8083 and :8086 answer 400 to an HTTP/1.1 GET - h2c gRPC - and are excluded.

Both gateways are HasActiveRole:false. Their /health/active does return 200,
but only because the active tier runs an EMPTY check set when no Active-tagged
check is registered, so an Active badge would assert a role neither has.
Central and site are separate groups because they are separate Akka clusters;
one shared group would manufacture a leader disagreement.

BaseUrl is localhost (probed by the service, on the box) while ManagementUrl
carries the box address (followed by the operator's browser, elsewhere).

Serilog's file path is absolute and outside the Server directory an upgrade
replaces - a relative path resolves against the service AppDirectory, so every
upgrade would silently eat the log history. Same class of defect that cost
HistorianGateway its secrets store on a swap.

Boot-verified locally under ASPNETCORE_ENVIRONMENT=Wonder: registry passes
OverviewOptionsValidator and the host starts.
2026-08-11 06:44:17 -04:00
Joseph Doherty 8f31184b7d chore(overview): bump Theme 0.3.1 -> 0.4.1, Health 0.2.0 -> 0.3.0
Family alignment ahead of the first wonder-app-vd03 deploy. Both bumps are
inert for this app, established rather than assumed:

- Theme 0.3.1 -> 0.4.1 is two commits: the upstreamed Bootstrap .btn sizing
  block and the rail-btn-block modifier. Overview answers NO to both of the
  questions the kit README says to ask - it declares no .btn sizing of its own
  and has ZERO call sites carrying a bare `btn` token (its single button is
  class="rail-btn"), so the new .btn rule matches nothing here. The .rail-btn
  refactor that introduced --rail-btn-inset resolves to the same 0.6rem the
  literal margin used, so the one call site is unchanged. Same answer as
  HistorianGateway. Note a naive \bbtn\b grep is a FALSE POSITIVE on
  "rail-btn" - the hyphen is a word boundary; match a whitespace-delimited
  token instead.

- Health 0.2.0 -> 0.3.0 is a pure version-number alignment: the core
  ZB.MOM.WW.Health package is byte-identical between the two, with the entire
  delta confined to ZB.MOM.WW.Health.Akka (ActiveNodeHealthCheck,
  ClusterActiveNode, AkkaActiveNodeGate), which this app does not reference.
  Verified with a positive control on the diff path. The 0.2.1 active-tier
  503 change therefore cannot reach this app's own endpoints; it registers no
  Active-tagged check, so /health/active runs an empty set and stays 200.

Overview consumes the 0.3.0 `activeNode` data key over the WIRE from other
apps' health JSON, which is independent of the package it compiles against -
LeaderResolver already ranked it (344bdc4) while pinned to 0.2.0.

Release build 0 warnings; 177/177 tests pass. All remaining outdated packages
are test-only (bunit, coverlet, test SDK); nothing the app ships is behind.
2026-08-11 06:35:56 -04:00
Joseph Doherty 8e99b3b4c0 docs(playbook): re-read comments inside your own diff; check modals for state of their own
A commit that removed a modal's self-closing behaviour shipped two comments in the
same file still describing it as self-closing — review checks whether the code is
right, not whether the surrounding prose survived. And the property that predicts
which modals break on refresh is 'holds no state of its own' (pure projection of a
list), not the presence of a detail field.
2026-08-11 06:18:00 -04:00
Joseph Doherty 7926e8bb46 docs(playbook): a limitations section lends unearned credibility to the claims above it
Readers take 'what I could not establish' as evidence the rest was held to that
standard. Write the claim the evidence licenses, not the broader one; and when two
items in your own not-established list contradict each other, flag it — that
contradiction is the strongest lead you have, not another unknown.
2026-08-11 05:59:59 -04:00
Joseph Doherty 76098ffcd4 docs(playbook): keyword grep is blind to unlabeled entropy — the reliable check is your own minted values
A bare 32-hex key id matches none of the six credential keywords (verified). Added
an entropy scan and a PEM check as generic complements, marked the prefix list
repo-specific, and made the primary rule explicit: a gate that mints anything must
grep the commit for the literal values it produced.
2026-08-11 05:58:07 -04:00
Joseph Doherty 1cc4ccd258 docs(playbook): scan commit content too, and record which repos lack a .claude/ ignore
A filename audit finds the wrong-file problem; it cannot find a credential pasted
into a legitimately-tracked doc. Both checks are cheap and catch disjoint failures.
Verified: .claude/ is gitignored in OtOpcUa and HistorianGateway, NOT in mxgw or
ScadaBridge — in those two, enumerate-first discipline is the only guard.
2026-08-11 05:54:41 -04:00
Joseph Doherty 20e16c934c docs(index): Theme 0.4.1 — all five consumers pushed; two on branches, ScadaBridge's not deploy-ready 2026-08-11 05:52:30 -04:00
Joseph Doherty ec4ad984c1 docs(playbook): enumerate untracked paths before staging a sweep commit
A sweep's landing commit is where unrelated local state gets swept in. One repo's
'78 modified' was 76 modified + 2 untracked, one of which was .claude/ holding
settings.local.json — a literal 'git add -A' would have put permission config into
a tracked commit feeding a production deploy, invisible in a 76-file diff.
2026-08-11 05:51:21 -04:00
Joseph Doherty 094307e18b docs(playbook): test the shared premise when all hypotheses die; eliminate at the right granularity
Two deadlocked investigations both inherited 'file exists => sink constructed =>
no events flowed'; the pinned sink version creates no file until the first emitted
event, so the artifact proved the opposite. And 'a second log wrote in the same
directory' rules out directory-level failure, not per-handle failure — a proof
about the container is not a proof about the member.
2026-08-11 05:12:31 -04:00
Joseph Doherty baaf0a183c docs: correct the correction — absence of a .btn block does not imply the bump restyles
HistorianGateway has no local block AND no Bootstrap button call sites (all 15
buttons use the kit's .rail-btn chassis class), so the kit's .btn rule matches
nothing and bumping it changes nothing visible. My previous correction claimed the
opposite. Checking this needs two greps — declaration AND call sites — which come
apart exactly when an app doesn't use Bootstrap buttons.
2026-08-11 04:18:20 -04:00
Joseph Doherty dc6410dfad docs(playbook): name the recurring failure — a set-wide claim from a partial check
Three instances in one round (btn-block universality, sweep behavior-preservation,
the mxgw Bootstrap exemption), all the same shape, all caught by the asserter
re-checking rather than by challenge. Fix is a cheap per-member check that runs,
or an honestly-scoped claim when it hasn't.
2026-08-11 04:15:10 -04:00
Joseph Doherty 3777e50564 docs(theme): correct the .btn upstreaming claim — it was not universal
HistorianGateway has no local .btn sizing in either its committed tree or its
sweep branch (only vendored bootstrap.min.css), so the shipped comment and README
naming all five consumers were wrong. Consequence worth stating: HG's bump to
0.4.0+ is a VISIBLE change (buttons shrink to .85rem), not the no-op cleanup it is
for consumers that already carried the block.
2026-08-11 04:12:34 -04:00
Joseph Doherty 40e07aa08e docs(theme): correct the --rail-btn-inset comment — it reduces the duplication, it does not remove it
The prior wording claimed the two rules 'cannot drift apart', which is only true
for the documented composition. 0.6rem still appears twice; the fallback confines
a stale copy to the undocumented modifier-alone path. Records why hoisting the
var to :root would be worse rather than better.
2026-08-11 03:22:24 -04:00
Joseph Doherty 9dd288b3ad docs(playbook): three caches not one — confirm against server output; mxgw is the check-first outlier 2026-08-11 03:21:14 -04:00
Joseph Doherty f313c5e62e refactor(theme): rail-btn inset as --rail-btn-inset so .rail-btn and .rail-btn-block cannot drift
Unreleased — behaviourally identical to the published 0.4.1 (browser-verified:
composed <a> and <button> both fill the rail foot exactly, 0.00px short). The
var() carries a 0.6rem fallback because --rail-btn-inset is declared on .rail-btn,
so rail-btn-block used alone would otherwise have an undefined var, invalidating
the whole width declaration at computed-value time and silently reverting to
width:auto — the exact <button> shrink-wrap bug 0.4.1 fixed.
2026-08-11 03:20:37 -04:00
Joseph Doherty f9d4e2f514 docs(index): Theme 0.4.0/0.4.1 — .btn sizing upstreamed, rail-btn-block fixed, 5 consumers 2026-08-11 03:18:25 -04:00
Joseph Doherty d603a2f66e docs(playbook): cache-bypass on the live gate + trim-don't-delete on upstreamed blocks
Two lessons from the Theme 0.4.0 rollout across five apps.

Live gate: after a kit-CSS change a normal reload can serve the cached previous
layout.css while the server serves the new one. .btn then computes at
Bootstrap's 16px — the exact signature of a failed upstreaming — so the natural
response is to restore the local rule just deleted, entrenching the duplication
the change removed. Hard-reload before trusting any computed-style probe.

Upstreaming: the kit ships sizing only, so trim the local .btn rather than
deleting it. mxgw's block also carried border-radius/font-weight/white-space,
which a wholesale delete would have dropped silently; three of five apps' blocks
were pure sizing and safe to delete outright, and the difference is invisible
without diffing. Also records why a retained border-radius must stay a literal.

Marks the .btn block as upstreamed in 0.4.0 rather than a candidate.
2026-08-11 03:02:17 -04:00