0.6.x refuses a secret store whose path is relative or inside the content root,
because a store in the deployment directory is destroyed by an ordinary upgrade —
the failure that wiped the MxGateway API-key store on 2026-08-09 and read as an
auth outage rather than a deployment error.
The pin alone would not have protected this repo. Program.cs expands ${secret:}
before the host exists, composing secrets into a throwaway ServiceCollection with
no IHostEnvironment, so the guard would not run at the moment the migrator creates
the store. That composition now lives in SecretsRegistration with an explicit
content root — resolved to match what the host resolves later, including the
Windows-Service case where the pre-host CWD is still system32 — and is covered by
PreHostSecretsContentRootTests, verified by simulating the regression and
confirming it fails on the leftover file rather than on the exception.
The docker rig needed a fix too: /app/data is absolute but inside the container's
content root, so all 8 nodes would have failed to boot. Each node's data directory
is now mounted a second time at /data; same host directory, so existing stores
carry over untouched.
Verified: build clean, 29 test assemblies green (Playwright's 159 failures are the
pre-existing SEC-36 login baseline). Not yet deployed — the rig runs the old
config until someone redeploys.
Pin all five ZB.MOM.WW.Secrets* packages 0.4.1 -> 0.5.0, which brings
SecretsGrpcHubClientOptions.FallbackEndpoints and the package's internal
FailoverSecretsHubReader. A site whose GrpcHub section lists fallback
endpoints now fails a sweep over to the next central instead of stalling
on a downed primary - safe ONLY because both central nodes serve one
shared SQL secret store (scadaproj#4), so either hub answers with the
same manifest; the appsettings comments say so and warn against listing
endpoints backed by independent stores.
appsettings.json gains "FallbackEndpoints": [] with a _fallbackEndpoints
comment, and the _endpoint note's single-endpoint-stall caveat is scoped
to the empty-list case it now only applies to.
Wiring pins (red first on 0.4.1): site + Grpc + one fallback resolves
ISecretsHubReader to FailoverSecretsHubReader with the
"zb-secrets-grpc-hub:fallback:0" keyed channel present; zero fallbacks
keeps the plain GrpcSecretsHubClient and no fallback channel - the
pre-0.5.0 container shape byte-identical.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
In Secrets:Replication:Mode=Grpc a central node's ISecretStore is now the
shared SQL-Server store (AddZbSecretsSqlServerStore) instead of a per-node
local SQLite store. Both central hubs read and write ONE copy of every row,
so they serve identical manifests by construction — the 2026-08-07 live gate
observed central-b answering an authenticated GetManifest with an EMPTY
manifest while central-a held every secret, which would turn site-side hub
failover into a silent convergence stop.
- SecretsRegistration: two fail-closed pre-checks before any registration on
the central+Grpc path — a blank Secrets:SqlServer:ConnectionString throws
naming the key (an independent store per central node is the recorded
defect), and a value containing ${secret: throws naming the bootstrap
circularity (the expander needs this store to resolve references). Site
registrations are byte-identical to before; SqlServer mode and
replication-off are untouched.
- Program.cs Layer-A expander follows the store swap: central+Grpc with a
non-blank connection string migrates and resolves pre-host ${secret:}
references through the shared SQL store, so expanded values can never
diverge from what the running node serves. Every other case keeps the
SQLite path unchanged; blank-connstr central deliberately falls through so
the clear AddScadaBridgeSecrets message is the one that fails the boot.
- appsettings.json: Secrets:SqlServer _comment now documents the Grpc-mode
central requirement (literal/env value only, sites leave it empty).
- SecretsReplicationWiringTests: +5 pins (shared store resolves, blank and
${secret:} connstrings fail naming the key, sites-have-no-SqlServer-types
descriptor sweep), central fixtures carry the now-required connstr.
Full suite green (7,474 passed across 30 projects, 0 warnings); the two
failures are pre-existing and unrelated: the Playwright live-rig suite fails
identically on unmodified main (cluster not running), and
GrpcCentralTransportTests.DeadlineExceeded_IsNotRetriedOnThePeer is a timing
flake that passes 3/3 in isolation and 470/470 on the first run of this code.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
scadaproj#3: the production secrets topology is central hosting a pull-only
gRPC hub with site nodes sweeping it. The SqlServer replicator stays in the
codebase and keeps working exactly as it did, but it is not the production
path — it needs every site node to hold a connection string to central's
database, which breaks ScadaBridge's standing rule that sites talk to central,
not to central's DB.
Selection is a new Secrets:Replication:Mode key alongside the existing
Secrets:Replication:Enabled flag. Absent or blank means SqlServer, so an
existing configuration that sets only Enabled behaves identically; an
unrecognised value is refused at startup naming the key, and refused whenever
it is present rather than only when replication is on — a typo should fail the
boot that introduced it, not some later boot that flips an unrelated flag.
Which HALF a node composes is a parameter, not a config key. Both composition
roots already know statically which they are (Program.cs is central,
SiteServiceRegistration is a site), and a role read from configuration is a
role that can be got wrong in the one direction that matters: a site hosting
the hub would serve central's whole secret inventory from inside the site
network to anything holding the shared token.
The hub is mapped onto the EXISTING central h2c control-plane listener
(ScadaBridge:Node:CentralGrpcPort, default 8083) beside CentralControlService
— the same listener and the same addressing convention sites already use, and
the same shape as the site's LocalDb sync endpoint sharing its gRPC port: two
disjoint service prefixes, two independent fail-closed gates. The
CentralControlAuthInterceptor on AddGrpc is prefix-scoped and passes hub calls
through; the hub's own SecretsHubAuthInterceptor, attached per-service by
AddZbSecretsGrpcHub, gates them on Secrets:GrpcHub:BearerToken.
Registration and mapping share one predicate (UsesGrpcHub) deliberately.
Mapping without registering would map a hub whose interceptor was never
attached — an anonymous endpoint serving every secret central holds, on a node
that looks completely healthy — so the two must not be able to drift.
gRPC mode fails CLOSED where SqlServer mode degrades: a missing endpoint or
bearer token is a startup failure, not a warning plus a local-only store. The
degraded outcome is precisely what the hub exists to prevent (a site quietly
serving secrets that never converge), and the package's own errors name the
exact key, so they are left unwrapped.
Templates are default-OFF: Enabled stays false, Mode stays SqlServer, and
Secrets:GrpcHub ships with an empty BearerToken and Endpoint. Empty is
fail-closed, not open. The token is documented as appsettings/env only — never
a ${secret:} reference, since resolving one is what the hub exists to make
possible — the same bootstrap rule the mesh pre-shared keys follow.
Known asymmetry, recorded in the template: CentralGrpcEndpoints is a LIST that
fails over across the central pair, but the hub client dials a SINGLE endpoint,
so a sweep against a downed central node stalls instead of failing over. That
is survivable — the sweep is best-effort and the site keeps serving its full
local last-known-good store — but secrets stop converging until that node is
back.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Routes both host-container secret registrations (central role in Program.cs,
site role in SiteServiceRegistration) through a new SecretsRegistration
composition seam that optionally enables ZB.MOM.WW.Secrets.Replicator.SqlServer
hub-replication mode: each node keeps a LOCAL store that syncs bidirectionally
with a shared central SQL hub, so a site cluster keeps resolving secrets
straight through a WAN outage to central.
OPT-IN GATE (the load-bearing part). AddZbSecretsSqlServerReplication validates
its options EAGERLY at registration time, so wiring it unconditionally would
make ScadaBridge fail to START anywhere Secrets:SqlServer:ConnectionString is
unset -- every dev box, every docker node, every existing deployment. The
SQL-Server package is therefore only touched when BOTH Secrets:Replication:
Enabled is true AND a non-blank connection string is present; otherwise the
registration is byte-identical to the previous plain AddZbSecrets call.
Enabled-without-a-connection-string falls back to local-only and logs a warning
rather than failing the node or silently looking healthy.
BOOTSTRAP CYCLE. The hub connection string is itself a secret and can never come
from the hub -- a node cannot read the hub to learn how to reach the hub. It must
arrive from outside the replicated set: an environment variable, or a ${secret:}
reference seeded in that node's own LOCAL store. appsettings.json therefore ships
ConnectionString empty with a _comment saying so (leaf keys starting with '_' are
skipped by the reference expander, verified in SecretReferenceExpander). No real
connection string is committed. The pre-host ${secret:} expander in Program.cs is
deliberately left on a plain local SQLite store for the same reason.
Per-node docker appsettings are intentionally NOT modified -- replication stays
off there for now.
Tests written before the wiring and confirmed red first (2 failed / 4 passed),
green after (6/6). They BUILD a container and RESOLVE from it rather than
asserting over ServiceDescriptors: a decorator can look correct as a descriptor
list and still throw on first resolve because the undecorated concrete store it
depends on is missing -- that exact defect shipped once in this library with all
descriptor-level tests green, so the undecorated SqliteSecretStore gets its own
resolution test.
Verified: Host.Tests 285/285 pass; full build (all projects except the
pre-existing AngleSharp NU1902 CentralUI.Tests restore break) 0 warnings,
0 errors.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Adds a "Dev Disable-Login Flag" subsection to Component-Security.md covering
ScadaBridge:Security:Auth:DisableLogin / User, the AutoLoginAuthenticationHandler
mechanism, and the no-environment-guard / startup-warning production risk.
Ships DisableLogin: false under ScadaBridge → Security → Auth in:
- src/.../Host/appsettings.json (canonical default)
- docker/central-node-a/appsettings.Central.json
- docker/central-node-b/appsettings.Central.json
Also records DL-3 commit SHAs in the plan tasks file.
Joseph Doherty