dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: dohertj2/ScadaBridge:4df3a55824f152fc614713f346fb414954023395
Branches Tags
dohertj2/ScadaBridge:main
...
compare: dohertj2/ScadaBridge:f6c3f7c59337ffa7207531187132ae7dbe309f31
Branches Tags
dohertj2/ScadaBridge:main

7 Commits
4df3a55824 ... f6c3f7c593

Author SHA1 Message Date
Joseph Doherty f6c3f7c593 test(secrets): live gate 4/4 — check 4 re-run and PASSES on 0.4.1 
Second pass on the 0.4.1 image, from a clean start with zero denial warnings on
central-a. Both auth negatives are still refused with a byte-identical
Unauthenticated status and detail - 0.4.1 changed what the server writes down,
not what a caller sees - and the no-bearer call now produces a WRN in the same
second it is refused, with the cause attributed. No interval has to elapse for
the first denial to be visible, which is the whole point: a follower with a
mis-rotated token is refused on every sweep, and central now says so immediately.

The rate limit was proven to DEFER rather than drop, not assumed to. The
wrong-bearer call one second later fell inside the 60 s window and produced no
line of its own; the window was waited out and one further wrong-bearer call
issued, whose summary reported TWO wrong-credential denials - the deferred one
plus the new one. Summing the two lines gives exactly the three negatives issued,
correctly attributed by cause. N is a per-window delta, so a reader must sum the
lines rather than quote the last one; that is recorded as a follow-up because it
is the kind of thing an alert gets wrong.

Log hygiene re-run fleet-wide and widened: all eight nodes' docker logs and every
on-disk Serilog file were grepped for the dev token, the dev KEK, all three secret
plaintexts AND both wrong tokens the negatives presented. Zero hits everywhere.
The presented-credential check is deliberate - echoing a rejected credential back
into a log is its own leak and a free oracle, and the new warning counts denials
by cause without carrying any credential material.

Checks 2 and 3 were not repeated: 0.4.1 touches the hub's denial logging and
nothing else - no wire change, no store change, no sweep change. Convergence was
re-smoked instead so the new image is not merely assumed to replicate: a fresh
secret reached both followers in 17 s, byte-identical and decrypt-verified on
both, and the first pass's live secret and tombstone survived the image swap
unchanged on all three nodes.

The first-pass FAIL evidence is kept intact rather than overwritten. The fix only
means anything against the failure it answers, and a gate doc that shows only the
green run cannot be audited.

Residuals stand as recorded: the hub client dials a single endpoint with no
failover, and the central pair does not converge with itself - central-b answered
an authenticated GetManifest with an empty manifest for the whole run. Those are
one question, not two.

4/4. Merging.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 08:51:39 -04:00
Joseph Doherty 6d38e89be0 deps(secrets): 0.4.1 — hub denial warning (live-gate check 4) 
The 2026-08-07 live gate's check 4 failed one clause of three: both auth
negatives were denied with a byte-identical Unauthenticated status and no token
or plaintext reached any log, but the hub recorded a denial only at Information
level, via Grpc.AspNetCore.Server rather than the gate itself. A follower whose
token was mis-rotated would therefore stop converging while central showed
nothing above INF.

That was a property of the library, not of this branch, so it was fixed there and
shipped as 0.4.1 (scadaproj main c86cead): SecretsHubAuthInterceptor now emits a
rate-limited Warning summarising each denial window, breaking the count down by
cause, with the first denial after startup or a quiet window warning immediately
so a single probe is never silent. Wire behaviour is untouched - the denial is
still one uniform Unauthenticated with one detail, and still carries no token
material - so this is additive on the server's diagnostics only and nothing a
follower observes changes.

All five ZB.MOM.WW.Secrets* pins move together. Splitting them is not an option
worth having: Abstractions carries the StoredSecret shape the Grpc wire mirror is
written against, so a mixed set is a silent structural mismatch rather than a
build error.

Build clean at 0 warnings; the secrets wiring + hub-mapping pins are 31/31.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 08:36:57 -04:00
Joseph Doherty 9d5cf7100e test(secrets): live gate for the gRPC secrets hub — 3/4 PASS, not merged 
Rig config enabling the pull-only hub on the docker cluster (central pair hosts,
site-a pair follows; site-b and site-c deliberately left off so the default-OFF
posture is proven side by side), plus the gate record.

Checks 1-3 PASS. A central write reaches both site-a nodes in 5 s with a
byte-identical ciphertext row and decrypts correctly on both; a site pair boots
and serves its full last-known-good store with the entire central pair stopped,
warning once per interval without crashing, and resumes convergence unaided when
central returns; a tombstone propagates in under 9 s and survives a pair restart
with central up and sweeping, without resurrecting.

Check 4 FAILS one clause of three. Both auth negatives - absent bearer and wrong
bearer - are denied with a byte-identical Unauthenticated status and detail, and
a fleet-wide grep of all eight nodes' docker logs and on-disk Serilog files finds
ZERO occurrences of the dev token, the dev KEK or either plaintext. But the
criterion also asks for a server-side WARNING on denial, and there is none: the
only record is one Information line per call from Grpc.AspNetCore.Server, because
SecretsHubAuthInterceptor deliberately logs nothing on a denial and warns only
when no token is configured at all. That is a property of the 0.4.0 library, not
of this branch, and it is not patched here - a host-side interceptor would
contradict a documented library decision at the wrong layer and put an unbounded
log write on an unauthenticated endpoint.

The merge condition is 4/4, so this branch is NOT merged. The library's denial
logging is the only thing between this result and a merge.

Two residuals worth carrying: the hub client dials a single endpoint and does not
fail over (observed live, and contrasted against CentralGrpcEndpoints failing over
on the same node in the same minute), and the central pair does not converge with
itself - central-b answered an authenticated GetManifest with an empty manifest
for the whole run while central-a held both secrets. Together those make "which
central node is authoritative for secrets" one question, not two.

Rig config notes: Secrets__SqlitePath points at /app/data because the appsettings
default resolves to /app inside the image's writable layer, so the central pair
gained the per-node data volume the site pairs already had. All values are
dev-only and committed under the same exception the mesh PSKs already use.

Also recorded: a gate-METHOD defect. Seeding the bind-mounted store from the macOS
host is not coherent with the running container - the row was visible to the host
and to a fresh container but never to the node, and was lost outright on restart.
Every store access was redone from a throwaway container. The failure mode is a
convincing false negative that looks exactly like a broken hub.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 08:01:55 -04:00
Joseph Doherty fc784b4137 fix(secrets): registration and hub mapping share the UsesGrpcHub predicate (review Important) 
AddScadaBridgeSecrets now branches on UsesGrpcHub instead of re-deriving the
same condition, so the doc's single-predicate claim is enforced rather than
aspirational; the role split is a switch with a default-throw so a future
third role must choose its hub half explicitly.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 07:33:47 -04:00
Joseph Doherty c8e90daafb test(secrets): pin both hub halves, the mode key, and the mapping gate 
Extends the existing build-and-resolve wiring suite rather than asserting over
ServiceDescriptors, for the reason that file already documents: a registration
can look correct as a descriptor list and still fail on first resolve.

Role separation is pinned in both directions — central registers the hub's
fail-closed interceptor and NOT the sweep, a site registers the sweep and NOT
the hub — because only one of those is a security failure and testing the
happy half would not catch it. The sweep is asserted by resolving IHostedService,
which builds its whole graph (reader, keyed channel, local store) and is where
a forgotten AddZbSecrets would surface; GrpcSecretSyncService is internal to
the package, so it is matched by assembly + type name the way the SqlServer
replicator's services already are.

SqlServer mode gets regression pins with the mode key both unset and named
explicitly, plus one asserting it stays role-agnostic — both nodes sync
bidirectionally against the same database, and the role parameter added for the
hub must not have quietly changed that.

The mapping tests assert over the app's real endpoint data sources, on the WIRE
route (/zb.mom.ww.secrets.hub.v1.SecretsHub/...) rather than the C# type, since
the route is what a follower addresses. One of them pins exactly the two READ
methods: pull-only is a property of the contract, and this is where a future
package version growing a write RPC would become visible instead of silently
opening a path for a site to overwrite central.

Numeric mode values get their own test. Enum.TryParse accepts any integer,
including ones outside the enum, so "7" would otherwise select a mode that does
not exist and fall through to the SqlServer branch.

Verified red-first by mutation on the finished implementation: swapping the two
role branches reds 8 (both role pins, both fail-closed pins, both mapping
pins); deleting the UsesGrpcHub check in the map extension reds exactly the two
"maps no hub endpoint" cases — the unauthenticated-hub scenario; dropping
Enum.IsDefined and letting UsesGrpcHub ignore Enabled reds the out-of-range
value and the flag-off-with-full-hub-config cases. 31/31 green restored.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 07:09:47 -04:00
Joseph Doherty 127ec25425 feat(secrets): wire the pull-only gRPC secrets hub — central hosts, sites sweep 
scadaproj#3: the production secrets topology is central hosting a pull-only
gRPC hub with site nodes sweeping it. The SqlServer replicator stays in the
codebase and keeps working exactly as it did, but it is not the production
path — it needs every site node to hold a connection string to central's
database, which breaks ScadaBridge's standing rule that sites talk to central,
not to central's DB.

Selection is a new Secrets:Replication:Mode key alongside the existing
Secrets:Replication:Enabled flag. Absent or blank means SqlServer, so an
existing configuration that sets only Enabled behaves identically; an
unrecognised value is refused at startup naming the key, and refused whenever
it is present rather than only when replication is on — a typo should fail the
boot that introduced it, not some later boot that flips an unrelated flag.

Which HALF a node composes is a parameter, not a config key. Both composition
roots already know statically which they are (Program.cs is central,
SiteServiceRegistration is a site), and a role read from configuration is a
role that can be got wrong in the one direction that matters: a site hosting
the hub would serve central's whole secret inventory from inside the site
network to anything holding the shared token.

The hub is mapped onto the EXISTING central h2c control-plane listener
(ScadaBridge:Node:CentralGrpcPort, default 8083) beside CentralControlService
— the same listener and the same addressing convention sites already use, and
the same shape as the site's LocalDb sync endpoint sharing its gRPC port: two
disjoint service prefixes, two independent fail-closed gates. The
CentralControlAuthInterceptor on AddGrpc is prefix-scoped and passes hub calls
through; the hub's own SecretsHubAuthInterceptor, attached per-service by
AddZbSecretsGrpcHub, gates them on Secrets:GrpcHub:BearerToken.

Registration and mapping share one predicate (UsesGrpcHub) deliberately.
Mapping without registering would map a hub whose interceptor was never
attached — an anonymous endpoint serving every secret central holds, on a node
that looks completely healthy — so the two must not be able to drift.

gRPC mode fails CLOSED where SqlServer mode degrades: a missing endpoint or
bearer token is a startup failure, not a warning plus a local-only store. The
degraded outcome is precisely what the hub exists to prevent (a site quietly
serving secrets that never converge), and the package's own errors name the
exact key, so they are left unwrapped.

Templates are default-OFF: Enabled stays false, Mode stays SqlServer, and
Secrets:GrpcHub ships with an empty BearerToken and Endpoint. Empty is
fail-closed, not open. The token is documented as appsettings/env only — never
a ${secret:} reference, since resolving one is what the hub exists to make
possible — the same bootstrap rule the mesh pre-shared keys follow.

Known asymmetry, recorded in the template: CentralGrpcEndpoints is a LIST that
fails over across the central pair, but the hub client dials a SINGLE endpoint,
so a sweep against a downed central node stalls instead of failing over. That
is survivable — the sweep is best-effort and the site keeps serving its full
local last-known-good store — but secrets stop converging until that node is
back.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 07:09:47 -04:00
Joseph Doherty 57f08c213b deps(secrets): bump ZB.MOM.WW.Secrets* to 0.4.0 and take the Grpc replicator 
0.4.0 adds ZB.MOM.WW.Secrets.Replicator.Grpc — the pull-only central secrets
hub scadaproj#3 selected as ScadaBridge's production topology. The four
existing pins move with it rather than straddling two versions: they share the
Abstractions surface the replicators bind to, and a mixed set is a restore that
resolves but composes types from two different builds of the same seam.

One package carries both halves (server + sweep client); which half a node
composes is a registration-time decision, so only the Host — the composition
root for both roles — references it.

nuget.config already maps ZB.MOM.WW.Secrets.* to the dohertj2-gitea feed, so
no source-mapping change was needed. Restore verified against the feed.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
 2026-08-07 07:09:17 -04:00
9 changed files with 1175 additions and 49 deletions
Expand all files Collapse all files
Directory.Packages.props
+5 -4
View File
@@ -104,10 +104,11 @@
<PackageVersion Include="ZB.MOM.WW.Auth.AspNetCore" Version="0.1.5" />
<PackageVersion Include="ZB.MOM.WW.Audit" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Theme" Version="0.3.1" />
<PackageVersion Include="ZB.MOM.WW.Secrets" Version="0.2.3" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.2.3" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.2.3" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" Version="0.2.3" />
<PackageVersion Include="ZB.MOM.WW.Secrets" Version="0.4.1" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.4.1" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.4.1" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" Version="0.4.1" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.Grpc" Version="0.4.1" />
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.3" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.3" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.1.3" />
docker/docker-compose.yml
+64
View File
@@ -1,3 +1,47 @@
# ── Clustered secret replication: the pull-only gRPC hub (scadaproj#3) ─────────
#
# Central hosts the hub on its EXISTING h2c control-plane listener (CentralGrpcPort
# 8083, alongside CentralControlService); each site node sweeps it on an interval and
# writes what it pulls into its OWN local SQLite store. Nothing but ciphertext crosses
# the wire, so every participating node must resolve the SAME KEK.
#
# ENABLED ON FOUR NODES ONLY: the central pair (hub) and the site-a pair (followers).
# site-b and site-c are deliberately left without it, so the default-OFF posture is
# proven side by side on one rig — exactly as site-a is the rig's only LocalDb-replicated
# pair. A node with no Secrets__* override keeps the shipped appsettings default
# (Replication:Enabled=false, Mode=SqlServer) and composes a plain local store.
#
# ALL VALUES HERE ARE DEV-ONLY and committed under the same exception as the mesh PSKs
# and the ApiKeyPepper above: a local docker rig needs a working credential in source
# control to boot. Production supplies the KEK out of band (ZB_SECRETS_MASTER_KEY, never
# committed) and the hub token from appsettings/env — NEVER as a ${secret:} reference,
# since resolving one is what the hub exists to make possible.
x-secrets-hub-env: &secrets-hub-env
# DEV-ONLY KEK — NOT a real key. Identical on all four participating nodes: only
# ciphertext replicates, so a node with a different KEK fails closed on resolve with a
# kek_id mismatch that reads like corruption but is a deployment error.
ZB_SECRETS_MASTER_KEY: "zZiBWuoaVMbJmGXToLk9Lakw0iJozXoL/7Gxac3GwJ4="
# The appsettings default is the relative "scadabridge-secrets.db", which resolves to
# /app — inside the image's writable layer, so it is destroyed by any container
# recreate and unreachable from the host. /app/data is the node's own mounted volume
# (the one LocalDb already uses on sites; added to the central pair for this).
Secrets__SqlitePath: "/app/data/scadabridge-secrets.db"
Secrets__Replication__Enabled: "true"
Secrets__Replication__Mode: "Grpc"
# DEV-ONLY shared bearer token — NOT a real secret. Presented by every follower and
# verified by the hub's fail-closed SecretsHubAuthInterceptor. Must be IDENTICAL on the
# hub and every follower; an unset token is a startup failure on both halves.
Secrets__GrpcHub__BearerToken: "secrets-hub-docker-dev-token"
# Site half of the same section. NOTE the asymmetry with
# ScadaBridge:Communication:CentralGrpcEndpoints, which is a LIST that fails over across
# the central pair: the hub client dials a SINGLE endpoint, so a sweep against a stopped
# central-a stalls rather than failing over to central-b. That is survivable — the sweep
# is best-effort and the node keeps serving its full local last-known-good store — but
# secrets stop converging until central-a returns.
x-secrets-hub-site-env: &secrets-hub-site-env
Secrets__GrpcHub__Endpoint: "http://scadabridge-central-a:8083"
services:
central-a:
image: scadabridge:latest
@@ -7,6 +51,8 @@ services:
stop_grace_period: 30s
container_name: scadabridge-central-a
environment:
# Hub half of the pull-only gRPC secrets hub (anchor at the top of this file).
<<: *secrets-hub-env
SCADABRIDGE_CONFIG: Central
ASPNETCORE_ENVIRONMENT: Development
ASPNETCORE_URLS: "http://+:5000"
@@ -42,6 +88,11 @@ services:
- "9013:8083" # gRPC control plane (CentralControlService, T1A.2)
volumes:
- ./central-node-a/appsettings.Central.json:/app/appsettings.Central.json:ro
# Added for the gRPC secrets hub: the node's local secret store lives at
# Secrets__SqlitePath=/app/data/scadabridge-secrets.db, so it needs the same
# per-node volume the site nodes already have. Without it the store sits in the
# image's writable layer and is destroyed by every container recreate.
- ./central-node-a/data:/app/data
- ./central-node-a/logs:/app/logs
networks:
- scadabridge-net
@@ -55,6 +106,8 @@ services:
stop_grace_period: 30s
container_name: scadabridge-central-b
environment:
# Hub half of the pull-only gRPC secrets hub (anchor at the top of this file).
<<: *secrets-hub-env
SCADABRIDGE_CONFIG: Central
ASPNETCORE_ENVIRONMENT: Development
ASPNETCORE_URLS: "http://+:5000"
@@ -90,6 +143,11 @@ services:
- "9014:8083" # gRPC control plane (CentralControlService, T1A.2)
volumes:
- ./central-node-b/appsettings.Central.json:/app/appsettings.Central.json:ro
# Added for the gRPC secrets hub: the node's local secret store lives at
# Secrets__SqlitePath=/app/data/scadabridge-secrets.db, so it needs the same
# per-node volume the site nodes already have. Without it the store sits in the
# image's writable layer and is destroyed by every container recreate.
- ./central-node-b/data:/app/data
- ./central-node-b/logs:/app/logs
networks:
- scadabridge-net
@@ -103,6 +161,9 @@ services:
stop_grace_period: 30s
container_name: scadabridge-site-a-a
environment:
# Follower half of the pull-only gRPC secrets hub (anchors at the top of this
# file). site-b and site-c deliberately carry neither.
<<: [*secrets-hub-env, *secrets-hub-site-env]
SCADABRIDGE_CONFIG: Site
ports:
- "9021:8082" # Akka remoting (host access for debugging)
@@ -123,6 +184,9 @@ services:
stop_grace_period: 30s
container_name: scadabridge-site-a-b
environment:
# Follower half of the pull-only gRPC secrets hub (anchors at the top of this
# file). site-b and site-c deliberately carry neither.
<<: [*secrets-hub-env, *secrets-hub-site-env]
SCADABRIDGE_CONFIG: Site
ports:
- "9022:8082" # Akka remoting
docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md
+455
View File
@@ -0,0 +1,455 @@
# Secrets gRPC hub — live gate on the docker cluster
**Result: PASS — 4/4 checks.** Run 2026-08-07 on the local 8-node docker cluster.
Run in two passes. The **first pass** (image `2958aa2c2bbb`, secrets libs **0.4.0**) passed
checks 1–3 and failed one clause of check 4: denials were recorded at Information, not
Warning. That was a property of the library, not of this branch, so it was fixed at the
library and shipped as **0.4.1** (`scadaproj` main `c86cead`). The **second pass** (image
`f866208c89e8`, secrets libs **0.4.1**) re-ran check 4 in full and re-smoked convergence to
prove the new image still converges. Both passes are recorded below, first-pass evidence
kept intact — the fix is only meaningful against the failure it answers.
## Purpose
Terminal gate (Task 7) for `feat/secrets-grpc-hub`, which wires
`ZB.MOM.WW.Secrets.Replicator.Grpc` 0.4.0 into ScadaBridge: the central role hosts a
**pull-only** `secrets_hub.v1` gRPC hub on its existing h2c control-plane listener, and a
site node runs a sweep that converges its **own local SQLite store** from that hub.
This is the ScadaBridge half of the production-topology decision in
[scadaproj#3](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/3) (design:
`scadaproj/docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md`). The
SqlServer replicator is not the production path here: it requires every site node to hold a
connection string to central's database, which breaks ScadaBridge's standing rule that
sites talk to central, not to central's DB.
**Scope: rig config only.** No product code was changed by this gate — see
[Defects](#defects-found).
## Rig state
| Fact | First pass | Second pass |
|---|---|---|
| Repo / branch | `ScadaBridge` @ `feat/secrets-grpc-hub` | same |
| Git SHA gated | `fc784b413713b1643a21d573e55c00bcc05bd8ee` (`fc784b41`) | `6d38e89be0b39a2224559dd50922751ccd8df2ed` (`6d38e89b`) |
| Image | `scadabridge:latest` = `2958aa2c2bbb`, built 11:39 UTC | `scadabridge:latest` = `f866208c89e8`, built 12:41 UTC |
| `ZB.MOM.WW.ScadaBridge.Host.dll` | `09cf76dc550d32fe…` — byte-identical on all 8 nodes | `bc8f17e7b5d6982c…` — byte-identical on all 8 nodes |
| Secrets libs | `ZB.MOM.WW.Secrets*` **0.4.0** | `ZB.MOM.WW.Secrets*` **0.4.1** |
| Working tree at build | only `docker/docker-compose.yml` modified (rig config, not in the image) + untracked `.claude/` | clean but for untracked `.claude/` |
Common to both passes:
| Fact | Value |
|---|---|
| Hub enabled on | `central-a`, `central-b` (host) and `site-a-a`, `site-a-b` (followers) |
| Hub NOT enabled on | `site-b-a`, `site-b-b`, `site-c-a`, `site-c-b` — no `Secrets__*` env, no store file, no hub/sweep log line (verified) |
| Sweep interval | `00:00:30` — the **product default**, deliberately not shortened, so the timings below are real |
| KEK | one shared committed **dev-only** key, identical on all four (`kek_id` sha256 `7451bcbc1f1f` on every row observed) |
`site-b` and `site-c` were left off on purpose: the default-OFF posture is proven side by
side on one rig, the same way `site-a` is the rig's only LocalDb-replicated pair.
### Rig-config change made by this gate
`docker/docker-compose.yml` gained two YAML anchors and applied them to four services:
- `x-secrets-hub-env` (central pair + site-a pair) — dev KEK, `Secrets__SqlitePath`,
`Secrets__Replication__Enabled=true`, `Secrets__Replication__Mode=Grpc`,
`Secrets__GrpcHub__BearerToken`.
- `x-secrets-hub-site-env` (site-a pair only) —
`Secrets__GrpcHub__Endpoint=http://scadabridge-central-a:8083`.
All values are DEV-ONLY and committed under the same exception the mesh PSKs and
`InboundApi:ApiKeyPepper` already use. `Secrets__SqlitePath` points at `/app/data` because
the appsettings default is a relative path resolving to `/app` — inside the image's
writable layer, destroyed by every container recreate. The **central pair gained a
`./central-node-*/data:/app/data` volume** for this; the site pairs already had one.
## Method
Secrets were seeded, read and deleted with the `ZB.MOM.WW.Secrets.Cli` (`secret`) published
from `scadaproj/ZB.MOM.WW.Secrets` (0.4.0) and run **in a throwaway container** mounting one
node's `data` volume, with that node's KEK. This is the method the sibling OtOpcUa gate used
the same day (`OtOpcUa/docs/plans/2026-08-07-secrets-pair-local-live-gate.md`).
The CentralUI `/admin/secrets` page was **not** used, and for this topology that costs
nothing: the hub is a pure reader of central's store on every call, with no write-triggered
path and no manifest cache. A row written by any process into central's store is on exactly
the same path a row written through the page would be. (Contrast the Akka replicator, where
the two differ — hence that gate's residual 1, which does not apply here.)
Ciphertext identity is evidenced as a **SHA-256 over the encrypted columns only**
(`ciphertext‖nonce‖tag‖wrapped_dek‖wrap_nonce‖wrap_tag`), plus a hash of `kek_id`. No
plaintext, no token and no key material appears in this document. The secret values used
were throwaway smoke strings.
gRPC calls for check 4 were made with `fullstorydev/grpcurl` on the `scadabridge-net`
network, against the committed `secrets_hub.proto`.
## Results
| # | Check | Pass | Result | Timing |
|---|---|---|---|---|
| 1 | Convergence — central write reaches both site-a nodes, byte-identical, decrypt-verified | 1st (0.4.0) | **PASS** | **5 s** (budget 30 s + margin) |
| 2 | Cold-boot offline — site pair boots and serves last-known-good with central down; convergence resumes | 1st (0.4.0) | **PASS** | resumed in **14 s** / **43 s** |
| 3 | Tombstone — delete propagates, survives a pair restart | 1st (0.4.0) | **PASS** | **≤ 9 s** |
| 4 | Auth negatives + log hygiene | 1st (0.4.0) | ~~FAIL~~ (1 clause of 3) | — |
| 4 | Auth negatives + log hygiene — **re-run in full** | 2nd (0.4.1) | **PASS** | denial warned immediately |
| — | Convergence re-smoke on the 0.4.1 image | 2nd (0.4.1) | **PASS** | **17 s** |
Baseline before seeding: all four stores empty.
### Check 1 — convergence · PASS
Seeded `hub-gate-smoke-1` on **`central-a` only** at 11:46:57Z.
```
central-node-a hub-gate-smoke-1 rev=0 del=0 cipher_sha256=00a14a2eb8de987fb75a4515 kek=7451bcbc1f1f updated=2026-08-07T11:46:57.1403922+00:00
site-a-node-a hub-gate-smoke-1 rev=0 del=0 cipher_sha256=00a14a2eb8de987fb75a4515 kek=7451bcbc1f1f updated=2026-08-07T11:46:57.1403922+00:00
site-a-node-b hub-gate-smoke-1 rev=0 del=0 cipher_sha256=00a14a2eb8de987fb75a4515 kek=7451bcbc1f1f updated=2026-08-07T11:46:57.1403922+00:00
```
Ciphertext hash, `kek_id`, `revision` and `updated_utc` are **identical on all three** — the
row is relayed verbatim, not re-encrypted, so the last-writer-wins ordering key survives the
hop. First poll after the seed (11:47:05Z) already showed both followers converged.
Both halves logged it, and central served the fetch:
```
site-a-a [11:47:02 INF] Secret hub sync converged: pulled 1 row(s) from the central hub.
site-a-b [11:47:02 INF] Secret hub sync converged: pulled 1 row(s) from the central hub.
central-a [11:47:02 INF] Request finished HTTP/2 POST .../SecretsHub/GetSecrets - 200 application/grpc 0.9899ms
```
**Convergence = 11:46:57Z → 11:47:02Z = 5 s**, one sweep tick.
**Decrypt-verified on both followers**: `secret get hub-gate-smoke-1` returned the exact
plaintext seeded on central-a, on `site-a-a` and `site-a-b`.
### Check 2 — cold-boot offline · PASS
Central pair stopped 11:48:27Z → 11:48:37Z.
**Sweep degrades, does not crash.** One warning per interval on each follower, carrying the
transport fault and nothing else:
```
site-a-a [11:49:02 WRN] Secret hub sync failed; the node continues serving its local store and will retry on the next interval.
Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error connecting to subchannel." …)
```
**Site pair restarted together** (the supported pattern) 11:49:31Z → 11:50:12Z, with central
still down. `site-a-a` booted at 11:49:42Z; its `SyncOnStartup` sweep warned at 11:49:42Z and
retried at 11:50:12 / 11:50:42 / 11:51:12 / 11:51:42 / 11:52:12 / 11:52:42. Both containers
`running`, `RestartCount=0`, **zero** `FATAL` / `Unhandled exception` /
`Application startup exception` lines.
**Last-known-good served with central down** — at 11:51:08Z, `secret get hub-gate-smoke-1`
returned the correct plaintext on **both** `site-a-a` and `site-a-b`.
Central pair started 11:51:37Z. `hub-gate-smoke-2` seeded on central-a at 11:52:29Z — **after
the site nodes were already up**, so this proves convergence *resumed* rather than
re-bootstrapped:
```
site-a-b [11:52:43 INF] Secret hub sync converged: pulled 1 row(s) from the central hub. → 14 s
site-a-a [11:53:12 INF] Secret hub sync converged: pulled 1 row(s) from the central hub. → 43 s
```
Both rows byte-identical to central's (`cipher_sha256=4f022e222cfc2f8f77dcfb0b`,
`updated=2026-08-07T11:52:29.6983529+00:00`).
**Honest note on the 43 s.** `site-a-a`'s first post-outage sweep (11:52:42Z) *still* failed,
with `SocketException: Name or service not known` — a stale DNS/subchannel view of a
container that had been `docker stop`ped and `docker start`ed. It recovered unaided on the
very next tick. So after a central outage, worst-case convergence on this rig is **two**
sweep intervals, not one. Bounded, self-healing, no intervention — but recorded so the 5 s
figure from check 1 is not read as the post-outage number.
### Check 3 — tombstone · PASS
`hub-gate-smoke-1` deleted on central-a at 11:54:05Z. `hub-gate-smoke-2` left live as a
control.
```
central-node-a hub-gate-smoke-1 rev=1 del=1 updated=2026-08-07T11:54:05.9812744+00:00 deleted=2026-08-07T11:54:05.9812744+00:00
site-a-node-a hub-gate-smoke-1 rev=1 del=1 updated=2026-08-07T11:54:05.9812744+00:00 deleted=2026-08-07T11:54:05.9812744+00:00
site-a-node-b hub-gate-smoke-1 rev=1 del=1 updated=2026-08-07T11:54:05.9812744+00:00 deleted=2026-08-07T11:54:05.9812744+00:00
```
Both followers held the tombstone at the 11:54:14Z poll — **≤ 9 s**. Delete is a soft
tombstone (`is_deleted=1`, `revision` bumped, `updated_utc` restamped), which is what lets it
win under last-writer-wins. `secret get` returned `{"error":"not-found"}` and `secret list`
returned only `hub-gate-smoke-2` on both followers.
**No resurrection across a pair restart.** Site pair restarted together 11:55:09Z →
11:55:50Z, **with central up and sweeping the whole time** — so the sweep had every
opportunity to re-pull the row. Re-checked at 11:57:05Z (75 s / 2+ sweeps later):
- `hub-gate-smoke-1` still `rev=1 del=1` on both, `updated_utc` unchanged;
`secret get` → `{"error":"not-found"}` on both.
- `hub-gate-smoke-2` survived on both and **decrypted to the correct plaintext** on both.
### Check 4, first pass (0.4.0) — auth negatives + log hygiene · FAIL (one clause)
Kept for the record; superseded by the second pass below.
Three calls to `SecretsHub/GetManifest` on `scadabridge-central-a:8083` at 11:57:31–32Z:
| Call | Result |
|---|---|
| **no** `authorization` header | `Code: Unauthenticated` / `Message: Unauthenticated.` |
| **wrong** bearer (`Bearer not-the-hub-token`) | `Code: Unauthenticated` / `Message: Unauthenticated.` |
| correct bearer (positive control) | `200`, manifest returned (2 entries) |
**PASS — identical denial.** Both negatives are byte-identical in code and detail: a caller
learns it was refused and nothing else. The positive control proves the endpoint is live and
the refusals are the auth gate, not a dead route.
**PASS — log hygiene, zero hits.** Every one of the **8** nodes' `docker logs` *and* every
on-disk Serilog file under `docker/*/logs/` was grepped for the dev bearer token value, the
dev KEK value, and both secret plaintexts:
```
central-a token=0 kek=0 plain1=0 plain2=0 authz-header-mentions=0
central-b token=0 kek=0 plain1=0 plain2=0 authz-header-mentions=0
site-a-a token=0 kek=0 plain1=0 plain2=0 authz-header-mentions=0
site-a-b token=0 kek=0 plain1=0 plain2=0 authz-header-mentions=0
site-b-a / site-b-b / site-c-a / site-c-b all zero
on-disk log files: zero matching files for all four search strings
```
The token appears only where it legitimately may: the committed rig config
(`docker/docker-compose.yml`) and the resulting container environment.
**FAIL — no server-side *warning*.** The criterion asks for a server-side warning on denial.
What the hub actually emits is one **Information**-level line per denial, from
`Grpc.AspNetCore.Server`, not from the hub's own gate:
```
[11:57:31 INF] Error status code 'Unauthenticated' with detail 'Unauthenticated.' raised.
[11:57:32 INF] Error status code 'Unauthenticated' with detail 'Unauthenticated.' raised.
```
Nothing at `WRN` or above was emitted by either denial — verified by dumping **every** log
line in the 11:57:31–32Z window, not by a filtered grep.
This is **not a wiring mistake and not fixable in this repo**: `SecretsHubAuthInterceptor`
(in `ZB.MOM.WW.Secrets.Replicator.Grpc` 0.4.0) deliberately logs *nothing* on a denial. Its
only warning is `WarnUnconfiguredOnce`, emitted once when the hub has **no token configured
at all** — which is not the case here. So the shipped design and this gate's criterion
disagree, and the disagreement lives in the library.
The consequence is real but narrow: a follower whose token was mis-rotated is refused on
every sweep and **stops converging**, while central records that only at Information. An
operator watching for warnings sees the follower's own `Secret hub sync failed` WRN (which
does fire), but central shows nothing above INF.
**Not patched in ScadaBridge.** A host-side interceptor could log the denial, but that would
(a) contradict a decision the library documents and reasons about explicitly, in the
consumer rather than at the layer that owns it, and (b) put an unbounded log write on an
unauthenticated endpoint. Both are worse than the gap. It was fixed at the layer that owns
it instead — see the second pass.
### Check 4, second pass (0.4.1) — auth negatives + log hygiene · PASS
Re-run in full on the 0.4.1 image (`f866208c89e8`, branch `6d38e89b`), from a clean start:
zero `The gRPC secrets hub denied` lines on `central-a` before the first negative.
**Denial semantics — unchanged, as intended.** Three calls to `SecretsHub/GetManifest` on
`scadabridge-central-a:8083`:
| Call | Time | Result |
|---|---|---|
| **no** `authorization` header | 12:46:09Z | `Code: Unauthenticated` / `Message: Unauthenticated.` |
| **wrong** bearer | 12:46:10Z | `Code: Unauthenticated` / `Message: Unauthenticated.` |
| correct bearer (positive control) | 12:46:10Z | `200`, manifest returned |
| **wrong** bearer, next window | 12:47:38Z | `Code: Unauthenticated` / `Message: Unauthenticated.` |
Byte-identical code and detail on every refusal, and the positive control proves the route
is live. The wire contract did not move: 0.4.1 changed only what the server writes down.
**The warning now fires, and fires on the first denial.** No interval had to elapse — the
no-bearer call at 12:46:09Z produced a `WRN` in the same second it was refused:
```
central-a [12:46:10 WRN] The gRPC secrets hub denied 1 call(s) since the previous such warning
(missing or malformed credential: 1; wrong credential: 0; hub token unconfigured: 0).
Denials are summarised at most once every 60s.
```
**Rate limiting defers, it does not drop.** The wrong-bearer call one second later fell
inside that 60 s window, so it produced no line of its own. Rather than assume it was
counted, the window was waited out and one further wrong-bearer call issued at 12:47:38Z:
```
central-a [12:47:38 WRN] The gRPC secrets hub denied 2 call(s) since the previous such warning
(missing or malformed credential: 0; wrong credential: 2; hub token unconfigured: 0).
```
The second summary reports **two** wrong-credential denials — the deferred 12:46:10Z one plus
the 12:47:38Z one. Summing across lines gives 1 missing + 2 wrong = **3**, exactly the three
negatives issued, with the causes correctly attributed. `N` is a per-window delta, so a
reader must sum the lines rather than quote the last one. Nothing was lost to the rate limit,
which is the only property that matters for the failure this fix exists to make visible: a
follower with a mis-rotated token is refused on every sweep, and central now says so at
`WRN` within a second of the first attempt.
**Log hygiene — zero hits, fleet-wide.** All **8** nodes' `docker logs` *and* every on-disk
Serilog file under `docker/*/logs/` were grepped for the dev bearer token, the dev KEK, all
three secret plaintexts, **and both wrong tokens presented by the negatives** — the last of
those deliberately, because echoing a rejected credential back into a log is its own leak and
a free oracle:
```
central-a token=0 kek=0 p1=0 p2=0 p3=0 wrong1=0 wrong2=0
central-b token=0 kek=0 p1=0 p2=0 p3=0 wrong1=0 wrong2=0
site-a-a token=0 kek=0 p1=0 p2=0 p3=0 wrong1=0 wrong2=0
site-a-b token=0 kek=0 p1=0 p2=0 p3=0 wrong1=0 wrong2=0
site-b-a / site-b-b / site-c-a / site-c-b all zero
on-disk log files: 0 files matched, for each of the 7 search strings
```
The new warning counts denials and names their cause; it carries no credential material,
presented or configured. The token still appears only where it legitimately may: the
committed rig config and the resulting container environment.
### Convergence re-smoke on the 0.4.1 image · PASS
Checks 2 and 3 were not repeated — 0.4.1 touches only the hub's denial logging. Convergence
was re-smoked to prove the new image still replicates end to end:
`hub-gate-smoke-3` seeded on `central-a` at 12:47:54Z:
```
central-node-a hub-gate-smoke-3 rev=0 del=0 cipher_sha256=fc3cee5431cab8f3a7df303c kek=7451bcbc1f1f updated=2026-08-07T12:47:54.3769041+00:00
site-a-node-a hub-gate-smoke-3 rev=0 del=0 cipher_sha256=fc3cee5431cab8f3a7df303c kek=7451bcbc1f1f updated=2026-08-07T12:47:54.3769041+00:00
site-a-node-b hub-gate-smoke-3 rev=0 del=0 cipher_sha256=fc3cee5431cab8f3a7df303c kek=7451bcbc1f1f updated=2026-08-07T12:47:54.3769041+00:00
site-a-a [12:48:11 INF] Secret hub sync converged: pulled 1 row(s) from the central hub.
site-a-b [12:48:11 INF] Secret hub sync converged: pulled 1 row(s) from the central hub.
```
**17 s**, both followers, byte-identical ciphertext and `updated_utc`, **decrypt-verified on
both**. The first pass's `hub-gate-smoke-2` and the `hub-gate-smoke-1` tombstone survived the
image swap unchanged on all three nodes.
## Defects found
**None in ScadaBridge product code.** One library defect, found by this gate and fixed
forward; one defect in the *gate method*, caught before any result was recorded.
**Library: the hub logged denials only at Information** (`ZB.MOM.WW.Secrets.Replicator.Grpc`
0.4.0). Found by check 4's first pass; detailed above. Fixed at the library and shipped as
**0.4.1** (`scadaproj` main `c86cead`): `SecretsHubAuthInterceptor` now emits a rate-limited
`Warning` per denial window, broken down by cause, with the first denial after startup or a
quiet window warning immediately. Wire behaviour is byte-identical — the refusal is still one
uniform `Unauthenticated` with one detail. ScadaBridge took the bump in `6d38e89b` (all five
`ZB.MOM.WW.Secrets*` pins moved together; build 0 warnings, secrets wiring + hub-mapping pins
31/31), and check 4 was re-run in full against it.
**Gate method: host-side seeding of a bind-mounted SQLite store is not coherent with the
running container** (Docker Desktop for macOS). The first attempt at check 1 seeded central-a's
store with the CLI running natively on the host. The row was visible to the host and to a
fresh helper container, but the **running node never saw it**: four consecutive sweeps
returned an empty manifest, no `GetSecrets` was ever issued, and a `wal_checkpoint(TRUNCATE)`
did not help. Restarting central-a then **lost the row entirely** — it was gone from both the
host and container views. Cause is the virtiofs/page-cache boundary between the macOS host
and the Linux guest: SQLite's WAL and `-shm` mappings are not shared across it.
Corrected by doing **every** store access — seed, read, delete — from a throwaway container
on the same guest kernel, which is what the sibling OtOpcUa gate already did. The rig was
returned to an empty-store baseline before check 1 was re-run. Recorded because the failure
mode is a convincing false negative: it looks exactly like "the hub is broken".
## Residuals
1. **The hub client dials a SINGLE endpoint — no failover.** Known and documented on the
branch. Observed live and directly contrasted on the same node in the same minute:
```
site-a-a [11:51:13 WRN] Central control-plane endpoint http://scadabridge-central-a:8083 is
unavailable; site site-a failed over to http://scadabridge-central-b:8083.
site-a-a [11:51:12 WRN] Secret hub sync failed; the node continues serving its local store …
```
`ScadaBridge:Communication:CentralGrpcEndpoints` is a list and failed over; the hub client
stalled. Survivable — the sweep is best-effort and the node keeps its full local
last-known-good store — but secrets stop converging until that one central node returns.
2. **The central pair does NOT converge with itself, and this makes residual 1 sharper than
it reads.** Both central nodes host the hub (verified: `central-b` answers, and refuses an
unauthenticated call identically). But `central-b`'s store stayed **empty** for the whole
run while `central-a` held both secrets:
```
central-b, correct bearer, 11:59:32Z: {} ← empty manifest, authenticated
central-node-b (store): (empty)
```
Nothing replicates central-a ↔ central-b in `Mode=Grpc` — the topology is pull-only
*central → sites*. So a hypothetical failover to `central-b` would not merely stall, it
would succeed against an empty hub. By the reconciler's pull-only algorithm that is not
data loss (a name absent from the manifest is simply not pulled, never deleted), but it
*is* a silent stop to convergence, and a deployment that authors a secret on the central
node the sites do **not** dial will see it reach nobody. **Not live-tested here** —
asserted from the algorithm plus the observed empty manifest. Any production enablement
needs an answer for how the central pair itself converges.
3. **Both site-a nodes dial the same central node.** Not a rig shortcut — it is the only
shape the client supports (residual 1). Worth restating so the run is not read as having
exercised two independent hub sources.
4. **The `/admin/secrets` write path was not exercised.** Seeding was CLI-into-central's-store.
For this topology that is the same path (see [Method](#method)), so it is a coverage note,
not a gap in the result — but the page's own Blazor/authz behaviour is unproven by this
gate. `ScadaBridge#22` covered it separately.
5. **Delete-while-a-follower-is-offline was not tested.** Check 3 restarts the pair *after*
both already held the tombstone. The harder resurrection case — deleting at central while
a follower is down, then bringing it back holding a live copy — is covered by the
library's last-writer-wins tests but not live here. Under a pull-only hub it is
structurally safer than in a bidirectional topology (the follower can never push its stale
live row back), which is why it was not prioritised.
6. **Replication stays default-OFF in the product.** `Secrets:Replication:Enabled` is `false`
and `Mode` is `SqlServer` in the shipped `appsettings.json`. It is enabled on this docker
rig only, on four of eight nodes, with a dev KEK and a dev token. Production enablement
additionally needs a real KEK supplied out of band and a real token — never committed, and
never a `${secret:}` reference.
7. **Checks 2 and 3 ran on 0.4.0 only.** 0.4.1 changes what the hub writes to its own log on
a denied call and nothing else — no wire change, no store change, no sweep change — so
re-running the outage and tombstone drills would have re-proven code the bump did not
touch. Convergence *was* re-smoked on the 0.4.1 image (17 s, decrypt-verified on both
followers), and the first pass's live secret and tombstone survived the image swap
unchanged, so the 0.4.1 image is not merely assumed to replicate.
## Follow-ups
- **File against ScadaBridge:** endpoint failover for the hub client, and/or a documented
answer for how the central pair's own stores converge (residual 2). Both are the same
underlying question — "which central node is authoritative for secrets".
- **Reading the new denial warning:** `N` is a **per-window delta**, not a running total.
Summing across lines gives the true count; quoting the last line understates it. Worth
knowing before anyone builds an alert on the `The gRPC secrets hub denied` anchor.
## Conclusion
The pull-only gRPC secrets hub **works, and works the way the branch says it does**:
convergence is one sweep tick, ciphertext is relayed verbatim so last-writer-wins ordering
survives the hop, a site node boots and serves its full last-known-good store with the entire
central pair stopped, convergence resumes unaided when central returns, tombstones propagate
and do not resurrect across a pair restart, the auth gate refuses uniformly and now *says so*
at `WRN` within a second of the first attempt, and no token, KEK or plaintext — including the
rejected credentials the negatives presented — reaches any log on any of the eight nodes.
The one clause that failed on the first pass was a library property, not a branch one. It was
fixed where it belonged, ScadaBridge took the bump, and check 4 was re-run in full rather than
argued around. **4/4 — merged to `main`.**
**Unchanged by this gate:** `Secrets:Replication:Enabled` remains **default-false in the
product**, with `Mode` defaulting to `SqlServer`. It is enabled on this docker rig only, on
four of eight nodes, with a dev KEK and a dev token. Production enablement additionally
requires a real KEK supplied out of band, a real hub token, and an answer to residual 2.
src/ZB.MOM.WW.ScadaBridge.Host/Program.cs
+25 -5
View File
@@ -255,11 +255,15 @@ try
.GetValue<bool>(nameof(ZB.MOM.WW.ScadaBridge.Security.Auth.AuthDisableLoginOptions.AllowOutsideDevelopment)));
builder.Services.AddSecurity(disableLogin);
builder.Services.AddCentralUI();
// Local SQLite store by default; a local store replicating against a shared SQL-Server hub
// only when Secrets:Replication:Enabled is true AND a hub connection string is present.
// See SecretsRegistration for why the gate exists (eager options validation) and why the
// hub connection string can never itself come from the hub.
builder.Services.AddScadaBridgeSecrets(builder.Configuration);
// Local SQLite store by default. With Secrets:Replication:Enabled true it becomes either a
// local store synced against a shared SQL-Server hub (Mode=SqlServer, and only when a hub
// connection string is present) or — the production topology, scadaproj#3 — the HOST half
// of the pull-only gRPC secrets hub (Mode=Grpc), which central serves and sites sweep.
// The Central role never registers the sweep client; passing the role explicitly is what
// makes that a compile-time property of this composition root rather than a config guess.
// See SecretsRegistration for why the SQL gate exists (eager options validation), why the
// hub credential can never itself come from the hub, and why gRPC mode fails closed.
builder.Services.AddScadaBridgeSecrets(builder.Configuration, SecretsNodeRole.Central);
// Secrets UI authorization: adds the named policies secrets:manage + secrets:reveal
// (role-based) consumed by the /admin/secrets page. AddSecretsAuthorization only ADDS
// these two policies via Configure<AuthorizationOptions> — it composes additively with
@@ -555,6 +559,22 @@ try
// connection to the right pipeline by listener/protocol.
app.MapGrpcService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlGrpcService>();
// The pull-only secrets hub (scadaproj#3), on the SAME h2c listener and the same addressing
// convention — a site dials central's CentralGrpcPort, never central's database. It shares
// the listener with CentralControlService the way the site's LocalDb sync endpoint shares
// its one: two disjoint service prefixes, two independent fail-closed gates. The
// CentralControlAuthInterceptor registered on AddGrpc is prefix-scoped to
// CentralControlService and passes hub calls straight through; the hub's own
// SecretsHubAuthInterceptor — attached per-service by AddZbSecretsGrpcHub — is what gates
// it, on the shared Secrets:GrpcHub:BearerToken.
//
// A NO-OP unless this node registered the hub. That is not an optimisation: mapping the
// service without registering it would map a hub whose interceptor was never attached,
// i.e. an unauthenticated endpoint serving every secret central holds. Both this call and
// the registration above route through SecretsRegistration.UsesGrpcHub so they cannot
// drift apart.
app.MapScadaBridgeSecretsHub(app.Configuration);
app.MapStaticAssets();
app.MapCentralUI<ZB.MOM.WW.ScadaBridge.Host.Components.App>();
app.MapInboundAPI();
src/ZB.MOM.WW.ScadaBridge.Host/SecretsRegistration.cs
+208 -18
View File
@@ -1,8 +1,44 @@
using Microsoft.AspNetCore.Routing;
using ZB.MOM.WW.Secrets.DependencyInjection;
using ZB.MOM.WW.Secrets.Replicator.Grpc.DependencyInjection;
using ZB.MOM.WW.Secrets.Replicator.SqlServer.DependencyInjection;
namespace ZB.MOM.WW.ScadaBridge.Host;
/// <summary>
/// Which half of a clustered secrets topology this node composes. Passed in by the caller rather
/// than read from configuration, because both composition roots already know statically which one
/// they are — <c>Program.cs</c> for central and <see cref="SiteServiceRegistration"/> for a site —
/// and a role read from a key could be got wrong in a way that silently turns a site into a hub.
/// </summary>
public enum SecretsNodeRole
{
/// <summary>The central cluster. Hosts the gRPC secrets hub; never sweeps from one.</summary>
Central,
/// <summary>A site cluster. Sweeps from central's hub; never hosts one.</summary>
Site,
}
/// <summary>
/// The clustered-replication transport, selected by <c>Secrets:Replication:Mode</c>.
/// </summary>
public enum SecretsReplicationMode
{
/// <summary>
/// Local store bidirectionally synced with a shared SQL-Server hub. The historical mode and
/// therefore the default, so an existing configuration that sets only
/// <c>Secrets:Replication:Enabled</c> keeps behaving exactly as it did.
/// </summary>
SqlServer,
/// <summary>
/// Local store on every node, converged by a pull-only gRPC hub that central hosts. The
/// production topology (scadaproj#3).
/// </summary>
Grpc,
}
/// <summary>
/// Single composition-root entry point for the host container's secret store, shared by the
/// central-role registrations in <c>Program.cs</c> and the site-role registrations in
@@ -11,25 +47,45 @@ namespace ZB.MOM.WW.ScadaBridge.Host;
/// <remarks>
/// <para>
/// Clustered replication is <b>opt-in and off by default</b>. ScadaBridge is hub-and-spoke — one
/// central cluster plus N separate site clusters — and hub mode gives each node a LOCAL store that
/// syncs bidirectionally with a shared central SQL hub, so a site keeps resolving secrets straight
/// through a WAN outage.
/// central cluster plus N separate site clusters — and every supported topology gives each node a
/// LOCAL store, so a site keeps resolving secrets straight through a WAN outage. What differs
/// between the two modes is only how that local store is kept converged.
/// </para>
/// <para>
/// The gate is not a style preference. <c>AddZbSecretsSqlServerReplication</c> validates its options
/// EAGERLY at registration time, so calling it unconditionally would throw at startup on every node
/// where <c>Secrets:SqlServer:ConnectionString</c> is unset — which is every dev box, every docker
/// node and every existing deployment. Both the explicit <c>Secrets:Replication:Enabled</c> flag and
/// a non-blank connection string are therefore required before the SQL-Server package is touched at
/// all; otherwise this is byte-for-byte the plain local-SQLite registration it has always been.
/// <b><see cref="SecretsReplicationMode.Grpc"/> is the production topology</b> (scadaproj#3):
/// central hosts a pull-only gRPC hub and each site node sweeps it. It exists because
/// <see cref="SecretsReplicationMode.SqlServer"/> requires every site node to hold a connection
/// string to central's database, which violates ScadaBridge's standing rule that sites talk to
/// central, not to central's DB. Pull-only is a property of the wire contract rather than a
/// setting: the proto carries no write RPC, so a site cannot push even if misconfigured, and
/// secrets therefore originate at central only.
/// </para>
/// <para>
/// <b>Bootstrap constraint.</b> The hub connection string is itself a secret, and it can never come
/// from the hub — a node cannot read the hub to learn how to reach the hub. It must arrive from
/// outside the replicated set: an environment variable, or a <c>${secret:}</c> reference that is
/// seeded in that node's own LOCAL store (the pre-host expander in <c>Program.cs</c> runs against a
/// plain local SQLite store before the host container exists, so such a reference does resolve).
/// That expander is deliberately left un-replicated for exactly this reason.
/// <b>The SQL-Server gate is not a style preference.</b> <c>AddZbSecretsSqlServerReplication</c>
/// validates its options EAGERLY at registration time, so calling it unconditionally would throw at
/// startup on every node where <c>Secrets:SqlServer:ConnectionString</c> is unset — which is every
/// dev box, every docker node and every existing deployment. Both the explicit
/// <c>Secrets:Replication:Enabled</c> flag and a non-blank connection string are therefore required
/// before the SQL-Server package is touched at all; otherwise this is byte-for-byte the plain
/// local-SQLite registration it has always been.
/// </para>
/// <para>
/// <b>The gRPC mode deliberately does NOT degrade the same way.</b> A missing connection string in
/// SQL-Server mode falls back to a local-only store with a loud warning, and that behaviour is
/// preserved untouched. gRPC mode instead fails at startup when its endpoint or bearer token is
/// missing, because the fallback is the exact outcome the hub exists to prevent — a site serving
/// secrets that silently never converge with central — and because selecting the mode takes an
/// explicit second key, so an operator who typed it meant it.
/// </para>
/// <para>
/// <b>Bootstrap constraint.</b> A replication credential can never itself come from the replicated
/// set — a node cannot read the hub to learn how to reach the hub. The SQL hub connection string
/// must arrive from outside it: an environment variable, or a <c>${secret:}</c> reference seeded in
/// that node's own LOCAL store (the pre-host expander in <c>Program.cs</c> runs against a plain
/// local SQLite store before the host container exists, so such a reference does resolve). The
/// same holds for <c>Secrets:GrpcHub:BearerToken</c>, which is supplied from appsettings or the
/// environment exactly as the mesh pre-shared keys are. That expander is deliberately left
/// un-replicated for exactly this reason.
/// </para>
/// </remarks>
public static class SecretsRegistration
@@ -40,27 +96,131 @@ public static class SecretsRegistration
/// <summary>Configuration section holding the SQL-Server hub options.</summary>
public const string SqlServerSectionPath = "Secrets:SqlServer";
/// <summary>
/// Configuration section holding the gRPC hub options. One section serves both halves: central
/// reads <c>BearerToken</c> + <c>MaxNamesPerRequest</c> from it, a site reads <c>Endpoint</c>,
/// <c>BearerToken</c> and the sweep timings.
/// </summary>
public const string GrpcHubSectionPath = "Secrets:GrpcHub";
/// <summary>Configuration key gating clustered replication. Absent or false = off.</summary>
public const string ReplicationEnabledKey = "Secrets:Replication:Enabled";
/// <summary>
/// Configuration key selecting the replication transport. Absent or blank =
/// <see cref="SecretsReplicationMode.SqlServer"/>, the historical behaviour.
/// </summary>
public const string ReplicationModeKey = "Secrets:Replication:Mode";
/// <summary>Configuration key holding the shared SQL-Server hub connection string.</summary>
public const string HubConnectionStringKey = "Secrets:SqlServer:ConnectionString";
/// <summary>
/// Reads <see cref="ReplicationModeKey"/>. An absent or blank value is
/// <see cref="SecretsReplicationMode.SqlServer"/>; anything unrecognised throws.
/// </summary>
/// <remarks>
/// The value is validated whenever it is present, even with replication switched off. A typo
/// only surfaces at the boot that introduces it that way — deferring the check to whenever
/// somebody later flips <see cref="ReplicationEnabledKey"/> would fail a node in an unrelated
/// change, pointing at a key nobody touched.
/// </remarks>
/// <param name="config">Application configuration.</param>
/// <returns>The selected mode.</returns>
/// <exception cref="InvalidOperationException">The configured value is not a known mode.</exception>
public static SecretsReplicationMode ResolveReplicationMode(IConfiguration config)
{
ArgumentNullException.ThrowIfNull(config);
var raw = config[ReplicationModeKey];
if (string.IsNullOrWhiteSpace(raw))
{
return SecretsReplicationMode.SqlServer;
}
if (!Enum.TryParse(raw.Trim(), ignoreCase: true, out SecretsReplicationMode mode)
|| !Enum.IsDefined(mode))
{
throw new InvalidOperationException(
$"{ReplicationModeKey} is '{raw}', which is not a known secrets replication mode. "
+ $"Valid values are '{nameof(SecretsReplicationMode.SqlServer)}' and "
+ $"'{nameof(SecretsReplicationMode.Grpc)}'.");
}
return mode;
}
/// <summary>
/// Whether this node participates in the pull-only gRPC secrets hub.
/// </summary>
/// <remarks>
/// The single predicate behind BOTH the service registration and
/// <see cref="MapScadaBridgeSecretsHub"/>, so the two can never disagree. That matters more
/// than tidiness: mapping the hub endpoint without registering it would map a service whose
/// fail-closed bearer interceptor was never attached — an unauthenticated hub serving every
/// secret central holds.
/// </remarks>
/// <param name="config">Application configuration.</param>
/// <returns><see langword="true"/> when replication is enabled in gRPC mode.</returns>
public static bool UsesGrpcHub(IConfiguration config)
{
ArgumentNullException.ThrowIfNull(config);
return config.GetValue<bool>(ReplicationEnabledKey)
&& ResolveReplicationMode(config) == SecretsReplicationMode.Grpc;
}
/// <summary>
/// Registers the host container's secret store: a plain local SQLite store by default, or a
/// local store replicating against a shared SQL-Server hub when replication is explicitly
/// enabled and a hub connection string is present.
/// replicating topology when <see cref="ReplicationEnabledKey"/> is set — a shared SQL-Server
/// hub, or the pull-only gRPC hub whose half is chosen by <paramref name="role"/>.
/// </summary>
/// <param name="services">The service collection to register into.</param>
/// <param name="config">Application configuration for options binding.</param>
/// <param name="role">Which half of a clustered topology this node composes.</param>
/// <returns>The same <paramref name="services"/> instance, for chaining.</returns>
public static IServiceCollection AddScadaBridgeSecrets(
this IServiceCollection services,
IConfiguration config)
IConfiguration config,
SecretsNodeRole role)
{
ArgumentNullException.ThrowIfNull(services);
ArgumentNullException.ThrowIfNull(config);
// UsesGrpcHub is the single predicate shared with MapScadaBridgeSecretsHub — the branch
// condition must never be re-derived here, or registration and endpoint mapping could
// disagree about whether the hub is on.
if (UsesGrpcHub(config))
{
// Neither gRPC extension registers the local store — pull-only means local writes
// never leave the node, so there is nothing to decorate and the store stays exactly as
// configured. Register it first; both halves resolve it.
services.AddZbSecrets(config, SecretsSectionPath);
switch (role)
{
case SecretsNodeRole.Central:
// Throws when Secrets:GrpcHub:BearerToken is unset. Deliberately not caught: a
// hub that starts and refuses every follower looks like a network fault from
// the site end, and would be discovered as a stale secret rather than as a
// boot failure.
services.AddZbSecretsGrpcHub(config, GrpcHubSectionPath);
break;
case SecretsNodeRole.Site:
// Throws when Secrets:GrpcHub:Endpoint or :BearerToken is unset. Same reasoning.
services.AddZbSecretsGrpcHubClient(config, GrpcHubSectionPath);
break;
default:
throw new ArgumentOutOfRangeException(nameof(role), role,
"Unknown SecretsNodeRole — a new role must decide explicitly which half " +
"of the gRPC hub it composes rather than fall into either branch.");
}
return services;
}
// Reached with the mode already validated (UsesGrpcHub resolves it on every path), so a
// false answer here means SqlServer mode or replication off — never an unvetted mode value.
var enabled = config.GetValue<bool>(ReplicationEnabledKey);
var connectionString = config[HubConnectionStringKey];
@@ -89,4 +249,34 @@ public static class SecretsRegistration
return services.AddZbSecrets(config, SecretsSectionPath);
}
/// <summary>
/// Maps the central-hosted secrets hub endpoint, but only on a node whose configuration
/// actually registered it. A no-op otherwise.
/// </summary>
/// <remarks>
/// Central-only by construction: <see cref="AddScadaBridgeSecrets"/> registers the hub service
/// and its bearer interceptor solely for <see cref="SecretsNodeRole.Central"/>, and a site node
/// never reaches this call. The endpoint lands on the shared endpoint routing, so it is served
/// on the dedicated h2c control-plane listener (<c>ScadaBridge:Node:CentralGrpcPort</c>,
/// default 8083) — the same listener and the same addressing convention sites already use to
/// reach <c>CentralControlService</c>.
/// </remarks>
/// <param name="endpoints">The endpoint route builder.</param>
/// <param name="config">Application configuration.</param>
/// <returns>The same <paramref name="endpoints"/> instance, for chaining.</returns>
public static IEndpointRouteBuilder MapScadaBridgeSecretsHub(
this IEndpointRouteBuilder endpoints,
IConfiguration config)
{
ArgumentNullException.ThrowIfNull(endpoints);
ArgumentNullException.ThrowIfNull(config);
if (!UsesGrpcHub(config))
{
return endpoints;
}
return endpoints.MapZbSecretsHub();
}
}
src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs
+10 -5
View File
@@ -105,11 +105,16 @@ public static class SiteServiceRegistration
services.AddZbLocalDbReplication(config);
services.AddDataConnectionLayer();
// Local SQLite store by default; a local store replicating against a shared SQL-Server hub
// only when Secrets:Replication:Enabled is true AND a hub connection string is present.
// Hub mode matters most here: a site node keeps resolving secrets from its local store
// straight through a WAN outage to central. See SecretsRegistration.
services.AddScadaBridgeSecrets(config);
// Local SQLite store by default. With Secrets:Replication:Enabled true it becomes either a
// local store synced against a shared SQL-Server hub (Mode=SqlServer) or — the production
// topology, scadaproj#3 — the FOLLOWER half of the pull-only gRPC secrets hub (Mode=Grpc),
// sweeping central over the same gRPC path the control plane already uses instead of
// holding a connection string to central's database. A site never hosts the hub; passing
// the role explicitly is what makes that a property of this composition root.
//
// Either way the node keeps a FULL local store, so it resolves secrets straight through a
// WAN outage to central and simply converges late. See SecretsRegistration.
services.AddScadaBridgeSecrets(config, SecretsNodeRole.Site);
// Adapter that surfaces the site id to
// StoreAndForwardService through DI WITHOUT introducing a
// StoreAndForward → HealthMonitoring project-reference cycle. Must be
src/ZB.MOM.WW.ScadaBridge.Host/ZB.MOM.WW.ScadaBridge.Host.csproj
+4
View File
@@ -41,6 +41,10 @@
<PackageReference Include="ZB.MOM.WW.Secrets.Abstractions" />
<PackageReference Include="ZB.MOM.WW.Secrets.Ui" />
<PackageReference Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" />
<!-- Both halves of the pull-only central secrets hub ship in one package: central
hosts the gRPC service, a site runs the sweep client. Which half a node composes
is decided at registration time by its role — see SecretsRegistration. -->
<PackageReference Include="ZB.MOM.WW.Secrets.Replicator.Grpc" />
<PackageReference Include="ZB.MOM.WW.LocalDb" />
<PackageReference Include="ZB.MOM.WW.LocalDb.Replication" />
</ItemGroup>
src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json
+17 -2
View File
@@ -14,13 +14,28 @@
"MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" },
"RunMigrationsOnStartup": true,
"ResolveCacheTtl": "00:00:30",
"Replication": { "Enabled": false },
"Replication": {
"_comment": "Default-OFF pin: no deployment enables clustered secret replication by shipping this file. Mode selects the transport when Enabled is true - 'SqlServer' (the default, kept so an existing config that sets only Enabled behaves exactly as before) or 'Grpc' (the PRODUCTION topology, scadaproj#3: central hosts a pull-only hub under Secrets:GrpcHub and sites sweep it, so a site never needs a connection string to central's database). Every node in either topology must carry the SAME ZB_SECRETS_MASTER_KEY - only ciphertext crosses the wire, so a node with a different KEK fails closed on resolve with a kek_id mismatch that reads like corruption but is a deployment error.",
"Enabled": false,
"Mode": "SqlServer"
},
"SqlServer": {
"ConnectionString": "",
"SchemaName": "zbsecrets",
"SyncInterval": "00:00:30",
"SyncOnStartup": true,
"_comment": "Hub connstr must be seeded in the LOCAL store (or supplied via env) - it cannot come from the hub itself. Enable via Secrets:Replication:Enabled."
"_comment": "Hub connstr must be seeded in the LOCAL store (or supplied via env) - it cannot come from the hub itself. Enable via Secrets:Replication:Enabled + Mode=SqlServer."
},
"GrpcHub": {
"_comment": "Read ONLY when Secrets:Replication:Enabled is true AND Mode=Grpc; inert otherwise. ONE section, both halves: CENTRAL reads BearerToken + MaxNamesPerRequest and hosts the hub on its CentralGrpcPort h2c listener (default 8083, alongside CentralControlService); a SITE reads Endpoint + BearerToken + the sweep timings and pulls. Replication is pull-only by wire contract - the proto has no write RPC - so secrets originate at central and a site cannot push. FAIL-CLOSED: unlike SqlServer mode there is no local-only fallback; a missing BearerToken (either role) or Endpoint (site) is a startup failure naming the key.",
"_bearerToken": "Shared credential every follower presents. Supply it from appsettings or the environment (Secrets__GrpcHub__BearerToken), NOT as a ${secret:...} reference - resolving that reference is what the hub exists to make possible, so it cannot come from the hub. Same rationale and same handling as the mesh pre-shared keys. Never commit a real value here; the empty default below is fail-closed, not open.",
"BearerToken": "",
"_endpoint": "SITE ONLY. Absolute http/https URI of a CENTRAL node's gRPC (h2c) port - the same address family as ScadaBridge:Communication:CentralGrpcEndpoints, e.g. 'http://central-a-host:8083'. NOT via Traefik (HTTP/1 only). NOTE the asymmetry with CentralGrpcEndpoints: that is a LIST and fails over across the central pair, whereas the hub client dials a SINGLE endpoint. A sweep against a downed central-a therefore stalls rather than failing over - which is survivable because the sweep is best-effort (one warning per interval, then retry) and the site keeps serving its full local last-known-good store, but it does mean secrets stop converging until that central node returns.",
"Endpoint": "",
"SyncInterval": "00:00:30",
"SyncOnStartup": true,
"CallDeadline": "00:00:30",
"MaxNamesPerRequest": 1000
}
},
"Serilog": {
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SecretsReplicationWiringTests.cs
+387 -15
View File
@@ -1,8 +1,11 @@
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Routing;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using ZB.MOM.WW.Secrets.Abstractions;
using ZB.MOM.WW.Secrets.Replication;
using ZB.MOM.WW.Secrets.Replicator.Grpc;
using ZB.MOM.WW.Secrets.Sqlite;
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
@@ -20,7 +23,9 @@ namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// Registration must not need a reachable SQL Server: the SQL-Server package validates its options
/// eagerly, but validation only checks that the connection string is non-empty and the schema name
/// is a legal identifier. Nothing connects until a sweep or a write runs, and these tests do
/// neither — so the dummy connection string below never resolves to a real host.
/// neither — so the dummy connection string below never resolves to a real host. The same holds for
/// the gRPC hub client: <c>GrpcChannel.ForAddress</c> performs no I/O, so the dummy endpoint is
/// never dialled either.
/// </para>
/// </summary>
public class SecretsReplicationWiringTests
@@ -29,6 +34,12 @@ public class SecretsReplicationWiringTests
private const string DummyConnectionString =
"Server=unused;Database=x;Integrated Security=true;";
// Syntactically valid, deliberately unreachable. Never dialled by these tests.
private const string DummyHubEndpoint = "http://unused.invalid:8083";
// Not a credential — a non-blank placeholder, which is all the fail-closed validators check.
private const string DummyBearerToken = "test-hub-token";
private static IConfiguration BuildConfig(params (string Key, string Value)[] overrides)
{
var settings = new Dictionary<string, string?>
@@ -47,11 +58,13 @@ public class SecretsReplicationWiringTests
return new ConfigurationBuilder().AddInMemoryCollection(settings).Build();
}
private static ServiceProvider BuildProvider(IConfiguration config)
private static ServiceProvider BuildProvider(
IConfiguration config,
SecretsNodeRole role = SecretsNodeRole.Central)
{
var services = new ServiceCollection();
services.AddLogging();
services.AddScadaBridgeSecrets(config);
services.AddScadaBridgeSecrets(config, role);
return services.BuildServiceProvider(validateScopes: true);
}
@@ -59,11 +72,40 @@ public class SecretsReplicationWiringTests
/// Any hosted service contributed by the SQL-Server replicator package. Asserted by assembly
/// rather than by type because <c>SqlServerHubMigrationHostedService</c> is internal to it.
/// </summary>
private static bool IsReplicatorHostedService(ServiceDescriptor descriptor) =>
private static bool IsSqlServerReplicatorHostedService(ServiceDescriptor descriptor) =>
descriptor.ServiceType == typeof(IHostedService)
&& descriptor.ImplementationType?.Assembly.GetName().Name
== "ZB.MOM.WW.Secrets.Replicator.SqlServer";
/// <summary>
/// The gRPC hub package's one hosted service — the follower sweep. Asserted by assembly and
/// type NAME because <c>GrpcSecretSyncService</c> is internal to the package. The hub SERVER
/// half contributes no hosted service at all, so presence of one from this assembly is
/// specifically the follower half and nothing else.
/// </summary>
private static bool IsGrpcHubSweepService(IHostedService service) =>
service.GetType().Assembly.GetName().Name == "ZB.MOM.WW.Secrets.Replicator.Grpc"
&& service.GetType().Name == "GrpcSecretSyncService";
private static (string Key, string Value)[] GrpcCentralConfig() =>
[
("Secrets:Replication:Enabled", "true"),
("Secrets:Replication:Mode", "Grpc"),
("Secrets:GrpcHub:BearerToken", DummyBearerToken),
];
private static (string Key, string Value)[] GrpcSiteConfig() =>
[
("Secrets:Replication:Enabled", "true"),
("Secrets:Replication:Mode", "Grpc"),
("Secrets:GrpcHub:Endpoint", DummyHubEndpoint),
("Secrets:GrpcHub:BearerToken", DummyBearerToken),
];
// ---------------------------------------------------------------------------------------
// Replication off — the default, and the shape every existing deployment is in.
// ---------------------------------------------------------------------------------------
[Fact]
public void ReplicationDisabledByDefault_ResolvesPlainSqliteStore()
{
@@ -80,11 +122,40 @@ public class SecretsReplicationWiringTests
{
var services = new ServiceCollection();
services.AddLogging();
services.AddScadaBridgeSecrets(BuildConfig());
services.AddScadaBridgeSecrets(BuildConfig(), SecretsNodeRole.Central);
Assert.DoesNotContain(services, IsReplicatorHostedService);
Assert.DoesNotContain(services, IsSqlServerReplicatorHostedService);
}
/// <summary>
/// Flag off ⇒ NEITHER half of the gRPC hub, on EITHER role — even with the hub section fully
/// populated. The mode key selects a transport; it must never be what turns replication on.
/// </summary>
[Theory]
[InlineData(SecretsNodeRole.Central)]
[InlineData(SecretsNodeRole.Site)]
public void ReplicationDisabled_WithGrpcModeAndFullHubConfig_WiresNeitherHubHalf(
SecretsNodeRole role)
{
IConfiguration config = BuildConfig(
("Secrets:Replication:Enabled", "false"),
("Secrets:Replication:Mode", "Grpc"),
("Secrets:GrpcHub:Endpoint", DummyHubEndpoint),
("Secrets:GrpcHub:BearerToken", DummyBearerToken));
using ServiceProvider provider = BuildProvider(config, role);
Assert.IsType<SqliteSecretStore>(provider.GetRequiredService<ISecretStore>());
Assert.Null(provider.GetService<SecretsHubAuthInterceptor>());
Assert.DoesNotContain(provider.GetServices<IHostedService>(), IsGrpcHubSweepService);
Assert.False(SecretsRegistration.UsesGrpcHub(config));
}
// ---------------------------------------------------------------------------------------
// SQL-Server mode — regression pins. Behaviour must be byte-for-byte what it was before the
// mode key existed, both when the mode is left unset and when it is named explicitly.
// ---------------------------------------------------------------------------------------
[Fact]
public void ReplicationEnabledWithoutConnectionString_StaysOnPlainSqliteStore()
{
@@ -96,20 +167,27 @@ public class SecretsReplicationWiringTests
IConfiguration config = BuildConfig(("Secrets:Replication:Enabled", "true"));
services.AddScadaBridgeSecrets(config);
services.AddScadaBridgeSecrets(config, SecretsNodeRole.Central);
using ServiceProvider provider = services.BuildServiceProvider(validateScopes: true);
Assert.IsType<SqliteSecretStore>(provider.GetRequiredService<ISecretStore>());
Assert.DoesNotContain(services, IsReplicatorHostedService);
Assert.DoesNotContain(services, IsSqlServerReplicatorHostedService);
}
[Fact]
public void ReplicationEnabledWithConnectionString_ResolvesReplicatingStore()
[Theory]
[InlineData(null)]
[InlineData("SqlServer")]
public void ReplicationEnabledWithConnectionString_ResolvesReplicatingStore(string? mode)
{
IConfiguration config = BuildConfig(
("Secrets:Replication:Enabled", "true"),
("Secrets:SqlServer:ConnectionString", DummyConnectionString));
IConfiguration config = mode is null
? BuildConfig(
("Secrets:Replication:Enabled", "true"),
("Secrets:SqlServer:ConnectionString", DummyConnectionString))
: BuildConfig(
("Secrets:Replication:Enabled", "true"),
("Secrets:Replication:Mode", mode),
("Secrets:SqlServer:ConnectionString", DummyConnectionString));
using ServiceProvider provider = BuildProvider(config);
@@ -142,9 +220,303 @@ public class SecretsReplicationWiringTests
("Secrets:Replication:Enabled", "true"),
("Secrets:SqlServer:ConnectionString", DummyConnectionString));
services.AddScadaBridgeSecrets(config);
services.AddScadaBridgeSecrets(config, SecretsNodeRole.Central);
// Both the hub-schema migration service and the bidirectional sweep.
Assert.Equal(2, services.Count(IsReplicatorHostedService));
Assert.Equal(2, services.Count(IsSqlServerReplicatorHostedService));
}
/// <summary>
/// SQL-Server mode is role-agnostic — both nodes of a hub topology sync bidirectionally
/// against the same shared database. The role parameter added for the gRPC hub must not have
/// quietly changed that.
/// </summary>
[Theory]
[InlineData(SecretsNodeRole.Central)]
[InlineData(SecretsNodeRole.Site)]
public void SqlServerMode_IsIdenticalOnBothRoles(SecretsNodeRole role)
{
IConfiguration config = BuildConfig(
("Secrets:Replication:Enabled", "true"),
("Secrets:Replication:Mode", "SqlServer"),
("Secrets:SqlServer:ConnectionString", DummyConnectionString));
using ServiceProvider provider = BuildProvider(config, role);
Assert.IsType<ReplicatingSecretStore>(provider.GetRequiredService<ISecretStore>());
Assert.Null(provider.GetService<SecretsHubAuthInterceptor>());
Assert.DoesNotContain(provider.GetServices<IHostedService>(), IsGrpcHubSweepService);
}
// ---------------------------------------------------------------------------------------
// gRPC mode — central hosts, site follows, and neither does the other's job.
// ---------------------------------------------------------------------------------------
[Fact]
public void GrpcMode_Central_RegistersHubAndItsAuthInterceptor()
{
IConfiguration config = BuildConfig(GrpcCentralConfig());
using ServiceProvider provider = BuildProvider(config, SecretsNodeRole.Central);
// The fail-closed bearer gate must RESOLVE, not merely be listed: an interceptor that
// cannot be activated is one gRPC would fail to attach, which is an ungated hub.
Assert.NotNull(provider.GetService<SecretsHubAuthInterceptor>());
Assert.True(SecretsRegistration.UsesGrpcHub(config));
// The local store is still the plain one — pull-only means nothing decorates it.
Assert.IsType<SqliteSecretStore>(provider.GetRequiredService<ISecretStore>());
}
[Fact]
public void GrpcMode_Central_DoesNotRegisterTheFollowerSweep()
{
// Central is the origin of every write. A sweep here would have it pull from itself, and
// on a misconfigured endpoint, from somebody else.
using ServiceProvider provider =
BuildProvider(BuildConfig(GrpcCentralConfig()), SecretsNodeRole.Central);
Assert.DoesNotContain(provider.GetServices<IHostedService>(), IsGrpcHubSweepService);
}
[Fact]
public void GrpcMode_Site_RegistersTheFollowerSweep()
{
using ServiceProvider provider =
BuildProvider(BuildConfig(GrpcSiteConfig()), SecretsNodeRole.Site);
// Resolving the hosted services builds the sweep's whole graph — reader, keyed channel,
// local store — which is where a missing AddZbSecrets would surface.
Assert.Contains(provider.GetServices<IHostedService>(), IsGrpcHubSweepService);
Assert.IsType<SqliteSecretStore>(provider.GetRequiredService<ISecretStore>());
}
[Fact]
public void GrpcMode_Site_DoesNotHostTheHub()
{
// A site hosting the hub would serve central's secrets to anything holding the shared
// token, from inside the site network.
using ServiceProvider provider =
BuildProvider(BuildConfig(GrpcSiteConfig()), SecretsNodeRole.Site);
Assert.Null(provider.GetService<SecretsHubAuthInterceptor>());
}
// ---------------------------------------------------------------------------------------
// gRPC mode fails closed. There is no local-only fallback here, unlike SQL-Server mode:
// a site quietly serving secrets that never converge is the outcome the hub exists to prevent.
// ---------------------------------------------------------------------------------------
[Fact]
public void GrpcMode_Central_WithoutBearerToken_FailsAtRegistration()
{
var services = new ServiceCollection();
services.AddLogging();
IConfiguration config = BuildConfig(
("Secrets:Replication:Enabled", "true"),
("Secrets:Replication:Mode", "Grpc"));
var ex = Assert.Throws<InvalidOperationException>(
() => services.AddScadaBridgeSecrets(config, SecretsNodeRole.Central));
// The package's own message, unwrapped, naming the exact key an operator has to set.
Assert.Contains("Secrets:GrpcHub:BearerToken", ex.Message, StringComparison.Ordinal);
}
[Fact]
public void GrpcMode_Site_WithoutEndpoint_FailsAtRegistration()
{
var services = new ServiceCollection();
services.AddLogging();
IConfiguration config = BuildConfig(
("Secrets:Replication:Enabled", "true"),
("Secrets:Replication:Mode", "Grpc"),
("Secrets:GrpcHub:BearerToken", DummyBearerToken));
var ex = Assert.Throws<InvalidOperationException>(
() => services.AddScadaBridgeSecrets(config, SecretsNodeRole.Site));
Assert.Contains("Secrets:GrpcHub:Endpoint", ex.Message, StringComparison.Ordinal);
}
[Fact]
public void GrpcMode_Site_WithoutBearerToken_FailsAtRegistration()
{
var services = new ServiceCollection();
services.AddLogging();
IConfiguration config = BuildConfig(
("Secrets:Replication:Enabled", "true"),
("Secrets:Replication:Mode", "Grpc"),
("Secrets:GrpcHub:Endpoint", DummyHubEndpoint));
var ex = Assert.Throws<InvalidOperationException>(
() => services.AddScadaBridgeSecrets(config, SecretsNodeRole.Site));
Assert.Contains("Secrets:GrpcHub:BearerToken", ex.Message, StringComparison.Ordinal);
}
// ---------------------------------------------------------------------------------------
// Mode key parsing.
// ---------------------------------------------------------------------------------------
[Theory]
[InlineData(null, SecretsReplicationMode.SqlServer)]
[InlineData("", SecretsReplicationMode.SqlServer)]
[InlineData("SqlServer", SecretsReplicationMode.SqlServer)]
[InlineData("sqlserver", SecretsReplicationMode.SqlServer)]
[InlineData("Grpc", SecretsReplicationMode.Grpc)]
[InlineData("grpc", SecretsReplicationMode.Grpc)]
[InlineData(" Grpc ", SecretsReplicationMode.Grpc)]
public void ResolveReplicationMode_ReadsTheKey(string? value, SecretsReplicationMode expected)
{
IConfiguration config = value is null
? BuildConfig()
: BuildConfig(("Secrets:Replication:Mode", value));
Assert.Equal(expected, SecretsRegistration.ResolveReplicationMode(config));
}
[Fact]
public void ResolveReplicationMode_RejectsAnUnknownValue_EvenWithReplicationOff()
{
// Caught at the boot that introduces the typo rather than at whichever later boot flips
// Secrets:Replication:Enabled — otherwise an unrelated change fails on a key nobody touched.
IConfiguration config = BuildConfig(("Secrets:Replication:Mode", "GrpcHub"));
var ex = Assert.Throws<InvalidOperationException>(
() => SecretsRegistration.ResolveReplicationMode(config));
Assert.Contains("Secrets:Replication:Mode", ex.Message, StringComparison.Ordinal);
}
/// <summary>
/// A numeric value must not be accepted. <c>Enum.TryParse</c> happily parses any integer,
/// including ones outside the enum, so "2" would otherwise select a mode that does not exist
/// and fall through the switch to the SQL-Server branch.
/// </summary>
[Fact]
public void ResolveReplicationMode_RejectsAnOutOfRangeNumericValue()
{
IConfiguration config = BuildConfig(("Secrets:Replication:Mode", "7"));
Assert.Throws<InvalidOperationException>(
() => SecretsRegistration.ResolveReplicationMode(config));
}
}
/// <summary>
/// Endpoint-mapping pins for the central-hosted secrets hub.
/// <para>
/// Registration and mapping are gated by one shared predicate on purpose, and this asserts the
/// consequence: the hub endpoint exists exactly when the node that serves it registered the
/// fail-closed interceptor. A mapped-but-unregistered hub would be an anonymous endpoint serving
/// every secret central holds, and it would look completely healthy.
/// </para>
/// </summary>
public class SecretsHubEndpointMappingTests
{
private const string DummyBearerToken = "test-hub-token";
private static WebApplication BuildCentralApp(params (string Key, string Value)[] overrides)
{
var builder = WebApplication.CreateBuilder();
builder.Configuration.Sources.Clear();
var settings = new Dictionary<string, string?>
{
["Secrets:SqlitePath"] = Path.Combine(
Path.GetTempPath(), $"sb-secrets-map-test-{Guid.NewGuid():N}.db"),
["Secrets:MasterKey:Source"] = "Environment",
["Secrets:MasterKey:EnvVarName"] = "ZB_SECRETS_MASTER_KEY",
["Secrets:RunMigrationsOnStartup"] = "false",
};
foreach ((string key, string value) in overrides)
{
settings[key] = value;
}
builder.Configuration.AddInMemoryCollection(settings);
builder.Services.AddGrpc();
builder.Services.AddScadaBridgeSecrets(builder.Configuration, SecretsNodeRole.Central);
WebApplication app = builder.Build();
app.MapScadaBridgeSecretsHub(app.Configuration);
return app;
}
/// <summary>
/// The gRPC service's WIRE name, from <c>package zb.mom.ww.secrets.hub.v1; service SecretsHub</c>.
/// Matched on the route pattern rather than on the C# type, because the route is the thing a
/// follower actually addresses and the thing a rename would break.
/// </summary>
private const string HubRoutePrefix = "/zb.mom.ww.secrets.hub.v1.SecretsHub/";
private static IReadOnlyList<string> HubRoutes(WebApplication app) =>
((IEndpointRouteBuilder)app).DataSources
.SelectMany(source => source.Endpoints)
.OfType<RouteEndpoint>()
.Select(endpoint => "/" + endpoint.RoutePattern.RawText?.TrimStart('/'))
.Where(route => route.StartsWith(HubRoutePrefix, StringComparison.Ordinal))
// Grpc.AspNetCore always adds a "{unimplementedMethod}" catch-all per service, which
// answers Unimplemented. It is framework scaffolding, not an RPC the contract declares.
.Where(route => !route.Contains('{', StringComparison.Ordinal))
.ToList();
private static bool HasHubEndpoint(WebApplication app) => HubRoutes(app).Count > 0;
[Fact]
public void GrpcMode_Central_MapsTheHubEndpoint()
{
using WebApplication app = BuildCentralApp(
("Secrets:Replication:Enabled", "true"),
("Secrets:Replication:Mode", "Grpc"),
("Secrets:GrpcHub:BearerToken", DummyBearerToken));
Assert.True(HasHubEndpoint(app));
}
/// <summary>
/// Pull-only is a property of the wire contract, and this is where it becomes observable to
/// ScadaBridge: central maps exactly the two READ methods and nothing that could accept a
/// pushed row. If a future package version adds a write RPC, this fails here rather than
/// silently opening a path for a site to overwrite central's secrets.
/// </summary>
[Fact]
public void GrpcMode_Central_MapsOnlyTheTwoReadMethods()
{
using WebApplication app = BuildCentralApp(
("Secrets:Replication:Enabled", "true"),
("Secrets:Replication:Mode", "Grpc"),
("Secrets:GrpcHub:BearerToken", DummyBearerToken));
Assert.Equal(
[HubRoutePrefix + "GetManifest", HubRoutePrefix + "GetSecrets"],
HubRoutes(app).Order(StringComparer.Ordinal).ToArray());
}
[Fact]
public void ReplicationDisabled_MapsNoHubEndpoint()
{
using WebApplication app = BuildCentralApp(
("Secrets:Replication:Enabled", "false"),
("Secrets:Replication:Mode", "Grpc"),
("Secrets:GrpcHub:BearerToken", DummyBearerToken));
Assert.False(HasHubEndpoint(app));
}
[Fact]
public void SqlServerMode_MapsNoHubEndpoint()
{
using WebApplication app = BuildCentralApp(
("Secrets:Replication:Enabled", "true"),
("Secrets:Replication:Mode", "SqlServer"),
("Secrets:SqlServer:ConnectionString", "Server=unused;Database=x;Integrated Security=true;"));
Assert.False(HasHubEndpoint(app));
}
}