ScadaBridge/docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md

Secrets gRPC hub — live gate on the docker cluster

Result: PASS — 4/4 checks. Run 2026-08-07 on the local 8-node docker cluster.

Run in two passes. The first pass (image 2958aa2c2bbb, secrets libs 0.4.0) passed checks 1–3 and failed one clause of check 4: denials were recorded at Information, not Warning. That was a property of the library, not of this branch, so it was fixed at the library and shipped as 0.4.1 (scadaproj main c86cead). The second pass (image f866208c89e8, secrets libs 0.4.1) re-ran check 4 in full and re-smoked convergence to prove the new image still converges. Both passes are recorded below, first-pass evidence kept intact — the fix is only meaningful against the failure it answers.

Purpose

Terminal gate (Task 7) for feat/secrets-grpc-hub, which wires ZB.MOM.WW.Secrets.Replicator.Grpc 0.4.0 into ScadaBridge: the central role hosts a pull-only secrets_hub.v1 gRPC hub on its existing h2c control-plane listener, and a site node runs a sweep that converges its own local SQLite store from that hub.

This is the ScadaBridge half of the production-topology decision in scadaproj#3 (design: scadaproj/docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md). The SqlServer replicator is not the production path here: it requires every site node to hold a connection string to central's database, which breaks ScadaBridge's standing rule that sites talk to central, not to central's DB.

Scope: rig config only. No product code was changed by this gate — see Defects.

Rig state

Fact	First pass	Second pass
Repo / branch	ScadaBridge @ feat/secrets-grpc-hub	same
Git SHA gated	fc784b413713b1643a21d573e55c00bcc05bd8ee (fc784b41)	6d38e89be0b39a2224559dd50922751ccd8df2ed (6d38e89b)
Image	scadabridge:latest = 2958aa2c2bbb, built 11:39 UTC	scadabridge:latest = f866208c89e8, built 12:41 UTC
ZB.MOM.WW.ScadaBridge.Host.dll	09cf76dc550d32fe… — byte-identical on all 8 nodes	bc8f17e7b5d6982c… — byte-identical on all 8 nodes
Secrets libs	ZB.MOM.WW.Secrets* 0.4.0	ZB.MOM.WW.Secrets* 0.4.1
Working tree at build	only docker/docker-compose.yml modified (rig config, not in the image) + untracked .claude/	clean but for untracked .claude/

Common to both passes:

Fact	Value
Hub enabled on	central-a, central-b (host) and site-a-a, site-a-b (followers)
Hub NOT enabled on	site-b-a, site-b-b, site-c-a, site-c-b — no Secrets__* env, no store file, no hub/sweep log line (verified)
Sweep interval	00:00:30 — the product default, deliberately not shortened, so the timings below are real
KEK	one shared committed dev-only key, identical on all four (kek_id sha256 7451bcbc1f1f on every row observed)

site-b and site-c were left off on purpose: the default-OFF posture is proven side by side on one rig, the same way site-a is the rig's only LocalDb-replicated pair.

Rig-config change made by this gate

docker/docker-compose.yml gained two YAML anchors and applied them to four services:

• x-secrets-hub-env (central pair + site-a pair) — dev KEK, Secrets__SqlitePath, Secrets__Replication__Enabled=true, Secrets__Replication__Mode=Grpc, Secrets__GrpcHub__BearerToken.
• x-secrets-hub-site-env (site-a pair only) — Secrets__GrpcHub__Endpoint=http://scadabridge-central-a:8083.

All values are DEV-ONLY and committed under the same exception the mesh PSKs and InboundApi:ApiKeyPepper already use. Secrets__SqlitePath points at /app/data because the appsettings default is a relative path resolving to /app — inside the image's writable layer, destroyed by every container recreate. The central pair gained a ./central-node-*/data:/app/data volume for this; the site pairs already had one.

Method

Secrets were seeded, read and deleted with the ZB.MOM.WW.Secrets.Cli (secret) published from scadaproj/ZB.MOM.WW.Secrets (0.4.0) and run in a throwaway container mounting one node's data volume, with that node's KEK. This is the method the sibling OtOpcUa gate used the same day (OtOpcUa/docs/plans/2026-08-07-secrets-pair-local-live-gate.md).

The CentralUI /admin/secrets page was not used, and for this topology that costs nothing: the hub is a pure reader of central's store on every call, with no write-triggered path and no manifest cache. A row written by any process into central's store is on exactly the same path a row written through the page would be. (Contrast the Akka replicator, where the two differ — hence that gate's residual 1, which does not apply here.)

Ciphertext identity is evidenced as a SHA-256 over the encrypted columns only (ciphertext‖nonce‖tag‖wrapped_dek‖wrap_nonce‖wrap_tag), plus a hash of kek_id. No plaintext, no token and no key material appears in this document. The secret values used were throwaway smoke strings.

gRPC calls for check 4 were made with fullstorydev/grpcurl on the scadabridge-net network, against the committed secrets_hub.proto.

Results

#	Check	Pass	Result	Timing
1	Convergence — central write reaches both site-a nodes, byte-identical, decrypt-verified	1st (0.4.0)	PASS	5 s (budget 30 s + margin)
2	Cold-boot offline — site pair boots and serves last-known-good with central down; convergence resumes	1st (0.4.0)	PASS	resumed in 14 s / 43 s
3	Tombstone — delete propagates, survives a pair restart	1st (0.4.0)	PASS	≤ 9 s
4	Auth negatives + log hygiene	1st (0.4.0)	FAIL (1 clause of 3)	—
4	Auth negatives + log hygiene — re-run in full	2nd (0.4.1)	PASS	denial warned immediately
—	Convergence re-smoke on the 0.4.1 image	2nd (0.4.1)	PASS	17 s

Baseline before seeding: all four stores empty.

Check 1 — convergence · PASS

Seeded hub-gate-smoke-1 on central-a only at 11:46:57Z.

central-node-a   hub-gate-smoke-1  rev=0 del=0 cipher_sha256=00a14a2eb8de987fb75a4515 kek=7451bcbc1f1f updated=2026-08-07T11:46:57.1403922+00:00
site-a-node-a    hub-gate-smoke-1  rev=0 del=0 cipher_sha256=00a14a2eb8de987fb75a4515 kek=7451bcbc1f1f updated=2026-08-07T11:46:57.1403922+00:00
site-a-node-b    hub-gate-smoke-1  rev=0 del=0 cipher_sha256=00a14a2eb8de987fb75a4515 kek=7451bcbc1f1f updated=2026-08-07T11:46:57.1403922+00:00

Ciphertext hash, kek_id, revision and updated_utc are identical on all three — the row is relayed verbatim, not re-encrypted, so the last-writer-wins ordering key survives the hop. First poll after the seed (11:47:05Z) already showed both followers converged.

Both halves logged it, and central served the fetch:

site-a-a  [11:47:02 INF] Secret hub sync converged: pulled 1 row(s) from the central hub.
site-a-b  [11:47:02 INF] Secret hub sync converged: pulled 1 row(s) from the central hub.
central-a [11:47:02 INF] Request finished HTTP/2 POST .../SecretsHub/GetSecrets - 200 application/grpc 0.9899ms

Convergence = 11:46:57Z → 11:47:02Z = 5 s, one sweep tick.

Decrypt-verified on both followers: secret get hub-gate-smoke-1 returned the exact plaintext seeded on central-a, on site-a-a and site-a-b.

Check 2 — cold-boot offline · PASS

Central pair stopped 11:48:27Z → 11:48:37Z.

Sweep degrades, does not crash. One warning per interval on each follower, carrying the transport fault and nothing else:

site-a-a [11:49:02 WRN] Secret hub sync failed; the node continues serving its local store and will retry on the next interval.
         Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error connecting to subchannel." …)

Site pair restarted together (the supported pattern) 11:49:31Z → 11:50:12Z, with central still down. site-a-a booted at 11:49:42Z; its SyncOnStartup sweep warned at 11:49:42Z and retried at 11:50:12 / 11:50:42 / 11:51:12 / 11:51:42 / 11:52:12 / 11:52:42. Both containers running, RestartCount=0, zero FATAL / Unhandled exception / Application startup exception lines.

Last-known-good served with central down — at 11:51:08Z, secret get hub-gate-smoke-1 returned the correct plaintext on both site-a-a and site-a-b.

Central pair started 11:51:37Z. hub-gate-smoke-2 seeded on central-a at 11:52:29Z — after the site nodes were already up, so this proves convergence resumed rather than re-bootstrapped:

site-a-b [11:52:43 INF] Secret hub sync converged: pulled 1 row(s) from the central hub.   →  14 s
site-a-a [11:53:12 INF] Secret hub sync converged: pulled 1 row(s) from the central hub.   →  43 s

Both rows byte-identical to central's (cipher_sha256=4f022e222cfc2f8f77dcfb0b, updated=2026-08-07T11:52:29.6983529+00:00).

Honest note on the 43 s. site-a-a's first post-outage sweep (11:52:42Z) still failed, with SocketException: Name or service not known — a stale DNS/subchannel view of a container that had been docker stopped and docker started. It recovered unaided on the very next tick. So after a central outage, worst-case convergence on this rig is two sweep intervals, not one. Bounded, self-healing, no intervention — but recorded so the 5 s figure from check 1 is not read as the post-outage number.

Check 3 — tombstone · PASS

hub-gate-smoke-1 deleted on central-a at 11:54:05Z. hub-gate-smoke-2 left live as a control.

central-node-a  hub-gate-smoke-1  rev=1 del=1 updated=2026-08-07T11:54:05.9812744+00:00 deleted=2026-08-07T11:54:05.9812744+00:00
site-a-node-a   hub-gate-smoke-1  rev=1 del=1 updated=2026-08-07T11:54:05.9812744+00:00 deleted=2026-08-07T11:54:05.9812744+00:00
site-a-node-b   hub-gate-smoke-1  rev=1 del=1 updated=2026-08-07T11:54:05.9812744+00:00 deleted=2026-08-07T11:54:05.9812744+00:00

Both followers held the tombstone at the 11:54:14Z poll — ≤ 9 s. Delete is a soft tombstone (is_deleted=1, revision bumped, updated_utc restamped), which is what lets it win under last-writer-wins. secret get returned {"error":"not-found"} and secret list returned only hub-gate-smoke-2 on both followers.

No resurrection across a pair restart. Site pair restarted together 11:55:09Z → 11:55:50Z, with central up and sweeping the whole time — so the sweep had every opportunity to re-pull the row. Re-checked at 11:57:05Z (75 s / 2+ sweeps later):

• hub-gate-smoke-1 still rev=1 del=1 on both, updated_utc unchanged; secret get → {"error":"not-found"} on both.
• hub-gate-smoke-2 survived on both and decrypted to the correct plaintext on both.

Check 4, first pass (0.4.0) — auth negatives + log hygiene · FAIL (one clause)

Kept for the record; superseded by the second pass below.

Three calls to SecretsHub/GetManifest on scadabridge-central-a:8083 at 11:57:31–32Z:

Call	Result
no authorization header	Code: Unauthenticated / Message: Unauthenticated.
wrong bearer (Bearer not-the-hub-token)	Code: Unauthenticated / Message: Unauthenticated.
correct bearer (positive control)	200, manifest returned (2 entries)

PASS — identical denial. Both negatives are byte-identical in code and detail: a caller learns it was refused and nothing else. The positive control proves the endpoint is live and the refusals are the auth gate, not a dead route.

PASS — log hygiene, zero hits. Every one of the 8 nodes' docker logs and every on-disk Serilog file under docker/*/logs/ was grepped for the dev bearer token value, the dev KEK value, and both secret plaintexts:

central-a  token=0 kek=0 plain1=0 plain2=0 authz-header-mentions=0
central-b  token=0 kek=0 plain1=0 plain2=0 authz-header-mentions=0
site-a-a   token=0 kek=0 plain1=0 plain2=0 authz-header-mentions=0
site-a-b   token=0 kek=0 plain1=0 plain2=0 authz-header-mentions=0
site-b-a / site-b-b / site-c-a / site-c-b   all zero
on-disk log files: zero matching files for all four search strings

The token appears only where it legitimately may: the committed rig config (docker/docker-compose.yml) and the resulting container environment.

FAIL — no server-side warning. The criterion asks for a server-side warning on denial. What the hub actually emits is one Information-level line per denial, from Grpc.AspNetCore.Server, not from the hub's own gate:

[11:57:31 INF] Error status code 'Unauthenticated' with detail 'Unauthenticated.' raised.
[11:57:32 INF] Error status code 'Unauthenticated' with detail 'Unauthenticated.' raised.

Nothing at WRN or above was emitted by either denial — verified by dumping every log line in the 11:57:31–32Z window, not by a filtered grep.

This is not a wiring mistake and not fixable in this repo: SecretsHubAuthInterceptor (in ZB.MOM.WW.Secrets.Replicator.Grpc 0.4.0) deliberately logs nothing on a denial. Its only warning is WarnUnconfiguredOnce, emitted once when the hub has no token configured at all — which is not the case here. So the shipped design and this gate's criterion disagree, and the disagreement lives in the library.

The consequence is real but narrow: a follower whose token was mis-rotated is refused on every sweep and stops converging, while central records that only at Information. An operator watching for warnings sees the follower's own Secret hub sync failed WRN (which does fire), but central shows nothing above INF.

Not patched in ScadaBridge. A host-side interceptor could log the denial, but that would (a) contradict a decision the library documents and reasons about explicitly, in the consumer rather than at the layer that owns it, and (b) put an unbounded log write on an unauthenticated endpoint. Both are worse than the gap. It was fixed at the layer that owns it instead — see the second pass.

Check 4, second pass (0.4.1) — auth negatives + log hygiene · PASS

Re-run in full on the 0.4.1 image (f866208c89e8, branch 6d38e89b), from a clean start: zero The gRPC secrets hub denied lines on central-a before the first negative.

Denial semantics — unchanged, as intended. Three calls to SecretsHub/GetManifest on scadabridge-central-a:8083:

Call	Time	Result
no authorization header	12:46:09Z	Code: Unauthenticated / Message: Unauthenticated.
wrong bearer	12:46:10Z	Code: Unauthenticated / Message: Unauthenticated.
correct bearer (positive control)	12:46:10Z	200, manifest returned
wrong bearer, next window	12:47:38Z	Code: Unauthenticated / Message: Unauthenticated.

Byte-identical code and detail on every refusal, and the positive control proves the route is live. The wire contract did not move: 0.4.1 changed only what the server writes down.

The warning now fires, and fires on the first denial. No interval had to elapse — the no-bearer call at 12:46:09Z produced a WRN in the same second it was refused:

central-a [12:46:10 WRN] The gRPC secrets hub denied 1 call(s) since the previous such warning
          (missing or malformed credential: 1; wrong credential: 0; hub token unconfigured: 0).
          Denials are summarised at most once every 60s.

Rate limiting defers, it does not drop. The wrong-bearer call one second later fell inside that 60 s window, so it produced no line of its own. Rather than assume it was counted, the window was waited out and one further wrong-bearer call issued at 12:47:38Z:

central-a [12:47:38 WRN] The gRPC secrets hub denied 2 call(s) since the previous such warning
          (missing or malformed credential: 0; wrong credential: 2; hub token unconfigured: 0).

The second summary reports two wrong-credential denials — the deferred 12:46:10Z one plus the 12:47:38Z one. Summing across lines gives 1 missing + 2 wrong = 3, exactly the three negatives issued, with the causes correctly attributed. N is a per-window delta, so a reader must sum the lines rather than quote the last one. Nothing was lost to the rate limit, which is the only property that matters for the failure this fix exists to make visible: a follower with a mis-rotated token is refused on every sweep, and central now says so at WRN within a second of the first attempt.

Log hygiene — zero hits, fleet-wide. All 8 nodes' docker logs and every on-disk Serilog file under docker/*/logs/ were grepped for the dev bearer token, the dev KEK, all three secret plaintexts, and both wrong tokens presented by the negatives — the last of those deliberately, because echoing a rejected credential back into a log is its own leak and a free oracle:

central-a  token=0 kek=0 p1=0 p2=0 p3=0 wrong1=0 wrong2=0
central-b  token=0 kek=0 p1=0 p2=0 p3=0 wrong1=0 wrong2=0
site-a-a   token=0 kek=0 p1=0 p2=0 p3=0 wrong1=0 wrong2=0
site-a-b   token=0 kek=0 p1=0 p2=0 p3=0 wrong1=0 wrong2=0
site-b-a / site-b-b / site-c-a / site-c-b   all zero
on-disk log files: 0 files matched, for each of the 7 search strings

The new warning counts denials and names their cause; it carries no credential material, presented or configured. The token still appears only where it legitimately may: the committed rig config and the resulting container environment.

Convergence re-smoke on the 0.4.1 image · PASS

Checks 2 and 3 were not repeated — 0.4.1 touches only the hub's denial logging. Convergence was re-smoked to prove the new image still replicates end to end:

hub-gate-smoke-3 seeded on central-a at 12:47:54Z:

central-node-a  hub-gate-smoke-3  rev=0 del=0 cipher_sha256=fc3cee5431cab8f3a7df303c kek=7451bcbc1f1f updated=2026-08-07T12:47:54.3769041+00:00
site-a-node-a   hub-gate-smoke-3  rev=0 del=0 cipher_sha256=fc3cee5431cab8f3a7df303c kek=7451bcbc1f1f updated=2026-08-07T12:47:54.3769041+00:00
site-a-node-b   hub-gate-smoke-3  rev=0 del=0 cipher_sha256=fc3cee5431cab8f3a7df303c kek=7451bcbc1f1f updated=2026-08-07T12:47:54.3769041+00:00

site-a-a [12:48:11 INF] Secret hub sync converged: pulled 1 row(s) from the central hub.
site-a-b [12:48:11 INF] Secret hub sync converged: pulled 1 row(s) from the central hub.

17 s, both followers, byte-identical ciphertext and updated_utc, decrypt-verified on both. The first pass's hub-gate-smoke-2 and the hub-gate-smoke-1 tombstone survived the image swap unchanged on all three nodes.

Defects found

None in ScadaBridge product code. One library defect, found by this gate and fixed forward; one defect in the gate method, caught before any result was recorded.

Library: the hub logged denials only at Information (ZB.MOM.WW.Secrets.Replicator.Grpc 0.4.0). Found by check 4's first pass; detailed above. Fixed at the library and shipped as 0.4.1 (scadaproj main c86cead): SecretsHubAuthInterceptor now emits a rate-limited Warning per denial window, broken down by cause, with the first denial after startup or a quiet window warning immediately. Wire behaviour is byte-identical — the refusal is still one uniform Unauthenticated with one detail. ScadaBridge took the bump in 6d38e89b (all five ZB.MOM.WW.Secrets* pins moved together; build 0 warnings, secrets wiring + hub-mapping pins 31/31), and check 4 was re-run in full against it.

Gate method: host-side seeding of a bind-mounted SQLite store is not coherent with the running container (Docker Desktop for macOS). The first attempt at check 1 seeded central-a's store with the CLI running natively on the host. The row was visible to the host and to a fresh helper container, but the running node never saw it: four consecutive sweeps returned an empty manifest, no GetSecrets was ever issued, and a wal_checkpoint(TRUNCATE) did not help. Restarting central-a then lost the row entirely — it was gone from both the host and container views. Cause is the virtiofs/page-cache boundary between the macOS host and the Linux guest: SQLite's WAL and -shm mappings are not shared across it.

Corrected by doing every store access — seed, read, delete — from a throwaway container on the same guest kernel, which is what the sibling OtOpcUa gate already did. The rig was returned to an empty-store baseline before check 1 was re-run. Recorded because the failure mode is a convincing false negative: it looks exactly like "the hub is broken".

Residuals

1. The hub client dials a SINGLE endpoint — no failover. Known and documented on the branch. Observed live and directly contrasted on the same node in the same minute:

   site-a-a [11:51:13 WRN] Central control-plane endpoint http://scadabridge-central-a:8083 is
            unavailable; site site-a failed over to http://scadabridge-central-b:8083.
   site-a-a [11:51:12 WRN] Secret hub sync failed; the node continues serving its local store …
   

   ScadaBridge:Communication:CentralGrpcEndpoints is a list and failed over; the hub client stalled. Survivable — the sweep is best-effort and the node keeps its full local last-known-good store — but secrets stop converging until that one central node returns.

2. The central pair does NOT converge with itself, and this makes residual 1 sharper than it reads. Both central nodes host the hub (verified: central-b answers, and refuses an unauthenticated call identically). But central-b's store stayed empty for the whole run while central-a held both secrets:

   central-b, correct bearer, 11:59:32Z:  {}          ← empty manifest, authenticated
   central-node-b (store):                (empty)
   

   Nothing replicates central-a ↔ central-b in Mode=Grpc — the topology is pull-only central → sites. So a hypothetical failover to central-b would not merely stall, it would succeed against an empty hub. By the reconciler's pull-only algorithm that is not data loss (a name absent from the manifest is simply not pulled, never deleted), but it is a silent stop to convergence, and a deployment that authors a secret on the central node the sites do not dial will see it reach nobody. Not live-tested here — asserted from the algorithm plus the observed empty manifest. Any production enablement needs an answer for how the central pair itself converges.

3. Both site-a nodes dial the same central node. Not a rig shortcut — it is the only shape the client supports (residual 1). Worth restating so the run is not read as having exercised two independent hub sources.

4. The /admin/secrets write path was not exercised. Seeding was CLI-into-central's-store. For this topology that is the same path (see Method), so it is a coverage note, not a gap in the result — but the page's own Blazor/authz behaviour is unproven by this gate. ScadaBridge#22 covered it separately.

5. Delete-while-a-follower-is-offline was not tested. Check 3 restarts the pair after both already held the tombstone. The harder resurrection case — deleting at central while a follower is down, then bringing it back holding a live copy — is covered by the library's last-writer-wins tests but not live here. Under a pull-only hub it is structurally safer than in a bidirectional topology (the follower can never push its stale live row back), which is why it was not prioritised.

6. Replication stays default-OFF in the product. Secrets:Replication:Enabled is false and Mode is SqlServer in the shipped appsettings.json. It is enabled on this docker rig only, on four of eight nodes, with a dev KEK and a dev token. Production enablement additionally needs a real KEK supplied out of band and a real token — never committed, and never a ${secret:} reference.

7. Checks 2 and 3 ran on 0.4.0 only. 0.4.1 changes what the hub writes to its own log on a denied call and nothing else — no wire change, no store change, no sweep change — so re-running the outage and tombstone drills would have re-proven code the bump did not touch. Convergence was re-smoked on the 0.4.1 image (17 s, decrypt-verified on both followers), and the first pass's live secret and tombstone survived the image swap unchanged, so the 0.4.1 image is not merely assumed to replicate.

Follow-ups

• File against ScadaBridge: endpoint failover for the hub client, and/or a documented answer for how the central pair's own stores converge (residual 2). Both are the same underlying question — "which central node is authoritative for secrets".
• Reading the new denial warning: N is a per-window delta, not a running total. Summing across lines gives the true count; quoting the last line understates it. Worth knowing before anyone builds an alert on the The gRPC secrets hub denied anchor.

Conclusion

The pull-only gRPC secrets hub works, and works the way the branch says it does: convergence is one sweep tick, ciphertext is relayed verbatim so last-writer-wins ordering survives the hop, a site node boots and serves its full last-known-good store with the entire central pair stopped, convergence resumes unaided when central returns, tombstones propagate and do not resurrect across a pair restart, the auth gate refuses uniformly and now says so at WRN within a second of the first attempt, and no token, KEK or plaintext — including the rejected credentials the negatives presented — reaches any log on any of the eight nodes.

The one clause that failed on the first pass was a library property, not a branch one. It was fixed where it belonged, ScadaBridge took the bump, and check 4 was re-run in full rather than argued around. 4/4 — merged to main.

Unchanged by this gate: Secrets:Replication:Enabled remains default-false in the product, with Mode defaulting to SqlServer. It is enabled on this docker rig only, on four of eight nodes, with a dev KEK and a dev token. Production enablement additionally requires a real KEK supplied out of band, a real hub token, and an answer to residual 2.