test(secrets): live gate for the gRPC secrets hub — 3/4 PASS, not merged

Rig config enabling the pull-only hub on the docker cluster (central pair hosts,
site-a pair follows; site-b and site-c deliberately left off so the default-OFF
posture is proven side by side), plus the gate record.

Checks 1-3 PASS. A central write reaches both site-a nodes in 5 s with a
byte-identical ciphertext row and decrypts correctly on both; a site pair boots
and serves its full last-known-good store with the entire central pair stopped,
warning once per interval without crashing, and resumes convergence unaided when
central returns; a tombstone propagates in under 9 s and survives a pair restart
with central up and sweeping, without resurrecting.

Check 4 FAILS one clause of three. Both auth negatives - absent bearer and wrong
bearer - are denied with a byte-identical Unauthenticated status and detail, and
a fleet-wide grep of all eight nodes' docker logs and on-disk Serilog files finds
ZERO occurrences of the dev token, the dev KEK or either plaintext. But the
criterion also asks for a server-side WARNING on denial, and there is none: the
only record is one Information line per call from Grpc.AspNetCore.Server, because
SecretsHubAuthInterceptor deliberately logs nothing on a denial and warns only
when no token is configured at all. That is a property of the 0.4.0 library, not
of this branch, and it is not patched here - a host-side interceptor would
contradict a documented library decision at the wrong layer and put an unbounded
log write on an unauthenticated endpoint.

The merge condition is 4/4, so this branch is NOT merged. The library's denial
logging is the only thing between this result and a merge.

Two residuals worth carrying: the hub client dials a single endpoint and does not
fail over (observed live, and contrasted against CentralGrpcEndpoints failing over
on the same node in the same minute), and the central pair does not converge with
itself - central-b answered an authenticated GetManifest with an empty manifest
for the whole run while central-a held both secrets. Together those make "which
central node is authoritative for secrets" one question, not two.

Rig config notes: Secrets__SqlitePath points at /app/data because the appsettings
default resolves to /app inside the image's writable layer, so the central pair
gained the per-node data volume the site pairs already had. All values are
dev-only and committed under the same exception the mesh PSKs already use.

Also recorded: a gate-METHOD defect. Seeding the bind-mounted store from the macOS
host is not coherent with the running container - the row was visible to the host
and to a fresh container but never to the node, and was lost outright on restart.
Every store access was redone from a throwaway container. The failure mode is a
convincing false negative that looks exactly like a broken hub.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1

docker/docker-compose.yml

@@ -1,3 +1,47 @@
+# ── Clustered secret replication: the pull-only gRPC hub (scadaproj#3) ─────────
+#
+# Central hosts the hub on its EXISTING h2c control-plane listener (CentralGrpcPort
+# 8083, alongside CentralControlService); each site node sweeps it on an interval and
+# writes what it pulls into its OWN local SQLite store. Nothing but ciphertext crosses
+# the wire, so every participating node must resolve the SAME KEK.
+#
+# ENABLED ON FOUR NODES ONLY: the central pair (hub) and the site-a pair (followers).
+# site-b and site-c are deliberately left without it, so the default-OFF posture is
+# proven side by side on one rig — exactly as site-a is the rig's only LocalDb-replicated
+# pair. A node with no Secrets__* override keeps the shipped appsettings default
+# (Replication:Enabled=false, Mode=SqlServer) and composes a plain local store.
+#
+# ALL VALUES HERE ARE DEV-ONLY and committed under the same exception as the mesh PSKs
+# and the ApiKeyPepper above: a local docker rig needs a working credential in source
+# control to boot. Production supplies the KEK out of band (ZB_SECRETS_MASTER_KEY, never
+# committed) and the hub token from appsettings/env — NEVER as a ${secret:} reference,
+# since resolving one is what the hub exists to make possible.
+x-secrets-hub-env: &secrets-hub-env
+  # DEV-ONLY KEK — NOT a real key. Identical on all four participating nodes: only
+  # ciphertext replicates, so a node with a different KEK fails closed on resolve with a
+  # kek_id mismatch that reads like corruption but is a deployment error.
+  ZB_SECRETS_MASTER_KEY: "zZiBWuoaVMbJmGXToLk9Lakw0iJozXoL/7Gxac3GwJ4="
+  # The appsettings default is the relative "scadabridge-secrets.db", which resolves to
+  # /app — inside the image's writable layer, so it is destroyed by any container
+  # recreate and unreachable from the host. /app/data is the node's own mounted volume
+  # (the one LocalDb already uses on sites; added to the central pair for this).
+  Secrets__SqlitePath: "/app/data/scadabridge-secrets.db"
+  Secrets__Replication__Enabled: "true"
+  Secrets__Replication__Mode: "Grpc"
+  # DEV-ONLY shared bearer token — NOT a real secret. Presented by every follower and
+  # verified by the hub's fail-closed SecretsHubAuthInterceptor. Must be IDENTICAL on the
+  # hub and every follower; an unset token is a startup failure on both halves.
+  Secrets__GrpcHub__BearerToken: "secrets-hub-docker-dev-token"
+
+# Site half of the same section. NOTE the asymmetry with
+# ScadaBridge:Communication:CentralGrpcEndpoints, which is a LIST that fails over across
+# the central pair: the hub client dials a SINGLE endpoint, so a sweep against a stopped
+# central-a stalls rather than failing over to central-b. That is survivable — the sweep
+# is best-effort and the node keeps serving its full local last-known-good store — but
+# secrets stop converging until central-a returns.
+x-secrets-hub-site-env: &secrets-hub-site-env
+  Secrets__GrpcHub__Endpoint: "http://scadabridge-central-a:8083"
+
 services:
   central-a:
     image: scadabridge:latest
@@ -7,6 +51,8 @@ services:
     stop_grace_period: 30s
     container_name: scadabridge-central-a
     environment:
+      # Hub half of the pull-only gRPC secrets hub (anchor at the top of this file).
+      <<: *secrets-hub-env
       SCADABRIDGE_CONFIG: Central
       ASPNETCORE_ENVIRONMENT: Development
       ASPNETCORE_URLS: "http://+:5000"
@@ -42,6 +88,11 @@ services:
       - "9013:8083"   # gRPC control plane (CentralControlService, T1A.2)
     volumes:
       - ./central-node-a/appsettings.Central.json:/app/appsettings.Central.json:ro
+      # Added for the gRPC secrets hub: the node's local secret store lives at
+      # Secrets__SqlitePath=/app/data/scadabridge-secrets.db, so it needs the same
+      # per-node volume the site nodes already have. Without it the store sits in the
+      # image's writable layer and is destroyed by every container recreate.
+      - ./central-node-a/data:/app/data
       - ./central-node-a/logs:/app/logs
     networks:
       - scadabridge-net
@@ -55,6 +106,8 @@ services:
     stop_grace_period: 30s
     container_name: scadabridge-central-b
     environment:
+      # Hub half of the pull-only gRPC secrets hub (anchor at the top of this file).
+      <<: *secrets-hub-env
       SCADABRIDGE_CONFIG: Central
       ASPNETCORE_ENVIRONMENT: Development
       ASPNETCORE_URLS: "http://+:5000"
@@ -90,6 +143,11 @@ services:
       - "9014:8083"   # gRPC control plane (CentralControlService, T1A.2)
     volumes:
       - ./central-node-b/appsettings.Central.json:/app/appsettings.Central.json:ro
+      # Added for the gRPC secrets hub: the node's local secret store lives at
+      # Secrets__SqlitePath=/app/data/scadabridge-secrets.db, so it needs the same
+      # per-node volume the site nodes already have. Without it the store sits in the
+      # image's writable layer and is destroyed by every container recreate.
+      - ./central-node-b/data:/app/data
       - ./central-node-b/logs:/app/logs
     networks:
       - scadabridge-net
@@ -103,6 +161,9 @@ services:
     stop_grace_period: 30s
     container_name: scadabridge-site-a-a
     environment:
+      # Follower half of the pull-only gRPC secrets hub (anchors at the top of this
+      # file). site-b and site-c deliberately carry neither.
+      <<: [*secrets-hub-env, *secrets-hub-site-env]
       SCADABRIDGE_CONFIG: Site
     ports:
       - "9021:8082"   # Akka remoting (host access for debugging)
@@ -123,6 +184,9 @@ services:
     stop_grace_period: 30s
     container_name: scadabridge-site-a-b
     environment:
+      # Follower half of the pull-only gRPC secrets hub (anchors at the top of this
+      # file). site-b and site-c deliberately carry neither.
+      <<: [*secrets-hub-env, *secrets-hub-site-env]
       SCADABRIDGE_CONFIG: Site
     ports:
       - "9022:8082"   # Akka remoting