scadaproj/docs/plans/2026-08-07-secrets-grpc-hub-implementation.md

# Secrets gRPC hub replicator — implementation plan

> **For Claude:** REQUIRED SUB-SKILL: execute with superpowers-extended-cc:subagent-driven-development
> (owner chose subagent execution, parallel where safe, 2026-08-07 — do not re-ask).
> Task metadata (`Classification`, `Parallelizable with`) drives the review chain and dispatch.

**Goal:** Ship `ZB.MOM.WW.Secrets.Replicator.Grpc` (central-hosted pull-only secrets hub), wire it
into ScadaBridge, live-gate it, and re-gate OtOpcUa's Akka pair-local topology — completing
[scadaproj#3](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/3).

**Design authority:** `docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md` (D1–D4).
Pull-only is an owner-confirmed invariant: **the proto exposes no write RPC — the narrower
contract is the enforcement.**

**Architecture:** central hosts a `secrets_hub.v1` gRPC service adapting its `ISecretStore` (read
half only) to the wire; site nodes run a `BackgroundService` sweep that pulls via
`SecretReplicationReconciler.ComputePullSet` → fetch → `ApplyAsync`. Ciphertext-only; offline =
last-known-good (sweep never throws out of `ExecuteAsync`).

**Verified seams (read before coding):**
- `ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/ISecretReplicationHub.cs` —
  `GetManifestAsync` / `GetManyAsync` / `ApplyReplicatedAsync`. The gRPC hub mirrors ONLY the
  first two.
- `ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/Replication/SecretReplicationReconciler.cs` —
  `ComputePullSet` (static), `ApplyAsync`, `ReadLocalAsync`. Pull-only = never call
  `ComputePushSet`.
- `SqlServerSecretSyncService.cs` — the sweep shape to mirror (SyncOnStartup, `PeriodicTimer`,
  never-throw, log + retry next tick).

**Standing security invariants (every task):** never echo bearer/KEK material in logs, errors, or
status; error messages name exact config keys; fail closed on every unresolvable auth case; auth
interceptors keep EXACTLY ONE public constructor (Grpc.AspNetCore silently stops invoking
interceptors with more); ciphertext only on the wire.

---

## Task 1: Package scaffold + `secrets_hub.v1` proto

**Classification:** standard · **Est:** ~5 min · **Parallelizable with:** Task 8
**Files:** Create `ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/ZB.MOM.WW.Secrets.Replicator.Grpc.csproj`,
`…/Protos/secrets_hub.proto`; Modify `ZB.MOM.WW.Secrets/ZB.MOM.WW.Secrets.slnx` (add project).

Proto (`package zb.mom.ww.secrets.hub.v1`), service `SecretsHub`, exactly two RPCs:
- `rpc GetManifest(GetManifestRequest) returns (GetManifestResponse)` — repeated entries mirroring
  `SecretManifestEntry` (read the Abstractions type and map EVERY field losslessly: name,
  revision, timestamp, tombstone flag, and whatever else it carries).
- `rpc GetSecrets(GetSecretsRequest {repeated string names}) returns (GetSecretsResponse)` —
  repeated rows mirroring `StoredSecret` verbatim (ciphertext + metadata; read the type first).
NO other RPC. csproj: `Grpc.AspNetCore` + `Grpc.Net.Client` + `Grpc.Tools` (match versions used
elsewhere in the family), `<Version>` inherits the family's shared version. Follow the SqlServer
package's csproj as the template. Build must pass; commit.

## Task 2: Server half — hub service + bearer auth + DI

**Classification:** high-risk · **Est:** ~5 min · **Parallelizable with:** Task 3
**Files:** Create `…Replicator.Grpc/SecretsHubService.cs`, `…/SecretsHubAuthInterceptor.cs`,
`…/DependencyInjection/SecretsGrpcHubServerExtensions.cs`, `…/SecretsGrpcHubServerOptions.cs`.

`SecretsHubService : SecretsHub.SecretsHubBase` adapts the registered `ISecretStore` (manifest +
bulk read — reuse/mirror how `SqlServerSecretStore` produces manifest entries). Auth: interceptor
validating `Authorization: Bearer <token>` against `SecretsGrpcHubServerOptions.BearerToken`
(bound from config; empty/unset ⇒ the service REFUSES to serve — fail closed, log names the
config key once). Constant-time comparison. Uniform denial (`Unauthenticated`, no detail).
`AddZbSecretsGrpcHub(IServiceCollection, IConfiguration, sectionPath)` +
`MapZbSecretsHub(IEndpointRouteBuilder)`. TDD: red-first unit tests for the interceptor (valid /
invalid / missing / unset-key-refuses) and options validation.

## Task 3: Client half — pull-only sweep service + DI

**Classification:** high-risk · **Est:** ~5 min · **Parallelizable with:** Task 2
**Files:** Create `…Replicator.Grpc/GrpcSecretsHubClient.cs`, `…/GrpcSecretSyncService.cs`,
`…/SecretsGrpcHubClientOptions.cs`, `…/DependencyInjection/SecretsGrpcHubClientExtensions.cs`.

`GrpcSecretsHubClient` wraps the generated client (bearer header per call; per-call deadline from
options). `GrpcSecretSyncService : BackgroundService` mirrors `SqlServerSecretSyncService`
EXCEPT pull-only: manifest → `SecretReplicationReconciler.ComputePullSet` → `GetSecrets` →
`ApplyAsync` (+ cache invalidation seam, same as SqlServer's ctor wiring). NEVER computes a push
set; NEVER throws out of `ExecuteAsync` (hub outage ⇒ warn + retry next tick — copy the
SqlServer service's log/exception discipline). Options: `Endpoint` (required), `BearerToken`
(required, fail-closed), `SyncInterval` (default 30 s), `SyncOnStartup` (default true),
`CallDeadline` (default 30 s); `Validate()` called at registration like
`SqlServerSecretsOptions`. TDD: red-first tests with a fake hub — pull convergence, tombstone
apply, outage → no throw + local store untouched, options validation.

## Task 4: Integration tests + no-write-surface pin

**Classification:** standard · **Est:** ~5 min · **Parallelizable with:** none (needs 2+3)
**Files:** Create `ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.Grpc.Tests/` (project +
suites); Modify slnx.

In-process server via `Microsoft.AspNetCore.TestHost` (real Kestrel-less gRPC): end-to-end
convergence (seed hub store → client store converges, byte-identical ciphertext), delete
propagates as tombstone, hub down at client boot ⇒ client serves last-known-good and converges
after the hub returns, auth negatives fail closed (bad/missing/unset both sides). Plus the
contract pin: a reflection test over the generated `SecretsHub` service asserting the RPC set is
EXACTLY `{GetManifest, GetSecrets}` — a third RPC (any write surface) fails the build. Full
`dotnet test` on the Secrets slnx green, 0 warnings.

## Task 5: Version bump + pack + publish 0.4.0

**Classification:** small · **Est:** ~4 min · **Parallelizable with:** none
**Files:** Modify `ZB.MOM.WW.Secrets/Directory.Build.props` (version → 0.4.0), CHANGELOG, pack docs.

Bump the family version to `0.4.0`, CHANGELOG entry (new package, no breaking changes; existing
packages unchanged behaviorally), `dotnet pack -c Release`, push ALL packages (now 6 + CLI as
applicable) to the `dohertj2-gitea` feed, restore-verify from a scratch consumer (the established
pattern from 0.2.x/0.3.0 publishes). Commit + push scadaproj `main`.

## Task 6: ScadaBridge wiring — central hub + site sweep client

**Classification:** high-risk · **Est:** ~5 min · **Parallelizable with:** none (needs Task 5's feed package)
**Repo:** `~/Desktop/ScadaBridge` (branch `feat/secrets-grpc-hub` off `main`).
**Files:** Modify the central gRPC endpoint composition (find where the ClusterClient→gRPC
migration's central services are mapped), the site-role composition root, `Directory.Packages.props`
(Secrets → 0.4.0 + new package ref), appsettings templates; Create DI pin tests.

Central: `AddZbSecretsGrpcHub` + `MapZbSecretsHub` on the existing central gRPC listener, bearer
token from appsettings/env (`${secret:}` is NOT usable here — chicken-and-egg; follow the mesh
PSK pattern), fail-closed. Site: register the sweep client under
`Secrets:Replication:Enabled=true` + a mode/section selecting Grpc — mirror how the existing
replicator registration is switched; central must NOT register the client, sites must NOT
register the hub. Red-first WebApplicationFactory/DI pin tests over the real Program.cs for both
roles (the ScadaBridge#22 test pattern). Full ScadaBridge offline suite green before commit.

## Task 7: ScadaBridge live gate (docker cluster)

**Classification:** high-risk · **Est:** live-rig, serial · **Parallelizable with:** Task 8
**Files:** Create `ScadaBridge/docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md` (gate record).

On the local 8-node docker cluster (rebuild via `bash docker/deploy.sh` with the flag enabled on
site-a + central): (1) secret added centrally converges to both site-a nodes ≤ sweep interval,
byte-identical, decrypt-verified; (2) site pair cold-boots with central stopped ⇒ serves
last-known-good, converges after central returns; (3) central delete tombstone propagates, no
resurrection on site restart; (4) auth negatives fail closed, zero key material in logs
(grep the logs to prove it). Record PASS/FAIL per check in the gate doc; fix-forward defects on
the branch. Merge `feat/secrets-grpc-hub` → `main`, push.

## Task 8: OtOpcUa Akka pair-local re-gate (three-mesh rig)

**Classification:** high-risk · **Est:** live-rig, config-only (no product code) · **Parallelizable with:** Tasks 1–7 (independent repo + rig)
**Repo:** `~/Desktop/OtOpcUa`. **Files:** Create `OtOpcUa/docs/plans/2026-08-07-secrets-pair-local-live-gate.md`.

The 2026-07-18 six-node convergence gate predates the mesh Phase 6 partition — re-gate pair-local
semantics on the current three-mesh docker-dev rig (fixtures on `10.100.0.35`; dev KEK): (1) seed
on one node of site-a ⇒ partner converges ≤60 s byte-identical; (2) **no cross-pair propagation**
to site-b/central (this is now CORRECT behavior); (3) delete propagates within the pair; (4)
clean pair restart, no resurrection. Record the gate doc, commit + push OtOpcUa `master`.

## Task 9: Closeout — issue, runbook, index

**Classification:** small · **Est:** ~4 min · **Parallelizable with:** none (last)

Check off the remaining boxes on scadaproj#3 (implementation, ScadaBridge gate, OtOpcUa re-gate,
rotation doc — verify the rotation fan-out paragraph in `clustered-secrets.md` §Production
enablement is still accurate post-implementation and extend if needed). Update
`components/secrets/GAPS.md` + the CLAUDE.md Secrets index row (0.4.0 published, hub shipped,
gates PASS with SHAs). Commit + push scadaproj `main`. Close #3 if every box is checked.

---

## Dispatch notes for the executor

- **Wave 1:** Task 1 ∥ Task 8 (different repos; Task 8 is the long pole — start it first).
- **Wave 2:** Task 2 ∥ Task 3 (disjoint files, same new project).
- **Wave 3:** Task 4 → Task 5 (serial).
- **Wave 4:** Task 6 → Task 7 (serial, ScadaBridge).
- **Wave 5:** Task 9 after everything.
- Live-gate tasks (7, 8) need docker orchestration + log inspection — run them via a single agent
  each (or main loop), never split one gate across agents. They use different docker hosts
  (ScadaBridge = local docker; OtOpcUa rig = `10.100.0.35` + local compose) so they may overlap.
- Rig credentials/keys: dev KEK and rig PSKs only; never commit rig overrides; never echo key
  material (standing rules).
- Commits end with the session trailer:
  `Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1`