docs(secrets): close out scadaproj#3 — gRPC hub shipped 0.4.1, both live gates PASS; residual filed as #4
Task 9 (closeout) of the Secrets gRPC-hub program. Bookkeeping only; no code. - clustered-secrets.md, Production enablement: the gRPC hub is no longer pending. It is BUILT, SHIPPED (0.4.1) and LIVE-GATED PASS 4/4 (ScadaBridge main @ f6c3f7c5, docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md), so the "until that ships, ScadaBridge replication stays off" line is replaced with the truth: the code path is merged and gated, and production enablement is a separate operator action behind a still-default-off flag. OtOpcUa's pair-local topology is recorded as live-gated (lmxopcua master @ 1bd79c8e) rather than pending re-gate. Rotation fan-out corrected: on ScadaBridge it is BOTH central nodes plus both nodes of every site pair, because each central node has its own independent store. - components/secrets/GAPS.md: new "gRPC hub delivered" section recording the 0.4.0/0.4.1 publish, both gates with SHAs and gate-doc paths, and the residual. Existing narrative left intact and amended, not rewritten. - CLAUDE.md Secrets row: lib version 0.3.0 -> 0.4.1 and the app-pin claim corrected (ScadaBridge pins 0.4.1 for the Grpc replicator; the other three still pin 0.2.3); the "production topologies DECIDED" region now records delivery, both gates, default-off, and the residual. - Plan ledger: task 9 completed. Residual carried forward as scadaproj#4 — ScadaBridge's central pair does not converge with itself (central-b served an authenticated but EMPTY manifest all gate while central-a held every secret) and the site client dials a single endpoint; these are one design question, since failover to an empty hub would succeed and silently stop convergence. scadaproj#3 is closed. Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
This commit is contained in:
Joseph Doherty
@@ -196,21 +196,44 @@ Which topology each app runs in production is decided — tracked as
|
||||
- **OtOpcUa: Akka peer-to-peer, pair-local.** Each application cluster is its own 2-node Akka
|
||||
mesh (mesh Phase 6), so replication converges within a pair and nowhere else — seed each pair
|
||||
independently (central via `/admin/secrets`; site pairs via the `secret` CLI or ciphertext-only
|
||||
bundle import).
|
||||
bundle import). **This topology is live-gated as such** — PASS 4/4 on the three-mesh rig
|
||||
(`OtOpcUa/docs/plans/2026-08-07-secrets-pair-local-live-gate.md`, `lmxopcua master @ 1bd79c8e`):
|
||||
pair convergence ≤7 s, intra-pair delete <1 s, clean pair restart, and **no cross-pair
|
||||
propagation proven with seeded isolation controls on the other pairs** rather than by mere
|
||||
absence. It supersedes the 2026-07-18 six-node gate as evidence, which predates the mesh
|
||||
Phase 6 partition.
|
||||
- **ScadaBridge: central gRPC hub (§2's SQL-hub mode is NOT used).** Site servers never talk to
|
||||
the DB directly — they talk to central for everything — so the SqlServer hub's site-side
|
||||
connection string violates the topology rule. A `ZB.MOM.WW.Secrets.Replicator.Grpc` package
|
||||
(central-hosted `secrets_hub.v1`, **pull-only: updates originate at central only, by design**)
|
||||
replaces it; until that ships, ScadaBridge replication stays off. The gRPC mode is **cached on
|
||||
the site clusters** exactly like §2's hub mode — each site node keeps a full local store,
|
||||
serves last-known-good secrets while central is unreachable, and converges on the first
|
||||
successful sweep.
|
||||
connection string violates the topology rule. The `ZB.MOM.WW.Secrets.Replicator.Grpc` package
|
||||
(central-hosted `secrets_hub.v1`, **pull-only: updates originate at central only, by design** —
|
||||
exactly two RPCs, `GetManifest`/`GetSecrets`, with no write RPC at the contract level) replaces
|
||||
it. **It is BUILT, SHIPPED (`0.4.1`) and LIVE-GATED PASS 4/4** — see
|
||||
`ScadaBridge/docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md` (`ScadaBridge main @ f6c3f7c5`):
|
||||
convergence in 5 s, byte-identical and decrypt-verified on both site-a nodes; a cold boot with
|
||||
central down serves last-known-good and resumes; tombstones propagate ≤9 s with no resurrection;
|
||||
auth negatives deny uniformly with zero credential material in the logs. The gRPC mode is
|
||||
**cached on the site clusters** exactly like §2's hub mode — each site node keeps a full local
|
||||
store, serves last-known-good secrets while central is unreachable, and converges on the first
|
||||
successful sweep. **The code path is merged and gated; production enablement is still a separate
|
||||
operator action** — `Secrets:Replication:Enabled` remains default-off, and `Secrets:Replication:Mode`
|
||||
defaults to `SqlServer` (legacy-equivalent), so `Grpc` must be selected deliberately. Do not
|
||||
enable ScadaBridge in production until
|
||||
[scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4) is settled — the central
|
||||
pair does not converge with itself and the site client dials a single endpoint.
|
||||
- **KEK: one fleet-wide production KEK, delivered per node via appsettings/environment override,
|
||||
never committed.** The committed dev KEK is dev-only. Verify per pair with the §0 digest
|
||||
compare. A single fleet KEK is what makes bundle-based seeding between pairs possible.
|
||||
- **Rotation fan-out:** re-wraps do not replicate, so `secret rewrap-all` runs against each
|
||||
node's store individually (both nodes of every pair; on ScadaBridge also central). The KEK
|
||||
doctor is the recovery path for a missed node.
|
||||
node's store individually. On **OtOpcUa** that is both nodes of every pair. On **ScadaBridge**
|
||||
it is **both central nodes plus both nodes of every site pair** — every central node has its
|
||||
own independent local store and nothing replicates between them (live-observed: `central-b`
|
||||
answered an authenticated `GetManifest` with an empty manifest for a whole gate run while
|
||||
`central-a` held every secret;
|
||||
[scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4)), so skipping the
|
||||
second central leaves a store that will be wrong the moment authority or endpoint failover
|
||||
moves to it. Note that the pull-only channel does not re-push re-wrapped rows either: a site
|
||||
node's own store must be re-wrapped directly. The KEK doctor is the recovery path for a
|
||||
missed node.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -179,12 +179,64 @@ project 38/38 green.
|
||||
sites-talk-to-central-not-the-DB rule. **The gRPC mode keeps the cached-on-site-clusters
|
||||
posture:** each site node holds a full local store and serves last-known-good secrets when
|
||||
central is unreachable, converging on the first successful sweep — same offline guarantee as
|
||||
the SQL hub mode, without the site-side DB connection. Enablement waits on the hub package (ScadaBridge) and the
|
||||
pair-local re-gate on the post-Phase-6 rig (OtOpcUa).
|
||||
the SQL hub mode, without the site-side DB connection. ~~Enablement waits on the hub package (ScadaBridge) and the
|
||||
pair-local re-gate on the post-Phase-6 rig (OtOpcUa).~~ **Both prerequisites are now DONE
|
||||
(2026-08-07) — see "gRPC hub delivered" below; scadaproj#3 is CLOSED. Enablement is now purely
|
||||
an operator action: the flag stays default-off in both products.**
|
||||
- ✅ ScadaBridge's hub topology **is** live-validated and technically safe to enable — but the
|
||||
2026-08-07 decision rejects it for production on topology grounds (site nodes must not hold a
|
||||
central SQL connection string); superseded by the gRPC hub above.
|
||||
|
||||
### gRPC hub delivered — scadaproj#3 CLOSED 2026-08-07
|
||||
|
||||
- ✅ **`ZB.MOM.WW.Secrets.Replicator.Grpc` BUILT and PUBLISHED.** Family `0.4.0`
|
||||
(scadaproj `main @ 321ab22`), then `0.4.1` (`main @ c86cead`) adding a rate-limited denial
|
||||
Warning found by the live gate — 6 packages each on the `dohertj2-gitea` feed, restore-verified.
|
||||
`ZB.MOM.WW.Secrets/CHANGELOG.md` created in the same pass. Grpc package suite 166 tests;
|
||||
solution 444 pass / 0 fail / 15 skipped.
|
||||
- **Pull-only is enforced at the contract, not by convention:** `secrets_hub.v1` declares
|
||||
exactly two RPCs — `GetManifest` and `GetSecrets` — and no write RPC exists to call. Pinned by
|
||||
a descriptor-reflection test in the package and a route pin on the ScadaBridge side.
|
||||
- Fail-closed bearer auth: constant-time compare, uniform denial (no oracle), a single public
|
||||
constructor so the auth-less shape is unconstructable, and a `GetSecrets` batch cap that
|
||||
answers `ResourceExhausted`.
|
||||
- Site side: `GrpcSecretSyncService`, a never-throw pull-only sweep with per-row parse isolation
|
||||
(one malformed row cannot poison the sweep) over a keyed `GrpcChannel`; offline =
|
||||
last-known-good.
|
||||
- ✅ **ScadaBridge wired both ends and live-gated — PASS 4/4.** `ScadaBridge main @ f6c3f7c5`
|
||||
(fast-forward merged + pushed; branch `feat/secrets-grpc-hub` retained). Gate doc
|
||||
`ScadaBridge/docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md`.
|
||||
- Wiring: the node role is a **method parameter** (`SecretsNodeRole.Central|Site`), not a config
|
||||
key — configuration cannot flip a site node into hosting the hub. Mode key
|
||||
`Secrets:Replication:Mode` ∈ {`SqlServer` (default, legacy-equivalent), `Grpc`}. The hub is
|
||||
mapped on central's existing h2c control-plane listener (`CentralGrpcPort`, default 8083) —
|
||||
no new port. Startup fails closed on a missing `Secrets:GrpcHub:BearerToken` or `Endpoint`.
|
||||
31 DI/endpoint pin tests.
|
||||
- Gate results: convergence 5 s, byte-identical and decrypt-verified on **both** site-a nodes;
|
||||
cold boot with central down serves last-known-good and resumes (14 s / 43 s); tombstone ≤9 s
|
||||
with no resurrection; auth negatives uniform, the 0.4.1 denial Warning fired immediately, the
|
||||
rate limit proven to **defer, not drop**, and zero credential material in any log — including
|
||||
the rejected tokens themselves.
|
||||
- Flag remains **default-off everywhere**; the rig enables the central pair and site-a only.
|
||||
- ✅ **OtOpcUa pair-local re-gate — PASS 4/4.** `lmxopcua master @ 1bd79c8e`, gate doc
|
||||
`OtOpcUa/docs/plans/2026-08-07-secrets-pair-local-live-gate.md`. Pair convergence ≤7 s;
|
||||
intra-pair delete <1 s; clean pair restart; and **no cross-pair propagation proven with seeded
|
||||
isolation controls on the other pairs**, not by mere absence of a row. This supersedes the
|
||||
2026-07-18 six-node gate as evidence — that gate predates the mesh Phase 6 partition, and
|
||||
Phase 6's own secrets leg was inference-only (no secret was ever seeded during it). Rig fix that
|
||||
landed with the gate: `Secrets__SqlitePath` moved onto each node's per-node `/app/data` volume,
|
||||
where it had been sharing a path.
|
||||
- ⚠️ **Residual, filed as [scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4):
|
||||
ScadaBridge's central pair does not converge with itself, and the site client dials a single
|
||||
`Endpoint`.** Each central node has an independent local store; live-observed, `central-b`
|
||||
answered an *authenticated* `GetManifest` with an **empty manifest** for the whole gate run
|
||||
while `central-a` held every secret. These are one design question, not two: adding endpoint
|
||||
failover alone would let a failover *succeed* against an empty hub — not data loss (pull-only
|
||||
never deletes) but a **silent convergence stop**, and secrets authored on the central node the
|
||||
sites do not dial reach nobody. Folded into the same issue: delete-while-follower-offline is
|
||||
untested, and alerting on the denial Warning must sum per-window deltas because the warning is
|
||||
deliberately rate-limited. **ScadaBridge production enablement should wait on #4.**
|
||||
|
||||
Harness: `scratchpad/livegate/` (two scenarios, `akka` / `sql`; not committed).
|
||||
|
||||
### Upstream defect found during adoption — `0.2.0` Akka replication was inert
|
||||
|
||||
@@ -5,11 +5,11 @@
|
||||
{"id": 2, "subject": "Task 2: Server half — hub service + bearer auth + DI", "status": "completed", "classification": "high-risk", "blockedBy": [1], "parallelizableWith": [3], "note": "7fc77f2 + 353af72 (review fixes: GetSecrets batch cap, DI single-registration guard) merged ff into feat/secrets-grpc-hub; spec ✅ 16/16, code review Approved; 73 server tests"},
|
||||
{"id": 3, "subject": "Task 3: Client half — pull-only sweep service + DI", "status": "completed", "classification": "high-risk", "blockedBy": [1], "parallelizableWith": [2], "note": "c5ea3a7 + 1c31e6b (review Critical fix: per-row parse isolation, WireToDomain batch methods deleted, real-client CallInvoker tests) merged 55e0178; spec ✅ 8/8, code review Approved on re-review; merged suite 140/140"},
|
||||
{"id": 4, "subject": "Task 4: Integration tests + no-write-surface pin", "status": "completed", "classification": "standard", "blockedBy": [2, 3], "note": "3e3748b + 5f81e65 (harness drift-fix via internal validateEagerly seam); spec ✅ 6/6, code review Approved; Grpc suite 158, solution 436/0/15"},
|
||||
{"id": 5, "subject": "Task 5: Version bump + pack + publish 0.4.0", "status": "pending", "classification": "small", "blockedBy": [4]},
|
||||
{"id": 6, "subject": "Task 6: ScadaBridge wiring — central hub + site sweep client", "status": "pending", "classification": "high-risk", "blockedBy": [5]},
|
||||
{"id": 7, "subject": "Task 7: ScadaBridge live gate (docker cluster)", "status": "pending", "classification": "high-risk", "blockedBy": [6], "parallelizableWith": [8]},
|
||||
{"id": 5, "subject": "Task 5: Version bump + pack + publish 0.4.0", "status": "completed", "classification": "small", "blockedBy": [4], "note": "321ab22 = main, pushed; 6 packages @ 0.4.0 on dohertj2-gitea feed incl. new Replicator.Grpc, restore-verified from fresh cache; CHANGELOG.md created (backfilled 0.2.0–0.3.0)"},
|
||||
{"id": 6, "subject": "Task 6: ScadaBridge wiring — central hub + site sweep client", "status": "completed", "classification": "high-risk", "blockedBy": [5], "note": "ScadaBridge feat/secrets-grpc-hub 57f08c21+127ec254+c8e90daa+fc784b41 (predicate-sharing review fix); role is a method parameter (not config); mode key Secrets:Replication:Mode default SqlServer; 31/31 wiring pins; spec ✅ 8/8, code review Approved; residual: sweep dials single endpoint (no central-pair failover) — carry to closeout"},
|
||||
{"id": 7, "subject": "Task 7: ScadaBridge live gate (docker cluster)", "status": "completed", "classification": "high-risk", "blockedBy": [6], "parallelizableWith": [8], "note": "PASS 4/4 (check 4 on 2nd pass after 0.4.1 denial-warning fix, scadaproj c86cead); ScadaBridge main @ f6c3f7c5 merged+pushed; gate doc docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md; residuals: central pair doesn't converge with itself + single-endpoint client = one design issue to file; delete-while-follower-offline untested"},
|
||||
{"id": 8, "subject": "Task 8: OtOpcUa Akka pair-local re-gate (three-mesh rig)", "status": "completed", "classification": "high-risk", "parallelizableWith": [1, 2, 3, 4, 5, 6, 7], "note": "PASS 4/4, OtOpcUa master @ 1bd79c8e pushed; gate doc docs/plans/2026-08-07-secrets-pair-local-live-gate.md; rig fix: Secrets__SqlitePath onto per-node volume; follow-up for Task 9: stale KNOWN-NON-FUNCTIONAL comment in OtOpcUa appsettings.json Secrets block"},
|
||||
{"id": 9, "subject": "Task 9: Closeout — issue #3, runbook, GAPS, index", "status": "pending", "classification": "small", "blockedBy": [7, 8]}
|
||||
{"id": 9, "subject": "Task 9: Closeout — issue #3, runbook, GAPS, index", "status": "completed", "classification": "small", "blockedBy": [7, 8], "note": "Runbook clustered-secrets.md Production-enablement + rotation fan-out updated (ScadaBridge = both central nodes + both nodes of every site pair); components/secrets/GAPS.md gained a 'gRPC hub delivered' section (0.4.0/0.4.1, both gates with SHAs, residual); CLAUDE.md Secrets row updated (lib 0.4.1, ScadaBridge pins 0.4.1, both gates PASS 4/4, default-off, residual). OtOpcUa stale appsettings _comment corrected: lmxopcua master @ 01256344, pushed. scadaproj#3 all 6 boxes checked with SHA annotations, closing comment posted, CLOSED. Residual filed as scadaproj#4 (central-pair convergence + client endpoint failover are one design question; plus delete-while-follower-offline untested and denial-warning alerting must sum per-window deltas)."}
|
||||
],
|
||||
"lastUpdated": "2026-08-07"
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user