diff --git a/components/secrets/GAPS.md b/components/secrets/GAPS.md index 7d8021a..b205dc8 100644 --- a/components/secrets/GAPS.md +++ b/components/secrets/GAPS.md @@ -84,10 +84,50 @@ construction). `Validate()` eagerly, so unconditional wiring would have broken startup on every node. - Bootstrap constraint recorded: the hub connstr **cannot come from the hub**; it must be seeded in each node's local store or supplied via env. - - ⬜ **Task 9, the G-2-class live gate, is NOT done** — write→resolve across nodes, tombstone - propagation (no resurrection), partition survival, and the negative-KEK fail-closed control. - Needs a 2-node OtOpcUa rig and a ScadaBridge central-pair + site + SQL hub. **The topology is - not "adopted" until this passes.** Do not enable `Secrets:Replication:Enabled` anywhere first. + - **Task 9 live gate RUN 2026-07-18. Split result — one topology passed, one FAILED.** + +### Live gate: SQL-Server hub (ScadaBridge topology) — ✅ 6/6 PASSED + +Run against a **real SQL Server 2022** (`10.100.0.35:31433`, DB `ZbSecretsGate`) with two nodes, +each with its own local SQLite store, using `AddZbSecretsSqlServerReplication` — the same extension +`AddScadaBridgeSecrets` calls: + +| Check | Result | +|---|---| +| hub schema provisions + sweeps start | ✅ | +| write on A → resolves on B via the hub | ✅ converged through the shared hub | +| delete on A → tombstone on B | ✅ no resurrection | +| write on B → resolves on A | ✅ sweep is bidirectional | +| **partitioned node still resolves from local state** | ✅ hub unreachable, local reads unaffected — **site autonomy holds**, which is the whole reason ScadaBridge uses hub mode | +| **wrong-KEK node fails closed** | ✅ `SecretDecryptionException`, not wrong plaintext | + +The KEK control fired spontaneously first: a re-run against a hub still holding rows from a prior +run (different KEK) threw `SecretDecryptionException` rather than returning garbage — unplanned but +real evidence of fail-closed. + +### Live gate: Akka peer-to-peer (OtOpcUa topology) — ❌ FAILED, blocking defect + +**Resolving `ISecretReplicator` hangs indefinitely** in a real clustered process. Reproduced twice +on a healthy 2-node cluster (both members confirmed `Up` before the hang). Isolated by resolving +`ISecretReplicator` *directly* rather than via `ISecretStore`, so it is **not** nested-resolution +contention — the hang is in constructing `AkkaSecretReplicator` → `SecretReplicationActorProvider.ActorRef` +→ `system.ActorOf(SecretReplicationActor.Props(...))`. Native stacks show `Monitor_Wait`. + +**Impact if enabled:** OtOpcUa would hang at startup, since the wiring resolves `ISecretStore` from a +startup hook. It is currently harmless only because `Secrets:Replication:Enabled` defaults false. + +**Not explained by the library's own suite:** `TwoNodeClusterReplicationTests` creates the same actor +on a real 2-node cluster and passes (33 tests green). The difference is creation through DI inside a +real host process. Root cause NOT yet identified — candidates: `DistributedPubSub.Get(...)` in +`PreStart` interacting with the `Lazy` (`ExecutionAndPublication`) lock, or missing +`Akka.Cluster.Tools` reference config in the composed HOCON. + +- ⬜ **Do NOT enable `Secrets:Replication:Enabled` in OtOpcUa.** The Akka topology is not adoptable + until this deadlock is root-caused and fixed, with a regression test that creates the actor + through DI in a hosted process rather than directly from a test. +- ✅ ScadaBridge's hub topology **is** live-validated and safe to enable (config + a shared KEK). + +Harness: `scratchpad/livegate/` (two scenarios, `akka` / `sql`; not committed). ### Upstream defect found during adoption — `0.2.0` Akka replication was inert diff --git a/docs/plans/2026-07-18-secrets-0.2.0-upgrade-and-clustering.md.tasks.json b/docs/plans/2026-07-18-secrets-0.2.0-upgrade-and-clustering.md.tasks.json index 1fd4a4e..852b228 100644 --- a/docs/plans/2026-07-18-secrets-0.2.0-upgrade-and-clustering.md.tasks.json +++ b/docs/plans/2026-07-18-secrets-0.2.0-upgrade-and-clustering.md.tasks.json @@ -99,7 +99,7 @@ "id": 9, "nativeId": "15", "subject": "Task 9: Live convergence gate (both clustered apps)", - "status": "pending", + "status": "completed", "classification": "high-risk", "blockedBy": [ 6, @@ -110,7 +110,7 @@ "id": 10, "nativeId": "16", "subject": "Task 10: Docs reconcile", - "status": "pending", + "status": "completed", "classification": "small", "blockedBy": [ 9 @@ -124,5 +124,5 @@ } ], "lastUpdated": "2026-07-18", - "notes": "Tasks 1-8 complete on local feat/secrets-0.2.0 branches in 4 repos (not pushed). Apps ended on Secrets 0.2.1, not 0.2.0: adoption uncovered that 0.2.0's Akka replicator was silently inert (TryAdd order); fixed and republished. Task 9 (live gate) NOT done - needs 2-node rigs. Replication is default-OFF everywhere, so nothing ships enabled." + "notes": "Tasks 1-10 complete. Live gate RUN 2026-07-18 with a SPLIT result: SQL-Server hub (ScadaBridge) PASSED 6/6 against a real SQL Server 2022 incl. partition survival and wrong-KEK fail-closed. Akka peer-to-peer (OtOpcUa) FAILED - resolving ISecretReplicator deadlocks in a real clustered process at ActorOf. Replication remains default-OFF everywhere; do NOT enable it in OtOpcUa until the deadlock is fixed." } \ No newline at end of file