docs(secrets): live convergence gate results - SQL hub PASSES 6/6, Akka FAILS

SQL-Server hub topology (ScadaBridge) is live-validated against a real SQL Server
2022 with two nodes: write->resolve via hub, tombstone propagation without
resurrection, bidirectional sweep, partitioned-node local resolution (site
autonomy - the reason hub mode was chosen), and wrong-KEK fail-closed via
SecretDecryptionException.

Akka peer-to-peer topology (OtOpcUa) FAILS the gate: resolving ISecretReplicator
hangs indefinitely in a real clustered process. Reproduced twice on a healthy
2-node cluster; isolated to constructing AkkaSecretReplicator -> ActorRef ->
system.ActorOf, NOT nested DI resolution. The library's own 2-node suite passes,
so the trigger is creation through DI in a hosted process. Root cause not yet
identified.

Impact: OtOpcUa would hang at startup if replication were enabled, since the
startup hook resolves ISecretStore. Harmless today only because the flag defaults
false. Do NOT enable Secrets:Replication:Enabled in OtOpcUa.

The gate did its job - it caught a blocking defect that every offline suite,
including the library's own real-cluster tests, had passed over.
This commit is contained in:
Joseph Doherty
2026-07-18 13:50:35 -04:00
parent 62a96e5f22
commit eb445ea7e7
2 changed files with 47 additions and 7 deletions
+44 -4
View File
@@ -84,10 +84,50 @@ construction).
`Validate()` eagerly, so unconditional wiring would have broken startup on every node.
- Bootstrap constraint recorded: the hub connstr **cannot come from the hub**; it must be seeded
in each node's local store or supplied via env.
- **Task 9, the G-2-class live gate, is NOT done** — write→resolve across nodes, tombstone
propagation (no resurrection), partition survival, and the negative-KEK fail-closed control.
Needs a 2-node OtOpcUa rig and a ScadaBridge central-pair + site + SQL hub. **The topology is
not "adopted" until this passes.** Do not enable `Secrets:Replication:Enabled` anywhere first.
- **Task 9 live gate RUN 2026-07-18. Split result — one topology passed, one FAILED.**
### Live gate: SQL-Server hub (ScadaBridge topology) — ✅ 6/6 PASSED
Run against a **real SQL Server 2022** (`10.100.0.35:31433`, DB `ZbSecretsGate`) with two nodes,
each with its own local SQLite store, using `AddZbSecretsSqlServerReplication` — the same extension
`AddScadaBridgeSecrets` calls:
| Check | Result |
|---|---|
| hub schema provisions + sweeps start | ✅ |
| write on A → resolves on B via the hub | ✅ converged through the shared hub |
| delete on A → tombstone on B | ✅ no resurrection |
| write on B → resolves on A | ✅ sweep is bidirectional |
| **partitioned node still resolves from local state** | ✅ hub unreachable, local reads unaffected — **site autonomy holds**, which is the whole reason ScadaBridge uses hub mode |
| **wrong-KEK node fails closed** | ✅ `SecretDecryptionException`, not wrong plaintext |
The KEK control fired spontaneously first: a re-run against a hub still holding rows from a prior
run (different KEK) threw `SecretDecryptionException` rather than returning garbage — unplanned but
real evidence of fail-closed.
### Live gate: Akka peer-to-peer (OtOpcUa topology) — ❌ FAILED, blocking defect
**Resolving `ISecretReplicator` hangs indefinitely** in a real clustered process. Reproduced twice
on a healthy 2-node cluster (both members confirmed `Up` before the hang). Isolated by resolving
`ISecretReplicator` *directly* rather than via `ISecretStore`, so it is **not** nested-resolution
contention — the hang is in constructing `AkkaSecretReplicator``SecretReplicationActorProvider.ActorRef`
`system.ActorOf(SecretReplicationActor.Props(...))`. Native stacks show `Monitor_Wait`.
**Impact if enabled:** OtOpcUa would hang at startup, since the wiring resolves `ISecretStore` from a
startup hook. It is currently harmless only because `Secrets:Replication:Enabled` defaults false.
**Not explained by the library's own suite:** `TwoNodeClusterReplicationTests` creates the same actor
on a real 2-node cluster and passes (33 tests green). The difference is creation through DI inside a
real host process. Root cause NOT yet identified — candidates: `DistributedPubSub.Get(...)` in
`PreStart` interacting with the `Lazy<IActorRef>` (`ExecutionAndPublication`) lock, or missing
`Akka.Cluster.Tools` reference config in the composed HOCON.
-**Do NOT enable `Secrets:Replication:Enabled` in OtOpcUa.** The Akka topology is not adoptable
until this deadlock is root-caused and fixed, with a regression test that creates the actor
through DI in a hosted process rather than directly from a test.
- ✅ ScadaBridge's hub topology **is** live-validated and safe to enable (config + a shared KEK).
Harness: `scratchpad/livegate/` (two scenarios, `akka` / `sql`; not committed).
### Upstream defect found during adoption — `0.2.0` Akka replication was inert
@@ -99,7 +99,7 @@
"id": 9,
"nativeId": "15",
"subject": "Task 9: Live convergence gate (both clustered apps)",
"status": "pending",
"status": "completed",
"classification": "high-risk",
"blockedBy": [
6,
@@ -110,7 +110,7 @@
"id": 10,
"nativeId": "16",
"subject": "Task 10: Docs reconcile",
"status": "pending",
"status": "completed",
"classification": "small",
"blockedBy": [
9
@@ -124,5 +124,5 @@
}
],
"lastUpdated": "2026-07-18",
"notes": "Tasks 1-8 complete on local feat/secrets-0.2.0 branches in 4 repos (not pushed). Apps ended on Secrets 0.2.1, not 0.2.0: adoption uncovered that 0.2.0's Akka replicator was silently inert (TryAdd order); fixed and republished. Task 9 (live gate) NOT done - needs 2-node rigs. Replication is default-OFF everywhere, so nothing ships enabled."
"notes": "Tasks 1-10 complete. Live gate RUN 2026-07-18 with a SPLIT result: SQL-Server hub (ScadaBridge) PASSED 6/6 against a real SQL Server 2022 incl. partition survival and wrong-KEK fail-closed. Akka peer-to-peer (OtOpcUa) FAILED - resolving ISecretReplicator deadlocks in a real clustered process at ActorOf. Replication remains default-OFF everywhere; do NOT enable it in OtOpcUa until the deadlock is fixed."
}