docs(secrets): live convergence gate results - SQL hub PASSES 6/6, Akka FAILS
SQL-Server hub topology (ScadaBridge) is live-validated against a real SQL Server 2022 with two nodes: write->resolve via hub, tombstone propagation without resurrection, bidirectional sweep, partitioned-node local resolution (site autonomy - the reason hub mode was chosen), and wrong-KEK fail-closed via SecretDecryptionException. Akka peer-to-peer topology (OtOpcUa) FAILS the gate: resolving ISecretReplicator hangs indefinitely in a real clustered process. Reproduced twice on a healthy 2-node cluster; isolated to constructing AkkaSecretReplicator -> ActorRef -> system.ActorOf, NOT nested DI resolution. The library's own 2-node suite passes, so the trigger is creation through DI in a hosted process. Root cause not yet identified. Impact: OtOpcUa would hang at startup if replication were enabled, since the startup hook resolves ISecretStore. Harmless today only because the flag defaults false. Do NOT enable Secrets:Replication:Enabled in OtOpcUa. The gate did its job - it caught a blocking defect that every offline suite, including the library's own real-cluster tests, had passed over.
This commit is contained in:
@@ -84,10 +84,50 @@ construction).
|
||||
`Validate()` eagerly, so unconditional wiring would have broken startup on every node.
|
||||
- Bootstrap constraint recorded: the hub connstr **cannot come from the hub**; it must be seeded
|
||||
in each node's local store or supplied via env.
|
||||
- ⬜ **Task 9, the G-2-class live gate, is NOT done** — write→resolve across nodes, tombstone
|
||||
propagation (no resurrection), partition survival, and the negative-KEK fail-closed control.
|
||||
Needs a 2-node OtOpcUa rig and a ScadaBridge central-pair + site + SQL hub. **The topology is
|
||||
not "adopted" until this passes.** Do not enable `Secrets:Replication:Enabled` anywhere first.
|
||||
- **Task 9 live gate RUN 2026-07-18. Split result — one topology passed, one FAILED.**
|
||||
|
||||
### Live gate: SQL-Server hub (ScadaBridge topology) — ✅ 6/6 PASSED
|
||||
|
||||
Run against a **real SQL Server 2022** (`10.100.0.35:31433`, DB `ZbSecretsGate`) with two nodes,
|
||||
each with its own local SQLite store, using `AddZbSecretsSqlServerReplication` — the same extension
|
||||
`AddScadaBridgeSecrets` calls:
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| hub schema provisions + sweeps start | ✅ |
|
||||
| write on A → resolves on B via the hub | ✅ converged through the shared hub |
|
||||
| delete on A → tombstone on B | ✅ no resurrection |
|
||||
| write on B → resolves on A | ✅ sweep is bidirectional |
|
||||
| **partitioned node still resolves from local state** | ✅ hub unreachable, local reads unaffected — **site autonomy holds**, which is the whole reason ScadaBridge uses hub mode |
|
||||
| **wrong-KEK node fails closed** | ✅ `SecretDecryptionException`, not wrong plaintext |
|
||||
|
||||
The KEK control fired spontaneously first: a re-run against a hub still holding rows from a prior
|
||||
run (different KEK) threw `SecretDecryptionException` rather than returning garbage — unplanned but
|
||||
real evidence of fail-closed.
|
||||
|
||||
### Live gate: Akka peer-to-peer (OtOpcUa topology) — ❌ FAILED, blocking defect
|
||||
|
||||
**Resolving `ISecretReplicator` hangs indefinitely** in a real clustered process. Reproduced twice
|
||||
on a healthy 2-node cluster (both members confirmed `Up` before the hang). Isolated by resolving
|
||||
`ISecretReplicator` *directly* rather than via `ISecretStore`, so it is **not** nested-resolution
|
||||
contention — the hang is in constructing `AkkaSecretReplicator` → `SecretReplicationActorProvider.ActorRef`
|
||||
→ `system.ActorOf(SecretReplicationActor.Props(...))`. Native stacks show `Monitor_Wait`.
|
||||
|
||||
**Impact if enabled:** OtOpcUa would hang at startup, since the wiring resolves `ISecretStore` from a
|
||||
startup hook. It is currently harmless only because `Secrets:Replication:Enabled` defaults false.
|
||||
|
||||
**Not explained by the library's own suite:** `TwoNodeClusterReplicationTests` creates the same actor
|
||||
on a real 2-node cluster and passes (33 tests green). The difference is creation through DI inside a
|
||||
real host process. Root cause NOT yet identified — candidates: `DistributedPubSub.Get(...)` in
|
||||
`PreStart` interacting with the `Lazy<IActorRef>` (`ExecutionAndPublication`) lock, or missing
|
||||
`Akka.Cluster.Tools` reference config in the composed HOCON.
|
||||
|
||||
- ⬜ **Do NOT enable `Secrets:Replication:Enabled` in OtOpcUa.** The Akka topology is not adoptable
|
||||
until this deadlock is root-caused and fixed, with a regression test that creates the actor
|
||||
through DI in a hosted process rather than directly from a test.
|
||||
- ✅ ScadaBridge's hub topology **is** live-validated and safe to enable (config + a shared KEK).
|
||||
|
||||
Harness: `scratchpad/livegate/` (two scenarios, `akka` / `sql`; not committed).
|
||||
|
||||
### Upstream defect found during adoption — `0.2.0` Akka replication was inert
|
||||
|
||||
|
||||
@@ -99,7 +99,7 @@
|
||||
"id": 9,
|
||||
"nativeId": "15",
|
||||
"subject": "Task 9: Live convergence gate (both clustered apps)",
|
||||
"status": "pending",
|
||||
"status": "completed",
|
||||
"classification": "high-risk",
|
||||
"blockedBy": [
|
||||
6,
|
||||
@@ -110,7 +110,7 @@
|
||||
"id": 10,
|
||||
"nativeId": "16",
|
||||
"subject": "Task 10: Docs reconcile",
|
||||
"status": "pending",
|
||||
"status": "completed",
|
||||
"classification": "small",
|
||||
"blockedBy": [
|
||||
9
|
||||
@@ -124,5 +124,5 @@
|
||||
}
|
||||
],
|
||||
"lastUpdated": "2026-07-18",
|
||||
"notes": "Tasks 1-8 complete on local feat/secrets-0.2.0 branches in 4 repos (not pushed). Apps ended on Secrets 0.2.1, not 0.2.0: adoption uncovered that 0.2.0's Akka replicator was silently inert (TryAdd order); fixed and republished. Task 9 (live gate) NOT done - needs 2-node rigs. Replication is default-OFF everywhere, so nothing ships enabled."
|
||||
"notes": "Tasks 1-10 complete. Live gate RUN 2026-07-18 with a SPLIT result: SQL-Server hub (ScadaBridge) PASSED 6/6 against a real SQL Server 2022 incl. partition survival and wrong-KEK fail-closed. Akka peer-to-peer (OtOpcUa) FAILED - resolving ISecretReplicator deadlocks in a real clustered process at ActorOf. Replication remains default-OFF everywhere; do NOT enable it in OtOpcUa until the deadlock is fixed."
|
||||
}
|
||||
Reference in New Issue
Block a user