dohertj2/lmxopcua
1
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity

Clustered secrets: Akka replication blocked by upstream deadlock (scadaproj#1) - keep Secrets:Replication:Enabled=false #482

New Issue
Closed
opened 2026-07-18 14:13:10 -04:00 by dohertj2 · 3 comments
dohertj2 commented 2026-07-18 14:13:10 -04:00
Owner
Copy Link
Copy Source

Summary

Tracker for the OtOpcUa side of a defect whose fix belongs upstream in the shared secrets library.

Upstream issue: scadaproj#1

ZB.MOM.WW.Secrets.Replicator.AkkaDotNet 0.2.1 deadlocks when its replication actor is created through DI in a real hosted process — resolving ISecretReplicator never returns. This blocks OtOpcUa from turning on the Akka clustered-secrets topology.

Current state in this repo — safe, but do not enable

The wiring is merged (master, secrets adoption at ZB.MOM.WW.Secrets 0.2.1) and is inert by design:

  • Secrets:Replication:Enabled defaults to false in src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/SecretsRegistration.cs.
  • With it false, AddOtOpcUaSecrets calls plain AddZbSecrets(...) — behaviour is byte-identical to before the adoption. Guarded by SecretsReplicationRegistrationTests.

⚠️ Do NOT set Secrets:Replication:Enabled=true on any OtOpcUa node until scadaproj#1 is fixed. The startup hook resolves ISecretStore, so the host would hang at startup — including driver-role nodes, whose only visible symptom would be drivers never opening sessions.

Why this is filed here rather than only upstream

The wiring, the config flag, and the live gate that must re-run all live in this repo. When scadaproj#1 is fixed:

  1. Bump ZB.MOM.WW.Secrets.Replicator.AkkaDotNet to the fixed version in Directory.Packages.props.
  2. Re-run the Akka live convergence gate (write→resolve across nodes, tombstone propagation without resurrection, partition survival, negative-KEK fail-closed).
  3. Only then consider enabling the flag, per environment.

Alternative if this stays blocked

The SQL-Server hub topology (ZB.MOM.WW.Secrets.Replicator.SqlServer) passed its live gate 6/6 against a real SQL Server 2022 and is unaffected by this defect. OtOpcUa already runs a central config DB, so shared-store or hub mode is a viable fallback if the Akka path cannot be fixed — at the cost of coupling driver-node secret resolution to database reachability, which is why peer-to-peer was preferred.

References

  • Gate results + full ruled-out list: scadaproj/components/secrets/GAPS.md
  • Plan: scadaproj/docs/plans/2026-07-18-secrets-0.2.0-upgrade-and-clustering.md (Task 9)
## Summary Tracker for the OtOpcUa side of a defect whose fix belongs upstream in the shared secrets library. **Upstream issue: [`scadaproj#1`](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/1)** `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` 0.2.1 deadlocks when its replication actor is created through DI in a real hosted process — resolving `ISecretReplicator` never returns. This blocks OtOpcUa from turning on the Akka clustered-secrets topology. ## Current state in this repo — safe, but do not enable The wiring is **merged** (`master`, secrets adoption at `ZB.MOM.WW.Secrets` 0.2.1) and is **inert by design**: - `Secrets:Replication:Enabled` **defaults to `false`** in `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/SecretsRegistration.cs`. - With it false, `AddOtOpcUaSecrets` calls plain `AddZbSecrets(...)` — behaviour is byte-identical to before the adoption. Guarded by `SecretsReplicationRegistrationTests`. ⚠️ **Do NOT set `Secrets:Replication:Enabled=true` on any OtOpcUa node until scadaproj#1 is fixed.** The startup hook resolves `ISecretStore`, so the host would **hang at startup** — including driver-role nodes, whose only visible symptom would be drivers never opening sessions. ## Why this is filed here rather than only upstream The wiring, the config flag, and the live gate that must re-run all live in this repo. When scadaproj#1 is fixed: 1. Bump `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` to the fixed version in `Directory.Packages.props`. 2. Re-run the Akka live convergence gate (write→resolve across nodes, tombstone propagation without resurrection, partition survival, negative-KEK fail-closed). 3. Only then consider enabling the flag, per environment. ## Alternative if this stays blocked The SQL-Server hub topology (`ZB.MOM.WW.Secrets.Replicator.SqlServer`) **passed its live gate 6/6** against a real SQL Server 2022 and is unaffected by this defect. OtOpcUa already runs a central config DB, so shared-store or hub mode is a viable fallback if the Akka path cannot be fixed — at the cost of coupling driver-node secret resolution to database reachability, which is why peer-to-peer was preferred. ## References - Gate results + full ruled-out list: `scadaproj/components/secrets/GAPS.md` - Plan: `scadaproj/docs/plans/2026-07-18-secrets-0.2.0-upgrade-and-clustering.md` (Task 9)
dohertj2 commented 2026-07-18 14:55:40 -04:00
Author
Owner
Copy Link
Copy Source

Unblocked: the upstream deadlock (scadaproj#1) is root-caused, fixed, and ZB.MOM.WW.Secrets.* 0.2.2 is published to the Gitea feed (restore-verified). It was a DI singleton cycle in the Akka package's wiring (store decorator → replicator → actor provider → cache invalidator → resolver → store), not anything Akka-side; fixed by deferring the invalidator edge to first eviction. The 2-node Akka live convergence gate now passes 6/6 against the fixed lib. Next here: bump OtOpcUa to 0.2.2, re-run the gate on the OtOpcUa topology, then consider enabling Secrets:Replication:Enabled.

Unblocked: the upstream deadlock (scadaproj#1) is root-caused, fixed, and **`ZB.MOM.WW.Secrets.*` `0.2.2` is published to the Gitea feed** (restore-verified). It was a DI singleton cycle in the Akka package's wiring (store decorator → replicator → actor provider → cache invalidator → resolver → store), not anything Akka-side; fixed by deferring the invalidator edge to first eviction. The 2-node Akka live convergence gate now passes **6/6** against the fixed lib. Next here: bump OtOpcUa to `0.2.2`, re-run the gate on the OtOpcUa topology, then consider enabling `Secrets:Replication:Enabled`.
dohertj2 commented 2026-07-18 15:03:54 -04:00
Author
Owner
Copy Link
Copy Source

Items 1–2 done — master @ 2254ae3d

1. Bumped. All four ZB.MOM.WW.Secrets.* pins 0.2.1 → 0.2.2 in Directory.Packages.props. Full solution builds with 0 errors.

2. Gate re-run — 6/6 PASS, against the published 0.2.2 feed packages (not source refs): cluster forms → write on A resolves on B (shared-KEK decrypt) → delete tombstones on B with no resurrection → tombstoned secret no longer resolves through B's cache (the check that proves the deferred invalidator evicts) → reverse-direction write converges → wrong-KEK node fails closed via SecretDecryptionException.

Bonus — the promised test now exists. SecretReplicationStarter's docs referenced a skipped test The_startup_hook_actually_creates_the_replication_actor that had never been written because the resolve hung. It's now a real provider-based test: container built exactly as the host builds it, SecretReplicationStarter run under a 20 s watchdog, and the replication actor proven to exist via ActorSelection on a self-joined single-node cluster (plain ActorSystem, no TestKit — sidesteps the xunit.v3/Akka.TestKit.Xunit2 conflict). On 0.2.0 that test fails (no actor); on 0.2.1 it hits the watchdog (the deadlock); on 0.2.2 it passes. SecretsReplicationRegistrationTests 8/8. Also corrected the stale comment that blamed the old hang on DistributedPubSub needing a joined cluster — the actor constructor was never reached; it was the DI cycle (scadaproj#1).

Remaining (item 3): Secrets:Replication:Enabled stays default-false. Enabling is a per-environment operational decision — needs the same KEK distributed to every node first (Secrets:MasterKey), per ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md. Leaving this issue open to track that decision.

## Items 1–2 done — `master` @ `2254ae3d` **1. Bumped.** All four `ZB.MOM.WW.Secrets.*` pins `0.2.1` → `0.2.2` in `Directory.Packages.props`. Full solution builds with 0 errors. **2. Gate re-run — 6/6 PASS**, against the **published 0.2.2 feed packages** (not source refs): cluster forms → write on A resolves on B (shared-KEK decrypt) → delete tombstones on B with no resurrection → tombstoned secret no longer resolves through B's cache (the check that proves the deferred invalidator evicts) → reverse-direction write converges → wrong-KEK node fails closed via `SecretDecryptionException`. **Bonus — the promised test now exists.** `SecretReplicationStarter`'s docs referenced a skipped test `The_startup_hook_actually_creates_the_replication_actor` that had never been written because the resolve hung. It's now a real provider-based test: container built exactly as the host builds it, `SecretReplicationStarter` run under a 20 s watchdog, and the replication actor proven to exist via `ActorSelection` on a self-joined single-node cluster (plain `ActorSystem`, no TestKit — sidesteps the xunit.v3/`Akka.TestKit.Xunit2` conflict). On 0.2.0 that test fails (no actor); on 0.2.1 it hits the watchdog (the deadlock); on 0.2.2 it passes. `SecretsReplicationRegistrationTests` 8/8. Also corrected the stale comment that blamed the old hang on `DistributedPubSub` needing a joined cluster — the actor constructor was never reached; it was the DI cycle (scadaproj#1). **Remaining (item 3):** `Secrets:Replication:Enabled` stays **default-false**. Enabling is a per-environment operational decision — needs the same KEK distributed to every node first (`Secrets:MasterKey`), per `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`. Leaving this issue open to track that decision.
dohertj2 commented 2026-07-18 16:00:52 -04:00
Author
Owner
Copy Link
Copy Source

Item 3 done — replication ENABLED and live-proven on the docker-dev rig (master @ c878fbbd)

All six host nodes (central-1/2, site-a-1/2, site-b-1/2) now run with Secrets:Replication:Enabled=true via one x-secrets-env compose anchor: replication on, AnnounceInterval=00:00:05, and the ONE shared dev-only KEK (${OTOPCUA_SECRETS_KEK:-…} committed default per the rig's zero-operator-step pattern; every node derives kek_id sha256:f3f0c2056e7a).

Evidence (rig left running in this state):

  • No startup deadlock — all six nodes reached Application started, repeatedly, across restarts; the scadaproj#1 hang signature never appeared. Mesh formed (all members Up).
  • Convergence proven at the strongest level: a secret (gate/dockerdev) seeded into site-a-1's local store only propagated to all five peers within ≤60 s. All six nodes' SQLite stores held the row with byte-identical ciphertext (same sha256 over ciphertext‖nonce‖tag, same kek_id, rev=0) — replication, not re-encryption — and the rows copied from central-2 and site-b-2 decrypt to the exact plaintext under the shared KEK.
  • Restart robustness: docker compose restart site-b-2 → rejoined cleanly, row intact, zero errors. The fleet also survived a full local Docker-engine restart with replication enabled (environmental OrbStack wedge mid-verification, not an app defect — no deadlock, no crash).

Found during verification, filed separately: #483 — the AdminUI /admin/secrets page renders but is non-interactive (Secrets.Ui RCL components have no @rendermode; the host applies render modes per-page), so "Add secret" does nothing. Pre-existing, unrelated to replication.

With bump (0.2.2), gate re-run (6/6 vs feed packages), and enablement all done and proven, this tracker is complete — closing. Production/wonder enablement, if wanted later, needs its own decision + real KEK distribution (NOT this dev key) per ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md, and deserves a fresh issue.

## Item 3 done — replication ENABLED and live-proven on the docker-dev rig (`master` @ `c878fbbd`) All six host nodes (central-1/2, site-a-1/2, site-b-1/2) now run with `Secrets:Replication:Enabled=true` via one `x-secrets-env` compose anchor: replication on, `AnnounceInterval=00:00:05`, and the ONE shared dev-only KEK (`${OTOPCUA_SECRETS_KEK:-…}` committed default per the rig's zero-operator-step pattern; every node derives `kek_id sha256:f3f0c2056e7a`). **Evidence (rig left running in this state):** - **No startup deadlock** — all six nodes reached `Application started`, repeatedly, across restarts; the scadaproj#1 hang signature never appeared. Mesh formed (all members Up). - **Convergence proven at the strongest level:** a secret (`gate/dockerdev`) seeded into **site-a-1's local store only** propagated to **all five peers within ≤60 s**. All six nodes' SQLite stores held the row with **byte-identical ciphertext** (same sha256 over ciphertext‖nonce‖tag, same kek_id, rev=0) — replication, not re-encryption — and the rows copied from central-2 and site-b-2 **decrypt to the exact plaintext** under the shared KEK. - **Restart robustness:** `docker compose restart site-b-2` → rejoined cleanly, row intact, zero errors. The fleet also survived a full local Docker-engine restart with replication enabled (environmental OrbStack wedge mid-verification, not an app defect — no deadlock, no crash). **Found during verification, filed separately:** #483 — the AdminUI `/admin/secrets` page renders but is non-interactive (Secrets.Ui RCL components have no `@rendermode`; the host applies render modes per-page), so "Add secret" does nothing. Pre-existing, unrelated to replication. With bump (0.2.2), gate re-run (6/6 vs feed packages), and enablement all done and proven, this tracker is complete — closing. Production/wonder enablement, if wanted later, needs its own decision + real KEK distribution (NOT this dev key) per `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`, and deserves a fresh issue.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
auto-managed
queue management
 cross-driver
queue management
 driver/abcip
queue management
 driver/ablegacy
queue management
 driver/focas
queue management
 driver/opcuaclient
queue management
 driver/s7
queue management
 driver/twincat
queue management
 phase/1
queue management
 phase/2
queue management
 phase/3
queue management
 phase/4
queue management
 phase/5
queue management
 phase/6
queue management
 queue/blocked
queue management
 queue/done
queue management
 queue/failed
queue management
 queue/in-progress
queue management
 queue/queued
queue management
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#482
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?