Re-ran the two design 9 checks that were blocked for OtOpcUa on 2026-07-24, against a
docker-dev rig from OtOpcUa master 412da27c (Health 0.3.0) with the dashboard in a
container on the rig network:
1. one Active per pair: exactly one /health/active 200 per mesh (MAIN central-2,
SITE-A site-a-1, SITE-B site-b-1), dashboard shows one ACTIVE + one STANDBY each;
2. leader chip on every group; network-partitioning site-b-2 from site-b-1 produced
two 1-member clusters, both Active, and the SPLIT BRAIN chip, which cleared once
site-b-2 was restarted (healing the link alone does not merge two Akka clusters,
the documented OtOpcUa recovery); stopping site-a-1 moved the SITE-A chip.
Evidence (screenshots, /health/active bodies, timeline) under
docs/plans/evidence/2026-11-12-overview-phase4/. Ledger task 20 -> completed (20/20).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Adds the family backlog standard to the root index (open work = Gitea issues,
shared label set, BTDB excluded) and converts components/*/GAPS, hosted-lib
CLAUDE.md notes, code-reviews, infra/glauth and plan docs to banner + issue
pointers (scadaproj #5-#19, ScadaBridge#79).
Program closeout: scadaproj#4 CLOSED (comment issuecomment-5751) — ScadaBridge's
central pair now runs ONE shared SQL-Server store in Grpc mode (0.5.0
FallbackEndpoints, 0.5.1 virgin-DB migrator race fix; ScadaBridge @ e697477c),
live-gated PASS 5/5 + re-drill. The pre-Serilog container-wedge defect the gate
reproduced twice is filed as ScadaBridge#34 (pre-existing, out of program scope).
- clustered-secrets.md: central pair = one shared SQL store (connstr literal/env,
never ${secret:}); rewrap = ONE central run + both nodes of each site pair;
FallbackEndpoints same-store-only rule; 0.5.1 minimum for central pairs
- components/secrets/GAPS.md: resolution section appended; stale wait-on-#4 struck
- CLAUDE.md Secrets row: #4 CLOSED, family 0.5.1, gate + residual recorded
- tasks.json: all 6 tasks completed
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Task 9 (closeout) of the Secrets gRPC-hub program. Bookkeeping only; no code.
- clustered-secrets.md, Production enablement: the gRPC hub is no longer
pending. It is BUILT, SHIPPED (0.4.1) and LIVE-GATED PASS 4/4 (ScadaBridge
main @ f6c3f7c5, docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md), so the
"until that ships, ScadaBridge replication stays off" line is replaced with
the truth: the code path is merged and gated, and production enablement is a
separate operator action behind a still-default-off flag. OtOpcUa's
pair-local topology is recorded as live-gated (lmxopcua master @ 1bd79c8e)
rather than pending re-gate. Rotation fan-out corrected: on ScadaBridge it is
BOTH central nodes plus both nodes of every site pair, because each central
node has its own independent store.
- components/secrets/GAPS.md: new "gRPC hub delivered" section recording the
0.4.0/0.4.1 publish, both gates with SHAs and gate-doc paths, and the
residual. Existing narrative left intact and amended, not rewritten.
- CLAUDE.md Secrets row: lib version 0.3.0 -> 0.4.1 and the app-pin claim
corrected (ScadaBridge pins 0.4.1 for the Grpc replicator; the other three
still pin 0.2.3); the "production topologies DECIDED" region now records
delivery, both gates, default-off, and the residual.
- Plan ledger: task 9 completed.
Residual carried forward as scadaproj#4 — ScadaBridge's central pair does not
converge with itself (central-b served an authenticated but EMPTY manifest all
gate while central-a held every secret) and the site client dials a single
endpoint; these are one design question, since failover to an empty hub would
succeed and silently stop convergence. scadaproj#3 is closed.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Bumps the ZB.MOM.WW.Secrets family from 0.3.0 to 0.4.0 and adds the new
ZB.MOM.WW.Secrets.Replicator.Grpc package to the packable set. Publishes
all six 0.4.0 nupkgs to the dohertj2-gitea feed and restore-verifies the
new package resolves from it in a fresh --packages directory. Existing
packages are behaviorally unchanged — no breaking changes.
Adds ZB.MOM.WW.Secrets/CHANGELOG.md (none existed previously) to record
this and prior releases, and lists the new package in README.md's
Packages table. Also includes the controller's pending ledger updates
for docs/plans/2026-08-07-secrets-grpc-hub-implementation.md (Tasks
1-4 and 8 marked completed).
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Design doc + runbook Production-enablement section + GAPS/index updates.
The previously-untracked production-KEK decision is now recorded: one
fleet-wide KEK via appsettings/env (never committed); ScadaBridge rejects
the SqlServer hub on topology grounds (sites talk to central, never the
DB) in favor of a new ZB.MOM.WW.Secrets.Replicator.Grpc package —
pull-only, updates originate at central only, by design.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
The InitJoin self-form watchdog (a timer calling Cluster.Join(SelfAddress) after a
window) was implemented and REJECTED after live testing: it sits outside Akka's join
handshake, so on a routine standby restart — peer alive, join stalled behind removal of
the node's own stale incarnation — it fires mid-join, abandons the in-flight join and
forms a permanent second cluster at the same address (measured: still split after 90s;
OtOpcUa hit it live in its manual-failover drill).
The safety argument 'a booting node only self-forms when no seed answers InitJoin' is
true of Akka's NATIVE first-seed rule (FirstSeedNodeProcess, inside the handshake) and
FALSE of the watchdog, which can fire while a seed is answering. That reasoning error is
now called out explicitly so no future plan re-derives the pattern.
- docs/plans/2026-07-22-initjoin-selfform-fallback.md: dated correction banner (history
preserved) + the two surviving false claims struck in place (behavior-spec row and the
Mechanism section's 'races are benign'); points at ScadaBridge
Component-ClusterInfrastructure.md as the surviving design.
- akka_failover.md 6.1: shipped design stated as self-first seed ordering plus the
opt-in, default-OFF simultaneous-cold-start bootstrap guard; added a
never-re-derive-the-watchdog note. 6.2: two items my 6.1 edit contradicted are marked
closed (OtOpcUa auto-down live gate, mesh election scope — both done 2026-07-24).
- CLAUDE.md: ScadaBridge issue #14 is CLOSED (2026-07-24), not open — v3 dual-namespace
live gates all passed, ScadaBridge consumes OtOpcUa v3 end-to-end.
Promotes the two apps' private replacements for ActiveNodeHealthCheck into one
shared primitive, and retires the leader/RoleLeader selection they were written
to avoid.
Both Akka consumers had already hand-rolled a replacement rather than use this
package (ScadaBridge OldestNodeActiveHealthCheck, OtOpcUa
ClusterPrimaryHealthCheck), so the entire active-node surface here —
ActiveNodeHealthCheck, AkkaActiveNodeGate — had ZERO consumers. It was not
merely unused: it was avoided, twice, for the same reason.
Leadership is address-ordered (host, then port) and has no relationship to time;
singleton placement is age-ordered. The two agree on a freshly-formed cluster,
which is why single-node and happy-path tests never caught it. They diverge
permanently after any restart: the restarted node rejoins as the youngest but
keeps its address, so if it holds the lower address it becomes leader while the
singletons — and all the work they own — stay on the other node.
New ClusterActiveNode is the single implementation: oldest Up member, optional
role scope, plus role-preference resolution for a fused node that must answer
for the role its singletons are pinned to. ActiveNodeHealthCheck and
AkkaActiveNodeGate both delegate to it, so an endpoint gate and the
/health/active probe an orchestrator routes by cannot disagree.
BREAKING (behaviour, not signature):
- Selection is by age, not leadership.
- The role-filtered mode no longer reports Healthy for a node LACKING the role.
That "not applicable => Healthy" made the tier answer 200 on every node and
silently broke leader-pinning (lmxopcua#494); it is now Unhealthy by default,
overridable via NoActiveRoleStatus. This case is reachable, not defensive —
OtOpcUa's RoleParser admits dev-only and cluster-role-only nodes.
- Identity compares UniqueAddress, so a node restarted on the same host:port is
correctly a different member during the overlap.
Results now carry activeRole/selfAddress/activeNode in the 0.2.0 per-entry data
object, so a standby reports WHO is active and a dashboard can render a pair
from either node's payload alone.
Tests: 45 Akka (was 39), 76 total. The age-vs-address divergence is pinned
against a real two-node cluster built so the oldest member is not the
lowest-addressed one, with a fixture sanity check so it cannot pass for the
wrong reason.
Found by the live rig, and invisible to every unit test that existed.
LeaderResolver and InstanceCard.ClusterDataLine both looked the cluster view up
under the check name "akka-cluster". ScadaBridge registers the shared
AkkaClusterHealthCheck under that name; OtOpcUa registers the SAME check as
"akka". So once the OtOpcUa rig was rebuilt on Health 0.2.0 and started
publishing data.leader, its groups still rendered with no leader chip and no
evidence line - and a group with no votes is a legitimate state, so nothing
looked wrong. ScadaBridge's four groups were fully populated the whole time,
which is what made it look like the feature worked.
Both now find the entry by the DATA KEY ("leader"), which is the part of the
contract the shared check actually owns rather than the part each consumer names
for itself.
Regression tests: the leader is found under akka-cluster, akka, and an arbitrary
future name; and an entry that publishes data WITHOUT a leader (ScadaBridge's
localdb check, whose data carries replication counters) does not get picked in
preference to the one that answers the question. Every prior fixture used
ScadaBridge's name, which is precisely why this got through.
After the fix all seven cluster groups render a leader chip with zero
split-brain flags - design section 9 check 2 now passes for BOTH products.
174 tests, 0 warnings.
Found during the live acceptance run. The host calls UseStaticWebAssets only when
the environment is literally "Development", so running the build output under any
other name - checking a staging registry locally, say - leaves the Theme RCL's
assets under _content/ unresolvable. The failure is silent and unhelpful: the page
returns 200 with its markup fully present and renders as a blank white screen,
because every stylesheet 404s. Calling it unconditionally is a no-op once
published (the manifest is gone and the assets are real files on disk), so the
only thing it changes is that failure. The container was never affected.
Phase 4 results recorded in the ledger: 7 of 9 design section 9 checks fully pass,
2 pass for ScadaBridge and are blocked for OtOpcUa pending a rig redeploy.
Notable live evidence:
- Leader chip moved from site-c-a to site-c-b when site-c-a was stopped, and the
split-brain chip appeared while the pair was self-formed then cleared once it
genuinely joined.
- Kill to Unreachable took two failing sweeps (the damping working as designed);
recovery landed on the first good poll.
- Five consecutive page loads in 0.00-0.02s all showed the same "last sweep"
timestamp, which is cache-first stated as a measurement.
- Freezing the probed endpoints while auto-refresh stayed RUNNING froze the poller
mid-sweep and the cards aged into STALE on their own - staleness is computed at
render, so a stuck poller needs no explicit stall detection.
The run also surfaced a pre-existing rig defect in both products: self-first Akka
seed lists mean a pair booted cold self-forms two 1-member clusters that never
join. After a --force-recreate every ScadaBridge node named ITSELF leader and every
node answered /health/active 200; restarting ONE node of each pair converged them
to one Leader plus one Active plus one Standby. That is the user's own open backlog
item (docs/plans/2026-07-22-initjoin-selfform-fallback.md, unexecuted), and the
dashboard made it visible fleet-wide in one screen.
170 tests, 0 warnings.