Task 26 — the P2 milestone gate. Ran the live /run verification on an isolated
docker-dev rig (project otopcua-mqtt, ports 9210/4850-4851/14350, image built
from this branch) against the real Mosquitto TLS+auth broker and the C#
Sparkplug edge-node simulator on 10.100.0.35, and recorded the result.
The gate found three defects. Two are the same defect class the P1 gate found
twice — a hand-maintained AdminUI surface left behind by a driver-side feature —
and the third is a pre-existing cross-driver bug that only a Sparkplug flow
could surface.
1. CRITICAL, FIXED — MqttDriverForm still shipped its P1 Sparkplug PLACEHOLDER.
Switching Mode to SparkplugB rendered a "not available yet" notice and NO
Group ID field, so with Sparkplug ingest fully shipped there was still no
way to author a Sparkplug driver from the AdminUI at all. Sparkplug.GroupId
is the driver's entire subscription filter (spBv1.0/{GroupId}/#): blank ⇒
connected, Healthy, ingesting nothing. Now authors all five Sparkplug keys,
MERGES over the existing sub-object rather than replacing it (so a key a
newer driver adds inside it survives an older AdminUI), leaves it untouched
in Plain mode, and validates the group id as the topic segment it is.
Pinned by 8 MqttDriverFormModelTests cases, verified falsifiable — 8 RED
with the fix stubbed out.
2. PRE-EXISTING + CROSS-DRIVER, FIXED — every node label in the shared
DriverBrowseTree was an <a href="#">. In a Blazor Web App, blazor.web.js's
enhanced-navigation click interceptor resolves a bare "#" against
<base href="/">, so clicking ANY browse-tree node label navigated the whole
AdminUI to "/", tore down the circuit, and destroyed the hosting modal —
losing the browse session AND the tag selection. @onclick:preventDefault
does not help: it suppresses the browser's default action, not Blazor's own
interceptor. Labels are now <button type="button">. Dates to the component's
introduction (2026-05-28) and affects EVERY driver's picker; it surfaced
only now because choosing a rebirth scope is the first flow that requires
clicking a label rather than the ▶ toggle (always a button, always fine).
3. FIXED (mitigation) — the browse tree rendered exactly once, at open, and
never again, while OpenAsync returns as soon as the SUBSCRIBE is granted.
On Sparkplug that means it is almost always empty forever, because births
are never retained. Added a Refresh button that re-reads the root against
the SAME session (nothing reconnects, nothing observed is lost, the tag
selection is kept, only the armed rebirth scope is cleared).
Live gate result (all against the real broker + simulator, group OtOpcUaSim):
- driver authored end-to-end through the fixed MqttDriverForm; blob is
Mode:SparkplugB + Sparkplug.GroupId:OtOpcUaSim with enums as names
- browse: BROWSER OPEN chip, Sparkplug-only rebirth panel, tree renders
Group → EdgeNode → [Device] → Metric with the device folder Filler1 and
node-level metric leaves SIDE BY SIDE, and "Node Control/Rebirth" as ONE
leaf (the metric-name-is-not-a-topic-segment rule holding)
- rebirth: group scope names the group + the 32-node cap, Cancel published
NOTHING (simulator log unchanged); edge-node scope → "1 command published"
and the simulator logged "rebirth NCMD #1 received; republishing metadata";
group scope → "2 commands published", both edge nodes re-birthed; a device
and a metric both resolve UP to "edge node EdgeA"; the panel does NOT render
for a Plain MQTT device
- browse-commit wrote bindable tuples with no topic/address key —
{"groupId","edgeNodeId","metricName","deviceId"} for the device metric and
deviceId ABSENT (not blank) for the node-level one, datatypes inherited from
the birth (Int64 / Float)
- deployed (Sealed), and both nodes served live changing values through OPC UA
at the 2 s cadence — Good 0x00000000, no BadNodeIdUnknown
- death→STALE: stopping the simulator drove both nodes to 0x80000000 with an
empty value ~2 s later, and restarting it recovered them to Good on the new
birth (FillCount back to its birth-declared 1000, then counting)
- rediscovery fired exactly TWICE (once per authored scope: OtOpcUaSim/EdgeA
7 metrics, OtOpcUaSim/EdgeA/Filler1 3 metrics) and was then gated across
many subsequent births/rebirths — the anti-storm change gate working, and
EdgeB correctly filtered out as an unauthored scope
- tag editor: opens in SparkplugB (mode inference on reopen) with all four
descriptor fields populated; blank group → "A Sparkplug group ID is
required."; "Plant/1" rejected; "Node Control/Rebirth" ACCEPTED and saved
with the slash whole; dataType override persists and its key is ABSENT again
after selecting "(from birth certificate)"
Two things are NOT verified, and are recorded rather than claimed:
- Rediscovery is INERT in v3 and this is a platform gap, not an MQTT one:
nothing subscribes to IRediscoverable.OnRediscoveryNeeded and
DriverHostActor.HandleDiscoveredNodes hard-returns. A DBIRTH introducing a
metric does NOT change the OPC UA tree; redeploy. Verified from the driver's
log line only, exactly as far as it can be verified.
- A metric name containing "/" cannot be BROWSE-COMMITTED: the derived raw tag
Name is a RawPath segment and may not contain "/", so the spec-mandatory
"Node Control/Rebirth" is refused ("Row 3: Name must not contain '/'"). The
refusal is loud and all-or-nothing, never a silently mis-bound tag, and
Manual entry is the working path. Fixing it needs a name-sanitisation policy
with a collision answer, so it was recorded rather than guessed at.
Also left open + documented: an empty browse tree cannot arm a rebirth (the
scope comes from a tree click, and the session side already accepts a bare
{group}/{edgeNode} — only a UI path is missing); _canRebirth is captured at
browse-open; the tag editor injects the Plain-only payloadFormat key into a
Sparkplug blob (cosmetic — the factory routes on the tuple).
Suites: offline 1528 passed / 0 failed (581 Driver.Mqtt + 809 AdminUI + 138
Core.Abstractions); live 15/15 against the broker + simulator.
Docs: docs/drivers/Mqtt.md brought fully up to date for P2 (Sparkplug config
sub-object, both tag shapes, the ingest state machine, rediscovery, browse +
rebirth + Refresh, a Known-gaps table); Mqtt-Test-Fixture.md gains the simulator
and drops its "Sparkplug has no fixture" claims; infra/README.md §3 gains the
simulator row; CLAUDE.md gains the simulator endpoint facts; the tracking doc
marks MQTT/Sparkplug COMPLETE. docker-dev/docker-compose.mqtt.yml is the
isolated-rig overlay the gate ran on.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
0.2.1 makes the shared ActiveNodeHealthCheck return Unhealthy (503), not
Degraded (200), for a node that carries the role but is not the role leader.
That is what the shared health spec always specified, what docs/ServiceHosting.md,
docs/v2/Architecture-v2.md and the 2026-05-26 alignment design all describe
("200 only on the Admin-role leader; 503 elsewhere"), and what
docker-dev/traefik-dynamic.yml + scripts/install/traefik-dynamic.yml assume when
they use /health/active as the load-balancer probe.
The implementation returned Degraded, which MapZbHealth maps to 200, so BOTH
central nodes always passed the probe and Traefik load-balanced the admin UI
across the pair instead of pinning it to the leader. The docs were right; the
code was wrong. Closes the Traefik half of #494.
No OtOpcUa code change — package pin only.
Live-verified on docker-dev, with the MAIN pair genuinely formed (memberCount 2,
both members naming one leader):
central-1 (leader) /health/active -> 200 traefik UP
central-2 (standby) /health/active -> 503 traefik DOWN
Failover drill (README step 2), which had never actually exercised anything:
stopping central-1 flipped Traefik to central-2 within 20 s and
http://localhost:9200/ answered 200 continuously across the swap — no outage.
Restarting central-1 reclaimed the role-leader and Traefik flipped back.
README corrected on two counts: the drill now states that exactly ONE backend is
UP by design, and step 3 no longer claims central-2 keeps leadership — RoleLeader
is the lowest-address member, so central-1 deterministically reclaims it. Added
the cold-boot caveat that starting both nodes at once can leave each self-forming
a 1-member cluster, in which case both are their own leader and both answer 200.
STILL OPEN in #494: driver-only nodes. The check is scoped to the admin role and
returns Healthy for any node that lacks it, so all four site nodes answer 200 and
the real per-Cluster redundancy Primary (oldest Up driver member, IsDriverPrimary)
is still not what this tier reports.
Closes the residual Phase-7 gap: when BOTH nodes of a 2-node pair cold-start at the
same instant, each self-first seed runs FirstSeedNodeProcess, times out waiting for
the other, and forms its own 1-node cluster — two Primaries in one pair (the Phase 6
live gate reproduced this reliably on docker). The prior mitigation was operational
(staggered start / compose depends_on), which does not exist on production hardware.
The guard (dark switch Cluster:BootstrapGuard:Enabled, default OFF) prevents the split
without giving up cold-start-alone:
- The lower-address node is the preferred founder: self-first, forms immediately.
- The higher node probes its partner's Akka port (TCP connect) up to PartnerProbeSeconds:
reachable => peer-first (join the founder, never race it); unreachable => self-first
(partner is dead, form alone). The order is decided BEFORE the single JoinSeedNodes,
from an explicit reachability signal — never re-formed mid-handshake (the retired
SelfFormAfter failure mode). When on, Akka gets no config seeds (BuildClusterOptions)
and ClusterBootstrapCoordinator drives the join.
Review-driven hardening: case-insensitive tie-break (a hostname-casing mismatch would
reopen the split); fail-fast validation of the timing knobs; the residual "founder dies
in the probe->join window" hang is documented and made operator-visible (warning + a
restart recovers). Real-ActorSystem coordinator tests cover the load-bearing higher-node
cold-start-alone case; 147/147 Cluster tests pass.
Live-gated on docker-dev (site-a = enablement demo, serialization removed, guard on;
site-b keeps depends_on-serialization + guard off as the A/B control): simultaneous
start -> site-a-1 founds, site-a-2 probes-reachable-joins -> 250/240, NO split;
higher-node-alone -> probes dead founder, self-forms -> 250; founder rejoin -> 240.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Two docker-dev-only defects the Phase 6 live gate surfaced (both invisible to
`docker compose config`, which validates the file but never boots the app):
1. Driver-only site nodes crashed at boot with OptionsValidationException — LDAP
transport None + AllowInsecure false. Only central carried the Security__Ldap__*
block, but the site nodes serve OPC UA endpoints and validate LDAP options too, and
they override `environment` wholesale (YAML `<<:` doesn't deep-merge it). Extracted
the block to a shared &ldap-env anchor merged into every host node.
2. Site pairs split-brained at simultaneous startup (both nodes "JOINING itself",
250/250 instead of 250/240). Both are self-first seeds, so both run Akka's
FirstSeedNodeProcess; the partner didn't depend on its pair founder, so they raced
and each formed a 1-node cluster. Added an &akka-founder-healthcheck (bash /dev/tcp
probe of Akka 4053 — no nc/curl in the image) to site-a-1/site-b-1 and switched the
partners to depends_on service_healthy, so the founder forms first and the partner
JOINS it. (Carry the underlying simultaneous-cold-start property to Phase 7 — it is
inherent to symmetric self-first seeding, present for central since #459.)
Live gate record: docs/plans/2026-07-24-mesh-phase6-live-gate.md (legs 1-2 PASS —
three isolated meshes, one Primary per pair at 250/240).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Rewrite docker-dev/docker-compose.yml from one shared six-node gossip ring
into three self-contained pairs (MAIN: central-1/2, SITE-A: site-a-1/2,
SITE-B: site-b-1/2), all still on the ActorSystem name `otopcua` and the one
shared docker network — separation is purely by per-pair self-first
Cluster:SeedNodes, faithful to how production splits over a WAN rather than
a firewall.
Per node:
- Cluster__Roles / OTOPCUA_ROLES gain a `cluster-<ClusterId>` role
(cluster-MAIN / cluster-SITE-A / cluster-SITE-B), the marker
SplitTopologyTransportValidator and ClusterRoleInfo key off.
- Cluster__SeedNodes now lists only the node's own pair (self first, partner
second); site nodes no longer seed off central-1.
- MeshTransport__Mode default flips Dps -> ClusterClient (still overridable
via OTOPCUA_MESH_MODE) on all six nodes — SplitTopologyTransportValidator
now refuses to boot a cluster-role node left on Dps.
- Telemetry__Mode=Grpc added on all six driver nodes; TelemetryDial__Mode=Grpc
added on central-1/central-2 only (the admin pair).
- ConfigSource/ConfigServe modes and MeshTransport contact points are
unchanged (already split-correct since Phases 3/4/2).
seed-clusters.sql needed no changes — the ClusterNode rows already carry the
correct ClusterId/Host/AkkaPort/GrpcPort per node. Validated with
`docker compose -f docker-dev/docker-compose.yml config`.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The four driver-only site nodes lose their ConfigDb connection string and hardcode
ConfigSource:Mode=FetchAndCache (Direct is now rejected by the validator for a
driver-only node). Central-1/2 keep ConfigDb + Direct + ConfigServe. Proves the
Phase-4 cut on the rig for the live gate.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 3 added an in-repo .proto compiled by Grpc.Tools; its bundled linux_arm64 protoc
segfaults (exit 139) on Apple-Silicon Docker, so the native-arm64 build stage could not
generate the gRPC stubs. dotnet publish is framework-dependent (portable IL + every RID's
native assets), so pinning ONLY the build stage to linux/amd64 leaves the runtime image
native arm64 (fast) while the one-time build is emulated. Matches the Dockerfile's stated
'targeting linux-x64' intent.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 3 Task 9. docker-dev rig: central-1/2 carry ConfigServe (:4055 h2c + committed
dev key) and stay Direct; the four site nodes carry ConfigSource (both central
endpoints + matching key) defaulting to Direct. Flip only the site nodes with
OTOPCUA_CONFIG_MODE=FetchAndCache at 'docker compose up' — central keeps SQL and models
the target topology. Binding :4055 activates the dedicated-h2c Kestrel takeover on
central (re-binds :9000 too). Docs: new docs/Configuration.md ConfigSource/ConfigServe
section; docs/Redundancy.md 'config bytes travel out-of-band' note; the program plan
Phase 3 section + tracking row flipped to CODE-COMPLETE; a CLAUDE.md 'Config source'
section.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Rig: all six docker-dev nodes carry MeshTransport__Mode plus both central
contact points. Mode is "${OTOPCUA_MESH_MODE:-Dps}", so the rig still comes up
on the transport it has always used and the live gate can flip the whole fleet
at `docker compose up` without a compose edit or a rebuild.
Docs: a MeshTransport section in Configuration.md (including why contact points
are addresses only, why the contact set does not scope delivery, and why
buffer-size=0 is a behaviour decision rather than a tuning knob), a "Command
transport" section in Redundancy.md contrasting the two modes, and a summary
beside the Redundancy notes in CLAUDE.md.
Corrects the design doc's claim that ScadaBridge replies with a typed failure to
every unhandled message. Neither of their comm actors has a ReceiveAny or an
Unhandled override; the idiom is per message type and fires on a missing
registration, and an unknown type dead-letters there as it does anywhere else.
Records the exit-gate deviation in the program plan: "an Ask timing out cleanly
against a stopped node" cannot be run, because no Ask crosses this boundary. The
honest equivalents are a stopped node failing the deploy at the apply deadline
and an unreachable contact dropping the command with a Warning.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
- docker-dev seed: AkkaPort = 4053 on all six ClusterNode rows, so a freshly
seeded rig matches a migrated one. GrpcPort left null.
- config-db-schema.md: both columns, why AkkaPort is NOT NULL/4053 and GrpcPort
is nullable, the unenforced duplication + its reconciler, and Enabled's new
second meaning as the deploy path's expected-ack set.
- Configuration.md: Cluster:Port / PublicHostname now flag that they are stored
twice, with the "update the row too" instruction and why the drift is silent.
- design doc §7: Phase 1 marked done, plus a "Phase 1 as shipped" note recording
both deviations rather than leaving the sketch reading as what happened.
- program plan: Phase 1 marked done; AdminUI node edit explicitly deferred.
- CLAUDE.md: the deploy-path behaviour change and its three consequences.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Akka runs FirstSeedNodeProcess -- the only bootstrap path that can form a NEW
cluster when no peer answers InitJoin -- exclusively when seed-nodes[0] is the
node's own address. Every other node runs JoinSeedNodeProcess and retries
InitJoin forever. That is why "both peers are seeds" never meant "either can
cold-start alone", and it is fixed here the way Akka itself intends: each seed
node lists ITSELF first.
- docker-dev central-2 now lists itself as SeedNodes__0 (central-1 was already
self-first). The site nodes are untouched -- they seed off central-1 only and
are deliberately not seeds.
- AkkaClusterOptionsValidator enforces the invariant at boot via the shared
ZB.MOM.WW.Configuration AddValidatedOptions/ValidateOnStart seam. The rule is
CONDITIONAL -- self must be seed[0] only IF self is in the list at all -- or it
would refuse to boot every driver-only site node. Identity is compared on
PublicHostname (falling back to Hostname when blank) AND port, i.e. the address
Akka puts in SelfAddress: matching the 0.0.0.0 bind address would find no seed
anywhere and leave the rule silently inert on the whole docker-dev rig.
- ClusterBootstrapFallback + Cluster:SelfFormAfter are DELETED. The watchdog sat
outside Akka's join handshake, so it could not distinguish "no seed answered"
from "a seed answered and the join is in flight" -- and Cluster.Join(SelfAddress)
is not ignored mid-handshake, it wins. The live gate (ea45ace1) caught it
islanding a node that a manual failover had bounced: InitJoinAck received, no
Welcome inside the window because the peer's ring still held the old
incarnation, watchdog fired, second cluster. The TCP reachability guard patched
that one shape; the race stayed. It was also inert for every non-seed node, so
after this change it can never usefully fire.
- SelfFormBootstrapTests -> SelfFirstSeedBootstrapTests: real in-process clusters
through the production bootstrap at production failure-detection timings, incl.
a falsifiability control proving the OLD peer-first ordering never forms (that
test failed at 11s against the watchdog, which is what proved the retirement),
the restart-into-a-live-peer case that killed the watchdog, and a simultaneous
cold start converging on ONE cluster.
Mirrors ScadaBridge 4a6341d8; anticipates per-cluster-mesh-program Phase 6, which
had this retirement queued.
Gate record: docs/plans/2026-07-20-localdb-phase2-live-gate.md.
Checks 1, 2, 5, 6 pass. Checks 3 and 4 are NOT satisfied — the defect they
were meant to confirm turned out to be the opposite of what the plan assumed.
Three defects crash-looped every driver node before check 1 could even run:
1. An empty ServerHistorian:ApiKey kills the host. ServerHistorianOptions-
Validator exists to turn exactly that class of failure into a named
OptionsValidationException, but its documented fail tier explicitly excluded
ApiKey on the reasoning that a keyless client "degrades — the gateway rejects
calls". It does not: the client validates its own options at construction, so
the process dies during Akka startup and never makes a call.
2. UseTls disagreeing with the endpoint scheme kills the host too, in both
directions (both messages confirmed in the shipped client assembly). Moving an
endpoint from https to http without clearing UseTls is an ordinary migration
slip.
3. Plaintext h2c was UNREACHABLE. HistorianGatewayClientAdapter forwarded the
TLS-only options unconditionally, and AllowUntrustedServerCertificate defaults
to false, so it always sent RequireCertificateValidation=true — which the
client rejects outright when UseTls=false. Every http:// deployment crashed,
though the scheme is documented as the supported way to select h2c, and the
only workaround was to assert a certificate posture for a connection that has
no certificate.
The fourth was the blocker, and it is Phase 2's own:
4. The drain gate deferred to a Primary that cannot deliver. Redundancy roles are
elected CLUSTER-WIDE; the alarm queue is PAIR-LOCAL. On the rig the elected
driver Primary is central-1 — it carries the driver Akka role, replicates
nobody's LocalDb and does not even run the alarm historian — so every driver
node logged "Historian drain suspended", including the two site-b nodes that
have no peer at all. Nothing drained anywhere, where before Phase 2 it drained
fine. The cost is not a duplicate; it is the buffer growing to the capacity
wall and evicting the audit trail it exists to protect.
Fixed in three layers: a separate ShouldDrainAlarmHistory policy (unknown role
drains; the two gates now deliberately disagree, and a test pins that); peer-
host matching in DriverHostActor so a node stands down only for a Primary
holding its rows; and AddAlarmHistorian short-circuiting the gate when
replication is unconfigured — testing BOTH Replication:PeerAddress and
SyncListenPort, since only the dialing half sets the former while both halves
share the queue.
Every one of these follows from the asymmetry: a false allow costs a duplicate
row, which at-least-once delivery already accepts and payload-hash ids
collapse; a false deny loses data silently.
A third vacuous test, caught by the same delete-the-guard discipline: the
role-view tests stayed green with the guard removed, because AwaitAssert polls
until an assertion passes and the assertion was "reads open" — which is the
SEEDED value, satisfied at the first poll before the actor processed anything.
They now assert the sequence of published values through a recording view; the
control then goes red for exactly the cases that matter.
Migration evidence: 11 legacy rows across two deliberately overlapping files
converged to exactly 9 identical rows on both nodes, proving D-6's payload-hash
identity on real nodes rather than in a fixture.
Open design fork, recorded in the gate doc rather than decided here: a pair
cannot currently identify its own Primary, so both halves drain. Safe in every
topology — nothing loses data — but the gate's de-duplication benefit is
unrealised until roles are scoped per pair.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Enables the alarm store-and-forward sink on all four driver nodes of the
docker-dev rig -- the replicating site-a pair and the default-OFF site-b pin
-- so the live gate has a real buffer to watch converge, and can see that
site-b's sink works as a plain node-local queue with no peer traffic.
Two rig details worth stating rather than rediscovering. The endpoint is
deliberately unresolvable: there is no HistorianGateway here, so every drain
attempt fails, which is exactly the historian-outage state the buffer exists
for. But it still has to be a syntactically valid absolute http(s) URI or the
host refuses to start, because ServerHistorianOptionsValidator is
consumer-gated -- AlarmHistorian:Enabled=true makes the endpoint required
even while ServerHistorian:Enabled=false. And MaxAttempts is raised far above
the production default of 10, which against a permanently unreachable gateway
would dead-letter the entire queue about five minutes in, turning a buffering
test into a dead-letter test.
Docs record what an operator now has to know: that a rising queue depth on a
Secondary is correct and on BOTH nodes is not (that shape is a redundancy
snapshot naming neither node -- check node identity before suspecting the
sink), that delivery is at-least-once across a failover by design with no
dedup layer, and that AlarmHistorian:DatabasePath is removed but should be
left in place through the upgrade because the migrator still reads it to find
the file to copy.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Every driver-role node gets a per-node named volume at /app/data and
LocalDb__Path=/app/data/otopcua-localdb.db so the pair-local config cache
survives container recreates. The site-a pair enables replication (SyncListenPort
9001, a-1 dials a-2, shared dev-only ApiKey, MaxBatchSize 16); the central and
site-b pairs stay default-OFF — site-b is the pin the live gate's check 8
verifies. docker compose config validates; rig not brought up (live gate owns runs).
Turns on Secrets:Replication:Enabled across all six docker-dev host nodes via
one x-secrets-env anchor: replication enabled, a 5s AnnounceInterval so
anti-entropy is observable during dev exercise, and the ONE shared dev-only
KEK every node requires (committed default per the rig's zero-operator-step
pattern, overridable via OTOPCUA_SECRETS_KEK; never reuse outside this rig).
Live-proven on the rig with Secrets 0.2.2: all six nodes start with NO
startup deadlock (the scadaproj#1 signature never appeared), the mesh forms,
and a secret seeded into site-a-1's local store alone converged to all five
peers within 60s — byte-identical ciphertext on all six nodes (same
sha256, kek_id sha256:f3f0c2056e7a), decrypt-verified on central-2 and
site-b-2. site-b-2 restart rejoined cleanly with the row intact; the fleet
also survived a full Docker-engine restart with replication enabled.
Found during verification, tracked separately: the AdminUI /admin/secrets
page renders but is non-interactive (Secrets.Ui RCL components carry no
@rendermode and the host applies render modes per-page), so Add secret does
nothing - pre-existing, unrelated to replication.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Corrects the #459 finding. 2-node keep-oldest recovery works fine (the ScadaBridge
sister project proves it); OtOpcUa was missing the supervision pieces that make it
automatic, and docs/Redundancy.md wrongly claimed in-place oldest-crash failover.
Mechanism (confirmed on a 2-container rig + by decompiling Akka KeepOldest.OldestDecision):
on an OLDEST-node crash keep-oldest downs the LONE survivor (DownReachable including
myself) — down-if-alone can't rescue a lone survivor (its branch needs >=2 survivors).
Recovery is exit-and-rejoin: run-coordinated-shutdown-when-down terminates the node and
the service supervisor restarts it. My earlier 'total outage' was a docker-dev artifact
(no restart policy); production Install-Services.ps1 already has sc.exe failure restart.
Changes (ScadaBridge parity):
- ActorSystemTerminationWatchdog (Host, registered after AddAkka): watches
ActorSystem.WhenTerminated and on an unexpected self-down calls StopApplication so the
process exits (supervisor restarts it) instead of idling with a dead actor system.
Distinguishes graceful shutdown via _stopRequested + ApplicationStopping. 3 unit tests.
- docker-dev: restart: unless-stopped on the host anchor (models production supervision) +
both redundancy peers in SeedNodes so a restarted node re-forms via either peer.
- docs/Redundancy.md: rewrote the split-brain recovery section — younger-loss = in-place
fast failover; oldest-loss = exit-and-rejoin under supervision (not in-place); the three
requirements (supervisor + watchdog + both-node seeds); flagged HardKillFailoverTests as
non-representative (Transport.Shutdown, not a real crash). Instant in-place takeover on
ANY single loss needs 3+ members.
Cluster.Tests 29/29 (SBR guards), watchdog tests 3/3, full solution builds.
Live re-verify of the watchdog image pending (host docker disk full).
The historian-gateway cutover made ServerHistorianOptions read Endpoint/ApiKey/UseTls,
but docker-dev compose still set Enabled=true with the retired Wonderware Host/Port/
SharedSecret keys → HistorianGatewayClientAdapter.Create built a Uri from an empty
Endpoint → unhandled UriFormatException crashed the Host at every bring-up, leaving
docker-dev unbootable against current code.
central-1 + central-2 now use the gateway-shape keys, DISABLED by default (the rig has no
historian gateway): Enabled=${OTOPCUA_HISTORIAN_ENABLED:-false}, Endpoint/ApiKey via env,
UseTls + AllowUntrustedServerCertificate. To enable, set OTOPCUA_HISTORIAN_ENABLED=true +
OTOPCUA_HISTORIAN_ENDPOINT + export ServerHistorian__ApiKey (never committed). Verified:
compose config valid, central-1 boots clean (no UriFormatException; OPC UA + AdminUI serving).
Found during the arch-review #10 live-gate rig session; branched off master (independent
of the archreview work).
CLAUDE.md: add scadaproj sister-project index, fix mxaccessgw path to
~/Desktop/MxAccessGateway, note the temporary CVE-2025-6965 NuGetAuditSuppress,
and correct the driver-typed tag-editor list (OpcUaClient is now mapped; only
Galaxy falls back). Also captures docker-dev data-plane GroupToRole + ServerHistorian
live-verify config and the HistorianGateway integration/backlog notes.
The seed re-inserted a Namespace with Kind='SystemPlatform' (+ a
GalaxyMxGateway driver + 3 mirror tags), but that NamespaceKind member was
removed when Galaxy became Equipment-kind (migration CleanupSystemPlatformNamespaces).
cluster-seed runs after the migrator, so a fresh down -v/up re-introduced a Kind
the current code can't EF-materialize — 500ing /deployments and failing every
publish (ConfigComposer reads db.Namespaces). Remove the obsolete inserts;
author an Equipment-kind Galaxy driver via the UI if a fixture is needed.
The three TestMachine_001 folder-path tag inserts were guarded by TagId, but
the violated constraint is UX_Tag_FolderPath — a unique filtered index on
(DriverInstanceId, FolderPath, Name) WHERE EquipmentId IS NULL. The Galaxy
driver auto-materialises these same folder-path tags at runtime under its own
generated TagIds (nw-mirror-*), so on a re-run the TagId guard found nothing,
the insert fired, and it collided on UX_Tag_FolderPath — cluster-seed exited 1.
Guard each insert on the index's natural key instead, so the row is skipped
whether it came from a prior seed or the driver's runtime mirror.
Verified by re-running cluster-seed against the populated dev volume (which
holds nw-mirror-testmachine_001-testalarm00{1,2,3}): exit 0, no UX_Tag_FolderPath
violation.
Adds a 'migrator' Dockerfile stage + Compose service that runs 'dotnet ef
database update' once on bring-up, so a fresh SQL volume gets the schema with no
operator step (quirk 1). cluster-seed + every host node depend on it via
service_completed_successfully, so the seed never races an in-progress migration
(quirk 2). Host build pinned to target: runtime (the migrator is now the last
stage). entrypoint + README updated; the manual 'dotnet ef' first-time step is
gone. Verified: down -v + up --build self-bootstraps (migrator+seed exit 0,
6 nodes up), deploy Sealed 6/6.
Rewrite docker-dev/README.md and update docker-dev/Dockerfile comment
to reflect the new topology: one Akka mesh seeded by central-1/central-2
(fused admin+driver, MAIN cluster, single UI at http://localhost:9200),
with site-a-*/site-b-* as driver-only members scoped by ClusterId.
Removes all references to the old three-mesh layout (admin-a, admin-b,
driver-a, driver-b, site-a.localhost, site-b.localhost).
Set Security__DeployApiKey on the 6 admin/fused nodes so POST /api/deployments works in the
dev fleet (reachable via traefik :9200). Smoke-tested: no/wrong key -> 401, correct key -> 202
with a real deployment id.
The docker-dev sql service had no volume, so its data lived in the container
writable layer. A recreate silently dropped the OtOpcUa database and every host
node failed its configdb health check (AdminUI 503) until an operator re-ran
'dotnet ef database update' + the cluster-seed. Add a named volume
(otopcua-mssql-data -> /var/opt/mssql) so the migrated schema + seeded clusters
survive 'docker compose up' cycles.
docker-dev un-stubbed → binds zb-shared-glauth on 10.100.0.35:3893 (dc=zb,dc=local)
via cn=serviceaccount; sign in multi-role/password (group→role seeded by
seed-clusters.sql). Per-VM C:\publish\glauth + base DNs dc=lmxopcua/dc=otopcua
obsolete. Source of truth: scadaproj/infra/glauth/.
The 6 admin/site host containers drop DevStubMode and bind the shared dev GLAuth
(scadaproj/infra/glauth/, dc=zb,dc=local) via cn=serviceaccount. seed-clusters.sql
seeds system-wide LdapGroupRoleMapping rows OtOpcUa-Admins->Administrator,
OtOpcUa-Designers->Designer, OtOpcUa-Viewers->Viewer (bare-RDN group keys, matching
the shared lib's ToGroupShortName). Verified: multi-role -> Viewer+Designer+
Administrator at :9200 via real LDAP.
The per-role overlay (appsettings.{role}.json) was appended after WebApplicationBuilder's
default sources, so it outranked environment variables — a baked role file could not be
overridden by a deployment env var. In otopcua-dev this meant appsettings.admin.json's
Security:Ldap:DevStubMode=false beat the compose's DevStub override, so every AdminUI login
attempted a real LDAP bind against a non-existent server and failed with 'Unexpected
authentication error'.
- Program.cs: re-append AddEnvironmentVariables() + AddCommandLine(args) after the role
overlay so deployment overrides keep top precedence (overlay still beats base appsettings).
- docker-dev/docker-compose.yml: the DevStub env var targeted the stale 'Authentication:Ldap'
section; the code reads 'Security:Ldap'. Corrected the prefix on every host node (+ comment).
Dev AdminUI login now signs in as Administrator via the DevStub bypass.
Host :80 collides with the sister scadabridge-traefik dev stack; bind the
OtOpcUa Traefik :80 entrypoint to host 9200 instead (admin UI now at
http://localhost:9200). Dashboard already on 8089 to avoid the same clash.
Replaces the old fallback (mxgw_otopcua_…UY_NKlBl3) with the freshly issued
mxgw_otopcua2_GI7-… on all 8 host services. Gateway endpoint stays at
http://10.100.0.48:5120 (seed-clusters.sql already points there). Operators
who set GALAXY_MXGW_API_KEY in their shell continue to override the default
unchanged.
Closes the gap where Tag rows with EquipmentId=NULL + Namespace.Kind=SystemPlatform
(Galaxy hierarchy) existed in ConfigDb but were never surfaced in the OPC UA
address space. Now they materialise as Variable nodes under a folder named for
their FolderPath, browseable through any OPC UA client.
Layers touched:
- IOpcUaAddressSpaceSink: new EnsureVariable(nodeId, parentFolderId, displayName,
dataType) signature on the sink interface, NullSink, DeferredSink, SdkSink.
- OtOpcUaNodeManager.EnsureVariable: creates a BaseDataVariableState parented
under the named folder (or root), initial Value=null +
StatusCode=BadWaitingForInitialData; resolves Tag.DataType strings to the
matching OPC UA built-in NodeId. Idempotent.
- Phase7CompositionResult: new GalaxyTags collection of GalaxyTagPlan records
carrying (TagId, DriverInstanceId, FolderPath, DisplayName, DataType,
MxAccessRef). Constructor overloads keep existing call sites compiling.
- Phase7Composer.Compose: now takes Tag + Namespace inputs, filters for
SystemPlatform-namespace tags with EquipmentId=NULL, emits GalaxyTagPlan
rows with MXAccess ref "FolderPath.Name".
- Phase7Plan: new AddedGalaxyTags / RemovedGalaxyTags / ChangedGalaxyTags
collections + GalaxyTagDelta record; IsEmpty + needsRebuild updated.
- Phase7Planner.Compute: diffs GalaxyTags by TagId via existing DiffById helper.
- DeploymentArtifact.ParseComposition: reads the Tags + Namespaces +
DriverInstances arrays the ConfigComposer already emits, applies the same
SystemPlatform filter, returns the same GalaxyTagPlan list as the composer
so artifact-side and compose-side plans agree.
- Phase7Applier: new MaterialiseGalaxyTags pass that ensures one folder per
distinct FolderPath then one Variable per tag. NodeId for the variable is
"<FolderPath>.<Name>" matching the MXAccess ref so the future Galaxy
SubscribeBulk wiring can address them directly.
- OpcUaPublishActor.RebuildAddressSpace: invokes MaterialiseGalaxyTags after
MaterialiseHierarchy. _lastApplied initialiser updated for the new ctor.
- seed-clusters.sql: pre-existing TestMachine_001.TestAlarm001..003 rows
needed no change — the composer/applier now picks them up automatically.
Verified end-to-end via docker-dev: deploy click → driver-a logs
"Phase7Applier: Galaxy tags materialised (tags=3, folders=1)" → OPC UA Client
CLI browses the three Variable nodes under TestMachine_001 folder. Reads
return BadWaitingForInitialData status (expected — Galaxy driver's
SubscribeBulk wiring to push values into the nodes is the remaining
follow-up).
Two bring-up issues found while clicking through the operator Deploy flow
on the docker-dev stack:
- ConfigPublishCoordinator computes expected-ack NodeIds from
Akka.Cluster.State.Members as "{host}:{port}" (e.g. "driver-a:4053") to
match ClusterRoleInfo's NodeId derivation. The seed had been using the
bare service name ("driver-a"), so NodeDeploymentState INSERT hit FK
violation 547 on NodeDeploymentState.NodeId → ClusterNode.NodeId. Seed
now writes the full host:port form for every ClusterNode row.
- Blazor Server uses SignalR (WebSocket upgrade after the initial GET).
Without sticky sessions, Traefik round-robins admin-a/admin-b and the
WebSocket upgrade lands on the wrong backend, returning "No Connection
with that ID: Status code '404'" so @onclick handlers never fire on the
client. Added sticky.cookie (otopcua_lb, SameSite=Lax) to all three
Traefik service loadBalancers so each session pins to one node.
Verified end-to-end: clicked "Deploy current configuration" on
/deployments → Deployment row sealed in ~70ms → driver-a + driver-b
spawn GalaxyMxGateway driver (stub=False) → GalaxyDriver connects to
http://10.100.0.48:5120 with the seeded ApiKeySecretRef=env:GALAXY_MXGW_API_KEY.
User confirmed the mxaccessgw client (Galaxy driver) doesn't need Windows
— only the gateway worker has that constraint. This wires the Galaxy
driver into the docker-dev fleet:
- docker-compose.yml: GALAXY_MXGW_API_KEY env var on every host service
(admin nodes harmlessly ignore it; driver-role nodes pick it up when
the seeded DriverInstance resolves ApiKeySecretRef=env:GALAXY_MXGW_API_KEY).
Default value matches the key the operator provided; override via shell
env (GALAXY_MXGW_API_KEY=... docker compose up -d) to rotate without
editing compose.
- seed-clusters.sql: now creates a SystemPlatform Namespace
(MAIN-galaxy, urn:zb:docker-dev:galaxy) plus a GalaxyMxGateway
DriverInstance (MAIN-galaxy-mxgw) in the MAIN cluster pointing at
http://10.100.0.48:5120 with UseTls=false. Idempotent via IF NOT EXISTS.
- DriverInstanceActor.ShouldStub: clarified the doc comment — only the
legacy "Galaxy" type name and "Historian.Wonderware" are Windows-only;
the v2 "GalaxyMxGateway" driver is .NET 10 cross-platform (gRPC to an
external gateway) and is NOT stubbed.
- README: documents the final operator step — sign in, click "Deploy
current configuration" on /deployments to materialise the seeded
Galaxy driver into a running gRPC connection. Raw DriverInstance rows
don't spawn drivers on their own; the v2 lifecycle requires a sealed
Deployment first.
Two fixes surfaced while bringing up the docker-dev stack end-to-end:
- HealthEndpoints.MapOtOpcUaHealth now calls .AllowAnonymous() on /health/ready,
/health/active, /healthz. Without it the AddOtOpcUaAuth fallback policy 401s
every probe and Traefik marks every backend unhealthy → all three cluster
routes return 503.
- cluster-seed entrypoint no longer attempts to apply Migrate-To-V2.sql via
sqlcmd. The EF-generated idempotent script puts CREATE PROCEDURE inside
IF NOT EXISTS BEGIN ... END blocks (procs must be first in their batch),
so sqlcmd fails with "Must declare the scalar variable @FromGenerationId".
EF's own runner handles this; sqlcmd doesn't. The seed now just waits for
the schema and applies row inserts. Migrations remain the operator's job:
dotnet ef database update --project src/Core/.../Configuration \
--startup-project src/Server/.../Host
Also: LDAP service removed (bitnami/openldap:2.6 image retired, legacy tag
crashes mid-setup with exit 68); every host now runs with
Authentication__Ldap__DevStubMode=true. Bumped LDAP+Traefik dashboard host
ports to avoid collisions with the sister scadalink dev stack (3893→3894,
8080→8089).
Confirmed working end-to-end: all three Traefik routes return HTTP 200,
cluster-seed populates ServerCluster (MAIN/SITE-A/SITE-B) + ClusterNode
(driver-a/b, site-a-1/2, site-b-1/2) rows on first boot.
Adds a one-shot cluster-seed service to docker-dev/docker-compose.yml
that pre-populates the three Akka clusters' scope rows in the shared
OtOpcUa ConfigDb so operators don't have to click through /clusters +
/hosts on every fresh bring-up.
Seed contents:
ServerCluster MAIN (Warm/2), SITE-A (Warm/2), SITE-B (Warm/2)
ClusterNode driver-a + driver-b → MAIN
site-a-1 + site-a-2 → SITE-A
site-b-1 + site-b-2 → SITE-B
NodeCount + RedundancyMode honour the CK_ServerCluster check constraint.
ApplicationUri follows the urn:OtOpcUa:<NodeId> convention; uniqueness
across the fleet satisfies UX_ClusterNode_ApplicationUri.
Mechanism:
- docker-dev/seed/seed-clusters.sql — idempotent INSERTs (IF NOT EXISTS
guards on every row).
- docker-dev/seed/entrypoint.sh — bash wrapper that waits for SQL to
accept connections, then polls until dbo.ServerCluster exists (the
host containers' EF auto-migration creates it on first boot), then
applies the SQL script.
- cluster-seed service uses mcr.microsoft.com/mssql-tools as the base
image (bash + sqlcmd available), restart: "no" so it runs once.
Re-running `docker compose up` is safe: the seed exits cleanly on the
second run because every INSERT is guarded.
Manual re-seed: `docker compose run --rm cluster-seed`.
The previous commit (961e094) gave each site cluster its own database
(OtOpcUa_SiteA / OtOpcUa_SiteB). That fights the architecture — ConfigDb
is multi-tenant by design: one schema with a ServerCluster table whose
rows scope the rest of the configuration via ClusterId. Per-cluster
databases would split the schema and force every singleton/coordinator
to point at a different connection string.
Correct model: one ConfigDb, three ServerCluster rows (MAIN / SITE-A /
SITE-B), each Akka cluster's ClusterNode rows pointing back at the
matching ClusterId. Akka mesh isolation is still enforced by the
disjoint seed-node lists (unchanged from the previous commit).
Compose: all eight host nodes now point at Server=sql,1433;Database=OtOpcUa
and the README documents the post-boot ServerCluster + ClusterNode rows
operators need to create via /clusters and /hosts before the runtime can
resolve its scope.
Extends the docker-dev compose with two additional, fully-isolated Akka
clusters representing distinct sites. Each site is a 2-node fused
admin+driver cluster (OTOPCUA_ROLES=admin,driver on both nodes), backed
by its own ConfigDb database so configuration state stays separate from
the main cluster and from the other site.
Cluster isolation: the three meshes share the same Akka system name
"otopcua" and remoting port 4053 (inside each container's own network
namespace), but their seed-node lists are disjoint — main seeds at
admin-a, site-a seeds at site-a-1, site-b seeds at site-b-1 — so gossip
doesn't cross between them.
Layout:
Main cluster ConfigDb=OtOpcUa admin-a, admin-b, driver-a, driver-b
Site A ConfigDb=OtOpcUa_SiteA site-a-1, site-a-2 (fused admin+driver)
Site B ConfigDb=OtOpcUa_SiteB site-b-1, site-b-2 (fused admin+driver)
OPC UA endpoints exposed on host ports 4840-4845. Admin UIs reachable
through Traefik via Host-header routing:
http://localhost → main cluster (PathPrefix default)
http://site-a.localhost → site A
http://site-b.localhost → site B
`*.localhost` auto-resolves on macOS; Linux users add the two hosts to
/etc/hosts (or rely on the resolver's RFC 6761 behaviour).
- scripts/install/traefik.yml + traefik-dynamic.yml: Traefik static + dynamic
config. One :80 entry point, one router on HostRegexp(otopcua.*), one
service load-balancing admin-a:9000 + admin-b:9000 with /health/active health
check (interval 5s, timeout 2s, expected 200). Followers return 503 from
/health/active so Traefik drops them within the next interval after a
leadership change.
- scripts/install/Install-Traefik.ps1: downloads Traefik for Windows, drops the
yml configs, registers the OtOpcUaTraefik Windows service via sc.exe with
restart-on-failure. Companion to Install-Services.ps1.
- docker-dev/{Dockerfile,docker-compose.yml,traefik-dynamic.yml,README.md}:
Mac-friendly four-node fleet (admin-a + admin-b + driver-a + driver-b) plus
SQL Server 2022 + OpenLDAP + Traefik. Single OtOpcUa.Host image built once;
Compose drives OTOPCUA_ROLES + Cluster:* per container to differentiate the
four hosts. README walks through bring-up + failover smoke + the dev LDAP
users.
Note: untested on macOS (no local Docker — see docs/v2/dev-environment.md).
Joseph Doherty