C1 (CRITICAL). AuthoredTable routed any wildcard-authored tag into `Wildcards`
and never into the concrete topic index. But the two paths that reconstruct
state from a SUBSCRIBED FILTER STRING — OnReconnectedAsync's re-subscribe set
and DegradeTopic — both looked up that concrete-only index, and
`_subscribedTopics` keys on the wildcard PATTERN ("f/+/temp"). So:
- reconnect resolved zero filters for a wildcard tag, hit a silent early
return, and issued NO subscribe. The cache keeps its last Good value, so the
tag never turns Bad — it just stops updating forever behind a connection
reporting healthy. Exactly the silent-death mode MqttConnection's own remarks
warn about, left unguarded for wildcards.
- DegradeTopic missed too, so a SUBACK rejection of a wildcard filter degraded
nothing and left the tag at BadWaitingForInitialData.
Fix: AuthoredTable now carries `ByFilter` (EVERY def, keyed by its authored
topic filter, wildcards included) alongside `ByExactTopic` (concrete only) and
`Wildcards`. Delivery matches an incoming published topic — which never carries
a wildcard — so it keeps using ByExactTopic + the comparer scan. Every path
keyed on a filter string uses ByFilter. The distinction is documented on the
record. The silent early return is now a Warning naming the orphaned topics.
Also in this pass:
- I2: bounded decode on the dispatcher thread. A body over `MaxPayloadBytes`
(default 1 MiB, ctor-settable) is refused BEFORE any GetString/JsonDocument
parse and degrades its own tags. Unbounded decode on the shared dispatcher is
paid by every subscription, not just the offending topic. NOTE: promoting this
to an operator-facing MqttDriverOptions key (+ WithMaximumPacketSize) needs a
.Contracts edit this task is scoped out of; flagged in the XML docs.
- I3: integral-valued reals now coerce to integer tags. JS/Python edge gateways
serialize an integer as 5.0, and TryGetInt32/int.TryParse are syntactic — an
Int32 tag fed by such a gateway silently never received data. Parsed via
decimal so integrality and range are exact across Int64/UInt64. A genuinely
fractional 5.5 is still refused, never rounded.
- I4: the JSON document is parsed ONCE per message and shared across the whole
fan-out (the documented "one document, one JSONPath per signal" shape was
re-parsing per tag). Zero-alloc via a ref struct when no Json tag matches.
- I5: pinned the documented JSON-string-holding-a-number coercion end-to-end.
- Minor: Register now prunes `_subscribedTopics` / `_handleByRawPath` entries for
tags a redeploy dropped, so a deleted topic stops being re-subscribed forever.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Three review findings, all "unbounded work/memory from untrusted broker input"
rather than correctness bugs - but this points at a live plant broker.
1. InferPayload decoded the WHOLE payload on MQTTnet's dispatcher thread before
trimming to 64 chars. The node cap never engages here: a multi-MB blob on an
already-known topic creates no node, it just updates one, so the cost repeats
per message forever. Now at most PayloadInspectMaxBytes (512) are examined.
A blind slice can cut a multi-byte sequence, which the strict decoder would
report as "binary" - TrimToUtf8Boundary backs the window off to the last whole
sequence so good UTF-8 is never mislabelled. A clipped payload is String by
construction rather than inferred from a partial view.
2. The node cap bounded COUNT, not bytes. MQTT topics run to ~65KB, so 50k nodes
near that ceiling is a multi-GB tree. Topics now bounded at 1024 chars and
segments at 256, rejected whole (a truncated topic is a DIFFERENT topic the
picker would commit as a tag address) and surfaced via an __oversized__ marker
kept distinct from __truncated__ - the operator's remedy differs.
3. Control characters in a snippet would break the picker's single-line row;
Trim() only strips the ends. Now scrubbed to spaces.
Each guard was falsified independently by neutering it and confirming exactly one
test reddens. That found a real gap: the oversized-topic test was passing via the
SEGMENT bound, leaving the whole-topic bound untested - the test now uses many
short segments so only the topic bound can reject it.
Also records the .Driver project reference as a known, deliberate exception to the
.Browser -> .Contracts pattern, with its cost (AdminUI gains Core + Polly + Serilog)
and the clean fix (a leaf transport project; NOT a move into .Contracts, which is
deliberately transport-free).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
MqttSubscriptionManager is the P1 plain-MQTT ingest path: it holds the authored
RawPath → MqttTagDefinition table behind the shared EquipmentTagRefResolver,
establishes one SUBSCRIBE per distinct authored topic, and turns each inbound
message into a LastValueCache update plus an OnDataChange notification.
Load-bearing choices, each pinned by a falsified test:
- The published reference IS the RawPath (def.Name), never the topic and never
the TagConfig blob — DriverHostActor's dual-namespace fan-out is RawPath-keyed,
so any other key passes every unit test and delivers nothing in production.
- One message fans out to EVERY tag on its topic; several tags routinely share a
topic with different jsonPaths, and stopping at the first match silently starves
the rest.
- An unauthored topic raises nothing: MQTT ingest never auto-provisions.
- Wildcard-authored topics (accepted by the parser, warned at deploy) match via
MQTTnet's own MqttTopicFilterComparer, so '+'/'#' behave as a broker would.
Concrete topics stay a single lookup; the wildcard scan runs only when one exists.
- Retained seeds are honoured natively on MQTT 5.0 (retain handling) AND filtered
client-side on the retain flag, because 3.1.1 brokers always replay.
- Nothing throws on MQTTnet's dispatcher thread: a malformed payload degrades that
tag (BadDecodingError / BadTypeMismatch), and a throwing OnDataChange subscriber
is contained without starving the tags behind it.
- The manager issues the FIRST subscribe itself — Reconnected deliberately does not
fire after the initial ConnectAsync.
- Reconnect re-subscribe: total failure THROWS (MqttConnection then tears the
session down and retries under backoff, rather than serving a connected-but-deaf
driver); a partial SUBACK rejection degrades only the denied refs, because
retrying forever over one ACL-denied topic would take every tag dark.
MqttConnection gains a MessageReceived observer (fired on the dispatcher, throwing
observers contained) and a bounded SubscribeAsync under its own linked-CTS deadline
— deliberately not MqttClientOptions.Timeout, which already governs every MQTTnet
op. A refused filter is an outcome, not an exception; only the SUBSCRIBE itself
failing throws. Classify() is parameterised by operation name (messages unchanged
for connect).
JSONPath is a documented subset ($, $.a.b, $['a'], $.a[0]); filters/slices/
recursive descent report a miss rather than pulling in a JSONPath package for an
expression form that selects a set where a tag needs one scalar.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
A will is published by the BROKER, so it is invisible to PublishCountForTest.
MqttDriverOptions carries none today, but Sparkplug NDEATH is a will message -
once P2 adds it, an ungraceful browse disconnect would fire NDEATH under the
plant's own edge-node identity. ToBrowseOptions is where it must be cleared.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
MQTT has no discovery protocol, so the AdminUI address picker learns the topic
space the only way there is: subscribe to the wildcard and accumulate what
arrives. MqttBrowseSession serves that observation as a segment tree (split on
'/'); MqttDriverBrowser opens it with a distinct "{clientId}-browse-{guid8}"
identity under a 5-30 s clamped budget.
Browsing publishes NOTHING - the load-bearing safety property, since the picker
runs against a live plant broker. Every outbound message must route through the
single PublishAsync seam that counts into PublishCountForTest; P2's operator-
triggered Sparkplug rebirth is the one intended exception. Sparkplug mode fails
fast rather than serving a raw spBv1.0 topic tree that looks like a metric tree.
Deviation from the plan's ref list: the project also references .Driver so the
browse CONNECT reuses MqttConnection.BuildClientOptions (TLS posture, CA-pin
chain validator, credentials). A second copy would be a security divergence.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Closes the connect-vs-connect defect: MQTTnet throws (and raises no DisconnectedAsync)
on a connect-while-connected, so a caller's ConnectAsync and the reconnect supervisor
corrupted each other in both directions. Both paths now check first; when the supervisor
finds the session already restored it stands down WITHOUT firing Reconnected.
Reconnected becomes Func<CancellationToken, Task>, fed from the lifetime token and capped
at ConnectTimeoutSeconds, so a hung re-subscribe can no longer park the supervisor.
Connected is published only after the re-subscribe succeeds; a supervisor that dies now
reports Faulted instead of Reconnecting forever.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
LastValueCache bridges MQTT's subscribe-first push model to the OPC UA
server's polled IReadable.ReadAsync: the subscription path calls Update()
per RawPath, the read path calls Read(). Read never throws; an unseen
RawPath returns BadWaitingForInitialData (0x80320000) rather than an
exception or null, so a batch covering many references degrades per-ref
instead of failing wholesale.
Deviates from the plan's GoodNoData snippet: GoodNoData is reserved
repo-wide for "the historian window held no samples" (NullHistorianDataSource,
OtOpcUaNodeManager HistoryRead paths); BadWaitingForInitialData is the
established convention for "no live value observed yet" (CalculationDriver,
VirtualTagEngine, FOCAS, AddressSpaceApplier). Keyed by RawPath per the v3
driver-reference identity, not a topic/JSON-path-derived key.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Review follow-ups on MqttConnection (Task 3):
- Every certificate test asserted rejection, so the accept branch was
unreachable-by-regression. Adds a leaf genuinely issued by the pinned CA and
asserts acceptance.
- ValidateAgainstPinnedCa never seeded ChainPolicy.ExtraStore from the incoming
chain, so a leaf behind an intermediate delivered during the handshake failed
despite a legitimate path to the pinned root. Seeds from both the incoming
chain's elements and its ExtraStore; CustomRootTrust still means only the
pinned roots may terminate the chain.
- A DisposeAsync racing an in-flight connect escaped as an unclassified
exception; it now folds into ObjectDisposedException.
- Promotes the single-caller concurrency invariant into the type remarks, with
the accurate blast radius (a leaked live connection, not a benign throw).
Serialising the lifecycle remains Task 4's job.
- X509Chain.Build can throw; an exception escaping a TLS validation callback is
an opaque handshake crash, so it is caught and refused.
- Adds a connect-retry test (Task 4's reconnect loop reuses the instance) and a
disposed-then-connect test.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Task 2 review follow-up. The plan specified MqttEquipmentTagParser.TryParse(reference)
with def.Name = the TagConfig blob, and told us to mirror a type named
ModbusEquipmentTagParser. Both are plan defects:
- EquipmentTagRefResolver documents that a v3 driver reference "is now always a
RawPath" and that the blob-parse fallback is retired. Keying Name by the blob
would make OnDataChange publish under a reference that never matches the
RawPath-keyed fan-out in DriverHostActor - silently dead in production with
every unit test still green.
- ModbusEquipmentTagParser does not exist. The six sibling drivers all use
<Driver>TagDefinitionFactory.FromTagConfig(tagConfig, rawPath, out def).
Changes:
- Rename MqttEquipmentTagParser -> MqttTagDefinitionFactory; TryParse(reference,
out def) -> FromTagConfig(tagConfig, rawPath, out def) setting Name: rawPath,
matching ModbusTagDefinitionFactory's structure, param docs and guard order.
- Pin the identity contract with a dedicated test so a regression to blob-keying
goes red.
- Read qos with the same strictness as the enums: a present-but-invalid qos
("high" / 1.5 / 5 / null) now rejects the tag and is warned by Inspect,
instead of being silently absorbed into the driver-level default and handing
the operator a weaker delivery guarantee than the one they authored.
- No ToTagConfig inverse: the siblings carry one solely for their Driver.<X>.Cli
project, the MQTT plan defines none, and the AdminUI editor template
references no driver factory. Recorded as an explicit YAGNI call in the type
doc rather than added speculatively.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Code review of f22db5d8 (approved-with-findings): the existing round-trip
tests only exercised explicit JSON payloads, so nothing failed if UseTls /
AllowUntrustedServerCertificate or the §5.1 numeric defaults were flipped.
Adds a defaults-pinning test plus a partial-JSON test that omits the TLS
knobs entirely (the production case — an operator config that just doesn't
mention TLS must not silently land insecure).
Also overrides the record's PrintMembers so ToString() no longer prints
Password in plaintext (verified RED before the fix: the new test failed
with the raw password in the rendered string). OpcUaClientDriverOptions has
the same unredacted-ToString() shape but is out of scope for this task per
the coordinator's note; flagging as a possible follow-up.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Per-plan copy-paste command (subagent-driven-development in a git worktree),
plus the slash-command and executing-plans/resume alternatives, and a note that
the four independent plans can run in concurrent worktrees.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
ClusterPrimaryHealthCheck was written three days' worth of debugging ago because
the shared ActiveNodeHealthCheck selected by RoleLeader and reported Healthy for
any node lacking the role. Health 0.3.0 fixes the shared check — oldest Up
member, role-preference scoping, Unhealthy when the node owns no active work —
so the private copy is now duplication rather than a workaround, and it is
deleted.
RolePreference [admin, driver] reproduces exactly what it did: a fused central
node answers for admin (where the singletons and the AdminUI are pinned), a
driver-only site node answers for driver.
SelectOldestUpMemberOfRole now delegates to the shared ClusterActiveNode instead
of re-implementing the age ordering. This is the point of the change rather than
a tidy-up: the redundancy snapshot drives the OPC UA ServiceLevel 250/240 split
while the health tier drives Traefik's admin routing, so two copies of "oldest Up
member of a role" would be two chances to advertise one node as authoritative
while gating the data plane on another. ControlPlane takes a ZB.MOM.WW.Health.Akka
reference for it; the layering trade-off is recorded at the PackageReference.
Behaviour note: a node carrying neither admin nor driver now answers 503 on the
active tier instead of 200. That case is reachable — RoleParser admits dev-only
and cluster-role-only nodes — and 503 is correct, since such a node owns no
active work and must not be in an active-tier pool. No docker-dev node is
affected: every rig node is admin+driver or driver-only.
Tests replaced in kind. The rule itself is now pinned in the library against a
real two-node cluster; what stays OtOpcUa's own decision is the wiring, so
ActiveTierRegistrationTests pins which check is on the active tag, that it is
registered on driver-only nodes too, and that it is not on the ready tier.
Verified: Host.Tests 25/25, ControlPlane redundancy 15/15.
Add a living progress tracker (docs/plans/2026-07-24-driver-expansion-tracking.md)
for the driver-expansion program's Waves 0-2, linked from the program design doc's
document map. One row per deliverable pointing at design + implementation plan + status.
Add executable implementation plans (plan .md + co-located .tasks.json) for the four
Wave 1/2 drivers, each authored by writing-plans methodology off its design doc and
mirroring the relevant existing driver:
- Modbus RTU 11 tasks (extends Driver.Modbus; RTU-over-TCP, direct-serial descoped)
- SQL poll 22 tasks (ISqlDialect seam, SQL Server only in v1; SQLite + central-SQL fixtures)
- MTConnect Agent 23 tasks (browse free via Wave-0 seam; TrakHound-vs-hand-rolled = Task 0)
- MQTT/Sparkplug B 27 tasks (P1 plain MQTT 0-14 then P2 Sparkplug 15-26; MQTTnet-5 pinning spike = Task 0)
Wave 0 (universal browser) remains merged (056887d6) with its live gate open (#468).
All four new plans are design-only -> plan-ready; nothing built yet.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Closes the remaining half of #494, and corrects the half fixed in 8dd9da7d.
The shared ActiveNodeHealthCheck(role: "admin") answered the wrong question twice
over. It returns Healthy for any node LACKING the role, so all four driver-only
site nodes called themselves active and no consumer could find the Primary of a
site Cluster. And it selects by RoleLeader - the lowest-ADDRESSED member - which
is not where Akka places singletons.
Replaced with ClusterPrimaryHealthCheck ("cluster-primary"). One rule: this node
is active iff it is the OLDEST Up member carrying its own active role, where the
active role is admin when the node has it and driver otherwise. That serves both
consumers correctly - a fused admin node answers for admin, so Traefik pins the
AdminUI to the node hosting the singletons; a driver-only site node answers for
driver, which is exactly SelectDriverPrimary, the same election behind
IsDriverPrimary and the OPC UA ServiceLevel 250/240 split. Per-mesh scoping is
free after Phase 6: ClusterState.Members already contains only this node's own
Cluster.
SelectDriverPrimary is generalised to SelectOldestUpMemberOfRole so the
age-ordering rule has one implementation. Two copies of "oldest Up member of a
role" would be two chances to silently disagree about who is in charge, and the
tier and the redundancy snapshot must never disagree.
THE ROLE-LEADER HALF MATTERED IN PRACTICE, not just in theory. On the rebuilt rig
the two orderings diverge right now:
akka leader (lowest address) = central-1
oldest admin member = central-2 <- hosts the singletons
8dd9da7d made the tier a real 503 but still selected by RoleLeader, so it would
have pinned Traefik to central-1 - the node NOT hosting the work. With this change
Traefik correctly routes to central-2.
Live-verified, exactly one 200 per mesh:
MAIN central-2 200 central-1 503 traefik: central-2 UP, central-1 DOWN
SITE-A site-a-1 200 site-a-2 503
SITE-B site-b-1 200 site-b-2 503
Tests: role-selection matrix and the startup-safe Degraded path in Host.Tests
(26 pass); parity between the tier's selector and the redundancy snapshot's, plus
role scoping, added to RedundancyPrimaryElectionTests, which forms a real two-node
cluster deliberately built so the oldest member is not the lowest-addressed one
(5 pass).
0.2.1 makes the shared ActiveNodeHealthCheck return Unhealthy (503), not
Degraded (200), for a node that carries the role but is not the role leader.
That is what the shared health spec always specified, what docs/ServiceHosting.md,
docs/v2/Architecture-v2.md and the 2026-05-26 alignment design all describe
("200 only on the Admin-role leader; 503 elsewhere"), and what
docker-dev/traefik-dynamic.yml + scripts/install/traefik-dynamic.yml assume when
they use /health/active as the load-balancer probe.
The implementation returned Degraded, which MapZbHealth maps to 200, so BOTH
central nodes always passed the probe and Traefik load-balanced the admin UI
across the pair instead of pinning it to the leader. The docs were right; the
code was wrong. Closes the Traefik half of #494.
No OtOpcUa code change — package pin only.
Live-verified on docker-dev, with the MAIN pair genuinely formed (memberCount 2,
both members naming one leader):
central-1 (leader) /health/active -> 200 traefik UP
central-2 (standby) /health/active -> 503 traefik DOWN
Failover drill (README step 2), which had never actually exercised anything:
stopping central-1 flipped Traefik to central-2 within 20 s and
http://localhost:9200/ answered 200 continuously across the swap — no outage.
Restarting central-1 reclaimed the role-leader and Traefik flipped back.
README corrected on two counts: the drill now states that exactly ONE backend is
UP by design, and step 3 no longer claims central-2 keeps leadership — RoleLeader
is the lowest-address member, so central-1 deterministically reclaims it. Added
the cold-boot caveat that starting both nodes at once can leave each self-forming
a 1-member cluster, in which case both are their own leader and both answer 200.
STILL OPEN in #494: driver-only nodes. The check is scoped to the admin role and
returns Healthy for any node that lacks it, so all four site nodes answer 200 and
the real per-Cluster redundancy Primary (oldest Up driver member, IsDriverPrimary)
is still not what this tier reports.
Picks up the additive per-entry `data` object in the canonical health JSON and
the cluster-view data the shared AkkaClusterHealthCheck now publishes (leader /
selfAddress / selfRoles / memberCount / unreachableCount).
No OtOpcUa code change is needed: the "akka" ready/active check is already the
shared ZB.MOM.WW.Health.Akka.AkkaClusterHealthCheck (Health/HealthEndpoints.cs),
so /health/ready starts carrying entries["akka"].data.leader on the bump alone.
Payloads from every other check are byte-identical — data is emitted only when a
check publishes some.
Prereq for the family overview dashboard (scadaproj
docs/plans/2026-07-22-overview-dashboard-impl-plan.md, Phase 2).
Verified: Host 21/21, AdminUI 692/692, Host builds 0 warnings.
Live rig check (admin node serves data.leader) deferred to the dashboard's
acceptance run.
The simultaneous-cold-start split-brain, carried as a Phase 7 "candidate follow-up,
not shipped", is now closed by the opt-in Cluster:BootstrapGuard (279d1d0f). Documents
the guard in CLAUDE.md's bootstrap section and marks the Phase 7 finding closed.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Closes the residual Phase-7 gap: when BOTH nodes of a 2-node pair cold-start at the
same instant, each self-first seed runs FirstSeedNodeProcess, times out waiting for
the other, and forms its own 1-node cluster — two Primaries in one pair (the Phase 6
live gate reproduced this reliably on docker). The prior mitigation was operational
(staggered start / compose depends_on), which does not exist on production hardware.
The guard (dark switch Cluster:BootstrapGuard:Enabled, default OFF) prevents the split
without giving up cold-start-alone:
- The lower-address node is the preferred founder: self-first, forms immediately.
- The higher node probes its partner's Akka port (TCP connect) up to PartnerProbeSeconds:
reachable => peer-first (join the founder, never race it); unreachable => self-first
(partner is dead, form alone). The order is decided BEFORE the single JoinSeedNodes,
from an explicit reachability signal — never re-formed mid-handshake (the retired
SelfFormAfter failure mode). When on, Akka gets no config seeds (BuildClusterOptions)
and ClusterBootstrapCoordinator drives the join.
Review-driven hardening: case-insensitive tie-break (a hostname-casing mismatch would
reopen the split); fail-fast validation of the timing knobs; the residual "founder dies
in the probe->join window" hang is documented and made operator-visible (warning + a
restart recovers). Real-ActorSystem coordinator tests cover the load-bearing higher-node
cold-start-alone case; 147/147 Cluster tests pass.
Live-gated on docker-dev (site-a = enablement demo, serialization removed, guard on;
site-b keeps depends_on-serialization + guard off as the A/B control): simultaneous
start -> site-a-1 founds, site-a-2 probes-reachable-joins -> 250/240, NO split;
higher-node-alone -> probes dead founder, self-forms -> 250; founder rejoin -> 240.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
All 6 failover drills pass on the docker-dev three-mesh rig:
- D1 MAIN graceful manual-failover via the AdminUI "Trigger failover" button —
Primary leaves, peer→250, restarts and rejoins with no split.
- D2 SITE-A auto-down failover, both directions; D3 SITE-B crash-the-oldest.
- D4 auto-down 1-vs-1 survival — every survivor stayed Up (closes the gate
deferred since Phase 0a).
- D5 self-first cold-start-alone — a node boots with its partner down and forms
its own mesh (gate b).
- D6 recovery — restarted nodes rejoin as Secondary (240).
Ships a one-page co-located operator runbook (OtOpcUa + ScadaBridge on shared
site VMs). Manual-failover button confirmed MAIN-only by construction; site
pairs (driver-only, no UI) fail over via auto-down. Carried Phase-6 finding —
simultaneous cold-start of both pair VMs can split — mitigated operationally in
the runbook (staggered start); a product-level join guard is a candidate
follow-up, not shipped.
The per-cluster mesh program (Phases 0a–7) is COMPLETE.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The cluster-redundancy singleton's role scope is computed at AddAkka-configurator
time and must come from AkkaClusterOptions, never by resolving IClusterRoleInfo
there (it depends on the ActorSystem being built → stack overflow at boot). Caught
by the Phase 6 live gate; fixed in 3a4ed8dd.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Records the full live-gate run against master on the docker-dev three-mesh rig.
All 8 legs pass: three isolated 2-node meshes, one Primary per pair (250/240),
deploy sealed acks=6 over per-cluster ClusterClient, telemetry gRPC :4056 up +
reconnect, reconciler own-cluster-scoped, secrets pair-local, split validator
refuses Dps, and per-pair failover with auto-down 1-vs-1 survival (also closes the
deferred Phase 0a gate). Three defects were caught and fixed forward (3a4ed8dd,
792f28ec) — see the live-gate doc's Findings section. Task 9 marked completed.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Two docker-dev-only defects the Phase 6 live gate surfaced (both invisible to
`docker compose config`, which validates the file but never boots the app):
1. Driver-only site nodes crashed at boot with OptionsValidationException — LDAP
transport None + AllowInsecure false. Only central carried the Security__Ldap__*
block, but the site nodes serve OPC UA endpoints and validate LDAP options too, and
they override `environment` wholesale (YAML `<<:` doesn't deep-merge it). Extracted
the block to a shared &ldap-env anchor merged into every host node.
2. Site pairs split-brained at simultaneous startup (both nodes "JOINING itself",
250/250 instead of 250/240). Both are self-first seeds, so both run Akka's
FirstSeedNodeProcess; the partner didn't depend on its pair founder, so they raced
and each formed a 1-node cluster. Added an &akka-founder-healthcheck (bash /dev/tcp
probe of Akka 4053 — no nc/curl in the image) to site-a-1/site-b-1 and switched the
partners to depends_on service_healthy, so the founder forms first and the partner
JOINS it. (Carry the underlying simultaneous-cold-start property to Phase 7 — it is
inherent to symmetric self-first seeding, present for central since #459.)
Live gate record: docs/plans/2026-07-24-mesh-phase6-live-gate.md (legs 1-2 PASS —
three isolated meshes, one Primary per pair at 250/240).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 6 wired the re-homed redundancy singleton in Program.cs as
`WithOtOpcUaClusterRedundancySingleton(sp.GetRequiredService<IClusterRoleInfo>())`
inside the AddAkka configurator lambda. That lambda runs WHILE the ActorSystem is
being built, but ClusterRoleInfo depends on the ActorSystem (it is a live
Cluster.State view) — so resolving it there recurses into ActorSystem construction
and every node dies at boot with StackOverflowException. It compiles cleanly (a
runtime DI cycle) and RedundancyStateSingletonRehomeTests passed a hand-built
FakeClusterRoleInfo straight into the extension, mocking around the composition-root
line that overflows. The docker-dev live gate caught it on first boot.
Derive the singleton's cluster-role scope from AkkaClusterOptions (pure config, no
ActorSystem) instead: BuildClusterRedundancySingletonOptions / the extension now take
AkkaClusterOptions and compute Role = Roles.FirstOrDefault(IsClusterRole) ?? driver —
identical to ClusterRoleInfo.ClusterRole, which derives from the same config. Program.cs
passes IOptions<AkkaClusterOptions>.Value, which has no ActorSystem dependency.
Live-verified: rebuilt image boots with zero stack-overflow lines, cluster singletons
form per mesh. 5/5 rehome tests pass; Host builds clean.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Redundancy.md: KNOWN-LIMITATION (per-Akka-cluster election) marked RESOLVED;
new 'Per-cluster meshes (Phase 6)' section + pair-local secrets note; manual-
failover mesh-scope caveat removed; auto-down 1-vs-1 note updated (every mesh is
now two nodes, drill deferred to Phase 7). IManualFailoverService + ClusterRedundancy
razor caveats updated to pair-local reality. Configuration.md documents the
cluster-{ClusterId} role, per-pair self-first seeds, SplitTopologyTransportValidator,
and the intentional Dps default. Program + design status tables + CLAUDE.md marked
Phase 6 DONE.
(Task 8 subagent completed the edits but died on an API error before committing;
reviewed and committed by the controller.)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Rewrite docker-dev/docker-compose.yml from one shared six-node gossip ring
into three self-contained pairs (MAIN: central-1/2, SITE-A: site-a-1/2,
SITE-B: site-b-1/2), all still on the ActorSystem name `otopcua` and the one
shared docker network — separation is purely by per-pair self-first
Cluster:SeedNodes, faithful to how production splits over a WAN rather than
a firewall.
Per node:
- Cluster__Roles / OTOPCUA_ROLES gain a `cluster-<ClusterId>` role
(cluster-MAIN / cluster-SITE-A / cluster-SITE-B), the marker
SplitTopologyTransportValidator and ClusterRoleInfo key off.
- Cluster__SeedNodes now lists only the node's own pair (self first, partner
second); site nodes no longer seed off central-1.
- MeshTransport__Mode default flips Dps -> ClusterClient (still overridable
via OTOPCUA_MESH_MODE) on all six nodes — SplitTopologyTransportValidator
now refuses to boot a cluster-role node left on Dps.
- Telemetry__Mode=Grpc added on all six driver nodes; TelemetryDial__Mode=Grpc
added on central-1/central-2 only (the admin pair).
- ConfigSource/ConfigServe modes and MeshTransport contact points are
unchanged (already split-correct since Phases 3/4/2).
seed-clusters.sql needed no changes — the ClusterNode rows already carry the
correct ClusterId/Host/AkkaPort/GrpcPort per node. Validated with
`docker compose -f docker-dev/docker-compose.yml config`.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Code review found SplitTopologyTransportValidator read the wrong role view,
defeating its fail-loud purpose two ways:
(a) Roles-source divergence. AkkaClusterOptions.Roles binds Cluster:Roles
but falls back to OTOPCUA_ROLES (RoleParser.Parse) when it is empty. A
node configured with only OTOPCUA_ROLES=driver,cluster-SITE-A genuinely
carries the cluster role in Akka, yet the validator saw no cluster role
and passed silently on MeshTransport:Mode=Dps (the documented incident).
(b) Case. The Cluster:Roles bind path is verbatim while RoleParser.Parse
lowercases the OTOPCUA_ROLES path; 'Cluster-SITE-A' / 'Driver' / 'Admin'
slipped past the ordinal checks.
Resolve effective roles the same way the node does — Cluster:Roles, else
RoleParser.Parse(env OTOPCUA_ROLES) exactly as Program.cs — then normalize
both (trim + ToLowerInvariant) before IsClusterRole/driver/admin. The message
still names the ClusterId in the operator's original spelling. Exemption and
all other behaviour unchanged. Contained to the validator.
Tests: OTOPCUA_ROLES-only cluster role on Dps fails; mixed-case roles on Dps
fail; cluster role with neither admin nor driver needs only ClusterClient;
Cluster:Roles wins over a stray OTOPCUA_ROLES. 16/16 green (124/124 project).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Per-cluster mesh Phase 6 (Task 5). A node carrying a cluster-{ClusterId}
role (RoleParser.IsClusterRole) is in the split topology, where the
Phase 2/3/5 DPS dark-switch branches deliver nothing across the mesh
boundary. SplitTopologyTransportValidator fails host start unless such a
node uses the mesh-crossing transports: MeshTransport:Mode=ClusterClient
always; Telemetry:Mode=Grpc if it has the driver role; TelemetryDial:Mode
=Grpc if it has the admin role. It is a no-op for any node with no
cluster-role (legacy / single-mesh / test). Roles + the three modes are
cross-read from IConfiguration, mirroring ConfigSourceOptionsValidator.
Registered via AddValidatedOptions on MeshTransportOptions with
ValidateOnStart, alongside the sibling validators.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The per-cluster contact map only gains a key when a row parses, so an
all-malformed cluster has no key at all and is caught by the absent-key arm;
the empty-set guard is defensive/unreachable for the DB producer. Comment-only
(Task 4 code-review minor follow-up).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Task 3 review: scoping only the rows/enabled DB queries by ClusterId left
the observed set (from Cluster.State.Members) filtered by the driver role
alone. In the mixed-mesh window — DB scoping live via a cluster role, but
the physical Akka mesh not yet split so central still sees foreign members
via gossip — a foreign-cluster driver member stays in observed while its row
is filtered out, hitting the RunningNodeHasNoRow branch, which logs at ERROR.
That traded the Warning-level false positive for a worse Error-level one.
Filter observed by the node's own cluster ROLE too. Extracted the projection
into a static FilterOwnClusterDriverMembers(members, ownClusterRole) so the
membership filter is unit-testable without seeding a live cluster (the actor
harness joins as admin only, so observed is always empty). Sourced at
registration from IClusterRoleInfo.ClusterRole alongside ClusterId. Null role
(legacy, no cluster role) keeps every driver member — reconcile-all, unchanged.
Also clarified the ownClusterId doc wording (non-null-and-non-empty).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Review follow-ups on the redundancy re-home (Phase 6, Task 2):
- Program.cs: the hasDriver comment claimed IClusterRoleInfo "throws at host
start if this driver node carries no cluster role" — false since the fallback
landed. Rewrite it to describe the cluster-{ClusterId} scope with the driver-role
fallback for legacy single-mesh / not-yet-migrated nodes.
- ClusterRoleInfo.cs: the SubscriberActor comment described RedundancyStateActor as
the "admin-role singleton" — stale. Note it is now a cluster-{ClusterId}-scoped
singleton spawned on every driver node (LeaderChanged stays a no-op here).
- Add a host/registry-level test for the driver-role FALLBACK path (roles ["driver"],
no cluster role) closing the coverage asymmetry — it was proven only in the pure
helper. Asserts the node boots and registers RedundancyStateActorKey, i.e. the boot
the earlier throw-based version would have aborted.
The boot helper now supplies an in-memory IDbContextFactory and a fake IClusterRoleInfo:
Akka.Hosting invokes singleton props factories eagerly at StartAsync, and the admin
ClusterNodeAddressReconciler factory reads both (the latter for its per-cluster reconcile
scope, added by a sibling Phase 6 task) — without them the admin boot NREs.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
After the fleet splits into one Akka mesh per application Cluster, a
ClusterClientReceptionist serves only its own mesh — so the single
fleet-wide client's SendToAll reached only the mesh whose receptionist
answered, leaving every other cluster silently on its old configuration.
Central now holds one ClusterClient per ClusterId and fans SendToAll
across all of them. LoadContactsFromDb selects ClusterId and groups the
receptionist contacts per cluster (keeping the per-row TryParse guard and
the enabled/non-maintenance filter); ContactsLoaded carries
ContactsByCluster; HandleContactsLoaded diffs per cluster (rebuild changed,
stop+drop vanished, warn-don't-create on empty); RebuildClient builds a
per-cluster client under a clusterId-derived actor name. IMeshClusterClientFactory.Create
gained a clusterId parameter so the per-cluster clients get distinct,
diagnosable names. ApplyAck handling and the DPS branch are unchanged; the
deploy path stays payload-free.
Tests: focused unit coverage of the grouping + fan-out (one-client-per-cluster,
fan-out SendToAll to every cluster client) via the recording factory double,
plus the real two-mesh boundary test extended to central + two separate site
meshes proving one dispatch reaches both and a client scoped to one cluster
never crosses into another. Red-before-green verified: crippling the fan-out
to a single client fails the reaches-both-meshes assertion.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 6 splits the fleet into one mesh per Cluster, so an admin (central)
node's Cluster.State.Members shows only its own pair — it no longer sees
site members via gossip. The ClusterNodeAddressReconciler singleton compared
live membership against EVERY ClusterNode row fleet-wide, so post-split every
foreign-cluster row would log EnabledRowNotInCluster forever.
Scope the actor's two DB queries to rows whose ClusterId matches the admin
node's own (IClusterRoleInfo.ClusterId, sourced at registration). A legacy
admin node with no cluster role (null ClusterId) still reconciles the whole
fleet — it genuinely sees every member via gossip. The pure Reconcile
function and AddressMismatchKind semantics are unchanged; scoping lives
entirely in the actor's queries.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 6 splits the single fleet-wide Akka mesh into one 2-node mesh per
application Cluster. RedundancyStateActor was an admin-scoped cluster singleton
that elected ONE fleet-wide driver Primary — but after the split a driver-only
site pair has no admin node in its mesh to host it, so its Primary would never
be elected.
Re-home it: remove the redundancy WithSingleton block from the admin
WithOtOpcUaControlPlaneSingletons set, and add WithOtOpcUaClusterRedundancySingleton,
a singleton spawned from Program.cs's hasDriver branch on EVERY driver node (the
fused central included). It scopes to the node's own cluster-{ClusterId} role when
present, and falls back to the fixed `driver` role otherwise. Each mesh then has
exactly one, electing its own pair-local Primary and publishing redundancy-state
on its own mesh's DistributedPubSub. The election logic (oldest Up driver member)
and the DPS publish are unchanged — they become pair-local after the split.
The driver-role fallback (Decision 2) deliberately does NOT throw when a node has
no cluster role: that is the pre-Phase-6 fleet-wide behavior on a legacy single
mesh, so legacy/harness (TwoNodeClusterHarness: admin,driver) and not-yet-migrated
deployments keep booting unchanged. On a genuinely split 2-node mesh the `driver`
role is already pair-local (the mesh IS the pair), so the fallback is correct in
both worlds. Cluster-scope, when present, additionally survives an accidental
two-mesh merge by keeping one singleton per cluster.
The role scope + driver-role fallback are pinned by unit tests against the
extracted BuildClusterRedundancySingletonOptions helper (the BuildDowningHocon
pattern); admin-removal + driver-registration are pinned behaviorally against a
booted node's ActorRegistry.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 6 Task 1. IClusterRoleInfo gains ClusterRole/ClusterId, derived from
the node's configured AkkaClusterOptions.Roles at construction time (not
live Cluster.State) so the identity is available before the cluster forms.
First cluster-scoped role wins when more than one is configured, logged as
a Warning. Updates the FakeClusterRoleInfo test double in
ServiceCollectionExtensionsTests to satisfy the new interface members.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 6 (per-cluster mesh split) needs nodes to carry a cluster-scoped
role like cluster-SITE-A; RoleParser previously rejected anything
outside the fixed admin/driver/dev set. Adds IsClusterRole/
ClusterIdFromRole plus well-known-role constants, and points the three
duplicate "driver" string literals (RedundancyStateActor,
ClusterNodeAddressReconcilerActor, ServiceCollectionExtensions) at the
new RoleParser.Driver so the value has one source, without renaming
any existing DriverRole symbol.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Plan + task ledger for splitting the single fleet mesh into three 2-node
meshes (central + per-site pairs), cluster-scoped roles + redundancy
singleton, per-cluster ClusterClient, own-cluster reconciler, split-topology
transport validator + flipped defaults, pair-local secrets, and the docker-dev
rig rewrite. Decisions settled with the user 2026-07-24.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Full 12-stream telemetry mesh formed in Grpc mode, AdminUI pill live (data path
proven), kill-and-reconnect recovered a node's stream in ~5s; Dps baseline dials
nothing. Surfaced+validated the ClusterNode.GrpcPort upgrade-backfill gotcha.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Joseph Doherty