Compare commits

..

44 Commits

Author SHA1 Message Date
Joseph Doherty ee12568cab refactor(modbus-rtu): distinct CrcMismatch/UnitMismatch desync reasons
v2-ci / build (pull_request) Successful in 4m1s
v2-ci / unit-tests (pull_request) Failing after 13m33s
The RTU framer overloaded DesyncReason.TruncatedFrame for three distinct
failures — a genuine short read, a CRC-16 validation failure, and a
CRC-valid reply from the wrong RS-485 slave. Split the latter two into
dedicated CrcMismatch / UnitMismatch enum members so a future .Reason
consumer (metrics, operator diagnostics) can tell a wiring/EMI CRC fault
apart from a mis-addressed multi-drop reply. Behaviour is unchanged — all
three still throw ModbusTransportDesyncException and tear the socket down.

Framing tests now assert the specific Reason on each path.

Follow-up #12 from the Modbus RTU-over-TCP plan.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-25 15:13:48 -04:00
Joseph Doherty e27eb77187 docs(modbus-rtu): record Wave-1 RTU-over-TCP live /run gate result (PASSED)
v2-ci / build (pull_request) Successful in 3m45s
v2-ci / unit-tests (pull_request) Failing after 13m54s
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 15:50:10 -04:00
Joseph Doherty fcc7ed698e harden(modbus-rtu): factory throws on unknown transport + document RTU FC-shape assumption
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 15:33:49 -04:00
Joseph Doherty 4a8c9badce docs(modbus-rtu): note the co-running :5021 rtu_over_tcp fixture in Docker README + Dockerfile
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 14:36:00 -04:00
Joseph Doherty 650e27075f test(modbus-rtu): add rtu_over_tcp pymodbus fixture + RTU-over-TCP integration tests
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 14:29:10 -04:00
Joseph Doherty 132e7b63f6 feat(modbus-rtu): add Transport selector to the AdminUI Modbus driver form
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 14:19:15 -04:00
Joseph Doherty 13bd9820b0 feat(modbus-rtu): route driver + probe transports through ModbusTransportFactory
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 14:12:12 -04:00
Joseph Doherty 1d90bf3611 feat(modbus-rtu): add ModbusTransportFactory switch on Transport mode
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 14:08:10 -04:00
Joseph Doherty 20be5416b9 feat(modbus-rtu): bind Transport mode on options/DTO/factory (string-enum guard)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 14:04:34 -04:00
Joseph Doherty 47e9cd56ef feat(modbus-rtu): add ModbusRtuOverTcpTransport (RTU framing over socket lifecycle)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 13:56:19 -04:00
Joseph Doherty 2f38b5b285 refactor(modbus): extract ModbusSocketLifecycle from ModbusTcpTransport (behaviour-preserving)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 13:47:18 -04:00
Joseph Doherty 20100c36ad feat(modbus-rtu): validate response unit + cover partial-read/truncation in ModbusRtuFraming
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 13:42:25 -04:00
Joseph Doherty eed6617784 feat(modbus-rtu): add ModbusRtuFraming (ADU build + FC-aware length-less deframe)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 13:33:59 -04:00
Joseph Doherty 8d0f60ec51 feat(modbus-rtu): add ModbusCrc CRC-16/MODBUS helper + golden vectors
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 13:30:05 -04:00
Joseph Doherty e787aa572b feat(modbus-rtu): add ModbusTransportMode enum (Tcp | RtuOverTcp)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 13:28:30 -04:00
Joseph Doherty 963eec1b32 chore: gitignore .worktrees/ for subagent-driven-development
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 13:26:01 -04:00
Joseph Doherty 7a9caf141c docs(drivers): add worktree + subagent execution commands to the tracking doc
v2-ci / build (push) Successful in 3m46s
v2-ci / unit-tests (push) Failing after 14m11s
Per-plan copy-paste command (subagent-driven-development in a git worktree),
plus the slash-command and executing-plans/resume alternatives, and a note that
the four independent plans can run in concurrent worktrees.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 13:24:36 -04:00
Joseph Doherty 88adec047b refactor(health): adopt the shared active-node check (Health 0.3.0)
ClusterPrimaryHealthCheck was written three days' worth of debugging ago because
the shared ActiveNodeHealthCheck selected by RoleLeader and reported Healthy for
any node lacking the role. Health 0.3.0 fixes the shared check — oldest Up
member, role-preference scoping, Unhealthy when the node owns no active work —
so the private copy is now duplication rather than a workaround, and it is
deleted.

RolePreference [admin, driver] reproduces exactly what it did: a fused central
node answers for admin (where the singletons and the AdminUI are pinned), a
driver-only site node answers for driver.

SelectOldestUpMemberOfRole now delegates to the shared ClusterActiveNode instead
of re-implementing the age ordering. This is the point of the change rather than
a tidy-up: the redundancy snapshot drives the OPC UA ServiceLevel 250/240 split
while the health tier drives Traefik's admin routing, so two copies of "oldest Up
member of a role" would be two chances to advertise one node as authoritative
while gating the data plane on another. ControlPlane takes a ZB.MOM.WW.Health.Akka
reference for it; the layering trade-off is recorded at the PackageReference.

Behaviour note: a node carrying neither admin nor driver now answers 503 on the
active tier instead of 200. That case is reachable — RoleParser admits dev-only
and cluster-role-only nodes — and 503 is correct, since such a node owns no
active work and must not be in an active-tier pool. No docker-dev node is
affected: every rig node is admin+driver or driver-only.

Tests replaced in kind. The rule itself is now pinned in the library against a
real two-node cluster; what stays OtOpcUa's own decision is the wiring, so
ActiveTierRegistrationTests pins which check is on the active tag, that it is
registered on driver-only nodes too, and that it is not on the ready tier.

Verified: Host.Tests 25/25, ControlPlane redundancy 15/15.
2026-07-24 13:21:11 -04:00
Joseph Doherty 76e55fc112 docs(drivers): Wave 0-2 tracking doc + Wave 1/2 implementation plans
Add a living progress tracker (docs/plans/2026-07-24-driver-expansion-tracking.md)
for the driver-expansion program's Waves 0-2, linked from the program design doc's
document map. One row per deliverable pointing at design + implementation plan + status.

Add executable implementation plans (plan .md + co-located .tasks.json) for the four
Wave 1/2 drivers, each authored by writing-plans methodology off its design doc and
mirroring the relevant existing driver:

- Modbus RTU        11 tasks  (extends Driver.Modbus; RTU-over-TCP, direct-serial descoped)
- SQL poll          22 tasks  (ISqlDialect seam, SQL Server only in v1; SQLite + central-SQL fixtures)
- MTConnect Agent   23 tasks  (browse free via Wave-0 seam; TrakHound-vs-hand-rolled = Task 0)
- MQTT/Sparkplug B  27 tasks  (P1 plain MQTT 0-14 then P2 Sparkplug 15-26; MQTTnet-5 pinning spike = Task 0)

Wave 0 (universal browser) remains merged (056887d6) with its live gate open (#468).
All four new plans are design-only -> plan-ready; nothing built yet.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 13:17:44 -04:00
Joseph Doherty 4550486144 fix(health): active tier reports the cluster Primary, not the admin role leader
Closes the remaining half of #494, and corrects the half fixed in 8dd9da7d.

The shared ActiveNodeHealthCheck(role: "admin") answered the wrong question twice
over. It returns Healthy for any node LACKING the role, so all four driver-only
site nodes called themselves active and no consumer could find the Primary of a
site Cluster. And it selects by RoleLeader - the lowest-ADDRESSED member - which
is not where Akka places singletons.

Replaced with ClusterPrimaryHealthCheck ("cluster-primary"). One rule: this node
is active iff it is the OLDEST Up member carrying its own active role, where the
active role is admin when the node has it and driver otherwise. That serves both
consumers correctly - a fused admin node answers for admin, so Traefik pins the
AdminUI to the node hosting the singletons; a driver-only site node answers for
driver, which is exactly SelectDriverPrimary, the same election behind
IsDriverPrimary and the OPC UA ServiceLevel 250/240 split. Per-mesh scoping is
free after Phase 6: ClusterState.Members already contains only this node's own
Cluster.

SelectDriverPrimary is generalised to SelectOldestUpMemberOfRole so the
age-ordering rule has one implementation. Two copies of "oldest Up member of a
role" would be two chances to silently disagree about who is in charge, and the
tier and the redundancy snapshot must never disagree.

THE ROLE-LEADER HALF MATTERED IN PRACTICE, not just in theory. On the rebuilt rig
the two orderings diverge right now:

  akka leader (lowest address) = central-1
  oldest admin member          = central-2   <- hosts the singletons

8dd9da7d made the tier a real 503 but still selected by RoleLeader, so it would
have pinned Traefik to central-1 - the node NOT hosting the work. With this change
Traefik correctly routes to central-2.

Live-verified, exactly one 200 per mesh:

  MAIN     central-2 200   central-1 503     traefik: central-2 UP, central-1 DOWN
  SITE-A   site-a-1  200   site-a-2  503
  SITE-B   site-b-1  200   site-b-2  503

Tests: role-selection matrix and the startup-safe Degraded path in Host.Tests
(26 pass); parity between the tier's selector and the redundancy snapshot's, plus
role scoping, added to RedundancyPrimaryElectionTests, which forms a real two-node
cluster deliberately built so the oldest member is not the lowest-addressed one
(5 pass).
2026-07-24 12:43:27 -04:00
Joseph Doherty 8dd9da7d4d fix(health): bump to ZB.MOM.WW.Health 0.2.1 — Traefik leader-pinning now works
0.2.1 makes the shared ActiveNodeHealthCheck return Unhealthy (503), not
Degraded (200), for a node that carries the role but is not the role leader.
That is what the shared health spec always specified, what docs/ServiceHosting.md,
docs/v2/Architecture-v2.md and the 2026-05-26 alignment design all describe
("200 only on the Admin-role leader; 503 elsewhere"), and what
docker-dev/traefik-dynamic.yml + scripts/install/traefik-dynamic.yml assume when
they use /health/active as the load-balancer probe.

The implementation returned Degraded, which MapZbHealth maps to 200, so BOTH
central nodes always passed the probe and Traefik load-balanced the admin UI
across the pair instead of pinning it to the leader. The docs were right; the
code was wrong. Closes the Traefik half of #494.

No OtOpcUa code change — package pin only.

Live-verified on docker-dev, with the MAIN pair genuinely formed (memberCount 2,
both members naming one leader):

  central-1 (leader)  /health/active -> 200   traefik UP
  central-2 (standby) /health/active -> 503   traefik DOWN

Failover drill (README step 2), which had never actually exercised anything:
stopping central-1 flipped Traefik to central-2 within 20 s and
http://localhost:9200/ answered 200 continuously across the swap — no outage.
Restarting central-1 reclaimed the role-leader and Traefik flipped back.

README corrected on two counts: the drill now states that exactly ONE backend is
UP by design, and step 3 no longer claims central-2 keeps leadership — RoleLeader
is the lowest-address member, so central-1 deterministically reclaims it. Added
the cold-boot caveat that starting both nodes at once can leave each self-forming
a 1-member cluster, in which case both are their own leader and both answer 200.

STILL OPEN in #494: driver-only nodes. The check is scoped to the admin role and
returns Healthy for any node that lacks it, so all four site nodes answer 200 and
the real per-Cluster redundancy Primary (oldest Up driver member, IsDriverPrimary)
is still not what this tier reports.
2026-07-24 11:30:50 -04:00
Joseph Doherty 9ed2251f37 chore(health): bump ZB.MOM.WW.Health* to 0.2.0
v2-ci / build (push) Successful in 3m38s
v2-ci / unit-tests (push) Failing after 14m15s
Picks up the additive per-entry `data` object in the canonical health JSON and
the cluster-view data the shared AkkaClusterHealthCheck now publishes (leader /
selfAddress / selfRoles / memberCount / unreachableCount).

No OtOpcUa code change is needed: the "akka" ready/active check is already the
shared ZB.MOM.WW.Health.Akka.AkkaClusterHealthCheck (Health/HealthEndpoints.cs),
so /health/ready starts carrying entries["akka"].data.leader on the bump alone.
Payloads from every other check are byte-identical — data is emitted only when a
check publishes some.

Prereq for the family overview dashboard (scadaproj
docs/plans/2026-07-22-overview-dashboard-impl-plan.md, Phase 2).

Verified: Host 21/21, AdminUI 692/692, Host builds 0 warnings.
Live rig check (admin node serves data.leader) deferred to the dashboard's
acceptance run.
2026-07-24 10:00:41 -04:00
Joseph Doherty d1dac87f6f docs: record the bootstrap guard (CLAUDE.md + Phase 7 note)
v2-ci / build (push) Successful in 4m38s
v2-ci / unit-tests (push) Failing after 16m7s
The simultaneous-cold-start split-brain, carried as a Phase 7 "candidate follow-up,
not shipped", is now closed by the opt-in Cluster:BootstrapGuard (279d1d0f). Documents
the guard in CLAUDE.md's bootstrap section and marks the Phase 7 finding closed.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 08:35:16 -04:00
Joseph Doherty 279d1d0fb1 feat(cluster): simultaneous-cold-start split-brain guard (opt-in)
v2-ci / build (push) Successful in 4m46s
v2-ci / unit-tests (push) Failing after 16m9s
Closes the residual Phase-7 gap: when BOTH nodes of a 2-node pair cold-start at the
same instant, each self-first seed runs FirstSeedNodeProcess, times out waiting for
the other, and forms its own 1-node cluster — two Primaries in one pair (the Phase 6
live gate reproduced this reliably on docker). The prior mitigation was operational
(staggered start / compose depends_on), which does not exist on production hardware.

The guard (dark switch Cluster:BootstrapGuard:Enabled, default OFF) prevents the split
without giving up cold-start-alone:
- The lower-address node is the preferred founder: self-first, forms immediately.
- The higher node probes its partner's Akka port (TCP connect) up to PartnerProbeSeconds:
  reachable => peer-first (join the founder, never race it); unreachable => self-first
  (partner is dead, form alone). The order is decided BEFORE the single JoinSeedNodes,
  from an explicit reachability signal — never re-formed mid-handshake (the retired
  SelfFormAfter failure mode). When on, Akka gets no config seeds (BuildClusterOptions)
  and ClusterBootstrapCoordinator drives the join.

Review-driven hardening: case-insensitive tie-break (a hostname-casing mismatch would
reopen the split); fail-fast validation of the timing knobs; the residual "founder dies
in the probe->join window" hang is documented and made operator-visible (warning + a
restart recovers). Real-ActorSystem coordinator tests cover the load-bearing higher-node
cold-start-alone case; 147/147 Cluster tests pass.

Live-gated on docker-dev (site-a = enablement demo, serialization removed, guard on;
site-b keeps depends_on-serialization + guard off as the A/B control): simultaneous
start -> site-a-1 founds, site-a-2 probes-reachable-joins -> 250/240, NO split;
higher-node-alone -> probes dead founder, self-forms -> 250; founder rejoin -> 240.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 08:34:29 -04:00
Joseph Doherty c50ebcf7dc docs(mesh): Phase 7 DONE — failover drills PASSED, program complete
v2-ci / build (push) Successful in 3m48s
v2-ci / unit-tests (push) Failing after 14m8s
All 6 failover drills pass on the docker-dev three-mesh rig:
- D1 MAIN graceful manual-failover via the AdminUI "Trigger failover" button —
  Primary leaves, peer→250, restarts and rejoins with no split.
- D2 SITE-A auto-down failover, both directions; D3 SITE-B crash-the-oldest.
- D4 auto-down 1-vs-1 survival — every survivor stayed Up (closes the gate
  deferred since Phase 0a).
- D5 self-first cold-start-alone — a node boots with its partner down and forms
  its own mesh (gate b).
- D6 recovery — restarted nodes rejoin as Secondary (240).

Ships a one-page co-located operator runbook (OtOpcUa + ScadaBridge on shared
site VMs). Manual-failover button confirmed MAIN-only by construction; site
pairs (driver-only, no UI) fail over via auto-down. Carried Phase-6 finding —
simultaneous cold-start of both pair VMs can split — mitigated operationally in
the runbook (staggered start); a product-level join guard is a candidate
follow-up, not shipped.

The per-cluster mesh program (Phases 0a–7) is COMPLETE.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 08:01:26 -04:00
Joseph Doherty ecf5410433 docs(claude): record the Phase 6 boot-crash wiring landmine
v2-ci / build (push) Successful in 5m6s
v2-ci / unit-tests (push) Failing after 16m29s
The cluster-redundancy singleton's role scope is computed at AddAkka-configurator
time and must come from AkkaClusterOptions, never by resolving IClusterRoleInfo
there (it depends on the ActorSystem being built → stack overflow at boot). Caught
by the Phase 6 live gate; fixed in 3a4ed8dd.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 07:09:38 -04:00
Joseph Doherty 7774797770 docs(mesh): Phase 6 live gate PASSED — all 8 legs green, exit gate MET
v2-ci / build (push) Successful in 5m13s
v2-ci / unit-tests (push) Failing after 16m32s
Records the full live-gate run against master on the docker-dev three-mesh rig.
All 8 legs pass: three isolated 2-node meshes, one Primary per pair (250/240),
deploy sealed acks=6 over per-cluster ClusterClient, telemetry gRPC :4056 up +
reconnect, reconciler own-cluster-scoped, secrets pair-local, split validator
refuses Dps, and per-pair failover with auto-down 1-vs-1 survival (also closes the
deferred Phase 0a gate). Three defects were caught and fixed forward (3a4ed8dd,
792f28ec) — see the live-gate doc's Findings section. Task 9 marked completed.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 07:08:52 -04:00
Joseph Doherty 792f28ec7b fix(docker-dev): Phase 6 rig — LDAP on all nodes + serialize site-pair startup
v2-ci / build (push) Successful in 3m43s
v2-ci / unit-tests (push) Failing after 14m27s
Two docker-dev-only defects the Phase 6 live gate surfaced (both invisible to
`docker compose config`, which validates the file but never boots the app):

1. Driver-only site nodes crashed at boot with OptionsValidationException — LDAP
   transport None + AllowInsecure false. Only central carried the Security__Ldap__*
   block, but the site nodes serve OPC UA endpoints and validate LDAP options too, and
   they override `environment` wholesale (YAML `<<:` doesn't deep-merge it). Extracted
   the block to a shared &ldap-env anchor merged into every host node.

2. Site pairs split-brained at simultaneous startup (both nodes "JOINING itself",
   250/250 instead of 250/240). Both are self-first seeds, so both run Akka's
   FirstSeedNodeProcess; the partner didn't depend on its pair founder, so they raced
   and each formed a 1-node cluster. Added an &akka-founder-healthcheck (bash /dev/tcp
   probe of Akka 4053 — no nc/curl in the image) to site-a-1/site-b-1 and switched the
   partners to depends_on service_healthy, so the founder forms first and the partner
   JOINS it. (Carry the underlying simultaneous-cold-start property to Phase 7 — it is
   inherent to symmetric self-first seeding, present for central since #459.)

Live gate record: docs/plans/2026-07-24-mesh-phase6-live-gate.md (legs 1-2 PASS —
three isolated meshes, one Primary per pair at 250/240).

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 06:54:27 -04:00
Joseph Doherty 3a4ed8ddb5 fix(mesh): break Phase 6 boot-crash — cluster-redundancy singleton scope must not resolve IClusterRoleInfo eagerly
Phase 6 wired the re-homed redundancy singleton in Program.cs as
`WithOtOpcUaClusterRedundancySingleton(sp.GetRequiredService<IClusterRoleInfo>())`
inside the AddAkka configurator lambda. That lambda runs WHILE the ActorSystem is
being built, but ClusterRoleInfo depends on the ActorSystem (it is a live
Cluster.State view) — so resolving it there recurses into ActorSystem construction
and every node dies at boot with StackOverflowException. It compiles cleanly (a
runtime DI cycle) and RedundancyStateSingletonRehomeTests passed a hand-built
FakeClusterRoleInfo straight into the extension, mocking around the composition-root
line that overflows. The docker-dev live gate caught it on first boot.

Derive the singleton's cluster-role scope from AkkaClusterOptions (pure config, no
ActorSystem) instead: BuildClusterRedundancySingletonOptions / the extension now take
AkkaClusterOptions and compute Role = Roles.FirstOrDefault(IsClusterRole) ?? driver —
identical to ClusterRoleInfo.ClusterRole, which derives from the same config. Program.cs
passes IOptions<AkkaClusterOptions>.Value, which has no ActorSystem dependency.

Live-verified: rebuilt image boots with zero stack-overflow lines, cluster singletons
form per mesh. 5/5 rehome tests pass; Host builds clean.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 06:54:14 -04:00
Joseph Doherty 2839deb5be docs(mesh): Phase 6 ledger — tasks 0-8 done, task 9 live gate in progress
v2-ci / build (push) Successful in 4m2s
v2-ci / unit-tests (push) Failing after 14m18s
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 03:03:30 -04:00
Joseph Doherty 8a173bbaf6 docs(mesh): Phase 6 — mark done, remove single-mesh caveats, document pair-local meshes
Redundancy.md: KNOWN-LIMITATION (per-Akka-cluster election) marked RESOLVED;
new 'Per-cluster meshes (Phase 6)' section + pair-local secrets note; manual-
failover mesh-scope caveat removed; auto-down 1-vs-1 note updated (every mesh is
now two nodes, drill deferred to Phase 7). IManualFailoverService + ClusterRedundancy
razor caveats updated to pair-local reality. Configuration.md documents the
cluster-{ClusterId} role, per-pair self-first seeds, SplitTopologyTransportValidator,
and the intentional Dps default. Program + design status tables + CLAUDE.md marked
Phase 6 DONE.

(Task 8 subagent completed the edits but died on an API error before committing;
reviewed and committed by the controller.)

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 03:02:35 -04:00
Joseph Doherty 2a057a466f docs(mesh): Phase 6 plan+ledger updates through Task 7
Records the Task 2/3 fallback+scoping corrections, the Task 6 no-flip decision
(data-backed), and task statuses 0-7.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 02:39:08 -04:00
Joseph Doherty 741ab97b89 feat(mesh-phase6): split docker-dev into three independent 2-node Akka meshes
Rewrite docker-dev/docker-compose.yml from one shared six-node gossip ring
into three self-contained pairs (MAIN: central-1/2, SITE-A: site-a-1/2,
SITE-B: site-b-1/2), all still on the ActorSystem name `otopcua` and the one
shared docker network — separation is purely by per-pair self-first
Cluster:SeedNodes, faithful to how production splits over a WAN rather than
a firewall.

Per node:
- Cluster__Roles / OTOPCUA_ROLES gain a `cluster-<ClusterId>` role
  (cluster-MAIN / cluster-SITE-A / cluster-SITE-B), the marker
  SplitTopologyTransportValidator and ClusterRoleInfo key off.
- Cluster__SeedNodes now lists only the node's own pair (self first, partner
  second); site nodes no longer seed off central-1.
- MeshTransport__Mode default flips Dps -> ClusterClient (still overridable
  via OTOPCUA_MESH_MODE) on all six nodes — SplitTopologyTransportValidator
  now refuses to boot a cluster-role node left on Dps.
- Telemetry__Mode=Grpc added on all six driver nodes; TelemetryDial__Mode=Grpc
  added on central-1/central-2 only (the admin pair).
- ConfigSource/ConfigServe modes and MeshTransport contact points are
  unchanged (already split-correct since Phases 3/4/2).

seed-clusters.sql needed no changes — the ClusterNode rows already carry the
correct ClusterId/Host/AkkaPort/GrpcPort per node. Validated with
`docker compose -f docker-dev/docker-compose.yml config`.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 02:38:26 -04:00
Joseph Doherty 344b0d3334 fix(cluster): validate split topology against EFFECTIVE roles, not raw Cluster:Roles
Code review found SplitTopologyTransportValidator read the wrong role view,
defeating its fail-loud purpose two ways:
 (a) Roles-source divergence. AkkaClusterOptions.Roles binds Cluster:Roles
     but falls back to OTOPCUA_ROLES (RoleParser.Parse) when it is empty. A
     node configured with only OTOPCUA_ROLES=driver,cluster-SITE-A genuinely
     carries the cluster role in Akka, yet the validator saw no cluster role
     and passed silently on MeshTransport:Mode=Dps (the documented incident).
 (b) Case. The Cluster:Roles bind path is verbatim while RoleParser.Parse
     lowercases the OTOPCUA_ROLES path; 'Cluster-SITE-A' / 'Driver' / 'Admin'
     slipped past the ordinal checks.

Resolve effective roles the same way the node does — Cluster:Roles, else
RoleParser.Parse(env OTOPCUA_ROLES) exactly as Program.cs — then normalize
both (trim + ToLowerInvariant) before IsClusterRole/driver/admin. The message
still names the ClusterId in the operator's original spelling. Exemption and
all other behaviour unchanged. Contained to the validator.

Tests: OTOPCUA_ROLES-only cluster role on Dps fails; mixed-case roles on Dps
fail; cluster role with neither admin nor driver needs only ClusterClient;
Cluster:Roles wins over a stray OTOPCUA_ROLES. 16/16 green (124/124 project).

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 02:29:41 -04:00
Joseph Doherty a886d5e6e0 feat(cluster): refuse to boot a split-topology node on a DPS transport mode
Per-cluster mesh Phase 6 (Task 5). A node carrying a cluster-{ClusterId}
role (RoleParser.IsClusterRole) is in the split topology, where the
Phase 2/3/5 DPS dark-switch branches deliver nothing across the mesh
boundary. SplitTopologyTransportValidator fails host start unless such a
node uses the mesh-crossing transports: MeshTransport:Mode=ClusterClient
always; Telemetry:Mode=Grpc if it has the driver role; TelemetryDial:Mode
=Grpc if it has the admin role. It is a no-op for any node with no
cluster-role (legacy / single-mesh / test). Roles + the three modes are
cross-read from IConfiguration, mirroring ConfigSourceOptionsValidator.
Registered via AddValidatedOptions on MeshTransportOptions with
ValidateOnStart, alongside the sibling validators.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 02:13:47 -04:00
Joseph Doherty d98b32640b docs(mesh): correct the all-malformed-cluster comments in CentralCommunicationActor
The per-cluster contact map only gains a key when a row parses, so an
all-malformed cluster has no key at all and is caught by the absent-key arm;
the empty-set guard is defensive/unreachable for the DB producer. Comment-only
(Task 4 code-review minor follow-up).

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 02:08:05 -04:00
Joseph Doherty 1ce2042e14 fix(mesh): also scope the reconciler's observed membership to the own cluster
Task 3 review: scoping only the rows/enabled DB queries by ClusterId left
the observed set (from Cluster.State.Members) filtered by the driver role
alone. In the mixed-mesh window — DB scoping live via a cluster role, but
the physical Akka mesh not yet split so central still sees foreign members
via gossip — a foreign-cluster driver member stays in observed while its row
is filtered out, hitting the RunningNodeHasNoRow branch, which logs at ERROR.
That traded the Warning-level false positive for a worse Error-level one.

Filter observed by the node's own cluster ROLE too. Extracted the projection
into a static FilterOwnClusterDriverMembers(members, ownClusterRole) so the
membership filter is unit-testable without seeding a live cluster (the actor
harness joins as admin only, so observed is always empty). Sourced at
registration from IClusterRoleInfo.ClusterRole alongside ClusterId. Null role
(legacy, no cluster role) keeps every driver member — reconcile-all, unchanged.

Also clarified the ownClusterId doc wording (non-null-and-non-empty).

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 02:07:02 -04:00
Joseph Doherty a4d3ab4005 docs(mesh): fix stale redundancy-singleton comments + prove the driver fallback end-to-end
Review follow-ups on the redundancy re-home (Phase 6, Task 2):

- Program.cs: the hasDriver comment claimed IClusterRoleInfo "throws at host
  start if this driver node carries no cluster role" — false since the fallback
  landed. Rewrite it to describe the cluster-{ClusterId} scope with the driver-role
  fallback for legacy single-mesh / not-yet-migrated nodes.
- ClusterRoleInfo.cs: the SubscriberActor comment described RedundancyStateActor as
  the "admin-role singleton" — stale. Note it is now a cluster-{ClusterId}-scoped
  singleton spawned on every driver node (LeaderChanged stays a no-op here).
- Add a host/registry-level test for the driver-role FALLBACK path (roles ["driver"],
  no cluster role) closing the coverage asymmetry — it was proven only in the pure
  helper. Asserts the node boots and registers RedundancyStateActorKey, i.e. the boot
  the earlier throw-based version would have aborted.

The boot helper now supplies an in-memory IDbContextFactory and a fake IClusterRoleInfo:
Akka.Hosting invokes singleton props factories eagerly at StartAsync, and the admin
ClusterNodeAddressReconciler factory reads both (the latter for its per-cluster reconcile
scope, added by a sibling Phase 6 task) — without them the admin boot NREs.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 01:58:48 -04:00
Joseph Doherty 6531ec1984 feat(mesh): one ClusterClient per Cluster in CentralCommunicationActor (Phase 6)
After the fleet splits into one Akka mesh per application Cluster, a
ClusterClientReceptionist serves only its own mesh — so the single
fleet-wide client's SendToAll reached only the mesh whose receptionist
answered, leaving every other cluster silently on its old configuration.

Central now holds one ClusterClient per ClusterId and fans SendToAll
across all of them. LoadContactsFromDb selects ClusterId and groups the
receptionist contacts per cluster (keeping the per-row TryParse guard and
the enabled/non-maintenance filter); ContactsLoaded carries
ContactsByCluster; HandleContactsLoaded diffs per cluster (rebuild changed,
stop+drop vanished, warn-don't-create on empty); RebuildClient builds a
per-cluster client under a clusterId-derived actor name. IMeshClusterClientFactory.Create
gained a clusterId parameter so the per-cluster clients get distinct,
diagnosable names. ApplyAck handling and the DPS branch are unchanged; the
deploy path stays payload-free.

Tests: focused unit coverage of the grouping + fan-out (one-client-per-cluster,
fan-out SendToAll to every cluster client) via the recording factory double,
plus the real two-mesh boundary test extended to central + two separate site
meshes proving one dispatch reaches both and a client scoped to one cluster
never crosses into another. Red-before-green verified: crippling the fan-out
to a single client fails the reaches-both-meshes assertion.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 01:53:20 -04:00
Joseph Doherty 2535df019b fix(mesh): scope ClusterNode address reconcile to the admin node's own cluster
Phase 6 splits the fleet into one mesh per Cluster, so an admin (central)
node's Cluster.State.Members shows only its own pair — it no longer sees
site members via gossip. The ClusterNodeAddressReconciler singleton compared
live membership against EVERY ClusterNode row fleet-wide, so post-split every
foreign-cluster row would log EnabledRowNotInCluster forever.

Scope the actor's two DB queries to rows whose ClusterId matches the admin
node's own (IClusterRoleInfo.ClusterId, sourced at registration). A legacy
admin node with no cluster role (null ClusterId) still reconciles the whole
fleet — it genuinely sees every member via gossip. The pure Reconcile
function and AddressMismatchKind semantics are unchanged; scoping lives
entirely in the actor's queries.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 01:39:06 -04:00
Joseph Doherty a2c5d57fa0 feat(mesh): re-home the redundancy singleton to a per-cluster driver singleton
Phase 6 splits the single fleet-wide Akka mesh into one 2-node mesh per
application Cluster. RedundancyStateActor was an admin-scoped cluster singleton
that elected ONE fleet-wide driver Primary — but after the split a driver-only
site pair has no admin node in its mesh to host it, so its Primary would never
be elected.

Re-home it: remove the redundancy WithSingleton block from the admin
WithOtOpcUaControlPlaneSingletons set, and add WithOtOpcUaClusterRedundancySingleton,
a singleton spawned from Program.cs's hasDriver branch on EVERY driver node (the
fused central included). It scopes to the node's own cluster-{ClusterId} role when
present, and falls back to the fixed `driver` role otherwise. Each mesh then has
exactly one, electing its own pair-local Primary and publishing redundancy-state
on its own mesh's DistributedPubSub. The election logic (oldest Up driver member)
and the DPS publish are unchanged — they become pair-local after the split.

The driver-role fallback (Decision 2) deliberately does NOT throw when a node has
no cluster role: that is the pre-Phase-6 fleet-wide behavior on a legacy single
mesh, so legacy/harness (TwoNodeClusterHarness: admin,driver) and not-yet-migrated
deployments keep booting unchanged. On a genuinely split 2-node mesh the `driver`
role is already pair-local (the mesh IS the pair), so the fallback is correct in
both worlds. Cluster-scope, when present, additionally survives an accidental
two-mesh merge by keeping one singleton per cluster.

The role scope + driver-role fallback are pinned by unit tests against the
extracted BuildClusterRedundancySingletonOptions helper (the BuildDowningHocon
pattern); admin-removal + driver-registration are pinned behaviorally against a
booted node's ActorRegistry.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 01:33:26 -04:00
Joseph Doherty 2d2f1decae feat(mesh): ClusterRoleInfo exposes the node's own cluster-scoped role
Phase 6 Task 1. IClusterRoleInfo gains ClusterRole/ClusterId, derived from
the node's configured AkkaClusterOptions.Roles at construction time (not
live Cluster.State) so the identity is available before the cluster forms.
First cluster-scoped role wins when more than one is configured, logged as
a Warning. Updates the FakeClusterRoleInfo test double in
ServiceCollectionExtensionsTests to satisfy the new interface members.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 01:17:58 -04:00
Joseph Doherty 87ff00fd30 feat(mesh): RoleParser accepts cluster-{ClusterId} roles; consolidate the driver-role literal
Phase 6 (per-cluster mesh split) needs nodes to carry a cluster-scoped
role like cluster-SITE-A; RoleParser previously rejected anything
outside the fixed admin/driver/dev set. Adds IsClusterRole/
ClusterIdFromRole plus well-known-role constants, and points the three
duplicate "driver" string literals (RedundancyStateActor,
ClusterNodeAddressReconcilerActor, ServiceCollectionExtensions) at the
new RoleParser.Driver so the value has one source, without renaming
any existing DriverRole symbol.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 01:10:42 -04:00
Joseph Doherty 02177fec7c docs(mesh): Phase 6 plan — mesh partition + co-location topology
Plan + task ledger for splitting the single fleet mesh into three 2-node
meshes (central + per-site pairs), cluster-scoped roles + redundancy
singleton, per-cluster ClusterClient, own-cluster reconciler, split-topology
transport validator + flipped defaults, pair-local secrets, and the docker-dev
rig rewrite. Decisions settled with the user 2026-07-24.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 01:07:54 -04:00
88 changed files with 8736 additions and 485 deletions
+3
View File
@@ -58,3 +58,6 @@ lib/
# Session working notes — never stage (pending.md's own hard rule; R2-12/U-4)
/pending.md
/current.md
# Subagent-driven-development git worktrees
.worktrees/
+22 -7
View File
@@ -226,9 +226,22 @@ The server supports non-transparent warm/hot redundancy via the `Redundancy` sec
**A third bootstrap gap closed 2026-07-22 — downing was never the whole story, and the fix changed twice.** Even under `auto-down`, a node cold-starting while its peer was dead **never came Up**: Akka runs `FirstSeedNodeProcess` — the only bootstrap path that can form a *new* cluster when no peer answers `InitJoin` — exclusively when `seed-nodes[0]` is the nodes own address; every other node runs `JoinSeedNodeProcess` and retries `InitJoin` forever. **`Cluster:SeedNodes` is therefore ORDERED: every node that is one of its own seeds lists ITSELF first, partner second** (docker-dev `central-2` was swapped; site nodes list only `central-1` and are legitimately not seeds). `AkkaClusterOptionsValidator` (`AddValidatedOptions`/`ValidateOnStart` in `AddOtOpcUaCluster`) enforces it at boot — **conditionally** (self must be entry 0 only *if* self is in the list, or every site node would refuse to start), matching on `PublicHostname`+`Port`, never the `0.0.0.0` bind address. The earlier fix — a `Cluster:SelfFormAfter` timer calling `Cluster.Join(SelfAddress)` — is **deleted**: it sat outside Akkas join handshake, so it could not tell "no seed answered" from "a seed answered and the join is in flight", and it islanded a manually-failed-over node live before a TCP reachability guard patched that one shape. Pinned by `SelfFirstSeedBootstrapTests` (real in-process clusters, incl. a falsifiability control proving peer-first ordering never forms) + `AkkaClusterOptionsValidatorTests`. See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering".
**Simultaneous cold-start splits — the bootstrap guard closes it (opt-in, `Cluster:BootstrapGuard:Enabled`, default OFF).** Self-first on BOTH nodes means when both cold-start at the same instant, each runs `FirstSeedNodeProcess`, times out, and forms its OWN 1-node cluster → two Primaries in one pair (the Phase 6 live gate reproduced this on docker; two separate clusters do NOT auto-merge). The guard prevents it without losing cold-start-alone: the lexicographically **lower** `host:port` node is the preferred founder (self-first, forms immediately); the **higher** node TCP-probes the partner's Akka port up to `PartnerProbeSeconds` (25s) — **reachable ⇒ peer-first (join the founder)**, **unreachable ⇒ self-first (partner dead, form alone)**. The order is chosen BEFORE the single `JoinSeedNodes`, from an explicit reachability signal — it never re-forms mid-handshake (the `SelfFormAfter` failure mode). When on, Akka gets NO config seeds (`BuildClusterOptions` empties `ClusterOptions.SeedNodes`) and `ClusterBootstrapCoordinator` drives the join. Residual trade (accepted, operator-visible via a warning + restart-recovers): the higher node hangs if the founder dies in the probe→join window. docker-dev: **site-a = guard demo** (guard on, serialization removed); **site-b = A/B control** (guard off, `depends_on: service_healthy` serialization). The guard is the production-faithful fix (compose `depends_on` doesn't exist on real VMs). Pinned by `ClusterBootstrapGuardTests` + `ClusterBootstrapCoordinatorTests` (real 2-node clusters, incl. the higher-node-cold-start-alone case). See `docs/Redundancy.md` §"Bootstrap guard".
Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b/1 done; 2 code-complete behind a dark switch; 37 not started).
**RESOLVED by per-cluster mesh Phase 6 (2026-07-24) — no longer per-Akka-cluster.** The "one Primary across the whole Akka mesh" limitation described above is gone; see the Phase 6 paragraph immediately below.
**Per-cluster mesh Phase 6 (2026-07-24) — the fleet is now three independent 2-node meshes, not one.** The single fleet-wide Akka mesh was split into one mesh per application `Cluster`: the central pair (`admin,driver,cluster-MAIN`), site-a (`driver,cluster-SITE-A`), site-b (`driver,cluster-SITE-B`) — same `ActorSystem` name, separated purely by per-pair self-first seeds (each node's `Cluster:SeedNodes` lists itself first, its own pair partner second, never a node from another cluster's pair). Consequences:
- **The redundancy singleton is cluster-scoped and spawned per driver node.** `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton (falls back to plain `driver` for a legacy node), so each pair elects and publishes its **own** Primary on its own mesh's DPS. **Two or more Primaries exist fleet-wide, one per pair — by design, not a bug.** ⚠️ **Wiring landmine (fixed `3a4ed8dd`, caught by the live gate):** the singleton's cluster-role *scope* is computed at `AddAkka`-configurator time, so it must come from `AkkaClusterOptions` (config) — **never** by resolving `IClusterRoleInfo` from the SP there. `ClusterRoleInfo` depends on the `ActorSystem`, and that lambda runs *while the ActorSystem is being built*, so resolving it recurses into ActorSystem construction and **stack-overflows every node at boot** (compiles clean; the rehome tests passed a hand-built fake straight into the extension, mocking around the composition-root line). Keep `WithOtOpcUaClusterRedundancySingleton`/`BuildClusterRedundancySingletonOptions` taking `AkkaClusterOptions`.
- `ClusterNodeAddressReconciler` is **own-cluster-scoped** — an admin node can only reconcile `ClusterNode` rows in its own cluster; it has no gossip visibility into another mesh.
- `CentralCommunicationActor` creates **one `ClusterClient` per application `Cluster`** and fans commands across them (closing the `TODO(Phase 6)` Phase 2 left behind), rather than one fleet-wide client.
- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}` role unless it also runs the split-safe transports: `MeshTransport:Mode=ClusterClient`, `Telemetry:Mode=Grpc` (driver), `TelemetryDial:Mode=Grpc` (admin). A DPS transport on a cluster-role node would try to gossip across a partition that no longer exists.
- **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`/`Telemetry:Mode`/`TelemetryDial:Mode` all still default to `Dps`; a blast-radius assessment showed flipping the default breaks every integration test and appsettings profile that leaves the key unset. The validator + explicit per-deployment config is the guardrail, not a default flip.
- **Secrets replication is pair-local** — each pair replicates its own Akka DPS secrets within itself; there is no fleet-wide/cross-cluster sync after the split (sites have no SQL, so no SQL-hub, consistent with Phase 4).
See `docs/Redundancy.md` §"Per-cluster meshes (Phase 6)", `docs/Configuration.md` §"Per-cluster mesh split (Phase 6)", and `docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`.
## Mesh command transport (`MeshTransport`)
@@ -246,12 +259,14 @@ change, not a redeploy. Things worth knowing before touching it:
`DeploymentArtifact.ResolveClusterScope`). `ClusterClient.Send` delivers to exactly **one** registered
actor — it would deploy to a single node while every other node silently kept its old configuration,
and the deployment could still seal green.
- **Exactly ONE fleet-wide ClusterClient while the fleet is one mesh.** A receptionist serves its whole
cluster, so `SendToAll` reaches every registered node-comm actor regardless of which node's address was
dialled. One client per application `Cluster` — the obviously-right-looking shape, and what Phase 6
wants — would fan each command out once per cluster (N× duplicate dispatch *and* N× duplicate ack).
Marked `TODO(Phase 6)` in `CentralCommunicationActor.RebuildClient`. **Corollary:** the contact set does
not scope delivery; excluding a `MaintenanceMode` node from the contacts does not stop it receiving.
- **RESOLVED by Phase 6 — one `ClusterClient` per application `Cluster`, not one fleet-wide.** When
this was written the fleet was still one mesh, so a receptionist served the whole cluster and
`SendToAll` reached every registered node-comm actor regardless of which node's address was dialled;
the `TODO(Phase 6)` in `CentralCommunicationActor.RebuildClient` asked for one client per application
`Cluster` instead. Phase 6 split the fleet into one mesh per `Cluster` and shipped exactly that — see
the [Redundancy](#redundancy) section's Phase 6 paragraph above. The `SendToAll`-within-a-mesh
behaviour below is otherwise unchanged; a `MaintenanceMode` node still isn't excluded from delivery
within its own mesh.
- **Inbound commands re-emit on the node's `EventStream`, not on their DPS topic** — DPS is mesh-wide and
central `SendToAll`s to every node, so a DPS re-publish would deliver N copies to every subscriber.
Node-side subscribers (`DriverHostActor`, `ScriptedAlarmHostActor`) accept both.
+3 -3
View File
@@ -140,7 +140,7 @@
<PackageVersion Include="xunit" Version="2.9.3" />
<PackageVersion Include="xunit.runner.visualstudio" Version="3.0.2" />
<PackageVersion Include="xunit.v3" Version="1.1.0" />
<PackageVersion Include="ZB.MOM.WW.Health" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Health" Version="0.3.0" />
<!--
LocalDb: embedded SQLite node-local cache + optional 2-node gRPC replication. The core
package floors the native SQLitePCLRaw.lib.e_sqlite3 at 2.1.12, so consumers inherit the
@@ -148,8 +148,8 @@
-->
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.3" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.3" />
<PackageVersion Include="ZB.MOM.WW.Health.Akka" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Health.EntityFrameworkCore" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Health.Akka" Version="0.3.0" />
<PackageVersion Include="ZB.MOM.WW.Health.EntityFrameworkCore" Version="0.3.0" />
<PackageVersion Include="ZB.MOM.WW.Telemetry" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Telemetry.Serilog" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.MxGateway.Client" Version="0.1.1" />
+5 -3
View File
@@ -100,9 +100,11 @@ The `-v` drops the SQL volume; remove it to keep ConfigDb state across restarts.
## Failover smoke
1. Watch the Traefik dashboard at `http://localhost:8089`. Both `central-1` and `central-2` should be listed as healthy in the `otopcua-admin` service.
2. `docker compose -f docker-dev/docker-compose.yml stop central-1``central-2` should pick up the admin role-leader within ~15 s (Akka split-brain stable-after). Traefik will route traffic to `central-2` once its `/health/active` returns 200.
3. `docker compose -f docker-dev/docker-compose.yml start central-1``central-1` rejoins as a follower; `central-2` keeps the leader role until something disturbs it.
1. Watch the Traefik dashboard at `http://localhost:8089`. Exactly ONE of `central-1` / `central-2` is UP in the `otopcua-admin` service — the admin role-leader. The standby is DOWN by design: `/health/active` answers 503 on it, which is how Traefik pins browser traffic to the leader. (Before `ZB.MOM.WW.Health` 0.2.1 the standby answered 200, so both showed UP and this pinning silently never worked — see `lmxopcua#494`.)
2. `docker compose -f docker-dev/docker-compose.yml stop central-1``central-2` picks up the admin role-leader within ~15 s (Akka split-brain stable-after) and Traefik flips it to UP. Verified 2026-07-24: the swap completed inside 20 s with **no gap**`http://localhost:9200/` answered 200 continuously across the failover.
3. `docker compose -f docker-dev/docker-compose.yml start central-1``central-1` rejoins and **reclaims** the admin role-leader, and Traefik flips back. It does not stay with `central-2`: Akka's `RoleLeader` is the lowest-address member, so the outcome is deterministic rather than sticky.
> **Cold-boot caveat.** Starting both central nodes simultaneously from stopped can leave each self-forming its own 1-member cluster (self-first seed lists), in which case BOTH are their own role-leader and both answer `/health/active` 200. Check `entries["akka"].data.memberCount` on `/health/ready` — it should read 2. Restarting either node makes it join the survivor.
## Resource limits & dev logging
+235 -103
View File
@@ -1,29 +1,67 @@
# docker-dev/ — Mac-friendly single-mesh hub-and-spoke fleet for v2 development + manual UI exercise.
# docker-dev/ — Mac-friendly THREE-mesh hub-and-spoke fleet for v2 development + manual UI exercise.
#
# Topology: ONE Akka mesh seeded by `central-1`. Logical separation between
# tenants is by ServerCluster.ClusterId rows (MAIN / SITE-A / SITE-B) in the one
# shared `OtOpcUa` ConfigDb — NOT by separate meshes. All six host nodes join the
# same gossip ring and the central UI deploys to every cluster over it.
# Topology (per-cluster mesh Phase 6, 2026-07-24): THREE independent 2-node Akka
# meshes, not one six-node mesh. Every node still runs the same ActorSystem name
# `otopcua` and all six containers still sit on the ONE shared docker-dev network
# — separation is purely by per-pair self-first Cluster__SeedNodes partitioning,
# never by network isolation. This is deliberately faithful to production, where
# the three meshes are separated by a WAN, not a firewall: nothing here stops
# central-1 opening a bare TCP connection to site-a-1's Akka port, it just never
# tries, because no seed list and no gossip ever names it.
#
# MAIN central-1 + central-2 (fused admin+driver, roles admin,driver,cluster-MAIN)
# SITE-A site-a-1 + site-a-2 (driver-only, roles driver,cluster-SITE-A)
# SITE-B site-b-1 + site-b-2 (driver-only, roles driver,cluster-SITE-B)
#
# Each pair seeds ONLY itself (self-first, partner-second) — no node lists a node
# outside its own mesh as a seed. Logical/tenant separation (ServerCluster.ClusterId
# rows MAIN / SITE-A / SITE-B in the one shared `OtOpcUa` ConfigDb) now COINCIDES
# with mesh separation: a `cluster-<ClusterId>` Akka role marks which mesh + which
# ServerCluster row a node belongs to (RoleParser.ClusterRolePrefix = "cluster-").
#
# Since central no longer shares a gossip ring with the site meshes, it reaches
# them over the three explicit split-topology transports instead of gossip:
# - Command plane: MeshTransport__Mode=ClusterClient (central dials each site
# mesh's ClusterClientReceptionist; DPS is gossip-only and
# would not cross the mesh boundary at all now)
# - Live telemetry: Telemetry__Mode=Grpc on every driver node (all six) +
# TelemetryDial__Mode=Grpc on the admin pair (central-1/2)
# — the site node hosts the gRPC server, central dials in
# - Deployed config: ConfigSource__Mode=FetchAndCache on the site nodes (already
# true since Phase 3/4 — driver-only nodes have no ConfigDb
# connection at all); central stays ConfigSource-less/Direct
# and keeps serving via ConfigServe.
# SplitTopologyTransportValidator (Cluster project) REFUSES to start any node
# carrying a `cluster-*` role that is left on a DPS-shaped transport — these
# three flips are therefore mandatory here, not optional dark switches.
#
# Stack:
# sql SQL Server 2022 — hosts the one ConfigDb every node uses
# cluster-seed one-shot mssql-tools job that INSERTs the ServerCluster +
# ClusterNode rows scoping each tenant, then exits (idempotent)
#
# central-1, central-2 OTOPCUA_ROLES=admin,driver — the ONLY UI + deploy
# singleton, plus the MAIN cluster's OPC UA publishers.
# Reachable at http://localhost:9200 (via Traefik).
# Both are Akka seed nodes, each listing ITSELF first
# (2026-07-22) so either can cold-start while the other
# is down; the site nodes seed off central-1 only.
# site-a-1, site-a-2 OTOPCUA_ROLES=driver — driver-only members of the same
# site-b-1, site-b-2 mesh, scoped to SITE-A / SITE-B by ClusterId. They
# serve no UI and authenticate no users; the central
# cluster manages and deploys to them over the mesh.
# central-1, central-2 OTOPCUA_ROLES=admin,driver,cluster-MAIN — the ONLY UI +
# deploy singleton, plus the MAIN cluster's OPC UA
# publishers. Reachable at http://localhost:9200 (via
# Traefik). Both are Akka seed nodes for the MAIN mesh
# ONLY, each listing ITSELF first (2026-07-22) so either
# can cold-start while the other is down.
# site-a-1, site-a-2 OTOPCUA_ROLES=driver,cluster-SITE-A — driver-only
# site-b-1, site-b-2 OTOPCUA_ROLES=driver,cluster-SITE-B — members of their
# OWN 2-node mesh (self-first seeded within the pair, NOT
# off central). They serve no UI, but they DO expose OPC UA
# data-plane endpoints (4842-4845) and authenticate those
# users against the same shared GLAuth (via the *ldap-env
# anchor); central manages + deploys to them over the
# ClusterClient/gRPC/FetchAndCache transports above, not
# gossip.
#
# Auth is real LDAP against the shared GLAuth on the Linux Docker host
# (10.100.0.35:3893, dc=zb,dc=local) — there is no LDAP container here.
# Only the admin-role central nodes carry the Security__Ldap__* block.
# EVERY host node carries the Security__Ldap__* block (shared *ldap-env anchor):
# the driver-only site nodes serve OPC UA endpoints too, and LDAP options are
# validated at boot on all of them (a node with no LDAP config crashes with
# OptionsValidationException — transport None + AllowInsecure false).
# Sign in `multi-role` / `password`.
#
# traefik PathPrefix(`/`) → central-1 / central-2 (the single UI route).
@@ -45,17 +83,12 @@
# listener, verified free against every other per-node port: Akka 4053, ConfigServe 4055 on
# central, LocalDb sync 9001 on site-a, OPC UA 4840, HTTP 9000) + Telemetry__ApiKey
# (committed dev key, matches the SQL password / secrets KEK / ConfigServe key exception).
# central-1/central-2 also carry the matching TelemetryDial__ApiKey. Both Telemetry__Mode and
# TelemetryDial__Mode are left UNSET here, so every node defaults to "Dps" (dark — telemetry
# keeps riding the existing DistributedPubSub topic). The ClusterNode.GrpcPort column (seeded
# by docker-dev/seed/seed-clusters.sql) points central's dial supervisor at each node's :4056.
# Unlike MeshTransport/ConfigSource, no `${VAR:-Dps}` interpolation is wired for either Mode key
# (a bare shell export does nothing here — these keys aren't referenced anywhere in this file, so
# there is nothing for it to substitute into). To run the Phase 5 live gate, manually add
# `Telemetry__Mode: "Grpc"` to the `environment:` block of all six driver nodes (central-1,
# central-2, site-a-1, site-a-2, site-b-1, site-b-2) and `TelemetryDial__Mode: "Grpc"` to
# central-1's and central-2's blocks — or supply both via a docker-compose.override.yml carrying
# the same keys — then `docker compose -f docker-dev/docker-compose.yml up -d` to recreate.
# central-1/central-2 also carry the matching TelemetryDial__ApiKey. As of Phase 6, Telemetry__Mode
# is set to "Grpc" on all six nodes and TelemetryDial__Mode is set to "Grpc" on central-1/central-2
# ONLY — SplitTopologyTransportValidator requires it once a node carries a cluster-* role, so this
# is no longer an optional live-gate step (see the Phase 5 comment history in git blame for the
# prior dark-switch shape). The ClusterNode.GrpcPort column (seeded by
# docker-dev/seed/seed-clusters.sql) points central's dial supervisor at each node's :4056.
#
# Usage:
# docker compose -f docker-dev/docker-compose.yml up -d --build
@@ -84,11 +117,57 @@ name: otopcua-dev
# AnnounceInterval is shortened to 5s (default 30s) so anti-entropy convergence is
# observable quickly during dev exercise. Each node keeps its own local SQLite store
# (Secrets:SqlitePath=otopcua-secrets.db, per-container); replication syncs them.
#
# Per-cluster mesh Phase 6: this DPS topic now rides each pair's OWN 2-node mesh, not one
# shared six-node mesh — so secret replication is PAIR-LOCAL. central-1/central-2 replicate
# with each other; site-a-1/site-a-2 replicate with each other; site-b-1/site-b-2 replicate
# with each other. There is no cross-cluster secret sync after the split (by design — the site
# meshes don't share a gossip ring with central or with each other, consistent with Phase 4's
# "driver-only nodes have no SQL, hence no SQL-backed cross-mesh hub" posture). No code change
# was needed for this — Enabled=true + the same KEK on every node is still correct; it just now
# converges within three independent 2-node groups instead of one six-node group.
x-secrets-env: &secrets-env
Secrets__Replication__Enabled: "true"
Secrets__Replication__AnnounceInterval: "00:00:05"
ZB_SECRETS_MASTER_KEY: "${OTOPCUA_SECRETS_KEK:-ZYGhIX0luS/XsevpCB2W18jYHMcqO6AjM9oXy+T6Zp4=}"
# EVERY node that serves an OPC UA endpoint authenticates data-plane users against the shared GLAuth —
# that includes the driver-only site nodes (ports 4842-4845), not just the admin/central pair. LDAP
# options are validated at host start on ALL of them (Enabled=true by appsettings default, Transport=None,
# AllowInsecure=false ⇒ OptionsValidationException), so a node that carries no LDAP config crashes at boot.
# This anchor is merged into every host node's environment (via `<<: [*secrets-env, *ldap-env]`). Central
# still also lists the block inline for readability; the explicit keys win over the merge, same values.
x-ldap-env: &ldap-env
Security__Ldap__Enabled: "true"
Security__Ldap__DevStubMode: "false"
Security__Ldap__Server: "10.100.0.35"
Security__Ldap__Port: "3893"
Security__Ldap__Transport: "None"
Security__Ldap__AllowInsecure: "true"
Security__Ldap__SearchBase: "dc=zb,dc=local"
Security__Ldap__ServiceAccountDn: "cn=serviceaccount,dc=zb,dc=local"
Security__Ldap__ServiceAccountPassword: "serviceaccount123"
Security__Ldap__GroupToRole__ReadOnly: "ReadOnly"
Security__Ldap__GroupToRole__WriteOperate: "WriteOperate"
Security__Ldap__GroupToRole__WriteTune: "WriteTune"
Security__Ldap__GroupToRole__WriteConfigure: "WriteConfigure"
Security__Ldap__GroupToRole__AlarmAck: "AlarmAck"
# Readiness gate for a pair FOUNDER node (site-a-1 / site-b-1): passes once its Akka port 4053 is bound.
# The partner (site-a-2 / site-b-2) waits on this via `depends_on: { condition: service_healthy }` so the
# founder forms its 2-node mesh FIRST and the partner JOINS it — instead of both self-first seeds racing
# and each forming its own single-node cluster (the split brain the Phase 6 live gate observed: 250/250,
# both Primary). `service_started` is NOT enough — it fires the instant the container launches, long
# before Akka binds. The image ships bash but no nc/curl, so the probe uses bash's /dev/tcp. start_period
# gives the founder a head start to not just bind but FORM (a 1-node cluster forms within the same second
# as the bind), so the partner's InitJoin lands on an existing cluster and is ack'd.
x-akka-founder-healthcheck: &akka-founder-healthcheck
test: ["CMD-SHELL", "bash -c '</dev/tcp/localhost/4053' || exit 1"]
interval: 3s
timeout: 2s
retries: 20
start_period: 8s
services:
sql:
@@ -155,8 +234,9 @@ services:
# ── Central cluster (2-node fused admin+driver) ─────────────────────────────
# The only UI + deploy singleton; also the MAIN cluster's OPC UA publishers.
# central-1 seeds the single Akka mesh that every other node joins; central-2 is a seed too
# (listing itself first), so it can form the mesh alone if central-1 is down.
# Per-cluster mesh Phase 6: central-1 and central-2 form their OWN 2-node "MAIN" Akka mesh,
# NOT a shared mesh with the site nodes — each lists itself first as seed, so either can
# cold-start while the other is down, and the site meshes are never in this pair's seed list.
central-1: &otopcua-host
build:
@@ -188,8 +268,8 @@ services:
sql: { condition: service_healthy }
migrator: { condition: service_completed_successfully }
environment:
<<: *secrets-env
OTOPCUA_ROLES: "admin,driver"
<<: [*secrets-env, *ldap-env]
OTOPCUA_ROLES: "admin,driver,cluster-MAIN"
ASPNETCORE_URLS: "http://+:9000"
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
Cluster__Hostname: "0.0.0.0"
@@ -203,18 +283,22 @@ services:
ConfigServe__GrpcListenPort: "4055"
ConfigServe__ApiKey: "configserve-docker-dev-key"
ConfigSource__Mode: "Direct"
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream). DARK SWITCH — Telemetry:Mode is left
# unset here (defaults to "Dps"), so the node keeps publishing telemetry on the existing
# DistributedPubSub topic; the dedicated gRPC listener below binds regardless (the node "always
# hosts" per docs/Telemetry.md) so flipping the mode later is a config change, not a redeploy.
# Port 4056 checked free against every other port this node binds: Akka 4053, ConfigServe 4055,
# OPC UA 4840 (container-internal), HTTP 9000. TelemetryDial__ApiKey (below) is central's
# dial-side key and must equal every node's Telemetry:ApiKey (fail-closed interceptor).
# Per-cluster mesh Phase 5/6 (gRPC live-telemetry stream). central-1 carries a cluster-*
# role + admin, so SplitTopologyTransportValidator requires BOTH Telemetry:Mode=Grpc
# (it is also a driver node, publishing MAIN's own telemetry) AND TelemetryDial:Mode=Grpc
# (it is the admin-role dialer for every site node). The dedicated gRPC listener below binds
# regardless of mode (the node "always hosts" per docs/Telemetry.md). Port 4056 checked free
# against every other port this node binds: Akka 4053, ConfigServe 4055, OPC UA 4840
# (container-internal), HTTP 9000. TelemetryDial__ApiKey (below) is central's dial-side key
# and must equal every node's Telemetry:ApiKey (fail-closed interceptor).
Telemetry__GrpcListenPort: "4056"
Telemetry__Mode: "Grpc"
Telemetry__ApiKey: "telemetry-docker-dev-key"
TelemetryDial__Mode: "Grpc"
TelemetryDial__ApiKey: "telemetry-docker-dev-key"
# Both redundancy peers are seeds (#459 / ScadaBridge parity) so a restarted node can
# re-join the mesh via EITHER peer, not only central-1.
# re-join the MAIN mesh via EITHER peer — NOT via a site node; the site meshes are never in
# this pair's seed list (per-cluster mesh Phase 6).
#
# ORDER IS LOAD-BEARING (2026-07-22): each node lists ITSELF first. Akka runs
# FirstSeedNodeProcess — the only path that can form a NEW cluster when no peer answers
@@ -223,19 +307,25 @@ services:
# boot by AkkaClusterOptionsValidator; see docs/Redundancy.md.
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
# Per-cluster mesh Phase 6: cluster-MAIN marks this node as a member of the MAIN mesh
# (RoleParser.ClusterRolePrefix = "cluster-"), read by SplitTopologyTransportValidator to
# require ClusterClient/Grpc transports, and by ClusterRoleInfo to scope redundancy +
# ClusterNodeAddressReconcilerActor to ClusterId=MAIN.
Cluster__Roles__0: "admin"
Cluster__Roles__1: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
# and its comm actor registered with the receptionist either way. Flip the whole rig with
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
Cluster__Roles__2: "cluster-MAIN"
# Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6). Central no
# longer shares a gossip ring with the site meshes, so DPS (gossip-only) cannot reach them —
# ClusterClient is the only transport that can. SplitTopologyTransportValidator refuses to
# start any node carrying a cluster-* role that is left on Dps, so this is fixed, not an
# env-var-overridable default, on every node in this file.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
@@ -297,8 +387,8 @@ services:
central-1: { condition: service_started }
migrator: { condition: service_completed_successfully }
environment:
<<: *secrets-env
OTOPCUA_ROLES: "admin,driver"
<<: [*secrets-env, *ldap-env]
OTOPCUA_ROLES: "admin,driver,cluster-MAIN"
ASPNETCORE_URLS: "http://+:9000"
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
Cluster__Hostname: "0.0.0.0"
@@ -309,29 +399,33 @@ services:
ConfigServe__GrpcListenPort: "4055"
ConfigServe__ApiKey: "configserve-docker-dev-key"
ConfigSource__Mode: "Direct"
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream) — see central-1. DARK SWITCH: Mode
# stays unset (⇒ Dps); the dedicated :4056 listener binds regardless.
# Per-cluster mesh Phase 5/6 (gRPC live-telemetry stream) — see central-1. central-2 carries
# a cluster-* role + admin, so both Telemetry:Mode=Grpc and TelemetryDial:Mode=Grpc are
# required (SplitTopologyTransportValidator); the dedicated :4056 listener binds regardless.
Telemetry__GrpcListenPort: "4056"
Telemetry__Mode: "Grpc"
Telemetry__ApiKey: "telemetry-docker-dev-key"
TelemetryDial__Mode: "Grpc"
TelemetryDial__ApiKey: "telemetry-docker-dev-key"
# Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1. SELF FIRST:
# central-2 lists itself as seed-nodes[0], which is what lets it cold-start while central-1
# is down (it previously listed central-1 first and simply never came Up in that case).
# is down (it previously listed central-1 first and simply never came Up in that case). Per
# per-cluster mesh Phase 6, central-2's seed list contains ONLY the MAIN pair — no site node.
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-2:4053"
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-1:4053"
# Per-cluster mesh Phase 6: cluster-MAIN marks this node as a member of the MAIN mesh — see
# central-1's Cluster__Roles comment.
Cluster__Roles__0: "admin"
Cluster__Roles__1: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
# and its comm actor registered with the receptionist either way. Flip the whole rig with
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
Cluster__Roles__2: "cluster-MAIN"
# Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see central-1.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
@@ -379,18 +473,20 @@ services:
- otopcua-localdb-central-2:/app/data
# ── Site A cluster (2-node driver-only) ─────────────────────────────────────
# Driver-only members of the single mesh, scoped to SITE-A by ClusterId. No UI,
# no user auth — managed + deployed to by the central cluster over the mesh.
# All site nodes seed central-1.
# Per-cluster mesh Phase 6: site-a-1 and site-a-2 form their OWN 2-node "SITE-A" Akka mesh,
# separate from MAIN and SITE-B — self-first seeded WITHIN the pair, never off central. No UI,
# no user auth; central manages + deploys to this mesh over ClusterClient/gRPC/FetchAndCache,
# not gossip (central and site-a share no seed list, so they never form one ring).
site-a-1:
<<: *otopcua-host
healthcheck: *akka-founder-healthcheck
depends_on:
sql: { condition: service_healthy }
central-1: { condition: service_started }
migrator: { condition: service_completed_successfully }
environment:
OTOPCUA_ROLES: "driver"
OTOPCUA_ROLES: "driver,cluster-SITE-A"
# Per-cluster mesh Phase 4: driver-only site nodes hold NO ConfigDb connection string. Their
# config comes from central via ConfigSource:Mode=FetchAndCache (gRPC fetch → LocalDb cache), their
# scripted-alarm condition state lives in LocalDb, and central persists their deploy acks. Direct
@@ -406,32 +502,41 @@ services:
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
ConfigSource__ApiKey: "configserve-docker-dev-key"
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream) — see central-1/central-2. DARK
# SWITCH: Telemetry:Mode stays unset (⇒ Dps); the dedicated :4056 listener binds regardless
# (the node "always hosts" per docs/Telemetry.md). Port checked free against this node's
# other ports: Akka 4053, OPC UA 4840, HTTP 9000, LocalDb sync 9001 (below). ApiKey must
# equal central's TelemetryDial:ApiKey (fail-closed interceptor).
# Per-cluster mesh Phase 5/6 (gRPC live-telemetry stream) — see central-1/central-2. This
# node carries cluster-SITE-A + driver, so SplitTopologyTransportValidator requires
# Telemetry:Mode=Grpc (mandatory, not a dark switch, as of Phase 6). The dedicated :4056
# listener binds regardless (the node "always hosts" per docs/Telemetry.md). Port checked
# free against this node's other ports: Akka 4053, OPC UA 4840, HTTP 9000, LocalDb sync 9001
# (below). ApiKey must equal central's TelemetryDial:ApiKey (fail-closed interceptor).
Telemetry__GrpcListenPort: "4056"
Telemetry__Mode: "Grpc"
Telemetry__ApiKey: "telemetry-docker-dev-key"
# Site nodes are deliberately NOT seeds — they join the central pair's mesh. The self-first
# seed rule is conditional for exactly this reason (it binds only when a node's own address
# is in its own seed list), so these configs are exempt rather than broken.
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
# Per-cluster mesh Phase 6: site-a-1 seeds its OWN "SITE-A" mesh (itself first, site-a-2
# second) — it no longer seeds off central-1. Central is reached over ClusterClient/gRPC/
# FetchAndCache, never gossip; central and site-a share no seed list.
Cluster__SeedNodes__0: "akka.tcp://otopcua@site-a-1:4053"
Cluster__SeedNodes__1: "akka.tcp://otopcua@site-a-2:4053"
# Simultaneous-cold-start split-brain guard (SITE-A enablement demo). With the pair no longer
# startup-serialized, this is what stops both self-first seeds forming their own 1-node cluster:
# the lower-address node (site-a-1) founds immediately; the higher (site-a-2) probes site-a-1 and
# joins it when reachable, or forms alone only if site-a-1 is truly down. Akka gets NO config
# seeds when this is on (BuildClusterOptions); ClusterBootstrapCoordinator issues JoinSeedNodes.
Cluster__BootstrapGuard__Enabled: "true"
Cluster__Roles__0: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
# and its comm actor registered with the receptionist either way. Flip the whole rig with
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
Cluster__Roles__1: "cluster-SITE-A"
# Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see
# central-1/central-2. Central dials this mesh's ClusterClientReceptionist via the contact
# points below; there is no gossip path anymore.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
<<: *secrets-env
<<: [*secrets-env, *ldap-env]
# Quiet EF/AspNetCore SQL flood — see central-1 (Serilog override). mem_limit/
# mem_reservation are inherited from the *otopcua-host anchor.
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
@@ -489,8 +594,13 @@ services:
sql: { condition: service_healthy }
central-1: { condition: service_started }
migrator: { condition: service_completed_successfully }
# SITE-A is the BOOTSTRAP-GUARD enablement demo: NO startup serialization on the pair — both
# nodes start simultaneously (the exact race that split-brained in the Phase 6 gate). The
# Cluster:BootstrapGuard (enabled in both site-a env blocks) is the ONLY thing preventing the
# split: it makes the lower-address node the founder and the higher-address node probe-then-join.
# site-b keeps the depends_on:service_healthy serialization + guard OFF, as the A/B control.
environment:
OTOPCUA_ROLES: "driver"
OTOPCUA_ROLES: "driver,cluster-SITE-A"
# Per-cluster mesh Phase 4: no ConfigDb string on driver-only site nodes (see site-a-1).
Cluster__Hostname: "0.0.0.0"
Cluster__Port: "4053"
@@ -501,26 +611,33 @@ services:
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
ConfigSource__ApiKey: "configserve-docker-dev-key"
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream) — see central-1/site-a-1. DARK
# SWITCH: Telemetry:Mode stays unset (⇒ Dps); the dedicated :4056 listener binds regardless.
# Per-cluster mesh Phase 5/6 (gRPC live-telemetry stream) — see central-1/site-a-1.
# Telemetry:Mode=Grpc is mandatory as of Phase 6 (cluster-SITE-A + driver); the dedicated
# :4056 listener binds regardless.
Telemetry__GrpcListenPort: "4056"
Telemetry__Mode: "Grpc"
Telemetry__ApiKey: "telemetry-docker-dev-key"
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
# Per-cluster mesh Phase 6: site-a-2 seeds its OWN "SITE-A" mesh (itself first, site-a-1
# second) — it no longer seeds off central-1.
Cluster__SeedNodes__0: "akka.tcp://otopcua@site-a-2:4053"
Cluster__SeedNodes__1: "akka.tcp://otopcua@site-a-1:4053"
# Bootstrap-guard enablement demo — see site-a-1. site-a-2 is the HIGHER address, so the guard
# makes it probe site-a-1 and join it (peer-first) rather than race-form its own cluster.
Cluster__BootstrapGuard__Enabled: "true"
Cluster__Roles__0: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
# and its comm actor registered with the receptionist either way. Flip the whole rig with
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
Cluster__Roles__1: "cluster-SITE-A"
# Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see
# central-1/site-a-1.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
<<: *secrets-env
<<: [*secrets-env, *ldap-env]
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
@@ -563,15 +680,18 @@ services:
- otopcua-localdb-site-a-2:/app/data
# ── Site B cluster (2-node driver-only) ─────────────────────────────────────
# Per-cluster mesh Phase 6: site-b-1 and site-b-2 form their OWN 2-node "SITE-B" Akka mesh,
# separate from MAIN and SITE-A — same shape as site-a-1/site-a-2, see that pair's header.
site-b-1:
<<: *otopcua-host
healthcheck: *akka-founder-healthcheck
depends_on:
sql: { condition: service_healthy }
central-1: { condition: service_started }
migrator: { condition: service_completed_successfully }
environment:
OTOPCUA_ROLES: "driver"
OTOPCUA_ROLES: "driver,cluster-SITE-B"
# Per-cluster mesh Phase 4: no ConfigDb string on driver-only site nodes (see site-a-1).
Cluster__Hostname: "0.0.0.0"
Cluster__Port: "4053"
@@ -582,26 +702,30 @@ services:
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
ConfigSource__ApiKey: "configserve-docker-dev-key"
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream) — see central-1/site-a-1. DARK
# SWITCH: Telemetry:Mode stays unset (⇒ Dps); the dedicated :4056 listener binds regardless.
# Per-cluster mesh Phase 5/6 (gRPC live-telemetry stream) — see central-1/site-a-1.
# Telemetry:Mode=Grpc is mandatory as of Phase 6 (cluster-SITE-B + driver); the dedicated
# :4056 listener binds regardless.
Telemetry__GrpcListenPort: "4056"
Telemetry__Mode: "Grpc"
Telemetry__ApiKey: "telemetry-docker-dev-key"
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
# Per-cluster mesh Phase 6: site-b-1 seeds its OWN "SITE-B" mesh (itself first, site-b-2
# second) — it no longer seeds off central-1.
Cluster__SeedNodes__0: "akka.tcp://otopcua@site-b-1:4053"
Cluster__SeedNodes__1: "akka.tcp://otopcua@site-b-2:4053"
Cluster__Roles__0: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
# and its comm actor registered with the receptionist either way. Flip the whole rig with
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
Cluster__Roles__1: "cluster-SITE-B"
# Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see
# central-1/site-a-1.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
<<: *secrets-env
<<: [*secrets-env, *ldap-env]
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
@@ -634,8 +758,12 @@ services:
sql: { condition: service_healthy }
central-1: { condition: service_started }
migrator: { condition: service_completed_successfully }
# Serialize the SITE-B pair's startup so site-b-1 forms the mesh first and site-b-2 JOINS it — see
# the identical comment on site-a-2. Without this both self-first seeds form their own single-node
# cluster on simultaneous start (split brain).
site-b-1: { condition: service_healthy }
environment:
OTOPCUA_ROLES: "driver"
OTOPCUA_ROLES: "driver,cluster-SITE-B"
# Per-cluster mesh Phase 4: no ConfigDb string on driver-only site nodes (see site-a-1).
Cluster__Hostname: "0.0.0.0"
Cluster__Port: "4053"
@@ -646,26 +774,30 @@ services:
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
ConfigSource__ApiKey: "configserve-docker-dev-key"
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream) — see central-1/site-a-1. DARK
# SWITCH: Telemetry:Mode stays unset (⇒ Dps); the dedicated :4056 listener binds regardless.
# Per-cluster mesh Phase 5/6 (gRPC live-telemetry stream) — see central-1/site-a-1.
# Telemetry:Mode=Grpc is mandatory as of Phase 6 (cluster-SITE-B + driver); the dedicated
# :4056 listener binds regardless.
Telemetry__GrpcListenPort: "4056"
Telemetry__Mode: "Grpc"
Telemetry__ApiKey: "telemetry-docker-dev-key"
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
# Per-cluster mesh Phase 6: site-b-2 seeds its OWN "SITE-B" mesh (itself first, site-b-1
# second) — it no longer seeds off central-1.
Cluster__SeedNodes__0: "akka.tcp://otopcua@site-b-2:4053"
Cluster__SeedNodes__1: "akka.tcp://otopcua@site-b-1:4053"
Cluster__Roles__0: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
# and its comm actor registered with the receptionist either way. Flip the whole rig with
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
Cluster__Roles__1: "cluster-SITE-B"
# Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see
# central-1/site-a-1.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
<<: *secrets-env
<<: [*secrets-env, *ldap-env]
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+29 -2
View File
@@ -103,13 +103,40 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
| `Hostname` | string | `0.0.0.0` | Bind hostname. |
| `Port` | int | `4053` | Cluster transport port. **Duplicated as `ClusterNode.AkkaPort` in the Config DB** — see the note below. |
| `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. **Duplicated as `ClusterNode.Host`** — see the note below. |
| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). |
| `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES`. Allowed values: `admin`, `driver`, `dev`. |
| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). **Per-pair since Phase 6:** each application `Cluster`'s pair is its own Akka mesh, so a node's `SeedNodes` lists only itself (first) and its own pair partner (second) — never a node from another cluster's pair. |
| `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES`. Allowed values: `admin`, `driver`, `dev`, plus a per-cluster `cluster-{ClusterId}` role (e.g. `cluster-MAIN`, `cluster-SITE-A`) — see [Per-cluster mesh split (Phase 6)](#per-cluster-mesh-split-phase-6-cluster-clusterid-roles--splittopologytransportvalidator) below. |
> The full redundancy model (ServiceLevel tiers, split-brain, peer discovery) is in [`Redundancy.md`](Redundancy.md). The OPC UA peer-URI advertising lives in the `OpcUa:PeerApplicationUris` key above.
> **`Port` / `PublicHostname` are stored twice.** The node binds from these keys; the fleet's `ClusterNode` row records the same address as `AkkaPort` / `Host` so that **central can dial the node without sharing a gossip ring with it** — which is what [per-cluster mesh Phase 2](plans/2026-07-21-per-cluster-mesh-design.md) needs. Nothing in the schema makes the two agree, so `ClusterNodeAddressReconcilerActor` (admin-role singleton) compares them against live membership and logs an Error on mismatch. If you change `Cluster:Port` or `Cluster:PublicHostname` on a node, **update its `ClusterNode` row too** — a node binding 4054 with a row saying 4053 still gossips fine today, and fails silently in Phase 2. The row's `NodeId` is `host:port` and is also the deploy path's ack identity, so a drifted node's deployment acks stop matching as well. See [`config-db-schema.md` § `ClusterNode`](v2/config-db-schema.md#clusternode).
### Per-cluster mesh split (Phase 6): `cluster-{ClusterId}` roles + `SplitTopologyTransportValidator`
- **Purpose:** [per-cluster mesh](plans/2026-07-22-per-cluster-mesh-program.md) **Phase 6** split the
single fleet-wide Akka mesh into one independent 2-node mesh per application `Cluster`. A node's
`Cluster:Roles` now carries a `cluster-{ClusterId}` role alongside `driver` (and `admin` for a fused
central node) — e.g. `driver,cluster-SITE-A` — and its `Cluster:SeedNodes` lists only itself and its
own pair partner (see the `Roles`/`SeedNodes` rows above). Nodes in different clusters never gossip
with each other.
- **`SplitTopologyTransportValidator`** (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`, wired at
`ValidateOnStart` alongside the other `Cluster`-family validators) fails host start on any node
carrying a `cluster-{ClusterId}` role unless it is also configured for the split-safe transports:
`MeshTransport:Mode = ClusterClient`, `Telemetry:Mode = Grpc` on a driver node, `TelemetryDial:Mode =
Grpc` on an admin node. A DPS-mode transport on a cluster-role node would try to gossip across a
partition boundary that no longer exists, so the validator fails closed rather than let commands or
telemetry silently vanish. A legacy node with no `cluster-{ClusterId}` role is exempt and may stay on
`Dps`.
- **The compiled transport-mode defaults were deliberately NOT flipped.** `MeshTransport:Mode`,
`Telemetry:Mode`, and `TelemetryDial:Mode` (documented below) all still default to `Dps` — a
blast-radius assessment found that flipping the compiled default would break every integration test
and every existing `appsettings.json` profile that leaves these keys unset. `SplitTopologyTransportValidator`
plus explicit per-deployment configuration is the guardrail, not a default flip; every node that
actually carries a `cluster-{ClusterId}` role must set the split-safe modes explicitly.
- See [`Redundancy.md` § Per-cluster meshes (Phase 6)](Redundancy.md#per-cluster-meshes-phase-6) for the
full runtime picture (one Primary per pair, the cluster-scoped `RedundancyStateActor` singleton,
`ClusterNodeAddressReconciler` own-cluster-scoping, and central's one-`ClusterClient`-per-`Cluster`
fan-out).
### `MeshTransport` (central↔node command transport)
- **Purpose:** selects how central's three command channels (`deployments`, `driver-control`, `alarm-commands`) and the node's `deployment-acks` reply reach the other side — over the mesh-wide DistributedPubSub, or over an Akka `ClusterClient` that does **not** require central and the node to share a gossip ring. The second option is what [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) Phase 6 needs once the meshes split.
+116 -30
View File
@@ -108,22 +108,59 @@ plane (writes, alarm acks, the alerts emit, the alarm-history drain) on the wron
*higher* address so the two orderings genuinely disagree. `NodeRedundancyState.IsRoleLeaderForDriver`
was renamed `IsDriverPrimary` to stop the name asserting the old derivation.
> **KNOWN LIMITATION — the election is per *Akka* cluster, not per application `Cluster`.**
> The election yields exactly **one** Primary across the whole Akka cluster. A fleet that
> runs several application clusters (each with its own redundant pair) inside one Akka cluster —
> which is what `docker-dev/docker-compose.yml` models, with MAIN, SITE-A and SITE-B — therefore has
> one Primary among *all* its driver nodes, not one per pair. Every consumer of the role inherits
> this: `ServiceLevel`, the `alerts` emit gate, and the inbound device-write gate below.
> **RESOLVED by Phase 6 — the election is now per application `Cluster`, not per Akka mesh.**
> Phase 6 of the per-cluster mesh program split the single fleet-wide Akka mesh into one independent
> 2-node mesh per application `Cluster` (see [Per-cluster meshes (Phase 6)](#per-cluster-meshes-phase-6)
> below). `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton spawned on every driver
> node, so each pair elects its own Primary — the whole-Akka-cluster mismatch this block used to describe
> no longer exists. See `docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`.
>
> The unit of redundancy everywhere else in the product is the application `Cluster` (`ClusterId`) —
> it owns drivers, devices and `ClusterNode` rows, and the LocalDb deployment-artifact cache is
> already keyed by it *so that a pair shares one entry*. Scoping the election the same way is the
> fix; `RedundancyStateActor` is an admin-role singleton in the same assembly as
> `AdminOperationsActor`, which already reads and groups `ClusterNodes` by cluster.
>
> Surfaced by the LocalDb Phase 2 live gate, where it stopped the alarm-history drain on every node
> in the fleet — see `docs/plans/2026-07-20-localdb-phase2-live-gate.md`. The alarm drain no longer
> depends on this (it defers only to a node that shares its queue), but the other three gates still do.
> Originally surfaced by the LocalDb Phase 2 live gate, where the old whole-mesh election stopped the
> alarm-history drain on every node in the fleet — see `docs/plans/2026-07-20-localdb-phase2-live-gate.md`.
## Per-cluster meshes (Phase 6)
The fleet no longer runs one Akka mesh. Phase 6 split it into **three independent 2-node meshes** — the
central pair (roles `admin,driver,cluster-MAIN`), the site-a pair (`driver,cluster-SITE-A`), and the
site-b pair (`driver,cluster-SITE-B`) — all sharing the same `ActorSystem` name (`otopcua`) on one network,
separated purely by **per-pair self-first seed partitioning**: each node's `Cluster:SeedNodes` lists only
itself and its own pair partner, so gossip never crosses a pair boundary (see
[Bootstrap: self-first seed ordering](#bootstrap-self-first-seed-ordering)).
Consequences:
- **One Primary per pair — two or more Primaries fleet-wide, by design.** Each mesh runs its own
election, so a three-mesh fleet has three Primaries, one per application `Cluster`. This is correct,
not a regression of the old single-Primary assumption.
- **`RedundancyStateActor` is a `cluster-{ClusterId}`-scoped singleton**, spawned on every driver node
(falling back to the plain `driver` role for a legacy node not carrying a cluster role). Each pair
elects and publishes its own `redundancy-state` on its own mesh's DPS — the DPS topic name is
unchanged, but it no longer reaches past the pair.
- **`ClusterNodeAddressReconciler` is own-cluster-scoped** — an admin node can only reconcile
`ClusterNode` rows belonging to its own cluster; it has no gossip visibility into another mesh to
reconcile against.
- **`CentralCommunicationActor` creates one `ClusterClient` per application `Cluster`** and fans
commands across them, rather than one fleet-wide client. Central reaches a site mesh over
`ClusterClient` (commands), gRPC (telemetry — see `docs/Telemetry.md`), and `ConfigServe` (config
fetch — see [`ConfigSource`/`ConfigServe`](../CLAUDE.md#config-source-configsource--configserve)),
never over gossip.
- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}`
role unless it is also configured for the split-safe transports: `MeshTransport:Mode = ClusterClient`,
`Telemetry:Mode = Grpc` (driver nodes), `TelemetryDial:Mode = Grpc` (admin nodes). A DPS-mode transport
on a cluster-role node would silently try to gossip across a partition that no longer exists, so the
validator fails closed instead. Legacy/non-cluster-role nodes are exempt.
- **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`,
`Telemetry:Mode`, and `TelemetryDial:Mode` all still default to `Dps`. A blast-radius assessment
found flipping the compiled default would break every integration test and every existing
`appsettings.json` profile that doesn't explicitly set these keys. The guardrail is the validator
above plus explicit per-deployment configuration, not a default flip.
### Secrets replication is pair-local
Akka DPS secret replication is scoped to each 2-node mesh: a pair replicates its own secrets to its
own partner and no further. There is no fleet-wide or cross-cluster secret sync after the split — by
design, consistent with Phase 4 (site nodes have no SQL, so there is no SQL-backed hub to fan a
cross-mesh sync through).
## Data flow
@@ -316,18 +353,17 @@ advertises `ServiceLevel` 250. OPC UA clients re-select.
| Confirmation | A dialog naming the node, and what follows (restart, ServiceLevel 250 moves, clients re-select). |
| Audit | Written through the shared `ZB.MOM.WW.Audit.IAuditWriter` seam **before** the Leave — action `cluster.manual-failover`, actor, target. A *refused* failover changes nothing and writes nothing. |
> **Mesh-scope caveat.** Until the per-cluster mesh work lands
> (`docs/plans/2026-07-21-per-cluster-mesh-design.md`), the Primary is elected once per Akka mesh rather than
> per application `Cluster` row — so on a fleet running several clusters in one mesh this button acts on the
> whole mesh's Primary, which may belong to a different cluster than the page you are on. The panel says so.
> Phase 6 of the mesh program removes both the caveat and this notice.
Phase 6 of the mesh program (see [Per-cluster meshes (Phase 6)](#per-cluster-meshes-phase-6)) split
the fleet into one 2-node mesh per application `Cluster`, so the button now acts on **this pair's own**
Primary — there is no longer a whole-mesh vs. per-cluster ambiguity to caveat.
> **Live gate outstanding.** The auto-down change is verified by unit tests against the effective
> configuration, and by the sister project's live drill on an equivalent Akka version and topology. It has
> **not** yet been drilled on an OtOpcUa two-node rig — the docker-dev rig runs a single six-node mesh, where
> the 1-vs-1 pathology cannot occur. The drill (kill the oldest container of a genuine two-node cluster;
> assert the survivor stays up, takes the singletons, and reports the primary `ServiceLevel`) should run
> alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See
> configuration, and by the sister project's live drill on an equivalent Akka version and topology. It
> was **not** drillable on an OtOpcUa rig before Phase 6, because the docker-dev rig ran a single
> six-node mesh where the 1-vs-1 pathology cannot occur. **Phase 6 has landed and every mesh is now
> exactly two nodes**, so the drill (kill the oldest container of a genuine two-node cluster; assert
> the survivor stays up, takes the singletons, and reports the primary `ServiceLevel`) is finally
> testable — deferred to Phase 7's failover drill. See
> `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a.
## Bootstrap: self-first seed ordering
@@ -365,11 +401,61 @@ Comparing against `Cluster:Hostname` would be worse than wrong: in docker-dev it
no seed URI anywhere, so the rule would silently exempt the entire rig.
Sequential recovery is island-free by construction: a `FirstSeedNodeProcess` node self-joins only when **no**
seed answered, so a peer booting after the survivor formed simply joins it as the youngest member. Both
nodes cold-starting *simultaneously* converge on one cluster while they are mutually reachable — the
`InitJoin` handshake resolves it before either self-join deadline expires. The residual risk is a boot-time
**partition** (both cold-starting, mutually unreachable): both form, which is the same dual-active class the
`auto-down` strategy already accepts, with the same recovery (restart one side).
seed answered, so a peer booting after the survivor formed simply joins it as the youngest member.
**Simultaneous cold-start CAN split — this is the residual gap the bootstrap guard below closes.** When
BOTH nodes start from nothing at the same instant, each runs `FirstSeedNodeProcess`, and if neither has
formed before the other's `seed-node-timeout` expires, EACH self-joins and forms its own single-node
cluster — two Primaries in one pair. On in-process loopback the `InitJoin` handshake usually resolves fast
enough to converge, but the Phase 6 live gate reproduced the split reliably on the docker-dev rig (real
container start timing + DNS): both nodes logged `JOINING itself … forming a new cluster` and both advertised
ServiceLevel 250. Two independent clusters do **not** auto-merge, so the split persists until an operator
restarts one side.
### Bootstrap guard (`Cluster:BootstrapGuard`, opt-in)
The guard eliminates the simultaneous-start split without giving up cold-start-alone. It is a **dark switch,
default off** — a node with `Cluster:BootstrapGuard:Enabled=false` (the default) keeps Akka's config-driven
self-first auto-join exactly as above.
When enabled, the node is given **no config seed nodes** (so Akka does not auto-join), and
`ClusterBootstrapCoordinator` picks the join order from a deterministic address tie-break plus a reachability
probe:
- The node with the lexicographically **lower** canonical `host:port` is the **preferred founder**: it joins
self-first and forms immediately if no peer answers — no probe, no delay.
- The **higher** node probes its partner's Akka port (a plain TCP connect; the partner binds its port well
before it forms) for up to `PartnerProbeSeconds` (default 25 s): **reachable ⇒ peer-first** (it joins the
founder, never races it); **unreachable after the window ⇒ self-first** (the partner is genuinely down, so
it forms alone — cold-start-alone preserved for the higher node too).
The order is decided **before** issuing a single `Cluster.JoinSeedNodes`, from an explicit reachability
signal — it never re-decides mid-handshake, the failure mode that retired the `SelfFormAfter` watchdog.
**Residual trade-off (accepted, operator-visible):** once the higher node commits peer-first it cannot
self-form (Akka's `JoinSeedNodeProcess` retries forever). If the founder dies in the small probe→join window,
the higher node hangs unjoined; the coordinator logs a clear WARNING after a bounded grace, and a **restart
recovers it** (the guard re-runs, finds the founder down, and forms alone). Config keys:
| Key | Default | Notes |
|---|---|---|
| `Cluster:BootstrapGuard:Enabled` | `false` | Dark switch. Only meaningful on a node that is one of its own two pair seeds; inert elsewhere. |
| `Cluster:BootstrapGuard:PartnerProbeSeconds` | `25` | Higher node's probe window. Must comfortably exceed the founder's process-start-to-Akka-bind time, or a slow-but-alive founder is mistaken for dead and the split re-opens. Validated `> 0` at boot. |
| `Cluster:BootstrapGuard:PartnerProbeIntervalMs` | `500` | Interval between probes. |
| `Cluster:BootstrapGuard:ProbeConnectTimeoutMs` | `1000` | Per-probe TCP connect timeout. |
On the docker-dev rig the **site-a pair is the enablement demo** (guard on, startup serialization removed so
both nodes cold-start simultaneously); **site-b keeps the `depends_on: service_healthy` startup serialization
with the guard off**, as the A/B control. Either mechanism prevents the split; the guard is the one that also
works on production hardware, where compose `depends_on` does not exist.
The alternative to the guard is **operational**: stagger the two VMs' service-manager start, or bring the
designated founder VM up first — see the Phase 7 co-located operator runbook
(`docs/plans/2026-07-24-mesh-phase7-failover-drills.md`).
The residual risk that remains even with the guard is a boot-time **partition** (both cold-starting, mutually
unreachable from the start): both form, which is the same dual-active class the `auto-down` strategy already
accepts, with the same recovery (restart one side).
Pinned by `SelfFirstSeedBootstrapTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`), which starts real
in-process clusters through the production bootstrap at production failure-detection timings: a lone
@@ -26,7 +26,9 @@ OpcUaClient, Historian.Gateway (`src/Drivers/`).
## 2. Document map
**Program:** this file.
**Program:** this file (architecture / shared contract / build order).
**Progress tracker (Waves 02):** [`2026-07-24-driver-expansion-tracking.md`](2026-07-24-driver-expansion-tracking.md)
— per-deliverable status + links to each design doc and implementation plan.
**Research reports** (`docs/research/drivers/`, index: [`README.md`](../research/drivers/README.md)):
- [`mtconnect-agent.md`](../research/drivers/mtconnect-agent.md) ·
@@ -312,7 +312,7 @@ Deliberately not a task plan — per-phase plans follow, one at a time.
| 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No |
| 4 | Cut the driver-side ConfigDb connection: re-home `EfAlarmConditionStateStore`, resolve the `DbHealthProbeActor` ServiceLevel input, audit `OpcUaPublishActor` | No |
| 5 | gRPC stream contract; migrate the seven observability topics | No |
| 6 | Mesh partition: per-cluster seed nodes, cluster-scoped roles + singletons; rig models the real topology | No |
| 6 | Mesh partition: per-cluster seed nodes, cluster-scoped roles + singletons; rig models the real topology | No**DONE 2026-07-24**, see `2026-07-22-per-cluster-mesh-program.md` §Phase 6 and `2026-07-24-mesh-phase6-partition-and-colocation.md` |
| 7 | Failover drill (both directions, per §6.2); live gate | No |
Phases 3 and 4 are the ones that change a running system's data path rather than its wiring, and each
@@ -263,25 +263,45 @@ kill-and-reconnect of a site node's dialer recovers every stream (driver-health/
from the node hub's snapshot replay). Treat Phase 5 as **code-complete, not verified**, until this
lands — see `docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` Task 12.
### Phase 6 — Mesh partition + co-location topology
### Phase 6 — Mesh partition + co-location topology — **DONE 2026-07-24**
**Scope:** per-cluster seed nodes — ~~adopt ScadaBridge's self-first ordering and RETIRE the
`SelfFormAfter` watchdog + TCP reachability guard~~ **DONE EARLY 2026-07-22** (docker-dev
`central-2` swapped to self-first, `ClusterBootstrapFallback`/`SelfFormAfter` deleted,
`AkkaClusterOptionsValidator` ports ScadaBridge's startup rule in its conditional form, and
`SelfFirstSeedBootstrapTests` replaces `SelfFormBootstrapTests`; see `docs/Redundancy.md`
§"Bootstrap: self-first seed ordering"). What remains for this phase is the per-pair
`Cluster__SeedNodes__*` matrix once the meshes actually split — every node in a pair is then a seed
of its own mesh, so the site-node exemption disappears; cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping,
central pair keeps the admin singletons; **docker-dev rig rewritten** to model the real topology —
including the co-location port table above (both products' compose files on shared per-site
networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the
election is pair-local now) and the fallback's site-node island-guard docs note (moot — every
node is a seed of its own mesh); `Cluster__SeedNodes__*` env matrix per pair.
**Exit gate:** rig up in the new shape; every existing live-gated behavior re-verified per pair
(deploy, redundancy 250/240 per pair — **two Primaries fleet-wide, one per pair, by design**);
secrets Akka replication re-verified or re-scoped (it rides DPS on the current single mesh — its
topology must be re-decided here, likely SQL-hub mode like ScadaBridge, since pub/sub cannot
cross separate meshes).
§"Bootstrap: self-first seed ordering"). The rest of the phase — actually splitting the single
fleet-wide mesh into three independent 2-node meshes (central `cluster-MAIN`, site-a
`cluster-SITE-A`, site-b `cluster-SITE-B`) — **shipped 2026-07-24**. See
`docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md` for the full plan and task list.
Delivered:
- Per-pair `Cluster:SeedNodes` (each node lists itself first, its pair partner second — no
cross-pair seeds) + `cluster-{ClusterId}` roles on every driver node + singleton re-scoping:
`RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton spawned on every driver
node (falls back to plain `driver` for legacy nodes), so **each pair elects its own Primary —
two or more Primaries fleet-wide, by design**; `ClusterNodeAddressReconciler` is own-cluster-scoped.
- `CentralCommunicationActor` now creates one `ClusterClient` per application `Cluster` and fans
commands across them (closes the `TODO(Phase 6)` left by Phase 2).
- New `SplitTopologyTransportValidator` fails host start on a `cluster-{ClusterId}`-role node left
on a DPS transport (`MeshTransport:Mode` must be `ClusterClient`; `Telemetry:Mode`/`TelemetryDial:Mode`
must be `Grpc`) — see `docs/Configuration.md` §"Per-cluster mesh split (Phase 6)".
- **Three decisions worth recording:**
1. **Secrets replication went pair-local, not SQL-hub.** The design flagged "likely SQL-hub mode
like ScadaBridge" as the fallback if DPS couldn't cross the split meshes — in the event, each
pair simply keeps replicating its own secrets within itself (no cross-cluster sync needed, and
no SQL-hub was built, consistent with Phase 4: site nodes have no SQL to hub through).
2. **The compiled transport-mode defaults (`MeshTransport:Mode`, `Telemetry:Mode`,
`TelemetryDial:Mode`) were deliberately left at `Dps`**, not flipped to the split-safe modes —
a blast-radius assessment found flipping the default breaks every integration test and every
appsettings profile that leaves the key unset. The guardrail is
`SplitTopologyTransportValidator` plus explicit per-deployment config, not a default flip.
3. **The docker-dev rig models three OtOpcUa-only meshes** (central/site-a/site-b), not the
ScadaBridge co-location topology sketched in the design's port table — co-location with
ScadaBridge's own site nodes stayed out of scope for this phase.
**Exit gate:** rig up in the new three-mesh shape; every existing live-gated behavior re-verified
per pair (deploy, redundancy 250/240 per pair, two-Primaries-fleet-wide confirmed); the
ClusterRedundancy page's mesh-scope caveat and `IManualFailoverService`'s doc-comment caveat
removed (the election is pair-local now); `docs/Redundancy.md`'s per-Akka-cluster KNOWN LIMITATION
marked resolved.
### Phase 7 — Failover drill + live gates
**Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both
@@ -318,5 +338,5 @@ resource sizing on the site VMs should be checked once both products run the ful
| 3 fetch-and-cache | **DONE 2026-07-23**, merged `d01b0695`, live gate PASSED, pushed to origin (scadaproj umbrella index updated + pushed `b5e7bc8`). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. |
| 4 cut driver ConfigDb | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase4` (Tasks 08 + 1b + 1011; Task 9 table-drop deferred). ConfigDb admin-only; driver-only ⇒ FetchAndCache (validator); `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central SQL down — proven live); alarm condition state in replicated LocalDb `alarm_condition_state`; central persists acks; `OpcUaPublish` guard split fixes the driver-only address-space wipe; driver-only LDAP maps from appsettings only (6th consumer found mid-phase). Gate: deploy sealed green w/ 4 DB-less site nodes acking, ServiceLevel held 240 w/ SQL down, restart booted last-known-good from the LocalDb pointer. See `2026-07-23-mesh-phase4-cut-driver-configdb.md` + `2026-07-23-mesh-phase4-live-gate.md`. |
| 5 gRPC telemetry | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase5`. 4-channel scope (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`), 3 deferred with rationale (`redundancy-state`, `fleet-status`, `deployment-acks`); dark switch `Telemetry:Mode`/`TelemetryDial:Mode` (Dps default); node hosts server / central dials; auth-from-day-one fail-closed bearer key, superseding design §6.3. Gate: full 12-stream mesh formed in Grpc mode (2 inbound per driver node, 6 outbound per central), AdminUI pill live (data path proven), kill-and-reconnect of a node recovered its stream in ~5s; Dps baseline dials nothing. Surfaced an upgrade gotcha — `ClusterNode.GrpcPort` must be populated on existing deployments (fresh installs seed it; nullable column doesn't backfill) — which live-validated the graceful null-`GrpcPort` skip. See `docs/Telemetry.md`, `2026-07-23-mesh-phase5-grpc-telemetry-stream.md` + `2026-07-23-mesh-phase5-live-gate.md`. |
| 6 mesh partition + co-location | not started |
| 7 drill + live gates | not started |
| 6 mesh partition + co-location | **DONE 2026-07-24** — three independent 2-node meshes (central/site-a/site-b), `cluster-{ClusterId}` roles, per-pair self-first seeds, cluster-scoped `RedundancyStateActor` singleton (two+ Primaries fleet-wide, by design), one `ClusterClient` per `Cluster`, `SplitTopologyTransportValidator`, pair-local secrets replication, compiled transport defaults intentionally kept `Dps`. See `2026-07-24-mesh-phase6-partition-and-colocation.md`. |
| 7 drill + live gates | **DONE 2026-07-24 — all 6 drills PASSED** on the docker-dev three-mesh rig. D1 MAIN graceful manual-failover via the AdminUI button (Primary leaves, peer→250, restarts+rejoins no split); D2 SITE-A auto-down failover both directions; D3 SITE-B crash-the-oldest; D4 **auto-down 1-vs-1 survival (closes the gate deferred since Phase 0a)** — every survivor stayed Up; D5 **self-first cold-start-alone (gate b)** — a node boots alone and forms its mesh; D6 recovery/rejoin. Manual-failover button confirmed MAIN-only by construction (site pairs are driver-only, fail over via auto-down). One-page co-located operator runbook shipped. Carried Phase-6 finding: simultaneous cold-start of both pair VMs can split — operational mitigation (staggered start) in the runbook; product-level guard is a candidate follow-up, not shipped. See `2026-07-24-mesh-phase7-failover-drills.md`. **Per-cluster mesh program COMPLETE.** |
@@ -0,0 +1,170 @@
# Driver-expansion — Waves 02 tracking
> **Living status tracker** for the driver-expansion program (Waves 0, 1, 2). One row per
> deliverable, each pointing at its **design doc**, its **implementation plan** (once written),
> and its current **status**. The authoritative *architecture* index is the program design doc;
> this file is the authoritative *progress* index.
>
> **Program design (architecture / shared contract / build order):**
> [`2026-07-15-driver-expansion-program-design.md`](2026-07-15-driver-expansion-program-design.md)
>
> Last updated: 2026-07-24.
## Legend
| Status | Meaning |
|---|---|
| ✅ **Done** | Merged to master, tests green. |
| 🟡 **Live gate open** | Code merged; a live `/run` or hardware-gated verification still outstanding. |
| 📝 **Plan ready** | Executable implementation plan (`*-implementation.md` + `.tasks.json`) written; not yet built. |
| 📐 **Design only** | Design doc exists; **no** implementation plan yet — writing-plans is the next step. |
| ⛔ **Not started** | No design, no plan. |
**Doc types** (per the writing-plans skill): a *design* states architecture, decisions, and risks;
an *implementation plan* is the bite-sized, TDD, file-path-level task list the subagent-driven
executor runs off, with a co-located `.tasks.json` for resume. A deliverable is only buildable once
it reaches 📝.
## Summary
| Wave | Deliverable | Design | Impl. plan | Status | Effort | Fixture / hardware |
|---|---|---|---|---|---|---|
| **0** | Universal Discover-backed browser | [design](2026-07-15-universal-discovery-browser-design.md) | [plan](2026-07-15-universal-discovery-browser-implementation.md) · [tasks](2026-07-15-universal-discovery-browser-implementation.md.tasks.json) | 🟡 **Live gate open** | SM | none (retrofits shipped drivers) |
| **1** | Modbus RTU (extend Modbus) | [design](2026-07-15-modbus-rtu-driver-design.md) | [plan](2026-07-24-modbus-rtu-driver.md) · [tasks](2026-07-24-modbus-rtu-driver.md.tasks.json) | 🟢 **Built — live gate PASSED** (11 tasks, branch `feat/modbus-rtu-driver`, pending merge) | **S (lowest)** | `pymodbus` `rtu_over_tcp` — CI-simulatable, no new infra |
| **1** | SQL poll | [design](2026-07-15-sql-poll-driver-design.md) | [plan](2026-07-24-sql-poll-driver.md) · [tasks](2026-07-24-sql-poll-driver.md.tasks.json) | 📝 **Plan ready** (22 tasks) | SM | **existing central SQL Server** `10.100.0.35,14330` + SQLite unit |
| **2** | MTConnect Agent | [design](2026-07-15-mtconnect-driver-design.md) | [plan](2026-07-24-mtconnect-driver.md) · [tasks](2026-07-24-mtconnect-driver.md.tasks.json) | 📝 **Plan ready** (23 tasks) | SM | Dockerized `mtconnect/cppagent` — CI-simulatable |
| **2** | MQTT / Sparkplug B | [design](2026-07-15-mqtt-sparkplug-driver-design.md) | [plan](2026-07-24-mqtt-sparkplug-driver.md) · [tasks](2026-07-24-mqtt-sparkplug-driver.md.tasks.json) | 📝 **Plan ready** (27 tasks, P1+P2) | ML | Mosquitto/EMQX + C# Sparkplug sim — CI-simulatable |
> Waves 3+ (BACnet/IP, Omron) and the deferred MELSEC are tracked in the program design doc §6/§8,
> not here. Omron CIP is the program's sole real-hardware wire gate; BACnet's broadcast/BBMD leg is
> env-gated live. Both are out of this file's scope until they're pulled forward.
---
## Executing a plan (git worktree + subagent-per-task)
Each 📝 plan is executed with the **subagent-driven-development** skill: it sets up an isolated
**git worktree** first (via `using-git-worktrees`), then dispatches a **fresh subagent per task**,
running the classification-driven review chain (`trivial` = implement only … `high-risk` =
spec-review → code-review → integration review) between tasks. The `.tasks.json` next to each plan
tracks progress and lets a later session resume.
**Paste one of these into Claude Code to build a driver:**
| Deliverable | Command |
|---|---|
| Modbus RTU (Wave 1) | `Use superpowers-extended-cc:subagent-driven-development to execute docs/plans/2026-07-24-modbus-rtu-driver.md in a new git worktree` |
| SQL poll (Wave 1) | `Use superpowers-extended-cc:subagent-driven-development to execute docs/plans/2026-07-24-sql-poll-driver.md in a new git worktree` |
| MTConnect (Wave 2) | `Use superpowers-extended-cc:subagent-driven-development to execute docs/plans/2026-07-24-mtconnect-driver.md in a new git worktree` |
| MQTT/Sparkplug (Wave 2) | `Use superpowers-extended-cc:subagent-driven-development to execute docs/plans/2026-07-24-mqtt-sparkplug-driver.md in a new git worktree` |
- **Slash-command form** (equivalent): `/superpowers-extended-cc:subagent-driven-development <plan-path>`.
- **Parallel-session / resume form** (batch execution with checkpoints, no fresh-subagent-per-task):
`/superpowers-extended-cc:executing-plans <plan-path>` — reads the same `.tasks.json` and continues
from the first pending task.
- **Recommended order:** Modbus RTU → SQL poll → MTConnect → MQTT/Sparkplug (lowest effort first;
see each wave below for the gating notes). Run one plan per worktree; the four plans are
independent, so separate worktrees may run concurrently.
---
## Wave 0 — Universal Discover-backed browser 🟡
**What it is.** One generic `DiscoveryDriverBrowser` (+ `CapturingAddressSpaceBuilder`,
`CapturedTreeBrowseSession`, `BrowserSessionService` fallback, and the
`ITagDiscovery.SupportsOnlineDiscovery` gate) that turns any driver's `ITagDiscovery.DiscoverAsync`
into an AdminUI browse tree. It is the **Wave-0 gate**: every browsable new driver depends on this
seam, and it retrofits browse to the already-shipped AbCip / TwinCAT / FOCAS drivers for near-zero
marginal cost.
- **Design:** [`2026-07-15-universal-discovery-browser-design.md`](2026-07-15-universal-discovery-browser-design.md)
- **Implementation plan:** [`2026-07-15-universal-discovery-browser-implementation.md`](2026-07-15-universal-discovery-browser-implementation.md) · [`.tasks.json`](2026-07-15-universal-discovery-browser-implementation.md.tasks.json)
- **Status: 🟡 code-complete + merged, live gate open.**
- Merged to master — `056887d6` (*Merge feat/universal-discovery-browser — Wave-0 universal Discover-backed browser*), 2026-07-15.
- Implementation plan: **19 / 19 tasks completed.**
- Lit up AbCip / TwinCAT / FOCAS pickers with zero per-driver browse code.
- **Outstanding:** the full tree-render live `/run` gate is **fixture-blocked** — tracked as **Gitea #468**. Complete this before leaning on browse in Wave 2/3.
---
## Wave 1 — low-effort / high-leverage pair 📝
Both are **fully CI-simulatable with zero new hardware**; Modbus RTU needs no new infra at all, and
SQL poll reuses the always-on central SQL Server. This is the recommended next build.
### Modbus RTU — 🟢 Built, live gate PASSED (branch `feat/modbus-rtu-driver`, pending merge)
- **Design:** [`2026-07-15-modbus-rtu-driver-design.md`](2026-07-15-modbus-rtu-driver-design.md)
- **Implementation plan:** [`2026-07-24-modbus-rtu-driver.md`](2026-07-24-modbus-rtu-driver.md) · [`.tasks.json`](2026-07-24-modbus-rtu-driver.md.tasks.json) — **11 tasks** (1 trivial / 5 small / 2 standard / 3 high-risk), all complete via subagent-driven-development.
- **Scope:** extend the existing Modbus driver with a `Transport` selector (RTU CRC-16 + FC-aware
length, no TxId) + a `rtu_over_tcp` `pymodbus` docker profile + the AdminUI selector.
**Direct-serial transport is descoped** (user, 2026-07-15) — RTU-over-TCP via a serial→Ethernet
gateway is the only shipped mode, so there is no serial hardware gate.
- **Fixture:** added the `rtu_over_tcp` profile (`framer=rtu`, host `:5021`, co-runs with `standard`)
to the Modbus integration fixture. **Unknown confirmed:** pymodbus 3.13's simulator DOES honour
`framer=rtu` on a TCP server (wire-probed a pure RTU FC03 response, no MBAP) — so the simple profile
path was taken; **no stdlib fallback server was needed.**
- **Verification (all green):** unit suite **344/344**; full solution build 0 errors; the 2 RTU
integration tests pass **live** against the real `framer=rtu` fixture (read HR5→5, write+readback
HR200←4242). Per-task review chain (spec + code reviewers) + a final whole-branch review, all
approved-to-merge.
- **Live `/run` gate — PASSED (2026-07-24, docker-dev rebuilt from this branch):**
- AdminUI `/raw` Modbus **driver** config modal renders the new **Transport** selector (with the
operator hint) — set to `RtuOverTcp`, saved.
- **Device Test Connect → `OK · 23 MS`** — the probe built a `ModbusRtuOverTcpTransport` via the
factory and spoke **FC03 over real RTU framing** to `10.100.0.35:5021`.
- Authored raw tags HR5 (r/o) + HR200 (r/w) → **deployment `06ec1632` Sealed** (all 6 nodes acked).
- Client.CLI on `opc.tcp://localhost:4840` (`multi-role`/`password`): **read** `ns=2;s=rtu-gate/Device1/HR5`
**5, Good**; **write** `ns=2;s=rtu-gate/Device1/HR200` = 4242 → readback **4242, Good**.
- **Effort:** **S — the lowest on the roadmap.** Delivered first, as recommended.
### SQL poll — 📝 Plan ready
- **Design:** [`2026-07-15-sql-poll-driver-design.md`](2026-07-15-sql-poll-driver-design.md)
- **Implementation plan:** [`2026-07-24-sql-poll-driver.md`](2026-07-24-sql-poll-driver.md) · [`.tasks.json`](2026-07-24-sql-poll-driver.md.tasks.json) — **22 tasks**. `ISqlDialect` seam in from day one; only the SQL Server dialect built in v1 (Postgres/ODBC deferred).
- **Scope:** a bespoke schema-browser driver polling a SQL table into equipment tags (SQL Server P1;
Postgres/ODBC land in P2/P3 behind `ISqlDialect`).
- **Fixture:** SQLite unit fixture (primary) + an **env-gated integration fixture against the
existing central SQL Server** (`10.100.0.35,14330`) with a seeded `SqlPollFixture` DB. The
blackhole/timeout live-gate pauses a **dedicated** `mssql` container — **never** the shared
central SQL Server (it hosts `ConfigDb`).
- **Effort:** SM.
---
## Wave 2 — strategic telemetry / UNS pair 📝
Both CI-simulatable on the shared docker host, no hardware.
### MTConnect Agent — 📝 Plan ready
- **Design:** [`2026-07-15-mtconnect-driver-design.md`](2026-07-15-mtconnect-driver-design.md)
- **Implementation plan:** [`2026-07-24-mtconnect-driver.md`](2026-07-24-mtconnect-driver.md) · [`.tasks.json`](2026-07-24-mtconnect-driver.md.tasks.json) — **23 tasks**. Task 0 is the TrakHound-vs-hand-rolled client decision; browse-picker live-verify is gated on Wave-0 #468.
- **Scope:** P1 Agent MVP (`IDriver`+`ITagDiscovery`+`IReadable`+`ISubscribable`+probe+rediscover),
browse **free via the Wave-0 universal browser** (`SupportsOnlineDiscovery=true`, no browser code),
typed editor, `UNAVAILABLE→BadNoCommunication` mapping, ring-buffer re-baseline paging.
- **Fixture:** canned XML unit fixtures (bulk of coverage) + a dockerized `mtconnect/cppagent`
integration fixture (env-gated). Depends on Wave 0's browse seam being live-verified (#468).
- **Effort:** SM (≈11.5 wk with TrakHound, ≈2.53 wk hand-rolled).
### MQTT / Sparkplug B — 📝 Plan ready
- **Design:** [`2026-07-15-mqtt-sparkplug-driver-design.md`](2026-07-15-mqtt-sparkplug-driver-design.md)
- **Implementation plan:** [`2026-07-24-mqtt-sparkplug-driver.md`](2026-07-24-mqtt-sparkplug-driver.md) · [`.tasks.json`](2026-07-24-mqtt-sparkplug-driver.md.tasks.json) — **27 tasks** (P1 plain MQTT = Tasks 014, a complete shippable milestone; P2 Sparkplug B = Tasks 1526). Task 0 is the mandatory MQTTnet-5/net10 + central-pinning validation spike.
- **Scope:** P1 plain MQTT (MQTTnet-5 connect/TLS/auth + hand-rolled reconnect, subscribe→
`OnDataChange`, retained last-value read, `#`-observation browser); P2 Sparkplug B ingest
(vendored Tahu proto, birth/alias/seq-gap/rebirth state machine, death→STALE).
- **Fixture:** Mosquitto/EMQX broker (TLS + real auth, not anonymous) + a **project-owned C#
Sparkplug edge-node simulator** for the rebirth/seq-gap/death test matrix + env-gated live suite
(`MQTT_FIXTURE_ENDPOINT`).
- **Effort:** ML. **Top risks:** Sparkplug state-machine correctness; MQTTnet-5/net10 + the repo's
fragile central pinning (mitigated by hand-rolling Tahu over one MQTTnet-5 client).
---
## Next actions
1. **Close the Wave-0 live gate** (Gitea #468) — unblocks browse verification for MTConnect/BACnet.
2. **Build Wave 1** — Modbus RTU first (lowest effort, no new infra), then SQL poll. Both plans are
📝 ready; run the command from the [Executing a plan](#executing-a-plan-git-worktree--subagent-per-task)
table (subagent-driven-development in a git worktree).
3. **Build Wave 2** (MTConnect, then MQTT/Sparkplug) — both plans 📝 ready. MTConnect's browse leg
wants #468 closed first; MQTT's Task 0 pinning spike gates everything after it.
Update this file's Summary table and per-wave status whenever a deliverable changes state.
@@ -0,0 +1,132 @@
# Per-cluster mesh Phase 6 — live gate record
> **Exit gate for Phase 6** (mesh partition). Bring up the rewritten docker-dev rig
> (three independent 2-node Akka meshes) against **merged master** and prove every
> previously-live-gated behaviour *per pair*, plus the split itself.
>
> Merge landed first (`origin/master` @ `2839deb5`, "merge now, gate after"); this gate
> runs against the merged code and **fixes forward on master** if anything surfaces.
**Rig topology** (docker-dev, local OrbStack):
| Mesh | Nodes | Roles | OPC UA (host) | Seeds (self-first, pair-local) |
|---|---|---|---|---|
| MAIN | central-1, central-2 | `admin,driver,cluster-MAIN` | 4840 / 4841 | own pair only |
| SITE-A | site-a-1, site-a-2 | `driver,cluster-SITE-A` | 4842 / 4843 | own pair only |
| SITE-B | site-b-1, site-b-2 | `driver,cluster-SITE-B` | 4844 / 4845 | own pair only |
- Akka port 4053 (per container); telemetry gRPC `:4056`; ConfigServe `:4055` (central);
LocalDb sync `:9001` (site-a); AdminUI `http://localhost:9200` (Traefik → central-1/2).
- Transports: `MeshTransport__Mode=ClusterClient` (all), `Telemetry__Mode=Grpc` (all six),
`TelemetryDial__Mode=Grpc` (central pair). Secrets replication pair-local.
- Deploy: `POST http://localhost:9200/api/deployments`, header `X-Api-Key: docker-dev-deploy-key`.
---
## Legs
| # | Leg | Status | Evidence |
|---|---|---|---|
| 1 | Three meshes form — each node sees ONLY its own pair (2 members) | ✅ PASS | each node handshakes ONLY its pair partner |
| 2 | Two+ Primaries, one per pair; ServiceLevel 250/240 within each pair | ✅ PASS | MAIN central-2=250/central-1=240; SITE-A site-a-1=250/site-a-2=240; SITE-B site-b-1=250/site-b-2=240 |
| 3 | Deploy reaches all clusters over per-cluster ClusterClient; seals green | ✅ PASS | "Deployment 019dbd99… sealed (acks=6)"; all 6 NodeDeploymentState rows acked; one ClusterClient per site cluster |
| 4 | Telemetry per cluster over gRPC (:4056); pills live; kill/reconnect ~5s | ✅ PASS | central-1 6 outbound :4056 dials, each site node 2 inbound (both admin dial in), 0 failures/30s; site-b restart failed→recovered the streams |
| 5 | Reconciler quiet — no `EnabledRowNotInCluster` for foreign rows | ✅ PASS | central-2 reconciler: "reconcile with cluster membership (2 driver members)" — own MAIN pair only; 0 foreign-row errors |
| 6 | Secrets pair-local — no cross-mesh replication / no unreachable-peer errors | ✅ PASS (by fallback criterion) | 0 secret errors fleet-wide; DPS topic rides each pair's isolated mesh (Leg 1) ⇒ structurally pair-local |
| 7 | Split validator — a site node flipped to `MeshTransport__Mode=Dps` refuses to boot | ✅ PASS | "Cluster:Roles declares a cluster-SITE-B role … MeshTransport:Mode is 'Dps' … set MeshTransport:Mode=ClusterClient" |
| 8 | Failover within a pair — kill a Primary, Secondary takes over (auto-down 1-vs-1) | ✅ PASS | killed site-a-1 (Primary); site-a-2 auto-downed it, took the singleton, 240→250, survived alone (auto-down 1-vs-1 = deferred Phase 0a gate, now proven) |
**Phase 6 exit gate: MET.** All 8 legs pass. Three findings fixed forward on master (`3a4ed8dd` product
boot-crash, `792f28ec` rig LDAP + split-brain serialization). One test-methodology note: the Leg 7
`compose run … site-b-2` throwaway shares the `site-b-2` network alias and briefly disrupted the SITE-B
Akka association — restart the affected pair after that probe (done), or run the validator check via a
container that does not join the compose network alias.
---
## Findings
### FINDING 1 (fix-forward on master) — boot-crash: `StackOverflowException` on EVERY node
**Symptom.** First `up` of the fresh Phase-6 images: all six nodes crash at startup with
`Stack overflow.` — an infinite recursion entering through
`ZB.MOM.WW.OtOpcUa.Cluster.ServiceCollectionExtensions.WithOtOpcUaClusterBootstrap`.
**Root cause (Phase 6, Task 2 wiring).** `Program.cs` resolved `IClusterRoleInfo` **eagerly, inside
the `AddAkka` configurator lambda**:
```csharp
ab.WithOtOpcUaClusterRedundancySingleton(sp.GetRequiredService<IClusterRoleInfo>()); // line 385
```
That lambda runs *while the `ActorSystem` is being constructed*, but `ClusterRoleInfo`'s ctor
**depends on the `ActorSystem`** (it is a live `Cluster.State` view). Resolving it there forces a
nested `ActorSystem` build → re-runs the same configurator → `sp.GetRequiredService<IClusterRoleInfo>()`
again → ∞. The cycle:
`WithOtOpcUaClusterBootstrap → GetService<IClusterRoleInfo> → ActorSystemFactory → WithOtOpcUaClusterBootstrap → …`
**Why merge + build + tests + reviews all missed it.** It compiles cleanly (a *runtime* DI cycle,
not a compile error). `RedundancyStateSingletonRehomeTests` boots a real host but passes a hand-built
`FakeClusterRoleInfo` **directly** into the extension — it never exercised the `sp.GetRequiredService`
line in the composition root that overflows. The bug lived in `Program.cs`, which every test mocked
around. Only a real boot of the shipped image surfaces it — precisely the live gate's job.
**Fix.** Derive the singleton's cluster-role scope from `AkkaClusterOptions` (pure config, no
ActorSystem) instead of `IClusterRoleInfo`:
- `BuildClusterRedundancySingletonOptions(AkkaClusterOptions)` + `WithOtOpcUaClusterRedundancySingleton(AkkaClusterOptions)`
— derive `Role = clusterOptions.Roles.FirstOrDefault(RoleParser.IsClusterRole) ?? RoleParser.Driver`
(identical to `ClusterRoleInfo.ClusterRole`, which derives from the same config).
- `Program.cs` passes `sp.GetRequiredService<IOptions<AkkaClusterOptions>>().Value``IOptions<>` has
no ActorSystem dependency, breaking the cycle.
- Tests updated to pass `AkkaClusterOptions { Roles = … }`. 5/5 rehome tests pass; Host builds clean.
Files: `Program.cs`, `ControlPlane/ServiceCollectionExtensions.cs`, `RedundancyStateSingletonRehomeTests.cs`.
Status: **FIXED + live-verified** — rebuilt image boots with zero stack-overflow lines; all six nodes
start; cluster singletons form (`redundancy-state` identified per mesh). 5/5 rehome tests pass.
### FINDING 2 (fix-forward on master) — driver-only site nodes crash: LDAP options fail validation
**Symptom.** site-a-1/site-a-2/site-b-1/site-b-2 crash at `Host.StartAsync` with
`OptionsValidationException: LDAP transport is None (plaintext) but AllowInsecure is false`.
**Root cause (Task 7 rig).** Only central-1/central-2 carried the `Security__Ldap__*` env block; the
compose even asserted "Only the admin-role central nodes carry" it. But the site nodes serve OPC UA
endpoints (4842-4845) and LDAP options are validated at boot on every node (appsettings default
`Enabled=true`, `Transport=None`, `AllowInsecure=false` ⇒ throw). The site nodes overrode `environment`
wholesale (YAML `<<:` does not deep-merge `environment`), so they inherited no LDAP config. `docker
compose config` validates the file but never boots the app, so this passed the Task 7 acceptance check.
**Fix (rig).** Extracted the LDAP block to a shared `&ldap-env` anchor merged into **every** host node
(`<<: [*secrets-env, *ldap-env]`); corrected the two misleading header comments. Site nodes now boot.
### FINDING 3 (fix-forward on master) — site pairs split-brain at simultaneous startup (250/250)
**Symptom.** Both nodes of each site pair logged `is JOINING itself ... and forming a new cluster` and
each elected itself Primary → ServiceLevel **250/250** instead of 250/240. Each pair was two independent
1-node clusters, not one 2-node mesh.
**Root cause (Task 7 rig — a startup race, not a code defect).** Both pair nodes are self-first seeds
(`seed[0]=self`), so both run Akka's `FirstSeedNodeProcess` and can form alone. The partner
(site-a-2/site-b-2) only `depends_on` sql/central/migrator — **not its pair founder** — so the pair
started simultaneously and each formed its own cluster before discovering the other. The central pair
avoids this only because central-2 `depends_on central-1` (and even then won the InitJoin race by luck).
`depends_on: service_started` is insufficient — it fires when the container launches, before Akka binds.
This is an inherent property of symmetric self-first seeding (present for central since #459), now
extended to sites; **carry to Phase 7** as a production concern (simultaneous cold-start of both pair
VMs). NOTE: two separate 1-node clusters do NOT merge via SBR — auto-down resolves partitions of one
cluster, not two independent clusters.
**Fix (rig).** Added an `&akka-founder-healthcheck` (bash `/dev/tcp` probe of Akka port 4053; the image
has no nc/curl) to site-a-1/site-b-1, and switched site-a-2/site-b-2 to `depends_on: { site-a-1:
service_healthy }`. The founder now binds + forms its mesh before the partner starts, so the partner
JOINS it. Verified: site-a-2/site-b-2 log `Welcome from <founder>` (joined), ServiceLevel 250/240.
---
## Evidence log
- **Leg 1** — join handshakes: central-1↔central-2, site-a-1↔site-a-2, site-b-1↔site-b-2; no node
handshakes a node outside its pair. central *dials* site nodes over ClusterClient/telemetry (Akka
association + gRPC), which is the split-topology transport, NOT gossip membership.
- **Leg 2**`Client.CLI redundancy` per endpoint: 4840 central-1=240, 4841 central-2=250, 4842
site-a-1=250, 4843 site-a-2=240, 4844 site-b-1=250, 4845 site-b-2=240. One Primary per pair.
@@ -0,0 +1,540 @@
# Per-Cluster Mesh Phase 6 — Mesh Partition + Co-location Topology
> **For Claude:** REQUIRED SUB-SKILL: execute this plan task-by-task with
> superpowers-extended-cc:subagent-driven-development (the established mode for this workstream).
**Goal:** Split OtOpcUa's single fleet-wide Akka mesh into **one 2-node mesh per application
`Cluster`** (central pair + one pair per site), so every Primary-gated decision and every gated
resource shares one pair-local scope — the ScadaBridge shape. Same ActorSystem name (`otopcua`)
everywhere; separation is by **seed-node partitioning** only.
**Architecture:** Each pair's two nodes list only each other as seeds (self-first), so Akka forms
three independent meshes even on a shared network. Driver nodes carry a new `cluster-{ClusterId}`
role; the one pair-local singleton (`RedundancyStateActor`) re-scopes to it and is spawned on every
driver node, so each pair elects its own Primary and publishes `redundancy-state` on its own mesh's
DPS. Central (admin) reaches each site mesh over the explicit Phase 2/3/5 transports (ClusterClient
+ ConfigServe gRPC + telemetry gRPC), now partitioned **one ClusterClient per cluster**. The five
admin singletons stay on the central pair. Two admin-side consumers that assumed fleet-wide gossip
visibility (`ClusterNodeAddressReconciler`, `CentralCommunicationActor.RebuildClient`) are
re-scoped. Secrets replication becomes **pair-local** automatically (Akka DPS within each 2-node
mesh; no fleet-wide sync, no central-SQL dependency — consistent with Phase 4). A startup validator
refuses to boot a split-configured node (one carrying a `cluster-` role) that is still pointed at a
DPS transport mode, and the compiled transport defaults flip to the split-correct values.
**Tech Stack:** .NET 10, Akka.NET + Akka.Hosting (`WithSingleton`/`WithActors`), Akka.Cluster.Tools
(ClusterClient + ClusterSingleton + DistributedPubSub), Akka.Cluster.Hosting `ClusterSingletonOptions`,
xUnit + Shouldly, docker-dev compose rig.
**Decisions settled with the user 2026-07-24 (recon-driven; the first contradicts the program doc's
own guess):**
1. **Secrets after split = pair-local Akka.** The program doc guessed "SQL-hub mode like ScadaBridge,"
but recon showed sites have **no SQL Server** (the point of Phase 4), so a SQL hub would force site
nodes back onto central SQL. Instead: leave the `AddZbSecretsAkkaReplication` wiring untouched — it
becomes per-pair for free once each mesh is two nodes. Fleet-wide secret sync is dropped by design;
document it. No library re-wire.
2. **DPS branch disposition = flip + change defaults + validator, keep the code.** The rig flips to
`ClusterClient`/`Grpc`/`FetchAndCache`; the compiled defaults change so a fresh deploy is
split-correct; the DPS branch code stays as dark switches; a new startup validator makes a
split-configured node refuse to boot on a DPS transport mode (neutralizes the silent-break footgun
without a large irreversible deletion). `redundancy-state`/`fleet-status` stay on DPS either way.
3. **Rig scope = OtOpcUa-only per-cluster meshes.** Rewrite docker-dev into 3 independent 2-node
meshes on a single shared network (faithful to production, where separation is by seeds over a WAN,
not by network isolation). Physical co-location with ScadaBridge is documented as the production
topology, not literally run in docker-dev.
**Authoritative context (read before implementing):**
- `docs/plans/2026-07-22-per-cluster-mesh-program.md` §"Phase 6" (lines 266284) + the tracking table.
- `docs/plans/2026-07-21-per-cluster-mesh-design.md` §4 (oldest-Up election, already fixed), §5
(target architecture), §7 (sequencing).
- Recon findings (this session) — summarized inline per task below.
---
## What Phase 6 does NOT change (scope guards)
- **ActorSystem name stays `otopcua`.** Do not rename it or make it per-cluster.
- **`AkkaClusterOptions` gains no new property.** `SeedNodes` + `Roles` already carry per-node values
from config; the split is in *how they're populated per pair* (rig/appsettings) + the `RoleParser`
allow-list + singleton scoping. Do not add a `ClusterId` option — every `ClusterNode` row is already
defined to be a driver node (Phase 1 decision), and roles are the declaration of record.
- **`SelectDriverPrimary` election logic is unchanged.** It filters on the `driver` role over
`Cluster.State.Members`; after the split that set is pair-local, so it elects the oldest of two.
No edit to the algorithm.
- **The five admin singletons stay admin-scoped** (`config-publish`, `admin-operations`,
`audit-writer`, `fleet-status`, `cluster-node-address-reconciler`). Only `redundancy-state`
re-homes.
- **`DriverMemberCount()`** in `DriverHostActor`/`ScriptedAlarmHostActor` needs **no code change**
it reads whole-mesh members filtered by `driver`, which becomes pair-local for free. (Note the
semantics change in docs; do not "fix" it in code.)
- **Do NOT delete the DPS command/telemetry branches** (decision 2). Do NOT switch the network to
per-site isolation or wire in ScadaBridge (decision 3). Do NOT re-wire secrets to SQL (decision 1).
---
### Task 0: `RoleParser` accepts `cluster-{ClusterId}`; consolidate the `driver` role literal
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** none (blocks 1, 2, 5, 7)
**Context:** `RoleParser.Allowed` is a fixed `HashSet {"admin","driver","dev"}`
(`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs:5-8`). A `cluster-{ClusterId}` role fails it, so
no node can carry one. Also, three independent `"driver"` string literals exist
(`RedundancyStateActor.DriverRole`, `ClusterNodeAddressReconcilerActor.DriverRole`,
`Runtime/ServiceCollectionExtensions.DriverRole`) — consolidate to a single source now since every
Phase 6 task touches role code.
**Files:**
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs`
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/RoleParserTests.cs` (create if absent)
**Steps:**
1. Add a well-known-roles surface to `RoleParser` (or a new `ClusterRoles` static): constants
`Admin = "admin"`, `Driver = "driver"`, `Dev = "dev"`, `ClusterRolePrefix = "cluster-"`, and a
helper `bool IsClusterRole(string role) => role.StartsWith(ClusterRolePrefix, StringComparison.Ordinal) && role.Length > ClusterRolePrefix.Length`.
2. Change `Parse` validation: a role is allowed if it is one of the three fixed roles **or**
`IsClusterRole(role)` is true. Keep the existing trim/lowercase/empty-filter behavior. A role of
exactly `"cluster-"` (empty ClusterId) is **rejected**.
3. Add `RoleParser.ClusterIdFromRole(string) => role.Substring(ClusterRolePrefix.Length)` returning
the ClusterId for a cluster role (used by later tasks).
4. Update `RedundancyStateActor.DriverRole`, `ClusterNodeAddressReconcilerActor.DriverRole`, and
`Runtime/ServiceCollectionExtensions.DriverRole` to reference the single `RoleParser.Driver` (keep
the public `const`/`static readonly` symbols as aliases if any test references them by their old
name — grep first; do not break existing test references).
5. Tests: `admin`/`driver`/`dev`/`cluster-MAIN`/`cluster-SITE-A` accepted; `cluster-` rejected;
`bogus` rejected; case/whitespace normalization preserved; `ClusterIdFromRole("cluster-SITE-A") == "SITE-A"`.
**Acceptance:** `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests` green; `cluster-{X}` roles
parse; the three literals resolve to one constant.
---
### Task 1: `ClusterRoleInfo` exposes the node's own cluster-scoped role
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** none (needs Task 0; blocks 2, 5)
**Context:** Singleton registration and the validator both need "what is *this* node's
`cluster-{ClusterId}` role." `ClusterRoleInfo` (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterRoleInfo.cs`,
`IClusterRoleInfo`, registered in `AddOtOpcUaCluster`) is the existing role/leader authority — extend
it.
**Files:**
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterRoleInfo.cs`
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/IClusterRoleInfo.cs` (or wherever the interface lives)
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/ClusterRoleInfoTests.cs`
**Steps:**
1. Add `string? ClusterRole { get; }` (e.g. `"cluster-SITE-A"`) and `string? ClusterId { get; }` (e.g.
`"SITE-A"`) to `IClusterRoleInfo`, derived from the configured `AkkaClusterOptions.Roles` via
`RoleParser.IsClusterRole` (first match; log a Warning and pick the first if >1 cluster role is
configured — that is a misconfiguration). Null when the node carries no cluster role (e.g. a
pre-split/admin-only-legacy node).
2. Do NOT read `Cluster.State` for this — it is the node's **own configured** identity, available
before the cluster forms (the singleton registration runs at host build time).
3. Tests: a node configured `["driver","cluster-SITE-A"]` reports `ClusterRole="cluster-SITE-A"`,
`ClusterId="SITE-A"`; a node `["admin","driver"]` reports both null; two cluster roles → first wins
+ warning path exercised.
**Acceptance:** `ClusterRoleInfo` reports the node's cluster role/id from config; tests green.
---
### Task 2: Re-home `RedundancyStateActor` to a `cluster-{ClusterId}` singleton on driver nodes
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 3, Task 4 (disjoint files)
**Context (the crux).** Today `RedundancyStateActor` is registered as an **admin** singleton
(`ControlPlane/ServiceCollectionExtensions.cs:133-136`, `singletonOptions.Role = "admin"`), electing a
**fleet-wide** Primary and publishing `redundancy-state` over DPS. After the split, central's mesh
sees only its own pair — so a site pair (driver-only, no admin node) would have **no one** to elect its
Primary. Fix: scope the singleton to the node's `cluster-{ClusterId}` role and spawn it on **every
driver node** (`hasDriver`), so each mesh (central pair = `cluster-MAIN`, each site pair =
`cluster-{SITE}`) has exactly one, electing its own pair-local Primary and publishing on its own mesh's
DPS. **De-risked:** `RedundancyStateActor` has zero injected deps (verified — it only uses
`Cluster.Get(Context.System)` + DPS), so it spawns cleanly on a driver-only node.
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs`
(remove `redundancy-state` from `WithOtOpcUaControlPlaneSingletons`; add a new
`WithOtOpcUaClusterRedundancySingleton(IClusterRoleInfo)` extension that registers it scoped to the
cluster role)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (call the new extension on `hasDriver`, not
`hasAdmin`; a fused `admin,driver` node calls it once via the driver branch)
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/...` singleton-role test; and confirm the
admin path no longer registers `redundancy-state`.
**Steps:**
1. In `ControlPlane/ServiceCollectionExtensions.cs`: delete the `WithSingleton` block for
`RedundancyStateActorKey`/`redundancy-state` from the admin method (lines ~133-136). Keep the
registry key type.
2. Add:
```csharp
public static AkkaConfigurationBuilder WithOtOpcUaClusterRedundancySingleton(
this AkkaConfigurationBuilder builder, IClusterRoleInfo roleInfo)
{
// Cluster-scoped when the node carries a cluster-{ClusterId} role (Phase-6 split: one
// redundancy singleton per pair, robust even if two meshes accidentally merged). Falls
// back to the driver role for a legacy single-mesh node or a test harness with no cluster
// role — where driver-scope is the pre-Phase-6 fleet-wide behavior, and on a genuinely
// split 2-node mesh the driver role is ALREADY pair-local because the mesh IS the pair.
// No throw: decision 2 requires legacy/harness nodes to keep booting unchanged.
var role = roleInfo.ClusterRole ?? RoleParser.Driver;
var opts = new ClusterSingletonOptions { Role = role };
return builder.WithSingleton<RedundancyStateActorKey>(
RedundancyStateActor.Topic, RedundancyStateActor.Props(), opts /* createProxyToo per existing pattern */);
}
```
Match the exact `WithSingleton` overload/signature the other five singletons use (check createProxy
arg + singleton name conventions).
3. In `Program.cs`: within the `if (hasDriver)` Akka-configurator block (near line 357-374, alongside
`WithOtOpcUaRuntimeActors`), call `ab.WithOtOpcUaClusterRedundancySingleton(clusterRoleInfo)`.
Resolve `IClusterRoleInfo` from `sp`. Ensure it runs on the fused central node too (central has
`hasDriver=true`). Do NOT also register it in the `hasAdmin` block.
4. **No fail-fast on a missing cluster role** — the fallback keeps legacy/harness nodes booting
(decision 2). A *misconfigured* Phase-6 node (split transports, no cluster role) is a non-issue for
the redundancy singleton: driver-scope is pair-local on a split mesh anyway.
5. Tests: assert the admin singleton set no longer contains `redundancy-state`; assert the new
extension registers a singleton scoped to `Role == "cluster-SITE-A"` when the roleInfo reports that;
assert that a roleInfo with null `ClusterRole` falls back to `Role == "driver"` (no throw).
**Acceptance:** each mesh hosts exactly one `RedundancyStateActor` scoped to its cluster role; a
driver-only site node elects its own Primary; admin path no longer owns it. Unit tests green.
**Live-verified later** (Task 9 gate): two Primaries fleet-wide, one per pair.
---
### Task 3: Re-scope `ClusterNodeAddressReconciler` to central-mesh-visible rows
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 2, Task 4 (disjoint files)
**Context:** The admin-singleton reconciler compares `Cluster.State.Members` (driver role) against
**every** `ClusterNode` row fleet-wide (`ClusterNodeAddressReconciler.cs:123-131`,
`ClusterNodeAddressReconcilerActor.cs:76-96`). After the split, central sees only its own pair, so
every site row logs `EnabledRowNotInCluster` forever (its own doc comment,
`ClusterNodeAddressReconciler.cs:60-65`, flags this). Fix: only reconcile rows whose `ClusterId`
matches central's own cluster — central can no longer observe other clusters via gossip, so it must
not assert anything about their rows.
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Fleet/ClusterNodeAddressReconciler.cs` (filter
the row set by own-cluster)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Fleet/ClusterNodeAddressReconcilerActor.cs`
(pass the admin node's own `ClusterId` in; source it from `IClusterRoleInfo.ClusterId`, or from the
set of `ClusterId`s present among its own observed driver members)
- Update the class doc comment (remove the "Phase 2 must revisit" note; state it is now own-cluster
scoped).
- Test: `tests/.../ClusterNodeAddressReconcilerTests.cs` — a row in another cluster is **ignored** (no
finding); an own-cluster enabled row with no member still yields `EnabledRowNotInCluster`; drift
detection within own cluster still works.
**The pure `ClusterNodeAddressReconciler.Reconcile` function is UNCHANGED** — the fix is entirely in
the actor's two DB queries (`ClusterNodeAddressReconcilerActor.Reconcile`, lines 89-97), which filter
rows by the admin node's own `ClusterId` before projecting. `ClusterNode` has a `ClusterId` column.
**Steps:**
1. Inject the admin node's own `ClusterId` into the actor: add `IClusterRoleInfo` (or just a
`string? ownClusterId`) to `Props`/ctor, sourced from `IClusterRoleInfo.ClusterId` at registration
(`ControlPlane/ServiceCollectionExtensions.cs` where the reconciler singleton is registered).
2. In the actor's `Reconcile`, add `.Where(n => n.ClusterId == ownClusterId)` to BOTH the `rows` and
`enabled` queries — but ONLY when `ownClusterId` is non-null.
3. **Null `ownClusterId` (legacy admin node with no cluster role) ⇒ reconcile ALL rows** — this is the
correct single-mesh behavior (a legacy admin node genuinely sees every driver member via gossip).
Do NOT skip. This preserves backward-compat, mirroring the Task 2 fallback philosophy.
4. `RowDialTargetDisagrees` / `RunningNodeHasNoRow` / `EnabledRowNotInCluster` semantics are unchanged;
they now simply operate on the own-cluster row subset (post-split) or the full set (legacy).
5. Update the class doc comment in `ClusterNodeAddressReconciler.cs:60-65`: replace the "Phase 2 must
revisit" note with the delivered own-cluster-scoped behavior.
6. Tests (`ClusterNodeAddressReconcilerActorTests` or the existing reconciler tests — pick the level
that can inject `ownClusterId`): a SITE-A row is **ignored** when `ownClusterId=="MAIN"` (no
`EnabledRowNotInCluster`); an own-cluster (MAIN) enabled row with no member still yields
`EnabledRowNotInCluster`; own-cluster drift still caught; `ownClusterId==null` reconciles the full
set (legacy). If the actor is hard to unit-test directly, a focused query-filter test + the
unchanged pure-function tests suffice — say which you chose.
**Acceptance:** central reconciles only MAIN rows; site rows never produce false
`EnabledRowNotInCluster`; own-cluster drift still caught.
---
### Task 4: One `ClusterClient` per application `Cluster` in `CentralCommunicationActor`
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 2, Task 3 (disjoint files)
**Context:** `RebuildClient` (`CentralCommunicationActor.cs:271-297`) creates **one fleet-wide**
`ClusterClient` dialing every `ClusterNode` row, and `HandleMeshCommand` (`:128-157`) does one
`ClusterClient.SendToAll(...)`. The `TODO(Phase 6)` (`:249-268`) states this is correct only on a
single mesh: a `ClusterClientReceptionist` serves its whole cluster, so `SendToAll` on one client
reaches everyone. After the split, `SendToAll` on one client reaches only the mesh whose receptionist
answered — so central needs **one client per cluster**, and dispatch fans `SendToAll` across all of
them.
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/MeshClusterClientBoundaryTests.cs`
(extend: two separate site meshes + central, a command reaches BOTH via per-cluster clients; and a
single-cluster client's `SendToAll` does NOT reach the other cluster)
**Steps:**
1. `LoadContactsFromDb` (`:177-221`): return contacts **grouped by `ClusterId`** — an
`ImmutableDictionary<string, ImmutableHashSet<string>>` (ClusterId → receptionist contact paths).
Keep the `Enabled && !MaintenanceMode` filter.
2. Replace the single `_client`/`_currentContacts` with a `Dictionary<string, (IActorRef Client,
ImmutableHashSet<string> Contacts)>` keyed by ClusterId. `HandleContactsLoaded` diffs **per cluster**
and rebuilds only the clusters whose contact set changed; stops+removes clients for clusters that
vanished.
3. `RebuildClient` becomes `RebuildClient(string clusterId, ImmutableHashSet<string> contacts)`
builds one `ClusterClient` per cluster (name it by clusterId so the actor paths are distinct).
4. `HandleMeshCommand` fans out: for each cluster client, `client.Tell(new ClusterClient.SendToAll(
MeshPaths.NodeCommunication, cmd.Message))`. Preserve the sender-relay idiom where it exists.
**A command with no live clients (all clusters down) drops with a Warning**, matching today's
"no reachable contact" behavior — do not buffer.
5. Update the `TODO(Phase 6)` doc block to describe the delivered per-cluster-client shape (remove the
TODO marker).
6. **Frame-size note:** the deploy path stays payload-free (notify-and-fetch), so no new frame-size
exposure — do not add payload to these commands.
7. Tests: extend `MeshClusterClientBoundaryTests` to two site meshes + central; assert a dispatch
reaches a node in each site mesh, and that per-cluster `SendToAll` is correctly partitioned.
**Acceptance:** central dispatches deploy-notify/driver-control/alarm-commands to every cluster via
one client each; the boundary test proves cross-mesh partitioned delivery; acks still return via the
site nodes' own `ClusterClient.Send` to central (unchanged).
---
### Task 5: Split-topology startup validator (cluster-role ⇒ non-DPS transport)
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 6 (disjoint files)
**Context (decision 2):** the DPS branches stay as dark switches, but a node configured for the split
topology (it carries a `cluster-{ClusterId}` role) must not silently run a transport still in DPS
mode, which cannot cross meshes. Fail host start instead. Marker of "split topology" = the presence of
a `cluster-` role (they exist only post-Phase-6).
**Files:**
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/SplitTopologyTransportValidator.cs` (an
`IValidateOptions<...>` or a startup validator wired via `AddValidatedOptions`/`ValidateOnStart`,
mirroring `AkkaClusterOptionsValidator`/`ConfigSourceOptionsValidator` structure)
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs` (`AddOtOpcUaCluster`
registers it)
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SplitTopologyTransportValidatorTests.cs`
**Rule (only fires when the node carries a `cluster-` role):**
- `MeshTransport:Mode` must be `ClusterClient` (not `Dps`) — every node.
- If `hasDriver`: `Telemetry:Mode` must be `Grpc` (node hosts the telemetry server).
- If `hasAdmin`: `TelemetryDial:Mode` must be `Grpc` (central dials).
- `ConfigSource:Mode`: **do not** add a rule here — Phase 4's `ConfigSourceOptionsValidator` already
requires `FetchAndCache` on driver-only nodes; a fused node legitimately stays `Direct`.
- A node with **no** `cluster-` role (legacy/pre-split) is exempt (rule does not fire), so existing
single-mesh deployments and tests are unaffected.
**Steps:**
1. Read `Cluster:Roles` (same direct-read pattern the other validators use — not `IClusterRoleInfo`,
to avoid a service-resolution ordering dependency at validation time), `MeshTransport:Mode`,
`Telemetry:Mode`, `TelemetryDial:Mode`.
2. Implement the rule above; each violation adds a precise message naming the offending key + required
value.
3. Register with `ValidateOnStart` so a misconfigured node refuses to boot.
4. Tests: cluster-role node on `MeshTransport:Mode=Dps` fails; on `ClusterClient` passes; driver
cluster node on `Telemetry:Mode=Dps` fails; admin cluster node on `TelemetryDial:Mode=Dps` fails;
a non-cluster-role node on all-Dps passes (exempt).
**Acceptance:** a split-configured node on any DPS transport refuses to start with a clear message;
legacy nodes unaffected.
---
### Task 6: Compiled transport-mode defaults — DECIDED: do NOT flip (data-backed deviation)
**Classification:** small → **resolved as a no-code decision 2026-07-24**
**Parallelizable with:** Task 5
**Original intent (decision 2):** change the compiled `MeshTransport:Mode`/`Telemetry:Mode`/
`TelemetryDial:Mode` defaults to the split-correct values so a fresh deploy is split-correct without
per-node overrides.
**What the blast-radius assessment found (read-only, this session):** flipping the compiled defaults
is **HIGH-RISK and low-value**. Every one of the three transport-option validators fail-closes when its
mode is the non-Dps value but the mode's prerequisites are unset — and *nothing in the repo sets those
sections*:
- `MeshTransport:Mode=ClusterClient` requires `CentralContactPoints` — no `appsettings*.json` and no
test harness sets it → `ValidateOnStart` throws at boot.
- `Telemetry:Mode=Grpc` requires `GrpcListenPort`+`ApiKey` on a driver node — same gap.
- `TelemetryDial:Mode=Grpc` requires `ApiKey` with **no role gate at all** — breaks even roleless
fixtures and docker-dev's 4 site nodes.
- Concretely this hard-fails **all 14 `Host.IntegrationTests`** (via `TwoNodeClusterHarness`, which
sets none of these), **all 5 shipped appsettings profiles**, and several unit tests asserting the
Dps default.
**Why not flipping is correct, not a compromise:** the Task 5 `SplitTopologyTransportValidator`
already forces `ClusterClient`/`Grpc`/`Grpc` onto **exactly** the nodes carrying a post-Phase-6
`cluster-{ClusterId}` role, fail-closed, with a precise message — which *is* "a fresh Phase-6 deploy is
split-correct," achieved more robustly than a default flip (a split node cannot even boot on the wrong
transport). Flipping the compiled defaults would merely duplicate that enforcement for split nodes
while indiscriminately breaking every legacy/test/admin node that legitimately still runs Dps. The
docker-dev rig (Task 7) sets the modes **explicitly** on its cluster-role nodes, so it never relies on
the default anyway.
**Decision:** keep `MeshTransportOptions.Mode`/`TelemetryOptions.Mode`/`TelemetryDialOptions.Mode`
defaults at `Dps`. No code change. Enforcement of split-correctness lives in Task 5's validator + the
explicit Task 7 rig config. Recorded in `docs/Configuration.md` (Task 8) so the next reader knows the
defaults are intentionally Dps and the validator is the guardrail.
**Acceptance:** MET by decision — the split validator (Task 5) is the guardrail; no default flip.
---
### Task 7: Rewrite the docker-dev rig into three 2-node meshes
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none (needs Tasks 06 landed to actually run; the file edit is independent
but sequence it after code so the rig is testable)
**Context (decision 3):** one shared network, seed-partitioned into three meshes. Central pair =
`cluster-MAIN`; site-a pair = `cluster-SITE-A`; site-b pair = `cluster-SITE-B`. Each pair lists only
its own two nodes as seeds (self-first). Flip the transports on. Secrets stays enabled (now
pair-local). Central must remain network-reachable to site nodes (shared network — already true; do
**not** isolate networks).
**Files:**
- Modify: `docker-dev/docker-compose.yml`
- Modify: `docker-dev/seed/seed-clusters.sql` if any seeded address/port changes (AkkaPort stays 4053;
GrpcPort stays 4056 — likely no seed change, but verify the ClusterId/host columns still match).
**Per-node changes:**
| Node | `Cluster__Roles` | `Cluster__SeedNodes__0` (self) | `__1` (partner) |
|---|---|---|---|
| central-1 | `admin`,`driver`,`cluster-MAIN` | `akka.tcp://otopcua@central-1:4053` | `...central-2:4053` |
| central-2 | `admin`,`driver`,`cluster-MAIN` | `akka.tcp://otopcua@central-2:4053` | `...central-1:4053` |
| site-a-1 | `driver`,`cluster-SITE-A` | `akka.tcp://otopcua@site-a-1:4053` | `...site-a-2:4053` |
| site-a-2 | `driver`,`cluster-SITE-A` | `akka.tcp://otopcua@site-a-2:4053` | `...site-a-1:4053` |
| site-b-1 | `driver`,`cluster-SITE-B` | `akka.tcp://otopcua@site-b-1:4053` | `...site-b-2:4053` |
| site-b-2 | `driver`,`cluster-SITE-B` | `akka.tcp://otopcua@site-b-2:4053` | `...site-b-1:4053` |
**Transport flips (all nodes unless noted):**
- `MeshTransport__Mode: ClusterClient` (remove the `${OTOPCUA_MESH_MODE:-Dps}` default or set the var
default to ClusterClient).
- `Telemetry__Mode: Grpc` (all); `TelemetryDial__Mode: Grpc` (central pair only — the dialers).
- `ConfigSource__Mode: FetchAndCache` already on sites; central stays `Direct`.
- `MeshTransport__CentralContactPoints__0/1` on **site** nodes point at central-1/central-2 (unchanged
— sites learn central from appsettings). Central nodes do not need CentralContactPoints for
themselves; central discovers *site* contacts from `ClusterNode` rows (per-cluster clients, Task 4).
- **Secrets** (`x-secrets-env` anchor): keep `Secrets__Replication__Enabled=true` — it becomes
pair-local automatically. Add a compose comment noting the new pair-local semantics.
- **Remove** the `central-1`-as-seed lines from every site node (the old cross-pair seed). Update the
header comment block (lines 1-23) to describe three meshes, not one.
- **LocalDb sync:** keep site-a's replication demo; optionally add a distinct sync port for site-b to
model per-site ports (nice-to-have, not required for the gate — if added, use a different host-safe
port and its own peer address).
**Steps:**
1. Rewrite the six nodes' `Cluster__Roles__*` and `Cluster__SeedNodes__*` per the table.
2. Flip the transport-mode env vars.
3. Update the header comment + secrets comment.
4. `docker compose -f docker-dev/docker-compose.yml config` to validate YAML (no live bring-up in this
task — that is the Task 9 gate).
**Acceptance:** `docker compose config` validates; the six nodes carry per-pair self-first seeds +
cluster roles + split-correct transport modes; header comment describes three meshes.
---
### Task 8: Docs sweep — status tables, caveat removal, pair-local secrets note
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 7 (docs only; no build)
**Files:**
- `docs/Redundancy.md` — new §"Per-cluster meshes (Phase 6)": one Primary per pair (**two+ Primaries
fleet-wide, by design**), redundancy-state singleton is now `cluster-{ClusterId}`-scoped and
spawned per driver node; remove any single-mesh caveat. Add §"Secrets replication is pair-local".
- `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/IManualFailoverService.cs:34-35` — remove the
"Mesh-scope caveat / until the per-cluster mesh work lands" doc comment (it landed).
- AdminUI ClusterRedundancy page — remove the mesh-scope caveat text (grep the Razor for the caveat
string).
- `docs/Configuration.md` — document the `cluster-{ClusterId}` role, per-pair `Cluster:SeedNodes`
ordering, the new transport-mode defaults, and the split-topology validator.
- `docs/plans/2026-07-22-per-cluster-mesh-program.md` — mark Phase 6 **DONE** in the section + the
tracking table; note the three decisions.
- `docs/plans/2026-07-21-per-cluster-mesh-design.md` §7 — mark row 6 DONE.
- `CLAUDE.md` — a "Per-cluster mesh (Phase 6)" note under the Redundancy section: three meshes, cluster
roles, one Primary per pair, pair-local secrets, per-cluster ClusterClient, reconciler own-cluster
scoped, transport defaults flipped + split validator.
**Steps:**
1. Write the Redundancy.md sections; remove the two caveats (IManualFailoverService + Razor).
2. Update Configuration.md.
3. Update the two plan status tables + CLAUDE.md.
4. Grep for any remaining "single mesh" / "fleet-wide Primary" / "until the per-cluster mesh work
lands" copy and reconcile.
**Acceptance:** no stale single-mesh caveats remain; status tables show Phase 6 done; secrets
pair-local + cluster-role documented.
---
### Task 9: Live gate — three meshes, per-pair redundancy, cross-mesh transports
**Classification:** high-risk
**Estimated implement time:** manual (controller-run against docker-dev)
**Parallelizable with:** none (final)
**Context:** the exit gate. Bring the rewritten rig up and prove every previously-live-gated behavior
per pair, plus the split itself. Gather TCP/log/CLI evidence; record in
`docs/plans/2026-07-24-mesh-phase6-live-gate.md`.
**Legs:**
1. **Three meshes form.** Each node's `Cluster.State.Members` shows **only its own pair** (2 members).
Evidence: redundancy CLI / logs per node, or `/proc/net/tcp` showing 4053 associations only within
each pair. central sees central-1/2 only; site-a sees site-a-1/2 only; site-b sees site-b-1/2 only.
2. **Two+ Primaries, one per pair.** Each pair elects its own oldest-Up driver Primary; ServiceLevel
**250/240** within each pair (Primary/Secondary), independently. Client.CLI `redundancy` per
endpoint (4840/4841 MAIN, 4842/4843 SITE-A, 4844/4845 SITE-B).
3. **Deploy reaches all clusters over per-cluster ClusterClient.** `POST :9200/api/deployments`
(X-Api-Key) seals green with all six nodes acking; central's logs show one client per cluster;
verify a SITE-A node applies the config (its address space serves).
4. **Telemetry per cluster over gRPC.** AdminUI `/alerts` + `/hosts` pills live; central dials each
site mesh's telemetry servers (2 inbound per driver node on 4056; central holds outbound per cluster).
Kill-and-reconnect a site node → its stream recovers (~5s).
5. **Reconciler quiet.** central's `ClusterNodeAddressReconciler` logs **no** `EnabledRowNotInCluster`
for SITE-A/SITE-B rows (own-cluster scoped now); MAIN drift still caught (probe with a deliberate
mismatch if practical).
6. **Secrets pair-local.** With `Secrets:Replication:Enabled=true`, a secret written on site-a-1
replicates to site-a-2 (same mesh) and does **not** reach central or site-b (separate meshes) — or,
if a live secret write is impractical, assert the DPS topic is mesh-scoped via the membership
evidence from leg 1 + a doc note. No errors from the secrets actor about unreachable peers.
7. **Split validator.** Flip one site node to `MeshTransport__Mode=Dps` → it refuses to boot with the
Task 5 message. Restore.
8. **Failover within a pair** (bridges toward Phase 7): kill a site pair's Primary → the Secondary
becomes Primary (250) and the pair survives alone (auto-down 1-vs-1 — the deferred Phase 0a gate is
now testable; run it if time permits and note the result for Phase 7).
**Record:** `docs/plans/2026-07-24-mesh-phase6-live-gate.md` with per-leg evidence + any findings.
**Cleanup:** restore the rig to a coherent committed state.
**Acceptance:** all legs pass (or any deviation is documented + justified as in prior phases). Phase 6
exit gate MET.
---
## Risk register (carry from design §8 + program)
- LocalDb is load-bearing for config (unchanged; #485 class).
- **Two+ Primaries fleet-wide is now correct, by design** — anything asserting "one Primary" is stale.
- Partial rollout hazard: a node whose mesh is split has pair-local `DriverMemberCount`, while a
not-yet-split node counts fleet-wide — do not run the fleet half-split in production without the
drill (Phase 7).
- Co-located VM loss takes out one node of **both** products (Phase 7 drill item; out of scope here).
- Frame size: deploy path stays payload-free — keep it that way.
@@ -0,0 +1,24 @@
{
"planPath": "docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md",
"program": "per-cluster-mesh",
"phase": 6,
"branch": "feat/mesh-phase6",
"decisions": {
"secrets": "pair-local Akka (no SQL-hub; sites have no SQL)",
"dpsBranches": "flip rig + split-topology validator; keep DPS code. NOTE: compiled defaults NOT flipped (Task 6) — blast-radius assessment showed a flip hard-fails all 14 Host.IntegrationTests + all 5 appsettings profiles + docker-dev site nodes; the Task 5 validator already enforces split-correctness fail-closed on cluster-role nodes, which is the actual intent.",
"rigScope": "OtOpcUa-only three 2-node meshes on one shared network"
},
"tasks": [
{"id": 0, "subject": "Task 0: RoleParser accepts cluster-{ClusterId}; consolidate driver literal", "classification": "small", "status": "completed", "commit": "87ff00fd"},
{"id": 1, "subject": "Task 1: ClusterRoleInfo exposes node's own ClusterRole/ClusterId", "classification": "small", "status": "completed", "commit": "2d2f1dec", "note": "interface in Commons/Interfaces/IClusterRoleInfo.cs; ctor (ActorSystem, IOptions<AkkaClusterOptions>, ILogger)"},
{"id": 2, "subject": "Task 2: re-home RedundancyStateActor to cluster-role singleton on drivers", "classification": "high-risk", "status": "completed", "commit": "a4d3ab40", "note": "re-home a2c5d57f + review fixes a4d3ab40 (comments + driver-fallback registry test + fixed a cross-task admin-boot NRE race); 14/14 tests"},
{"id": 3, "subject": "Task 3: re-scope ClusterNodeAddressReconciler to own-cluster rows", "classification": "standard", "status": "completed", "commit": "1ce2042e", "note": "impl 2535df01 + review-fix 1ce2042e (observed ALSO scoped by cluster role via testable FilterOwnClusterDriverMembers helper; 17/17)"},
{"id": 4, "subject": "Task 4: one ClusterClient per cluster in CentralCommunicationActor", "classification": "high-risk", "status": "completed", "commit": "6531ec19", "note": "impl 6531ec19 + comment fix d98b3264; two-mesh boundary test proves partitioned delivery, red-before-green"},
{"id": 5, "subject": "Task 5: split-topology startup validator (cluster-role => non-DPS)", "classification": "standard", "status": "completed", "commit": "344b0d33", "note": "impl a886d5e6 + review-fix 344b0d33 (resolve EFFECTIVE roles incl OTOPCUA_ROLES fallback matching Program.cs, case-normalize -> guardrail can't be defeated by role-source/case mismatch); 16/16 filter, 124/124 Cluster.Tests"},
{"id": 6, "subject": "Task 6: compiled transport-mode defaults", "classification": "small", "status": "resolved-no-flip", "note": "DECIDED not to flip — blast-radius assessment: high-risk (breaks all integration tests + appsettings profiles + docker-dev site nodes), low-value (Task 5 validator already enforces split-correctness fail-closed). Documented in plan Task 6 + Configuration.md."},
{"id": 7, "subject": "Task 7: rewrite docker-dev rig into three 2-node meshes", "classification": "standard", "status": "completed", "commit": "741ab97b", "note": "3 meshes, per-pair self-first seeds, cluster roles, transports flipped (ClusterClient/Grpc), pair-local secrets note; docker compose config validates; seed SQL already correct (GrpcPort=4056)"},
{"id": 8, "subject": "Task 8: docs sweep — status tables, caveat removal, pair-local secrets note", "classification": "small", "status": "completed", "commit": "8a173bba", "note": "all 7 edits (Redundancy/Config/CLAUDE/2 plans/IManualFailoverService/ClusterRedundancy.razor); subagent died at commit step, edits reviewed+committed by controller; full-solution build clean (0 errors)"},
{"id": 9, "subject": "Task 9: live gate — three meshes, per-pair redundancy, cross-mesh transports","classification": "high-risk", "status": "completed", "note": "PASSED — all 8 legs green (docs/plans/2026-07-24-mesh-phase6-live-gate.md). Caught + fixed-forward 3 defects: (1) product boot-crash StackOverflow — cluster-redundancy singleton resolved IClusterRoleInfo eagerly inside the AddAkka lambda while the ActorSystem was building (fix 3a4ed8dd: derive scope from AkkaClusterOptions); (2) rig — site nodes crashed on LDAP options (no Security__Ldap__* block); (3) rig — site pairs split-brained on simultaneous start (both self-first seeds), fixed with a founder healthcheck + service_healthy gating (2+3 in 792f28ec). Legs: 3 isolated meshes, one Primary/pair 250/240, deploy sealed acks=6 over per-cluster ClusterClient, telemetry gRPC :4056 up + reconnect, reconciler own-cluster-scoped, secrets pair-local, split validator refuses Dps, failover+auto-down-1v1 survival proven. Master 792f28ec."}
],
"lastUpdated": "2026-07-24T11:10:00Z"
}
@@ -0,0 +1,94 @@
# Per-cluster mesh Phase 7 — failover drills + closing live gates
> **Final phase of the per-cluster mesh program.** Run the failover drill per pair type, close the two
> outstanding live gates (auto-down 1-vs-1 crash-the-oldest; self-first cold-start-alone), re-verify the
> manual-failover button, and ship a one-page co-located operator runbook.
>
> Runs against merged master on the docker-dev three-mesh rig (same rig as the Phase 6 gate).
## Scope clarified by recon
- **Failover mechanism is the same for every pair**: the oldest Up `driver` member leaves/dies, the peer
becomes oldest, takes the `cluster-{ClusterId}` redundancy singleton, and advertises ServiceLevel 250.
- **Manual-failover button (`IManualFailoverService`) is MAIN-scoped by construction** — it acts on the
local node's `Cluster.State` (graceful `Leave` of the oldest Up driver), and the AdminUI runs only on
the central (MAIN) pair. Site pairs are **driver-only, no UI**; they fail over via **auto-down** (kill)
only. This is the documented pair-local caveat from Phase 6, not a gap to fix.
- **Graceful (`docker stop`, SIGTERM → CoordinatedShutdown → cluster Leave) vs. ungraceful (`docker kill`,
SIGKILL → peer auto-downs)** are the two failover paths. Manual failover = graceful Leave; a crashed
node = auto-down.
## Drills
| # | Drill | Status | Evidence |
|---|---|---|---|
| D1 | MAIN graceful failover (manual-failover path) — Primary leaves, peer takes over | ✅ PASS | AdminUI "Trigger failover" on `/clusters/MAIN/redundancy`: central-2 "Exiting completed" → central-1 became Primary (250) → central-2 restarted, "Welcome from central-1" (rejoined, no split). Roles swapped 250↔240 |
| D2 | SITE-A auto-down failover, BOTH directions (kill Primary, then kill new Primary) | ✅ PASS | leg 8 killed site-a-1→site-a-2 Primary; D2 killed site-a-2→site-a-1 Primary (250), survived alone |
| D3 | SITE-B auto-down failover (crash-the-oldest) | ✅ PASS | killed site-b-1 (Primary/oldest) → site-b-2 "Member removed [site-b-1]", BecomingOldest→Oldest, 250 |
| D4 | Gate (a): auto-down 1-vs-1 crash-the-oldest — survivor stays Up, does not self-down | ✅ PASS | every survivor across D2/D3/D5/leg8 stayed Up (e.g. site-a-1 "Up 49 minutes", site-b-2 "Up 56 minutes") — no self-down. Closes the gate deferred since Phase 0a |
| D5 | Gate (b): self-first cold-start-alone — a node boots ALONE (partner down) and forms its mesh | ✅ PASS | stopped both site-b, started only site-b-1: "JOINING itself … forming a new cluster" → leader → Up → serves 250 as sole Primary |
| D6 | Recovery — a downed/restarted node rejoins its pair as Secondary (240) | ✅ PASS | central-2, site-a-2, site-b-2 all restarted → "Welcome from <founder>" → rejoined 240 |
**Phase 7 exit gate: MET.** All 6 drills pass on every pair type. Final fleet: three healthy 2-node
pairs, each one Primary (250) + one Secondary (240). Roles are wherever the last drill left them — each
pair has exactly one Primary, which is what matters.
## Findings / notes
- **Manual-failover button is MAIN-only, by construction** (acts on the local `Cluster.State`; AdminUI
runs only on central). The UI now correctly states "this application Cluster's own Primary … each
cluster's redundant pair runs its own independent 2-node Akka mesh" (Phase 6 text, verified live).
Site pairs (driver-only, no UI) fail over via auto-down (crash) only — there is no manual-failover
surface for them, and that is correct for the topology.
- **Graceful failover restarts the node in-place and it rejoins** (no split), because a single peer that
is already Up answers the restarting node's InitJoin. The **simultaneous cold-start** split (Phase 6
finding) only bites when BOTH pair nodes start from nothing at once — see the runbook mitigation.
- **Carried Phase-6 finding — simultaneous cold-start of both pair VMs — NOW CLOSED by a product guard
(`279d1d0f`).** On the real co-located VMs a site's two VMs can power-cycle together; two self-first
seeds can then each form a 1-node cluster (split brain, two Primaries in one pair). Two mitigations now
exist: (1) **operational** — stagger the two VMs' service-manager start / start the founder first (the
docker-dev `depends_on: service_healthy` serialization, still used by site-b); (2) **product** — the
opt-in `Cluster:BootstrapGuard` (default off), which makes the lower-address node the founder and the
higher node probe-then-join, arbitrating the race inside the join decision. Live-gated on the site-a
pair (guard on, serialization removed): simultaneous start → 250/240, no split; higher-node cold-start
-alone → self-forms → 250. See `docs/Redundancy.md` §"Bootstrap guard". The guard does NOT need the
compose `depends_on` that production hardware lacks — it is the production-faithful fix.
---
## Operator runbook — co-located OtOpcUa + ScadaBridge site failover (one page)
**Topology.** Each site = 2 Windows VMs. Each VM runs **one OtOpcUa driver node** (`driver,cluster-SITE-X`,
Akka `:4053`, OPC UA `:4840`) **and one ScadaBridge site node** — two independent 2-node Akka clusters
sharing the hardware, never a shared mesh. Central = 2 VMs, each an OtOpcUa `admin,driver,cluster-MAIN`
node (hosts the AdminUI) + a ScadaBridge central node.
**Normal state.** Each pair: one node Primary (OPC UA ServiceLevel **250**), one Secondary (**240**).
Check from any OPC UA client: `otopcua-cli redundancy -u opc.tcp://<node>:4840`.
**Planned failover (move the Primary off a VM — e.g. for patching):**
- *Central (MAIN) pair:* AdminUI → **Clusters → MAIN → Redundancy → Trigger failover → Confirm**. The
Primary gracefully leaves, restarts, rejoins as Secondary; its peer becomes Primary (250) in a few
seconds. OPC UA clients re-select the new Primary automatically.
- *Site pairs (SITE-A/SITE-B):* no UI (driver-only). Stop the OtOpcUa service on the Primary VM
(graceful SIGTERM) — the peer auto-downs it and becomes Primary. Restart the service to rejoin.
**Unplanned failover (a VM dies).** The peer detects the loss (~10-15 s: failure detector + auto-down),
removes the dead node, takes the redundancy singleton, and advertises 250. **A 1-node pair keeps
serving** (auto-down 1-vs-1 survival — verified). Because the VM is shared, **both products fail over
together**: confirm the ScadaBridge site node on the surviving VM also took over (its own runbook).
**Recovery.** Bring the dead VM back. The OtOpcUa node cold-starts, finds its peer Up, and rejoins as
Secondary (240) — "Welcome from <peer>" in its log. No manual step.
**⚠️ Both VMs down at once (site power event).** Start the VMs **staggered**, or bring the **designated
founder VM up first** and wait for its OtOpcUa node to reach ServiceLevel 250 before starting the second
VM. Starting both OtOpcUa services simultaneously can split the pair into two 1-node clusters (two
Primaries). If that happens: stop the OtOpcUa service on the non-founder VM, confirm the founder is the
lone Primary (250), then restart the non-founder — it rejoins as Secondary.
**Health checks.**
- ServiceLevel per node: `otopcua-cli redundancy -u opc.tcp://<node>:4840` (250 Primary / 240 Secondary).
- MAIN membership + Primary: AdminUI → Clusters → MAIN → Redundancy.
- A pair showing **250/250** = split brain — apply the "both VMs down" recovery above.
- A pair showing **240/240 or a lone 240** = no Primary elected — check the singleton host is Up.
+729
View File
@@ -0,0 +1,729 @@
# Modbus RTU Driver Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans (or subagent-driven-development) to implement this plan task-by-task.
**Goal:** Add a Modbus **RTU-over-TCP** transport (RTU CRC-16 framing tunnelled to a serial→Ethernet gateway) to the existing `ModbusDriver`, selected by a new `Transport` config field — reusing every codec/planner/health/materialisation surface unchanged.
**Architecture:** Purely additive behind the existing `IModbusTransport` seam. The application protocol (function codes, codecs, planner, coalescing, write path, probe, materialisation, HistoryRead) is byte-for-byte identical across TCP and RTU — only the wire framing differs (`[addr][PDU][CRC-16]`, no MBAP, no TxId). Work = a CRC-16 helper + a length-less FC-aware framer + one new socket transport that composes a socket lifecycle extracted from `ModbusTcpTransport` + a transport-mode selector on the config + one AdminUI field. **Zero** changes to codecs/planner/health.
**Tech Stack:** existing `ZB.MOM.WW.OtOpcUa.Driver.Modbus` (+ `.Contracts`, `.Addressing`), xUnit + Shouldly, `pymodbus[simulator]==3.13.0` Docker fixture (RTU-framed-TCP profile; stdlib fallback server modelled on `exception_injector.py`), Blazor `ModbusDriverForm.razor` (AdminUI). No new NuGet dependency.
**Source design:** docs/plans/2026-07-15-modbus-rtu-driver-design.md
**Program design:** docs/plans/2026-07-15-driver-expansion-program-design.md (§3 shared contract, §7 fixtures) — this is **Wave 1**.
**Progress tracker:** docs/plans/2026-07-24-driver-expansion-tracking.md
## Scope (P1 shippable v1 only)
RTU-over-TCP via a serial→Ethernet gateway (Moxa/Digi/Lantronix), RTU CRC-16 (poly `0xA001`) + FC-aware length framing, a `Transport` selector (`Tcp` | `RtuOverTcp`) on the Modbus config, the `rtu_over_tcp` pymodbus docker profile, and the AdminUI selector. Read **and** write supported (same as TCP).
## Deferred / out of scope
- **Direct-serial transport** (`ModbusRtuTransport` / `System.IO.Ports`) — descoped (user, 2026-07-15). RTU buses are reached exclusively via a serial→Ethernet gateway. Design record kept in **design §2b / §9 P2**; the `ModbusRtuFraming` built here is byte-stream-agnostic so a future serial leg would reuse it unchanged. Do **not** number-squat an `Rtu` enum member for it.
- **Modbus ASCII** — a third framing (`:`-delimited hex + LRC); would be another `IModbusTransport` if ever needed (design §2c). Not planned.
- **Per-tag `TagConfig` changes** — none. `ModbusTagDefinition`/`ModbusTagDto` already carry the per-tag `UnitId` override that makes an RS-485 multi-drop bus work; the read planner already refuses to coalesce across UnitIds. Additions are **driver-level only** (design §4).
- **`ModbusDriver.BuildSlaveHostName` change** — NOT needed. It already emits `host:port/unit{n}`, the correct per-slave resilience key for a gateway (design §6). The design §1 table listed it only for the descoped serial (COM) case.
## Cross-cutting rules (program §3.1 — mandatory)
- **Enum-serialization trap**`Transport` MUST round-trip as a **name string**, never a number, on both the AdminUI serializer (`ModbusDriverForm._jsonOpts` already carries `JsonStringEnumConverter`) and the factory (DTO field typed `string?`, parsed via the existing `ParseEnum<T>` — mirror `Family`/`MelsecSubFamily`; **do not** type the DTO field as the enum). Guarded by an explicit `"transport":"RtuOverTcp"` string assertion (design §5).
- **Per-op deadline (R2-01)** — the RTU transport MUST run every transaction under a linked-CTS `CancelAfter(Options.Timeout)` — a frozen gateway must never wedge a poll. There is **no MBAP length field to trust**; FC-aware sizing is the only length signal, so getting it right (or timing out) is the top correctness risk (design §7).
- **Single-flight mandatory on RTU** — no TxId means at most one transaction on the bus at a time. Keep the `_gate` `SemaphoreSlim` in the new transport.
- **Ctor is connection-free** — the transport constructor must not connect; all I/O happens in `ConnectAsync`/`SendAsync`.
- **Live-verify discipline** — the picker/editor + deploy path get a docker-dev `/run` before trust (Task 10).
---
## Task 0: Add the `ModbusTransportMode` enum
**Classification:** trivial
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 1, Task 3
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Contracts/ModbusTransportMode.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests/ModbusTransportModeTests.cs`
The enum lives in `.Contracts` (backend-dep-free), namespace `ZB.MOM.WW.OtOpcUa.Driver.Modbus` — same namespace as `ModbusDriverOptions`.
**Step 1 — failing test:**
```csharp
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests;
public sealed class ModbusTransportModeTests
{
[Fact]
public void Tcp_is_the_default_zero_member()
=> ((ModbusTransportMode)0).ShouldBe(ModbusTransportMode.Tcp);
[Fact]
public void Exactly_two_members_ship_in_v1()
=> Enum.GetNames<ModbusTransportMode>().ShouldBe(["Tcp", "RtuOverTcp"]);
}
```
**Step 2 — run & expect FAIL** (type does not exist / does not compile):
`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests --filter "FullyQualifiedName~ModbusTransportModeTests"`
**Step 3 — minimal impl:**
```csharp
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus;
/// <summary>
/// Wire transport for a Modbus driver instance. <see cref="Tcp"/> = Modbus/TCP (MBAP + TxId);
/// <see cref="RtuOverTcp"/> = RTU framing (address + CRC-16, no MBAP) tunnelled over a socket to
/// a serial→Ethernet gateway. Default <see cref="Tcp"/> — existing configs omit the field.
/// A direct-serial <c>Rtu</c> member is intentionally NOT reserved here (descoped 2026-07-15);
/// do not number-squat it.
/// </summary>
public enum ModbusTransportMode
{
/// <summary>Modbus/TCP — 7-byte MBAP header + transaction id (the historical default).</summary>
Tcp,
/// <summary>RTU framing (<c>[addr][PDU][CRC-16]</c>) tunnelled over a socket to a serial→Ethernet gateway.</summary>
RtuOverTcp,
}
```
**Step 4 — run & expect PASS.**
**Step 5 — commit:**
`git commit -am "feat(modbus-rtu): add ModbusTransportMode enum (Tcp | RtuOverTcp)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW"`
---
## Task 1: `ModbusCrc` — CRC-16 (poly 0xA001) helper + golden vectors
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 0, Task 3
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusCrc.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests/ModbusCrcTests.cs`
CRC-16/MODBUS: reflected poly `0xA001`, init `0xFFFF`, appended **low byte first**. This is the top-of-stack of the length-less framer — golden vectors are the whole point.
**Step 1 — failing test** (table-driven against published Modbus CRC vectors):
```csharp
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests;
public sealed class ModbusCrcTests
{
// Known CRC-16/MODBUS vectors (init 0xFFFF, poly 0xA001). CRC returned as ushort;
// on the wire it is appended low-byte-first.
[Theory]
// FC03 read HR: unit 1, addr 0, qty 1 -> CRC 0x0A84 (bytes 84 0A on the wire)
[InlineData(new byte[] { 0x01, 0x03, 0x00, 0x00, 0x00, 0x01 }, (ushort)0x0A84)]
// "123456789" canonical check value for CRC-16/MODBUS = 0x4B37
[InlineData(new byte[] { 0x31,0x32,0x33,0x34,0x35,0x36,0x37,0x38,0x39 }, (ushort)0x4B37)]
public void Compute_matches_known_vectors(byte[] frame, ushort expected)
=> ModbusCrc.Compute(frame).ShouldBe(expected);
[Fact]
public void AppendLowByteFirst_writes_lo_then_hi()
{
var body = new byte[] { 0x01, 0x03, 0x00, 0x00, 0x00, 0x01 };
var withCrc = ModbusCrc.Append(body);
withCrc.Length.ShouldBe(body.Length + 2);
withCrc[^2].ShouldBe((byte)0x84); // CRC low byte
withCrc[^1].ShouldBe((byte)0x0A); // CRC high byte
}
}
```
**Step 2 — run & expect FAIL:**
`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests --filter "FullyQualifiedName~ModbusCrcTests"`
**Step 3 — minimal impl** (`ModbusCrc.cs`): a `static ushort Compute(ReadOnlySpan<byte>)` (init `0xFFFF`, xor byte, 8× shift-right, xor `0xA001` on carry) and `static byte[] Append(ReadOnlySpan<byte> body)` returning `body + [crc & 0xFF, crc >> 8]`. ~30 lines. If a golden vector disagrees, fix the CRC math — never the vector.
**Step 4 — run & expect PASS.**
**Step 5 — commit:**
`git commit -am "feat(modbus-rtu): add ModbusCrc CRC-16/MODBUS helper + golden vectors
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW"`
---
## Task 2: `ModbusRtuFraming` — ADU build + FC-aware length-less deframe
**Classification:** high-risk (framing)
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusRtuFraming.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests/ModbusRtuFramingTests.cs`
The one genuinely new correctness risk (design §2a/§7): RTU frames carry **no length field**, so response size is parsed from the function code. Read `addr(1)`+`fc(1)`, then: **exception** (`fc & 0x80`) → `excCode(1)+CRC(2)`; **reads FC01/02/03/04**`byteCount(1)` then `byteCount + CRC(2)`; **write echoes FC05/06/15/16** → fixed `4 + CRC(2)`. Validate trailing CRC, strip `addr`+`CRC`, return the bare PDU `[fc, ...data]`. CRC mismatch / truncation → `ModbusTransportDesyncException`; exception PDU (after CRC-validate) → `ModbusException`. Test directly against an in-memory `MemoryStream` of canned bytes.
**Step 1 — failing test:**
```csharp
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests;
public sealed class ModbusRtuFramingTests
{
private static MemoryStream Canned(params byte[] frame) => new(frame);
[Fact]
public void BuildAdu_prefixes_unit_and_appends_crc()
{
var adu = ModbusRtuFraming.BuildAdu(0x01, new byte[] { 0x03, 0x00, 0x00, 0x00, 0x01 });
adu.ShouldBe(new byte[] { 0x01, 0x03, 0x00, 0x00, 0x00, 0x01, 0x84, 0x0A });
}
[Fact]
public async Task ReadResponse_FC03_returns_bare_pdu()
{
// addr=01 fc=03 byteCount=02 data=00 0A + CRC(lo,hi)
var body = new byte[] { 0x01, 0x03, 0x02, 0x00, 0x0A };
var frame = ModbusCrc.Append(body);
await using var s = Canned(frame);
var pdu = await ModbusRtuFraming.ReadResponsePduAsync(s, 0x01, TestContext.Current.CancellationToken);
pdu.ShouldBe(new byte[] { 0x03, 0x02, 0x00, 0x0A }); // fc + byteCount + data, no addr/CRC
}
[Fact]
public async Task ReadResponse_exception_frame_throws_ModbusException()
{
// addr=01 fc=0x83 exc=0x02 + CRC
var frame = ModbusCrc.Append(new byte[] { 0x01, 0x83, 0x02 });
await using var s = Canned(frame);
var ex = await Should.ThrowAsync<ModbusException>(
ModbusRtuFraming.ReadResponsePduAsync(s, 0x01, TestContext.Current.CancellationToken));
ex.FunctionCode.ShouldBe((byte)0x03);
ex.ExceptionCode.ShouldBe((byte)0x02);
}
[Fact]
public async Task ReadResponse_bad_crc_throws_desync()
{
var frame = ModbusCrc.Append(new byte[] { 0x01, 0x03, 0x02, 0x00, 0x0A });
frame[^1] ^= 0xFF; // corrupt CRC high byte
await using var s = Canned(frame);
await Should.ThrowAsync<ModbusTransportDesyncException>(
ModbusRtuFraming.ReadResponsePduAsync(s, 0x01, TestContext.Current.CancellationToken));
}
[Fact]
public async Task ReadResponse_FC06_write_echo_returns_four_byte_pdu()
{
// addr=01 fc=06 addr=00 07 value=00 2A + CRC -> PDU = 06 00 07 00 2A
var frame = ModbusCrc.Append(new byte[] { 0x01, 0x06, 0x00, 0x07, 0x00, 0x2A });
await using var s = Canned(frame);
var pdu = await ModbusRtuFraming.ReadResponsePduAsync(s, 0x01, TestContext.Current.CancellationToken);
pdu.ShouldBe(new byte[] { 0x06, 0x00, 0x07, 0x00, 0x2A });
}
}
```
**Step 2 — run & expect FAIL:**
`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests --filter "FullyQualifiedName~ModbusRtuFramingTests"`
**Step 3 — minimal impl** (`ModbusRtuFraming.cs`): `static byte[] BuildAdu(byte unitId, ReadOnlySpan<byte> pdu)` = `ModbusCrc.Append([unitId, ..pdu])`; `static async Task<byte[]> ReadResponsePduAsync(Stream, byte expectedUnit, CancellationToken)` doing the FC-aware read described above with a `ReadExactlyAsync` helper (mirror `ModbusTcpTransport.ReadExactlyAsync`). Reuse `ModbusException` + `ModbusTransportDesyncException` (both already in the project). Keep framing byte-stream-agnostic (no socket knowledge) so a future serial transport can reuse it.
**Step 4 — run & expect PASS.**
**Step 5 — commit:**
`git commit -am "feat(modbus-rtu): add ModbusRtuFraming (ADU build + FC-aware length-less deframe)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW"`
---
## Task 3: Extract `ModbusSocketLifecycle` from `ModbusTcpTransport` (behaviour-preserving)
**Classification:** high-risk (transport / concurrency)
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 0, Task 1, Task 2
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusSocketLifecycle.cs`
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusTcpTransport.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests/ModbusSocketLifecycleTests.cs`
Mechanical refactor: move the IPv4-preference connect (`ConnectAsync`), `EnableKeepAlive`/`ClampToWholeSeconds`, `ConnectWithBackoffAsync`, `TearDownAsync`, and the `_lastSuccessUtc`/idle-disconnect tracking out of `ModbusTcpTransport` into a reusable `ModbusSocketLifecycle` that exposes the current `NetworkStream`. `ModbusTcpTransport` keeps its MBAP `SendOnceAsync` and composes the lifecycle. **Behaviour must be byte-for-byte unchanged** — the existing `ModbusTcpReconnectTests` + `ModbusConnectionOptionsTests` are the regression net; run the whole Modbus.Tests suite green before committing.
**Step 1 — failing test** (pins the extracted surface; `ClampToWholeSeconds` moves with it):
```csharp
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests;
public sealed class ModbusSocketLifecycleTests
{
[Theory]
[InlineData(0.4, 1)] // sub-second rounds up to 1 (the int-cast truncation guard)
[InlineData(2.0, 2)]
[InlineData(-5.0, 1)] // negative clamps to 1
public void ClampToWholeSeconds_rounds_up_min_one(double seconds, int expected)
=> ModbusSocketLifecycle.ClampToWholeSeconds(TimeSpan.FromSeconds(seconds)).ShouldBe(expected);
[Fact]
public async Task Connect_to_dead_port_surfaces_socket_failure()
{
var life = new ModbusSocketLifecycle("127.0.0.1", 1, TimeSpan.FromMilliseconds(300),
autoReconnect: false);
await Should.ThrowAsync<System.Net.Sockets.SocketException>(
life.ConnectAsync(TestContext.Current.CancellationToken));
}
}
```
**Step 2 — run & expect FAIL:**
`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests --filter "FullyQualifiedName~ModbusSocketLifecycleTests"`
**Step 3 — minimal impl:** create `ModbusSocketLifecycle` holding `_host/_port/_timeout/_autoReconnect/_keepAlive/_idleDisconnect/_reconnect`, the `TcpClient`/`NetworkStream`, and `_lastSuccessUtc`; expose `ConnectAsync`, `ConnectWithBackoffAsync`, `TearDownAsync`, `Stream` (current), `MarkSuccess()`, `ShouldReconnectForIdle()`, and the `static int ClampToWholeSeconds`. Re-point `ModbusTcpTransport` to delegate connect/reconnect/teardown/idle to it while keeping MBAP `SendOnceAsync` in place. Move (don't duplicate) `ClampToWholeSeconds` — update its one internal caller/test reference.
**Step 4 — run & expect PASS** — then run the **whole** suite to prove no regression:
`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests`
**Step 5 — commit:**
`git commit -am "refactor(modbus): extract ModbusSocketLifecycle from ModbusTcpTransport (behaviour-preserving)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW"`
---
## Task 4: `ModbusRtuOverTcpTransport` — RTU framing over the socket lifecycle
**Classification:** high-risk (framing / transport / concurrency)
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusRtuOverTcpTransport.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests/ModbusRtuOverTcpTransportTests.cs`
`IModbusTransport` implementation: composes `ModbusSocketLifecycle` (Task 3) for connect/reconnect/keepalive/idle and `ModbusRtuFraming` (Task 2) for send/receive. Delta from `ModbusTcpTransport`: CRC framing instead of MBAP, **no TxId**. Keep the `_gate` `SemaphoreSlim` single-flight (mandatory — no TxId), the single reconnect-retry shape, and a linked-CTS `CancelAfter(Options.Timeout)` per-op deadline (design §7, R2-01). **Seam choice for unit test:** add an `internal` test ctor accepting a pre-connected `Stream` (an in-memory duplex fake) that bypasses `ConnectAsync` — this is "the fake one level below the `IModbusTransport` fakes" the design §8 calls for, and lets the send/receive orchestration + single-flight + deadline be tested with no socket.
**Step 1 — failing test** (duplex fake: canned response stream + capture of written request bytes):
```csharp
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests;
public sealed class ModbusRtuOverTcpTransportTests
{
[Fact]
public async Task SendAsync_writes_rtu_adu_and_returns_deframed_pdu()
{
// Response the fake gateway will emit: FC03, 1 reg = 0x000A.
var response = ModbusCrc.Append(new byte[] { 0x01, 0x03, 0x02, 0x00, 0x0A });
var fake = new CapturingDuplexStream(response);
await using var transport = ModbusRtuOverTcpTransport.ForTest(fake, TimeSpan.FromSeconds(2));
var pdu = await transport.SendAsync(0x01, new byte[] { 0x03, 0x00, 0x00, 0x00, 0x01 },
TestContext.Current.CancellationToken);
pdu.ShouldBe(new byte[] { 0x03, 0x02, 0x00, 0x0A });
// The request went out as an RTU ADU: unit + pdu + CRC, NO MBAP header.
fake.Written.ShouldBe(new byte[] { 0x01, 0x03, 0x00, 0x00, 0x00, 0x01, 0x84, 0x0A });
}
[Fact]
public async Task SendAsync_exception_pdu_surfaces_ModbusException()
{
var response = ModbusCrc.Append(new byte[] { 0x01, 0x83, 0x02 });
var fake = new CapturingDuplexStream(response);
await using var transport = ModbusRtuOverTcpTransport.ForTest(fake, TimeSpan.FromSeconds(2));
await Should.ThrowAsync<ModbusException>(
transport.SendAsync(0x01, new byte[] { 0x03, 0x00, 0x00, 0x00, 0x01 },
TestContext.Current.CancellationToken));
}
[Fact]
public async Task SendAsync_stalled_gateway_hits_per_op_deadline()
{
var fake = new CapturingDuplexStream(respondBytes: Array.Empty<byte>(), stall: true);
await using var transport = ModbusRtuOverTcpTransport.ForTest(fake, TimeSpan.FromMilliseconds(200));
await Should.ThrowAsync<ModbusTransportDesyncException>(
transport.SendAsync(0x01, new byte[] { 0x03, 0x00, 0x00, 0x00, 0x01 },
TestContext.Current.CancellationToken));
}
}
```
(The `CapturingDuplexStream` test double — a `Stream` that records writes and replays `respondBytes` on read, or blocks forever when `stall` — lives in the test file.)
**Step 2 — run & expect FAIL:**
`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests --filter "FullyQualifiedName~ModbusRtuOverTcpTransportTests"`
**Step 3 — minimal impl:** production ctor takes the same connection params as `ModbusTcpTransport` and builds a `ModbusSocketLifecycle`; `internal static ForTest(Stream, TimeSpan)` injects a pre-connected stream. `SendAsync`: `_gate.WaitAsync`; idle-reconnect check (skipped in test seam); write `ModbusRtuFraming.BuildAdu(unitId, pdu)`, flush, `ModbusRtuFraming.ReadResponsePduAsync(...)` under a linked `CancelAfter(_timeout)`; timeout-vs-caller-cancel distinction + teardown mirror `ModbusTcpTransport.SendOnceAsync`; single reconnect-retry on socket-level failure.
**Step 4 — run & expect PASS.**
**Step 5 — commit:**
`git commit -am "feat(modbus-rtu): add ModbusRtuOverTcpTransport (RTU framing over socket lifecycle)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW"`
---
## Task 5: `ModbusTransportFactory.Create` — switch on `Transport`
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 6
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusTransportFactory.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests/ModbusTransportFactoryTests.cs`
One place that maps `ModbusDriverOptions` → the right `IModbusTransport`, consumed by both the driver default closure (Task 7) and the probe (Task 7). `Tcp``ModbusTcpTransport`, `RtuOverTcp``ModbusRtuOverTcpTransport`; both get `Host/Port/Timeout/AutoReconnect/KeepAlive/IdleDisconnect/Reconnect`.
**Step 1 — failing test:**
```csharp
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests;
public sealed class ModbusTransportFactoryTests
{
[Fact]
public void Tcp_mode_builds_tcp_transport()
=> ModbusTransportFactory.Create(new ModbusDriverOptions { Transport = ModbusTransportMode.Tcp })
.ShouldBeOfType<ModbusTcpTransport>();
[Fact]
public void RtuOverTcp_mode_builds_rtu_transport()
=> ModbusTransportFactory.Create(new ModbusDriverOptions { Transport = ModbusTransportMode.RtuOverTcp })
.ShouldBeOfType<ModbusRtuOverTcpTransport>();
[Fact]
public void Default_options_build_tcp_transport()
=> ModbusTransportFactory.Create(new ModbusDriverOptions())
.ShouldBeOfType<ModbusTcpTransport>();
}
```
(Depends on Task 6's `Transport` property existing on `ModbusDriverOptions` — sequence Task 6 or add the property first. If Task 5 lands before Task 6, add the property in this task instead.)
**Step 2 — run & expect FAIL.**
`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests --filter "FullyQualifiedName~ModbusTransportFactoryTests"`
**Step 3 — minimal impl:** `static IModbusTransport Create(ModbusDriverOptions o)` = `switch (o.Transport) { RtuOverTcp => new ModbusRtuOverTcpTransport(...), _ => new ModbusTcpTransport(...) }`, passing the shared connection params.
**Step 4 — run & expect PASS.**
**Step 5 — commit:**
`git commit -am "feat(modbus-rtu): add ModbusTransportFactory switch on Transport mode
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW"`
---
## Task 6: Add `Transport` to options + DTO + factory (string-enum guard)
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 5
**Files:**
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Contracts/ModbusDriverOptions.cs`
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusDriverFactoryExtensions.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests/ModbusTransportConfigRoundTripTests.cs`
Add `public ModbusTransportMode Transport { get; init; } = ModbusTransportMode.Tcp;` to `ModbusDriverOptions`. On the DTO: add `public string? Transport { get; init; }` (typed **`string?`**, per the Modbus factory pattern — design §5) and map it with the existing `ParseEnum<ModbusTransportMode>(...)` helper, defaulting to `Tcp` when null. The guard test asserts the string wire form.
**Step 1 — failing test:**
```csharp
using System.Text.Json;
using System.Text.Json.Serialization;
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests;
public sealed class ModbusTransportConfigRoundTripTests
{
// Mirrors the AdminUI ModbusDriverForm serializer: camelCase + string enums.
private static readonly JsonSerializerOptions _adminOpts = new()
{
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
Converters = { new JsonStringEnumConverter() },
};
[Fact]
public void Transport_serializes_as_a_string_not_a_number()
{
var opts = new ModbusDriverOptions { Host = "10.20.0.50", Port = 4001, Transport = ModbusTransportMode.RtuOverTcp };
var json = JsonSerializer.Serialize(opts, _adminOpts);
json.ShouldContain("\"transport\":\"RtuOverTcp\"");
json.ShouldNotContain("\"transport\":1", Case.Sensitive);
}
[Fact]
public void Factory_binds_RtuOverTcp_from_string_config()
{
const string json = """{ "host": "10.20.0.50", "port": 4001, "transport": "RtuOverTcp" }""";
using var driver = ModbusDriverFactoryExtensions.CreateInstance("mb-rtu", json);
var opts = (ModbusDriverOptions)typeof(ModbusDriver)
.GetField("_options", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance)!
.GetValue(driver)!;
opts.Transport.ShouldBe(ModbusTransportMode.RtuOverTcp);
}
[Fact]
public void Factory_defaults_Transport_to_Tcp_when_omitted()
{
const string json = """{ "host": "10.0.0.10" }""";
using var driver = ModbusDriverFactoryExtensions.CreateInstance("mb-tcp", json);
var opts = (ModbusDriverOptions)typeof(ModbusDriver)
.GetField("_options", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance)!
.GetValue(driver)!;
opts.Transport.ShouldBe(ModbusTransportMode.Tcp);
}
}
```
**Step 2 — run & expect FAIL:**
`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests --filter "FullyQualifiedName~ModbusTransportConfigRoundTripTests"`
**Step 3 — minimal impl:** add the `Transport` property (options) + DTO `string?` field + factory mapping `Transport = dto.Transport is null ? ModbusTransportMode.Tcp : ParseEnum<ModbusTransportMode>(dto.Transport, "<driver-level>", driverInstanceId, "Transport")`.
**Step 4 — run & expect PASS.**
**Step 5 — commit:**
`git commit -am "feat(modbus-rtu): bind Transport mode on options/DTO/factory (string-enum guard)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW"`
---
## Task 7: Wire the driver default closure + probe to `ModbusTransportFactory`
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 8
**Files:**
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusDriver.cs`
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusDriverProbe.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests/ModbusTransportWiringTests.cs`
Replace the driver ctor's hardcoded `new ModbusTcpTransport(...)` default closure with `o => ModbusTransportFactory.Create(o)`. In the probe, replace the hardcoded `new ModbusTcpTransport(host, port, timeout, autoReconnect: false)` (line ~77) with a factory build carrying `Transport` (parse it into `ProbeTarget` from the DTO), `autoReconnect: false`. Now an `RtuOverTcp`-authored config probes over RTU framing — matching how it will actually run.
**Step 1 — failing test** (the default closure honours the mode; the probe carries it):
```csharp
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests;
public sealed class ModbusTransportWiringTests
{
private static IModbusTransport BuildDefaultTransport(ModbusDriverOptions opts)
{
using var driver = new ModbusDriver(opts, "wire-test");
var factory = (Func<ModbusDriverOptions, IModbusTransport>)typeof(ModbusDriver)
.GetField("_transportFactory", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance)!
.GetValue(driver)!;
return factory(opts);
}
[Fact]
public void Default_closure_builds_rtu_transport_for_RtuOverTcp()
=> BuildDefaultTransport(new ModbusDriverOptions { Transport = ModbusTransportMode.RtuOverTcp })
.ShouldBeOfType<ModbusRtuOverTcpTransport>();
[Fact]
public void Default_closure_builds_tcp_transport_for_Tcp()
=> BuildDefaultTransport(new ModbusDriverOptions())
.ShouldBeOfType<ModbusTcpTransport>();
}
```
(If `_transportFactory` reflection proves brittle, assert instead that a `RtuOverTcp` driver initialised against an unreachable host degrades without ever constructing an MBAP frame — but the closure-reflection test is the tightest.)
**Step 2 — run & expect FAIL:**
`dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests --filter "FullyQualifiedName~ModbusTransportWiringTests"`
**Step 3 — minimal impl:** driver ctor default = `?? (o => ModbusTransportFactory.Create(o))`; probe: add `ModbusTransportMode Transport` to `ProbeTarget`, parse `dto.Transport` via `ParseEnum`, and build the probe transport through `ModbusTransportFactory.Create(new ModbusDriverOptions { Host, Port, Timeout, Transport, AutoReconnect = false })`.
**Step 4 — run & expect PASS** (+ run the full Modbus.Tests suite green).
**Step 5 — commit:**
`git commit -am "feat(modbus-rtu): route driver + probe transports through ModbusTransportFactory
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW"`
---
## Task 8: AdminUI — `Transport` selector on `ModbusDriverForm`
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 7
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Shared/Drivers/Forms/ModbusDriverForm.razor`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/Uns/ModbusDriverFormModelTests.cs`
Add a `Transport` `<InputSelect>` to the **Protocol** panel (design's target `ModbusDriverPage.razor` is retired; `ModbusDriverForm.razor` is the live host — it serializes `ModbusDriverOptions` directly through `_jsonOpts`, which already carries `JsonStringEnumConverter`, so `Transport` emits as a name string automatically; `Transport` is **not** an endpoint key so it is not stripped by `GetConfigJson`). Add `public ModbusTransportMode Transport { get; set; } = ModbusTransportMode.Tcp;` to `FormModel`, map it in `FromOptions`/`ToOptions`. Host/Port stay on `ModbusDeviceForm` — a `RtuOverTcp` gateway uses the same `host:port` shape. Extend the existing `ModbusDriverFormModelTests` (round-trip + string-enum guard).
**Step 1 — failing test** (append to `ModbusDriverFormModelTests`):
```csharp
[Fact]
public void Round_trip_preserves_Transport_mode()
{
var form = new ModbusDriverForm.FormModel { Transport = ModbusTransportMode.RtuOverTcp };
var json = JsonSerializer.Serialize(form.ToOptions(), JsonOpts);
var back = ModbusDriverForm.FormModel.FromOptions(
JsonSerializer.Deserialize<ModbusDriverOptions>(json, JsonOpts)!);
back.Transport.ShouldBe(ModbusTransportMode.RtuOverTcp);
json.ShouldContain("\"transport\":\"RtuOverTcp\""); // name string, never a number
}
```
**Step 2 — run & expect FAIL:**
`dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests --filter "FullyQualifiedName~ModbusDriverFormModelTests"`
**Step 3 — minimal impl:** add the `Transport` `FormModel` property + `FromOptions`/`ToOptions` mapping, and an `<InputSelect @bind-Value="_form.Transport" @bind-Value:after="EmitAsync">` looping `Enum.GetValues<ModbusTransportMode>()` in the Protocol panel, with a form-text hint: "RtuOverTcp = talk raw RTU frames to a serial→Ethernet gateway; must match the gateway's mode."
**Step 4 — run & expect PASS.** (AdminUI has no bUnit — the razor binding itself is proven live in Task 10; this model test is the reflection substitute per the AdminUI live-verify memory.)
**Step 5 — commit:**
`git commit -am "feat(modbus-rtu): add Transport selector to the AdminUI Modbus driver form
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW"`
---
## Task 9: Docker `rtu_over_tcp` fixture + RTU-over-TCP integration test
**Classification:** standard (new component / multi-file)
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests/Docker/profiles/rtu_over_tcp.json`
- Modify: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests/Docker/docker-compose.yml`
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests/ModbusRtuOverTcpFixture.cs`
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests/ModbusRtuOverTcpTests.cs`
- Modify (only if the pymodbus RTU-framer confirmation fails): `Docker/Dockerfile` + `Docker/rtu_over_tcp_server.py`
**FIRST sub-step — confirm the unknown (design §8 #1):** verify the pymodbus simulator actually exposes the RTU framer on a TCP server. The checked-in `profiles/standard.json` already sets `server_list.srv.framer: "socket"`, so the knob exists; the RTU profile sets `"framer": "rtu"` (comm stays `"tcp"`). Bring it up and probe with the new driver:
```bash
lmxopcua-fix sync modbus
lmxopcua-fix up modbus rtu_over_tcp
# then run the integration test (below) against MODBUS_RTU_SIM_ENDPOINT=10.100.0.35:5021
```
If pymodbus 3.13's simulator does **not** honour `framer=rtu` on a TCP server (garbled/MBAP-framed replies), **fall back** to a stdlib `rtu_over_tcp_server.py` modelled on the existing `exception_injector.py` (same asyncio TCP-server shape, but frame with `[addr][PDU][CRC-16]` via a Python CRC-16/MODBUS instead of the MBAP header) + a `COPY rtu_over_tcp_server.py` line in the Dockerfile and a `command:` override on the service. Record which path was taken in the test-class summary.
Fixture details: bind host port **`5021`** (NOT the shared `:5020` — the `rtu_over_tcp` service must co-run with `standard` and sidestep the shared-port stale-container trap). Add `labels: { project: lmxopcua }` per the program fixture convention (existing services carry none — `lmxopcua-fix sync` normally owns the label host-side, but add it in-file for this service so it is discoverable). `ModbusRtuOverTcpFixture` mirrors `ModbusSimulatorFixture` but reads `MODBUS_RTU_SIM_ENDPOINT` (default `10.100.0.35:5021`) with the same skip-clean pattern.
**Step 1 — failing test** (`ModbusRtuOverTcpTests.cs` — read + write round-trip over RTU-over-TCP):
```csharp
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests;
[Collection(ModbusRtuOverTcpCollection.Name)]
[Trait("Category", "Integration")]
[Trait("Device", "RtuOverTcp")]
public sealed class ModbusRtuOverTcpTests(ModbusRtuOverTcpFixture sim)
{
[Fact]
public async Task Reads_a_seeded_holding_register_over_rtu_framing()
{
if (sim.SkipReason is not null) Assert.Skip(sim.SkipReason);
var opts = new ModbusDriverOptions
{
Host = sim.Host, Port = sim.Port, UnitId = 1,
Transport = ModbusTransportMode.RtuOverTcp,
Timeout = TimeSpan.FromSeconds(2),
Probe = new ModbusProbeOptions { Enabled = false },
RawTags = ModbusRawTags.Entries([
new ModbusTagDefinition("HR5", ModbusRegion.HoldingRegisters, Address: 5,
DataType: ModbusDataType.UInt16, Writable: false),
]),
};
await using var driver = new ModbusDriver(opts, "modbus-rtu-int");
await driver.InitializeAsync("{}", TestContext.Current.CancellationToken);
var results = await driver.ReadAsync(["HR5"], TestContext.Current.CancellationToken);
results[0].StatusCode.ShouldBe(0u); // Good
results[0].Value.ShouldBe((ushort)5); // HR[5] seeded = address-as-value
}
[Fact]
public async Task Writes_and_reads_back_a_holding_register_over_rtu_framing()
{
if (sim.SkipReason is not null) Assert.Skip(sim.SkipReason);
var opts = new ModbusDriverOptions
{
Host = sim.Host, Port = sim.Port, UnitId = 1,
Transport = ModbusTransportMode.RtuOverTcp,
Timeout = TimeSpan.FromSeconds(2),
Probe = new ModbusProbeOptions { Enabled = false },
RawTags = ModbusRawTags.Entries([
new ModbusTagDefinition("HR200", ModbusRegion.HoldingRegisters, Address: 200,
DataType: ModbusDataType.UInt16, Writable: true),
]),
};
await using var driver = new ModbusDriver(opts, "modbus-rtu-int-w");
await driver.InitializeAsync("{}", TestContext.Current.CancellationToken);
var writes = await driver.WriteAsync([new WriteRequest("HR200", (ushort)4242)],
TestContext.Current.CancellationToken);
writes[0].StatusCode.ShouldBe(0u);
var back = await driver.ReadAsync(["HR200"], TestContext.Current.CancellationToken);
back[0].Value.ShouldBe((ushort)4242);
}
}
```
(Seed `rtu_over_tcp.json` with the same `HR[0..31]=address-as-value` + `HR[200..209]=scratch` layout as `standard.json` so these addresses resolve.)
**Step 2 — run & expect FAIL / SKIP** without the fixture, then bring it up:
```bash
lmxopcua-fix sync modbus && lmxopcua-fix up modbus rtu_over_tcp
MODBUS_RTU_SIM_ENDPOINT=10.100.0.35:5021 dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests --filter "FullyQualifiedName~ModbusRtuOverTcpTests"
```
**Step 3 — minimal impl:** the `rtu_over_tcp.json` profile + compose service (+ stdlib server fallback only if the framer confirmation failed) + the fixture class. Iterate until both facts hold on the wire.
**Step 4 — run & expect PASS** against the live fixture.
**Step 5 — commit:**
`git commit -am "test(modbus-rtu): add rtu_over_tcp pymodbus fixture + RTU-over-TCP integration tests
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW"`
---
## Task 10: Live `/run` verification on docker-dev
**Classification:** standard (live gate)
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Modify (docs only): `docs/plans/2026-07-24-driver-expansion-tracking.md` (record Wave-1 Modbus-RTU live-gate result)
No new code — the live-verify discipline (program §3.1) proves the Razor binding + deploy path that unit tests can't. Against the local docker-dev rig (AdminUI `http://localhost:9200`, login disabled — drive it yourself; do not wait for the user to sign in):
1. Bring up the `rtu_over_tcp` fixture (`lmxopcua-fix up modbus rtu_over_tcp`, host `:5021`).
2. In `/raw`, author a Modbus device pointing Host/Port at the docker host `:5021`, and set the driver **Transport = RtuOverTcp** in the driver-config modal (the Task 8 selector). Run **Test Connect** — expect green (the probe now runs FC03 over RTU framing).
3. Author a holding-register raw tag (e.g. `HR5`), reference it into a UNS equipment, and deploy.
4. Read it back via Client.CLI against the local server, or the `/uns` value panel:
`dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://localhost:4840 -n "ns=2;s=<RawPath-of-HR5>"`
— expect Good quality + the seeded value.
5. Write to a writable register (`HR200`) via Client.CLI `write` and confirm it round-trips.
6. Record PASS/FAIL + evidence in the tracking doc.
**Step 5 — commit:**
`git commit -am "docs(modbus-rtu): record Wave-1 RTU-over-TCP live /run gate result
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW"`
---
## Ordering summary
Framer/CRC unit layer (Tasks 02) → transport (Tasks 34) → factory + config + driver/probe wiring (Tasks 57) → AdminUI selector (Task 8) → docker integration fixture (Task 9) → live `/run` (Task 10). Tasks 1 & 3 can run alongside Task 0; Task 5 alongside Task 6; Task 7 alongside Task 8. Commit after every task. DRY: `ModbusRtuFraming` and `ModbusSocketLifecycle` are each written once and composed; no codec/planner/health code is touched. YAGNI: no direct-serial, no ASCII, no per-tag config, no `BuildSlaveHostName` change.
@@ -0,0 +1,17 @@
{
"planPath": "docs/plans/2026-07-24-modbus-rtu-driver.md",
"tasks": [
{"id": 0, "subject": "Task 0: Add the ModbusTransportMode enum (Tcp | RtuOverTcp)", "status": "pending"},
{"id": 1, "subject": "Task 1: ModbusCrc CRC-16/MODBUS helper + golden vectors", "status": "pending"},
{"id": 2, "subject": "Task 2: ModbusRtuFraming ADU build + FC-aware length-less deframe", "status": "pending", "blockedBy": [1]},
{"id": 3, "subject": "Task 3: Extract ModbusSocketLifecycle from ModbusTcpTransport (behaviour-preserving)", "status": "pending"},
{"id": 4, "subject": "Task 4: ModbusRtuOverTcpTransport (RTU framing over socket lifecycle)", "status": "pending", "blockedBy": [2, 3]},
{"id": 5, "subject": "Task 5: ModbusTransportFactory.Create switch on Transport mode", "status": "pending", "blockedBy": [0, 4]},
{"id": 6, "subject": "Task 6: Bind Transport on options/DTO/factory (string-enum guard)", "status": "pending", "blockedBy": [0]},
{"id": 7, "subject": "Task 7: Route driver default closure + probe through ModbusTransportFactory", "status": "pending", "blockedBy": [5, 6]},
{"id": 8, "subject": "Task 8: AdminUI Transport selector on ModbusDriverForm", "status": "pending", "blockedBy": [6]},
{"id": 9, "subject": "Task 9: Docker rtu_over_tcp fixture + RTU-over-TCP integration test", "status": "pending", "blockedBy": [4, 7]},
{"id": 10, "subject": "Task 10: Live /run verification on docker-dev", "status": "pending", "blockedBy": [8, 9]}
],
"lastUpdated": "2026-07-24"
}
@@ -0,0 +1,969 @@
# MQTT / Sparkplug B Driver Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans (or subagent-driven-development) to implement this plan task-by-task.
**Goal:** Ship a standard Equipment-kind `Mqtt` driver that ingests plain MQTT (P1) then Sparkplug B (P2) into the OtOpcUa dual-namespace address space, with a bespoke observation browser, typed AdminUI editor, and TLS+auth-enforced fixtures.
**Architecture:** Three new `net10.0` projects mirror the OpcUaClient triad — `.Contracts` (transport-free options/DTOs/parser/enums), `.Driver` (subscribe-first `IDriver` holding one live MQTTnet-5 client for its whole lifetime, raising `OnDataChange` from the receive callback, with a hand-rolled reconnect loop since v5 dropped `ManagedMqttClient`), and `.Browser` (a passive `#`/birth observation-window `IBrowseSession`). Sparkplug B decodes vendored Eclipse Tahu protobuf via `Google.Protobuf`/`Grpc.Tools` over that same single client — no SparkplugNet.
**Tech Stack:** MQTTnet v5 (MIT, .NET Foundation, ships `net10.0`), vendored Eclipse Tahu `sparkplug_b.proto` + `Google.Protobuf` (already pinned 3.34.1) + `Grpc.Tools` (already pinned 2.76.0, build-time), bespoke browser.
**Source design:** docs/plans/2026-07-15-mqtt-sparkplug-driver-design.md
**Phases:** P1 (plain MQTT) = Tasks 014 (a complete shippable milestone); P2 (Sparkplug B ingest) = Tasks 1526 (built on the P1 skeleton). Write-through (NCMD/DCMD/plain-publish) is deferred — see "Deferred / out of scope".
---
## Cross-cutting rules (apply to every task — program design §3.1)
- **Enum-serialization trap (systemic bug):** every JSON seam that (de)serializes `MqttDriverOptions` or a tag config — factory, probe, browser, **and** the AdminUI driver-config page + probe DTO — MUST share a `JsonSerializerOptions` carrying `new JsonStringEnumConverter()` + `PropertyNameCaseInsensitive=true` + `UnmappedMemberHandling=Skip`. `MqttMode`, `MqttPayloadFormat`, `protocolVersion` are enums; serialize them as **names**. Mirror `OpcUaClientDriverFactoryExtensions`.
- **Per-op deadline (R2-01 frozen-peer lesson):** every network call (connect / subscribe / probe / rebirth-publish) has a bounded linked-CTS deadline. No unbounded waits on a dead broker.
- **Ctor is connection-free:** `MqttDriver`/`MqttDriverBrowser` constructors touch no network; all connects happen in `InitializeAsync`/`OpenAsync` (the universal-browser `CanBrowse` throwaway-instance pattern depends on this).
- **Secrets from env:** broker `password` is blank in committed JSON, supplied via env (mirror `ServerHistorian__ApiKey`). Never commit or log creds.
- **Broker security — never ship anonymous/public-broker defaults:** default `useTls=true`, real auth. `allowUntrustedServerCertificate` + `caCertificatePath` mirror the ServerHistorian TLS knobs (dev/on-prem escape hatch, off by default). Fixtures run auth+TLS.
- **DriverType string is `"Mqtt"`** everywhere (driver page, probe, factory, editor map, validator, browser). One constant `DriverTypeNames.Mqtt`; grep to enforce (heed the `ModbusTcp`/`Modbus` mismatch lesson).
- **Live-verify discipline:** Razor binding + deploy-inertness bugs pass unit tests. Every picker/editor/deploy path gets a docker-dev `/run` live-verify (Tasks 14 + 26).
- **New-project csproj:** `net10.0`, `Nullable` + `ImplicitUsings` enabled, `TreatWarningsAsErrors=true` opted in per-csproj (not global).
---
## Task 0 (P1): Dependency-validation spike — MQTTnet-5 pin + net10 restore/build
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none (this is the gate — the design's #1 dependency risk; nothing else starts until it is green)
**Files:**
- Modify: `Directory.Packages.props` (add `<PackageVersion Include="MQTTnet" Version="5.x" />`)
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Contracts/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Contracts.csproj` (temporary spike form — references `MQTTnet` only to prove the graph restores; the transport ref is removed in Task 1, `.Contracts` is transport-free)
- Modify: `ZB.MOM.WW.OtOpcUa.slnx` (add the project)
**Why first:** The design (§2.1, §10) makes this the top dependency risk — the repo uses central package management with `CentralPackageTransitivePinningEnabled` **deliberately OFF** (Roslyn 5.0.0/4.12.0 split, `Directory.Packages.props:103` comment + "Transitive pinning breaks Roslyn build" memory). We must prove **MQTTnet v5 restores and builds clean on net10 in this exact pinning configuration** before writing any driver code. Fresh-restore red-vs-local-green (the NU1903 memory) also means we validate a clean restore, not just an incremental one.
### Steps
1. Confirm on NuGet the exact latest **MQTTnet v5** version carrying a `net10.0` TFM; pin that exact version in `Directory.Packages.props` (alphabetical position near `MessagePack`).
2. Scaffold the `.Contracts` csproj (net10.0, nullable, implicit usings, TWAE) with a single `<PackageReference Include="MQTTnet" />` (no `Version` — central management supplies it). Add to `slnx`.
3. Re-verify the two external-library record claims (design §2.1 checkbox): (a) MQTTnet v5 net10.0 TFM exists; (b) SparkplugNet still transitively pins MQTTnet 4.3.x with no net10.0 TFM. Record the confirmed versions in the tasks.json note. The hand-roll decision does not hinge on (b) — it is a confirm-the-record check.
4. Prove a **clean** restore + build:
```bash
dotnet restore src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Contracts # expect: no NU1605/NU1608 version-conflict, no NU1903 audit error
dotnet build src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Contracts # expect: Build succeeded, 0 warnings (TWAE on)
rm -rf ~/.nuget/packages/mqttnet && dotnet restore src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Contracts # fresh-restore proof
```
**Expected:** all three succeed with zero version-conflict / transitive-pin / audit errors. If restore fails on a 4/5 diamond or a missing net10 TFM, STOP — the hand-roll-Tahu decision (§2.1) is vindicated but the MQTTnet-5 pin itself is the blocker; resolve the exact version before proceeding.
5. Commit:
```bash
git commit -am "feat(mqtt): validate MQTTnet-5/net10 restore under central pinning (spike, P1 gate)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW"
```
---
## Task 1 (P1): `.Contracts` — enums + `MqttDriverOptions` DTO
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** none (Task 2 depends on this)
**Files:**
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Contracts/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Contracts.csproj` (drop the temporary MQTTnet ref from Task 0 — `.Contracts` is transport-free; reference only `Core.Abstractions`)
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Contracts/MqttMode.cs` (`Plain | SparkplugB`)
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Contracts/MqttPayloadFormat.cs` (`Json | Raw | Scalar`)
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Contracts/MqttProtocolVersion.cs` (`V311 | V500`)
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Contracts/MqttDriverOptions.cs` (broker conn + mode + `sparkplug`/`plain` sub-objects, §5.1)
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests.csproj` (net10, xUnit + Shouldly)
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests/MqttDriverOptionsTests.cs`
- Modify: `ZB.MOM.WW.OtOpcUa.slnx`
### Steps (TDD)
1. Write the failing test — options round-trip enum-as-name:
```csharp
public sealed class MqttDriverOptionsTests
{
private static readonly JsonSerializerOptions J = new()
{
PropertyNameCaseInsensitive = true,
UnmappedMemberHandling = JsonUnmappedMemberHandling.Skip,
Converters = { new JsonStringEnumConverter() },
};
[Fact]
public void Deserialize_SparkplugConfig_ReadsModeAndSubObject()
{
const string json = """
{ "host":"10.100.0.35","port":8883,"useTls":true,"mode":"SparkplugB",
"sparkplug":{"groupId":"Plant1","hostId":"h1","requestRebirthOnGap":true} }
""";
var o = JsonSerializer.Deserialize<MqttDriverOptions>(json, J)!;
o.Mode.ShouldBe(MqttMode.SparkplugB);
o.UseTls.ShouldBeTrue();
o.Sparkplug!.GroupId.ShouldBe("Plant1");
o.Plain.ShouldBeNull();
}
[Fact]
public void Serialize_WritesEnumsAsNames_NotOrdinals()
{
var s = JsonSerializer.Serialize(new MqttDriverOptions { Mode = MqttMode.Plain }, J);
s.ShouldContain("\"Plain\"");
s.ShouldNotContain("\"mode\":0");
}
}
```
2. Run — expect FAIL (types don't exist):
```bash
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests # RED
```
3. Implement the enums + `MqttDriverOptions` (with nested `MqttSparkplugOptions` / `MqttPlainOptions`) matching §5.1 defaults (`useTls=true`, `connectTimeoutSeconds=15`, `reconnectMin/MaxBackoffSeconds=1/30`). Wire the test csproj into `slnx`.
4. Run — expect PASS.
5. Commit: `git commit -am "feat(mqtt): Contracts options DTO + enums (name-serialized)"` (+ session trailer).
---
## Task 2 (P1): `.Contracts``MqttTagDefinition` + `MqttEquipmentTagParser` (plain)
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none (Task 6 depends on the resolver)
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Contracts/MqttTagDefinition.cs` (parsed per-tag descriptor — carries plain OR sparkplug variant fields; plain fields populated in P1)
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Contracts/MqttEquipmentTagParser.cs` (`TryParse(reference, out def)` + `Inspect(reference)`, mirrors `ModbusEquipmentTagParser` / `ModbusTagDefinitionFactory`)
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests/MqttEquipmentTagParserTests.cs`
### Steps (TDD)
1. Failing tests — parse a plain TagConfig blob, reject a typo'd enum (strict), and reject a wildcard topic:
```csharp
[Fact]
public void TryParse_PlainJsonBlob_PopulatesTopicAndPath()
{
const string r = """{"topic":"factory/oven/temp","payloadFormat":"Json","jsonPath":"$.value","dataType":"Double","qos":1}""";
MqttEquipmentTagParser.TryParse(r, out var def).ShouldBeTrue();
def!.Topic.ShouldBe("factory/oven/temp");
def.PayloadFormat.ShouldBe(MqttPayloadFormat.Json);
def.Name.ShouldBe(r); // the def Name == reference string (forward-router key)
}
[Fact]
public void TryParse_TypoedPayloadFormat_RejectsStrict()
=> MqttEquipmentTagParser.TryParse(
"""{"topic":"a/b","payloadFormat":"Jason","dataType":"Double"}""", out _).ShouldBeFalse();
[Fact]
public void Inspect_WildcardTopic_ReturnsWarning()
=> MqttEquipmentTagParser.Inspect("""{"topic":"a/+/c","payloadFormat":"Raw"}""").ShouldNotBeEmpty();
```
2. Run — expect FAIL.
3. Implement: a leading `{` marks a TagConfig blob; use `TagConfigJson.TryReadEnumStrict` for `payloadFormat`/`dataType` (typo → `false``BadNodeIdUnknown` upstream). `def.Name = reference`. `Inspect` returns a deploy-time warning list for present-but-invalid enums / unparseable blobs / wildcard tag-topics. Leave Sparkplug-descriptor parsing as a stub the P2 tasks fill (`groupId`/`edgeNodeId`/`deviceId`/`metricName`).
4. Run — expect PASS.
5. Commit: `git commit -am "feat(mqtt): plain tag parser + strict-enum descriptor"` (+ trailer).
---
## Task 3 (P1): `.Driver` project + `MqttConnection` connect/TLS/auth (bounded)
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none (Task 4 extends `MqttConnection`)
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.csproj` (net10, refs `.Contracts` + `Core.Abstractions` + `Core` + `MQTTnet`)
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt/MqttConnection.cs` (MQTTnet-5 client wrapper: build options from `MqttDriverOptions` incl. TLS + CA-pin + credentials; `ConnectAsync(ct)` under `connectTimeoutSeconds` linked-CTS)
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests/MqttConnectionTests.cs`
- Modify: `ZB.MOM.WW.OtOpcUa.slnx`
### Steps (TDD)
1. Failing test — bounded connect against a dead endpoint fails fast (no hang), and TLS-option assembly honours the knobs. Use a closed loopback port for the deadline test:
```csharp
[Fact]
public async Task ConnectAsync_DeadBroker_FailsWithinDeadline_NoHang()
{
var opts = new MqttDriverOptions { Host = "127.0.0.1", Port = 1, UseTls = false, ConnectTimeoutSeconds = 2 };
var conn = new MqttConnection(opts, driverId: "t", logger: null);
var sw = Stopwatch.StartNew();
await Should.ThrowAsync<Exception>(() => conn.ConnectAsync(CancellationToken.None));
sw.Elapsed.ShouldBeLessThan(TimeSpan.FromSeconds(5)); // deadline honoured, not wedged
}
[Fact]
public void BuildClientOptions_UseTlsWithCaPin_SetsTlsAndValidatesChain()
{
var opts = new MqttDriverOptions { Host="h", Port=8883, UseTls=true,
AllowUntrustedServerCertificate=false, CaCertificatePath="/tmp/ca.pem" };
var built = MqttConnection.BuildClientOptions(opts, clientIdSuffix: null);
built.ChannelOptions.ShouldBeOfType<MqttClientTcpOptions>().TlsOptions.UseTls.ShouldBeTrue();
}
```
2. Run — expect FAIL.
3. Implement `MqttConnection`: static `BuildClientOptions(opts, clientIdSuffix)` (host/port, protocol version, clientId + optional suffix, keep-alive, clean-session, credentials, TLS with `AllowUntrustedServerCertificate` → custom cert validator, `CaCertificatePath` → chain pin). `ConnectAsync(ct)` links `ct` with a `connectTimeoutSeconds` CTS. Ctor stores options only — connection-free.
4. Run — expect PASS.
5. Commit: `git commit -am "feat(mqtt): MqttConnection connect/TLS/auth under bounded deadline"` (+ trailer).
---
## Task 4 (P1): `MqttConnection` hand-rolled reconnect loop (backoff + resubscribe)
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt/MqttConnection.cs` (reconnect loop: exponential backoff `min→max`; on `DisconnectedAsync` schedule reconnect; on reconnect fire a `Reconnected` callback so the driver re-subscribes; expose `State` = Connected/Reconnecting/Faulted + `LastMessageAgeUtc`)
- Modify: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests/MqttConnectionTests.cs`
**Why high-risk:** MQTTnet v5 dropped v4's `ManagedMqttClient` (§8) — there is no library-managed reconnect. Subscriptions do not survive a clean session; a reconnect that forgets to re-subscribe silently goes dark. Backoff that ignores its cap can hot-loop a dead broker.
### Steps (TDD)
1. Failing tests — backoff schedule is bounded + monotone-to-cap, and a `Reconnected` event drives a re-subscribe callback. Test the backoff calculator as a pure function (no live broker):
```csharp
[Theory]
[InlineData(0, 1)] [InlineData(1, 2)] [InlineData(2, 4)] [InlineData(10, 30)] // caps at max=30
public void NextBackoff_IsExponentialClampedToMax(int attempt, int expectedSeconds)
=> MqttConnection.NextBackoff(attempt, minSeconds: 1, maxSeconds: 30)
.ShouldBe(TimeSpan.FromSeconds(expectedSeconds));
[Fact]
public async Task OnReconnect_InvokesResubscribeCallback()
{
var conn = new MqttConnection(new MqttDriverOptions(), "t", null);
var fired = 0; conn.Reconnected += () => { fired++; return Task.CompletedTask; };
await conn.RaiseReconnectedForTest(); // test seam
fired.ShouldBe(1);
}
```
2. Run — expect FAIL.
3. Implement: `NextBackoff(attempt, min, max)` pure; a reconnect worker that loops on disconnect with backoff, honours `cleanSession`/session-expiry, re-subscribes via the `Reconnected` callback (idempotent insurance even for persistent sessions), and (Sparkplug, P2) re-requests rebirth. `State` transitions Connected↔Reconnecting; Faulted only on unrecoverable config. Never block the MQTTnet dispatcher thread.
4. Run — expect PASS.
5. Commit: `git commit -am "feat(mqtt): hand-rolled reconnect loop with bounded backoff + resubscribe"` (+ trailer).
---
## Task 5 (P1): `LastValueCache` + `IReadable`
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 6
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt/LastValueCache.cs` (`FullReference → DataValueSnapshot`, thread-safe)
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests/LastValueCacheTests.cs`
### Steps (TDD)
1. Failing test — unseen reference returns a per-ref `GoodNoData`/uncertain snapshot (never throws the batch); a seeded ref returns last value:
```csharp
[Fact]
public void Read_UnseenRef_ReturnsPerRefNoData_DoesNotThrow()
{
var c = new LastValueCache();
var snap = c.Read("factory/oven/temp#$.value");
snap.StatusCode.ShouldBe(StatusCodes.GoodNoData); // or Uncertain — per IReadable contract, per-ref status
}
[Fact]
public void Update_ThenRead_ReturnsLastValue()
{
var c = new LastValueCache();
c.Update("k", DataValueSnapshot.Good(42.0, DateTime.UtcNow));
c.Read("k").Value.ShouldBe(42.0);
}
```
2. Run — expect FAIL.
3. Implement `LastValueCache` (concurrent dict); `MqttDriver.ReadAsync` (Task 7) returns `references.Select(cache.Read)` — batch never throws, per-ref `StatusCode` carries "not yet observed".
4. Run — expect PASS.
5. Commit: `git commit -am "feat(mqtt): last-value cache backing IReadable (per-ref no-data)"` (+ trailer).
---
## Task 6 (P1): `ISubscribable` — plain topic subscribe → `OnDataChange` + retained seed
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 5
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt/MqttSubscriptionManager.cs` (registers `FullReference → MqttTagDefinition` via the shared `EquipmentTagRefResolver<MqttTagDefinition>`; dedupes/coalesces topics; on message: match topic → authored tag(s), extract value at `jsonPath`/raw/scalar, raise `OnDataChange`; seed from retained message on subscribe)
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt/MqttConnection.cs` (expose `SubscribeAsync(filters, ct)` under a bounded deadline + a message-received event)
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests/MqttSubscriptionManagerTests.cs`
### Steps (TDD)
1. Failing test — feed a synthetic received message through the manager (no live broker); assert routing + JSONPath extraction fire `OnDataChange` with the right `FullReference`:
```csharp
[Fact]
public void OnMessage_MatchingTopic_RaisesDataChangeAtJsonPath()
{
var mgr = new MqttSubscriptionManager();
var handle = mgr.Register(new[] { """{"topic":"f/oven/temp","payloadFormat":"Json","jsonPath":"$.value","dataType":"Double"}""" });
string? gotRef = null; object? gotVal = null;
mgr.OnDataChange += (_, e) => { gotRef = e.FullReference; gotVal = e.Snapshot.Value; };
mgr.HandleMessage("f/oven/temp", """{"value":21.5}"""u8.ToArray(), retained: false);
gotVal.ShouldBe(21.5);
gotRef.ShouldContain("f/oven/temp");
}
[Fact]
public void OnMessage_UnauthoredTopic_RaisesNothing() // chatty broker must not auto-provision
{
var mgr = new MqttSubscriptionManager();
mgr.Register(Array.Empty<string>());
var fired = false; mgr.OnDataChange += (_, _) => fired = true;
mgr.HandleMessage("random/topic", "1"u8.ToArray(), retained: false);
fired.ShouldBeFalse();
}
```
2. Run — expect FAIL.
3. Implement using `EquipmentTagRefResolver<MqttTagDefinition>` (parse-once cache, as Modbus does). `HandleMessage` matches topic → tag(s), extracts (Json→JSONPath token→typed value; Raw→bytes-as-string; Scalar→parse), updates `LastValueCache`, raises `OnDataChange`. Retained-flagged messages seed initial value. `SubscribeAsync` returns an `ISubscriptionHandle` whose `DiagnosticId` names mode+filter; completes under a bounded SUBACK deadline (SUBACK failure → per-ref Bad, not hang).
4. Run — expect PASS.
5. Commit: `git commit -am "feat(mqtt): plain subscribe→OnDataChange with retained seed + ref resolver"` (+ trailer).
---
## Task 7 (P1): `MqttDriver` shell — `IDriver` + authored-only `ITagDiscovery` + `IHostConnectivityProbe`
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none (Task 8/9 wire it)
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt/MqttDriver.cs` (`IDriver, ITagDiscovery, ISubscribable, IReadable, IHostConnectivityProbe, IRediscoverable`; composes `MqttConnection` + `MqttSubscriptionManager` + `LastValueCache`)
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests/MqttDriverDiscoveryTests.cs`
### Steps (TDD)
1. Failing test — `DiscoverAsync` streams **only authored tags** into a capturing `IAddressSpaceBuilder`; plain-mode `RediscoverPolicy == Once`; `SupportsOnlineDiscovery == false`:
```csharp
[Fact]
public async Task DiscoverAsync_Plain_StreamsAuthoredTagsOnly_PolicyOnce()
{
var driver = new MqttDriver(new MqttDriverOptions { Mode = MqttMode.Plain }, "d", null);
driver.SetAuthoredTagsForTest(new[] { """{"topic":"f/t","payloadFormat":"Raw","dataType":"String"}""" });
var b = new CapturingAddressSpaceBuilder();
await driver.DiscoverAsync(b, CancellationToken.None);
b.Variables.Count.ShouldBe(1);
driver.RediscoverPolicy.ShouldBe(DiscoveryRediscoverPolicy.Once);
((ITagDiscovery)driver).SupportsOnlineDiscovery.ShouldBeFalse();
}
```
2. Run — expect FAIL.
3. Implement: `InitializeAsync` deserializes options (shared `JsonSerializerOptions`), builds + connects `MqttConnection`, subscribes the mode-appropriate filter. `ReinitializeAsync` applies deltas in place (never crash → Faulted). `ShutdownAsync` disconnects/disposes. `GetHealth` = Connected/Reconnecting/Faulted + last-message-age. `GetMemoryFootprint` = caches; `FlushOptionalCachesAsync` = no-op (birth/alias are correctness state — forbidden to flush; last-value backs `IReadable`). `DiscoverAsync` replays authored tags only; `RediscoverPolicy => Once` (plain). `IRediscoverable.OnRediscoveryNeeded` never fires in plain mode.
4. Run — expect PASS.
5. Commit: `git commit -am "feat(mqtt): MqttDriver shell — lifecycle + authored-only discovery (Once)"` (+ trailer).
---
## Task 8 (P1): `MqttDriverProbe` — CONNECT handshake
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 10
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt/MqttDriverProbe.cs` (`IDriverProbe`, `DriverType => "Mqtt"`; parse config with the shared options, open a CONNECT under the timeout, green + latency on CONNACK-accepted, targeted error on refused/TLS/auth/timeout — mirrors `ModbusDriverProbe`)
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests/MqttDriverProbeTests.cs`
### Steps (TDD)
1. Failing test — `DriverType` is `"Mqtt"`; a dead endpoint yields a failed (not thrown) probe result within the deadline:
```csharp
[Fact]
public void DriverType_IsCanonicalMqtt() => new MqttDriverProbe().DriverType.ShouldBe("Mqtt");
[Fact]
public async Task ProbeAsync_DeadBroker_ReturnsFailedResult_WithinDeadline()
{
var r = await new MqttDriverProbe().ProbeAsync(
"""{"host":"127.0.0.1","port":1,"useTls":false,"connectTimeoutSeconds":2}""", CancellationToken.None);
r.Success.ShouldBeFalse();
r.Message.ShouldNotBeNullOrEmpty();
}
```
2. Run — expect FAIL.
3. Implement; CONNACK-accepted is the MQTT "device is answering" proof. Use the **shared** `JsonSerializerOptions` (enum-as-name).
4. Run — expect PASS.
5. Commit: `git commit -am "feat(mqtt): MqttDriverProbe CONNECT handshake"` (+ trailer).
---
## Task 9 (P1): Factory + `DriverTypeNames.Mqtt` + Host registration
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt/MqttDriverFactoryExtensions.cs` (`DriverTypeName = "Mqtt"`, shared `JsonSerializerOptions`, `Register(registry, loggerFactory)` — mirror `OpcUaClientDriverFactoryExtensions`)
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/DriverTypeNames.cs` (add `public const string Mqtt = "Mqtt";` + append to the `All` list)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs` (add `Driver.Mqtt.MqttDriverFactoryExtensions.Register(registry, loggerFactory);` in `Register(...)` near line 146; add `using MqttProbe = Driver.Mqtt.MqttDriverProbe;` + `services.TryAddEnumerable(ServiceDescriptor.Singleton<IDriverProbe, MqttProbe>());` in `AddOtOpcUaDriverProbes` near line 124)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj` (ProjectReference the `.Driver` assembly)
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests/MqttDriverFactoryExtensionsTests.cs`
### Steps (TDD)
1. Failing test — `Register` binds the `"Mqtt"` type, and the factory builds an `MqttDriver` from JSON with string enums:
```csharp
[Fact]
public void Register_ThenCreate_BuildsMqttDriver()
{
var registry = new DriverFactoryRegistry();
MqttDriverFactoryExtensions.Register(registry);
var d = registry.Create("Mqtt", "d1", """{"host":"h","port":1883,"mode":"Plain"}""");
d.ShouldBeOfType<MqttDriver>();
}
[Fact]
public void DriverTypeName_MatchesConstant()
=> MqttDriverFactoryExtensions.DriverTypeName.ShouldBe(DriverTypeNames.Mqtt);
```
2. Run — expect FAIL.
3. Implement + wire both host sites. Build the whole solution to confirm registration compiles:
```bash
dotnet build ZB.MOM.WW.OtOpcUa.slnx # expect: Build succeeded
```
4. Run — expect PASS.
5. Commit: `git commit -am "feat(mqtt): factory + DriverTypeNames.Mqtt + host factory/probe registration"` (+ trailer).
---
## Task 10 (P1): `.Browser` — bespoke `#`-observation browser (passive)
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 8
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Browser/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Browser.csproj` (net10, refs `.Contracts` + `Commons(.Browsing)` + `MQTTnet`)
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Browser/MqttDriverBrowser.cs` (`IDriverBrowser`, `DriverType => "Mqtt"`; `OpenAsync` connects with a `{clientId}-browse-{guid8}` suffix under a clamped 530 s budget; subscribes `#`/`{topicPrefix}#`; returns `MqttBrowseSession`; **publishes nothing**)
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Browser/MqttBrowseSession.cs` (`IBrowseSession` over a thread-safe accumulating observed tree; `RootAsync`/`ExpandAsync` split topic segments on `/`, leaf = `Kind=Leaf`; `AttributesAsync` = synthetic attribute w/ inferred type + last payload snippet; `DisposeAsync` disconnects best-effort)
- Modify: `ZB.MOM.WW.OtOpcUa.slnx`
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests/MqttBrowseSessionTests.cs`
### Steps (TDD)
1. Failing test — feed observed topics into the session's accumulating tree (test seam, no live broker); assert `RootAsync`/`ExpandAsync` build the segment tree and **no publish** occurs on any browse call:
```csharp
[Fact]
public async Task ExpandAsync_BuildsTopicSegmentTree_FromObservedTopics()
{
var s = new MqttBrowseSession(MqttMode.Plain);
s.ObserveTopicForTest("factory/line3/oven/temp");
var root = await s.RootAsync(CancellationToken.None);
root.ShouldContain(n => n.BrowseName == "factory");
var lvl2 = await s.ExpandAsync("factory", CancellationToken.None);
lvl2.ShouldContain(n => n.BrowseName == "line3");
}
[Fact]
public async Task BrowseCalls_PublishNothing() // browse is read-only
{
var s = new MqttBrowseSession(MqttMode.Plain);
await s.RootAsync(CancellationToken.None);
s.PublishCountForTest.ShouldBe(0);
}
```
2. Run — expect FAIL.
3. Implement the accumulating tree + passive `OpenAsync`. In P1 only the plain `#` path is live; leave a Sparkplug hook the P2 tasks fill (`Group→EdgeNode→Device→Metric` + `RequestRebirthAsync`).
4. Run — expect PASS.
5. Commit: `git commit -am "feat(mqtt): bespoke passive #-observation browser (plain)"` (+ trailer).
---
## Task 11 (P1): Register the bespoke browser (overrides universal fallback)
**Classification:** trivial
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 12
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/EndpointRouteBuilderExtensions.cs` (add `services.AddSingleton<IDriverBrowser, MqttDriverBrowser>();` alongside the existing two near line 7576)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/ZB.MOM.WW.OtOpcUa.AdminUI.csproj` (ProjectReference `.Browser`)
### Steps
1. Add the registration + project reference. Registering for `DriverType="Mqtt"` overrides the universal fallback (`BrowserSessionService` resolves bespoke-first).
2. Build: `dotnet build src/Server/ZB.MOM.WW.OtOpcUa.AdminUI` — expect success.
3. Commit: `git commit -am "feat(mqtt): register MqttDriverBrowser (bespoke-first for Mqtt)"` (+ trailer).
---
## Task 12 (P1): Typed AdminUI editor + validator (plain shape)
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 11
**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/TagEditors/MqttTagConfigModel.cs` (thin typed model over a preserved `JsonObject` key bag; `FromJson`/`ToJson`/`Validate`, preserves unknown keys incl. history keys; carries a `Mode` field selecting sub-shape — P1 implements Plain `Validate`, Sparkplug stub filled in Task 24)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/TagEditors/TagConfigEditorMap.cs` (`[DriverTypeNames.Mqtt] = typeof(Components.Shared.Uns.TagEditors.MqttTagConfigEditor)`)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/TagEditors/TagConfigValidator.cs` (`[DriverTypeNames.Mqtt] = j => MqttTagConfigModel.FromJson(j).Validate()`)
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Shared/Uns/TagEditors/MqttTagConfigEditor.razor` (copy the Modbus editor template — mode-switch at top, per-mode field group, client-side `Validate()`)
- Create: `tests/Server/.../MqttTagConfigModelTests.cs` (co-locate with the existing AdminUI test project — verify path with `find tests/Server -name "*TagConfigModel*Tests.cs"`)
### Steps (TDD)
1. Failing test — round-trip preserves unknown/history keys; plain `Validate` requires concrete topic + jsonPath-when-Json; re-derives `FullName`:
```csharp
[Fact]
public void FromJson_ToJson_PreservesUnknownKeys()
{
var m = MqttTagConfigModel.FromJson("""{"topic":"a/b","payloadFormat":"Raw","dataType":"String","isHistorized":true}""");
m.ToJson().ShouldContain("isHistorized");
}
[Fact]
public void Validate_PlainWildcardTopic_Fails()
=> MqttTagConfigModel.FromJson("""{"topic":"a/+/c","payloadFormat":"Raw","dataType":"String"}""")
.Validate().ShouldNotBeEmpty();
[Fact]
public void Validate_JsonWithoutPath_Fails()
=> MqttTagConfigModel.FromJson("""{"topic":"a/b","payloadFormat":"Json","dataType":"Double"}""")
.Validate().ShouldNotBeEmpty();
```
2. Run — expect FAIL.
3. Implement the model (mirror `OpcUaClientTagConfigModel`); `ToJson` writes PascalCase `FullName` re-derived from descriptor (`{topic}#{jsonPath}` plain). Build the `.razor`. Register both map entries.
4. Run — expect PASS + `dotnet build src/Server/ZB.MOM.WW.OtOpcUa.AdminUI`.
5. Commit: `git commit -am "feat(mqtt): typed tag editor + validator (plain mode)"` (+ trailer).
---
## Task 13 (P1): Mosquitto + JSON-publisher fixture (TLS+auth) + env-gated live suite
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.IntegrationTests/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.IntegrationTests.csproj`
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.IntegrationTests/Docker/docker-compose.yml` (`eclipse-mosquitto` with **auth + TLS** on `:8883`, plain `:1883` for smoke only; a JSON-publisher container emitting `retain=true` JSON on a few topics — `mosquitto_pub` loop or `paho-mqtt` script; `project=lmxopcua` label applied host-side)
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.IntegrationTests/Docker/mosquitto.conf` + password/cert material generator script (creds via env, never real secrets committed)
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.IntegrationTests/PlainMqttLiveTests.cs` (category `LiveIntegration`, gated on `MQTT_FIXTURE_ENDPOINT` — skips clean when unset → macOS-offline-safe)
- Modify: `ZB.MOM.WW.OtOpcUa.slnx`
### Steps
1. Write the compose with **auth + TLS** (never anonymous). Add the JSON publisher with `retain=true`.
2. Write env-gated tests: connect/TLS/auth; plain subscribe + retained-read seed; unauthored-topic silence. `[SkippableFact]` reading `MQTT_FIXTURE_ENDPOINT`.
3. Confirm offline skip:
```bash
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.IntegrationTests # expect: all Skipped (no env var)
```
4. Deploy the fixture to the docker host and run live (design §9):
```bash
lmxopcua-fix sync mqtt && lmxopcua-fix up mqtt
export MQTT_FIXTURE_ENDPOINT=10.100.0.35:8883
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.IntegrationTests --filter "Category=LiveIntegration" # expect: PASS
```
5. Commit: `git commit -am "test(mqtt): Mosquitto TLS+auth fixture + env-gated plain live suite"` (+ trailer).
---
## Task 14 (P1): Live `/run` verify on docker-dev — **P1 MILESTONE COMPLETE**
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Modify: `CLAUDE.md` (add the MQTT fixture endpoint to the Docker Workflow endpoint list: `10.100.0.35:1883/8883`, `MQTT_FIXTURE_ENDPOINT`)
- Modify: `docs/plans/2026-07-15-driver-expansion-tracking.md` (mark MQTT P1 code-complete)
### Steps
1. On docker-dev (`:9200`, login disabled — run it yourself, do not defer): author an `Mqtt` driver (Plain mode) + a tag via the `/uns` picker (bespoke `#`-observation browser) or the typed editor; deploy.
2. Confirm via Client.CLI the node carries live broker values:
```bash
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://localhost:4840 -n "ns=2;s=<RawPath>"
```
3. Confirm the typed editor renders (Razor binding bugs pass unit tests — live-verify is mandatory). Update CLAUDE.md + tracking doc.
4. Commit: `git commit -am "docs(mqtt): P1 plain-MQTT milestone live-verified; endpoint recorded"` (+ trailer).
---
# ── P2: Sparkplug B ingest (builds on the P1 skeleton) ──
## Task 15 (P2): Vendor Tahu proto + `Grpc.Tools` codegen
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none (P2 root)
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Contracts/Protos/sparkplug_b.proto` (vendored Eclipse Tahu `sparkplug_b.proto`, pinned to a known Tahu commit with a provenance comment header)
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Contracts/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Contracts.csproj` (add `Google.Protobuf` + build-time `Grpc.Tools` (`PrivateAssets=all`); `<Protobuf Include="Protos/sparkplug_b.proto" GrpcServices="None" />` — message-only codegen, this repo's first in-repo protoc step)
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests/SparkplugProtoCodegenTests.cs`
### Steps (TDD)
1. Failing test — the generated `Payload`/`Metric` types exist and round-trip a hand-built payload:
```csharp
[Fact]
public void GeneratedPayload_RoundTrips()
{
var p = new Org.Eclipse.Tahu.Protobuf.Payload { Seq = 3 };
p.Metrics.Add(new Org.Eclipse.Tahu.Protobuf.Payload.Types.Metric { Name = "Temperature", Alias = 5 });
var back = Org.Eclipse.Tahu.Protobuf.Payload.Parser.ParseFrom(p.ToByteArray());
back.Seq.ShouldBe(3ul);
back.Metrics[0].Alias.ShouldBe(5ul);
}
```
2. Run — expect FAIL (no generated types).
3. Vendor the proto (provenance comment: Tahu commit hash + URL); wire `Grpc.Tools`. If in-repo protoc proves objectionable, fall back to checking in the generated C# with the same provenance comment (design §2.1). Build:
```bash
dotnet build src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Contracts # expect: proto compiles, 0 warnings
```
4. Run — expect PASS.
5. Commit: `git commit -am "feat(mqtt): vendor Tahu sparkplug_b.proto + Grpc.Tools codegen"` (+ trailer).
---
## Task 16 (P2): `SparkplugCodec` decode + golden payloads
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 17
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt/Sparkplug/SparkplugCodec.cs` (decode `Payload`/`Metric` from wire bytes → a driver-side struct; encode NCMD deferred to `RebirthRequester` Task 20 / write-through P3)
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests/SparkplugCodecTests.cs`
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests/Golden/nbirth.bin` + `ndata.bin` (golden byte vectors, generated once from a hand-built payload and committed)
### Steps (TDD)
1. Failing test — decode a golden NBIRTH: seq + metric name/alias/datatype/value survive:
```csharp
[Fact]
public void Decode_GoldenNbirth_ExtractsMetrics()
{
var payload = SparkplugCodec.Decode(File.ReadAllBytes("Golden/nbirth.bin"));
payload.Seq.ShouldBe((byte)0);
payload.Metrics.ShouldContain(m => m.Name == "Temperature" && m.Alias == 5 && m.DataType == SparkplugDataType.Float);
}
```
2. Run — expect FAIL.
3. Implement `SparkplugCodec.Decode`; generate the golden vectors in a one-off `[Fact(Skip="generator")]` or a small helper, commit the `.bin` files.
4. Run — expect PASS.
5. Commit: `git commit -am "feat(mqtt): SparkplugCodec decode + golden payload vectors"` (+ trailer).
---
## Task 17 (P2): `SparkplugTopic` + `SparkplugDataType.ToDriverDataType`
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 16
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt/Sparkplug/SparkplugTopic.cs` (parse/format `spBv1.0/{group}/{type}/{node}[/{device}]`; `type` ∈ NBIRTH/DBIRTH/NDATA/DDATA/NDEATH/DDEATH/NCMD/DCMD/STATE)
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Contracts/SparkplugDataType.cs` (enum + `ToDriverDataType()` per the §3.5 map: Int8→Int16, UInt8→UInt16, DataSet/Template unsupported, `*Array`→element+IsArray)
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests/SparkplugTopicTests.cs`
### Steps (TDD)
1. Failing tests — topic parse extracts group/type/node/device; datatype map widens Int8→Int16 and marks DataSet unsupported:
```csharp
[Fact]
public void Parse_DeviceData_ExtractsAllSegments()
{
var t = SparkplugTopic.Parse("spBv1.0/Plant1/DDATA/EdgeA/Filler1");
t.GroupId.ShouldBe("Plant1"); t.Type.ShouldBe(SparkplugMessageType.DDATA);
t.EdgeNodeId.ShouldBe("EdgeA"); t.DeviceId.ShouldBe("Filler1");
}
[Theory]
[InlineData(SparkplugDataType.Int8, DriverDataType.Int16)]
[InlineData(SparkplugDataType.UInt8, DriverDataType.UInt16)]
[InlineData(SparkplugDataType.Float, DriverDataType.Float32)]
public void ToDriverDataType_MapsAndWidens(SparkplugDataType s, DriverDataType d)
=> s.ToDriverDataType().ShouldBe(d);
```
2. Run — expect FAIL.
3. Implement.
4. Run — expect PASS.
5. Commit: `git commit -am "feat(mqtt): Sparkplug topic parse + datatype map"` (+ trailer).
---
## Task 18 (P2): `BirthCache` + `AliasTable` — bind-by-name, rebuild-per-birth
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 19
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt/Sparkplug/AliasTable.cs` (per `(edgeNode,device)` alias→(name,datatype); **rebuilt wholesale each birth, never merged**)
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt/Sparkplug/BirthCache.cs` (NBIRTH/DBIRTH metric catalog: name/alias/datatype/last value)
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests/AliasBindingTests.cs`
**Why high-risk:** §3.6 invariants #1#2 — binding data by alias silently mis-routes when an alias is reused for a different metric across a rebirth. Bind by **stable metric NAME**; the alias is a per-birth cache only.
### Steps (TDD)
1. Failing tests — the two load-bearing invariants:
```csharp
[Fact]
public void Rebirth_ReusesAliasForDifferentMetric_ResolvesByName_NotAlias()
{
var t = new AliasTable();
t.RebuildFromBirth(new[] { (name:"Temperature", alias:5ul, dt:SparkplugDataType.Float) });
t.Resolve(alias: 5).Name.ShouldBe("Temperature");
// rebirth: alias 5 now means a DIFFERENT metric
t.RebuildFromBirth(new[] { (name:"Pressure", alias:5ul, dt:SparkplugDataType.Float) });
t.Resolve(alias: 5).Name.ShouldBe("Pressure"); // wholesale replace, not merge
}
[Fact]
public void RebuildFromBirth_DoesNotMergeStaleAliases()
{
var t = new AliasTable();
t.RebuildFromBirth(new[] { (name:"A", alias:1ul, dt:SparkplugDataType.Int32) });
t.RebuildFromBirth(new[] { (name:"B", alias:2ul, dt:SparkplugDataType.Int32) });
t.TryResolve(alias: 1, out _).ShouldBeFalse(); // alias 1 gone after rebirth
}
```
2. Run — expect FAIL.
3. Implement — `RebuildFromBirth` replaces the whole map. `BirthCache` keeps the metric catalog + last value keyed by name.
4. Run — expect PASS.
5. Commit: `git commit -am "feat(mqtt): AliasTable/BirthCache — bind-by-name, rebuild-per-birth"` (+ trailer).
---
## Task 19 (P2): `SequenceTracker` — seq-gap + bdSeq death-tie
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 18
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt/Sparkplug/SequenceTracker.cs` (per edge-node `seq` 0255 wrap gap detection; `bdSeq` ties NDEATH↔NBIRTH)
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests/SequenceTrackerTests.cs`
**Why high-risk:** §3.6 invariant #3 — a wrong wrap boundary (255→0 is NOT a gap) either misses real gaps (stale data) or false-positives every wrap (rebirth-storm).
### Steps (TDD)
1. Failing tests — contiguous ok, wrap ok, gap detected:
```csharp
[Fact]
public void Sequence_WrapAt255IsContiguous_NotAGap()
{
var s = new SequenceTracker();
s.Accept(254).ShouldBeTrue(); s.Accept(255).ShouldBeTrue(); s.Accept(0).ShouldBeTrue(); // wrap
}
[Fact]
public void Sequence_SkippedValue_IsGap()
{
var s = new SequenceTracker();
s.Accept(10).ShouldBeTrue();
s.Accept(12).ShouldBeFalse(); // gap → caller requests rebirth
}
```
2. Run — expect FAIL.
3. Implement (next-expected = `(last+1) & 0xFF`); `bdSeq` compare helper.
4. Run — expect PASS.
5. Commit: `git commit -am "feat(mqtt): SequenceTracker seq-gap + bdSeq death-tie"` (+ trailer).
---
## Task 20 (P2): `RebirthRequester` — NCMD encode
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt/Sparkplug/RebirthRequester.cs` (encode an NCMD writing `Node Control/Rebirth = true` to `spBv1.0/{group}/NCMD/{node}`; publish under a bounded deadline)
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests/RebirthRequesterTests.cs`
### Steps (TDD)
1. Failing test — encoded NCMD carries the `Node Control/Rebirth`=true metric and targets the right topic:
```csharp
[Fact]
public void BuildRebirthNcmd_EncodesControlMetric_AndTopic()
{
var (topic, bytes) = RebirthRequester.Build("Plant1", "EdgeA");
topic.ShouldBe("spBv1.0/Plant1/NCMD/EdgeA");
var p = SparkplugCodec.Decode(bytes);
p.Metrics.ShouldContain(m => m.Name == "Node Control/Rebirth" && Equals(m.Value, true));
}
```
2. Run — expect FAIL.
3. Implement (encode path via the generated proto).
4. Run — expect PASS.
5. Commit: `git commit -am "feat(mqtt): RebirthRequester NCMD encode"` (+ trailer).
---
## Task 21 (P2): Sparkplug ingest state machine — the §3.6 matrix
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** none (integrates Tasks 1620 into the driver)
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt/Sparkplug/SparkplugIngestor.cs` (routes decoded messages: N/DBIRTH → rebuild `AliasTable` + `BirthCache`; N/DDATA → resolve alias→name → map to authored `FullReference` **by (group,node,device,metricName)**`OnDataChange`; N/DDEATH → emit STALE/Bad snapshots; seq-gap / unknown-alias / data-before-birth → `RebirthRequester` gated on `requestRebirthOnGap`; STATE/primary-host handling; late-join rebirth on connect)
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt/MqttDriver.cs` (Sparkplug-mode subscribe `spBv1.0/{group}/#` (+ STATE) → `SparkplugIngestor`; reconnect → re-subscribe + request rebirth)
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests/SparkplugIngestorTests.cs`
**Why high-risk:** this is the §3.6 correctness core — the #1 risk in the design. Test the full matrix: rebirth, missed-birth, seq-gap, alias-reuse-across-rebirth, death→stale→rebirth.
### Steps (TDD)
1. Failing tests — the §3.6 matrix, driven through the ingestor with synthetic decoded messages (no live broker):
```csharp
[Fact]
public void Birth_Then_Data_ResolvesByAlias_RaisesOnDataChangeByName()
{
var ing = new SparkplugIngestor(...); // authored tag: Plant1/EdgeA/Filler1:Temperature
string? firedRef = null; ing.OnDataChange += (_, e) => firedRef = e.FullReference;
ing.HandleDbirth("Plant1","EdgeA","Filler1", new[] { (name:"Temperature", alias:5ul, dt:SparkplugDataType.Float, val:(object)0f) });
ing.HandleDdata("Plant1","EdgeA","Filler1", seq:1, new[] { (alias:5ul, val:(object)21.5f) });
firedRef.ShouldContain("Filler1:Temperature");
}
[Fact]
public void DataBeforeBirth_RequestsRebirth()
{
var ncmds = new List<string>();
var ing = new SparkplugIngestor(..., publishNcmd: (t,_) => ncmds.Add(t));
ing.HandleDdata("Plant1","EdgeA","Filler1", seq:0, new[] { (alias:9ul, val:(object)1f) });
ncmds.ShouldContain("spBv1.0/Plant1/NCMD/EdgeA");
}
[Fact]
public void Ddeath_EmitsStaleForDeviceMetrics_NextBirthRestoresGood()
{
var ing = new SparkplugIngestor(...);
var quals = new List<StatusCode>(); ing.OnDataChange += (_, e) => quals.Add(e.Snapshot.StatusCode);
ing.HandleDbirth("Plant1","EdgeA","Filler1", new[] { (name:"Temperature", alias:5ul, dt:SparkplugDataType.Float, val:(object)0f) });
ing.HandleDdeath("Plant1","EdgeA","Filler1");
quals.ShouldContain(q => StatusCode.IsBad(q)); // STALE/Bad on death
}
```
2. Run — expect FAIL.
3. Implement the ingestor holding the §3.6 invariants. Route via `EquipmentTagRefResolver` keyed by `(group,node,device,metricName)`.
4. Run — expect PASS (all matrix cases).
5. Commit: `git commit -am "feat(mqtt): Sparkplug ingest state machine (birth/alias/seq/rebirth/death→stale)"` (+ trailer).
---
## Task 22 (P2): `ITagDiscovery` `UntilStable` + `IRediscoverable`
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt/MqttDriver.cs` (`RediscoverPolicy => UntilStable` in Sparkplug mode — authored tags' datatypes fill in as births arrive; `DiscoverAsync` re-streams authored tags with resolved datatypes from `BirthCache`; fire `OnRediscoveryNeeded` on a **new DBIRTH** or a rebirth with a changed metric set, `ScopeHint` = edge-node/device folder path)
- Modify: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests/MqttDriverDiscoveryTests.cs`
### Steps (TDD)
1. Failing tests — Sparkplug policy is `UntilStable`; a new DBIRTH fires `OnRediscoveryNeeded`; a tag authored before its birth picks up the birth datatype:
```csharp
[Fact]
public void SparkplugMode_RediscoverPolicy_IsUntilStable()
=> new MqttDriver(new MqttDriverOptions { Mode = MqttMode.SparkplugB }, "d", null)
.RediscoverPolicy.ShouldBe(DiscoveryRediscoverPolicy.UntilStable);
[Fact]
public async Task NewDbirth_FiresOnRediscoveryNeeded()
{
var d = new MqttDriver(new MqttDriverOptions { Mode = MqttMode.SparkplugB }, "d", null);
var fired = false; ((IRediscoverable)d).OnRediscoveryNeeded += (_, _) => fired = true;
d.SimulateNewDbirthForTest("Plant1","EdgeA","Filler2");
fired.ShouldBeTrue();
}
```
2. Run — expect FAIL.
3. Implement.
4. Run — expect PASS.
5. Commit: `git commit -am "feat(mqtt): Sparkplug UntilStable discovery + rediscover-on-DBIRTH"` (+ trailer).
---
## Task 23 (P2): Sparkplug browser tree + `AttributesAsync` + `RequestRebirthAsync`
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 24
**Files:**
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Browser/MqttBrowseSession.cs` (Sparkplug tree Group→EdgeNode→Device→Metric from observed births; `AttributesAsync` = self-describing metric `AttributeInfo{Name, DriverDataType from birth, IsArray, SecurityClass}`; `RequestRebirthAsync(scope)` — the **only** publishing session member, scoped to selected group/edge-node, via the `RebirthRequester` path)
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Browser/MqttDriverBrowser.cs` (Sparkplug discovery filter `spBv1.0/{groupId}/#`; `OpenAsync` still publishes nothing)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Browsing/BrowserSessionService.cs` (dispatch the MQTT-specific `RequestRebirthAsync` for MQTT sessions, gated by the same `DriverOperator` policy that gates the picker; Info-log the target scope)
- Modify: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.Tests/MqttBrowseSessionTests.cs`
### Steps (TDD)
1. Failing tests — birth-driven tree fills; `OpenAsync`/`RootAsync`/`ExpandAsync` publish **nothing**; `RequestRebirthAsync` publishes exactly one scoped NCMD:
```csharp
[Fact]
public async Task SparkplugTree_FillsFromObservedBirths_BrowseNeverPublishes()
{
var s = new MqttBrowseSession(MqttMode.SparkplugB);
s.ObserveBirthForTest("Plant1","EdgeA","Filler1","Temperature", SparkplugDataType.Float);
var groups = await s.RootAsync(default);
groups.ShouldContain(n => n.BrowseName == "Plant1");
s.PublishCountForTest.ShouldBe(0); // passive
}
[Fact]
public async Task RequestRebirthAsync_ScopedToNode_PublishesOneNcmd()
{
var s = new MqttBrowseSession(MqttMode.SparkplugB);
s.ObserveBirthForTest("Plant1","EdgeA","Filler1","T", SparkplugDataType.Float);
await s.RequestRebirthAsync(scope: "Plant1/EdgeA");
s.PublishCountForTest.ShouldBe(1);
}
```
2. Run — expect FAIL.
3. Implement. `RequestRebirthAsync` is never fired by `OpenAsync`/`RootAsync`/`ExpandAsync` — only on explicit operator click.
4. Run — expect PASS + `dotnet build` the AdminUI.
5. Commit: `git commit -am "feat(mqtt): Sparkplug birth-driven browser tree + scoped Request-rebirth action"` (+ trailer).
---
## Task 24 (P2): AdminUI editor Sparkplug mode shape + validator
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 23
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/TagEditors/MqttTagConfigModel.cs` (Sparkplug `Validate`: `groupId`/`edgeNodeId`/`metricName` required, `deviceId` optional, `dataType` a known Sparkplug type; `ToJson` re-derives `FullName` = `{group}/{node}[/{device}]:{metric}`)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Shared/Uns/TagEditors/MqttTagConfigEditor.razor` (Sparkplug field group under the mode switch)
- Modify: `tests/Server/.../MqttTagConfigModelTests.cs`
### Steps (TDD)
1. Failing tests — Sparkplug validation + FullName derivation:
```csharp
[Fact]
public void Validate_SparkplugMissingMetricName_Fails()
=> MqttTagConfigModel.FromJson("""{"groupId":"Plant1","edgeNodeId":"EdgeA"}""").Validate().ShouldNotBeEmpty();
[Fact]
public void ToJson_Sparkplug_DerivesFullName()
=> MqttTagConfigModel.FromJson("""{"groupId":"Plant1","edgeNodeId":"EdgeA","deviceId":"Filler1","metricName":"Temperature","dataType":"Float"}""")
.ToJson().ShouldContain("Plant1/EdgeA/Filler1:Temperature");
```
2. Run — expect FAIL.
3. Implement.
4. Run — expect PASS + AdminUI build.
5. Commit: `git commit -am "feat(mqtt): typed tag editor Sparkplug mode + validation"` (+ trailer).
---
## Task 25 (P2): C# edge-node simulator fixture + §3.6 live matrix
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.IntegrationTests/SparkplugSimulator/` (a project-owned C# edge-node simulator using the **same** MQTTnet-5 + Tahu-proto path the driver uses — publishes NBIRTH/DBIRTH → periodic N/DDATA; honours a rebirth NCMD; injects seq-gap / alias-reuse / N/DDEATH on demand)
- Modify: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.IntegrationTests/Docker/docker-compose.yml` (add the simulator container against the same TLS+auth Mosquitto)
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.IntegrationTests/SparkplugLiveTests.cs` (category `LiveIntegration`, env-gated: birth→data→alias-resolve→OnDataChange; rebirth recovery; death→STALE; browser passive window asserts **no NCMD published by open/browse**; explicit `RequestRebirthAsync` node-vs-group enumeration)
### Steps
1. Build the simulator (encode via the same generated proto — validates encode/decode symmetry).
2. Write env-gated live tests covering the §3.6 matrix + browser passivity.
3. Offline skip proof:
```bash
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.IntegrationTests # expect: Skipped without MQTT_FIXTURE_ENDPOINT
```
4. Deploy + run live:
```bash
lmxopcua-fix sync mqtt && lmxopcua-fix up mqtt sparkplug
export MQTT_FIXTURE_ENDPOINT=10.100.0.35:8883
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.IntegrationTests --filter "Category=LiveIntegration" # expect: PASS
```
5. Commit: `git commit -am "test(mqtt): C# Sparkplug edge-node simulator + §3.6 live matrix"` (+ trailer).
---
## Task 26 (P2): Live `/run` verify Sparkplug — **P2 COMPLETE**
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Modify: `docs/plans/2026-07-15-driver-expansion-tracking.md` (mark MQTT/Sparkplug P1+P2 code-complete + live-verified)
- Modify: `CLAUDE.md` (if a driver-fleet or endpoint fact changed; propagate to `../scadaproj/CLAUDE.md` OtOpcUa entry per the cross-repo rule)
### Steps
1. On docker-dev (`:9200`): author an `Mqtt` driver (SparkplugB mode) against the simulator; browse the birth-driven picker, click **Request rebirth**, pick a metric, deploy.
2. Confirm live values + rediscover-on-DBIRTH + death→STALE via Client.CLI reads/subscribe.
3. Confirm the typed editor renders both modes (live-verify — Razor bugs pass unit tests).
4. Update tracking + (if needed) CLAUDE.md/scadaproj index.
5. Commit: `git commit -am "docs(mqtt): Sparkplug P2 live-verified; MQTT driver complete"` (+ trailer).
---
## Deferred / out of scope
- **Write-through (P3 in the design):** `IWritable` — Sparkplug NCMD/DCMD + plain publish, optimistic-Good + self-correct on echo (mirrors Galaxy fire-and-forget), `WriteIdempotent` default-off for non-idempotent Sparkplug commands, write-topic config. See design §3.7 + §10 (P3 row). `MqttDriver` ships **read/subscribe/discover only** in P1+P2 — its absence of `IWritable` means nodes materialize read-only.
- **`DataSet`/`Template` Sparkplug metrics** — unsupported v1 (skip + warn, or raw-JSON as String). §3.5.
- **`Bytes`/`File` metrics** beyond a base64/raw-String fallback. §3.5.
- **EMQX broker** — Mosquitto is the default fixture; EMQX is the documented alternative when a dashboard/built-in Sparkplug tooling helps (§9), not built here.
## Notes on ordering / parallelism
- **P2 is blocked on the P1 milestone (Task 14).** Every P2 task's `blockedBy` chain roots at Task 14 so plain MQTT ships and is live-verified before Sparkplug work begins — and MQTTnet-5/net10 is proven (Task 0) before any proto/Sparkplug effort.
- Genuine parallel pairs: 5‖6, 8‖10, 11‖12, 16‖17, 18‖19, 23‖24.
- DRY: reuse `EquipmentTagRefResolver<MqttTagDefinition>` (as Modbus/OpcUaClient do), the shared `JsonSerializerOptions`, and one `SparkplugCodec` for both decode (ingest) and encode (rebirth NCMD). YAGNI: no write-through plumbing, no DataSet/Template, no EMQX until a need lands.
@@ -0,0 +1,34 @@
{
"planPath": "docs/plans/2026-07-24-mqtt-sparkplug-driver.md",
"note": "Wave 2 MQTT/Sparkplug B driver. Two phases both in-scope: P1 (plain MQTT, Tasks 0-14) is a complete shippable milestone; P2 (Sparkplug B ingest, Tasks 15-26) builds on it and every P2 task chains back to Task 14. Write-through (NCMD/DCMD/plain publish) is DEFERRED (design P3). Hard rules: Task 0 (MQTTnet-5/net10 restore+build under central pinning with transitive pinning OFF) is the mandatory FIRST gate. Hand-roll Tahu proto over ONE MQTTnet-5 client (NOT SparkplugNet — its MQTTnet-4.x transitive pin collides with the repo's deliberately-OFF CentralPackageTransitivePinning). DriverType string is 'Mqtt' everywhere (grep-enforced). Enum-serialization trap: every JSON seam uses shared JsonSerializerOptions with JsonStringEnumConverter. Fixtures enforce TLS+auth, never anonymous/public-broker. Google.Protobuf (3.34.1) + Grpc.Tools (2.76.0) already pinned; MQTTnet is the one new pin. High-risk tasks: 4 (hand-rolled reconnect loop), 18/19 (alias/seq state-machine components), 21 (the full 3.6 ingest matrix).",
"tasks": [
{"id": 0, "subject": "Task 0 (P1): Dependency-validation spike — MQTTnet-5 pin + net10 restore/build under central pinning", "status": "pending", "classification": "standard", "parallelizableWith": []},
{"id": 1, "subject": "Task 1 (P1): .Contracts enums + MqttDriverOptions DTO (name-serialized)", "status": "pending", "classification": "small", "parallelizableWith": [], "blockedBy": [0]},
{"id": 2, "subject": "Task 2 (P1): .Contracts MqttTagDefinition + MqttEquipmentTagParser (plain, strict enum)", "status": "pending", "classification": "standard", "parallelizableWith": [], "blockedBy": [1]},
{"id": 3, "subject": "Task 3 (P1): .Driver + MqttConnection connect/TLS/auth (bounded deadline)", "status": "pending", "classification": "standard", "parallelizableWith": [], "blockedBy": [1]},
{"id": 4, "subject": "Task 4 (P1): MqttConnection hand-rolled reconnect loop (backoff + resubscribe)", "status": "pending", "classification": "high-risk", "parallelizableWith": [], "blockedBy": [3]},
{"id": 5, "subject": "Task 5 (P1): LastValueCache + IReadable (per-ref no-data)", "status": "pending", "classification": "small", "parallelizableWith": [6], "blockedBy": [3]},
{"id": 6, "subject": "Task 6 (P1): ISubscribable plain topic subscribe→OnDataChange + retained seed", "status": "pending", "classification": "standard", "parallelizableWith": [5], "blockedBy": [2, 4]},
{"id": 7, "subject": "Task 7 (P1): MqttDriver shell — IDriver + authored-only ITagDiscovery (Once) + probe interface", "status": "pending", "classification": "standard", "parallelizableWith": [], "blockedBy": [5, 6]},
{"id": 8, "subject": "Task 8 (P1): MqttDriverProbe CONNECT handshake", "status": "pending", "classification": "small", "parallelizableWith": [10], "blockedBy": [3]},
{"id": 9, "subject": "Task 9 (P1): Factory + DriverTypeNames.Mqtt + Host factory/probe registration", "status": "pending", "classification": "standard", "parallelizableWith": [], "blockedBy": [7, 8]},
{"id": 10, "subject": "Task 10 (P1): .Browser bespoke #-observation browser (passive)", "status": "pending", "classification": "standard", "parallelizableWith": [8], "blockedBy": [2]},
{"id": 11, "subject": "Task 11 (P1): Register MqttDriverBrowser (bespoke-first for Mqtt)", "status": "pending", "classification": "trivial", "parallelizableWith": [12], "blockedBy": [10]},
{"id": 12, "subject": "Task 12 (P1): Typed AdminUI editor + validator (plain shape)", "status": "pending", "classification": "standard", "parallelizableWith": [11], "blockedBy": [2]},
{"id": 13, "subject": "Task 13 (P1): Mosquitto+JSON-publisher fixture (TLS+auth) + env-gated live suite", "status": "pending", "classification": "standard", "parallelizableWith": [], "blockedBy": [9]},
{"id": 14, "subject": "Task 14 (P1): Live /run verify on docker-dev — P1 MILESTONE COMPLETE", "status": "pending", "classification": "small", "parallelizableWith": [], "blockedBy": [9, 11, 12, 13]},
{"id": 15, "subject": "Task 15 (P2): Vendor Tahu sparkplug_b.proto + Grpc.Tools codegen", "status": "pending", "classification": "standard", "parallelizableWith": [], "blockedBy": [14]},
{"id": 16, "subject": "Task 16 (P2): SparkplugCodec decode + golden payload vectors", "status": "pending", "classification": "standard", "parallelizableWith": [17], "blockedBy": [15]},
{"id": 17, "subject": "Task 17 (P2): SparkplugTopic parse/format + SparkplugDataType.ToDriverDataType map", "status": "pending", "classification": "small", "parallelizableWith": [16], "blockedBy": [15]},
{"id": 18, "subject": "Task 18 (P2): BirthCache + AliasTable — bind-by-name, rebuild-per-birth", "status": "pending", "classification": "high-risk", "parallelizableWith": [19], "blockedBy": [16, 17]},
{"id": 19, "subject": "Task 19 (P2): SequenceTracker seq-gap + bdSeq death-tie", "status": "pending", "classification": "high-risk", "parallelizableWith": [18], "blockedBy": [16, 17]},
{"id": 20, "subject": "Task 20 (P2): RebirthRequester NCMD encode", "status": "pending", "classification": "small", "parallelizableWith": [], "blockedBy": [16]},
{"id": 21, "subject": "Task 21 (P2): Sparkplug ingest state machine — the 3.6 matrix", "status": "pending", "classification": "high-risk", "parallelizableWith": [], "blockedBy": [18, 19, 20]},
{"id": 22, "subject": "Task 22 (P2): ITagDiscovery UntilStable + IRediscoverable (rediscover-on-DBIRTH)", "status": "pending", "classification": "standard", "parallelizableWith": [], "blockedBy": [21]},
{"id": 23, "subject": "Task 23 (P2): Sparkplug browser tree + AttributesAsync + scoped RequestRebirthAsync", "status": "pending", "classification": "standard", "parallelizableWith": [24], "blockedBy": [20, 21]},
{"id": 24, "subject": "Task 24 (P2): AdminUI editor Sparkplug mode shape + validator", "status": "pending", "classification": "small", "parallelizableWith": [23], "blockedBy": [12, 17]},
{"id": 25, "subject": "Task 25 (P2): C# edge-node simulator fixture + 3.6 live matrix", "status": "pending", "classification": "standard", "parallelizableWith": [], "blockedBy": [21, 22, 23]},
{"id": 26, "subject": "Task 26 (P2): Live /run verify Sparkplug — P2 COMPLETE", "status": "pending", "classification": "small", "parallelizableWith": [], "blockedBy": [23, 24, 25]}
],
"lastUpdated": "2026-07-24"
}
+771
View File
@@ -0,0 +1,771 @@
# MTConnect Agent Driver Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans (or subagent-driven-development) to implement this plan task-by-task.
**Goal:** Ship a read-only, browsable `DriverType = "MTConnect"` Equipment-kind driver (P1 Agent MVP) that surfaces an MTConnect Agent's `/probe` device model as OPC UA nodes and serves live values via `/current` (read) + `/sample` long-poll (subscribe), with browse coming free from the Wave-0 universal discovery browser.
**Architecture:** One `MTConnectDriver` per instance implements `IDriver`, `ITagDiscovery` (`SupportsOnlineDiscovery=true`, `RediscoverPolicy=Once`), `IReadable`, `ISubscribable`, `IHostConnectivityProbe`, `IRediscoverable`**not** `IWritable` (Agent surface is read-only). A thin `IMTConnectAgentClient` seam wraps the HTTP/XML transport (probe/current/sample) so the whole driver is unit-tested against canned XML with no network. A per-instance `MTConnectObservationIndex` holds `dataItemId → latest DataValueSnapshot`, updated by `/current` and the `/sample` pump; `UNAVAILABLE → BadNoCommunication`. Browse is served by the already-shipped `DiscoveryDriverBrowser` — this driver ships **no browser code**, only the `SupportsOnlineDiscovery` opt-in.
**Tech Stack:** `HttpClient` behind the `IMTConnectAgentClient` seam — **TrakHound MTConnect.NET (`-Common` + `-HTTP`, MIT, netstandard2.0) pending the Task 0 verification, else a hand-rolled `System.Xml.Linq` + multipart boundary-reader fallback (drop-in behind the same seam)**; `System.Text.Json` for config DTOs (enum-as-name, `JsonStringEnumConverter` on the probe, `ParseEnum<T>` on the factory); the Wave-0 universal browser seam (`ITagDiscovery.SupportsOnlineDiscovery`).
**Source design:** docs/plans/2026-07-15-mtconnect-driver-design.md
**Program context:** docs/plans/2026-07-15-driver-expansion-program-design.md (§3 shared contract, §4 two-tier browse, §7 fixtures)
**Wave-0 dependency:** docs/plans/2026-07-15-universal-discovery-browser-design.md — the browse picker live-verify (Task 21) is gated on the Wave-0 universal-browser live gate (Gitea #468) being closed.
---
## Cross-cutting constraints (apply to every task)
- **Per-op deadline (R2-01 frozen-peer lesson).** Every agent HTTP call carries a bounded deadline: `/probe` + `/current` wrap in a `CancellationTokenSource(RequestTimeoutMs)` linked to the caller's `ct` AND set `HttpClient.Timeout`; the `/sample` long-poll cannot use `HttpClient.Timeout`, so it runs under a heartbeat watchdog (§7 of the design). No unbounded waits.
- **Enum-serialization trap (project-wide MEMORY).** Enums on any config/tag surface serialize as **name strings**: the AdminUI model writes them via `TagConfigJson.Set` (name), the probe parses with `new JsonStringEnumConverter()`, and the factory keeps enum-carrying DTO fields `string?` + parses via `ParseEnum<T>` (no converter in the factory `JsonOptions`). A numerically-serialized enum faults the driver at deploy.
- **Ctor is connection-free.** The `MTConnectDriver` constructor opens no sockets — every connect happens in `InitializeAsync`. The universal browser's throwaway-instance `CanBrowse` pattern depends on this.
- **Never throw across a capability boundary.** `IReadable` reports per-ref failures as Bad-coded snapshots (throws only if the driver itself is unreachable); `IDriverProbe.ProbeAsync` never throws (returns `Ok=false`).
- **Secrets from env/secret-refs.** No agent URL creds committed or logged.
- **`Float64`/`Int64` are the real `DriverDataType` member names** (verified in `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/DriverDataType.cs`).
---
## Task 0: Verify TrakHound MTConnect.NET pin (or record the hand-roll decision)
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** none (gates the `.csproj` package refs in Task 1)
**Files:**
- Modify: `docs/plans/2026-07-24-mtconnect-driver.md` (record the decision inline under this task)
The design (§2 "Library verification checklist") flags three research-sourced facts as unverifiable offline. Verify against real NuGet before committing to the dependency:
1. `dotnet package search MTConnect.NET-Common --exact-match` (and `-HTTP`) — confirm the pinned version (`6.9.0.2` or then-current) **exists on nuget.org**.
2. In a throwaway project, `dotnet add package MTConnect.NET-Common -v <pin>` + `dotnet restore` — confirm it **restores** and its TFM set includes `netstandard2.0` (loads on net10).
3. Open the restored package's embedded `LICENSE` — confirm the *pinned version* is **MIT** (older releases carried mixed MIT/Apache/"all rights reserved").
**Decision rule:** all three pass ⇒ TrakHound path (Task 1 adds the two `PackageReference`s). Any fail ⇒ **hand-roll fallback**: `HttpClient` + `System.Xml.Linq` for probe/current + a `multipart/x-mixed-replace` boundary reader for sample (~300500 LoC behind the same `IMTConnectAgentClient` seam; the driver above the seam is byte-identical either way). Record the chosen path and the observed version/TFM/license as a one-paragraph **DECISION** note appended to this task in the plan file.
**No test / no commit** for this task (it's a decision + a doc edit). Commit the plan edit with the next task, or standalone:
```bash
git add docs/plans/2026-07-24-mtconnect-driver.md
git commit -m "docs(mtconnect): record TrakHound-vs-hand-rolled client decision (Task 0)"
```
---
## Task 1: Scaffold the two driver projects + the test project
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** none (every later task builds on these projects)
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Contracts/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Contracts.csproj`
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.csproj`
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests.csproj`
- Modify: `ZB.MOM.WW.OtOpcUa.slnx` (add the three `<Project Path=...>` lines)
Steps:
- Copy the `.Contracts` csproj boilerplate from `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Contracts/…csproj`: `net10.0`, `Nullable=enable`, `ImplicitUsings=enable`, `TreatWarningsAsErrors=true`. `.Contracts` references **only** `Core.Abstractions` (no backend NuGet). Add `GenerateDocumentationFile=true` (matches the fleet).
- The runtime `Driver.MTConnect.csproj` references `Core`, `Core.Abstractions`, `.Contracts`, and — **per the Task 0 decision** — either the two TrakHound packages or nothing extra (hand-roll uses only the BCL). Add `InternalsVisibleTo` the Tests project (mirror Modbus).
- The `.Tests` csproj: xUnit + Shouldly (copy `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests/…csproj` package block), ProjectReferences to `Driver.MTConnect` + `.Contracts`. Mark the future `Fixtures/*.xml` as `CopyToOutputDirectory` (`<Content Include="Fixtures\**\*.xml"><CopyToOutputDirectory>PreserveNewest…`).
- Add all three to `ZB.MOM.WW.OtOpcUa.slnx` beside the Modbus entries (lines ~2992 pattern).
Verify:
```bash
dotnet build src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.csproj # expect: PASS (empty compile)
```
```bash
git add -A && git commit -m "build(mtconnect): scaffold Contracts+Driver+Tests projects, add to slnx (Task 1)"
```
---
## Task 2: `MTConnectDriverOptions` + `MTConnectTagDefinition` in `.Contracts`
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 3, Task 4
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Contracts/MTConnectDriverOptions.cs`
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Contracts/MTConnectTagDefinition.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests/MTConnectDriverOptionsTests.cs`
Mirror `ModbusDriverOptions`: strongly-typed record/POCO holding `AgentUri` (required), optional `DeviceName` scope, `RequestTimeoutMs`, `SampleIntervalMs`, `SampleCount`, `HeartbeatMs`, a `Probe` sub-record (mirror `ModbusProbeOptions`: `Enabled`/`Interval`/`Timeout`), and a `Reconnect` sub-record (`MinBackoffMs`/`MaxBackoffMs`/multiplier — mirror `ModbusReconnectOptions`). `MTConnectTagDefinition`: one record per authored tag — `FullName` (= `dataItemId`), `DriverDataType`, `IsArray`, `ArrayDim`, plus `mtCategory`/`mtType`/`mtSubType`/`units` metadata.
TDD — write the failing test first (defaults + required-field shape):
```csharp
[Fact]
public void Options_default_sample_and_heartbeat_are_sane()
{
var o = new MTConnectDriverOptions { AgentUri = "http://agent:5000" };
o.SampleIntervalMs.ShouldBeGreaterThan(0);
o.HeartbeatMs.ShouldBeGreaterThan(0);
o.Probe.Enabled.ShouldBeTrue();
}
```
```bash
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests --filter "FullyQualifiedName~MTConnectDriverOptionsTests" # RED, then GREEN
git add -A && git commit -m "feat(mtconnect): driver options + tag definition records (Task 2)"
```
---
## Task 3: `MTConnectDataTypeInference.Infer` + golden type-map test
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 2, Task 4
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Contracts/MTConnectDataTypeInference.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests/MTConnectDataTypeInferenceTests.cs`
Pure `static DriverDataType Infer(string category, string? type, string? units, string? representation)` implementing the §3.3 table. It lives in `.Contracts` so the driver, browser-commit, and the typed editor all agree. Return `(DriverDataType, bool isArray, uint? arrayDim)` (or a small record) so `TIME_SERIES` can carry `IsArray=true` + declared `sampleCount`.
Golden table (design §3.3):
| category / shape | result |
|---|---|
| `SAMPLE` numeric with `units` | `Float64` |
| `SAMPLE` `representation=TIME_SERIES` | `Float64` array (`IsArray=true`, `ArrayDim=sampleCount` or null) |
| `EVENT` numeric type (`PartCount`, `Line`) | `Int64` |
| `EVENT` controlled-vocab (`Execution`, `ControllerMode`, `Availability`) | `String` |
| `EVENT` free text (`Program`, `Block`, `Message`) | `String` |
| `CONDITION` | `String` |
TDD — one `[Theory]` covering every row (golden):
```csharp
[Theory]
[InlineData("SAMPLE", "Position", "MILLIMETER", null, DriverDataType.Float64)]
[InlineData("EVENT", "PartCount", null, null, DriverDataType.Int64)]
[InlineData("EVENT", "Execution", null, null, DriverDataType.String)]
[InlineData("EVENT", "Program", null, null, DriverDataType.String)]
[InlineData("CONDITION", "Temperature", null, null, DriverDataType.String)]
public void Infer_maps_the_v1_table(string cat, string type, string? units, string? rep, DriverDataType expected)
=> MTConnectDataTypeInference.Infer(cat, type, units, rep).DataType.ShouldBe(expected);
[Fact]
public void TimeSeries_sample_is_a_float64_array_with_declared_count()
{
var r = MTConnectDataTypeInference.Infer("SAMPLE", "Position", "MM", "TIME_SERIES", sampleCount: 8);
r.DataType.ShouldBe(DriverDataType.Float64);
r.IsArray.ShouldBeTrue();
r.ArrayDim.ShouldBe(8u);
}
```
```bash
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests --filter "FullyQualifiedName~MTConnectDataTypeInferenceTests" # RED, then GREEN
git add -A && git commit -m "feat(mtconnect): pure data-type inference table + golden test (Task 3)"
```
---
## Task 4: Capture the canned XML fixtures
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 2, Task 3
**Files:**
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests/Fixtures/probe.xml`
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests/Fixtures/current.xml`
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests/Fixtures/sample.xml`
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests/Fixtures/sample-gap.xml`
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests/Fixtures/current-unavailable.xml`
These canned docs drive the bulk of unit coverage with no network (design §8.1). Capture from a demo agent (`https://demo.mtconnect.org/probe` / `/current` / `/sample`) or hand-author minimal valid MTConnect 1.x XML covering:
- `probe.xml` — one `MTConnectDevices` with a `Device` → nested `Component``DataItem`s spanning every §3.3 category (a `SAMPLE` w/ units, a `TIME_SERIES` w/ `sampleCount`, an `EVENT` numeric, an `EVENT` controlled-vocab, a `CONDITION`), each with a distinct `id`.
- `current.xml` — an `MTConnectStreams` with a `Header instanceId=... nextSequence=...` and one observation per data item, including at least one `<... >UNAVAILABLE</...>`.
- `sample.xml` — an `MTConnectStreams` chunk with several observations and a `Header nextSequence` that continues contiguously from `current.xml`.
- `sample-gap.xml` — a `Header` whose `firstSequence` is **newer** than the `from` the driver would request (the ring-buffer-overflow / forced-`nextSequence`-gap case Task 7 + Task 11 assert re-baseline on).
- `current-unavailable.xml` — every observation `UNAVAILABLE` (Task 8's quality-mapping fixture).
No code, no test yet — verify the files copy to bin:
```bash
dotnet build tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests # PASS; then confirm bin/.../Fixtures/*.xml exist
git add -A && git commit -m "test(mtconnect): canned probe/current/sample XML fixtures incl. forced-gap (Task 4)"
```
---
## Task 5: `IMTConnectAgentClient` seam + return DTOs
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 2, Task 3, Task 4
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect/IMTConnectAgentClient.cs`
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect/MTConnectDtos.cs` (parsed model + observation records)
Define the seam the whole driver is tested behind:
```csharp
public interface IMTConnectAgentClient
{
Task<MTConnectProbeModel> ProbeAsync(CancellationToken ct);
Task<MTConnectStreamsResult> CurrentAsync(CancellationToken ct);
IAsyncEnumerable<MTConnectStreamsResult> SampleAsync(long from, CancellationToken ct);
}
```
`MTConnectProbeModel` — devices → components (nested) → data items (`Id`, `Name?`, `Category`, `Type`, `SubType?`, `Units?`, `Representation?`, `SampleCount?`). `MTConnectStreamsResult``long InstanceId`, `long NextSequence`, `long FirstSequence`, and `IReadOnlyList<MTConnectObservation>` (`DataItemId`, `Value` (string or `"UNAVAILABLE"`), `TimestampUtc`). These DTOs are the neutral shape both the TrakHound adapter and the hand-roll fallback produce, so the driver never sees TrakHound types (or `XElement`) directly.
Verify (compiles only):
```bash
dotnet build src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect # PASS
git add -A && git commit -m "feat(mtconnect): IMTConnectAgentClient seam + neutral parse DTOs (Task 5)"
```
---
## Task 6: `MTConnectAgentClient` — parse `/probe` into the device model
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect/MTConnectAgentClient.cs` (probe leg only this task)
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests/MTConnectProbeParseTests.cs`
Implement `ProbeAsync` — either via TrakHound's `MTConnectHttpClient` (map its `IDevice`/`IComponent`/`IDataItem` into the neutral `MTConnectProbeModel`) or the hand-roll `System.Xml.Linq` walk of `probe.xml`. The `/probe` call carries the linked-CTS deadline. To keep the parse itself unit-testable without a socket, factor the byte→model parse into an internal static (`MTConnectProbeParser.Parse(Stream|string)`) and feed it the fixture directly.
TDD:
```csharp
[Fact]
public async Task Probe_parse_yields_nested_components_and_all_dataitems()
{
var xml = await File.ReadAllTextAsync("Fixtures/probe.xml");
var model = MTConnectProbeParser.Parse(xml);
model.Devices.ShouldHaveSingleItem();
var ids = model.Devices[0].AllDataItems().Select(d => d.Id).ToList();
ids.ShouldContain("<the CONDITION dataItem id from the fixture>");
model.Devices[0].Components.ShouldNotBeEmpty(); // nesting preserved
}
```
```bash
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests --filter "FullyQualifiedName~MTConnectProbeParseTests" # RED, then GREEN
git add -A && git commit -m "feat(mtconnect): agent client /probe parse to device model (Task 6)"
```
---
## Task 7: `MTConnectAgentClient` — parse `/current` + `/sample`, detect the sequence gap
**Classification:** high-risk (ring-buffer / sequence paging correctness)
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect/MTConnectAgentClient.cs` (add current/sample legs + `MTConnectStreamsParser`)
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests/MTConnectStreamsParseTests.cs`
`CurrentAsync` parses `current.xml``MTConnectStreamsResult` (Header `instanceId`/`nextSequence`/`firstSequence` + observations). `SampleAsync(from, ct)` yields one `MTConnectStreamsResult` per multipart chunk and **exposes the gap signal**: expose a helper `bool IsSequenceGap(long requestedFrom, MTConnectStreamsResult chunk) => chunk.FirstSequence > requestedFrom` so Task 11's pump can re-baseline. Advance the caller's `from` to `chunk.NextSequence` (contiguous). Chunk framing (the multipart boundary reader) is TrakHound-internal or the hand-roll's boundary reader — either way the parser is fed one chunk's XML.
TDD (the forced-gap fixture is the load-bearing case):
```csharp
[Fact]
public void Current_parse_reads_header_sequences_and_observations()
{
var r = MTConnectStreamsParser.Parse(File.ReadAllText("Fixtures/current.xml"));
r.NextSequence.ShouldBeGreaterThan(0);
r.Observations.ShouldNotBeEmpty();
}
[Fact]
public void Sequence_gap_is_detected_when_firstSequence_exceeds_requested_from()
{
var chunk = MTConnectStreamsParser.Parse(File.ReadAllText("Fixtures/sample-gap.xml"));
MTConnectAgentClient.IsSequenceGap(requestedFrom: 1, chunk).ShouldBeTrue();
}
```
```bash
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests --filter "FullyQualifiedName~MTConnectStreamsParseTests" # RED, then GREEN
git add -A && git commit -m "feat(mtconnect): current/sample parse + sequence-gap detection (Task 7)"
```
---
## Task 8: `MTConnectObservationIndex` + `UNAVAILABLE → BadNoCommunication`
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect/MTConnectObservationIndex.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests/MTConnectObservationIndexTests.cs`
Thread-safe `dataItemId → DataValueSnapshot` map updated by `/current` and `/sample`. The snapshot builder maps a raw observation → `DataValueSnapshot`:
- value `UNAVAILABLE``new(null, BadNoCommunication, ts, now)` where `private const uint BadNoCommunication = 0x80310000u;` (design §3.3 — declared as a const, the Modbus `StatusBadCommunicationError` pattern; renders by name in the CLI's `SnapshotFormatter`).
- a present value → coerced to the tag's `DriverDataType` with `StatusCode = Good (0)`, `SourceTimestampUtc = observation.timestamp`.
- a `dataItemId` never seen / empty condition → `Bad`-coded snapshot.
TDD (drive off `current.xml` + `current-unavailable.xml`):
```csharp
[Fact]
public void Unavailable_observation_maps_to_BadNoCommunication_with_null_value()
{
var idx = new MTConnectObservationIndex();
idx.Apply(MTConnectStreamsParser.Parse(File.ReadAllText("Fixtures/current-unavailable.xml")));
var snap = idx.Get("<a dataItemId from the fixture>");
snap.Value.ShouldBeNull();
snap.StatusCode.ShouldBe(0x80310000u);
}
[Fact]
public void Present_value_indexes_good_with_source_timestamp()
{
var idx = new MTConnectObservationIndex();
idx.Apply(MTConnectStreamsParser.Parse(File.ReadAllText("Fixtures/current.xml")));
var snap = idx.Get("<a good dataItemId>");
snap.StatusCode.ShouldBe(0u);
snap.SourceTimestampUtc.ShouldNotBeNull();
}
```
```bash
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests --filter "FullyQualifiedName~MTConnectObservationIndexTests" # RED, then GREEN
git add -A && git commit -m "feat(mtconnect): observation index + UNAVAILABLE->BadNoCommunication (Task 8)"
```
---
## Task 9: `MTConnectDriver` shell — `IDriver` lifecycle
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect/MTConnectDriver.cs` (IDriver members only this task)
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests/MTConnectDriverLifecycleTests.cs`
Ctor `MTConnectDriver(MTConnectDriverOptions options, string driverInstanceId, Func<MTConnectDriverOptions, IMTConnectAgentClient>? agentClientFactory = null, ILogger<MTConnectDriver>? logger = null)`**connection-free**. `agentClientFactory` defaults to the real `MTConnectAgentClient`; tests inject a canned-XML fake implementing `IMTConnectAgentClient`. Implement:
- `InitializeAsync`: build the client, run one `/probe` under the deadline (cache `IDevice[]` model + `Header.instanceId`), prime the index with one `/current`; `DriverState.Healthy` on success, rethrow → Faulted on failure.
- `ReinitializeAsync`: stop the sample stream, re-Initialize (config-only unless `AgentUri`/`DeviceName` changed).
- `ShutdownAsync`: stop stream, dispose client.
- `GetHealth`: `DriverHealth(State, LastSuccessfulRead, LastError)``LastSuccessfulRead` = last `/current`|`/sample` chunk time.
- `GetMemoryFootprint` / `FlushOptionalCachesAsync`: footprint ≈ probe model + index; flush drops the probe/browse cache, keeps the index.
TDD with a canned-XML fake client:
```csharp
[Fact]
public async Task Initialize_primes_index_and_reports_healthy()
{
var fake = CannedAgentClient.FromFixtures(); // test helper: serves probe.xml + current.xml
var d = new MTConnectDriver(Opts(), "mt1", _ => fake);
await d.InitializeAsync("{\"agentUri\":\"http://x\"}", default);
d.GetHealth().State.ShouldBe(DriverState.Healthy);
}
```
```bash
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests --filter "FullyQualifiedName~MTConnectDriverLifecycleTests" # RED, then GREEN
git add -A && git commit -m "feat(mtconnect): driver shell IDriver lifecycle over agent-client seam (Task 9)"
```
---
## Task 10: `IReadable.ReadAsync``/current`, ordered snapshots
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 12, Task 13
**Files:**
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect/MTConnectDriver.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests/MTConnectReadTests.cs`
`ReadAsync(fullReferences, ct)`: one `/current` under the per-call deadline, index the whole-device response, return **one `DataValueSnapshot` per requested ref in order**; a ref absent from the response → a `Bad`-coded snapshot (never a throw). Batch reads cost one round-trip.
TDD:
```csharp
[Fact]
public async Task Read_returns_one_snapshot_per_ref_in_order_absent_ref_is_bad()
{
var d = await InitializedDriver();
var res = await d.ReadAsync(new[] { "<good id>", "does-not-exist" }, default);
res.Count.ShouldBe(2);
res[0].StatusCode.ShouldBe(0u);
(res[1].StatusCode & 0x80000000u).ShouldBe(0x80000000u); // Bad
}
```
```bash
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests --filter "FullyQualifiedName~MTConnectReadTests" # RED, then GREEN
git add -A && git commit -m "feat(mtconnect): IReadable via /current, ordered per-ref snapshots (Task 10)"
```
---
## Task 11: `ISubscribable``/sample` long-poll pump + ring-buffer re-baseline
**Classification:** high-risk (streaming state + re-baseline correctness)
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect/MTConnectDriver.cs`
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect/MTConnectSampleHandle.cs` (`ISubscriptionHandle`)
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests/MTConnectSubscribeTests.cs`
- `SubscribeAsync`: on the **first** subscription, start the shared sample stream from `nextSequence`; record the subscribed ref set; immediately fire `OnDataChange` for each subscribed ref from the primed `/current` (initial-data convention); return a handle (monotonic id + `DiagnosticId`). The stream-start handshake must complete inside the invoker's Subscribe timeout; the long-lived pump then runs under the heartbeat watchdog, not that timeout.
- Pump: each chunk → for each observation whose `dataItemId ∈` subscribed set, update the index + raise `OnDataChange(handle, dataItemId, snapshot)`; advance `from = nextSequence`.
- **Ring-buffer overflow:** when `IsSequenceGap(from, chunk)` (Task 7) → re-`/current` to re-baseline, then resume from the new `nextSequence`.
- `UnsubscribeAsync`: drop the handle's refs; stop the shared stream when the set empties.
TDD — assert the initial-data callback AND that a forced gap triggers a re-baseline `/current` (drive `SampleAsync` to yield `sample-gap.xml`):
```csharp
[Fact]
public async Task Subscribe_fires_initial_data_from_current()
{
var d = await InitializedDriver();
var seen = new List<string>();
d.OnDataChange += (_, e) => seen.Add(e.FullReference);
await d.SubscribeAsync(new[] { "<good id>" }, TimeSpan.FromMilliseconds(500), default);
seen.ShouldContain("<good id>");
}
[Fact]
public async Task Sequence_gap_triggers_recurrent_current_rebaseline()
{
var fake = CannedAgentClient.WithGapThenResume(); // sample-gap.xml then a contiguous chunk
var d = await InitializedDriver(fake);
await d.SubscribeAsync(new[] { "<good id>" }, TimeSpan.FromMilliseconds(50), default);
await fake.PumpOnce();
fake.CurrentCallCount.ShouldBeGreaterThan(1); // initial prime + re-baseline
}
```
```bash
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests --filter "FullyQualifiedName~MTConnectSubscribeTests" # RED, then GREEN
git add -A && git commit -m "feat(mtconnect): ISubscribable /sample pump + ring-buffer re-baseline (Task 11)"
```
---
## Task 12: `ITagDiscovery.DiscoverAsync` + `SupportsOnlineDiscovery=true`
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 10, Task 13
**Files:**
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect/MTConnectDriver.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests/MTConnectDiscoverTests.cs`
`public bool SupportsOnlineDiscovery => true;` + `public DiscoveryRediscoverPolicy RediscoverPolicy => DiscoveryRediscoverPolicy.Once;` (probe is synchronous + complete — this one member override is the whole browse opt-in; the Wave-0 universal browser does the rest). `DiscoverAsync(builder, ct)` streams the cached/`/probe` model into the builder:
- each `Device``builder.Folder(name, name)` → child builder;
- each nested `Component` → recursive `Folder(name, displayName)` on the parent's child builder;
- each `DataItem``child.Variable(browseName, displayName, attr)` with `browseName = dataItem.Name ?? dataItem.Id`, and
`attr = new DriverAttributeInfo(FullName: dataItem.Id, DriverDataType: <Infer(...).DataType>, IsArray: rep==TIME_SERIES, ArrayDim: <declared sampleCount or null>, SecurityClass: SecurityClassification.ViewOnly, IsHistorized: false, IsAlarm: category=="CONDITION")`.
- If `DeviceName` is set, stream only that device's subtree.
**Critical:** `FullName == dataItem.Id` — the value the universal browser commits as `TagConfig.FullName` and the key read/subscribe resolve against, aligned by construction.
TDD with a capturing builder (a test double, or reuse Wave-0's `CapturingAddressSpaceBuilder` if referenceable):
```csharp
[Fact]
public async Task Discover_streams_device_component_dataitem_tree_leaf_fullname_is_dataitemid()
{
var d = await InitializedDriver();
var cap = new CapturingBuilder();
await d.DiscoverAsync(cap, default);
cap.Variables.Select(v => v.Attr.FullName).ShouldContain("<CONDITION dataItem id>");
cap.Variables.Single(v => v.Attr.IsAlarm).Attr.DriverDataType.ShouldBe(DriverDataType.String);
d.SupportsOnlineDiscovery.ShouldBeTrue();
d.RediscoverPolicy.ShouldBe(DiscoveryRediscoverPolicy.Once);
}
```
```bash
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests --filter "FullyQualifiedName~MTConnectDiscoverTests" # RED, then GREEN
git add -A && git commit -m "feat(mtconnect): ITagDiscovery streams tree + SupportsOnlineDiscovery opt-in (Task 12)"
```
---
## Task 13: `IHostConnectivityProbe` + `IRediscoverable` (instanceId watch)
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 10, Task 12
**Files:**
- Modify: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect/MTConnectDriver.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests/MTConnectHostAndRediscoverTests.cs`
- `IHostConnectivityProbe`: one `HostConnectivityStatus` per Agent (`HostName = AgentUri`); a cheap periodic `/probe` (or the sample-stream heartbeat) flips `Running ↔ Stopped`; raise `OnHostStatusChanged` on transition. Enable/interval from `MTConnectDriverOptions.Probe`.
- `IRediscoverable`: cache `Header.instanceId` from Initialize; on every `/current`/`/sample` chunk, a **changed** `instanceId` means the Agent restarted / its model changed → raise `OnRediscoveryNeeded(new("MTConnect agent instanceId changed", null))`. This is why `RediscoverPolicy = Once` is safe — instanceId change, not polling, drives re-discovery.
TDD:
```csharp
[Fact]
public async Task InstanceId_change_raises_rediscovery()
{
var fake = CannedAgentClient.WithInstanceIdChangeOnNextChunk();
var d = await InitializedDriver(fake);
RediscoveryEventArgs? got = null;
((IRediscoverable)d).OnRediscoveryNeeded += (_, e) => got = e;
await d.SubscribeAsync(new[] { "<good id>" }, TimeSpan.FromMilliseconds(50), default);
await fake.PumpOnce();
got.ShouldNotBeNull();
}
```
```bash
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests --filter "FullyQualifiedName~MTConnectHostAndRediscoverTests" # RED, then GREEN
git add -A && git commit -m "feat(mtconnect): host-connectivity probe + instanceId rediscover (Task 13)"
```
---
## Task 14: `MTConnectDriverProbe : IDriverProbe`
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 15
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect/MTConnectDriverProbe.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests/MTConnectDriverProbeTests.cs`
Mirror `ModbusDriverProbe`: `DriverType => "MTConnect"`; parse the config DTO with a `JsonSerializerOptions` carrying `new JsonStringEnumConverter()` (copy `ModbusDriverProbe._opts` — the enum-serialization gotcha); one-shot `GET {AgentUri}/probe` under `timeout` (linked-CTS); `Ok=true`+latency on any valid `MTConnectDevices` response; `Ok=false`+message on TCP/HTTP/timeout failure. **Never throws** (`IDriverProbe` contract).
TDD (unit — parse + never-throw; live reachability is Task 20's integration job):
```csharp
[Fact]
public async Task Probe_on_unreachable_agent_returns_not_ok_and_does_not_throw()
{
var probe = new MTConnectDriverProbe();
var r = await probe.ProbeAsync("{\"agentUri\":\"http://127.0.0.1:1/\"}", TimeSpan.FromMilliseconds(300), default);
r.Ok.ShouldBeFalse();
r.Message.ShouldNotBeNull();
}
[Fact]
public async Task Probe_on_blank_config_returns_not_ok()
=> (await new MTConnectDriverProbe().ProbeAsync("{}", TimeSpan.FromSeconds(1), default)).Ok.ShouldBeFalse();
```
```bash
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests --filter "FullyQualifiedName~MTConnectDriverProbeTests" # RED, then GREEN
git add -A && git commit -m "feat(mtconnect): IDriverProbe one-shot /probe reachability (Task 14)"
```
---
## Task 15: `MTConnectDriverFactoryExtensions`
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 14
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect/MTConnectDriverFactoryExtensions.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests/MTConnectFactoryTests.cs`
Copy the Modbus factory: `public const string DriverTypeName = "MTConnect";` `Register(DriverFactoryRegistry registry, ILoggerFactory? loggerFactory = null)``registry.Register(DriverTypeName, (id, json) => CreateInstance(id, json, loggerFactory))`. `CreateInstance` deserializes an `MTConnectDriverConfigDto` (nullable-init DTO with `AgentUri`, `DeviceName`, timeouts, `Tags`, `Probe`), validates `AgentUri` present, builds `MTConnectDriverOptions`, returns `new MTConnectDriver(options, id, agentClientFactory: null, logger: loggerFactory?.CreateLogger<MTConnectDriver>())`. Factory `JsonOptions` mirror Modbus (`PropertyNameCaseInsensitive`, `ReadCommentHandling=Skip`, `AllowTrailingCommas`, **no enum converter** — enum-carrying DTO fields stay `string?` + go through `ParseEnum<T>`).
TDD:
```csharp
[Fact]
public void Factory_builds_a_driver_from_minimal_config()
{
var d = MTConnectDriverFactoryExtensions.CreateInstance("mt1", "{\"agentUri\":\"http://a:5000\"}");
d.DriverType.ShouldBe("MTConnect");
d.DriverInstanceId.ShouldBe("mt1");
}
[Fact]
public void Factory_rejects_config_without_agentUri()
=> Should.Throw<InvalidOperationException>(() => MTConnectDriverFactoryExtensions.CreateInstance("mt1", "{}"));
```
```bash
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.Tests --filter "FullyQualifiedName~MTConnectFactoryTests" # RED, then GREEN
git add -A && git commit -m "feat(mtconnect): driver factory + config DTO (Task 15)"
```
---
## Task 16: Host registration + `DriverTypeNames.MTConnect` + guard-test parity
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/DriverTypeNames.cs` (add `MTConnect` const + append to `All`)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs` (Register call + probe alias + `TryAddEnumerable`)
- Modify: `tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests.csproj` (add ProjectReference to `Driver.MTConnect`)
Three coupled edits that **must land together** to keep `DriverTypeNamesGuardTests` green (it asserts exact-set parity between `DriverTypeNames.All` and the reflection-discovered registered factories, scanning `ZB.MOM.WW.OtOpcUa.Driver.*.dll` in the test's bin):
1. Add `public const string MTConnect = "MTConnect";` to `DriverTypeNames` and append it to `All` (design uses `DriverTypeNames.MTConnect` in the editor map/validator per the CLAUDE.md "keyed off constants" rule).
2. In `DriverFactoryBootstrap`: add `using MTConnectProbe = Driver.MTConnect.MTConnectDriverProbe;`, `Driver.MTConnect.MTConnectDriverFactoryExtensions.Register(registry, loggerFactory);` in `Register(...)`, and `services.TryAddEnumerable(ServiceDescriptor.Singleton<IDriverProbe, MTConnectProbe>());` in `AddOtOpcUaDriverProbes` (so the probe reaches admin-only nodes — the admin-pinned Test-Connect singleton; `TryAddEnumerable` prevents the fused-node double-register that would make `ToDictionary(p=>p.DriverType)` throw).
3. Add the `Driver.MTConnect` ProjectReference to the guard-test project — **without it the new assembly isn't in bin, the guard doesn't discover the factory, and the new constant fails `Every_constant_matches_a_registered_factory`.**
Verify (the guard test is the gate here):
```bash
dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests --filter "FullyQualifiedName~DriverTypeNamesGuardTests" # RED before the ProjectReference/Register land together, GREEN after
dotnet build src/Server/ZB.MOM.WW.OtOpcUa.Host # PASS
git add -A && git commit -m "feat(mtconnect): host registration + DriverTypeNames const + guard parity (Task 16)"
```
---
## Task 17: AdminUI typed model `MTConnectTagConfigModel`
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/TagEditors/MTConnectTagConfigModel.cs`
- Test: `tests/Server/<AdminUI test project>/MTConnectTagConfigModelTests.cs` (mirror the Modbus model test's location)
Copy `ModbusTagConfigModel`: pure `FromJson`/`ToJson`/`Validate`, **preserves unknown keys** via the `TagConfigJson` bag. Fields: `FullName` (dataItemId, required), `MtCategory`/`MtType`/`MtSubType` (from probe — read-only in UI), `DataType` (`DriverDataType` override), `Units`, `MtDevice`/`MtComponent` (author context). `ToJson` writes enums via `TagConfigJson.Set` (**name strings** — the enum-serialization trap). `Validate()` returns an error when `FullName` is blank.
TDD:
```csharp
[Fact]
public void Roundtrip_preserves_unknown_keys_and_writes_enum_as_name()
{
var json = "{\"fullName\":\"dev1_pos\",\"dataType\":\"Float64\",\"somethingUnknown\":42}";
var m = MTConnectTagConfigModel.FromJson(json);
var outJson = m.ToJson();
outJson.ShouldContain("\"somethingUnknown\":42");
outJson.ShouldContain("\"dataType\":\"Float64\""); // name, never a number
}
[Fact]
public void Validate_blocks_blank_fullname()
=> MTConnectTagConfigModel.FromJson("{}").Validate().ShouldNotBeNull();
```
```bash
dotnet test tests/Server/<AdminUI test project> --filter "FullyQualifiedName~MTConnectTagConfigModelTests" # RED, then GREEN
git add -A && git commit -m "feat(mtconnect): typed AdminUI tag-config model (Task 17)"
```
---
## Task 18: AdminUI editor razor + map/validator registration
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Shared/Uns/TagEditors/MTConnectTagConfigEditor.razor`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/TagEditors/TagConfigEditorMap.cs`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/TagEditors/TagConfigValidator.cs`
Thin razor shell over `MTConnectTagConfigModel` (mirror `ModbusTagConfigEditor.razor`): `FullName` text, read-only `mtCategory`/`mtType`, a `DataType` override `<select>`. Because most tags arrive via the browse picker (which fills the model), the editor is mostly a confirm/override surface. Register:
- `[DriverTypeNames.MTConnect] = typeof(Components.Shared.Uns.TagEditors.MTConnectTagConfigEditor)` in `TagConfigEditorMap`.
- `[DriverTypeNames.MTConnect] = j => MTConnectTagConfigModel.FromJson(j).Validate()` in `TagConfigValidator`.
(No AdminUI `IDriverBrowser` DI line — browse is the universal browser, already registered.)
Verify (AdminUI has no bUnit — Razor binding is validated live in Task 21; here just compile):
```bash
dotnet build src/Server/ZB.MOM.WW.OtOpcUa.AdminUI # PASS
git add -A && git commit -m "feat(mtconnect): typed tag editor razor + map/validator entries (Task 18)"
```
---
## Task 19: `mtconnect/cppagent` docker fixture
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 17, Task 18
**Files:**
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.IntegrationTests/Docker/docker-compose.yml`
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.IntegrationTests/Docker/Devices.xml`
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.IntegrationTests/Docker/agent.cfg` (if the image needs it)
Compose one `mtconnect/cppagent` service, **`project: lmxopcua` label on the service** (host-side `lmxopcua-fix` convention), seeded with a canned `Devices.xml` (+ the image's built-in adapter/simulator so `/current` returns live-ish data), exposed on the shared docker host `10.100.0.35`. Deploy via `lmxopcua-fix sync mtconnect` + `lmxopcua-fix up mtconnect` (repo `Docker/` is source of truth; `/opt/otopcua-mtconnect/` is the mirror). Model the compose on the Modbus fixture's shape.
No .NET test in this task (fixture only). Verify the compose parses:
```bash
docker compose -f tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.IntegrationTests/Docker/docker-compose.yml config # PASS
git add -A && git commit -m "test(mtconnect): cppagent docker fixture + seeded Devices.xml (Task 19)"
```
---
## Task 20: Env-gated integration suite against cppagent
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.IntegrationTests/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.IntegrationTests.csproj`
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.IntegrationTests/MTConnectAgentIntegrationTests.cs`
- Modify: `ZB.MOM.WW.OtOpcUa.slnx`
A `*.IntegrationTests` suite that reads the agent endpoint from an env var (e.g. `MTCONNECT_AGENT_ENDPOINT`, default `http://10.100.0.35:5000`) and **skips cleanly when unset/down** (the Modbus/S7 pattern — `Skip.If(...)` or a fixture-reachability guard). Cover the real HTTP round-trips a canned fixture can't: `MTConnectDriverProbe` reachability green, `InitializeAsync` + `DiscoverAsync` build a non-empty tree, `ReadAsync` returns live values, `SubscribeAsync` delivers at least one `OnDataChange` from the live `/sample` stream.
Verify it skips offline (safe on macOS with no fixture up):
```bash
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.IntegrationTests # all tests SKIP when the env var/endpoint is absent
git add -A && git commit -m "test(mtconnect): env-gated cppagent integration suite (Task 20)"
```
---
## Task 21: Live `/run` verify on docker-dev — browse picker, editor, read, subscribe, deploy
**Classification:** high-risk (live; Razor + deploy-inertness bugs pass unit tests)
**Estimated implement time:** ~5 min (driving; excludes fixture spin-up)
**Parallelizable with:** none
**Files:**
- Modify: `docs/plans/2026-07-24-mtconnect-driver.md` (append a LIVE-GATE RESULT note)
> **DEPENDENCY:** this task's browse-picker leg is gated on the **Wave-0 universal-browser live gate (Gitea #468)** being closed — the picker's Browse button, `DiscoveryDriverBrowser.CanBrowse`, and `CapturedTreeBrowseSession` are Wave-0 machinery this driver only opts into via `SupportsOnlineDiscovery=true`. If #468 is still open, run the read/subscribe/deploy legs and record the browse leg as blocked-on-#468.
Bring the cppagent fixture up (`lmxopcua-fix up mtconnect`), author an MTConnect driver on docker-dev (`http://localhost:9200`, auto-authenticated admin — no login), and verify **in the running AdminUI + against `opc.tcp://localhost:4840`** (per the live-verify discipline — Razor binding bugs pass unit tests):
1. **Test Connect** on the MTConnect driver goes green (probe reaches the fixture).
2. **Browse picker** (`/uns` TagModal → Browse): the Device→Component→DataItem tree renders; a picked leaf commits `TagConfig.FullName = <dataItemId>` (Wave-0 dependency).
3. **Typed editor** shows `FullName`/read-only `mtCategory`/`DataType` override and round-trips.
4. **Deploy** the config; via Client.CLI read a picked node (`dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://localhost:4840 -n "ns=…;s=…"`) — value present, `UNAVAILABLE` items render `BadNoCommunication`.
5. **Subscribe** (`... subscribe ...`) — live updates flow from the `/sample` stream.
```bash
git add docs/plans/2026-07-24-mtconnect-driver.md
git commit -m "test(mtconnect): live /run gate on docker-dev — browse/read/subscribe/deploy (Task 21)"
```
---
## Task 22: Docs + deferred-writeback note; update the tracking doc
**Classification:** trivial
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `docs/drivers/MTConnect.md` (or a short section — mirror an existing driver doc)
- Modify: `docs/plans/2026-07-24-driver-expansion-tracking.md` (mark MTConnect P1 done + link this plan)
Document the P1 Agent MVP: config keys, browse-via-universal, the `UNAVAILABLE→BadNoCommunication` semantics, CONDITION-as-String, and the fixture recipe. Record **write-back (MTConnect Interfaces) as deferred** — point at `docs/plans/2026-07-15-mtconnect-driver-design.md` §3.6 + §9 (P1.5 native-alarm CONDITION, P2 SHDR ingest are also deferred there). Update the driver-expansion tracking doc's Wave-2 row.
```bash
git add -A && git commit -m "docs(mtconnect): P1 driver guide + tracking-doc update, defer write-back (Task 22)"
```
---
## Deferred (out of P1 — pointers, not scope)
- **Write-back (MTConnect Interfaces)** — design §3.6: read-only Agent surface; revisit only on a concrete deployment need.
- **CONDITION → native Part-9 alarms** (`IAlarmSource`), **`TIME_SERIES` SAMPLE arrays** materialized as OPC UA arrays, **EVENT controlled-vocab → OPC UA enumerations** — design §9 P1.5 fast-follow.
- **SHDR adapter ingest** (`SourceMode: "Agent" | "Shdr"`, loses auto-discovery) — design §9 P2.
@@ -0,0 +1,29 @@
{
"planPath": "docs/plans/2026-07-24-mtconnect-driver.md",
"tasks": [
{"id": 0, "subject": "Task 0: Verify TrakHound MTConnect.NET pin (or record the hand-roll decision)", "status": "pending"},
{"id": 1, "subject": "Task 1: Scaffold the two driver projects + the test project", "status": "pending", "blockedBy": [0]},
{"id": 2, "subject": "Task 2: MTConnectDriverOptions + MTConnectTagDefinition in .Contracts", "status": "pending", "blockedBy": [1]},
{"id": 3, "subject": "Task 3: MTConnectDataTypeInference.Infer + golden type-map test", "status": "pending", "blockedBy": [1]},
{"id": 4, "subject": "Task 4: Capture the canned XML fixtures", "status": "pending", "blockedBy": [1]},
{"id": 5, "subject": "Task 5: IMTConnectAgentClient seam + return DTOs", "status": "pending", "blockedBy": [1]},
{"id": 6, "subject": "Task 6: MTConnectAgentClient — parse /probe into the device model", "status": "pending", "blockedBy": [4, 5]},
{"id": 7, "subject": "Task 7: MTConnectAgentClient — parse /current + /sample, detect the sequence gap", "status": "pending", "blockedBy": [4, 5, 6]},
{"id": 8, "subject": "Task 8: MTConnectObservationIndex + UNAVAILABLE -> BadNoCommunication", "status": "pending", "blockedBy": [7]},
{"id": 9, "subject": "Task 9: MTConnectDriver shell — IDriver lifecycle", "status": "pending", "blockedBy": [2, 8]},
{"id": 10, "subject": "Task 10: IReadable.ReadAsync — /current, ordered snapshots", "status": "pending", "blockedBy": [9]},
{"id": 11, "subject": "Task 11: ISubscribable — /sample long-poll pump + ring-buffer re-baseline", "status": "pending", "blockedBy": [9, 7]},
{"id": 12, "subject": "Task 12: ITagDiscovery.DiscoverAsync + SupportsOnlineDiscovery=true", "status": "pending", "blockedBy": [9, 6]},
{"id": 13, "subject": "Task 13: IHostConnectivityProbe + IRediscoverable (instanceId watch)", "status": "pending", "blockedBy": [9]},
{"id": 14, "subject": "Task 14: MTConnectDriverProbe : IDriverProbe", "status": "pending", "blockedBy": [2, 5]},
{"id": 15, "subject": "Task 15: MTConnectDriverFactoryExtensions", "status": "pending", "blockedBy": [2, 9]},
{"id": 16, "subject": "Task 16: Host registration + DriverTypeNames.MTConnect + guard-test parity", "status": "pending", "blockedBy": [14, 15]},
{"id": 17, "subject": "Task 17: AdminUI typed model MTConnectTagConfigModel", "status": "pending", "blockedBy": [3, 16]},
{"id": 18, "subject": "Task 18: AdminUI editor razor + map/validator registration", "status": "pending", "blockedBy": [17]},
{"id": 19, "subject": "Task 19: mtconnect/cppagent docker fixture", "status": "pending", "blockedBy": [16]},
{"id": 20, "subject": "Task 20: Env-gated integration suite against cppagent", "status": "pending", "blockedBy": [19]},
{"id": 21, "subject": "Task 21: Live /run verify on docker-dev — browse picker, editor, read, subscribe, deploy", "status": "pending", "blockedBy": [18, 20]},
{"id": 22, "subject": "Task 22: Docs + deferred-writeback note; update the tracking doc", "status": "pending", "blockedBy": [21]}
],
"lastUpdated": "2026-07-24"
}
+670
View File
@@ -0,0 +1,670 @@
# SQL Poll Driver Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans (or subagent-driven-development) to implement this plan task-by-task.
**Goal:** Ship a read-only `Sql` Equipment-kind driver that polls SQL Server tables/views on an interval and publishes selected columns/rows as OPC UA variable nodes, authored through the standard equipment Tags flow with a bespoke schema browser.
**Architecture:** Three new projects mirroring the Modbus split — `Driver.Sql.Contracts` (options/tag DTOs + parser, zero transport deps), `Driver.Sql` (the `IDriver`/`ITagDiscovery`/`IReadable`/`ISubscribable`/`IHostConnectivityProbe` runtime + `ISqlDialect` seam + `SqlServerDialect`, owning `Microsoft.Data.SqlClient`), and `Driver.Sql.Browser` (schema-walk `IBrowseSession`). The `PollGroupEngine`, `EquipmentTagRefResolver<TDef>`, health state machine, and factory shape are all reused from Core; the driver batches tags by query-group (one round-trip per group per poll) and slices result sets back to per-tag snapshots. Every value binds as a `DbParameter`; identifiers are dialect-quoted from catalog-validated names only.
**Tech Stack:** `Microsoft.Data.SqlClient` (6.1.1, already in `Directory.Packages.props`) via `System.Data.Common` base types behind a `DbProviderFactory`; `ISqlDialect` + `SqlProvider` enum seam (only `SqlServer` implemented in v1); `Microsoft.Data.Sqlite` (10.0.7, test-only) for the offline unit fixture; xunit.v3 + Shouldly. **Zero new NuGet dependencies.**
**Source design:** docs/plans/2026-07-15-sql-poll-driver-design.md
**Program design (shared contract):** docs/plans/2026-07-15-driver-expansion-program-design.md §3, §7
**Conventions for every task below:** all projects `net10.0`, `Nullable`/`ImplicitUsings` enabled, `TreatWarningsAsErrors` on product projects. Every commit message ends with the repo trailer `Claude-Session: <session-url>` (omitted from the `git commit` examples for brevity). Run `dotnet build ZB.MOM.WW.OtOpcUa.slnx` green before each commit. New test projects use `xunit.v3` + `Shouldly` (see `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests/*.csproj`).
---
## Task 0: Scaffold `Driver.Sql.Contracts` project + register in slnx
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** none (root of the tree)
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Contracts/ZB.MOM.WW.OtOpcUa.Driver.Sql.Contracts.csproj`
- Modify: `ZB.MOM.WW.OtOpcUa.slnx` (add the project under `/src/Drivers/`)
**Steps:**
1. Create the csproj — refs `Core.Abstractions` ONLY (no transport deps, so AdminUI + browser can reference it without dragging `Microsoft.Data.SqlClient`). `InternalsVisibleTo` the Contracts is not needed. `RootNamespace = ZB.MOM.WW.OtOpcUa.Driver.Sql.Contracts`.
```xml
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<GenerateDocumentationFile>true</GenerateDocumentationFile>
<NoWarn>$(NoWarn);CS1591</NoWarn>
<RootNamespace>ZB.MOM.WW.OtOpcUa.Driver.Sql.Contracts</RootNamespace>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.Abstractions\ZB.MOM.WW.OtOpcUa.Core.Abstractions.csproj" />
</ItemGroup>
</Project>
```
2. Add to `ZB.MOM.WW.OtOpcUa.slnx` under the `/src/Drivers/` folder, beside the `Driver.Modbus.Contracts` line.
3. Run `dotnet build ZB.MOM.WW.OtOpcUa.slnx` — expect PASS (empty project compiles).
4. Commit: `git commit -m "feat(sql): scaffold Driver.Sql.Contracts project"`
---
## Task 1: `SqlProvider` + `SqlTagModel` enums + `SqlDriverConfigDto` (string-enum round-trip)
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none (blocks most downstream)
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Contracts/SqlProvider.cs`
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Contracts/SqlDriverConfigDto.cs`
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Tests/*` is NOT created here — this task's test lives temporarily in the Contracts consumer; create the Tests project in Task 6. **Instead**, put this task's test in a throwaway `Contracts`-adjacent xunit project? No — to avoid a project just for one test, fold this task's serialization assertion into Task 9's factory enum-serialization test. **This task ships DTOs only; its guard is Task 9.**
> **Note:** Tasks 12 produce pure records/enums with no I/O. Their behavioural guard (string-enum round-trip, strict parse) is the Task 9 factory test + Task 2's parser test (Task 2 creates its own micro-test via the Tests project created in Task 6). To keep TDD honest without a premature test project, **reorder locally**: implement Task 1 DTOs, then Task 6 (Tests project) can be pulled forward if the implementer prefers. The `blockedBy` graph permits this. The concrete failing-test-first discipline begins at Task 2.
**Steps:**
1. `SqlProvider.cs`:
```csharp
namespace ZB.MOM.WW.OtOpcUa.Driver.Sql.Contracts;
/// <summary>Which ADO.NET provider/dialect backs a Sql driver instance. v1 constructs only SqlServer.</summary>
public enum SqlProvider { SqlServer, Postgres, MySql, Odbc, Oracle }
/// <summary>Tag→value mapping model. v1 ships KeyValue + WideRow; Query is deferred (design §5.4 / P3).</summary>
public enum SqlTagModel { KeyValue, WideRow, Query }
```
2. `SqlDriverConfigDto.cs` — the driver-config blob shape (§5.1). Enum-typed fields (`Provider`) so the string converter round-trips names; timeouts as `TimeSpan?`; `ConnectionStringRef` (NOT the connection string). Fields: `Provider`, `ConnectionStringRef`, `DefaultPollInterval`, `OperationTimeout`, `CommandTimeout`, `MaxConcurrentGroups`, `NullIsBad`, `AllowWrites`, `Probe` (`SqlProbeDto{Enabled, Interval}`). `[JsonStringEnumConverter]` is applied at the factory (Task 9), not on the DTO.
3. Build — expect PASS.
4. Commit: `git commit -m "feat(sql): SqlProvider/SqlTagModel enums + SqlDriverConfigDto"`
---
## Task 2: `SqlTagDefinition` + `SqlEquipmentTagParser.TryParse` (strict enum reads, malicious-input safe)
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Contracts/SqlTagDefinition.cs`
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Contracts/SqlEquipmentTagParser.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Tests/SqlEquipmentTagParserTests.cs` (Tests project scaffolded in Task 6 — **pull the csproj creation forward from Task 6 for this task**, or defer this task's test until Task 6 and mark Task 2 blockedBy nothing but verified-at-6. Recommended: create the Tests project now as part of this task's setup.)
**TDD:**
1. **Failing test** — parse a KeyValue blob and a malicious `keyValue`; assert the value is captured verbatim (parser does NOT sanitize — binding is the reader's job) and a typo'd `model` enum rejects:
```csharp
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Sql.Contracts;
public class SqlEquipmentTagParserTests
{
[Fact]
public void TryParse_KeyValue_readsFields_andKeepsMaliciousValueVerbatim()
{
var json = """
{"driver":"Sql","model":"KeyValue","table":"dbo.TagValues","keyColumn":"tag_name",
"keyValue":"'; DROP TABLE x --","valueColumn":"num_value","timestampColumn":"sample_ts","type":"Float64"}
""";
SqlEquipmentTagParser.TryParse(json, "Plant/Sql/dev1/Speed", out var def).ShouldBeTrue();
def.Model.ShouldBe(SqlTagModel.KeyValue);
def.Table.ShouldBe("dbo.TagValues");
def.KeyValue.ShouldBe("'; DROP TABLE x --"); // captured as-is; the reader BINDS it, never concatenates
def.DeclaredType.ShouldBe(DriverDataType.Float64);
def.Name.ShouldBe("Plant/Sql/dev1/Speed"); // Name == RawPath (forward-router key)
}
[Fact]
public void TryParse_invalidModelEnum_rejectsTag()
=> SqlEquipmentTagParser.TryParse(
"""{"driver":"Sql","model":"Nonsense","table":"t","keyColumn":"k","keyValue":"v","valueColumn":"c"}""",
"raw", out _).ShouldBeFalse();
}
```
2. Run `dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Tests` — expect FAIL (types don't exist).
3. **Minimal impl**`SqlTagDefinition` record (`Name` = RawPath, `Model`, `Table`, `KeyColumn`, `KeyValue`, `ValueColumn`, `TimestampColumn`, `ColumnName`, `RowSelectorColumn`, `RowSelectorValue`, `RowSelectorTopByTimestamp`, `DeclaredType` = `DriverDataType?`). `SqlEquipmentTagParser.TryParse(string reference, string rawPath, out SqlTagDefinition def)` — mirror `ModbusTagDefinitionFactory.FromTagConfig`: recognize by leading `{`, read the `model` discriminator via `TagConfigJson.TryReadEnumStrict` (a present-but-invalid enum → `false` → the driver surfaces `BadNodeIdUnknown`, per R2-11), read model-specific fields, read the optional `type` override strictly, set `Name = rawPath`. Wrap in try/catch(JsonException/FormatException) → `false`. **No SQL is built here** — the parser only captures strings.
4. Run test — expect PASS.
5. Commit: `git commit -m "feat(sql): SqlTagDefinition + strict-enum equipment-tag parser"`
---
## Task 3: Scaffold `Driver.Sql` runtime project + register in slnx
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 2
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql/ZB.MOM.WW.OtOpcUa.Driver.Sql.csproj`
- Modify: `ZB.MOM.WW.OtOpcUa.slnx`
**Steps:**
1. Create the csproj per design §2.1 — refs `Core.Abstractions` + `Core` (for `DriverFactoryRegistry` in `ZB.MOM.WW.OtOpcUa.Core.Hosting`) + `Driver.Sql.Contracts`; packages `Microsoft.Data.SqlClient` + `Microsoft.Extensions.Logging.Abstractions`; `InternalsVisibleTo` the `.Tests`.
2. Add to `ZB.MOM.WW.OtOpcUa.slnx` under `/src/Drivers/`.
3. Build — expect PASS.
4. Commit: `git commit -m "feat(sql): scaffold Driver.Sql runtime project"`
---
## Task 4: `ISqlDialect` seam + `SqlServerDialect` (QuoteIdentifier, catalog SQL, MapColumnType) — golden queries
**Classification:** high-risk (identifier quoting is the injection boundary)
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 2
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql/ISqlDialect.cs`
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql/SqlServerDialect.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Tests/SqlServerDialectTests.cs`
**TDD:**
1. **Failing test** — golden catalog SQL, `QuoteIdentifier` escape + reject, `MapColumnType` table:
```csharp
[Fact]
public void QuoteIdentifier_bracketsAndEscapesEmbeddedBrackets()
{
var d = new SqlServerDialect();
d.QuoteIdentifier("Speed").ShouldBe("[Speed]");
d.QuoteIdentifier("a]b").ShouldBe("[a]]b]"); // ] doubled per T-SQL rules
}
[Fact]
public void QuoteIdentifier_rejectsControlCharsAndNul()
=> Should.Throw<ArgumentException>(() => new SqlServerDialect().QuoteIdentifier("x\0y"));
[Theory]
[InlineData("bit", DriverDataType.Boolean)]
[InlineData("int", DriverDataType.Int32)]
[InlineData("bigint", DriverDataType.Int64)]
[InlineData("real", DriverDataType.Float32)]
[InlineData("float", DriverDataType.Float64)]
[InlineData("decimal", DriverDataType.Float64)]
[InlineData("nvarchar", DriverDataType.String)]
[InlineData("datetime2", DriverDataType.DateTime)]
[InlineData("uniqueidentifier", DriverDataType.String)]
public void MapColumnType_mapsFamilies(string sql, DriverDataType expected)
=> new SqlServerDialect().MapColumnType(sql).ShouldBe(expected);
[Fact]
public void CatalogSql_isInformationSchemaAndParameterized()
{
var d = new SqlServerDialect();
d.LivenessSql.ShouldBe("SELECT 1");
d.ListTablesSql.ShouldContain("INFORMATION_SCHEMA.TABLES");
d.ListTablesSql.ShouldContain("@schema"); // never string-interpolated
d.ListColumnsSql.ShouldContain("@table");
}
```
2. Run — expect FAIL.
3. **Minimal impl**`ISqlDialect` per design §2.2 (`Provider`, `Factory` = `DbProviderFactory`, `QuoteIdentifier`, `LivenessSql`, `ListSchemasSql`, `ListTablesSql`, `ListColumnsSql`, `MapColumnType`). `SqlServerDialect`: `Factory => SqlClientFactory.Instance`; `QuoteIdentifier` doubles `]`, rejects embedded NUL/control chars via `ArgumentException`; catalog SQL uses `INFORMATION_SCHEMA` with `@schema`/`@table` markers; `MapColumnType` folds SQL type families to `DriverDataType` per design §3.7 (`decimal`/`numeric`/`money``Float64` with the documented precision caveat as an XML remark).
4. Run — expect PASS.
5. Commit: `git commit -m "feat(sql): ISqlDialect + SqlServerDialect with golden catalog SQL"`
---
## Task 5: `SqlQueryPlan` grouping + parameter binding (pure, no DB)
**Classification:** high-risk (the query-mapping core; GroupKey correctness governs correctness of every read)
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 10
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql/SqlQueryPlan.cs`
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql/SqlGroupPlanner.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Tests/SqlGroupPlannerTests.cs`
**TDD:**
1. **Failing test** — KeyValue tags on the same `(table,keyColumn,valueColumn,timestampColumn)` fold into ONE plan whose SQL is `... WHERE [tag_name] IN (@k0,@k1)` with two bound params; WideRow tags on the same `(table,rowSelector)` fold into one `SELECT <cols> ... WHERE [station_id]=@w`; different tables → different plans:
```csharp
[Fact]
public void KeyValueTags_sameSource_foldIntoOnePlan_withBoundInList()
{
var dialect = new SqlServerDialect();
var tags = new[] {
KvTag("A","dbo.TagValues","tag_name","Line1.Speed","num_value","sample_ts"),
KvTag("B","dbo.TagValues","tag_name","Line1.Temp","num_value","sample_ts"),
};
var plans = SqlGroupPlanner.Plan(tags, dialect).ToList();
plans.Count.ShouldBe(1);
plans[0].SqlText.ShouldContain("IN (@k0, @k1)");
plans[0].SqlText.ShouldContain("[tag_name]");
plans[0].Members.Count.ShouldBe(2);
plans[0].Parameters.ShouldBe(new object[] { "Line1.Speed", "Line1.Temp" }); // values are params, not text
}
```
2. Run — expect FAIL.
3. **Minimal impl**`SqlQueryPlan { object GroupKey, string SqlText, IReadOnlyList<object?> Parameters, IReadOnlyList<SqlTagDefinition> Members, Func<DbDataReader, SqlTagDefinition, ...> }` (indexer supplied by the reader in Task 7; here just the shape + SQL/param builder). `SqlGroupPlanner.Plan(tags, dialect)`: KeyValue → GroupKey `(table,keyColumn,valueColumn,timestampColumn)`, emit `SELECT <keyColumn>,<valueColumn>[,<timestampColumn>] FROM <table> WHERE <keyColumn> IN (@k0..@kN)` with distinct-key params; WideRow → GroupKey `(table, rowSelector)`, emit `SELECT <distinct member columns>[,<selector cols>] FROM <table> WHERE <whereColumn>=@w` (or `ORDER BY <col> DESC` + dialect TOP-1 for `topByTimestamp`). Identifiers via `dialect.QuoteIdentifier`; `<table>` split on `.` and each part quoted. **Every value is a param; zero interpolation.**
4. Run — expect PASS.
5. Commit: `git commit -m "feat(sql): SqlGroupPlanner folds tags into one parameterized query per group"`
---
## Task 6: Scaffold `Driver.Sql.Tests` + `SqliteDialect` (test) + `SqlitePollFixture`
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 10, Task 12
**Files:**
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Tests/ZB.MOM.WW.OtOpcUa.Driver.Sql.Tests.csproj`
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Tests/SqliteDialect.cs`
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Tests/SqlitePollFixture.cs`
- Modify: `ZB.MOM.WW.OtOpcUa.slnx`
> If Task 2 already created the csproj, this task adds the `SqliteDialect`/fixture and the slnx entry only.
**Steps:**
1. csproj — `xunit.v3` + `Shouldly` + `Microsoft.NET.Test.Sdk` + `xunit.runner.visualstudio`; `PackageReference Microsoft.Data.Sqlite` (10.0.7, test-only — no product dep on SQLite); project-refs `Driver.Sql` + `Driver.Sql.Contracts`. Add to slnx under `/tests/Drivers/`.
2. `SqliteDialect : ISqlDialect``Provider = SqlServer` shim is wrong; add a test-only `Provider` value is not needed — implement `ISqlDialect` directly with `Factory => SqliteFactory.Instance`, `QuoteIdentifier` doubling `"`, `LivenessSql = "SELECT 1"`, catalog SQL over `sqlite_schema` + `PRAGMA table_info(...)`, `MapColumnType` mapping declared affinity (design §9 caveat: SQLite is dynamically typed — keep seed columns explicitly typed). This dialect is also linked by the Browser.Tests (Task 13) via `<Compile Include="..\..\Driver.Sql.Tests\SqliteDialect.cs" Link="SqliteDialect.cs" />`.
3. `SqlitePollFixture` — opens an in-memory (or temp-file) SQLite connection, seeds a KeyValue table `TagValues(tag_name TEXT, num_value REAL, sample_ts TEXT)` and a wide-row `LatestStatus(station_id INT, oven_temp REAL, sample_ts TEXT)` with a few rows. Exposes the open `SqliteConnection` + a `DbProviderFactory` shim so the driver-under-test can be injected with an already-open connection provider (see Task 8's ctor seam).
4. Build + run (no tests yet) — expect PASS/empty.
5. Commit: `git commit -m "test(sql): Sql.Tests project + SqliteDialect + poll fixture"`
---
## Task 7: `SqlPollReader` core — one query per group, bounded deadline, slice back in input order
**Classification:** high-risk (connection/timeout/result-slicing core)
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql/SqlPollReader.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Tests/SqlPollReaderTests.cs`
**TDD:**
1. **Failing test** (against `SqlitePollFixture`) — read two KeyValue refs + one absent key; assert N-in/N-out order, correct values/types, absent key → `BadNoData`, NULL cell → `Uncertain` (default `nullIsBad=false`), source timestamp from `timestampColumn`:
```csharp
[Fact]
public async Task ReadAsync_returnsSnapshotsInInputOrder_absentKeyIsBadNoData()
{
await using var fx = await SqlitePollFixture.CreateAsync(); // seeds Line1.Speed=42.0, Line1.Temp=NULL
var reader = new SqlPollReader(fx.Factory, fx.ConnectionString, new SqliteDialect(),
commandTimeout: TimeSpan.FromSeconds(10), operationTimeout: TimeSpan.FromSeconds(15),
maxConcurrentGroups: 4, nullIsBad: false, resolve: fx.Resolve);
var refs = new[] { "Speed", "Missing", "Temp" };
var snaps = await reader.ReadAsync(refs, CancellationToken.None);
snaps.Count.ShouldBe(3);
snaps[0].Value.ShouldBe(42.0);
snaps[1].StatusCode.ShouldBe(SqlStatusCodes.BadNoData); // key deleted/absent
snaps[2].Value.ShouldBeNull();
StatusCode.IsUncertain(snaps[2].StatusCode).ShouldBeTrue(); // NULL cell, nullIsBad=false
}
```
2. Run — expect FAIL.
3. **Minimal impl**`SqlPollReader.ReadAsync(refs, ct)`: resolve each ref → `SqlTagDefinition` (miss → `BadNodeIdUnknown`); `SqlGroupPlanner.Plan(...)`; per group under a `SemaphoreSlim(maxConcurrentGroups)`: `await using var conn = factory.CreateConnection(); conn.ConnectionString = connStr; await conn.OpenAsync(linkedCt)`; build `DbCommand` with `CommandTimeout = (int)commandTimeout.TotalSeconds` and bound params; `using var linked = CancellationTokenSource.CreateLinkedTokenSource(ct); linked.CancelAfter(operationTimeout); await using var rdr = await cmd.ExecuteReaderAsync(linked.Token)`; index rows by key/selector; map each member's cell → `DataValueSnapshot` (type per `DeclaredType` ?? `dialect.MapColumnType`-inferred; NULL → `nullIsBad ? Bad : Uncertain`; absent key → `BadNoData`; source timestamp from `timestampColumn` else read wall-clock). **Reassemble in input order** (`DataValueSnapshot[refs.Count]`). Per-tag failure = Bad-coded snapshot, NOT an exception; the whole call throws only if the connection itself is unreachable (→ engine backoff). `await using` both connection and reader — never leak. Add `SqlStatusCodes` static (BadNoData/BadTimeout/BadNodeIdUnknown/BadCommunicationError/Uncertain constants) if not reusing an existing helper.
4. Run — expect PASS. Add a NULL-cell test with `nullIsBad:true``Bad`.
5. Commit: `git commit -m "feat(sql): SqlPollReader — grouped read, bounded deadline, ordered slice-back"`
---
## Task 8: `SqlDriver` shell — IDriver / ITagDiscovery / IReadable / ISubscribable / IHostConnectivityProbe
**Classification:** high-risk (lifecycle + poll-engine wiring + connection validation)
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql/SqlDriver.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Tests/SqlDriverTests.cs`
**TDD:**
1. **Failing test** — construct `SqlDriver` with an injected dialect + factory + already-resolved connection string (SQLite fixture) + authored `RawTagEntry` list; `InitializeAsync``Healthy`; `DiscoverAsync` streams one Variable per authored tag with `SecurityClass=ViewOnly` + `RediscoverPolicy=Once` + `SupportsOnlineDiscovery=false`; `ReadAsync` delegates to the reader; a subscribe fires an initial change:
```csharp
[Fact]
public async Task Initialize_thenDiscover_streamsAuthoredTagsReadOnly()
{
await using var fx = await SqlitePollFixture.CreateAsync();
var driver = SqlDriver.ForTest(fx, rawTags: fx.AuthoredRawTags);
await driver.InitializeAsync(fx.ConfigJson, CancellationToken.None);
driver.GetHealth().State.ShouldBe(DriverState.Healthy);
((ITagDiscovery)driver).SupportsOnlineDiscovery.ShouldBeFalse();
((ITagDiscovery)driver).RediscoverPolicy.ShouldBe(DiscoveryRediscoverPolicy.Once);
var cap = new CapturingBuilder();
await ((ITagDiscovery)driver).DiscoverAsync(cap, CancellationToken.None);
cap.Variables.ShouldContain(v => v.FullName == "Speed" && v.SecurityClass == SecurityClassification.ViewOnly);
}
```
2. Run — expect FAIL.
3. **Minimal impl**`SqlDriver : IDriver, ITagDiscovery, IReadable, ISubscribable, IHostConnectivityProbe, IAsyncDisposable`. Mirror `ModbusDriver`: fields grouped up top; `_resolver = new EquipmentTagRefResolver<SqlTagDefinition>(...)`; `_poll = new PollGroupEngine(reader: ReadAsync, onChange: ..., onError: HandlePollError, backoffCap: 30s)`. `InitializeAsync`: build `_tagsByRawPath` from `_options.RawTags` via `SqlEquipmentTagParser.TryParse` (miss = logged skip, never throw); open ONE connection, run `dialect.LivenessSql` under `CommandTimeout` + linked CTS, dispose; success → `Healthy`, failure → `Faulted` + `LastError`. `ReadAsync``_reader.ReadAsync`. `SubscribeAsync``_poll.Subscribe`; `UnsubscribeAsync``_poll.Unsubscribe`. `IHostConnectivityProbe`: single logical host (server host from the connection string) with `Running/Stopped` from the last poll/liveness outcome + `OnHostStatusChanged` on transitions. `GetMemoryFootprint => 0`; `FlushOptionalCachesAsync => Task.CompletedTask`; `ReinitializeAsync` = teardown+init. `DriverType => DriverTypeNames.Sql`. Add a `ForTest(...)` internal factory that injects dialect+factory+connString (production path resolves those in the factory, Task 9). **Pooling: open-use-dispose per poll (already in the reader); no long-lived connection.**
4. Run — expect PASS.
5. Commit: `git commit -m "feat(sql): SqlDriver shell wiring PollGroupEngine + probe + read-only discovery"`
---
## Task 9: `SqlDriverFactoryExtensions` + `connectionStringRef` env resolution + enum-serialization guard
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 10
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql/SqlDriverFactoryExtensions.cs`
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql/SqlConnectionStringResolver.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Tests/SqlDriverFactoryTests.cs`
**TDD:**
1. **Failing test**`CreateInstance` with a config carrying `"provider":"SqlServer"` builds a driver; a missing `connectionStringRef` throws a clean error; the connection-string ref resolves from env `Sql__ConnectionStrings__<ref>`; **enum-serialization guard** — a config author serializing `provider` as a NUMBER still parses (`JsonStringEnumConverter` accepts ordinals) AND the round-trip writes the NAME:
```csharp
[Fact]
public void CreateInstance_missingConnectionStringRef_throwsActionable()
=> Should.Throw<InvalidOperationException>(() =>
SqlDriverFactoryExtensions.CreateInstance("d1", """{"provider":"SqlServer"}""", null))
.Message.ShouldContain("connectionStringRef");
[Fact]
public void ConnectionStringRef_resolvesFromEnv()
{
Environment.SetEnvironmentVariable("Sql__ConnectionStrings__MesStaging", "Server=x;Database=y;");
SqlConnectionStringResolver.Resolve("MesStaging").ShouldBe("Server=x;Database=y;");
}
[Fact]
public void Provider_serializesAsNameNotNumber()
{
var json = JsonSerializer.Serialize(new SqlDriverConfigDto { Provider = SqlProvider.SqlServer },
SqlDriverFactoryExtensions.JsonOptionsForTest);
json.ShouldContain("\"SqlServer\"");
json.ShouldNotContain("\"provider\":0");
}
```
2. Run — expect FAIL.
3. **Minimal impl**`SqlConnectionStringResolver.Resolve(string @ref)``Environment.GetEnvironmentVariable($"Sql__ConnectionStrings__{@ref}")` (direct env read, deliberate — the factory closure has no `IConfiguration`, per design §8.2), throws actionable if absent. `SqlDriverFactoryExtensions`: `const DriverTypeName = DriverTypeNames.Sql`; `Register(registry, loggerFactory)``registry.Register(DriverTypeName, (id,json)=>CreateInstance(id,json,loggerFactory))`; `CreateInstance` deserializes `SqlDriverConfigDto` with `JsonOptions` carrying `JsonStringEnumConverter` + `UnmappedMemberHandling.Skip`, validates `ConnectionStringRef` present, builds `SqlServerDialect` from `Provider` (P1: only `SqlServer` constructible; others throw "provider not available in this build"), resolves the connection string, constructs the `SqlPollReader` + `SqlDriver`. **Never log the resolved connection string** (log provider + server host + database only). Expose `JsonOptionsForTest` internal.
4. Run — expect PASS.
5. Commit: `git commit -m "feat(sql): factory + connectionStringRef env resolution + string-enum guard"`
---
## Task 10: `SqlDriverProbe` (SELECT 1 liveness, bounded timeout)
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 5, Task 9, Task 12
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql/SqlDriverProbe.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Tests/SqlDriverProbeTests.cs`
**TDD:**
1. **Failing test** — against the SQLite fixture (inject dialect+factory), `ProbeAsync` returns green with latency; a bad connection string returns a red result with the error message (never throws):
```csharp
[Fact]
public async Task Probe_liveConnection_returnsGreen()
{
await using var fx = await SqlitePollFixture.CreateAsync();
var probe = SqlDriverProbe.ForTest(fx.Factory, new SqliteDialect());
var r = await probe.ProbeAsync(fx.ConfigJson, TimeSpan.FromSeconds(5), CancellationToken.None);
r.Success.ShouldBeTrue();
}
```
2. Run — expect FAIL.
3. **Minimal impl**`SqlDriverProbe : IDriverProbe`, `DriverType => DriverTypeNames.Sql`. Parse config via the SAME factory DTO shape + `JsonStringEnumConverter` (R2-11 factory parity, mirroring `ModbusDriverProbe`), resolve the connection string ref, open a connection under a linked CTS bounded by `timeout`, run `dialect.LivenessSql` (`SELECT 1`), return `DriverProbeResult(true, "SQL SELECT 1 OK", latency)`; catch → red result naming the failure. Never throws. `ForTest` injects factory+dialect for the SQLite path.
4. Run — expect PASS.
5. Commit: `git commit -m "feat(sql): SqlDriverProbe SELECT-1 liveness check"`
---
## Task 11: Host registration — `DriverTypeNames.Sql` + factory + probe + guard test
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** none
**Files:**
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/DriverTypeNames.cs`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj` (project-ref `Driver.Sql` if not transitively present)
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/DriverTypeNamesGuardTests.cs` is EXISTING — it asserts bidirectional parity between `DriverTypeNames` constants and the factory-registered set; this task makes it pass by adding both sides.
**TDD:**
1. **Failing step** — add `public const string Sql = "Sql";` to `DriverTypeNames` + include in `All`, but DON'T yet register the factory. Run `DriverTypeNamesGuardTests` → expect FAIL (constant with no registered factory).
2. **Impl** — in `DriverFactoryBootstrap.Register(...)` add `Driver.Sql.SqlDriverFactoryExtensions.Register(registry, loggerFactory);`; in `AddOtOpcUaDriverProbes` add `services.TryAddEnumerable(ServiceDescriptor.Singleton<IDriverProbe, SqlProbe>());` with a `using SqlProbe = Driver.Sql.SqlDriverProbe;` alias; add the `Driver.Sql` project-ref to the Host csproj. Default tier `DriverTier.A` (no explicit arg — SQL client is cross-platform managed, runs on macOS dev, so `ShouldStub()` needs no change).
3. Run guard test + `dotnet build ZB.MOM.WW.OtOpcUa.slnx` — expect PASS.
4. Commit: `git commit -m "feat(sql): register Sql factory + probe in Host, add DriverTypeNames.Sql"`
---
## Task 12: Scaffold `Driver.Sql.Browser` project + register in slnx
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 5, Task 6, Task 9, Task 10
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Browser/ZB.MOM.WW.OtOpcUa.Driver.Sql.Browser.csproj`
- Modify: `ZB.MOM.WW.OtOpcUa.slnx`
**Steps:**
1. csproj refs `Commons` (`ZB.MOM.WW.OtOpcUa.Commons``IDriverBrowser`/`IBrowseSession`/`BrowseNode`) + `Driver.Sql.Contracts` + **`Driver.Sql`** (for `ISqlDialect`/`SqlServerDialect` — the dialect's catalog SQL *is* the browse engine; `Microsoft.Data.SqlClient` comes transitively). This runtime-project ref is the deliberate deviation from `OpcUaClient.Browser` recorded in design §2 — add an XML comment on the ref citing the design so nobody "fixes" it. `InternalsVisibleTo` the `.Browser.Tests`.
2. Add to slnx under `/src/Drivers/`.
3. Build — expect PASS.
4. Commit: `git commit -m "feat(sql): scaffold Driver.Sql.Browser project"`
---
## Task 13: `SqlBrowseSession` — schema walk over the dialect catalog
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 16, Task 17
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Browser/SqlBrowseSession.cs`
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Browser.Tests/ZB.MOM.WW.OtOpcUa.Driver.Sql.Browser.Tests.csproj` (+ slnx entry, + link `SqliteDialect.cs` from Sql.Tests)
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Browser.Tests/SqlBrowseSessionTests.cs`
**TDD:**
1. **Failing test** — over a seeded SQLite DB + `SqliteDialect`: `RootAsync` yields schema folder(s); `ExpandAsync(schema)` yields the seeded tables; `ExpandAsync(table)` yields columns as leaves; `AttributesAsync(column)` yields the mapped `DriverDataType`:
```csharp
[Fact]
public async Task ExpandTable_yieldsColumnsAsLeaves_withMappedType()
{
await using var db = await SqliteBrowseFixture.CreateAsync(); // TagValues(tag_name TEXT, num_value REAL)
var session = new SqlBrowseSession(db.Connection, new SqliteDialect());
var tableNode = (await session.ExpandAsync(db.SchemaNodeId, default)).First(n => n.DisplayName.Contains("TagValues"));
var cols = await session.ExpandAsync(tableNode.NodeId, default);
cols.ShouldContain(c => c.DisplayName == "num_value" && c.IsLeaf);
var attrs = await session.AttributesAsync(cols.First(c => c.DisplayName == "num_value").NodeId, default);
attrs.DriverDataType.ShouldBe(nameof(DriverDataType.Float64));
}
```
2. Run — expect FAIL.
3. **Minimal impl**`SqlBrowseSession : IBrowseSession` (match the `Commons.Browsing` interface shape used by `OpcUaClientBrowseSession`): `RootAsync`/`ExpandAsync(nodeId)`/`AttributesAsync(nodeId)` run the dialect catalog queries with bound `@schema`/`@table` params; encode `NodeId` as `schema` / `schema.table` / `schema.table|column`; a `SemaphoreSlim _gate` serializes (one ADO.NET connection is not concurrent); per-call work bounded by the AdminUI's existing 20 s per-call timeout. Column leaf `AttributesAsync` returns `AttributeInfo(Name: column, DriverDataType: dialect.MapColumnType(dataType).ToString(), IsArray:false, SecurityClass:"ViewOnly")` (mirrors Galaxy two-stage pick).
4. Run — expect PASS.
5. Commit: `git commit -m "feat(sql): SqlBrowseSession schema-walk over dialect catalog"`
---
## Task 14: `SqlDriverBrowser` — IDriverBrowser open (env ref OR pasted literal, transient connection)
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 16, Task 17
**Files:**
- Create: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Browser/SqlDriverBrowser.cs`
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Browser.Tests/SqlDriverBrowserTests.cs`
**TDD:**
1. **Failing test**`OpenAsync` with a form JSON whose `connectionStringRef` resolves from env opens a session; an UNresolvable ref fails with a message naming the EXACT missing env var; a pasted literal connection string is used session-only (never persisted):
```csharp
[Fact]
public async Task Open_unresolvableRef_failsNamingExactEnvVar()
{
var browser = new SqlDriverBrowser();
var ex = await Should.ThrowAsync<InvalidOperationException>(() =>
browser.OpenAsync("""{"provider":"SqlServer","connectionStringRef":"MesStaging"}""", default));
ex.Message.ShouldContain("Sql__ConnectionStrings__MesStaging");
}
```
2. Run — expect FAIL.
3. **Minimal impl**`SqlDriverBrowser : IDriverBrowser`, `DriverType => DriverTypeNames.Sql`. `OpenAsync(configJson, ct)`: deserialize with the same `JsonStringEnumConverter` options; resolve `connectionStringRef` via a direct env read (`Sql__ConnectionStrings__<ref>`) IN THE ADMINUI PROCESS — on miss throw an actionable message naming the exact missing env var (design §4.4); accept an optional pasted literal `connectionString` for ad-hoc browse (session-only, **never persisted, never logged, no `_lastConfigJson` cached field**); select `SqlServerDialect` (P1); open ONE transient `DbConnection`, return a `SqlBrowseSession`. Reuses the AdminUI `BrowseSessionRegistry` + reaper + idle TTL unchanged.
4. Run — expect PASS.
5. Commit: `git commit -m "feat(sql): SqlDriverBrowser transient-connection open with env-ref + literal"`
---
## Task 15: AdminUI browser DI + `SqlAddressPickerBody.razor` (picker body)
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 16, Task 17
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/EndpointRouteBuilderExtensions.cs` (add `services.AddSingleton<IDriverBrowser, SqlDriverBrowser>();` beside the OpcUaClient/Galaxy lines ~:75)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/ZB.MOM.WW.OtOpcUa.AdminUI.csproj` (project-ref `Driver.Sql.Browser`)
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Shared/Uns/AddressPickers/SqlAddressPickerBody.razor` (reuse `DriverBrowseTree.razor` for the tree + an attribute side-panel + model/selector fields; gated by the existing `DriverOperator` policy; manual entry retained)
**Steps (no unit test — Razor is live-verified in Task 21):**
1. Register the bespoke `IDriverBrowser` (the universal-browser DI line is NOT added for `Sql` — its `CanBrowse` is false and bespoke wins by construction).
2. `SqlAddressPickerBody.razor` — thin body: `DriverBrowseTree` in picker mode + attribute side-panel; on column pick, compose the `TagConfig` blob (DisplayName default `table.column`, prefill `type` from `AttributesAsync`, operator picks model + fills key/row selector; `SecurityClass=ViewOnly`).
3. `dotnet build` — expect PASS.
4. Commit: `git commit -m "feat(sql): AdminUI browser DI + Sql address picker body"`
---
## Task 16: Env-gated central-SQL integration fixture (`Driver.Sql.IntegrationTests`)
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 13, Task 14, Task 15, Task 17
**Files:**
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.IntegrationTests/ZB.MOM.WW.OtOpcUa.Driver.Sql.IntegrationTests.csproj` (+ slnx entry)
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.IntegrationTests/SqlPollServerFixture.cs`
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.IntegrationTests/SqlServerReadTests.cs`
**TDD:**
1. **Failing/skipping test** — against the always-on central SQL Server (`10.100.0.35,14330`), seed a `SqlPollFixture` database with the two sample tables (KeyValue + wide-row), then validate the real `Microsoft.Data.SqlClient` path (read round-trip, `INFORMATION_SCHEMA` browse, `CommandTimeout`/cancellation, pooling). **Env-gated** via `SQL_TEST_ENDPOINT` / a full `Sql__ConnectionStrings__Fixture` — absent ⇒ skip cleanly (use `Assert.Skip(...)` / xunit.v3 `[Fact(Skip=...)]` dynamic skip), like every other `*.IntegrationTests`:
```csharp
[Fact]
public async Task ReadAsync_realSqlServer_roundTripsSeededKeyValue()
{
var connStr = Environment.GetEnvironmentVariable("Sql__ConnectionStrings__Fixture");
Assert.SkipWhen(string.IsNullOrEmpty(connStr), "Sql__ConnectionStrings__Fixture not set — offline skip.");
await using var fx = await SqlPollServerFixture.EnsureSeededAsync(connStr!);
var driver = SqlDriver.ForProduction(connStr!, new SqlServerDialect(), fx.AuthoredRawTags);
await driver.InitializeAsync(fx.ConfigJson, default);
var snaps = await driver.ReadAsync(new[] { "Line1.Speed" }, default);
snaps[0].StatusCode.ShouldBe(0u);
}
```
2. Run offline — expect SKIP (green).
3. **Impl** — the fixture (create-if-missing DB + tables + seed rows via `Microsoft.Data.SqlClient`) and the read tests. **The seed DB is `SqlPollFixture`, NOT `ConfigDb`** — the shared central server hosts ConfigDb; the fixture uses its own database on that server.
4. Commit: `git commit -m "test(sql): env-gated central-SQL integration fixture + read round-trip"`
---
## Task 17: Injection regression test (bind-harmless value / reject identifier, never execute)
**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 13, Task 14, Task 15, Task 16
**Files:**
- Test: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.Tests/SqlInjectionRegressionTests.cs`
**TDD:**
1. **Failing test** (offline, SQLite fixture) — a malicious `keyValue` (`'; DROP TABLE TagValues; --`) reads harmlessly as a bound param and the seed table still exists afterward; a malicious `table`/`column` identifier that is not catalog-valid is REJECTED (tag → `BadNodeIdUnknown`), never executed:
```csharp
[Fact]
public async Task MaliciousKeyValue_bindsHarmlessly_tableSurvives()
{
await using var fx = await SqlitePollFixture.CreateAsync();
var reader = fx.NewReader();
var snaps = await reader.ReadAsync(new[] { fx.RefWithKeyValue("'; DROP TABLE TagValues; --") }, default);
snaps[0].StatusCode.ShouldBe(SqlStatusCodes.BadNoData); // no such key — bound, not executed
(await fx.TableRowCountAsync("TagValues")).ShouldBeGreaterThan(0); // table intact
}
```
2. Run — expect FAIL (until reader binds correctly; it should already PASS from Task 7 if binding is right — this test LOCKS the guarantee).
3. **Impl** — none if Task 7 binds correctly; otherwise fix. Add a review-checklist note: "any code path building SQL by string-concatenating a tag field is a defect."
4. Commit: `git commit -m "test(sql): injection regression — bind values, reject unknown identifiers"`
---
## Task 18: Blackhole / timeout live-gate against a DEDICATED mssql container
**Classification:** high-risk (frozen-peer resilience — the single highest-value integration test)
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.IntegrationTests/Docker/docker-compose.yml` (a dedicated disposable `mssql` stack)
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.IntegrationTests/Docker/seed.sql`
- Create: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.IntegrationTests/SqlBlackholeTimeoutTests.cs`
> **CRITICAL:** this gate `docker pause`s a **dedicated** `mcr.microsoft.com/mssql/server` container it owns — **NEVER** the shared central SQL Server on `10.100.0.35,14330`, which hosts `ConfigDb` for the whole rig (design §9). The compose stack is deployed via `lmxopcua-fix sync`; `lmxopcua-fix` applies the `project=lmxopcua` label host-side.
**TDD:**
1. **Failing/skipping test** — mirror the R2-01 S7 blackhole (`S7_1500ConnectTimeoutOutageTests`): start reading against the dedicated mssql, then `docker pause` it mid-poll; assert the next read surfaces `BadTimeout` within `operationTimeout` + a small margin (client-side linked-CTS cancellation, not the poll thread wedging), the driver degrades, and the poll loop backs off; `docker unpause` recovers. Env-gated (`SQL_BLACKHOLE_ENDPOINT`) — offline skip:
```csharp
[Fact]
public async Task PausedServer_surfacesBadTimeout_withinDeadline_notWedged()
{
var ep = Environment.GetEnvironmentVariable("SQL_BLACKHOLE_ENDPOINT");
Assert.SkipWhen(string.IsNullOrEmpty(ep), "SQL_BLACKHOLE_ENDPOINT not set — offline skip.");
// ... start driver, docker pause <dedicated-container>, read, assert BadTimeout within ~operationTimeout, unpause ...
}
```
2. Run offline — expect SKIP.
3. **Impl** — dedicated `docker-compose.yml` (`mssql` service, own container name, `restart: "no"`), `seed.sql` (the two sample tables), and the test that shells `docker pause`/`docker unpause` against the OWNED container. Assert the wall-clock bound (the frozen-peer contract): `CommandTimeout` (server-side backstop) + `ExecuteReaderAsync(linkedCt)` bounded by `operationTimeout` (client-side real cancellation) both fire.
4. Commit: `git commit -m "test(sql): blackhole/timeout live-gate on a dedicated mssql container"`
---
## Task 19: AdminUI typed editor model + validator (timeout-order rule, string enums)
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 16, Task 17, Task 18
**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/TagEditors/SqlTagConfigModel.cs`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/TagEditors/TagConfigEditorMap.cs` (add `[DriverTypeNames.Sql] = typeof(...SqlTagConfigEditor)`)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/TagEditors/TagConfigValidator.cs` (add `[DriverTypeNames.Sql] = j => SqlTagConfigModel.FromJson(j).Validate()`)
- Test: `tests/.../AdminUI.Tests/.../SqlTagConfigModelTests.cs` (locate the existing AdminUI tag-editor test project; mirror `ModbusTagConfigModel` tests)
**TDD:**
1. **Failing test**`FromJson``ToJson` round-trips the model, preserves unknown keys, writes enums (`model`/`type`) as NAME strings not numbers; `Validate()` rejects a WideRow without a row selector and enforces the driver-level timeout-order rule surrogate at the tag layer if the tag carries timeouts (the primary timeout-order check is at the driver page — the model validates model/selector shape):
```csharp
[Fact]
public void ToJson_writesModelAndTypeAsStrings_preservesUnknownKeys()
{
var m = SqlTagConfigModel.FromJson("""{"driver":"Sql","model":"KeyValue","table":"t","keyColumn":"k","keyValue":"v","valueColumn":"c","type":"Float64","customX":1}""");
var json = m.ToJson();
json.ShouldContain("\"KeyValue\"");
json.ShouldContain("\"Float64\"");
json.ShouldContain("customX"); // unknown key survives load→save
json.ShouldNotContain("\"model\":0");
}
```
2. Run — expect FAIL.
3. **Minimal impl**`SqlTagConfigModel` (thin, over `TagConfigJson.ParseOrNew`, preserving the `_bag`): `Model`, `Table`, `KeyColumn`/`KeyValue`/`ValueColumn`/`TimestampColumn`, `ColumnName` + `RowSelector`, `Type` (`DriverDataType?`), `FromJson`/`ToJson` (enums as name strings via `TagConfigJson.Set`) `/Validate` (model discriminator ⇒ required-field shape; WideRow needs a selector). Register in `TagConfigEditorMap` + `TagConfigValidator`. **Enum-serialization trap (systemic):** `model`/`type` MUST round-trip as strings on both sides — the editor `ToJson` and the factory `SqlEquipmentTagParser` both use string enums; add the assertion above.
4. Run — expect PASS.
5. Commit: `git commit -m "feat(sql): typed AdminUI Sql tag-config model + validator (string enums)"`
---
## Task 20: `SqlTagConfigEditor.razor` (thin shell over the model)
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Shared/Uns/TagEditors/SqlTagConfigEditor.razor`
**Steps (Razor — live-verified in Task 21):**
1. Copy the Modbus editor template; bind over `SqlTagConfigModel`: a `Model` dropdown (KeyValue/WideRow) that shows the matching field group, `Table`, the key/value/timestamp fields (KeyValue) OR `ColumnName` + row-selector fields (WideRow), and a `Type` override dropdown. `@bind` string component params need the `@` prefix (the Global-UNS gotcha).
2. `dotnet build` — expect PASS.
3. Commit: `git commit -m "feat(sql): SqlTagConfigEditor razor shell"`
---
## Task 21: Live `/run` verification on docker-dev (picker + editor + deploy + read)
**Classification:** trivial (procedure — no product code; may surface fix-forward tasks)
**Estimated implement time:** ~5 min (verification, excluding any bugs found)
**Parallelizable with:** none (last)
**Files:** none (verification only; any defect becomes a new fix-forward task)
**Procedure (docker-dev rig, AdminUI auto-authenticated at `http://localhost:9200` — login disabled, so drive it yourself, don't defer to the user):**
1. Set `Sql__ConnectionStrings__DevSql` on the docker-dev central + driver nodes pointing at a dev SQL Server + seeded table (the dedicated mssql from Task 18, or the central-SQL fixture DB). Rebuild BOTH central-1/central-2 (`:9200` round-robins).
2. In `/raw`, add a `Sql` driver + device (paste a literal connection string or use the ref); Test-Connect → green (`SELECT 1`).
3. Open the Sql address picker → browse schema→table→column → commit a KeyValue tag and a WideRow tag; confirm the typed editor renders and validates.
4. Reference the raw tags into an equipment on `/uns`; deploy via `POST :9200/api/deployments` (X-Api-Key) or the deploy button; confirm the deployment seals green.
5. Read via Client.CLI (`read -u opc.tcp://localhost:4840 -n "ns=2;s=<RawPath>"`) — confirm the live SQL value + quality + source timestamp.
6. Record the run outcome in the plan's completion note. If anything is deploy-inert / mis-bound, open a fix-forward task (Razor binding + deploy-inertness bugs pass unit tests + review — this live gate is the real proof).
---
## Deferred / out of scope (pointers to the design, NOT executable tasks here)
These are recorded so nobody re-derives them; each is a later phase in `docs/plans/2026-07-15-sql-poll-driver-design.md`. The `ISqlDialect` seam is built in from day one (Task 4) precisely so the provider tail below is additive and cheap.
- **Named-query model** (design §5.4, §3.6(c) / design-P3) — the arbitrary-`SELECT` escape hatch (`queries: {...}` + a tag referencing a query by name + projection). Not built in v1; the `SqlTagModel.Query` enum member exists but the reader/planner path is deferred.
- **PostgreSQL + ODBC dialects** (design §2.2, §8.5 / design-P3) — `PostgresDialect` (`Npgsql`) + `OdbcDialect` (`System.Data.Odbc`), each a NEW gated `PackageVersion`. v1 constructs only `SqlServerDialect`; the other `SqlProvider` members throw "provider not available in this build."
- **MySQL/MariaDB + native Oracle** (design §10 / design-P5) — `MySqlConnector` + an `ALL_TAB_COLUMNS` Oracle dialect. Demand-driven.
- **Write mode (`IWritable`)** (design §3.4 / design-P4) — opt-in, triple-gated (`WriteOperate` role + driver `allowWrites` master switch + `TagConfig.writable`), parameterized UPSERT/StoredProc, `WriteIdempotent` retry policy. v1 is read-only; `SecurityClass=ViewOnly` everywhere and `allowWrites` defaults `false`.
- **Split-role env-parity operational note** (design §4.4) — admin nodes must carry the same `Sql__ConnectionStrings__<ref>` env vars as driver nodes for any browseable ref; the browser-open failure names the exact missing env var. This is a deployment/runbook item, not a code task.
@@ -0,0 +1,28 @@
{
"planPath": "docs/plans/2026-07-24-sql-poll-driver.md",
"tasks": [
{"id": 0, "subject": "Task 0: Scaffold Driver.Sql.Contracts project + register in slnx", "status": "pending"},
{"id": 1, "subject": "Task 1: SqlProvider/SqlTagModel enums + SqlDriverConfigDto", "status": "pending", "blockedBy": [0]},
{"id": 2, "subject": "Task 2: SqlTagDefinition + SqlEquipmentTagParser (strict enums, malicious-input safe)", "status": "pending", "blockedBy": [1]},
{"id": 3, "subject": "Task 3: Scaffold Driver.Sql runtime project + register in slnx", "status": "pending", "blockedBy": [0]},
{"id": 4, "subject": "Task 4: ISqlDialect + SqlServerDialect (QuoteIdentifier, catalog SQL, MapColumnType)", "status": "pending", "blockedBy": [1, 3]},
{"id": 5, "subject": "Task 5: SqlQueryPlan grouping + parameter binding (pure)", "status": "pending", "blockedBy": [2, 4]},
{"id": 6, "subject": "Task 6: Scaffold Driver.Sql.Tests + SqliteDialect + SqlitePollFixture", "status": "pending", "blockedBy": [3, 4]},
{"id": 7, "subject": "Task 7: SqlPollReader core — grouped read, bounded deadline, ordered slice-back", "status": "pending", "blockedBy": [5, 6]},
{"id": 8, "subject": "Task 8: SqlDriver shell — IDriver/ITagDiscovery/IReadable/ISubscribable/IHostConnectivityProbe", "status": "pending", "blockedBy": [7]},
{"id": 9, "subject": "Task 9: SqlDriverFactoryExtensions + connectionStringRef env resolution + enum guard", "status": "pending", "blockedBy": [8]},
{"id": 10, "subject": "Task 10: SqlDriverProbe (SELECT 1 liveness)", "status": "pending", "blockedBy": [4, 6]},
{"id": 11, "subject": "Task 11: Host registration — DriverTypeNames.Sql + factory + probe + guard test", "status": "pending", "blockedBy": [9, 10]},
{"id": 12, "subject": "Task 12: Scaffold Driver.Sql.Browser project + register in slnx", "status": "pending", "blockedBy": [3]},
{"id": 13, "subject": "Task 13: SqlBrowseSession — schema walk over dialect catalog", "status": "pending", "blockedBy": [4, 12, 6]},
{"id": 14, "subject": "Task 14: SqlDriverBrowser — env-ref/literal transient-connection open", "status": "pending", "blockedBy": [13]},
{"id": 15, "subject": "Task 15: AdminUI browser DI + SqlAddressPickerBody.razor", "status": "pending", "blockedBy": [14]},
{"id": 16, "subject": "Task 16: Env-gated central-SQL integration fixture + read round-trip", "status": "pending", "blockedBy": [8]},
{"id": 17, "subject": "Task 17: Injection regression test (bind value / reject identifier)", "status": "pending", "blockedBy": [7]},
{"id": 18, "subject": "Task 18: Blackhole/timeout live-gate on a dedicated mssql container", "status": "pending", "blockedBy": [16]},
{"id": 19, "subject": "Task 19: AdminUI typed Sql tag-config model + validator (string enums)", "status": "pending", "blockedBy": [1, 11]},
{"id": 20, "subject": "Task 20: SqlTagConfigEditor.razor shell", "status": "pending", "blockedBy": [19]},
{"id": 21, "subject": "Task 21: Live /run verification on docker-dev (picker + editor + deploy + read)", "status": "pending", "blockedBy": [11, 15, 20]}
],
"lastUpdated": "2026-07-24"
}
@@ -79,4 +79,46 @@ public sealed class AkkaClusterOptions
/// </para>
/// </remarks>
public string SplitBrainResolverStrategy { get; set; } = "auto-down";
/// <summary>
/// The simultaneous-cold-start split-brain guard (<c>Cluster:BootstrapGuard</c>). Default OFF — a
/// dark switch, so existing deployments and tests keep Akka's config-driven self-first auto-join
/// unchanged. See <see cref="ClusterBootstrapGuard"/> for the decision logic and
/// <c>ClusterBootstrapCoordinator</c> for the runtime.
/// </summary>
public ClusterBootstrapGuardOptions BootstrapGuard { get; set; } = new();
}
/// <summary>
/// Configuration for the simultaneous-cold-start split-brain guard. When
/// <see cref="Enabled"/>, the node does NOT auto-join from its config seeds; a coordinator picks the
/// join order (founder self-first / joiner peer-first) after a reachability probe. See
/// <see cref="ClusterBootstrapGuard"/>.
/// </summary>
public sealed class ClusterBootstrapGuardOptions
{
/// <summary>
/// Gets or sets whether the guard is active. Default <see langword="false"/> — Akka auto-joins from
/// the config seed list exactly as before. Only meaningful on a node that is one of its own two
/// pair seeds; inert everywhere else.
/// </summary>
public bool Enabled { get; set; }
/// <summary>
/// Gets or sets how long the higher-address node probes its partner's Akka endpoint before
/// concluding the partner is dead and forming alone. Must comfortably exceed the partner's
/// worst-case process-start-to-Akka-bind time so a slow-but-alive partner is never mistaken for a
/// dead one (which would re-open the split). Default 25 s.
/// </summary>
public int PartnerProbeSeconds { get; set; } = 25;
/// <summary>
/// Gets or sets the interval between partner reachability probes, in milliseconds. Default 500 ms.
/// </summary>
public int PartnerProbeIntervalMs { get; set; } = 500;
/// <summary>
/// Gets or sets the per-probe TCP connect timeout, in milliseconds. Default 1000 ms.
/// </summary>
public int ProbeConnectTimeoutMs { get; set; } = 1000;
}
@@ -41,6 +41,8 @@ public sealed class AkkaClusterOptionsValidator : OptionsValidatorBase<AkkaClust
{
ArgumentNullException.ThrowIfNull(options);
ValidateBootstrapGuard(builder, options.BootstrapGuard);
var seeds = options.SeedNodes ?? Array.Empty<string>();
if (seeds.Length == 0)
{
@@ -68,6 +70,25 @@ public sealed class AkkaClusterOptionsValidator : OptionsValidatorBase<AkkaClust
+ "the entries — see docs/Redundancy.md → 'Bootstrap: self-first seed ordering'.");
}
/// <summary>
/// When the bootstrap guard is enabled, its timing knobs must be positive. A zero/negative
/// <c>PartnerProbeSeconds</c> in particular silently degrades the guard to "never wait, always
/// conclude the partner is dead" — it would form alone immediately and re-open the very split it
/// exists to close. Fail-fast at boot rather than producing silence (the same rationale every
/// sibling options validator in this project cites).
/// </summary>
private static void ValidateBootstrapGuard(ValidationBuilder builder, ClusterBootstrapGuardOptions guard)
{
if (guard is null || !guard.Enabled) return;
if (guard.PartnerProbeSeconds <= 0)
builder.Add($"Cluster:BootstrapGuard:PartnerProbeSeconds must be > 0 when the guard is enabled (was {guard.PartnerProbeSeconds}).");
if (guard.PartnerProbeIntervalMs <= 0)
builder.Add($"Cluster:BootstrapGuard:PartnerProbeIntervalMs must be > 0 when the guard is enabled (was {guard.PartnerProbeIntervalMs}).");
if (guard.ProbeConnectTimeoutMs <= 0)
builder.Add($"Cluster:BootstrapGuard:ProbeConnectTimeoutMs must be > 0 when the guard is enabled (was {guard.ProbeConnectTimeoutMs}).");
}
/// <summary>
/// True when <paramref name="seed"/> addresses this node itself — host AND port.
/// </summary>
@@ -0,0 +1,201 @@
using System.Diagnostics;
using System.Net.Sockets;
using Akka.Actor;
using Akka.Cluster;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
namespace ZB.MOM.WW.OtOpcUa.Cluster;
/// <summary>
/// Runs the simultaneous-cold-start split-brain guard: when <c>Cluster:BootstrapGuard:Enabled</c>, the
/// node starts with NO config seed nodes (see <c>BuildClusterOptions</c>), and this coordinator picks
/// the join order and issues the single <see cref="Akka.Cluster.Cluster.JoinSeedNodes"/> once the
/// ActorSystem is up. See <see cref="ClusterBootstrapGuard"/> for the decision logic and the
/// split-brain rationale.
/// </summary>
/// <remarks>
/// <para>
/// The founder (lower canonical address) joins its self-first order immediately. The higher
/// node probes its partner's Akka endpoint (a plain TCP connect — the partner has bound its
/// port well before it forms) up to <see cref="ClusterBootstrapGuardOptions.PartnerProbeSeconds"/>:
/// reachable ⇒ peer-first (join the founder); unreachable ⇒ self-first (partner is dead, form
/// alone). The join runs on a background task so it never blocks host startup, and it is issued
/// exactly once — the coordinator never re-forms a node mid-handshake, the failure mode of the
/// retired <c>SelfFormAfter</c> watchdog.
/// </para>
/// </remarks>
public sealed class ClusterBootstrapCoordinator : IHostedService
{
private readonly Func<ActorSystem> _system;
private readonly AkkaClusterOptions _options;
private readonly ILogger<ClusterBootstrapCoordinator> _log;
private readonly CancellationTokenSource _cts = new();
private Task? _joinTask;
/// <summary>Creates the coordinator.</summary>
/// <param name="system">Lazy accessor for the node's ActorSystem (resolved after Akka's hosted service starts it).</param>
/// <param name="options">The bound cluster options (seed list + guard settings).</param>
/// <param name="log">Logger for the bootstrap decision.</param>
public ClusterBootstrapCoordinator(
Func<ActorSystem> system, IOptions<AkkaClusterOptions> options, ILogger<ClusterBootstrapCoordinator> log)
{
_system = system ?? throw new ArgumentNullException(nameof(system));
_options = options?.Value ?? throw new ArgumentNullException(nameof(options));
_log = log ?? throw new ArgumentNullException(nameof(log));
}
/// <inheritdoc/>
public Task StartAsync(CancellationToken cancellationToken)
{
if (!_options.BootstrapGuard.Enabled)
return Task.CompletedTask; // dark switch off — Akka auto-joined from config seeds already.
// Fire-and-forget so a higher node with a dead partner (which waits the full probe window)
// never blocks host startup. Exceptions are logged; a failed join leaves the node unjoined,
// which is visible and recoverable, never a silent split.
_joinTask = Task.Run(() => RunAsync(_cts.Token), _cts.Token);
return Task.CompletedTask;
}
/// <inheritdoc/>
public async Task StopAsync(CancellationToken cancellationToken)
{
await _cts.CancelAsync().ConfigureAwait(false);
if (_joinTask is not null)
{
try { await _joinTask.ConfigureAwait(false); }
catch (OperationCanceledException) { /* expected on shutdown */ }
}
}
private async Task RunAsync(CancellationToken ct)
{
try
{
var seeds = _options.SeedNodes ?? Array.Empty<string>();
var role = ClusterBootstrapGuard.Analyze(seeds, _options.PublicHostname, _options.Port);
string[] order;
var committedPeerFirst = false;
if (!role.Applies)
{
// Not a pair seed (single-seed legacy site node, self absent, malformed, or 3+ seeds):
// join the configured seeds unchanged — the guard arbitrates only the 2-node pair race.
order = seeds;
_log.LogInformation(
"Bootstrap guard: not a pair seed ({SeedCount} seed(s)); joining configured seeds unchanged.",
seeds.Length);
}
else if (role.IsFounder)
{
order = role.SelfFirstOrder;
_log.LogInformation(
"Bootstrap guard: this node is the preferred founder (lower address); joining self-first, forming immediately if no peer answers.");
}
else
{
var reachable = await ProbePartnerAsync(role.PartnerHost!, role.PartnerPort, ct).ConfigureAwait(false);
order = ClusterBootstrapGuard.HigherNodeOrder(role, reachable);
committedPeerFirst = reachable;
_log.LogInformation(
"Bootstrap guard: this node is the higher address; partner {PartnerHost}:{PartnerPort} was {Reachability} within the probe window; joining {Order}.",
role.PartnerHost, role.PartnerPort, reachable ? "REACHABLE" : "UNREACHABLE",
reachable ? "peer-first (join the founder)" : "self-first (partner down, forming alone)");
}
if (order.Length == 0)
{
_log.LogWarning("Bootstrap guard: no seed nodes to join; node stays unjoined until a seed is configured.");
return;
}
var cluster = Akka.Cluster.Cluster.Get(_system());
var addresses = order.Select(Address.Parse).ToArray();
cluster.JoinSeedNodes(addresses);
// Peer-first is a one-way commitment: JoinSeedNodeProcess never self-forms. If the founder
// died in the probe→join window, this node hangs unjoined forever. We do NOT re-form here
// (the retired SelfFormAfter mid-handshake failure mode) — we make the hang operator-visible
// so a restart, which re-runs the guard against the now-dead founder and self-forms, recovers
// it. Nothing is done on the founder / self-first paths, which always self-form on their own.
if (committedPeerFirst)
await WarnIfNotUpAsync(cluster, ct).ConfigureAwait(false);
}
catch (OperationCanceledException)
{
// Host shutting down before the join completed — nothing to do.
}
catch (Exception ex)
{
_log.LogError(ex, "Bootstrap guard: failed to issue JoinSeedNodes; node remains unjoined.");
}
}
/// <summary>
/// After committing peer-first, waits a bounded time for this node to reach <c>Up</c> and logs a
/// clear warning if it does not — the founder must have died in the probe→join window, leaving this
/// node hung in <c>JoinSeedNodeProcess</c>. Read-only: it never re-forms or re-joins (that is the
/// retired mid-handshake failure mode); it only surfaces the condition so an operator restarts the node.
/// </summary>
private async Task WarnIfNotUpAsync(Akka.Cluster.Cluster cluster, CancellationToken ct)
{
// Give the join a generous grace: the founder's own self-form (seed-node-timeout) plus this
// node's join round-trip. Two probe windows is comfortably beyond both.
var grace = TimeSpan.FromSeconds(Math.Max(10, _options.BootstrapGuard.PartnerProbeSeconds * 2));
var sw = Stopwatch.StartNew();
while (!ct.IsCancellationRequested && sw.Elapsed < grace)
{
if (cluster.SelfMember.Status == MemberStatus.Up) return; // joined — all good
try { await Task.Delay(TimeSpan.FromSeconds(1), ct).ConfigureAwait(false); }
catch (OperationCanceledException) { return; }
}
if (!ct.IsCancellationRequested && cluster.SelfMember.Status != MemberStatus.Up)
_log.LogWarning(
"Bootstrap guard: committed peer-first but this node is still {Status} after {Grace}s — its founder likely died in the probe→join window. This node will NOT self-form on its own (peer-first never does). RESTART it to recover: the guard will re-run, find the founder down, and form alone.",
cluster.SelfMember.Status, (int)grace.TotalSeconds);
}
/// <summary>
/// Polls the partner's Akka endpoint with a plain TCP connect until it is reachable or the probe
/// window elapses. Reachable-at-TCP is a sound proxy for "the partner is coming up": a node binds
/// its Akka port near the start of startup, well before it forms or joins a cluster.
/// </summary>
private async Task<bool> ProbePartnerAsync(string host, int port, CancellationToken ct)
{
var window = TimeSpan.FromSeconds(Math.Max(0, _options.BootstrapGuard.PartnerProbeSeconds));
var interval = TimeSpan.FromMilliseconds(Math.Max(50, _options.BootstrapGuard.PartnerProbeIntervalMs));
var sw = Stopwatch.StartNew();
while (!ct.IsCancellationRequested)
{
if (await TryConnectAsync(host, port, ct).ConfigureAwait(false)) return true;
if (sw.Elapsed >= window) return false;
try { await Task.Delay(interval, ct).ConfigureAwait(false); }
catch (OperationCanceledException) { return false; }
}
return false;
}
private async Task<bool> TryConnectAsync(string host, int port, CancellationToken ct)
{
try
{
using var client = new TcpClient();
using var cts = CancellationTokenSource.CreateLinkedTokenSource(ct);
cts.CancelAfter(Math.Max(100, _options.BootstrapGuard.ProbeConnectTimeoutMs));
await client.ConnectAsync(host, port, cts.Token).ConfigureAwait(false);
return client.Connected;
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
throw; // host shutdown — propagate
}
catch
{
return false; // connect refused / timed out / DNS not resolvable yet — partner not up
}
}
}
@@ -0,0 +1,153 @@
using System.Text.RegularExpressions;
namespace ZB.MOM.WW.OtOpcUa.Cluster;
/// <summary>
/// Pure decision logic for the simultaneous-cold-start split-brain guard.
/// </summary>
/// <remarks>
/// <para>
/// Both nodes of a 2-node pair are self-first seeds so <b>either</b> can cold-start alone when
/// its partner is dead (see <see cref="AkkaClusterOptions.SeedNodes"/>). The cost is that when
/// BOTH cold-start at the same instant, each runs Akka's <c>FirstSeedNodeProcess</c>, times out
/// waiting for the other, and forms its OWN single-node cluster — a split brain (two Primaries
/// in one pair). Two independent clusters do not auto-merge, so the split persists until an
/// operator intervenes.
/// </para>
/// <para>
/// This guard breaks the symmetry deterministically without giving up cold-start-alone. The
/// node with the lexicographically <b>lower</b> canonical address is the <i>preferred
/// founder</i>: it always uses self-first order and forms immediately. The <b>higher</b> node
/// first probes whether its partner's Akka endpoint is reachable (see
/// <see cref="ClusterBootstrapCoordinator"/>): reachable ⇒ peer-first order so it JOINS the
/// founder rather than racing it; unreachable after the probe window ⇒ the partner is genuinely
/// down, so it falls back to self-first and forms alone. Exactly one node founds when both are
/// present; the lower node always founds when it starts alone.
/// </para>
/// <para>
/// <b>Residual trade-off (accepted).</b> Once the higher node observes the partner reachable it
/// commits to peer-first — Akka's <c>JoinSeedNodeProcess</c> retries <c>InitJoin</c> forever and
/// never self-forms. If the founder dies in the small window between the probe succeeding and
/// the join completing, the higher node hangs unjoined until it is restarted (a restart re-runs
/// the guard, finds the founder unreachable, and self-forms). We deliberately do NOT re-decide
/// mid-handshake — that is exactly the retired <c>SelfFormAfter</c> failure mode. Instead
/// <see cref="ClusterBootstrapCoordinator"/> makes the hang operator-visible with a warning if
/// the node has not come Up within a bounded time after committing peer-first.
/// </para>
/// <para>
/// This decides the join order BEFORE issuing a single <c>JoinSeedNodes</c>, from an explicit
/// reachability signal — unlike the retired <c>SelfFormAfter</c> watchdog, which fired a
/// <c>Join(self)</c> mid-handshake on a bare timeout and could not tell "no seed answered" from
/// "a join is in flight", islanding a node a failover had just bounced.
/// </para>
/// </remarks>
public static class ClusterBootstrapGuard
{
private static readonly Regex SeedPattern =
new(@"^akka(?:\.[a-z0-9]+)?://[^@/]+@(?<host>[^:/]+):(?<port>\d+)", RegexOptions.IgnoreCase | RegexOptions.Compiled);
/// <summary>The analyzed bootstrap role of this node within its seed list.</summary>
/// <param name="Applies">True only when the guard engages: exactly two seeds, one of them this node.</param>
/// <param name="IsFounder">True when this node is the preferred founder (lower address) — form immediately, no probe.</param>
/// <param name="PartnerHost">The partner's advertised host (to probe when this node is the higher one); null when not applicable.</param>
/// <param name="PartnerPort">The partner's Akka port.</param>
/// <param name="SelfFirstOrder"><c>[self, partner]</c> — self-first; forms a new cluster when no peer answers.</param>
/// <param name="PeerFirstOrder"><c>[partner, self]</c> — peer-first; joins the partner's cluster, never self-forms while it is up.</param>
public sealed record BootstrapRole(
bool Applies,
bool IsFounder,
string? PartnerHost,
int PartnerPort,
string[] SelfFirstOrder,
string[] PeerFirstOrder);
/// <summary>
/// Analyzes the configured seed list to decide this node's bootstrap role. The guard engages only
/// for the Phase-6 pair shape (exactly two seeds, one of them this node); any other shape
/// (single-seed legacy site node, three+ seeds, self absent) returns <see cref="BootstrapRole.Applies"/>
/// = false and the caller joins the configured seeds unchanged.
/// </summary>
/// <param name="seeds">The configured seed URIs (self-first by convention, but order is not relied on here).</param>
/// <param name="selfHost">This node's advertised host (<see cref="AkkaClusterOptions.PublicHostname"/>).</param>
/// <param name="selfPort">This node's Akka port (<see cref="AkkaClusterOptions.Port"/>).</param>
/// <returns>The analyzed role; never null.</returns>
public static BootstrapRole Analyze(IReadOnlyList<string> seeds, string selfHost, int selfPort)
{
ArgumentNullException.ThrowIfNull(seeds);
ArgumentException.ThrowIfNullOrWhiteSpace(selfHost);
var na = new BootstrapRole(false, false, null, 0, Array.Empty<string>(), Array.Empty<string>());
if (seeds.Count != 2) return na;
var selfKey = Key(selfHost, selfPort);
string? self = null, partner = null;
string? partnerHost = null;
var partnerPort = 0;
foreach (var seed in seeds)
{
if (!TryParse(seed, out var host, out var port)) return na;
if (string.Equals(Key(host, port), selfKey, StringComparison.OrdinalIgnoreCase))
self = seed;
else
{
partner = seed;
partnerHost = host;
partnerPort = port;
}
}
// Self must be exactly one of the two, and the other must be a distinct partner.
if (self is null || partner is null || partnerHost is null) return na;
var selfFirst = new[] { self, partner };
var peerFirst = new[] { partner, self };
// Deterministic tie-break: the lower canonical address is the preferred founder. Both nodes
// compute the same comparison (same two seeds), so exactly one is the founder. Case-INSENSITIVE
// to match how self/partner are classified above (and AkkaClusterOptionsValidator.IsSelf):
// container/DNS hostnames are conventionally case-insensitive, and a casing difference between
// the two sides' seed config must NOT make both think they are the founder (which would reopen
// the very split this guard closes).
var isFounder = string.Compare(
Key(selfHost, selfPort),
Key(partnerHost, partnerPort),
StringComparison.OrdinalIgnoreCase) < 0;
return new BootstrapRole(true, isFounder, partnerHost, partnerPort, selfFirst, peerFirst);
}
/// <summary>
/// The order the higher (non-founder) node uses once it has learned whether its partner is
/// reachable: peer-first when the founder is up (join it), self-first when the founder is absent
/// (form alone — cold-start-alone preserved).
/// </summary>
/// <param name="role">The analyzed role (must be applicable and NOT the founder).</param>
/// <param name="partnerReachable">Whether the partner's Akka endpoint became reachable within the probe window.</param>
/// <returns>The seed order to pass to <c>Cluster.JoinSeedNodes</c>.</returns>
public static string[] HigherNodeOrder(BootstrapRole role, bool partnerReachable)
{
ArgumentNullException.ThrowIfNull(role);
return partnerReachable ? role.PeerFirstOrder : role.SelfFirstOrder;
}
/// <summary>Parses <c>akka.tcp://system@host:port[/...]</c> into host + port.</summary>
/// <param name="seed">The seed URI.</param>
/// <param name="host">The parsed advertised host.</param>
/// <param name="port">The parsed Akka port.</param>
/// <returns>True when the seed matched the expected shape.</returns>
public static bool TryParse(string? seed, out string host, out int port)
{
host = string.Empty;
port = 0;
if (string.IsNullOrWhiteSpace(seed)) return false;
var m = SeedPattern.Match(seed.Trim());
if (!m.Success) return false;
if (!int.TryParse(m.Groups["port"].Value, out port)) return false;
host = m.Groups["host"].Value;
return true;
}
private static string Key(string host, int port) => $"{host}:{port}";
}
@@ -20,6 +20,8 @@ public sealed class ClusterRoleInfo : IClusterRoleInfo, IDisposable
private readonly ILogger<ClusterRoleInfo> _logger;
private readonly CommonsNodeId _localNode;
private readonly HashSet<string> _localRoles;
private readonly string? _clusterRole;
private readonly string? _clusterId;
private readonly object _lock = new();
private readonly Dictionary<string, Member?> _roleLeaders = new(StringComparer.Ordinal);
private readonly Dictionary<string, HashSet<Member>> _membersByRole = new(StringComparer.Ordinal);
@@ -39,6 +41,23 @@ public sealed class ClusterRoleInfo : IClusterRoleInfo, IDisposable
_localNode = CommonsNodeId.Parse($"{options.Value.PublicHostname}:{options.Value.Port}");
_localRoles = new HashSet<string>(options.Value.Roles, StringComparer.Ordinal);
// Derived from the node's OWN configured roles (not live Cluster.State) so this identity is
// available at host-build time, before the cluster forms. Iterate the ORIGINAL array — not
// _localRoles — because first-wins requires configuration order, which HashSet does not
// guarantee.
var clusterRoles = options.Value.Roles.Where(RoleParser.IsClusterRole).ToArray();
if (clusterRoles.Length > 0)
{
_clusterRole = clusterRoles[0];
_clusterId = RoleParser.ClusterIdFromRole(_clusterRole);
if (clusterRoles.Length > 1)
{
_logger.LogWarning(
"Node configured with {Count} cluster-scoped roles ({Roles}); using the first ({ClusterRole})",
clusterRoles.Length, string.Join(", ", clusterRoles), _clusterRole);
}
}
SeedFromCurrentState();
_subscriber = system.ActorOf(Props.Create(() => new SubscriberActor(this)), "clusterroleinfo-subscriber");
}
@@ -49,6 +68,12 @@ public sealed class ClusterRoleInfo : IClusterRoleInfo, IDisposable
/// <inheritdoc />
public IReadOnlySet<string> LocalRoles => _localRoles;
/// <inheritdoc />
public string? ClusterRole => _clusterRole;
/// <inheritdoc />
public string? ClusterId => _clusterId;
/// <inheritdoc />
public bool HasRole(string role) => _localRoles.Contains(role);
@@ -186,7 +211,9 @@ public sealed class ClusterRoleInfo : IClusterRoleInfo, IDisposable
Receive<ClusterEvent.IMemberEvent>(e => owner.HandleMemberEvent(e));
Receive<ClusterEvent.RoleLeaderChanged>(e => owner.HandleRoleLeaderChanged(e));
// LeaderChanged is intentionally a no-op here: ServiceLevel lives in RedundancyStateActor
// (admin-role singleton) and has no consumer on this ClusterRoleInfo path by design.
// (per-cluster mesh Phase 6: a cluster-{ClusterId}-scoped singleton spawned on every
// driver node, no longer the admin-role singleton it once was) and has no consumer on this
// ClusterRoleInfo path by design.
Receive<ClusterEvent.LeaderChanged>(_ => { });
Receive<ClusterEvent.CurrentClusterState>(_ => { /* seeded from initial snapshot */ });
}
@@ -2,11 +2,36 @@ namespace ZB.MOM.WW.OtOpcUa.Cluster;
public static class RoleParser
{
/// <summary>The admin-role name.</summary>
public const string Admin = "admin";
/// <summary>The driver-role name. Single source of truth — downstream <c>DriverRole</c>
/// constants (<c>RedundancyStateActor</c>, <c>ClusterNodeAddressReconcilerActor</c>,
/// <c>ServiceCollectionExtensions</c>) alias this value rather than duplicating the literal.</summary>
public const string Driver = "driver";
/// <summary>The dev-role name.</summary>
public const string Dev = "dev";
/// <summary>Prefix identifying a cluster-scoped role, e.g. <c>cluster-SITE-A</c> (per-cluster mesh, Phase 6).</summary>
public const string ClusterRolePrefix = "cluster-";
private static readonly HashSet<string> Allowed = new(StringComparer.Ordinal)
{
"admin", "driver", "dev",
Admin, Driver, Dev,
};
/// <summary>Returns true when <paramref name="role"/> is a cluster-scoped role — starts with
/// <see cref="ClusterRolePrefix"/> and carries a non-empty ClusterId suffix.</summary>
/// <param name="role">The role name to test.</param>
public static bool IsClusterRole(string role) =>
role.StartsWith(ClusterRolePrefix, StringComparison.Ordinal) && role.Length > ClusterRolePrefix.Length;
/// <summary>Extracts the ClusterId from a cluster-scoped role, e.g. <c>cluster-SITE-A</c> → <c>SITE-A</c>.
/// Assumes <see cref="IsClusterRole"/> is true for <paramref name="role"/>.</summary>
/// <param name="role">The cluster-scoped role name.</param>
public static string ClusterIdFromRole(string role) => role[ClusterRolePrefix.Length..];
/// <summary>Parses a comma-separated string of role names into a validated array.</summary>
/// <param name="raw">The raw role string to parse.</param>
/// <returns>The distinct, lower-cased, validated role names.</returns>
@@ -22,9 +47,10 @@ public static class RoleParser
foreach (var r in roles)
{
if (!Allowed.Contains(r))
if (!Allowed.Contains(r) && !IsClusterRole(r))
throw new ArgumentException(
$"Unknown role '{r}'. Allowed: {string.Join(", ", Allowed)}.", nameof(raw));
$"Unknown role '{r}'. Allowed: {string.Join(", ", Allowed)}, or '{ClusterRolePrefix}<ClusterId>'.",
nameof(raw));
}
return roles;
@@ -56,6 +56,14 @@ public static class ServiceCollectionExtensions
services.AddValidatedOptions<TelemetryDialOptions, TelemetryDialOptionsValidator>(
configuration, TelemetryDialOptions.SectionName);
// Per-cluster mesh Phase 6: a node carrying a cluster-{ClusterId} role (the split topology)
// must use the mesh-crossing transports — the DPS dark-switch branches cannot cross a split
// mesh and would deliver nothing silently. Hung on MeshTransportOptions (which already has
// ValidateOnStart from its own registration above) so it fires at boot alongside the sibling
// validators; TryAddEnumerable inside AddValidatedOptions keeps the second validator additive.
services.AddValidatedOptions<MeshTransportOptions, SplitTopologyTransportValidator>(
configuration, MeshTransportOptions.SectionName);
services.AddSingleton<IClusterRoleInfo, ClusterRoleInfo>();
return services;
@@ -233,7 +241,13 @@ public static class ServiceCollectionExtensions
return new ClusterOptions
{
SeedNodes = options.SeedNodes,
// When the bootstrap guard is on, DO NOT hand Akka the seed list: Akka.Cluster.Hosting
// auto-joins from ClusterOptions.SeedNodes the instant the ActorSystem starts, which is
// exactly the self-first race the guard exists to arbitrate. Left empty, the node starts
// unjoined and ClusterBootstrapCoordinator picks the order (founder self-first / joiner
// peer-first, after a reachability probe) and issues the single Cluster.JoinSeedNodes.
// The config seed list is still read — by the coordinator, off AkkaClusterOptions.SeedNodes.
SeedNodes = options.BootstrapGuard.Enabled ? Array.Empty<string>() : options.SeedNodes,
Roles = options.Roles,
SplitBrainResolver = IsKeepOldest(options) ? new KeepOldestOption { DownIfAlone = true } : null,
};
@@ -0,0 +1,137 @@
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Options;
namespace ZB.MOM.WW.OtOpcUa.Cluster;
/// <summary>
/// Fails the host at startup when a node configured for the <b>split topology</b> (per-cluster
/// mesh, Phase 6) is left on a DistributedPubSub transport mode that cannot cross the mesh
/// boundary.
/// </summary>
/// <remarks>
/// <para>
/// Phase 6 splits the single fleet mesh into one 2-node mesh per application cluster. The
/// Phase 2/3/5 dark switches keep their DPS branches alive, but DPS is mesh-wide — it delivers
/// nothing between a central node and a site node that no longer share a gossip ring. The
/// marker of the split topology is a <c>cluster-{ClusterId}</c> role
/// (<see cref="RoleParser.IsClusterRole"/>); those roles exist only post-Phase-6. A node that
/// carries one <b>must</b> use the mesh-crossing transports, and this validator refuses to
/// boot it otherwise — the alternative is a node that runs, listens, and silently delivers no
/// commands or telemetry, which has no stack trace and no failing node to point at.
/// </para>
/// <para>
/// <b>The rule is conditional and must stay that way.</b> It fires only when this node's
/// <i>effective</i> Akka roles contain at least one cluster-scoped role — resolved the same
/// way the node resolves them (<c>Cluster:Roles</c>, falling back to <c>OTOPCUA_ROLES</c> when
/// that is empty, normalized case-insensitively). A legacy / single-mesh / test node has none,
/// so DPS is still legitimate and the validator passes unconditionally.
/// </para>
/// <para>
/// <b>Not validated here:</b> <c>ConfigSource:Mode</c>. Phase 4's
/// <see cref="ConfigSourceOptionsValidator"/> already requires <c>FetchAndCache</c> on a
/// driver-only node, and a fused node legitimately stays <c>Direct</c> — duplicating that here
/// would only risk the two rules drifting apart.
/// </para>
/// </remarks>
public sealed class SplitTopologyTransportValidator : IValidateOptions<MeshTransportOptions>
{
private readonly IConfiguration _configuration;
/// <summary>
/// DI-constructed by <c>AddValidatedOptions</c> (a plain <c>AddSingleton</c>), so a
/// constructor dependency is safe here. <see cref="IConfiguration"/> — not
/// <c>IClusterRoleInfo</c> — is the source of this node's roles and the three transport modes:
/// <c>IClusterRoleInfo</c>'s implementation needs the <c>ActorSystem</c>, which does not exist
/// yet at <c>ValidateOnStart</c> time. Reading the modes from configuration too (rather than
/// from three injected <c>IOptions</c>) mirrors how the sibling validators cross-read
/// <c>Cluster:Roles</c>, and avoids a service-resolution ordering dependency at validation
/// time.
/// </summary>
public SplitTopologyTransportValidator(IConfiguration configuration)
{
_configuration = configuration;
}
/// <inheritdoc />
public ValidateOptionsResult Validate(string? name, MeshTransportOptions options)
{
// The options instance is intentionally unused: every value this rule depends on is cross-read
// from IConfiguration, the same way ConfigSourceOptionsValidator reads Cluster:Roles.
//
// Resolve the EFFECTIVE Akka roles exactly as the node does, or this "fail loud" rule fires on
// the wrong role view and passes silently on the two shapes it exists to catch:
// (a) Roles-source divergence. AkkaClusterOptions.Roles binds Cluster:Roles but falls back to
// OTOPCUA_ROLES (via RoleParser.Parse — the same call Program.cs makes) when Cluster:Roles
// is empty. A node configured with only OTOPCUA_ROLES=driver,cluster-SITE-A genuinely
// carries the cluster role in Akka, so it must be validated too.
// (b) Case. The Cluster:Roles bind path is verbatim (no lowercasing), whereas RoleParser.Parse
// lowercases the OTOPCUA_ROLES path. Normalize BOTH here (trim + ToLowerInvariant) so
// 'Cluster-SITE-A' / 'Driver' / 'Admin' cannot slip past the ordinal IsClusterRole /
// driver / admin checks.
var configured = _configuration.GetSection("Cluster:Roles").Get<string[]>() ?? Array.Empty<string>();
var effective = configured.Length > 0
? configured
: RoleParser.Parse(Environment.GetEnvironmentVariable("OTOPCUA_ROLES"));
var normalized = Array.ConvertAll(effective, r => r.Trim().ToLowerInvariant());
var clusterRoleIndex = Array.FindIndex(normalized, RoleParser.IsClusterRole);
if (clusterRoleIndex < 0)
{
// No cluster-{ClusterId} role → not the split topology → DPS is legitimate. This is the
// exemption that keeps the validator a no-op for every legacy / single-mesh / test node.
return ValidateOptionsResult.Success;
}
// Classification is normalized (above); the ClusterId in the message is taken from the
// operator's original spelling so it names what they actually wrote.
var clusterId = RoleParser.ClusterIdFromRole(effective[clusterRoleIndex].Trim());
var isDriver = Array.IndexOf(normalized, RoleParser.Driver) >= 0;
var isAdmin = Array.IndexOf(normalized, RoleParser.Admin) >= 0;
var errors = new List<string>();
// Modes are read from configuration directly; an unset key inherits the class default (Dps for
// all three), so a split node that simply never set the key fails exactly as an explicit Dps
// would — which is the point.
var meshMode = _configuration["MeshTransport:Mode"] ?? MeshTransportOptions.ModeDps;
var telemetryMode = _configuration["Telemetry:Mode"] ?? TelemetryOptions.ModeDps;
var telemetryDialMode = _configuration["TelemetryDial:Mode"] ?? TelemetryDialOptions.ModeDps;
if (!string.Equals(meshMode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase))
{
errors.Add(
$"Cluster:Roles declares a cluster-{clusterId} role (split topology) but "
+ $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.Mode)} is "
+ $"'{meshMode}'; a split mesh cannot use DistributedPubSub — set "
+ $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.Mode)}="
+ $"{MeshTransportOptions.ModeClusterClient}.");
}
if (isDriver
&& !string.Equals(telemetryMode, TelemetryOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase))
{
errors.Add(
$"Cluster:Roles declares a cluster-{clusterId} role (split topology) with 'driver' but "
+ $"{TelemetryOptions.SectionName}:{nameof(TelemetryOptions.Mode)} is "
+ $"'{telemetryMode}'; a split-topology driver node hosts the telemetry gRPC server — "
+ $"set {TelemetryOptions.SectionName}:{nameof(TelemetryOptions.Mode)}="
+ $"{TelemetryOptions.ModeGrpc}.");
}
if (isAdmin
&& !string.Equals(telemetryDialMode, TelemetryDialOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase))
{
errors.Add(
$"Cluster:Roles declares a cluster-{clusterId} role (split topology) with 'admin' but "
+ $"{TelemetryDialOptions.SectionName}:{nameof(TelemetryDialOptions.Mode)} is "
+ $"'{telemetryDialMode}'; a split-topology central node dials nodes for telemetry — set "
+ $"{TelemetryDialOptions.SectionName}:{nameof(TelemetryDialOptions.Mode)}="
+ $"{TelemetryDialOptions.ModeGrpc}.");
}
return errors.Count == 0
? ValidateOptionsResult.Success
: ValidateOptionsResult.Fail(string.Join(" ", errors));
}
}
@@ -14,6 +14,21 @@ public interface IClusterRoleInfo
NodeId LocalNode { get; }
/// <summary>Gets the set of roles assigned to the local node.</summary>
IReadOnlySet<string> LocalRoles { get; }
/// <summary>
/// Gets the local node's cluster-scoped role (e.g. <c>cluster-SITE-A</c>), or <c>null</c> when
/// the node carries none. Derived from the node's OWN configured roles at construction time —
/// NOT from live cluster membership — so it is available before the cluster forms (per-cluster
/// mesh, Phase 6). When more than one cluster-scoped role is configured, the first one wins.
/// </summary>
string? ClusterRole { get; }
/// <summary>
/// Gets the ClusterId extracted from <see cref="ClusterRole"/> (e.g. <c>SITE-A</c>), or
/// <c>null</c> when the node carries no cluster-scoped role.
/// </summary>
string? ClusterId { get; }
/// <summary>Checks if the local node has the specified role.</summary>
/// <param name="role">Role name to check.</param>
/// <returns>True if the local node has the role; otherwise, false.</returns>
@@ -130,6 +130,14 @@ public sealed class ModbusDriverOptions
/// </summary>
public MelsecFamily MelsecSubFamily { get; init; } = MelsecFamily.Q_L_iQR;
/// <summary>
/// Wire transport selector. <see cref="ModbusTransportMode.Tcp"/> (default) is the
/// historical Modbus TCP/MBAP framing. <see cref="ModbusTransportMode.RtuOverTcp"/>
/// frames Modbus RTU PDUs (CRC16, no MBAP header) over the same TCP socket — for
/// serial-to-Ethernet gateways that bridge RS-485 without re-encapsulating to MBAP.
/// </summary>
public ModbusTransportMode Transport { get; init; } = ModbusTransportMode.Tcp;
/// <summary>
/// When <c>true</c>, the driver suppresses redundant writes: if the most recent
/// successful write to a tag carried value V and a new write of V arrives, the second
@@ -0,0 +1,17 @@
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus;
/// <summary>
/// Wire transport for a Modbus driver instance. <see cref="Tcp"/> = Modbus/TCP (MBAP + TxId);
/// <see cref="RtuOverTcp"/> = RTU framing (address + CRC-16, no MBAP) tunnelled over a socket to
/// a serial→Ethernet gateway. Default <see cref="Tcp"/> — existing configs omit the field.
/// A direct-serial <c>Rtu</c> member is intentionally NOT reserved here (descoped 2026-07-15);
/// do not number-squat it.
/// </summary>
public enum ModbusTransportMode
{
/// <summary>Modbus/TCP — 7-byte MBAP header + transaction id (the historical default).</summary>
Tcp,
/// <summary>RTU framing (<c>[addr][PDU][CRC-16]</c>) tunnelled over a socket to a serial→Ethernet gateway.</summary>
RtuOverTcp,
}
@@ -0,0 +1,48 @@
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus;
/// <summary>
/// CRC-16/MODBUS helper: reflected polynomial <c>0xA001</c>, initial value <c>0xFFFF</c>.
/// On the wire the CRC is appended low byte first, high byte second — <see cref="Append"/>
/// does that; <see cref="Compute"/> just returns the 16-bit value. Byte-stream-agnostic:
/// no socket or framing knowledge lives here.
/// </summary>
public static class ModbusCrc
{
/// <summary>Computes the CRC-16/MODBUS checksum over <paramref name="data"/>.</summary>
/// <param name="data">The bytes to checksum (e.g. the RTU frame minus the trailing CRC).</param>
/// <returns>The 16-bit CRC value.</returns>
public static ushort Compute(ReadOnlySpan<byte> data)
{
ushort crc = 0xFFFF;
foreach (var b in data)
{
crc ^= b;
for (var i = 0; i < 8; i++)
{
var carry = (crc & 0x0001) != 0;
crc >>= 1;
if (carry)
{
crc ^= 0xA001;
}
}
}
return crc;
}
/// <summary>
/// Returns <paramref name="body"/> with its CRC-16/MODBUS appended, low byte first.
/// </summary>
/// <param name="body">The frame body to checksum.</param>
/// <returns>A new array: <paramref name="body"/> followed by <c>[crc &amp; 0xFF, crc &gt;&gt; 8]</c>.</returns>
public static byte[] Append(ReadOnlySpan<byte> body)
{
var crc = Compute(body);
var result = new byte[body.Length + 2];
body.CopyTo(result);
result[^2] = (byte)(crc & 0xFF);
result[^1] = (byte)(crc >> 8);
return result;
}
}
@@ -94,7 +94,8 @@ public sealed class ModbusDriver
/// <summary>Initializes a new Modbus TCP driver with the specified options and transport factory.</summary>
/// <param name="options">Driver configuration options.</param>
/// <param name="driverInstanceId">Unique identifier for this driver instance.</param>
/// <param name="transportFactory">Factory to create the Modbus transport; defaults to ModbusTcpTransport.</param>
/// <param name="transportFactory">Factory to create the Modbus transport; defaults to
/// <see cref="ModbusTransportFactory.Create"/>, which honours <see cref="ModbusDriverOptions.Transport"/>.</param>
/// <param name="logger">Logger instance; defaults to null logger if not provided.</param>
public ModbusDriver(ModbusDriverOptions options, string driverInstanceId,
Func<ModbusDriverOptions, IModbusTransport>? transportFactory = null,
@@ -107,11 +108,7 @@ public sealed class ModbusDriver
_resolver = new EquipmentTagRefResolver<ModbusTagDefinition>(
r => _tagsByRawPath.TryGetValue(r, out var t) ? t : null);
_transportFactory = transportFactory
?? (o => new ModbusTcpTransport(
o.Host, o.Port, o.Timeout, o.AutoReconnect,
keepAlive: o.KeepAlive,
idleDisconnect: o.IdleDisconnectTimeout,
reconnect: o.Reconnect));
?? (o => ModbusTransportFactory.Create(o));
_poll = new PollGroupEngine(
reader: ReadAsync,
onChange: (handle, tagRef, snapshot) =>
@@ -74,6 +74,8 @@ public static class ModbusDriverFactoryExtensions
: ParseEnum<ModbusFamily>(dto.Family, "<driver-level>", driverInstanceId, "Family"),
MelsecSubFamily = dto.MelsecSubFamily is null ? MelsecFamily.Q_L_iQR
: ParseEnum<MelsecFamily>(dto.MelsecSubFamily, "<driver-level>", driverInstanceId, "MelsecSubFamily"),
Transport = dto.Transport is null ? ModbusTransportMode.Tcp
: ParseEnum<ModbusTransportMode>(dto.Transport, "<driver-level>", driverInstanceId, "Transport"),
AutoProhibitReprobeInterval = dto.AutoProhibitReprobeMs is { } reprobeMs ? TimeSpan.FromMilliseconds(reprobeMs) : null,
AutoReconnect = dto.AutoReconnect ?? true,
// v3: the deploy artifact (Wave C) populates rawTags with the cluster's authored raw Modbus
@@ -159,6 +161,8 @@ public static class ModbusDriverFactoryExtensions
public string? Family { get; init; }
/// <summary>Gets or sets the Melsec subfamily.</summary>
public string? MelsecSubFamily { get; init; }
/// <summary>Gets or sets the wire transport mode (Tcp / RtuOverTcp). Null defaults to Tcp.</summary>
public string? Transport { get; init; }
/// <summary>Gets or sets the automatic prohibition reprobei interval in milliseconds.</summary>
public int? AutoProhibitReprobeMs { get; init; }
/// <summary>Gets or sets a value indicating whether automatic reconnection is enabled.</summary>
@@ -29,15 +29,19 @@ public sealed class ModbusDriverProbe : IDriverProbe
/// <inheritdoc />
public string DriverType => "Modbus";
/// <summary>The parsed probe target — host/port/unitId + the effective connection timeout, all read
/// from the SAME factory DTO shape (<c>ModbusDriverConfigDto</c>) the driver factory parses, so
/// <c>timeoutMs</c> binds identically to the factory (R2-11, 05/CONV-2; the OpcUaClient parity rule).</summary>
internal readonly record struct ProbeTarget(string Host, int Port, byte UnitId, TimeSpan Timeout);
/// <summary>The parsed probe target — host/port/unitId + the effective connection timeout + the wire
/// transport mode, all read from the SAME factory DTO shape (<c>ModbusDriverConfigDto</c>) the driver
/// factory parses, so <c>timeoutMs</c> binds identically to the factory (R2-11, 05/CONV-2; the
/// OpcUaClient parity rule) and an <c>RtuOverTcp</c>-authored config probes over RTU framing —
/// matching how the driver will actually run.</summary>
internal readonly record struct ProbeTarget(string Host, int Port, byte UnitId, TimeSpan Timeout, ModbusTransportMode Transport);
/// <summary>Parse the driver config JSON into a <see cref="ProbeTarget"/> using the factory DTO shape.
/// Returns a null target + an error message when the JSON is invalid or has no host/port. The effective
/// timeout mirrors the factory (<c>TimeSpan.FromMilliseconds(dto.TimeoutMs ?? …)</c>); when the config
/// omits <c>timeoutMs</c> the caller's <paramref name="fallbackTimeout"/> is used.</summary>
/// omits <c>timeoutMs</c> the caller's <paramref name="fallbackTimeout"/> is used. <c>Transport</c>
/// mirrors the factory's null-defaults-to-Tcp convention (<see cref="ModbusDriverFactoryExtensions"/>);
/// an unrecognized value is reported as a probe-target error rather than silently falling back.</summary>
/// <param name="configJson">The driver config JSON (factory DTO shape).</param>
/// <param name="fallbackTimeout">The timeout used when the config omits <c>timeoutMs</c>.</param>
/// <returns>The parsed target, or a null target with an error string.</returns>
@@ -52,8 +56,15 @@ public sealed class ModbusDriverProbe : IDriverProbe
if (string.IsNullOrWhiteSpace(dto.Host) || port <= 0)
return (null, "Config has no host/port to probe.");
var transportMode = ModbusTransportMode.Tcp;
if (dto.Transport is not null && !Enum.TryParse(dto.Transport, ignoreCase: true, out transportMode))
{
return (null, $"Config has unknown Transport '{dto.Transport}'. " +
$"Expected one of {string.Join(", ", Enum.GetNames<ModbusTransportMode>())}");
}
var timeout = dto.TimeoutMs is { } ms && ms > 0 ? TimeSpan.FromMilliseconds(ms) : fallbackTimeout;
return (new ProbeTarget(dto.Host!, port, dto.UnitId ?? 1, timeout), null);
return (new ProbeTarget(dto.Host!, port, dto.UnitId ?? 1, timeout, transportMode), null);
}
/// <inheritdoc />
@@ -72,9 +83,17 @@ public sealed class ModbusDriverProbe : IDriverProbe
using var cts = CancellationTokenSource.CreateLinkedTokenSource(ct);
cts.CancelAfter(timeout);
// Phase 1 — TCP connect (using ModbusTcpTransport which handles IPv4 preference).
// Phase 1 — TCP connect, over whichever transport the config's Transport mode selects
// (ModbusTransportFactory is the single mapping point — same switch the live driver uses).
// autoReconnect=false: this is a one-shot probe, no retry loops.
var transport = new ModbusTcpTransport(host, port, timeout, autoReconnect: false);
var transport = ModbusTransportFactory.Create(new ModbusDriverOptions
{
Host = host,
Port = port,
Timeout = timeout,
Transport = t.Transport,
AutoReconnect = false,
});
await using (transport.ConfigureAwait(false))
{
try
@@ -0,0 +1,187 @@
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus;
/// <summary>
/// Modbus RTU framing: builds a request ADU (<c>[unitId] + PDU + CRC</c>) and deframes a
/// response back to its bare PDU. Byte-stream-agnostic — it operates on a <see cref="Stream"/>
/// and knows nothing about sockets or serial ports, so a future serial transport can reuse it.
/// </summary>
/// <remarks>
/// <para>
/// The single genuinely-new correctness risk of the RTU path: an RTU frame carries <b>no
/// length field</b> (unlike TCP's MBAP header), so the response size must be derived from
/// the function code. After reading <c>addr(1)</c> + <c>fc(1)</c>, the remaining byte count
/// is one of three shapes:
/// </para>
/// <list type="table">
/// <listheader><term>Shape</term><description>Trailing bytes after <c>addr,fc</c></description></listheader>
/// <item>
/// <term>Exception (<c>fc &amp; 0x80</c>)</term>
/// <description><c>excCode(1)</c> + <c>CRC(2)</c></description>
/// </item>
/// <item>
/// <term>Read (FC 01/02/03/04)</term>
/// <description><c>byteCount(1)</c> then <c>byteCount</c> data bytes + <c>CRC(2)</c></description>
/// </item>
/// <item>
/// <term>Write echo (FC 05/06/15/16)</term>
/// <description>fixed <c>4</c> bytes + <c>CRC(2)</c></description>
/// </item>
/// </list>
/// <para>
/// The trailing CRC is validated over <c>[addr, ...pdu]</c> (everything except the two CRC
/// bytes) <b>before</b> the frame is interpreted — so a corrupt exception frame surfaces as a
/// desync, never as a bogus <see cref="ModbusException"/>. A CRC mismatch or a short read
/// (truncation / stream closed mid-frame) throws <see cref="ModbusTransportDesyncException"/>;
/// a CRC-valid exception PDU throws <see cref="ModbusException"/> carrying the original
/// function code (<c>fc &amp; 0x7F</c>) and exception code.
/// </para>
/// <para>
/// Once the CRC passes, the response's address byte is validated against the addressed unit.
/// On an RS-485 multi-drop bus a CRC-valid reply from the <b>wrong</b> slave would otherwise be
/// silently accepted as the addressed unit's data; such a frame is a unit-mismatch
/// <see cref="ModbusTransportDesyncException"/>. This ordering is deliberate — a corrupt frame
/// stays a CRC/desync failure, and only a coherent-but-mis-addressed frame becomes a
/// unit-mismatch desync.
/// </para>
/// </remarks>
public static class ModbusRtuFraming
{
/// <summary>
/// Builds an RTU request ADU: the unit id, the PDU, and the trailing CRC-16/MODBUS
/// (appended low byte first).
/// </summary>
/// <param name="unitId">The RTU slave/unit address.</param>
/// <param name="pdu">The bare PDU (function code + data).</param>
/// <returns>A new array: <c>[unitId, ...pdu, crcLo, crcHi]</c>.</returns>
public static byte[] BuildAdu(byte unitId, ReadOnlySpan<byte> pdu)
{
var frame = new byte[1 + pdu.Length];
frame[0] = unitId;
pdu.CopyTo(frame.AsSpan(1));
return ModbusCrc.Append(frame);
}
/// <summary>
/// Reads a single RTU response frame from <paramref name="stream"/> and returns its bare PDU
/// (<c>[fc, ...data]</c>), with the leading unit-id and trailing CRC stripped.
/// </summary>
/// <param name="stream">The byte stream to read the response from.</param>
/// <param name="expectedUnit">
/// The unit id the request was addressed to. The response's address byte is validated against
/// it after the CRC passes; a mismatch is a unit-mismatch desync.
/// </param>
/// <param name="ct">A token to cancel the read.</param>
/// <returns>The bare response PDU (function code followed by its data).</returns>
/// <exception cref="ModbusTransportDesyncException">
/// The frame was truncated (stream closed mid-frame), its trailing CRC did not validate, or its
/// address byte did not match <paramref name="expectedUnit"/>.
/// </exception>
/// <exception cref="ModbusException">
/// The frame was a CRC-valid Modbus exception PDU (high bit set on the function code).
/// </exception>
public static async Task<byte[]> ReadResponsePduAsync(
Stream stream, byte expectedUnit, CancellationToken ct)
{
// Header: addr(1) + fc(1). The function code selects how many more bytes to read.
var header = new byte[2];
await ReadExactlyAsync(stream, header, ct).ConfigureAwait(false);
var fc = header[1];
int trailing; // bytes after addr+fc, up to and including the 2 CRC bytes.
if ((fc & 0x80) != 0)
{
// Exception frame: excCode(1) + CRC(2).
trailing = 1 + 2;
}
else if (fc is 0x01 or 0x02 or 0x03 or 0x04)
{
// Read response: byteCount(1) then byteCount data bytes + CRC(2).
var bc = new byte[1];
await ReadExactlyAsync(stream, bc, ct).ConfigureAwait(false);
var byteCount = bc[0];
var rest = new byte[byteCount + 2];
await ReadExactlyAsync(stream, rest, ct).ConfigureAwait(false);
return ValidateAndStrip(expectedUnit, header, bc, rest);
}
else
{
// Fixed 4-byte echo: correct for the write FCs this driver emits (05/06/0F/10). A future
// variable-length response FC outside 01-04 (e.g. FC23) would be mis-sized here and
// surface as a desync — revisit the FC-shape table if the driver starts emitting one.
trailing = 4 + 2;
}
var tail = new byte[trailing];
await ReadExactlyAsync(stream, tail, ct).ConfigureAwait(false);
return ValidateAndStrip(expectedUnit, header, ReadOnlySpan<byte>.Empty, tail);
}
/// <summary>
/// Assembles the full frame from its pieces, validates the trailing CRC over everything
/// except the CRC bytes, then validates the response address against
/// <paramref name="expectedUnit"/> and strips <c>addr</c> + <c>CRC</c> to return the bare PDU.
/// A CRC-valid exception PDU throws <see cref="ModbusException"/>.
/// </summary>
private static byte[] ValidateAndStrip(
byte expectedUnit, ReadOnlySpan<byte> header, ReadOnlySpan<byte> middle, ReadOnlySpan<byte> tail)
{
// Reassemble the whole frame: header(addr,fc) + middle(optional byteCount) + tail.
var frame = new byte[header.Length + middle.Length + tail.Length];
header.CopyTo(frame);
middle.CopyTo(frame.AsSpan(header.Length));
tail.CopyTo(frame.AsSpan(header.Length + middle.Length));
// CRC covers everything except the trailing 2 CRC bytes.
var body = frame.AsSpan(0, frame.Length - 2);
var expected = ModbusCrc.Compute(body);
var actual = (ushort)(frame[^2] | (frame[^1] << 8));
if (actual != expected)
throw new ModbusTransportDesyncException(
ModbusTransportDesyncException.DesyncReason.CrcMismatch,
$"Modbus RTU CRC mismatch: computed {expected:X4} got {actual:X4}");
// Unit-address validation runs only AFTER the CRC passes: a corrupt frame is a CRC/desync
// failure, and only a coherent-but-mis-addressed frame (a CRC-valid reply from the wrong
// RS-485 slave) is a unit-mismatch desync. Never silently accept another unit's data.
var addr = frame[0];
if (addr != expectedUnit)
throw new ModbusTransportDesyncException(
ModbusTransportDesyncException.DesyncReason.UnitMismatch,
$"Modbus RTU unit mismatch: expected {expectedUnit} got {addr}");
// Bare PDU = frame minus leading addr and trailing CRC.
var pdu = body[1..].ToArray();
// Exception PDU: high bit set on the function code. The CRC already validated, so this is a
// coherent protocol-level error — surface the ORIGINAL function code (fc & 0x7F).
if ((pdu[0] & 0x80) != 0)
{
var fc = (byte)(pdu[0] & 0x7F);
var exCode = pdu[1];
throw new ModbusException(fc, exCode, $"Modbus exception fc={fc} code={exCode}");
}
return pdu;
}
/// <summary>
/// Fills <paramref name="buf"/> completely from <paramref name="stream"/>, throwing a
/// truncation desync if the stream ends first. Mirrors
/// <c>ModbusTcpTransport.ReadExactlyAsync</c>, but normalises the end-of-stream to a
/// <see cref="ModbusTransportDesyncException"/> so a length-less RTU frame that arrives
/// short is classified as a desync rather than a bare <see cref="EndOfStreamException"/>.
/// </summary>
private static async Task ReadExactlyAsync(Stream stream, byte[] buf, CancellationToken ct)
{
var read = 0;
while (read < buf.Length)
{
var n = await stream.ReadAsync(buf.AsMemory(read), ct).ConfigureAwait(false);
if (n == 0)
throw new ModbusTransportDesyncException(
ModbusTransportDesyncException.DesyncReason.TruncatedFrame,
$"Modbus RTU frame truncated: expected {buf.Length} bytes, got {read}");
read += n;
}
}
}
@@ -0,0 +1,190 @@
using System.Net.Sockets;
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus;
/// <summary>
/// Concrete Modbus RTU-over-TCP transport. Composes <see cref="ModbusSocketLifecycle"/> for the
/// connect / reconnect / keep-alive / idle machinery and <see cref="ModbusRtuFraming"/> for the
/// wire layout — the same lifecycle <see cref="ModbusTcpTransport"/> uses, but with CRC framing
/// instead of MBAP.
/// </summary>
/// <remarks>
/// <para>
/// Delta from <see cref="ModbusTcpTransport"/>: an RTU ADU is <c>[unitId][PDU][CRC]</c> with
/// <b>no MBAP header and no transaction id</b>. Because there is no TxId to correlate
/// interleaved responses, at most one transaction may be on the bus at a time — the
/// <see cref="_gate"/> single-flight is therefore <b>mandatory</b>, not merely a
/// diagnostics convenience.
/// </para>
/// <para>
/// Survives mid-transaction socket drops the same way the TCP transport does: a socket-level
/// failure (<see cref="IOException"/> / <see cref="SocketException"/> /
/// <see cref="EndOfStreamException"/>, and the <see cref="ModbusTransportDesyncException"/>
/// that derives from <see cref="IOException"/>) triggers a single reconnect-then-retry; a
/// second failure bubbles up so the driver's health surface reflects the real state.
/// </para>
/// <para>
/// Every transaction runs under a per-op deadline (a linked
/// <see cref="CancellationTokenSource.CancelAfter(TimeSpan)"/>, design §7 / R2-01) so a
/// frozen gateway can never wedge a poll: the deadline firing is normalised to a
/// <see cref="ModbusTransportDesyncException"/> and tears the socket down so the next attempt
/// reconnects. A <b>caller</b> cancellation is distinguished from that timeout and propagates
/// as a plain <see cref="OperationCanceledException"/> with no teardown.
/// </para>
/// <para>
/// The constructor is connection-free; all socket I/O happens in <see cref="ConnectAsync"/> /
/// <see cref="SendAsync"/>. The <see cref="ForTest"/> seam injects a pre-connected
/// <see cref="Stream"/> (an in-memory duplex fake) so the send/receive orchestration,
/// single-flight, and deadline can be exercised with no socket — in that path
/// <see cref="_life"/> is <see langword="null"/> and the idle / reconnect machinery is
/// skipped (a raw injected stream cannot reconnect).
/// </para>
/// </remarks>
public sealed class ModbusRtuOverTcpTransport : IModbusTransport
{
private readonly ModbusSocketLifecycle? _life;
private readonly Stream? _testStream;
private readonly TimeSpan _timeout;
private readonly SemaphoreSlim _gate = new(1, 1);
private bool _disposed;
/// <summary>Initializes a new instance of the <see cref="ModbusRtuOverTcpTransport"/> class.</summary>
/// <param name="host">The host address or hostname of the Modbus gateway.</param>
/// <param name="port">The TCP port of the Modbus gateway.</param>
/// <param name="timeout">The timeout for socket operations and the per-op response deadline.</param>
/// <param name="autoReconnect">Whether to automatically reconnect on socket failures.</param>
/// <param name="keepAlive">Optional keep-alive configuration for the socket.</param>
/// <param name="idleDisconnect">Optional duration after which an idle socket is disconnected.</param>
/// <param name="reconnect">Optional reconnect backoff configuration.</param>
public ModbusRtuOverTcpTransport(
string host, int port, TimeSpan timeout, bool autoReconnect = true,
ModbusKeepAliveOptions? keepAlive = null,
TimeSpan? idleDisconnect = null,
ModbusReconnectOptions? reconnect = null)
{
_timeout = timeout;
_life = new ModbusSocketLifecycle(host, port, timeout, autoReconnect, keepAlive, idleDisconnect, reconnect);
}
private ModbusRtuOverTcpTransport(Stream testStream, TimeSpan timeout)
{
_timeout = timeout;
_testStream = testStream;
}
/// <summary>
/// Test seam: builds a transport over a pre-connected, in-memory duplex <paramref name="stream"/>,
/// bypassing <see cref="ConnectAsync"/> and the idle / reconnect machinery so the send/receive
/// orchestration, single-flight, and per-op deadline can be exercised with no real socket.
/// </summary>
/// <param name="stream">A pre-connected duplex stream standing in for the socket.</param>
/// <param name="timeout">The per-op response deadline.</param>
/// <returns>A transport driving <paramref name="stream"/> directly.</returns>
internal static ModbusRtuOverTcpTransport ForTest(Stream stream, TimeSpan timeout) => new(stream, timeout);
/// <inheritdoc />
public Task ConnectAsync(CancellationToken ct) =>
// The ForTest seam is handed a pre-connected stream — nothing to dial.
_life is null ? Task.CompletedTask : _life.ConnectAsync(ct);
/// <inheritdoc />
public async Task<byte[]> SendAsync(byte unitId, byte[] pdu, CancellationToken ct)
{
if (_disposed) throw new ObjectDisposedException(nameof(ModbusRtuOverTcpTransport));
if (CurrentStream is null) throw new InvalidOperationException("Transport not connected");
// Single-flight: RTU carries no transaction id, so at most one transaction may be on the bus
// at a time — serialise every send/receive under the gate.
await _gate.WaitAsync(ct).ConfigureAwait(false);
try
{
// Proactive idle-disconnect (real socket only): if the socket has been quiet longer than
// the configured threshold, tear it down + reconnect before this PDU lands. Defends
// against silent NAT / firewall reaping.
if (_life is not null && _life.ShouldReconnectForIdle())
{
await _life.TearDownAsync().ConfigureAwait(false);
await _life.ConnectWithBackoffAsync(ct).ConfigureAwait(false);
}
try
{
var result = await SendOnceAsync(unitId, pdu, ct).ConfigureAwait(false);
_life?.MarkSuccess();
return result;
}
catch (Exception ex) when (_life is not null && _life.AutoReconnect && ModbusSocketLifecycle.IsSocketLevelFailure(ex))
{
// Mid-transaction drop: tear down the dead socket, reconnect (with backoff if
// configured), resend. Single retry — a second failure propagates so health/status
// reflect reality. Never reached in the ForTest seam (a raw injected stream has no
// lifecycle to reconnect).
await _life.TearDownAsync().ConfigureAwait(false);
await _life.ConnectWithBackoffAsync(ct).ConfigureAwait(false);
var result = await SendOnceAsync(unitId, pdu, ct).ConfigureAwait(false);
_life.MarkSuccess();
return result;
}
}
finally
{
_gate.Release();
}
}
/// <summary>The live stream: the lifecycle's <see cref="NetworkStream"/>, or the injected test stream.</summary>
private Stream? CurrentStream => _life?.Stream ?? _testStream;
/// <summary>
/// Executes exactly one RTU transaction on the current stream: build the ADU, write + flush,
/// read back one deframed response PDU. See the class remarks for the desync / timeout /
/// caller-cancel handling.
/// </summary>
private async Task<byte[]> SendOnceAsync(byte unitId, byte[] pdu, CancellationToken ct)
{
var stream = CurrentStream;
if (stream is null) throw new InvalidOperationException("Transport not connected");
// RTU ADU: [unitId] + PDU + CRC — no MBAP header, no TxId.
var adu = ModbusRtuFraming.BuildAdu(unitId, pdu);
using var cts = CancellationTokenSource.CreateLinkedTokenSource(ct);
cts.CancelAfter(_timeout);
try
{
await stream.WriteAsync(adu.AsMemory(), cts.Token).ConfigureAwait(false);
await stream.FlushAsync(cts.Token).ConfigureAwait(false);
// Framing owns the length-less RTU deframe + CRC + unit-address validation. A CRC-valid
// exception PDU throws ModbusException (socket still coherent — propagates, no teardown);
// truncation / CRC / unit-mismatch throw ModbusTransportDesyncException.
return await ModbusRtuFraming.ReadResponsePduAsync(stream, unitId, cts.Token).ConfigureAwait(false);
}
catch (ModbusTransportDesyncException)
{
// Framing violation: the stream is desynchronized — never reuse it.
if (_life is not null) await _life.TearDownAsync().ConfigureAwait(false);
throw;
}
catch (OperationCanceledException) when (!ct.IsCancellationRequested)
{
// Per-op timeout fired (NOT the caller). The response is unknown / may still land later
// and corrupt the next read — tear the socket down and normalise as a desync so the
// reconnect retry / status mapping engages.
if (_life is not null) await _life.TearDownAsync().ConfigureAwait(false);
throw new ModbusTransportDesyncException(
ModbusTransportDesyncException.DesyncReason.Timeout,
$"Modbus per-op response timeout after {_timeout.TotalMilliseconds:0}ms");
}
}
/// <summary>Asynchronously disposes the transport and underlying socket resources.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
public async ValueTask DisposeAsync()
{
if (_disposed) return;
_disposed = true;
if (_life is not null) await _life.DisposeAsync().ConfigureAwait(false);
_gate.Dispose();
}
}
@@ -0,0 +1,221 @@
using System.Net.Sockets;
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus;
/// <summary>
/// Owns the raw TCP socket lifecycle shared by every Modbus-over-TCP transport (MBAP and
/// RTU-over-TCP alike): the IPv4-preference connect, OS-level keep-alive, geometric-backoff
/// reconnect, idle-disconnect tracking, and teardown. It exposes the current
/// <see cref="NetworkStream"/> so the composing transport can drive its own framing on top —
/// the lifecycle knows nothing about MBAP / RTU wire layout.
/// </summary>
/// <remarks>
/// <para>
/// Extracted verbatim from <see cref="ModbusTcpTransport"/> so the connect / reconnect /
/// keep-alive / idle behaviour is written once and composed by both transports. The
/// constructor is connection-free — all socket I/O happens in <see cref="ConnectAsync"/> /
/// <see cref="ConnectWithBackoffAsync"/>.
/// </para>
/// <para>
/// Why keep-alive matters for DL205/DL260: the AutomationDirect H2-ECOM100 does NOT send
/// TCP keepalives per <c>docs/v2/dl205.md</c> §behavioral-oddities, so any NAT/firewall
/// between the gateway and PLC can silently close an idle socket after 2-5 minutes.
/// Enabling OS-level <c>SO_KEEPALIVE</c> lets the driver's own side detect a stuck socket
/// in reasonable time even when the application is mostly idle.
/// </para>
/// </remarks>
public sealed class ModbusSocketLifecycle : IAsyncDisposable
{
private readonly string _host;
private readonly int _port;
private readonly TimeSpan _timeout;
private readonly bool _autoReconnect;
private readonly ModbusKeepAliveOptions _keepAlive;
private readonly TimeSpan? _idleDisconnect;
private readonly ModbusReconnectOptions _reconnect;
private TcpClient? _client;
private NetworkStream? _stream;
private DateTime _lastSuccessUtc = DateTime.UtcNow;
/// <summary>Initializes a new instance of the <see cref="ModbusSocketLifecycle"/> class.</summary>
/// <param name="host">The host address or hostname of the Modbus server.</param>
/// <param name="port">The TCP port of the Modbus server.</param>
/// <param name="timeout">The timeout for socket operations.</param>
/// <param name="autoReconnect">Whether to automatically reconnect on socket failures.</param>
/// <param name="keepAlive">Optional keep-alive configuration for the socket.</param>
/// <param name="idleDisconnect">Optional duration after which an idle socket is disconnected.</param>
/// <param name="reconnect">Optional reconnect backoff configuration.</param>
public ModbusSocketLifecycle(
string host, int port, TimeSpan timeout, bool autoReconnect = true,
ModbusKeepAliveOptions? keepAlive = null,
TimeSpan? idleDisconnect = null,
ModbusReconnectOptions? reconnect = null)
{
_host = host;
_port = port;
_timeout = timeout;
_autoReconnect = autoReconnect;
_keepAlive = keepAlive ?? new ModbusKeepAliveOptions();
_idleDisconnect = idleDisconnect;
_reconnect = reconnect ?? new ModbusReconnectOptions();
}
/// <summary>Gets the current live <see cref="NetworkStream"/>, or <see langword="null"/> when not connected.</summary>
public NetworkStream? Stream => _stream;
/// <summary>Gets a value indicating whether auto-reconnect on socket failures is enabled.</summary>
public bool AutoReconnect => _autoReconnect;
/// <summary>Establishes a connection to the Modbus server, preferring IPv4.</summary>
/// <param name="ct">A cancellation token to observe for cancellation.</param>
/// <returns>A task representing the asynchronous connection operation.</returns>
public async Task ConnectAsync(CancellationToken ct)
{
// Resolve the host explicitly + prefer IPv4. .NET's TcpClient default-constructor is
// dual-stack (IPv6 first, fallback to IPv4) — but most Modbus TCP devices (PLCs and
// simulators like pymodbus) bind 0.0.0.0 only, so the IPv6 attempt times out and we
// burn the entire ConnectAsync budget before even trying IPv4. Resolving first +
// dialing the IPv4 address directly sidesteps that.
var addresses = await System.Net.Dns.GetHostAddressesAsync(_host, ct).ConfigureAwait(false);
var ipv4 = System.Linq.Enumerable.FirstOrDefault(addresses,
a => a.AddressFamily == System.Net.Sockets.AddressFamily.InterNetwork);
var target = ipv4 ?? (addresses.Length > 0 ? addresses[0] : System.Net.IPAddress.Loopback);
_client = new TcpClient(target.AddressFamily);
EnableKeepAlive(_client, _keepAlive);
using var cts = CancellationTokenSource.CreateLinkedTokenSource(ct);
cts.CancelAfter(_timeout);
await _client.ConnectAsync(target, _port, cts.Token).ConfigureAwait(false);
_stream = _client.GetStream();
_lastSuccessUtc = DateTime.UtcNow;
}
/// <summary>
/// Connect attempt with the configured geometric backoff. The first attempt fires after
/// <see cref="ModbusReconnectOptions.InitialDelay"/> (default zero — immediate); each
/// subsequent attempt sleeps for the previous delay times <c>BackoffMultiplier</c>,
/// capped at <c>MaxDelay</c>. Caller's cancellation token aborts the loop.
/// </summary>
/// <param name="ct">A cancellation token to observe for cancellation.</param>
/// <returns>A task representing the asynchronous reconnect operation.</returns>
public async Task ConnectWithBackoffAsync(CancellationToken ct)
{
var delay = _reconnect.InitialDelay;
var attempt = 0;
while (true)
{
if (delay > TimeSpan.Zero)
await Task.Delay(delay, ct).ConfigureAwait(false);
try
{
await ConnectAsync(ct).ConfigureAwait(false);
return;
}
catch (Exception ex) when (IsSocketLevelFailure(ex) && _autoReconnect)
{
attempt++;
// Geometric growth, capped. Use Math.Min on ticks so we don't overflow with
// pathological multipliers / long deployments.
var nextTicks = (long)(Math.Max(delay.Ticks, TimeSpan.FromMilliseconds(100).Ticks) * _reconnect.BackoffMultiplier);
delay = TimeSpan.FromTicks(Math.Min(nextTicks, _reconnect.MaxDelay.Ticks));
if (attempt >= 10)
{
// Bail after 10 attempts to surface persistent failure to the caller. With
// the default backoff (1s base, 2.0x mult, 30s cap) this is roughly 4 minutes
// of attempts; with InitialDelay=0 it's immediate up to the same cap.
throw;
}
}
}
}
/// <summary>
/// Reports whether the socket has been idle longer than the configured idle-disconnect
/// threshold and should be proactively torn down + reconnected before the next transaction.
/// Always <see langword="false"/> when no idle-disconnect timeout is configured.
/// </summary>
/// <returns><see langword="true"/> when the idle threshold has been exceeded.</returns>
public bool ShouldReconnectForIdle() =>
_idleDisconnect.HasValue && DateTime.UtcNow - _lastSuccessUtc > _idleDisconnect.Value;
/// <summary>Records that a transaction just succeeded, resetting the idle-disconnect clock.</summary>
public void MarkSuccess() => _lastSuccessUtc = DateTime.UtcNow;
/// <summary>Tears down the current socket + stream, leaving the lifecycle ready to reconnect.</summary>
/// <returns>A task representing the asynchronous teardown operation.</returns>
public async Task TearDownAsync()
{
try { if (_stream is not null) await _stream.DisposeAsync().ConfigureAwait(false); }
catch { /* best-effort */ }
_stream = null;
try { _client?.Dispose(); } catch { }
_client = null;
}
/// <summary>
/// Enable SO_KEEPALIVE with aggressive probe timing. DL205/DL260 doesn't send keepalives
/// itself; having the OS probe the socket every ~30s lets the driver notice a dead PLC
/// or broken NAT path long before the default 2-hour Windows idle timeout fires.
/// Non-fatal if the underlying OS rejects the option (some older Linux / container
/// sandboxes don't expose the fine-grained timing levers — the driver still works,
/// application-level probe still detects problems).
/// </summary>
private static void EnableKeepAlive(TcpClient client, ModbusKeepAliveOptions opts)
{
if (!opts.Enabled) return;
try
{
client.Client.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.KeepAlive, true);
// A TimeSpan < 1s previously truncated to 0 via the int cast,
// which Windows / Linux interpret as "use the default" — silently defeating the
// configured keep-alive timing. Round up to at least 1 second so a sub-second
// configuration still produces a real keep-alive cadence. Negative values are
// also clamped to 1 to avoid surfacing as OS errors.
client.Client.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.TcpKeepAliveTime,
ClampToWholeSeconds(opts.Time));
client.Client.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.TcpKeepAliveInterval,
ClampToWholeSeconds(opts.Interval));
client.Client.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.TcpKeepAliveRetryCount, opts.RetryCount);
}
catch { /* best-effort; older OSes may not expose the granular knobs */ }
}
/// <summary>
/// Cast a <see cref="TimeSpan"/> to a whole number of seconds with a
/// minimum of 1 — protects callers from the int-cast truncation that turned 500 ms
/// keep-alive timing into "use the default" on most OSes.
/// </summary>
/// <param name="ts">The timespan to clamp to whole seconds.</param>
/// <returns>The clamped duration expressed as a whole number of seconds, never less than 1.</returns>
internal static int ClampToWholeSeconds(TimeSpan ts)
{
var seconds = (int)Math.Ceiling(ts.TotalSeconds);
return seconds < 1 ? 1 : seconds;
}
/// <summary>
/// Distinguish socket-layer failures (eligible for reconnect-and-retry) from
/// protocol-layer failures (must propagate — retrying the same PDU won't help if the
/// PLC just returned exception 02 Illegal Data Address).
/// </summary>
/// <param name="ex">The exception to classify.</param>
/// <returns><see langword="true"/> when the exception is a socket-level failure.</returns>
internal static bool IsSocketLevelFailure(Exception ex) =>
ex is EndOfStreamException
|| ex is IOException
|| ex is SocketException
|| ex is ObjectDisposedException;
/// <summary>Asynchronously disposes the underlying socket + stream resources.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
public async ValueTask DisposeAsync()
{
try
{
if (_stream is not null) await _stream.DisposeAsync().ConfigureAwait(false);
}
catch { /* best-effort */ }
_client?.Dispose();
}
}
@@ -3,13 +3,19 @@ using System.Net.Sockets;
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus;
/// <summary>
/// Concrete Modbus TCP transport. Wraps a single <see cref="TcpClient"/> and serializes
/// requests so at most one transaction is in-flight at a time — Modbus servers typically
/// support concurrent transactions, but the single-flight model keeps the wire trace
/// Concrete Modbus TCP transport. Wraps a single socket (owned by <see cref="ModbusSocketLifecycle"/>)
/// and serializes requests so at most one transaction is in-flight at a time — Modbus servers
/// typically support concurrent transactions, but the single-flight model keeps the wire trace
/// easy to diagnose and avoids interleaved-response correlation bugs.
/// </summary>
/// <remarks>
/// <para>
/// Owns the MBAP framing (<see cref="SendOnceAsync"/>: 7-byte header + transaction-id
/// pairing) and composes <see cref="ModbusSocketLifecycle"/> for the connect / reconnect /
/// keep-alive / idle-disconnect machinery — the same lifecycle the RTU-over-TCP transport
/// reuses with different framing.
/// </para>
/// <para>
/// Survives mid-transaction socket drops: when a send/read fails with a socket-level
/// error (<see cref="IOException"/>, <see cref="SocketException"/>, <see cref="EndOfStreamException"/>)
/// the transport disposes the dead socket, reconnects, and retries the PDU exactly
@@ -26,19 +32,11 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus;
/// </remarks>
public sealed class ModbusTcpTransport : IModbusTransport
{
private readonly string _host;
private readonly int _port;
private readonly ModbusSocketLifecycle _life;
private readonly TimeSpan _timeout;
private readonly bool _autoReconnect;
private readonly ModbusKeepAliveOptions _keepAlive;
private readonly TimeSpan? _idleDisconnect;
private readonly ModbusReconnectOptions _reconnect;
private readonly SemaphoreSlim _gate = new(1, 1);
private TcpClient? _client;
private NetworkStream? _stream;
private ushort _nextTx;
private bool _disposed;
private DateTime _lastSuccessUtc = DateTime.UtcNow;
/// <summary>Initializes a new instance of the <see cref="ModbusTcpTransport"/> class.</summary>
/// <param name="host">The host address or hostname of the Modbus server.</param>
@@ -54,84 +52,18 @@ public sealed class ModbusTcpTransport : IModbusTransport
TimeSpan? idleDisconnect = null,
ModbusReconnectOptions? reconnect = null)
{
_host = host;
_port = port;
_timeout = timeout;
_autoReconnect = autoReconnect;
_keepAlive = keepAlive ?? new ModbusKeepAliveOptions();
_idleDisconnect = idleDisconnect;
_reconnect = reconnect ?? new ModbusReconnectOptions();
_life = new ModbusSocketLifecycle(host, port, timeout, autoReconnect, keepAlive, idleDisconnect, reconnect);
}
/// <inheritdoc />
public async Task ConnectAsync(CancellationToken ct)
{
// Resolve the host explicitly + prefer IPv4. .NET's TcpClient default-constructor is
// dual-stack (IPv6 first, fallback to IPv4) — but most Modbus TCP devices (PLCs and
// simulators like pymodbus) bind 0.0.0.0 only, so the IPv6 attempt times out and we
// burn the entire ConnectAsync budget before even trying IPv4. Resolving first +
// dialing the IPv4 address directly sidesteps that.
var addresses = await System.Net.Dns.GetHostAddressesAsync(_host, ct).ConfigureAwait(false);
var ipv4 = System.Linq.Enumerable.FirstOrDefault(addresses,
a => a.AddressFamily == System.Net.Sockets.AddressFamily.InterNetwork);
var target = ipv4 ?? (addresses.Length > 0 ? addresses[0] : System.Net.IPAddress.Loopback);
_client = new TcpClient(target.AddressFamily);
EnableKeepAlive(_client, _keepAlive);
using var cts = CancellationTokenSource.CreateLinkedTokenSource(ct);
cts.CancelAfter(_timeout);
await _client.ConnectAsync(target, _port, cts.Token).ConfigureAwait(false);
_stream = _client.GetStream();
_lastSuccessUtc = DateTime.UtcNow;
}
/// <summary>
/// Enable SO_KEEPALIVE with aggressive probe timing. DL205/DL260 doesn't send keepalives
/// itself; having the OS probe the socket every ~30s lets the driver notice a dead PLC
/// or broken NAT path long before the default 2-hour Windows idle timeout fires.
/// Non-fatal if the underlying OS rejects the option (some older Linux / container
/// sandboxes don't expose the fine-grained timing levers — the driver still works,
/// application-level probe still detects problems).
/// </summary>
private static void EnableKeepAlive(TcpClient client, ModbusKeepAliveOptions opts)
{
if (!opts.Enabled) return;
try
{
client.Client.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.KeepAlive, true);
// A TimeSpan < 1s previously truncated to 0 via the int cast,
// which Windows / Linux interpret as "use the default" — silently defeating the
// configured keep-alive timing. Round up to at least 1 second so a sub-second
// configuration still produces a real keep-alive cadence. Negative values are
// also clamped to 1 to avoid surfacing as OS errors.
client.Client.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.TcpKeepAliveTime,
ClampToWholeSeconds(opts.Time));
client.Client.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.TcpKeepAliveInterval,
ClampToWholeSeconds(opts.Interval));
client.Client.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.TcpKeepAliveRetryCount, opts.RetryCount);
}
catch { /* best-effort; older OSes may not expose the granular knobs */ }
}
/// <summary>
/// Cast a <see cref="TimeSpan"/> to a whole number of seconds with a
/// minimum of 1 — protects callers from the int-cast truncation that turned 500 ms
/// keep-alive timing into "use the default" on most OSes.
/// </summary>
/// <param name="ts">The timespan to clamp to whole seconds.</param>
/// <returns>The clamped duration expressed as a whole number of seconds, never less than 1.</returns>
internal static int ClampToWholeSeconds(TimeSpan ts)
{
var seconds = (int)Math.Ceiling(ts.TotalSeconds);
return seconds < 1 ? 1 : seconds;
}
public Task ConnectAsync(CancellationToken ct) => _life.ConnectAsync(ct);
/// <inheritdoc />
public async Task<byte[]> SendAsync(byte unitId, byte[] pdu, CancellationToken ct)
{
if (_disposed) throw new ObjectDisposedException(nameof(ModbusTcpTransport));
if (_stream is null) throw new InvalidOperationException("Transport not connected");
if (_life.Stream is null) throw new InvalidOperationException("Transport not connected");
await _gate.WaitAsync(ct).ConfigureAwait(false);
try
@@ -140,27 +72,27 @@ public sealed class ModbusTcpTransport : IModbusTransport
// threshold, tear it down + reconnect before this PDU lands. Defends against silent
// NAT / firewall reaping where the socket looks alive locally but the upstream side
// dropped it minutes ago.
if (_idleDisconnect.HasValue && DateTime.UtcNow - _lastSuccessUtc > _idleDisconnect.Value)
if (_life.ShouldReconnectForIdle())
{
await TearDownAsync().ConfigureAwait(false);
await ConnectWithBackoffAsync(ct).ConfigureAwait(false);
await _life.TearDownAsync().ConfigureAwait(false);
await _life.ConnectWithBackoffAsync(ct).ConfigureAwait(false);
}
try
{
var result = await SendOnceAsync(unitId, pdu, ct).ConfigureAwait(false);
_lastSuccessUtc = DateTime.UtcNow;
_life.MarkSuccess();
return result;
}
catch (Exception ex) when (_autoReconnect && IsSocketLevelFailure(ex))
catch (Exception ex) when (_life.AutoReconnect && ModbusSocketLifecycle.IsSocketLevelFailure(ex))
{
// Mid-transaction drop: tear down the dead socket, reconnect (with backoff if
// configured), resend. Single retry — if it fails again, let it propagate so
// health/status reflect reality.
await TearDownAsync().ConfigureAwait(false);
await ConnectWithBackoffAsync(ct).ConfigureAwait(false);
await _life.TearDownAsync().ConfigureAwait(false);
await _life.ConnectWithBackoffAsync(ct).ConfigureAwait(false);
var result = await SendOnceAsync(unitId, pdu, ct).ConfigureAwait(false);
_lastSuccessUtc = DateTime.UtcNow;
_life.MarkSuccess();
return result;
}
}
@@ -170,43 +102,6 @@ public sealed class ModbusTcpTransport : IModbusTransport
}
}
/// <summary>
/// Connect attempt with the configured geometric backoff. The first attempt fires after
/// <see cref="ModbusReconnectOptions.InitialDelay"/> (default zero — immediate); each
/// subsequent attempt sleeps for the previous delay times <c>BackoffMultiplier</c>,
/// capped at <c>MaxDelay</c>. Caller's cancellation token aborts the loop.
/// </summary>
private async Task ConnectWithBackoffAsync(CancellationToken ct)
{
var delay = _reconnect.InitialDelay;
var attempt = 0;
while (true)
{
if (delay > TimeSpan.Zero)
await Task.Delay(delay, ct).ConfigureAwait(false);
try
{
await ConnectAsync(ct).ConfigureAwait(false);
return;
}
catch (Exception ex) when (IsSocketLevelFailure(ex) && _autoReconnect)
{
attempt++;
// Geometric growth, capped. Use Math.Min on ticks so we don't overflow with
// pathological multipliers / long deployments.
var nextTicks = (long)(Math.Max(delay.Ticks, TimeSpan.FromMilliseconds(100).Ticks) * _reconnect.BackoffMultiplier);
delay = TimeSpan.FromTicks(Math.Min(nextTicks, _reconnect.MaxDelay.Ticks));
if (attempt >= 10)
{
// Bail after 10 attempts to surface persistent failure to the caller. With
// the default backoff (1s base, 2.0x mult, 30s cap) this is roughly 4 minutes
// of attempts; with InitialDelay=0 it's immediate up to the same cap.
throw;
}
}
}
}
/// <summary>
/// Executes exactly one Modbus transaction on the current socket.
/// </summary>
@@ -230,7 +125,8 @@ public sealed class ModbusTcpTransport : IModbusTransport
/// </remarks>
private async Task<byte[]> SendOnceAsync(byte unitId, byte[] pdu, CancellationToken ct)
{
if (_stream is null) throw new InvalidOperationException("Transport not connected");
var stream = _life.Stream;
if (stream is null) throw new InvalidOperationException("Transport not connected");
var txId = ++_nextTx;
// MBAP: [TxId(2)][Proto=0(2)][Length(2)][UnitId(1)] + PDU
@@ -248,11 +144,11 @@ public sealed class ModbusTcpTransport : IModbusTransport
cts.CancelAfter(_timeout);
try
{
await _stream.WriteAsync(adu.AsMemory(), cts.Token).ConfigureAwait(false);
await _stream.FlushAsync(cts.Token).ConfigureAwait(false);
await stream.WriteAsync(adu.AsMemory(), cts.Token).ConfigureAwait(false);
await stream.FlushAsync(cts.Token).ConfigureAwait(false);
var header = new byte[7];
await ReadExactlyAsync(_stream, header, cts.Token).ConfigureAwait(false);
await ReadExactlyAsync(stream, header, cts.Token).ConfigureAwait(false);
var respTxId = (ushort)((header[0] << 8) | header[1]);
if (respTxId != txId)
throw new ModbusTransportDesyncException(
@@ -264,7 +160,7 @@ public sealed class ModbusTcpTransport : IModbusTransport
ModbusTransportDesyncException.DesyncReason.TruncatedFrame,
$"Modbus response length too small: {respLen}");
var respPdu = new byte[respLen - 1];
await ReadExactlyAsync(_stream, respPdu, cts.Token).ConfigureAwait(false);
await ReadExactlyAsync(stream, respPdu, cts.Token).ConfigureAwait(false);
// Exception PDU: function code has high bit set. This is a well-formed protocol-level
// error — the socket is still coherent, so it MUST propagate without teardown.
@@ -280,7 +176,7 @@ public sealed class ModbusTcpTransport : IModbusTransport
catch (ModbusTransportDesyncException)
{
// Framing violation: the socket is desynchronized — never reuse it.
await TearDownAsync().ConfigureAwait(false);
await _life.TearDownAsync().ConfigureAwait(false);
throw;
}
catch (OperationCanceledException) when (!ct.IsCancellationRequested)
@@ -288,33 +184,13 @@ public sealed class ModbusTcpTransport : IModbusTransport
// Per-op timeout fired (NOT the caller). The response is unknown / may still land later
// and corrupt the next read — tear the socket down and normalize as a desync so the
// reconnect retry / status mapping engages.
await TearDownAsync().ConfigureAwait(false);
await _life.TearDownAsync().ConfigureAwait(false);
throw new ModbusTransportDesyncException(
ModbusTransportDesyncException.DesyncReason.Timeout,
$"Modbus per-op response timeout after {_timeout.TotalMilliseconds:0}ms");
}
}
/// <summary>
/// Distinguish socket-layer failures (eligible for reconnect-and-retry) from
/// protocol-layer failures (must propagate — retrying the same PDU won't help if the
/// PLC just returned exception 02 Illegal Data Address).
/// </summary>
private static bool IsSocketLevelFailure(Exception ex) =>
ex is EndOfStreamException
|| ex is IOException
|| ex is SocketException
|| ex is ObjectDisposedException;
private async Task TearDownAsync()
{
try { if (_stream is not null) await _stream.DisposeAsync().ConfigureAwait(false); }
catch { /* best-effort */ }
_stream = null;
try { _client?.Dispose(); } catch { }
_client = null;
}
private static async Task ReadExactlyAsync(Stream s, byte[] buf, CancellationToken ct)
{
var read = 0;
@@ -332,12 +208,7 @@ public sealed class ModbusTcpTransport : IModbusTransport
{
if (_disposed) return;
_disposed = true;
try
{
if (_stream is not null) await _stream.DisposeAsync().ConfigureAwait(false);
}
catch { /* best-effort */ }
_client?.Dispose();
await _life.DisposeAsync().ConfigureAwait(false);
_gate.Dispose();
}
}
@@ -1,9 +1,10 @@
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus;
/// <summary>
/// Raised when a Modbus TCP transaction leaves the single-flight socket in an unknown /
/// Raised when a Modbus transaction leaves the single-flight socket in an unknown /
/// desynchronized state — a per-op response timeout, or a framing violation (transaction-id
/// mismatch or truncated MBAP length). Unlike a Modbus <em>exception PDU</em> (a well-formed
/// mismatch, truncated frame, RTU CRC mismatch, or RTU unit-address mismatch). Unlike a Modbus
/// <em>exception PDU</em> (a well-formed
/// protocol-level error where the socket is still coherent and the request must simply
/// propagate), a desync means bytes may still be in flight or half-read, so the socket is
/// unusable and MUST be torn down before this is thrown.
@@ -27,8 +28,20 @@ public sealed class ModbusTransportDesyncException : IOException
/// <summary>The response MBAP transaction id did not match the request's.</summary>
TxIdMismatch,
/// <summary>The response MBAP length field was below the mandatory minimum (truncated frame).</summary>
/// <summary>
/// The frame ended before it was complete — the MBAP length field was below the mandatory
/// minimum (TCP), or the stream closed mid-frame while reading a length-less RTU frame.
/// </summary>
TruncatedFrame,
/// <summary>The RTU frame's trailing CRC-16 did not validate over the received bytes.</summary>
CrcMismatch,
/// <summary>
/// A CRC-valid RTU frame carried a slave/unit address other than the one addressed — on an
/// RS-485 multi-drop bus, a reply from the wrong slave.
/// </summary>
UnitMismatch,
}
/// <summary>Gets the reason the socket was classified desynchronized.</summary>
@@ -0,0 +1,43 @@
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus;
/// <summary>
/// Single mapping point from <see cref="ModbusDriverOptions"/> to the concrete
/// <see cref="IModbusTransport"/> its <see cref="ModbusDriverOptions.Transport"/> selects.
/// Consumed by both <see cref="ModbusDriver"/>'s default transport-factory closure and the
/// AdminUI Test-Connect probe, so the <see cref="ModbusTransportMode"/> switch lives in
/// exactly one place.
/// </summary>
public static class ModbusTransportFactory
{
/// <summary>
/// Builds the transport <paramref name="options"/>.<see cref="ModbusDriverOptions.Transport"/>
/// selects, threading every shared connection setting (Host/Port/Timeout/AutoReconnect/
/// KeepAlive/IdleDisconnectTimeout/Reconnect) through to whichever concrete transport is built.
/// </summary>
/// <param name="options">Driver configuration options.</param>
/// <returns>A <see cref="ModbusRtuOverTcpTransport"/> when <see cref="ModbusTransportMode.RtuOverTcp"/>
/// is selected; a <see cref="ModbusTcpTransport"/> when <see cref="ModbusTransportMode.Tcp"/> is selected.</returns>
/// <exception cref="ArgumentOutOfRangeException">
/// <paramref name="options"/>.<see cref="ModbusDriverOptions.Transport"/> is not a recognized
/// <see cref="ModbusTransportMode"/> member — fail loudly rather than silently falling back to TCP/MBAP.
/// </exception>
public static IModbusTransport Create(ModbusDriverOptions options)
{
ArgumentNullException.ThrowIfNull(options);
return options.Transport switch
{
ModbusTransportMode.RtuOverTcp => new ModbusRtuOverTcpTransport(
options.Host, options.Port, options.Timeout, options.AutoReconnect,
keepAlive: options.KeepAlive,
idleDisconnect: options.IdleDisconnectTimeout,
reconnect: options.Reconnect),
ModbusTransportMode.Tcp => new ModbusTcpTransport(
options.Host, options.Port, options.Timeout, options.AutoReconnect,
keepAlive: options.KeepAlive,
idleDisconnect: options.IdleDisconnectTimeout,
reconnect: options.Reconnect),
_ => throw new ArgumentOutOfRangeException(
nameof(options), options.Transport, "Unknown Modbus transport mode."),
};
}
}
@@ -68,10 +68,10 @@ else
</div>
<p class="text-muted small mt-2 mb-0">
Read live from cluster state on this node. <strong>Mesh-wide scope:</strong> the Primary is
elected once per Akka mesh, not per cluster row — in the current single-mesh topology this
acts on the whole mesh's Primary, which may be a node of another
<span class="mono">Cluster</span>. See <span class="mono">docs/Redundancy.md</span>.
Read live from cluster state on this node. The Primary shown is
<strong>this application <span class="mono">Cluster</span>'s own Primary</strong> — since
the per-cluster mesh split, each cluster's redundant pair runs its own independent 2-node
Akka mesh and elects its own Primary. See <span class="mono">docs/Redundancy.md</span>.
</p>
<AuthorizeView Policy="@ManualFailoverPageModel.RequiredPolicy">
@@ -27,6 +27,16 @@
}
</InputSelect>
</div>
<div class="col-md-3">
<label class="form-label">Transport</label>
<InputSelect @bind-Value="_form.Transport" @bind-Value:after="EmitAsync" class="form-select form-select-sm">
@foreach (var e in Enum.GetValues<ModbusTransportMode>())
{
<option value="@e">@e</option>
}
</InputSelect>
<div class="form-text">RtuOverTcp = talk raw RTU frames to a serial→Ethernet gateway; must match the gateway's mode.</div>
</div>
<div class="col-md-3">
<label class="form-label">MELSEC sub-family</label>
<InputSelect @bind-Value="_form.MelsecSubFamily" @bind-Value:after="EmitAsync" class="form-select form-select-sm">
@@ -290,6 +300,9 @@
public ModbusFamily Family { get; set; } = ModbusFamily.Generic;
public MelsecFamily MelsecSubFamily { get; set; } = MelsecFamily.Q_L_iQR;
// Wire transport (Tcp = Modbus/TCP MBAP; RtuOverTcp = RTU framing over a serial→Ethernet gateway)
public ModbusTransportMode Transport { get; set; } = ModbusTransportMode.Tcp;
// Transport flags
public bool AutoReconnect { get; set; } = true;
public int IdleDisconnectTimeoutSeconds { get; set; } = 0;
@@ -337,6 +350,7 @@
TimeoutSeconds = (int)o.Timeout.TotalSeconds,
Family = o.Family,
MelsecSubFamily = o.MelsecSubFamily,
Transport = o.Transport,
AutoReconnect = o.AutoReconnect,
IdleDisconnectTimeoutSeconds = o.IdleDisconnectTimeout.HasValue ? (int)o.IdleDisconnectTimeout.Value.TotalSeconds : 0,
MaxRegistersPerRead = o.MaxRegistersPerRead,
@@ -387,6 +401,7 @@
MaxReadGap = (ushort)Math.Clamp(MaxReadGap, 0, 65535),
Family = Family,
MelsecSubFamily = MelsecSubFamily,
Transport = Transport,
WriteOnChangeOnly = WriteOnChangeOnly,
AutoReconnect = AutoReconnect,
KeepAlive = new ModbusKeepAliveOptions
@@ -25,8 +25,11 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
/// rotation would fail silently against the other.
/// </para>
/// <para>
/// <b>Exactly one ClusterClient, fleet-wide.</b> See <see cref="RebuildClient"/> — this is
/// the single most important thing to understand before changing this actor.
/// <b>One ClusterClient per application <c>Cluster</c> (Phase 6).</b> The fleet is now one
/// Akka mesh per cluster, so a single fleet-wide client's <c>SendToAll</c> would reach only
/// the one mesh whose receptionist answered. This actor therefore holds a client per cluster
/// and fans <c>SendToAll</c> across all of them. See <see cref="RebuildClient"/> — this is the
/// single most important thing to understand before changing this actor.
/// </para>
/// </remarks>
public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
@@ -40,8 +43,12 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
private readonly ILoggingAdapter _log = Context.GetLogger();
private readonly bool _useClusterClient;
private IActorRef? _client;
private ImmutableHashSet<string> _currentContacts = ImmutableHashSet<string>.Empty;
// One client per application Cluster, keyed by ClusterId. A ClusterClient reaches only the mesh
// whose receptionist it dialled, and after Phase 6 every cluster is its own mesh — so central must
// hold a client per cluster and fan SendToAll across all of them.
private readonly Dictionary<string, (IActorRef Client, ImmutableHashSet<string> Contacts)> _clients =
new();
private CancellationTokenSource _lifecycle = new();
/// <summary>Gets the timer scheduler driving the periodic contact refresh.</summary>
@@ -121,6 +128,9 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
/// <inheritdoc />
protected override void PostStop()
{
foreach (var (client, _) in _clients.Values) Context.Stop(client);
_clients.Clear();
_lifecycle.Cancel();
_lifecycle.Dispose();
}
@@ -133,7 +143,7 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
return;
}
if (_client is null)
if (_clients.Count == 0)
{
// App-level drop, matching the sister project's decision and this repo's buffer-size = 0.
// The command is NOT queued and will NOT be retried: a deploy fails at the coordinator's
@@ -141,19 +151,21 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
// command arriving minutes late against a config the DB already recorded as failed.
_log.Warning(
"No mesh ClusterClient — dropping {MessageType}. The last contact refresh produced "
+ "no usable contact point from the enabled, non-maintenance ClusterNode rows; the "
+ "command is not buffered and will not be retried",
+ "no usable contact point from the enabled, non-maintenance ClusterNode rows in any "
+ "cluster; the command is not buffered and will not be retried",
cmd.Message.GetType().Name);
return;
}
// SendToAll, NEVER Send. Today's DPS publish reaches EVERY DriverHostActor — there is no
// ClusterId or node filter on the node side at all; scoping happens later, inside the
// artifact (DeploymentArtifact.ResolveClusterScope). ClusterClient.Send delivers to exactly
// ONE registered actor, which would deploy to a single node of the fleet while every other
// node silently kept its old configuration — and the deployment would still seal green if
// the ack set happened to be satisfied.
_client.Tell(new ClusterClient.SendToAll(MeshPaths.NodeCommunication, cmd.Message));
// SendToAll, NEVER Send, fanned across EVERY cluster's client. Today's DPS publish reaches
// EVERY DriverHostActor — there is no ClusterId or node filter on the node side at all; scoping
// happens later, inside the artifact (DeploymentArtifact.ResolveClusterScope). Each ClusterClient
// reaches only the one mesh it dialled, so central must fan across all of them to preserve the
// fleet-wide reach DPS gives. ClusterClient.Send delivers to exactly ONE registered actor, which
// would deploy to a single node while every other node silently kept its old configuration — and
// the deployment would still seal green if the ack set happened to be satisfied.
foreach (var (client, _) in _clients.Values)
client.Tell(new ClusterClient.SendToAll(MeshPaths.NodeCommunication, cmd.Message));
}
private void HandleApplyAck(ApplyAck ack)
@@ -198,11 +210,14 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
var rows = await db.ClusterNodes
.AsNoTracking()
.Where(n => n.Enabled && !n.MaintenanceMode)
.Select(n => new { n.NodeId, n.Host, n.AkkaPort })
.Select(n => new { n.NodeId, n.ClusterId, n.Host, n.AkkaPort })
.ToListAsync(ct)
.ConfigureAwait(false);
var contacts = new List<string>(rows.Count);
// Grouped BY ClusterId: central now dials one ClusterClient per application Cluster, so the
// contact set must be partitioned the same way. A row's contacts join only its own cluster's
// client.
var byCluster = new Dictionary<string, List<string>>();
var malformed = new List<string>();
foreach (var row in rows)
{
@@ -212,11 +227,21 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
// not abort the whole refresh and leave the contact set half-built — the sister
// project shipped that bug and its regression test deliberately orders the bad row
// first.
if (ActorPath.TryParse(address, out _)) contacts.Add(address);
else malformed.Add($"{row.NodeId} -> {address}");
if (ActorPath.TryParse(address, out _))
{
if (!byCluster.TryGetValue(row.ClusterId, out var list))
byCluster[row.ClusterId] = list = new List<string>();
list.Add(address);
}
else
{
malformed.Add($"{row.NodeId} ({row.ClusterId}) -> {address}");
}
}
return new ContactsLoaded(contacts, malformed);
var contactsByCluster = byCluster.ToDictionary(
kv => kv.Key, kv => (IReadOnlyList<string>)kv.Value);
return new ContactsLoaded(contactsByCluster, malformed);
}, ct).PipeTo(self);
}
@@ -229,70 +254,111 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
+ "refresh (other nodes are unaffected). Check the row's Host and AkkaPort", bad);
}
var contacts = msg.Contacts.ToImmutableHashSet();
var byCluster = msg.ContactsByCluster.ToDictionary(
kv => kv.Key, kv => kv.Value.ToImmutableHashSet());
if (contacts.IsEmpty)
// Clusters that dropped out of the DB entirely, or whose only rows were malformed this refresh:
// stop and forget their clients so a vanished cluster stops receiving. LoadContactsFromDb only
// creates a ByCluster key when at least one of its rows parsed, so an all-malformed cluster has
// NO key at all and is caught here by the absent-key (!TryGetValue) arm — not by an empty value.
var gone = _clients.Keys
.Where(id => !byCluster.TryGetValue(id, out var c) || c.IsEmpty)
.ToList();
foreach (var clusterId in gone)
{
_log.Warning(
"No usable ClusterNode contact points; the mesh ClusterClient was not created and "
+ "every central→node command will be dropped until a refresh finds one");
return;
_log.Info(
"Cluster {ClusterId} no longer yields any usable contact point; stopping its "
+ "ClusterClient", clusterId);
Context.Stop(_clients[clusterId].Client);
_clients.Remove(clusterId);
}
if (_client is not null && _currentContacts.SetEquals(contacts)) return;
var anyUsable = false;
foreach (var (clusterId, contacts) in byCluster)
{
if (contacts.IsEmpty)
{
// Defensive: LoadContactsFromDb never emits an empty-but-present cluster (a key is
// created only on a first successfully-parsed row), so this arm is unreachable for the
// DB producer — an all-malformed cluster is handled by the absent-key branch above. It
// guards a ContactsLoaded (a public record) constructed elsewhere with an empty list.
_log.Warning(
"Cluster {ClusterId} has no usable ClusterNode contact points; no ClusterClient "
+ "was created for it and its nodes' commands will be dropped until a refresh finds "
+ "one", clusterId);
continue;
}
RebuildClient(contacts);
anyUsable = true;
if (_clients.TryGetValue(clusterId, out var existing) && existing.Contacts.SetEquals(contacts))
continue;
RebuildClient(clusterId, contacts);
}
if (!anyUsable && _clients.Count == 0)
{
_log.Warning(
"No usable ClusterNode contact points in any cluster; no mesh ClusterClient was "
+ "created and every central→node command will be dropped until a refresh finds one");
}
}
/// <summary>Stops any existing client and creates a replacement for the new contact set.</summary>
/// <summary>Stops a cluster's existing client and creates a replacement for its new contact set.</summary>
/// <remarks>
/// <para>
/// <b>TODO(Phase 6): one client per application <c>Cluster</c>.</b> Today this actor
/// creates exactly <b>one</b> client whose contacts are the whole fleet, and that is not
/// a compromise — it is the only correct shape while the fleet is a single Akka mesh.
/// <b>One client per application <c>Cluster</c> (delivered in Phase 6).</b> This actor now
/// holds one <c>ClusterClient</c> per cluster and <see cref="HandleMeshCommand"/> fans
/// <c>SendToAll</c> across all of them. Once the fleet is split into one Akka mesh per
/// cluster, a <c>ClusterClientReceptionist</c> serves only <i>its own</i> mesh, so a single
/// client's <c>SendToAll</c> would reach the one mesh whose receptionist answered and leave
/// every other cluster silently on its old configuration. Fanning across a client per
/// cluster restores the fleet-wide reach the pre-split DPS broadcast gave.
/// </para>
/// <para>
/// A <c>ClusterClientReceptionist</c> serves its entire cluster, so
/// <c>SendToAll</c> reaches every registered node-comm actor in the mesh <i>regardless of
/// which node's address was used as the contact point</i>. One client per Cluster would
/// therefore fan each command out once per cluster — N× duplicate
/// <c>DispatchDeployment</c> and N× duplicate <c>ApplyAck</c>. Phase 6 splits the meshes,
/// at which point per-cluster clients become both correct and necessary.
/// Why this is not N× duplicate dispatch: each client dials only its own cluster's
/// receptionists, so a command is delivered <b>once per mesh</b>, not once per cluster into
/// every mesh. (While the fleet was a single mesh, the same code would have duplicated —
/// which is exactly why the fan-out could not land until the meshes were partitioned.)
/// </para>
/// <para>
/// Corollary worth stating because it is easy to assume otherwise: <b>the contact set
/// does not scope delivery.</b> Excluding a maintenance-mode node from the contacts does
/// not stop it receiving commands — it still receives them, exactly as it does under
/// today's DPS broadcast. The filter is about which receptionists are worth <i>dialling</i>
/// (a node in maintenance may be switched off), not about who gets the message.
/// does not scope delivery within a mesh.</b> Excluding a maintenance-mode node from a
/// cluster's contacts does not stop it receiving commands — its receptionist still delivers
/// to every registered node-comm actor in that mesh. The filter is about which receptionists
/// are worth <i>dialling</i> (a node in maintenance may be switched off), not about who gets
/// the message.
/// </para>
/// </remarks>
/// <param name="contacts">The receptionist addresses to dial.</param>
private void RebuildClient(ImmutableHashSet<string> contacts)
/// <param name="clusterId">The application <c>Cluster</c> whose client is being (re)built.</param>
/// <param name="contacts">The receptionist addresses to dial for that cluster.</param>
private void RebuildClient(string clusterId, ImmutableHashSet<string> contacts)
{
if (_client is not null)
if (_clients.TryGetValue(clusterId, out var existing))
{
_log.Info("Mesh contact set changed; replacing the ClusterClient");
Context.Stop(_client);
_log.Info("Contact set for cluster {ClusterId} changed; replacing its ClusterClient", clusterId);
Context.Stop(existing.Client);
// Cleared now: if the create below throws, a stale ref would route commands into a
// Removed now: if the create below throws, a stale ref would route commands into a
// stopping actor and they would vanish without a warning.
_client = null;
_currentContacts = ImmutableHashSet<string>.Empty;
_clients.Remove(clusterId);
}
try
{
var paths = contacts.Select(ActorPath.Parse).ToImmutableHashSet();
_client = _clientFactory.Create(Context.System, paths);
_currentContacts = contacts;
_log.Info("Mesh ClusterClient created with {Count} contact point(s)", contacts.Count);
var client = _clientFactory.Create(Context.System, clusterId, paths);
_clients[clusterId] = (client, contacts);
_log.Info(
"Mesh ClusterClient for cluster {ClusterId} created with {Count} contact point(s)",
clusterId, contacts.Count);
}
catch (Exception ex)
{
_log.Error(ex,
"Failed to create the mesh ClusterClient; central→node commands are dropped until "
+ "the next contact refresh succeeds");
"Failed to create the mesh ClusterClient for cluster {ClusterId}; its nodes' commands "
+ "are dropped until the next contact refresh succeeds", clusterId);
}
}
@@ -300,9 +366,12 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
public sealed record RefreshContacts;
/// <summary>Result of a contact-point load, piped back from the DB read.</summary>
/// <param name="Contacts">Receptionist addresses that parsed cleanly.</param>
/// <param name="Malformed">Rows that did not, described for the log.</param>
/// <param name="ContactsByCluster">
/// Receptionist addresses that parsed cleanly, grouped by the owning application
/// <c>Cluster</c>'s id. One <c>ClusterClient</c> is built per key.
/// </param>
/// <param name="Malformed">Rows that did not parse, described for the log.</param>
public sealed record ContactsLoaded(
IReadOnlyList<string> Contacts,
IReadOnlyDictionary<string, IReadOnlyList<string>> ContactsByCluster,
IReadOnlyList<string> Malformed);
}
@@ -13,9 +13,14 @@ public interface IMeshClusterClientFactory
{
/// <summary>Creates a client for the given receptionist contact points.</summary>
/// <param name="system">The actor system to spawn the client in.</param>
/// <param name="clusterId">
/// The application <c>Cluster</c> these contacts belong to. Central now creates one client per
/// cluster (Phase 6), so the id is woven into the client's actor name to keep the per-cluster
/// clients distinct and diagnosable.
/// </param>
/// <param name="contacts">Receptionist actor paths, one per reachable node.</param>
/// <returns>The new client's <see cref="IActorRef"/>.</returns>
IActorRef Create(ActorSystem system, ImmutableHashSet<ActorPath> contacts);
IActorRef Create(ActorSystem system, string clusterId, ImmutableHashSet<ActorPath> contacts);
}
/// <inheritdoc />
@@ -35,13 +40,28 @@ public sealed class DefaultMeshClusterClientFactory : IMeshClusterClientFactory
private long _generation;
/// <inheritdoc />
public IActorRef Create(ActorSystem system, ImmutableHashSet<ActorPath> contacts)
public IActorRef Create(ActorSystem system, string clusterId, ImmutableHashSet<ActorPath> contacts)
{
ArgumentNullException.ThrowIfNull(system);
ArgumentException.ThrowIfNullOrEmpty(clusterId);
ArgumentNullException.ThrowIfNull(contacts);
var settings = ClusterClientSettings.Create(system).WithInitialContacts(contacts);
var name = $"mesh-client-{Interlocked.Increment(ref _generation)}";
var name = $"mesh-client-{SanitizeForActorName(clusterId)}-{Interlocked.Increment(ref _generation)}";
return system.ActorOf(ClusterClient.Props(settings), name);
}
/// <summary>
/// Reduces a cluster id to characters safe in an Akka actor name. The id is operator-authored
/// (e.g. a GUID or a slug), so it may carry characters Akka rejects; anything outside
/// <c>[A-Za-z0-9-]</c> becomes <c>-</c>. The trailing generation counter still guarantees
/// uniqueness, so a lossy sanitisation cannot cause a name collision.
/// </summary>
/// <param name="clusterId">The raw cluster id.</param>
/// <returns>An actor-name-safe rendering of the id.</returns>
private static string SanitizeForActorName(string clusterId)
{
var chars = clusterId.Select(c => char.IsAsciiLetterOrDigit(c) || c == '-' ? c : '-');
return new string(chars.ToArray());
}
}
@@ -58,10 +58,23 @@ public sealed record ClusterNodeAddress(string NodeId, string Host, int AkkaPort
/// to be deleted again.
/// </para>
/// <para>
/// <b>Phase 2 must revisit this.</b> Once the fleet splits into one mesh per cluster, an
/// admin node no longer sees site members at all, and every site row would report
/// <see cref="AddressMismatchKind.EnabledRowNotInCluster"/> forever. The check then has to
/// move to whatever transport replaces gossip — or be scoped to the central mesh.
/// <b>Scoped to the admin node's own cluster (Phase 6).</b> Once the fleet splits into one
/// mesh per cluster, an admin node no longer sees site members at all, and a fleet-wide sweep
/// would report every other cluster's row as
/// <see cref="AddressMismatchKind.EnabledRowNotInCluster"/> forever. The actor therefore scopes
/// <b>both</b> inputs to this function to the admin node's own cluster: the <c>rows</c>/enabled
/// set to rows whose <c>ClusterId</c> matches
/// <see cref="ZB.MOM.WW.OtOpcUa.Commons.Interfaces.IClusterRoleInfo.ClusterId"/>, and the
/// observed membership to members carrying its own
/// <see cref="ZB.MOM.WW.OtOpcUa.Commons.Interfaces.IClusterRoleInfo.ClusterRole"/>
/// (<c>FilterOwnClusterDriverMembers</c>). Scoping only the rows would swap the
/// <see cref="AddressMismatchKind.EnabledRowNotInCluster"/> false positive for a worse
/// Error-level <see cref="AddressMismatchKind.RunningNodeHasNoRow"/> during the mixed-mesh
/// window, where the DB scope is live but central still sees foreign members via gossip. A
/// legacy admin node carrying no cluster role (null <c>ClusterId</c>/<c>ClusterRole</c>) still
/// sees every member via gossip and reconciles the full fleet unchanged. The pure function
/// below is untouched — scoping happens entirely in the actor's DB queries and its
/// membership filter.
/// </para>
/// </remarks>
public static class ClusterNodeAddressReconciler
@@ -2,6 +2,7 @@ using Akka.Actor;
using Akka.Cluster;
using Akka.Event;
using Microsoft.EntityFrameworkCore;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Configuration;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
@@ -20,7 +21,7 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
public sealed class ClusterNodeAddressReconcilerActor : ReceiveActor, IWithTimers
{
/// <summary>Driver role name — only members carrying it are expected to have a ClusterNode row.</summary>
public const string DriverRole = "driver";
public const string DriverRole = RoleParser.Driver;
/// <summary>Collapses a burst of membership events (a rolling restart) into one reconcile.</summary>
public static readonly TimeSpan DebounceWindow = TimeSpan.FromSeconds(2);
@@ -29,6 +30,8 @@ public sealed class ClusterNodeAddressReconcilerActor : ReceiveActor, IWithTimer
public static readonly TimeSpan DefaultSweepInterval = TimeSpan.FromMinutes(5);
private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
private readonly string? _ownClusterId;
private readonly string? _ownClusterRole;
private readonly Akka.Cluster.Cluster _cluster;
private readonly TimeSpan _sweepInterval;
private readonly ILoggingAdapter _log = Context.GetLogger();
@@ -39,19 +42,49 @@ public sealed class ClusterNodeAddressReconcilerActor : ReceiveActor, IWithTimer
/// <summary>Creates Props for the reconciler.</summary>
/// <param name="dbFactory">Factory for the config database context.</param>
/// <param name="ownClusterId">
/// The admin node's own <c>ClusterId</c> (e.g. <c>MAIN</c>), from
/// <see cref="IClusterRoleInfo.ClusterId"/>. When non-null-and-non-empty this scopes the row
/// side of the reconcile (<c>rows</c>/<c>enabled</c>) to that cluster — the only members
/// central's own mesh can see after the Phase 6 split. A null or empty value (a legacy admin
/// node with no cluster role) reconciles the whole fleet, unchanged.
/// </param>
/// <param name="ownClusterRole">
/// The admin node's own cluster role (e.g. <c>cluster-MAIN</c>), from
/// <see cref="IClusterRoleInfo.ClusterRole"/>. Scopes the membership (<c>observed</c>) side to
/// match the row side, so a foreign-cluster driver member central still sees via gossip in the
/// mixed-mesh window is not mis-reported. Null/empty keeps every driver member (legacy).
/// </param>
/// <param name="sweepInterval">Periodic sweep interval; defaults to <see cref="DefaultSweepInterval"/>.</param>
/// <returns>Props for actor creation.</returns>
public static Props Props(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, TimeSpan? sweepInterval = null) =>
Akka.Actor.Props.Create(() => new ClusterNodeAddressReconcilerActor(dbFactory, sweepInterval));
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
string? ownClusterId = null,
string? ownClusterRole = null,
TimeSpan? sweepInterval = null) =>
Akka.Actor.Props.Create(() =>
new ClusterNodeAddressReconcilerActor(dbFactory, ownClusterId, ownClusterRole, sweepInterval));
/// <summary>Initializes a new instance of the <see cref="ClusterNodeAddressReconcilerActor"/> class.</summary>
/// <param name="dbFactory">Factory for the config database context.</param>
/// <param name="ownClusterId">
/// The admin node's own <c>ClusterId</c>; scopes the row side to that cluster's rows when
/// non-null-and-non-empty, else reconciles the whole fleet (legacy behaviour).
/// </param>
/// <param name="ownClusterRole">
/// The admin node's own cluster role; scopes the observed-membership side to that role when
/// non-null-and-non-empty, else keeps every driver member (legacy behaviour).
/// </param>
/// <param name="sweepInterval">Periodic sweep interval; defaults to <see cref="DefaultSweepInterval"/>.</param>
public ClusterNodeAddressReconcilerActor(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, TimeSpan? sweepInterval = null)
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
string? ownClusterId = null,
string? ownClusterRole = null,
TimeSpan? sweepInterval = null)
{
_dbFactory = dbFactory;
_ownClusterId = string.IsNullOrEmpty(ownClusterId) ? null : ownClusterId;
_ownClusterRole = string.IsNullOrEmpty(ownClusterRole) ? null : ownClusterRole;
_cluster = Akka.Cluster.Cluster.Get(Context.System);
_sweepInterval = sweepInterval ?? DefaultSweepInterval;
@@ -75,24 +108,27 @@ public sealed class ClusterNodeAddressReconcilerActor : ReceiveActor, IWithTimer
private void Reconcile()
{
var observed = _cluster.State.Members
.Where(m => m.Status == MemberStatus.Up && m.Roles.Contains(DriverRole))
.Select(m => (Host: m.Address.Host ?? string.Empty, Port: m.Address.Port ?? 0))
.Where(a => !string.IsNullOrWhiteSpace(a.Host) && a.Port > 0)
.ToList();
var observed = FilterOwnClusterDriverMembers(_cluster.State.Members, _ownClusterRole);
List<ClusterNodeAddress> rows;
List<string> enabled;
try
{
using var db = _dbFactory.CreateDbContext();
rows = db.ClusterNodes.AsNoTracking()
// Per-cluster mesh Phase 6: scope to the admin node's OWN cluster's rows. After the split,
// central's mesh sees only its own pair, so a row in another cluster it cannot observe would
// false-report EnabledRowNotInCluster forever. A legacy admin node with no cluster role
// (_ownClusterId == null) still sees every member via gossip, so it reconciles the full fleet.
var scoped = db.ClusterNodes.AsNoTracking();
if (_ownClusterId is not null)
scoped = scoped.Where(n => n.ClusterId == _ownClusterId);
rows = scoped
.Select(n => new ClusterNodeAddress(n.NodeId, n.Host, n.AkkaPort))
.ToList();
// Same predicate the coordinator's expected-ack set uses — a node the deploy path will
// not wait for must not be reported as missing either, or the maintenance hatch trades a
// failed deployment for a permanent warning.
enabled = db.ClusterNodes.AsNoTracking()
enabled = scoped
.Where(n => n.Enabled && !n.MaintenanceMode).Select(n => n.NodeId).ToList();
}
catch (Exception ex)
@@ -126,6 +162,35 @@ public sealed class ClusterNodeAddressReconcilerActor : ReceiveActor, IWithTimer
}
}
/// <summary>
/// Projects the Up, driver-role members central should reconcile against, scoped to the admin
/// node's OWN cluster role when it carries one. Extracted so the observed-membership filter —
/// the membership half of the Phase 6 own-cluster scope, mirroring the <c>ClusterId</c> row
/// scope in <see cref="Reconcile"/> — is unit-testable without seeding a live cluster.
/// </summary>
/// <param name="members">The live cluster members (from <c>Cluster.State.Members</c>).</param>
/// <param name="ownClusterRole">
/// The admin node's own cluster role (e.g. <c>cluster-MAIN</c>). When non-null-and-non-empty,
/// driver members that do not also carry it are excluded — matching the row scope so a foreign
/// driver member central still sees via gossip in the mixed-mesh window (row filtered out, but
/// member still visible) is NOT mis-reported as an Error-level <c>RunningNodeHasNoRow</c>. A
/// null or empty value (a legacy node with no cluster role) keeps every driver member —
/// reconcile-all, unchanged.
/// </param>
/// <returns>The <c>host</c>/<c>port</c> of each in-scope Up driver member.</returns>
public static IReadOnlyList<(string Host, int Port)> FilterOwnClusterDriverMembers(
IEnumerable<Member> members, string? ownClusterRole)
{
var driverMembers = members
.Where(m => m.Status == MemberStatus.Up && m.Roles.Contains(DriverRole));
if (!string.IsNullOrEmpty(ownClusterRole))
driverMembers = driverMembers.Where(m => m.Roles.Contains(ownClusterRole));
return driverMembers
.Select(m => (Host: m.Address.Host ?? string.Empty, Port: m.Address.Port ?? 0))
.Where(a => !string.IsNullOrWhiteSpace(a.Host) && a.Port > 0)
.ToList();
}
/// <summary>Self-message: run a reconcile pass now.</summary>
public sealed class ReconcileNow
{
@@ -31,10 +31,9 @@ public sealed record ManualFailoverSnapshot(string? PrimaryAddress, IReadOnlyLis
/// moves with it and OPC UA clients re-select.
/// </para>
/// <para>
/// <b>Mesh-scope caveat.</b> Until the per-cluster mesh work lands
/// (<c>docs/plans/2026-07-21-per-cluster-mesh-design.md</c>) the Primary is elected once per
/// Akka mesh, not per application <c>Cluster</c> row. On a fleet running several application
/// clusters in one mesh this acts on the whole mesh's Primary. The UI says so.
/// Post-Phase-6 (per-cluster mesh split): the Primary is elected per application
/// <c>Cluster</c>, because each cluster's redundant pair now runs its own independent
/// 2-node Akka mesh. This service therefore acts on <b>this pair's own</b> Primary.
/// </para>
/// </remarks>
public interface IManualFailoverService
@@ -2,6 +2,8 @@ using Akka.Actor;
using Akka.Cluster;
using Akka.Cluster.Tools.PublishSubscribe;
using Akka.Event;
using ZB.MOM.WW.Health.Akka;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using CommonsRedundancyRole = ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy.RedundancyRole;
@@ -112,7 +114,7 @@ public sealed class RedundancyStateActor : ReceiveActor, IWithTimers
}
/// <summary>The cluster role that marks a node as carrying the driver runtime.</summary>
public const string DriverRole = "driver";
public const string DriverRole = RoleParser.Driver;
/// <summary>
/// Selects the driver Primary: the <b>oldest</b> Up member carrying the <see cref="DriverRole"/>.
@@ -140,15 +142,39 @@ public sealed class RedundancyStateActor : ReceiveActor, IWithTimers
/// </remarks>
/// <param name="members">The current cluster members, in any order.</param>
/// <returns>The oldest Up driver member's address, or <c>null</c> when there is none.</returns>
public static Address? SelectDriverPrimary(IEnumerable<Member> members)
public static Address? SelectDriverPrimary(IEnumerable<Member> members) =>
SelectOldestUpMemberOfRole(members, DriverRole);
/// <summary>
/// Selects the oldest Up member carrying <paramref name="role"/> — the node that
/// <c>ClusterSingletonManager</c> would place a role-scoped singleton on.
/// </summary>
/// <param name="members">The current cluster members, in any order.</param>
/// <param name="role">The cluster role to scope the selection to.</param>
/// <returns>The oldest Up member's address for that role, or <c>null</c> when there is none.</returns>
/// <remarks>
/// <para>
/// The age-ordering rationale in <see cref="SelectDriverPrimary"/> applies verbatim to any
/// role: oldest, never <c>RoleLeader</c>. The rule itself lives in the shared
/// <see cref="ClusterActiveNode"/> (<c>ZB.MOM.WW.Health.Akka</c> 0.3.0) so it has exactly
/// one implementation family-wide — this method is the control plane's entry point into it.
/// </para>
/// <para>
/// Delegating rather than re-implementing is what keeps the redundancy snapshot and the
/// <c>/health/active</c> tier structurally in agreement. They answer for different
/// consumers — the OPC UA <c>ServiceLevel</c> 250/240 split reads this election, while an
/// orchestrator routes admin traffic by the health tier — so a divergence would advertise
/// one node as authoritative while gating the data plane on another. Both apps in the
/// family previously kept private copies of this rule, and both got it wrong in a
/// different way.
/// </para>
/// </remarks>
public static Address? SelectOldestUpMemberOfRole(IEnumerable<Member> members, string role)
{
ArgumentNullException.ThrowIfNull(members);
ArgumentException.ThrowIfNullOrWhiteSpace(role);
return members
.Where(m => m.Status == MemberStatus.Up && m.Roles.Contains(DriverRole))
.OrderBy(m => m, Member.AgeOrdering)
.FirstOrDefault()
?.Address;
return ClusterActiveNode.OldestUpMember(members, role)?.Address;
}
private IReadOnlyList<NodeRedundancyState> BuildSnapshot()
@@ -13,6 +13,7 @@ using ZB.MOM.WW.OtOpcUa.ControlPlane.AdminOperations;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Audit;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Coordinators;
using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
@@ -130,10 +131,12 @@ public static class ServiceCollectionExtensions
(system, registry, resolver) => FleetStatusBroadcaster.Props(),
singletonOptions);
builder.WithSingleton<RedundancyStateActorKey>(
RedundancyStateSingletonName,
(system, registry, resolver) => RedundancyStateActor.Props(),
singletonOptions);
// Per-cluster mesh Phase 6: the redundancy-state singleton is NO LONGER registered here.
// It was an admin-scoped, fleet-wide singleton — one Primary elected across the whole mesh.
// After the mesh split a driver-only site pair has no admin node to host it, so it moved to
// WithOtOpcUaClusterRedundancySingleton, a cluster-{ClusterId}-scoped singleton spawned on
// EVERY driver node (the fused central included). One per mesh, electing its own pair-local
// Primary and publishing on its own mesh's DistributedPubSub.
// Per-cluster mesh Phase 1: ClusterNode.AkkaPort duplicates the node's own Cluster:Port and
// nothing makes them agree. Runs here rather than on each driver node because Phase 4 removes
@@ -150,13 +153,87 @@ public static class ServiceCollectionExtensions
(system, registry, resolver) =>
{
var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>();
return ClusterNodeAddressReconcilerActor.Props(dbFactory);
// Per-cluster mesh Phase 6: scope BOTH halves of the reconcile to the admin node's OWN
// cluster — rows by ClusterId, observed membership by cluster role. After the mesh split
// central sees only its own pair via gossip, so an unscoped sweep would false-report
// every other cluster's rows (EnabledRowNotInCluster). Scoping only the rows would swap
// that for a WORSE Error-level RunningNodeHasNoRow in the mixed-mesh window (rows already
// filtered, but foreign members still visible via gossip). A legacy node with no cluster
// role has null ClusterId/ClusterRole ⇒ full-fleet reconcile, unchanged.
var roleInfo = resolver.GetService<IClusterRoleInfo>();
return ClusterNodeAddressReconcilerActor.Props(
dbFactory, roleInfo.ClusterId, roleInfo.ClusterRole);
},
singletonOptions,
createProxyToo: false);
return builder;
}
/// <summary>
/// Registers <see cref="RedundancyStateActor"/> as a cluster singleton scoped to the local node's
/// OWN <c>cluster-{ClusterId}</c> role (or the fixed <c>driver</c> role when the node carries no
/// cluster role), and spawns it on EVERY driver node (per-cluster mesh, Phase 6). Wire from the
/// driver branch of the fused Host's Program.cs — the fused central node is <c>hasDriver</c> too,
/// so it hosts exactly one redundancy singleton scoped to its own cluster role (e.g.
/// <c>cluster-MAIN</c>), while the admin singleton set it also registers no longer carries
/// redundancy.
/// </summary>
/// <param name="builder">The Akka configuration builder.</param>
/// <param name="roleInfo">The local node's cluster-role view (Task 1's <c>ClusterRoleInfo</c>).</param>
/// <returns>The builder for fluent chaining.</returns>
/// <remarks>
/// After the mesh split, central's mesh sees only its own pair — a driver-only site pair therefore
/// has NO admin node to elect its Primary. Scoping this singleton to the node's cluster role and
/// spawning it on every driver node gives each mesh exactly one, electing its own pair-local
/// Primary and publishing <c>redundancy-state</c> on its own mesh's DistributedPubSub.
/// </remarks>
public static AkkaConfigurationBuilder WithOtOpcUaClusterRedundancySingleton(
this AkkaConfigurationBuilder builder, AkkaClusterOptions clusterOptions)
{
var singletonOptions = BuildClusterRedundancySingletonOptions(clusterOptions);
builder.WithSingleton<RedundancyStateActorKey>(
RedundancyStateSingletonName,
(system, registry, resolver) => RedundancyStateActor.Props(),
singletonOptions);
return builder;
}
/// <summary>
/// Builds the <see cref="ClusterSingletonOptions"/> that scope the redundancy-state singleton to
/// the local node's <c>cluster-{ClusterId}</c> role, falling back to the fixed <c>driver</c> role
/// when the node carries no cluster role. Extracted (like
/// <c>SplitBrainResolverActivationTests</c>'s <c>BuildDowningHocon</c>) so the exact role scope is
/// unit-assertable without booting a cluster.
/// </summary>
/// <remarks>
/// Cluster-scope is the robust choice — it survives an accidental two-mesh merge by keeping one
/// singleton per cluster. The <c>driver</c>-role fallback deliberately does NOT throw when a node
/// has no cluster role, because (a) that is the pre-Phase-6 fleet-wide behavior on a legacy single
/// mesh, keeping legacy/harness nodes and any not-yet-migrated deployment booting unchanged, and
/// (b) on a genuinely split 2-node mesh the <c>driver</c> role is already pair-local — the mesh IS
/// the pair. So the fallback is correct in both worlds and costs no availability.
/// </remarks>
/// <param name="clusterOptions">The node's own cluster configuration (roles), read without the
/// ActorSystem so this is safe to call inside the Akka configurator lambda.</param>
/// <returns>Singleton options scoped to the node's cluster role, or the <c>driver</c> role.</returns>
public static ClusterSingletonOptions BuildClusterRedundancySingletonOptions(AkkaClusterOptions clusterOptions)
{
ArgumentNullException.ThrowIfNull(clusterOptions);
// Derive the cluster-{ClusterId} scope from the node's OWN configured roles — NOT from
// IClusterRoleInfo. IClusterRoleInfo's implementation depends on the ActorSystem (it is a live
// Cluster.State view), and this runs inside the AddAkka configurator lambda WHILE the
// ActorSystem is being built; resolving IClusterRoleInfo there recurses back into ActorSystem
// construction and stack-overflows the host at boot (the Phase 6 live gate caught exactly this
// on the fused central node). AkkaClusterOptions is pure configuration, available before the
// cluster forms — the same source ClusterRoleInfo.ClusterRole itself derives from — so the
// scope is identical. First cluster role wins, in configuration order (matching ClusterRoleInfo).
var clusterRole = clusterOptions.Roles.FirstOrDefault(RoleParser.IsClusterRole);
return new ClusterSingletonOptions { Role = clusterRole ?? RoleParser.Driver };
}
}
/// <summary>Marker key types used by <c>Akka.Hosting</c> to resolve singletons from the registry.</summary>
@@ -19,6 +19,14 @@
Google.Protobuf + Grpc.Core.Api flow transitively from Commons (the generated client). -->
<PackageReference Include="Grpc.Net.Client"/>
<PackageReference Include="ZB.MOM.WW.Audit"/>
<!-- For ClusterActiveNode — the shared "oldest Up member of role X" primitive. The control
plane and the /health/active tier must give the SAME answer for which node is in charge,
or the tier reports one node active while the Primary-gated data plane runs on another;
delegating here is what makes that structural rather than a convention. It lives in
Health.Akka because that is the family's Akka-cluster utility package (it already hosts
AkkaClusterStatusPolicy and an endpoint gate); a dedicated cluster package would be a
better home if a third such primitive ever appears. -->
<PackageReference Include="ZB.MOM.WW.Health.Akka"/>
</ItemGroup>
<ItemGroup>
@@ -8,6 +8,7 @@ using ZB.MOM.WW.Health;
using ZB.MOM.WW.Health.Akka;
using ZB.MOM.WW.Health.EntityFrameworkCore;
using ZB.MOM.WW.LocalDb.Replication;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Host.Configuration;
@@ -17,7 +18,7 @@ public static class HealthEndpoints
{
/// <summary>
/// Registers the shared ZB.MOM.WW health probes. Tier semantics preserved: configdb + akka on
/// ready+active; admin-leader on active only. The configdb probe is admin-only (per-cluster mesh
/// ready+active; cluster-primary on active only. The configdb probe is admin-only (per-cluster mesh
/// Phase 4): a driver-only node holds no ConfigDb, so it is registered iff <paramref name="hasAdmin"/>.
/// </summary>
/// <param name="services">The service collection to register the health checks on.</param>
@@ -53,11 +54,33 @@ public static class HealthEndpoints
failureStatus: null,
tags: new[] { ZbHealthTags.Ready, ZbHealthTags.Active },
args: AkkaClusterStatusPolicy.OtOpcUaCompat)
// "Is this node in charge of its mesh?" — Healthy on the one node that owns the active
// work for its Cluster, Unhealthy (503) on its partner, so Traefik pins the AdminUI to the
// node actually hosting the cluster singletons.
//
// Role preference admin-then-driver: a fused central node answers for `admin` (the role
// the singletons and the AdminUI are pinned to), while a driver-only site node answers for
// `driver` — which is exactly RedundancyStateActor.SelectDriverPrimary, the election behind
// IsDriverPrimary and the OPC UA ServiceLevel 250/240 split. Both now route through the
// shared ClusterActiveNode, so the tier and the ServiceLevel cannot disagree.
//
// Scoping is per-mesh for free: after per-cluster mesh Phase 6 a node's ClusterState
// contains only its own application Cluster, so "oldest Up member" is already "oldest Up
// member of this Cluster" — exactly one node answers 200 per Cluster, by construction.
//
// Before Health 0.3.0 this was ActiveNodeHealthCheck(role: "admin"), which answered a
// different question and got it wrong twice: it returned Healthy for any node lacking the
// admin role, so every driver-only site node called itself active, and it selected by
// RoleLeader (lowest address) rather than by age, which is not where Akka places
// singletons. See lmxopcua#494.
.AddTypeActivatedCheck<ActiveNodeHealthCheck>(
"admin-leader",
"cluster-primary",
failureStatus: null,
tags: new[] { ZbHealthTags.Active },
args: "admin")
args: new ActiveNodeHealthCheckOptions
{
RolePreference = new[] { RoleParser.Admin, RoleParser.Driver },
})
// Registered on every node regardless of role (unlike the admin-only configdb probe above):
// the check itself resolves ISyncStatus optionally and reports Healthy when LocalDb is absent
// (admin-only graphs) or replication is default-OFF, so a plain node is never degraded by it.
@@ -2,6 +2,7 @@ using Akka.Actor;
using Akka.Hosting;
using Microsoft.AspNetCore.Server.Kestrel.Core;
using Microsoft.Extensions.DependencyInjection.Extensions;
using Microsoft.Extensions.Options;
using Serilog;
using ZB.MOM.WW.LocalDb.Replication;
using Serilog.Events;
@@ -12,6 +13,7 @@ using ZB.MOM.WW.OtOpcUa.AdminUI.Grpc;
using ZB.MOM.WW.OtOpcUa.AdminUI.Hubs;
using ZB.MOM.WW.OtOpcUa.AdminUI.ScriptAnalysis;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Configuration.Services;
using ZB.MOM.WW.OtOpcUa.ControlPlane;
@@ -371,9 +373,33 @@ builder.Services.AddAkka("otopcua", (ab, sp) =>
ab.WithOtOpcUaSignalRBridges();
}
if (hasDriver)
{
ab.WithOtOpcUaRuntimeActors();
// Per-cluster mesh Phase 6: re-home the redundancy-state singleton off the admin node onto
// EVERY driver node, so each mesh elects its own pair-local Primary (a driver-only site pair
// has no admin node to do it). It scopes to this node's own cluster-{ClusterId} role, falling
// back to the driver role for legacy single-mesh / not-yet-migrated nodes (already pair-local
// on a split mesh, since the mesh IS the pair). Runs on the fused central node too (it is
// hasDriver), where it scopes to cluster-MAIN — exactly one redundancy singleton per node,
// since the admin branch above no longer registers one. Pass AkkaClusterOptions (pure config),
// NOT IClusterRoleInfo: this lambda runs WHILE the ActorSystem is being built, and
// IClusterRoleInfo depends on the ActorSystem — resolving it here recurses into ActorSystem
// construction and stack-overflows the host at boot. The singleton scope only needs the node's
// configured cluster role, which AkkaClusterOptions carries.
ab.WithOtOpcUaClusterRedundancySingleton(sp.GetRequiredService<IOptions<AkkaClusterOptions>>().Value);
}
});
// Simultaneous-cold-start split-brain guard (dark switch Cluster:BootstrapGuard:Enabled, default off).
// Registered AFTER AddAkka so its StartAsync runs once Akka's hosted service has built (and, when the
// guard is off, already auto-joined) the ActorSystem. When on, the node started with no config seeds
// (BuildClusterOptions) and this coordinator issues the single reachability-gated JoinSeedNodes. It
// no-ops when the guard is off, so registering it unconditionally is safe.
builder.Services.AddHostedService(sp => new ClusterBootstrapCoordinator(
() => sp.GetRequiredService<ActorSystem>(),
sp.GetRequiredService<IOptions<AkkaClusterOptions>>(),
sp.GetRequiredService<ILogger<ClusterBootstrapCoordinator>>()));
// Down-if-alone recovery watchdog (#459). Registered AFTER AddAkka so it starts after Akka's own
// hosted service has built the ActorSystem; it resolves the system lazily (never at construction) so
// it can't race startup. On an unexpected SBR self-down it stops the host so the service supervisor
@@ -36,7 +36,7 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime;
public static class ServiceCollectionExtensions
{
public const string DriverRole = "driver";
public const string DriverRole = RoleParser.Driver;
public const string DriverHostActorName = "driver-host";
public const string DbHealthProbeActorName = "db-health";
@@ -174,4 +174,44 @@ public sealed class AkkaClusterOptionsValidatorTests
result.Failed.ShouldBeFalse(result.FailureMessage);
}
/// <summary>The bootstrap-guard timing knobs default to positive values and pass when enabled.</summary>
[Fact]
public void Bootstrap_guard_defaults_pass()
{
var options = CentralNode("site-a-1", Seed("site-a-1"), Seed("site-a-2"));
options.BootstrapGuard = new ClusterBootstrapGuardOptions { Enabled = true };
var result = new AkkaClusterOptionsValidator().Validate(null, options);
result.Failed.ShouldBeFalse(result.FailureMessage);
}
/// <summary>
/// A zero <c>PartnerProbeSeconds</c> silently degrades the guard to "never wait, form alone
/// immediately" — re-opening the split. It must fail the host at boot, not run degraded.
/// </summary>
[Fact]
public void Bootstrap_guard_with_non_positive_probe_seconds_fails()
{
var options = CentralNode("site-a-1", Seed("site-a-1"), Seed("site-a-2"));
options.BootstrapGuard = new ClusterBootstrapGuardOptions { Enabled = true, PartnerProbeSeconds = 0 };
var result = new AkkaClusterOptionsValidator().Validate(null, options);
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("PartnerProbeSeconds");
}
/// <summary>The timing knobs are NOT validated when the guard is off — a disabled guard is inert.</summary>
[Fact]
public void Bootstrap_guard_disabled_does_not_validate_timings()
{
var options = CentralNode("site-a-1", Seed("site-a-1"), Seed("site-a-2"));
options.BootstrapGuard = new ClusterBootstrapGuardOptions { Enabled = false, PartnerProbeSeconds = 0 };
var result = new AkkaClusterOptionsValidator().Validate(null, options);
result.Failed.ShouldBeFalse(result.FailureMessage);
}
}
@@ -0,0 +1,204 @@
using System.Net;
using System.Net.Sockets;
using Akka.Actor;
using Akka.Cluster;
using Akka.Hosting;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
/// <summary>
/// Real-ActorSystem tests for the simultaneous-cold-start split-brain guard
/// (<see cref="ClusterBootstrapCoordinator"/> + <see cref="ClusterBootstrapGuard"/>), started through
/// the SAME wiring as production (<see cref="ServiceCollectionExtensions.WithOtOpcUaClusterBootstrap"/>
/// with <c>Cluster:BootstrapGuard:Enabled</c> + the coordinator hosted service). This subsystem has a
/// history of bugs invisible to pure unit tests, so the load-bearing correctness properties are
/// asserted against running nodes, mirroring <see cref="SelfFirstSeedBootstrapTests"/>.
/// </summary>
/// <remarks>
/// Ephemeral loopback ports are 5-digit, so numeric order equals the guard's ordinal "host:port"
/// string order — <c>Min(port)</c> is the founder, <c>Max(port)</c> is the higher (probing) node.
/// </remarks>
public sealed class ClusterBootstrapCoordinatorTests
{
private static int FreePort()
{
var listener = new TcpListener(IPAddress.Loopback, 0);
listener.Start();
var port = ((IPEndPoint)listener.LocalEndpoint).Port;
listener.Stop();
return port;
}
/// <summary>
/// Starts a node with the bootstrap guard ENABLED, wired exactly as Program.cs does: empty Akka
/// seed list (via <c>BuildClusterOptions</c>) plus the coordinator hosted service that issues the
/// reachability-gated <c>JoinSeedNodes</c>. Seeds are listed self-first (as every shipped config
/// is); the guard, not the order, decides founder vs. joiner.
/// </summary>
private static async Task<IHost> StartGuardedNodeAsync(
string systemName, int selfPort, int peerPort, int probeSeconds = 4)
{
var self = $"akka.tcp://{systemName}@127.0.0.1:{selfPort}";
var peer = $"akka.tcp://{systemName}@127.0.0.1:{peerPort}";
var options = new AkkaClusterOptions
{
SystemName = systemName,
Hostname = "127.0.0.1",
PublicHostname = "127.0.0.1",
Port = selfPort,
Roles = new[] { "driver" },
SeedNodes = new[] { self, peer },
BootstrapGuard = new ClusterBootstrapGuardOptions
{
Enabled = true,
PartnerProbeSeconds = probeSeconds,
PartnerProbeIntervalMs = 250,
ProbeConnectTimeoutMs = 500,
},
};
var builder = Host.CreateDefaultBuilder();
builder.ConfigureServices(services =>
{
services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(options));
services.AddAkka(systemName, (ab, sp) => ab.WithOtOpcUaClusterBootstrap(sp));
// Mirror Program.cs — the coordinator is what drives the guarded join.
services.AddHostedService(sp => new ClusterBootstrapCoordinator(
() => sp.GetRequiredService<ActorSystem>(),
sp.GetRequiredService<IOptions<AkkaClusterOptions>>(),
sp.GetRequiredService<ILogger<ClusterBootstrapCoordinator>>()));
});
var host = builder.Build();
await host.StartAsync();
return host;
}
private static async Task<bool> WaitForUpMembersAsync(IHost host, int expected, TimeSpan timeout)
{
var cluster = Akka.Cluster.Cluster.Get(host.Services.GetRequiredService<ActorSystem>());
var deadline = DateTime.UtcNow + timeout;
while (DateTime.UtcNow < deadline)
{
if (cluster.State.Members.Count(m => m.Status == MemberStatus.Up) >= expected) return true;
await Task.Delay(200);
}
return cluster.State.Members.Count(m => m.Status == MemberStatus.Up) >= expected;
}
private static async Task StopAsync(IHost host)
{
try { await host.StopAsync(); }
catch (Exception) { /* teardown only */ }
host.Dispose();
}
/// <summary>
/// The FOUNDER (lower address) with the guard on forms a cluster alone when its partner is dead —
/// exactly as un-guarded self-first does. The guard must not regress the preferred founder.
/// </summary>
[Fact]
public async Task Founder_forms_alone_when_partner_is_dead()
{
var low = FreePort();
var high = FreePort();
var founderPort = Math.Min(low, high);
var deadPeerPort = Math.Max(low, high); // nothing listening
var host = await StartGuardedNodeAsync("otopcua-guard-1", founderPort, deadPeerPort);
try
{
(await WaitForUpMembersAsync(host, 1, TimeSpan.FromSeconds(30)))
.ShouldBeTrue("the founder (lower address) must self-form immediately when its partner is down");
}
finally { await StopAsync(host); }
}
/// <summary>
/// THE LOAD-BEARING CASE. The HIGHER node with the guard on must still cold-start ALONE when its
/// partner is genuinely dead: it probes the (dead) founder, times out, and falls back to self-first.
/// If the guard's peer-first logic were wrong, this node would hang in JoinSeedNodeProcess forever —
/// the very regression a naive "always let the lower node found" rule would introduce.
/// </summary>
[Fact]
public async Task Higher_node_forms_alone_when_partner_is_dead_after_probing()
{
var low = FreePort();
var high = FreePort();
var higherPort = Math.Max(low, high);
var deadFounderPort = Math.Min(low, high); // the founder is dead
var host = await StartGuardedNodeAsync("otopcua-guard-2", higherPort, deadFounderPort, probeSeconds: 3);
try
{
// Must come Up AFTER the probe window (~3 s) expires and it falls back to self-first.
(await WaitForUpMembersAsync(host, 1, TimeSpan.FromSeconds(30)))
.ShouldBeTrue("the higher node must fall back to self-first and form alone once the dead partner never answers the probe");
}
finally { await StopAsync(host); }
}
/// <summary>
/// FALSIFIABILITY CONTROL for the test above: the higher node does NOT form alone during the probe
/// window — it is genuinely waiting/probing, not self-forming immediately (which would mean the
/// guard is inert). If this ever starts seeing a member before the window, the probe is not gating.
/// </summary>
[Fact]
public async Task Higher_node_does_not_form_before_the_probe_window_expires()
{
var low = FreePort();
var high = FreePort();
var higherPort = Math.Max(low, high);
var deadFounderPort = Math.Min(low, high);
var host = await StartGuardedNodeAsync("otopcua-guard-3", higherPort, deadFounderPort, probeSeconds: 10);
try
{
// Well within the 10 s probe window: still no cluster, because it is probing the dead founder.
(await WaitForUpMembersAsync(host, 1, TimeSpan.FromSeconds(3)))
.ShouldBeFalse("the higher node must probe for the founder, not self-form immediately");
}
finally { await StopAsync(host); }
}
/// <summary>
/// The higher node JOINS a live founder rather than forming a second cluster: start the founder,
/// then the higher node — its probe finds the founder reachable, it commits peer-first, and the
/// pair converges to ONE cluster of two.
/// </summary>
[Fact]
public async Task Higher_node_joins_a_live_founder()
{
const string systemName = "otopcua-guard-4";
var low = FreePort();
var high = FreePort();
var founderPort = Math.Min(low, high);
var higherPort = Math.Max(low, high);
var founder = await StartGuardedNodeAsync(systemName, founderPort, higherPort);
IHost? higher = null;
try
{
(await WaitForUpMembersAsync(founder, 1, TimeSpan.FromSeconds(30)))
.ShouldBeTrue("the founder must be Up before the higher node probes it");
higher = await StartGuardedNodeAsync(systemName, higherPort, founderPort);
(await WaitForUpMembersAsync(higher, 2, TimeSpan.FromSeconds(60)))
.ShouldBeTrue("the higher node must join the founder, forming one cluster of two");
(await WaitForUpMembersAsync(founder, 2, TimeSpan.FromSeconds(60)))
.ShouldBeTrue("the founder must see the higher node as a member of ITS cluster");
}
finally
{
if (higher is not null) await StopAsync(higher);
await StopAsync(founder);
}
}
}
@@ -0,0 +1,137 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Cluster;
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
/// <summary>
/// Unit tests for <see cref="ClusterBootstrapGuard"/> — the pure decision core of the
/// simultaneous-cold-start split-brain guard. The runtime probe + JoinSeedNodes wiring lives in
/// <c>ClusterBootstrapCoordinator</c> and is covered by the live gate.
/// </summary>
public sealed class ClusterBootstrapGuardTests
{
private const string A = "akka.tcp://otopcua@site-a-1:4053";
private const string B = "akka.tcp://otopcua@site-a-2:4053";
[Fact]
public void Lower_address_node_is_the_founder_and_needs_no_probe()
{
// site-a-1 < site-a-2, so on site-a-1 the guard makes it the founder.
var role = ClusterBootstrapGuard.Analyze(new[] { A, B }, "site-a-1", 4053);
role.Applies.ShouldBeTrue();
role.IsFounder.ShouldBeTrue();
role.SelfFirstOrder.ShouldBe(new[] { A, B });
}
[Fact]
public void Higher_address_node_is_not_the_founder_and_targets_the_partner_for_probing()
{
// On site-a-2, the partner is site-a-1 (the lower/founder).
var role = ClusterBootstrapGuard.Analyze(new[] { B, A }, "site-a-2", 4053);
role.Applies.ShouldBeTrue();
role.IsFounder.ShouldBeFalse();
role.PartnerHost.ShouldBe("site-a-1");
role.PartnerPort.ShouldBe(4053);
}
[Fact]
public void Tie_break_is_symmetric_regardless_of_seed_order()
{
// Both nodes must agree on who founds no matter how each lists its seeds.
var onLower = ClusterBootstrapGuard.Analyze(new[] { A, B }, "site-a-1", 4053);
var onHigher = ClusterBootstrapGuard.Analyze(new[] { B, A }, "site-a-2", 4053);
onLower.IsFounder.ShouldBeTrue();
onHigher.IsFounder.ShouldBeFalse();
// Exactly one founder.
(onLower.IsFounder ^ onHigher.IsFounder).ShouldBeTrue();
}
[Fact]
public void Higher_node_joins_the_founder_when_partner_is_reachable()
{
var role = ClusterBootstrapGuard.Analyze(new[] { B, A }, "site-a-2", 4053);
// Reachable ⇒ peer-first ⇒ JoinSeedNodeProcess ⇒ joins the founder, never self-forms.
ClusterBootstrapGuard.HigherNodeOrder(role, partnerReachable: true)
.ShouldBe(new[] { A, B }); // partner (site-a-1) first
}
[Fact]
public void Higher_node_forms_alone_when_partner_is_unreachable()
{
var role = ClusterBootstrapGuard.Analyze(new[] { B, A }, "site-a-2", 4053);
// Unreachable after the window ⇒ partner is dead ⇒ self-first ⇒ cold-start-alone preserved.
ClusterBootstrapGuard.HigherNodeOrder(role, partnerReachable: false)
.ShouldBe(new[] { B, A }); // self (site-a-2) first
}
[Fact]
public void Guard_does_not_apply_to_a_single_seed_node()
{
// A driver-only site node seeded solely by central-1 is not a pair seed — guard is inert.
var role = ClusterBootstrapGuard.Analyze(new[] { "akka.tcp://otopcua@central-1:4053" }, "site-x", 4053);
role.Applies.ShouldBeFalse();
}
[Fact]
public void Guard_does_not_apply_when_self_is_absent_from_the_two_seeds()
{
var role = ClusterBootstrapGuard.Analyze(new[] { A, B }, "central-1", 4053);
role.Applies.ShouldBeFalse();
}
[Fact]
public void Guard_does_not_apply_to_three_or_more_seeds()
{
var role = ClusterBootstrapGuard.Analyze(
new[] { A, B, "akka.tcp://otopcua@site-a-3:4053" }, "site-a-1", 4053);
role.Applies.ShouldBeFalse();
}
[Fact]
public void Guard_does_not_apply_when_a_seed_is_malformed()
{
var role = ClusterBootstrapGuard.Analyze(new[] { A, "not-a-seed" }, "site-a-1", 4053);
role.Applies.ShouldBeFalse();
}
[Theory]
[InlineData("akka.tcp://otopcua@site-a-1:4053", "site-a-1", 4053)]
[InlineData("akka.tcp://otopcua@10.0.0.5:4053/", "10.0.0.5", 4053)]
[InlineData(" akka.tcp://otopcua@host.lan:1234 ", "host.lan", 1234)]
public void TryParse_extracts_host_and_port(string seed, string expectedHost, int expectedPort)
{
ClusterBootstrapGuard.TryParse(seed, out var host, out var port).ShouldBeTrue();
host.ShouldBe(expectedHost);
port.ShouldBe(expectedPort);
}
[Theory]
[InlineData("")]
[InlineData("garbage")]
[InlineData("akka.tcp://otopcua@host")] // no port
public void TryParse_rejects_malformed_seeds(string seed)
{
ClusterBootstrapGuard.TryParse(seed, out _, out _).ShouldBeFalse();
}
[Fact]
public void Port_participates_in_the_tie_break_when_hosts_are_equal()
{
// Shared-host loopback pair (test rig): same host, different ports — port breaks the tie.
const string low = "akka.tcp://otopcua@127.0.0.1:4053";
const string high = "akka.tcp://otopcua@127.0.0.1:4054";
ClusterBootstrapGuard.Analyze(new[] { low, high }, "127.0.0.1", 4053).IsFounder.ShouldBeTrue();
ClusterBootstrapGuard.Analyze(new[] { high, low }, "127.0.0.1", 4054).IsFounder.ShouldBeFalse();
}
}
@@ -0,0 +1,105 @@
using Akka.Actor;
using Akka.Hosting;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Options;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
/// <summary>
/// Guards <see cref="ClusterRoleInfo.ClusterRole"/> / <see cref="ClusterRoleInfo.ClusterId"/> — the
/// node's OWN cluster-scoped role (per-cluster mesh, Phase 6), derived from the node's configured
/// <see cref="AkkaClusterOptions.Roles"/> at construction time, NOT from live cluster membership.
/// That distinction matters: this identity must be readable before the cluster forms, so later
/// Phase-6 work (singleton registration, a startup validator) can act on it at host-build time.
/// </summary>
public sealed class ClusterRoleInfoTests
{
private static AkkaClusterOptions SampleOptions(params string[] roles) => new()
{
Port = 0,
Hostname = "127.0.0.1",
PublicHostname = "127.0.0.1",
SeedNodes = Array.Empty<string>(),
Roles = roles,
};
/// <summary>
/// Boots a real host through the production <see cref="ServiceCollectionExtensions.WithOtOpcUaClusterBootstrap"/>
/// entry point (mirrors <c>SplitBrainResolverActivationTests</c>) — <see cref="ClusterRoleInfo"/>'s
/// constructor needs a real clustered <see cref="ActorSystem"/> to subscribe to cluster events,
/// even though the properties under test never touch live membership — resolves
/// <see cref="IClusterRoleInfo"/> from DI, and returns its <c>(ClusterRole, ClusterId)</c> pair
/// after tearing the host down. Both start/stop live in this helper rather than the test methods
/// so the CancellationToken-less calls stay off xunit1051's radar (mirrors
/// <c>SplitBrainResolverActivationTests.EffectiveConfigAsync</c>).
/// </summary>
private static async Task<(string? ClusterRole, string? ClusterId)> ResolveAsync(AkkaClusterOptions options)
{
var builder = Host.CreateDefaultBuilder();
builder.ConfigureServices(services =>
{
services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(options));
services.AddAkka("otopcua-role-info-test", (ab, sp) => ab.WithOtOpcUaClusterBootstrap(sp));
services.AddSingleton<IClusterRoleInfo, ClusterRoleInfo>();
});
using var host = builder.Build();
await host.StartAsync();
try
{
var roleInfo = host.Services.GetRequiredService<IClusterRoleInfo>();
return (roleInfo.ClusterRole, roleInfo.ClusterId);
}
finally
{
await host.StopAsync();
}
}
/// <summary>A node configured with a single cluster-scoped role exposes it and its ClusterId.</summary>
[Fact]
public async Task Single_cluster_role_exposes_role_and_id()
{
var (clusterRole, clusterId) = await ResolveAsync(SampleOptions("driver", "cluster-SITE-A"));
clusterRole.ShouldBe("cluster-SITE-A");
clusterId.ShouldBe("SITE-A");
}
/// <summary>A node configured with only fixed roles carries no cluster identity.</summary>
[Fact]
public async Task No_cluster_role_configured_yields_null()
{
var (clusterRole, clusterId) = await ResolveAsync(SampleOptions("admin", "driver"));
clusterRole.ShouldBeNull();
clusterId.ShouldBeNull();
}
/// <summary>
/// More than one cluster-scoped role is a misconfiguration; the first configured wins rather
/// than throwing.
/// </summary>
[Fact]
public async Task Multiple_cluster_roles_first_wins()
{
var (clusterRole, clusterId) = await ResolveAsync(SampleOptions("driver", "cluster-MAIN", "cluster-SITE-B"));
clusterRole.ShouldBe("cluster-MAIN");
clusterId.ShouldBe("MAIN");
}
/// <summary>An empty role list carries no cluster identity either.</summary>
[Fact]
public async Task Empty_roles_yields_null()
{
var (clusterRole, clusterId) = await ResolveAsync(SampleOptions());
clusterRole.ShouldBeNull();
clusterId.ShouldBeNull();
}
}
@@ -57,4 +57,57 @@ public sealed class RoleParserTests
{
Should.Throw<ArgumentException>(() => RoleParser.Parse("admin,master"));
}
/// <summary>Verifies that a single cluster-scoped role is accepted.</summary>
[Theory]
[InlineData("cluster-MAIN")]
[InlineData("cluster-SITE-A")]
public void Cluster_role_accepted(string role)
{
RoleParser.Parse(role).ShouldBe(new[] { role.ToLowerInvariant() });
}
/// <summary>Verifies that cluster roles mix freely with the fixed roles.</summary>
[Fact]
public void Fixed_and_cluster_roles_mix()
{
RoleParser.Parse("driver,cluster-SITE-A").ShouldBe(new[] { "driver", "cluster-site-a" });
}
/// <summary>Verifies that a cluster role with an empty ClusterId ("cluster-") is rejected.</summary>
[Fact]
public void Empty_cluster_id_throws()
{
Should.Throw<ArgumentException>(() => RoleParser.Parse("cluster-"));
}
/// <summary>Verifies IsClusterRole returns true for valid cluster roles and false otherwise.</summary>
[Theory]
[InlineData("cluster-MAIN", true)]
[InlineData("cluster-SITE-A", true)]
[InlineData("cluster-", false)]
[InlineData("driver", false)]
[InlineData("admin", false)]
[InlineData("clusterx", false)]
public void IsClusterRole_classifies_correctly(string role, bool expected)
{
RoleParser.IsClusterRole(role).ShouldBe(expected);
}
/// <summary>Verifies ClusterIdFromRole extracts the ClusterId suffix.</summary>
[Fact]
public void ClusterIdFromRole_extracts_suffix()
{
RoleParser.ClusterIdFromRole("cluster-SITE-A").ShouldBe("SITE-A");
}
/// <summary>Verifies the three well-known role constants match the historical literal values.</summary>
[Fact]
public void Well_known_role_constants()
{
RoleParser.Admin.ShouldBe("admin");
RoleParser.Driver.ShouldBe("driver");
RoleParser.Dev.ShouldBe("dev");
RoleParser.ClusterRolePrefix.ShouldBe("cluster-");
}
}
@@ -0,0 +1,267 @@
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Options;
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
/// <summary>
/// A split-topology node (one carrying a <c>cluster-{ClusterId}</c> role) left on a DPS transport
/// mode does not error — the transport simply delivers nothing across the mesh boundary, and the
/// symptom is a deployment that never applies or telemetry that never arrives. These tests make
/// that misconfiguration a loud host-start failure instead. A node with no cluster-role is exempt
/// (legacy / single-mesh / test), and the validator is a no-op for it.
/// </summary>
public class SplitTopologyTransportValidatorTests
{
private static ValidateOptionsResult Validate(
string? meshMode,
string? telemetryMode,
string? telemetryDialMode,
params string[] roles)
{
var pairs = new Dictionary<string, string?>();
for (var i = 0; i < roles.Length; i++)
{
pairs[$"Cluster:Roles:{i}"] = roles[i];
}
if (meshMode is not null) pairs["MeshTransport:Mode"] = meshMode;
if (telemetryMode is not null) pairs["Telemetry:Mode"] = telemetryMode;
if (telemetryDialMode is not null) pairs["TelemetryDial:Mode"] = telemetryDialMode;
var configuration = new ConfigurationBuilder().AddInMemoryCollection(pairs).Build();
// The validated options instance is never read (all four values come from IConfiguration), so a
// default MeshTransportOptions is a fine stand-in.
return new SplitTopologyTransportValidator(configuration)
.Validate(MeshTransportOptions.SectionName, new MeshTransportOptions());
}
[Fact]
public void Cluster_role_driver_node_on_Dps_mesh_transport_fails()
{
// Red-before-green anchor: a split-topology node on DistributedPubSub cannot deliver commands.
var result = Validate(
meshMode: MeshTransportOptions.ModeDps,
telemetryMode: TelemetryOptions.ModeGrpc,
telemetryDialMode: null,
"driver", "cluster-SITE-A");
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("MeshTransport:Mode");
result.FailureMessage.ShouldContain(MeshTransportOptions.ModeClusterClient);
result.FailureMessage.ShouldContain("SITE-A");
}
[Fact]
public void Cluster_role_driver_node_fully_configured_passes()
{
Validate(
meshMode: MeshTransportOptions.ModeClusterClient,
telemetryMode: TelemetryOptions.ModeGrpc,
telemetryDialMode: null,
"driver", "cluster-SITE-A").Succeeded.ShouldBeTrue();
}
[Fact]
public void Cluster_role_admin_node_fully_configured_passes()
{
Validate(
meshMode: MeshTransportOptions.ModeClusterClient,
telemetryMode: null,
telemetryDialMode: TelemetryDialOptions.ModeGrpc,
"admin", "cluster-SITE-A").Succeeded.ShouldBeTrue();
}
[Fact]
public void Cluster_role_driver_node_on_Dps_telemetry_fails()
{
var result = Validate(
meshMode: MeshTransportOptions.ModeClusterClient,
telemetryMode: TelemetryOptions.ModeDps,
telemetryDialMode: null,
"driver", "cluster-SITE-A");
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("Telemetry:Mode");
result.FailureMessage.ShouldContain(TelemetryOptions.ModeGrpc);
}
[Fact]
public void Cluster_role_admin_node_on_Dps_telemetry_dial_fails()
{
var result = Validate(
meshMode: MeshTransportOptions.ModeClusterClient,
telemetryMode: null,
telemetryDialMode: TelemetryDialOptions.ModeDps,
"admin", "cluster-SITE-A");
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("TelemetryDial:Mode");
result.FailureMessage.ShouldContain(TelemetryDialOptions.ModeGrpc);
}
[Fact]
public void Fused_cluster_role_node_needs_all_three_to_pass()
{
Validate(
meshMode: MeshTransportOptions.ModeClusterClient,
telemetryMode: TelemetryOptions.ModeGrpc,
telemetryDialMode: TelemetryDialOptions.ModeGrpc,
"admin", "driver", "cluster-MAIN").Succeeded.ShouldBeTrue();
}
[Fact]
public void Fused_cluster_role_node_missing_telemetry_grpc_fails()
{
var result = Validate(
meshMode: MeshTransportOptions.ModeClusterClient,
telemetryMode: TelemetryOptions.ModeDps,
telemetryDialMode: TelemetryDialOptions.ModeGrpc,
"admin", "driver", "cluster-MAIN");
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("Telemetry:Mode");
}
[Fact]
public void Fused_cluster_role_node_missing_telemetry_dial_grpc_fails()
{
var result = Validate(
meshMode: MeshTransportOptions.ModeClusterClient,
telemetryMode: TelemetryOptions.ModeGrpc,
telemetryDialMode: TelemetryDialOptions.ModeDps,
"admin", "driver", "cluster-MAIN");
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("TelemetryDial:Mode");
}
[Fact]
public void Non_cluster_role_node_on_all_Dps_passes()
{
// The exemption that keeps the validator a no-op for every legacy / single-mesh / test node:
// no cluster-{ClusterId} role means the split-topology rule does not apply at all.
Validate(
meshMode: MeshTransportOptions.ModeDps,
telemetryMode: TelemetryOptions.ModeDps,
telemetryDialMode: TelemetryDialOptions.ModeDps,
"admin", "driver").Succeeded.ShouldBeTrue();
}
[Fact]
public void Node_with_no_roles_on_all_Dps_passes()
{
// Absolute default: nothing declared, nothing to enforce.
Validate(
meshMode: MeshTransportOptions.ModeDps,
telemetryMode: TelemetryOptions.ModeDps,
telemetryDialMode: TelemetryDialOptions.ModeDps).Succeeded.ShouldBeTrue();
}
[Fact]
public void Unset_modes_on_a_cluster_role_node_fail_as_the_Dps_default()
{
// A split node that never sets the mode keys inherits the Dps defaults — which the split
// topology cannot use. Leaving them unset must fail exactly as setting them to Dps would.
var result = Validate(
meshMode: null,
telemetryMode: null,
telemetryDialMode: null,
"admin", "driver", "cluster-MAIN");
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("MeshTransport:Mode");
}
[Fact]
public void Mode_matching_is_case_insensitive()
{
Validate(
meshMode: "clusterclient",
telemetryMode: "grpc",
telemetryDialMode: "grpc",
"admin", "driver", "cluster-MAIN").Succeeded.ShouldBeTrue();
}
[Fact]
public void Cluster_role_from_OTOPCUA_ROLES_only_on_Dps_fails()
{
// Roles-source divergence: Cluster:Roles is empty, so the node's effective Akka roles come from
// OTOPCUA_ROLES (via RoleParser.Parse) exactly as Program.cs resolves them. A validator that
// read only Cluster:Roles would see no cluster role and pass silently on Dps — the documented
// production-incident shape.
using (new EnvVarScope("OTOPCUA_ROLES", "driver,cluster-SITE-A"))
{
var pairs = new Dictionary<string, string?> { ["MeshTransport:Mode"] = MeshTransportOptions.ModeDps };
var configuration = new ConfigurationBuilder().AddInMemoryCollection(pairs).Build();
var result = new SplitTopologyTransportValidator(configuration)
.Validate(MeshTransportOptions.SectionName, new MeshTransportOptions());
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("MeshTransport:Mode");
}
}
[Fact]
public void Cluster_role_from_OTOPCUA_ROLES_is_ignored_when_Cluster_Roles_is_set()
{
// When Cluster:Roles is populated it wins outright — the OTOPCUA_ROLES fallback is not consulted,
// mirroring AkkaClusterOptions.Roles. A non-cluster Cluster:Roles therefore stays exempt even if
// a stray OTOPCUA_ROLES carries a cluster role.
using (new EnvVarScope("OTOPCUA_ROLES", "driver,cluster-SITE-A"))
{
Validate(
meshMode: MeshTransportOptions.ModeDps,
telemetryMode: MeshTransportOptions.ModeDps,
telemetryDialMode: MeshTransportOptions.ModeDps,
"admin", "driver").Succeeded.ShouldBeTrue();
}
}
[Fact]
public void Mixed_case_cluster_and_role_names_on_Dps_fail()
{
// Case-sensitivity: the Cluster:Roles bind path is verbatim, so mixed-case entries must be
// normalized here or they slip past the ordinal IsClusterRole / driver checks and pass silently.
var result = Validate(
meshMode: MeshTransportOptions.ModeDps,
telemetryMode: TelemetryOptions.ModeGrpc,
telemetryDialMode: null,
"Driver", "Cluster-SITE-A");
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("MeshTransport:Mode");
}
[Fact]
public void Cluster_role_node_with_neither_admin_nor_driver_only_needs_ClusterClient()
{
// A cluster-role node that is neither admin nor driver has no telemetry surface to constrain —
// only the always-on MeshTransport=ClusterClient rule applies. Telemetry/TelemetryDial left on
// their Dps defaults must not fail it.
Validate(
meshMode: MeshTransportOptions.ModeClusterClient,
telemetryMode: null,
telemetryDialMode: null,
"cluster-SITE-A").Succeeded.ShouldBeTrue();
}
/// <summary>Sets an environment variable for the scope's lifetime and restores its prior value on dispose.</summary>
private sealed class EnvVarScope : IDisposable
{
private readonly string _name;
private readonly string? _previous;
public EnvVarScope(string name, string? value)
{
_name = name;
_previous = Environment.GetEnvironmentVariable(name);
Environment.SetEnvironmentVariable(name, value);
}
public void Dispose() => Environment.SetEnvironmentVariable(_name, _previous);
}
}
@@ -21,7 +21,10 @@ COPY profiles/ /fixtures/
# compose profile. See Docker/README.md §exception injection.
COPY exception_injector.py /fixtures/
EXPOSE 5020
# 5020 = the shared port for the standard/dl205/mitsubishi/s7_1500/exception_injection
# profiles (only one binds it at a time); 5021 = the rtu_over_tcp profile, which co-runs
# with standard. Image-metadata honesty only — compose's ports: mapping doesn't need EXPOSE.
EXPOSE 5020 5021
# Default to the standard profile; docker-compose.yml overrides per service.
# --http_port intentionally omitted; pymodbus 3.13's web UI binds on a
@@ -2,15 +2,16 @@
The Modbus driver's integration tests talk to a
[`pymodbus`](https://pymodbus.readthedocs.io/) simulator running as a
pinned Docker container. One image, per-profile service in compose, same
port binding (`5020`) regardless of which profile is live. Docker is the
only supported launch path — a fresh clone needs Docker Desktop and
nothing else.
pinned Docker container. One image, per-profile service in compose. Most
profiles bind `:5020`, so only one of *those* runs at a time; the
`rtu_over_tcp` profile binds `:5021` and is designed to co-run alongside
`standard`. Docker is the only supported launch path — a fresh clone needs
Docker Desktop and nothing else.
| File | Purpose |
|---|---|
| [`Dockerfile`](Dockerfile) | `python:3.12-slim-bookworm` + `pymodbus[simulator]==3.13.0` + every profile JSON + `exception_injector.py` |
| [`docker-compose.yml`](docker-compose.yml) | One service per profile (`standard` / `dl205` / `mitsubishi` / `s7_1500` / `exception_injection`); all bind `:5020` so only one runs at a time |
| [`docker-compose.yml`](docker-compose.yml) | One service per profile (`standard` / `dl205` / `mitsubishi` / `s7_1500` / `exception_injection` bind `:5020`, so only one of those runs at a time; `rtu_over_tcp` binds `:5021` and can run alongside `standard`) |
| [`profiles/*.json`](profiles/) | Same seed-register definitions the native launcher uses — canonical source |
| [`exception_injector.py`](exception_injector.py) | Pure-stdlib Modbus/TCP server that emits arbitrary exception codes per rule — used by the `exception_injection` profile |
@@ -43,10 +44,11 @@ docker compose -f tests\...\Docker\docker-compose.yml --profile dl205 up -d
docker compose -f tests\...\Docker\docker-compose.yml --profile dl205 down
```
Only one profile binds `:5020` at a time; switch by stopping the current
service + starting another. The integration tests discriminate by a
separate `MODBUS_SIM_PROFILE` env var so they skip correctly when the
wrong profile is live.
Only one of the `:5020` profiles binds at a time; switch by stopping the
current service + starting another. (`rtu_over_tcp` is the exception — it
binds `:5021` and co-runs with `standard`.) The integration tests
discriminate by a separate `MODBUS_SIM_PROFILE` env var so they skip
correctly when the wrong profile is live.
> **⚠️ Profile-cycling gotcha (bit us in the 2026-07 sweep — see
> `archreview/plans/INTEGRATION-SWEEP-STATUS.md`).** Each profile is a
@@ -78,6 +78,30 @@ services:
"--json_file", "/fixtures/s7_1500.json"
]
# RTU-over-TCP profile. Same pymodbus simulator, but the server is framed
# RTU (framer=rtu) instead of socket/MBAP — this is what a serial→Ethernet
# gateway presents. Binds :5021 (NOT the shared :5020) so it co-runs with
# the `standard` profile: two families, two ports, no conflict. Serves
# rtu_over_tcp.json (byte-for-byte standard.json + framer/port swap).
rtu_over_tcp:
profiles: ["rtu_over_tcp"]
image: otopcua-pymodbus:3.13.0
build:
context: .
dockerfile: Dockerfile
container_name: otopcua-pymodbus-rtu_over_tcp
labels:
project: lmxopcua
restart: "no"
ports:
- "5021:5021"
command: [
"pymodbus.simulator",
"--modbus_server", "srv",
"--modbus_device", "dev",
"--json_file", "/fixtures/rtu_over_tcp.json"
]
# Exception-injection profile. Runs the standalone pure-stdlib Modbus/TCP
# server shipped as exception_injector.py instead of the pymodbus
# simulator — pymodbus naturally emits only exception codes 02 + 03, and
@@ -0,0 +1,97 @@
{
"_comment": "rtu_over_tcp.json — RTU-over-TCP Modbus server for the integration suite (RTU framing tunnelled over a socket, as a serial→Ethernet gateway presents it). Byte-for-byte standard.json EXCEPT the server block: framer=\"rtu\" (not \"socket\") + port 5021 (not 5020), so it co-runs with the standard profile on :5020. pymodbus 3.13's simulator honors framer=rtu on a TCP server — confirmed on the wire (controller spike, 2026-07-24): raw RTU FC03 read of HR[5] returned `01 03 02 00 05 78 47`, a pure RTU frame with no MBAP header. Layout is identical to standard.json: HR[0..31]=address-as-value, HR[100]=auto-increment, HR[200..209]=scratch, coils 1024..1055=alternating, coils 1100..1109=scratch. NOTE: pymodbus rejects unknown keys at device-list / setup level; explanatory comments live in the README + git history.",
"server_list": {
"srv": {
"comm": "tcp",
"host": "0.0.0.0",
"port": 5021,
"framer": "rtu",
"device_id": 1
}
},
"device_list": {
"dev": {
"setup": {
"co size": 2048,
"di size": 2048,
"hr size": 2048,
"ir size": 2048,
"shared blocks": true,
"type exception": false,
"defaults": {
"value": {"bits": 0, "uint16": 0, "uint32": 0, "float32": 0.0, "string": " "},
"action": {"bits": null, "uint16": null, "uint32": null, "float32": null, "string": null}
}
},
"invalid": [],
"write": [
[0, 31],
[100, 100],
[200, 209],
[1024, 1055],
[1100, 1109]
],
"uint16": [
{"addr": 0, "value": 0}, {"addr": 1, "value": 1},
{"addr": 2, "value": 2}, {"addr": 3, "value": 3},
{"addr": 4, "value": 4}, {"addr": 5, "value": 5},
{"addr": 6, "value": 6}, {"addr": 7, "value": 7},
{"addr": 8, "value": 8}, {"addr": 9, "value": 9},
{"addr": 10, "value": 10}, {"addr": 11, "value": 11},
{"addr": 12, "value": 12}, {"addr": 13, "value": 13},
{"addr": 14, "value": 14}, {"addr": 15, "value": 15},
{"addr": 16, "value": 16}, {"addr": 17, "value": 17},
{"addr": 18, "value": 18}, {"addr": 19, "value": 19},
{"addr": 20, "value": 20}, {"addr": 21, "value": 21},
{"addr": 22, "value": 22}, {"addr": 23, "value": 23},
{"addr": 24, "value": 24}, {"addr": 25, "value": 25},
{"addr": 26, "value": 26}, {"addr": 27, "value": 27},
{"addr": 28, "value": 28}, {"addr": 29, "value": 29},
{"addr": 30, "value": 30}, {"addr": 31, "value": 31},
{"addr": 100, "value": 0,
"action": "increment",
"parameters": {"minval": 0, "maxval": 65535}},
{"addr": 200, "value": 0}, {"addr": 201, "value": 0},
{"addr": 202, "value": 0}, {"addr": 203, "value": 0},
{"addr": 204, "value": 0}, {"addr": 205, "value": 0},
{"addr": 206, "value": 0}, {"addr": 207, "value": 0},
{"addr": 208, "value": 0}, {"addr": 209, "value": 0}
],
"bits": [
{"addr": 1024, "value": 1}, {"addr": 1025, "value": 0},
{"addr": 1026, "value": 1}, {"addr": 1027, "value": 0},
{"addr": 1028, "value": 1}, {"addr": 1029, "value": 0},
{"addr": 1030, "value": 1}, {"addr": 1031, "value": 0},
{"addr": 1032, "value": 1}, {"addr": 1033, "value": 0},
{"addr": 1034, "value": 1}, {"addr": 1035, "value": 0},
{"addr": 1036, "value": 1}, {"addr": 1037, "value": 0},
{"addr": 1038, "value": 1}, {"addr": 1039, "value": 0},
{"addr": 1040, "value": 1}, {"addr": 1041, "value": 0},
{"addr": 1042, "value": 1}, {"addr": 1043, "value": 0},
{"addr": 1044, "value": 1}, {"addr": 1045, "value": 0},
{"addr": 1046, "value": 1}, {"addr": 1047, "value": 0},
{"addr": 1048, "value": 1}, {"addr": 1049, "value": 0},
{"addr": 1050, "value": 1}, {"addr": 1051, "value": 0},
{"addr": 1052, "value": 1}, {"addr": 1053, "value": 0},
{"addr": 1054, "value": 1}, {"addr": 1055, "value": 0},
{"addr": 1100, "value": 0}, {"addr": 1101, "value": 0},
{"addr": 1102, "value": 0}, {"addr": 1103, "value": 0},
{"addr": 1104, "value": 0}, {"addr": 1105, "value": 0},
{"addr": 1106, "value": 0}, {"addr": 1107, "value": 0},
{"addr": 1108, "value": 0}, {"addr": 1109, "value": 0}
],
"uint32": [],
"float32": [],
"string": [],
"repeat": []
}
}
}
@@ -0,0 +1,82 @@
using System.Net.Sockets;
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests;
/// <summary>
/// Reachability probe for the RTU-over-TCP Modbus simulator (pymodbus in Docker with
/// <c>framer=rtu</c>, see <c>Docker/docker-compose.yml</c> profile <c>rtu_over_tcp</c>) or a
/// real serial→Ethernet Modbus gateway. Mirrors <see cref="ModbusSimulatorFixture"/> but reads
/// <c>MODBUS_RTU_SIM_ENDPOINT</c> (default <c>10.100.0.35:5021</c> — the shared Docker host, a
/// distinct port from the standard profile's <c>:5020</c> so the two fixtures co-run). TCP-connects
/// once at fixture construction; each test checks <see cref="SkipReason"/> and calls
/// <c>Assert.Skip</c> when the endpoint was unreachable, so a dev box without a running simulator
/// still passes `dotnet test` cleanly.
/// </summary>
/// <remarks>
/// Same one-shot-probe discipline as <see cref="ModbusSimulatorFixture"/>: the probe socket is
/// not held for the life of the fixture (tests open their own <see cref="ModbusRtuOverTcpTransport"/>
/// against the same endpoint), and the fixture is a collection fixture so the probe runs once per
/// session rather than per test.
/// </remarks>
public sealed class ModbusRtuOverTcpFixture : IAsyncDisposable
{
// :5021 (not the standard profile's :5020) so the rtu_over_tcp container can co-run with the
// standard container on the shared Docker host. 10.100.0.35 = the shared Docker host (see
// CLAUDE.md "Docker Workflow"). Override with MODBUS_RTU_SIM_ENDPOINT to point at a real
// serial→Ethernet Modbus gateway, or a locally-running container.
private const string DefaultEndpoint = "10.100.0.35:5021";
private const string EndpointEnvVar = "MODBUS_RTU_SIM_ENDPOINT";
/// <summary>Gets the host address of the RTU-over-TCP Modbus simulator.</summary>
public string Host { get; }
/// <summary>Gets the port of the RTU-over-TCP Modbus simulator.</summary>
public int Port { get; }
/// <summary>Gets the skip reason if the simulator is unreachable; otherwise null.</summary>
public string? SkipReason { get; }
/// <summary>Initializes a new instance of the <see cref="ModbusRtuOverTcpFixture"/> class.</summary>
public ModbusRtuOverTcpFixture()
{
var raw = Environment.GetEnvironmentVariable(EndpointEnvVar) ?? DefaultEndpoint;
var parts = raw.Split(':', 2);
Host = parts[0];
Port = parts.Length == 2 && int.TryParse(parts[1], out var p) ? p : 502;
try
{
// Force IPv4 on the probe — pymodbus binds 0.0.0.0 (IPv4 only) while .NET default-resolves
// "localhost" → IPv6 ::1 first and only then tries IPv4, surfacing as a 2s timeout under
// .NET 10. Resolving + dialing explicit IPv4 sidesteps the dual-stack ordering. (Same
// rationale as ModbusSimulatorFixture.)
using var client = new TcpClient(System.Net.Sockets.AddressFamily.InterNetwork);
var task = client.ConnectAsync(
System.Net.Dns.GetHostAddresses(Host)
.FirstOrDefault(a => a.AddressFamily == System.Net.Sockets.AddressFamily.InterNetwork)
?? System.Net.IPAddress.Loopback,
Port);
if (!task.Wait(TimeSpan.FromSeconds(2)) || !client.Connected)
{
SkipReason = $"RTU-over-TCP Modbus simulator at {Host}:{Port} did not accept a TCP connection within 2s. " +
$"Start the pymodbus Docker container (docker compose -f Docker/docker-compose.yml --profile rtu_over_tcp up -d --build) " +
$"or override {EndpointEnvVar}, then re-run.";
}
}
catch (Exception ex)
{
SkipReason = $"RTU-over-TCP Modbus simulator at {Host}:{Port} unreachable: {ex.GetType().Name}: {ex.Message}. " +
$"Start the pymodbus Docker container (docker compose -f Docker/docker-compose.yml --profile rtu_over_tcp up -d --build) " +
$"or override {EndpointEnvVar}, then re-run.";
}
}
/// <inheritdoc />
public ValueTask DisposeAsync() => ValueTask.CompletedTask;
}
[Xunit.CollectionDefinition(Name)]
public sealed class ModbusRtuOverTcpCollection : Xunit.ICollectionFixture<ModbusRtuOverTcpFixture>
{
public const string Name = "ModbusRtuOverTcp";
}
@@ -0,0 +1,94 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests;
/// <summary>
/// End-to-end round-trip against the <c>rtu_over_tcp.json</c> pymodbus profile (or a real
/// serial→Ethernet Modbus gateway when <c>MODBUS_RTU_SIM_ENDPOINT</c> points at one), driving the
/// full <see cref="ModbusDriver"/> + real <see cref="ModbusRtuOverTcpTransport"/> stack with
/// <see cref="ModbusTransportMode.RtuOverTcp"/>. Proves the driver reads a seeded holding register
/// and writes+reads-back a scratch register when the wire carries RTU framing
/// (<c>[addr][PDU][CRC-16]</c>, no MBAP header) rather than Modbus/TCP.
/// </summary>
/// <remarks>
/// pymodbus 3.13's simulator honors <c>framer=rtu</c> on a TCP server — confirmed on the wire
/// (controller spike, 2026-07-24): a raw RTU FC03 read of HR[5] returned
/// <c>01 03 02 00 05 78 47</c>, a pure RTU frame with no MBAP header (data 0x0005 = 5). So the
/// fixture is the plain pymodbus simulator with framer/port swapped — no stdlib fallback server.
/// </remarks>
[Collection(ModbusRtuOverTcpCollection.Name)]
[Trait("Category", "Integration")]
[Trait("Device", "RtuOverTcp")]
public sealed class ModbusRtuOverTcpTests(ModbusRtuOverTcpFixture sim)
{
/// <summary>Reads a seeded holding register (HR[5]=5) over RTU-over-TCP framing.</summary>
[Fact]
public async Task Reads_a_seeded_holding_register_over_rtu_framing()
{
if (sim.SkipReason is not null) Assert.Skip(sim.SkipReason);
var opts = new ModbusDriverOptions
{
Host = sim.Host,
Port = sim.Port,
UnitId = 1,
Transport = ModbusTransportMode.RtuOverTcp,
Timeout = TimeSpan.FromSeconds(2),
Probe = new ModbusProbeOptions { Enabled = false },
RawTags = ModbusRawTags.Entries([
new ModbusTagDefinition(
Name: "HR5",
Region: ModbusRegion.HoldingRegisters,
Address: 5,
DataType: ModbusDataType.UInt16,
Writable: false),
]),
};
await using var driver = new ModbusDriver(opts, driverInstanceId: "modbus-rtu-int");
await driver.InitializeAsync(driverConfigJson: "{}", TestContext.Current.CancellationToken);
var results = await driver.ReadAsync(["HR5"], TestContext.Current.CancellationToken);
results.Count.ShouldBe(1);
results[0].StatusCode.ShouldBe(0u); // Good
results[0].Value.ShouldBe((ushort)5); // HR[5] seeded = address-as-value
}
/// <summary>Writes then reads back a scratch holding register (HR[200]) over RTU-over-TCP framing.</summary>
[Fact]
public async Task Writes_and_reads_back_a_holding_register_over_rtu_framing()
{
if (sim.SkipReason is not null) Assert.Skip(sim.SkipReason);
var opts = new ModbusDriverOptions
{
Host = sim.Host,
Port = sim.Port,
UnitId = 1,
Transport = ModbusTransportMode.RtuOverTcp,
Timeout = TimeSpan.FromSeconds(2),
Probe = new ModbusProbeOptions { Enabled = false },
RawTags = ModbusRawTags.Entries([
new ModbusTagDefinition(
Name: "HR200",
Region: ModbusRegion.HoldingRegisters,
Address: 200,
DataType: ModbusDataType.UInt16,
Writable: true),
]),
};
await using var driver = new ModbusDriver(opts, driverInstanceId: "modbus-rtu-int-w");
await driver.InitializeAsync(driverConfigJson: "{}", TestContext.Current.CancellationToken);
var writes = await driver.WriteAsync(
[new WriteRequest("HR200", (ushort)4242)],
TestContext.Current.CancellationToken);
writes.Count.ShouldBe(1);
writes[0].StatusCode.ShouldBe(0u);
var back = await driver.ReadAsync(["HR200"], TestContext.Current.CancellationToken);
back.Count.ShouldBe(1);
back[0].StatusCode.ShouldBe(0u);
back[0].Value.ShouldBe((ushort)4242);
}
}
@@ -0,0 +1,27 @@
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests;
public sealed class ModbusCrcTests
{
// Known CRC-16/MODBUS vectors (init 0xFFFF, poly 0xA001). CRC returned as ushort;
// on the wire it is appended low-byte-first.
[Theory]
// FC03 read HR: unit 1, addr 0, qty 1 -> CRC 0x0A84 (bytes 84 0A on the wire)
[InlineData(new byte[] { 0x01, 0x03, 0x00, 0x00, 0x00, 0x01 }, (ushort)0x0A84)]
// "123456789" canonical check value for CRC-16/MODBUS = 0x4B37
[InlineData(new byte[] { 0x31,0x32,0x33,0x34,0x35,0x36,0x37,0x38,0x39 }, (ushort)0x4B37)]
public void Compute_matches_known_vectors(byte[] frame, ushort expected)
=> ModbusCrc.Compute(frame).ShouldBe(expected);
[Fact]
public void AppendLowByteFirst_writes_lo_then_hi()
{
var body = new byte[] { 0x01, 0x03, 0x00, 0x00, 0x00, 0x01 };
var withCrc = ModbusCrc.Append(body);
withCrc.Length.ShouldBe(body.Length + 2);
withCrc[^2].ShouldBe((byte)0x84); // CRC low byte
withCrc[^1].ShouldBe((byte)0x0A); // CRC high byte
}
}
@@ -14,7 +14,7 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests;
/// (2) Sub-second <see cref="TimeSpan"/> values on <c>ModbusKeepAliveOptions.Time</c> /
/// <c>Interval</c> — the int-cast in <c>EnableKeepAlive</c> truncated <c>500&#160;ms</c> to
/// <c>0</c>, which most OSes interpret as "use the default", silently defeating the
/// configured timing. <c>ModbusTcpTransport.ClampToWholeSeconds</c> rounds up to a minimum
/// configured timing. <c>ModbusSocketLifecycle.ClampToWholeSeconds</c> rounds up to a minimum
/// of 1&#160;second.
/// </summary>
[Trait("Category", "Unit")]
@@ -70,7 +70,7 @@ public sealed class ModbusEdgeCaseValidationTests
[InlineData(60_000, 60)]
public void ClampToWholeSeconds_rounds_up_to_at_least_one_second(int ms, int expected)
{
ModbusTcpTransport.ClampToWholeSeconds(TimeSpan.FromMilliseconds(ms)).ShouldBe(expected);
ModbusSocketLifecycle.ClampToWholeSeconds(TimeSpan.FromMilliseconds(ms)).ShouldBe(expected);
}
/// <summary>Verifies that negative time spans are treated as one second.</summary>
@@ -80,6 +80,6 @@ public sealed class ModbusEdgeCaseValidationTests
// Defensive — operators occasionally configure a negative TimeSpan thinking it disables
// the feature. The OS would reject the negative int — clamping to 1 keeps the socket
// valid until the operator fixes the config.
ModbusTcpTransport.ClampToWholeSeconds(TimeSpan.FromSeconds(-5)).ShouldBe(1);
ModbusSocketLifecycle.ClampToWholeSeconds(TimeSpan.FromSeconds(-5)).ShouldBe(1);
}
}
@@ -0,0 +1,134 @@
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests;
public sealed class ModbusRtuFramingTests
{
private static MemoryStream Canned(params byte[] frame) => new(frame);
[Fact]
public void BuildAdu_prefixes_unit_and_appends_crc()
{
var adu = ModbusRtuFraming.BuildAdu(0x01, new byte[] { 0x03, 0x00, 0x00, 0x00, 0x01 });
adu.ShouldBe(new byte[] { 0x01, 0x03, 0x00, 0x00, 0x00, 0x01, 0x84, 0x0A });
}
[Fact]
public async Task ReadResponse_FC03_returns_bare_pdu()
{
// addr=01 fc=03 byteCount=02 data=00 0A + CRC(lo,hi)
var body = new byte[] { 0x01, 0x03, 0x02, 0x00, 0x0A };
var frame = ModbusCrc.Append(body);
await using var s = Canned(frame);
var pdu = await ModbusRtuFraming.ReadResponsePduAsync(s, 0x01, TestContext.Current.CancellationToken);
pdu.ShouldBe(new byte[] { 0x03, 0x02, 0x00, 0x0A }); // fc + byteCount + data, no addr/CRC
}
[Fact]
public async Task ReadResponse_exception_frame_throws_ModbusException()
{
// addr=01 fc=0x83 exc=0x02 + CRC
var frame = ModbusCrc.Append(new byte[] { 0x01, 0x83, 0x02 });
await using var s = Canned(frame);
var ex = await Should.ThrowAsync<ModbusException>(
ModbusRtuFraming.ReadResponsePduAsync(s, 0x01, TestContext.Current.CancellationToken));
ex.FunctionCode.ShouldBe((byte)0x03);
ex.ExceptionCode.ShouldBe((byte)0x02);
}
[Fact]
public async Task ReadResponse_bad_crc_throws_desync()
{
var frame = ModbusCrc.Append(new byte[] { 0x01, 0x03, 0x02, 0x00, 0x0A });
frame[^1] ^= 0xFF; // corrupt CRC high byte
await using var s = Canned(frame);
var ex = await Should.ThrowAsync<ModbusTransportDesyncException>(
ModbusRtuFraming.ReadResponsePduAsync(s, 0x01, TestContext.Current.CancellationToken));
ex.Reason.ShouldBe(ModbusTransportDesyncException.DesyncReason.CrcMismatch);
}
[Fact]
public async Task ReadResponse_FC06_write_echo_returns_four_byte_pdu()
{
// addr=01 fc=06 addr=00 07 value=00 2A + CRC -> PDU = 06 00 07 00 2A
var frame = ModbusCrc.Append(new byte[] { 0x01, 0x06, 0x00, 0x07, 0x00, 0x2A });
await using var s = Canned(frame);
var pdu = await ModbusRtuFraming.ReadResponsePduAsync(s, 0x01, TestContext.Current.CancellationToken);
pdu.ShouldBe(new byte[] { 0x06, 0x00, 0x07, 0x00, 0x2A });
}
[Fact]
public async Task ReadResponse_wrong_unit_address_throws_desync()
{
// CRC-valid FC03 frame addressed to unit 0x02, but we asked for 0x01 -> unit-mismatch desync.
var frame = ModbusCrc.Append(new byte[] { 0x02, 0x03, 0x02, 0x00, 0x0A });
await using var s = Canned(frame);
var ex = await Should.ThrowAsync<ModbusTransportDesyncException>(
ModbusRtuFraming.ReadResponsePduAsync(s, 0x01, TestContext.Current.CancellationToken));
ex.Reason.ShouldBe(ModbusTransportDesyncException.DesyncReason.UnitMismatch);
}
[Fact]
public async Task ReadResponse_FC03_deframes_correctly_when_delivered_in_dribbles()
{
// Same valid FC03 frame, but the stream hands back only 1-2 bytes per ReadAsync — proves
// the ReadExactlyAsync accumulation loop reassembles a length-less frame across many reads.
var frame = ModbusCrc.Append(new byte[] { 0x01, 0x03, 0x02, 0x00, 0x0A });
await using var s = new DribbleStream(frame, chunk: 2);
var pdu = await ModbusRtuFraming.ReadResponsePduAsync(s, 0x01, TestContext.Current.CancellationToken);
pdu.ShouldBe(new byte[] { 0x03, 0x02, 0x00, 0x0A });
}
[Fact]
public async Task ReadResponse_stream_ends_early_throws_desync()
{
// Only the first 4 bytes of a 7-byte FC03 frame are available, then EOF (ReadAsync returns 0)
// -> the truncation path in ReadExactlyAsync must surface a desync, not hang or mis-parse.
var frame = ModbusCrc.Append(new byte[] { 0x01, 0x03, 0x02, 0x00, 0x0A });
var truncated = frame[..4];
await using var s = new DribbleStream(truncated, chunk: 2);
var ex = await Should.ThrowAsync<ModbusTransportDesyncException>(
ModbusRtuFraming.ReadResponsePduAsync(s, 0x01, TestContext.Current.CancellationToken));
ex.Reason.ShouldBe(ModbusTransportDesyncException.DesyncReason.TruncatedFrame);
}
/// <summary>
/// A read-only test double that dribbles its backing bytes out at most <c>chunk</c> at a time
/// per <see cref="ReadAsync(Memory{byte}, CancellationToken)"/> call, then returns 0 (EOF)
/// once exhausted. Exercises the partial-read accumulation loop and the truncation → desync
/// path that a single-shot <see cref="MemoryStream"/> never reaches.
/// </summary>
private sealed class DribbleStream(byte[] data, int chunk) : Stream
{
private int _pos;
public override int Read(byte[] buffer, int offset, int count)
{
var n = Math.Min(Math.Min(chunk, count), data.Length - _pos);
if (n <= 0) return 0;
Array.Copy(data, _pos, buffer, offset, n);
_pos += n;
return n;
}
public override ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken cancellationToken = default)
{
var n = Math.Min(Math.Min(chunk, buffer.Length), data.Length - _pos);
if (n <= 0) return ValueTask.FromResult(0);
data.AsSpan(_pos, n).CopyTo(buffer.Span);
_pos += n;
return ValueTask.FromResult(n);
}
public override bool CanRead => true;
public override bool CanSeek => false;
public override bool CanWrite => false;
public override long Length => data.Length;
public override long Position { get => _pos; set => throw new NotSupportedException(); }
public override void Flush() { }
public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException();
public override void SetLength(long value) => throw new NotSupportedException();
public override void Write(byte[] buffer, int offset, int count) => throw new NotSupportedException();
}
}
@@ -0,0 +1,110 @@
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests;
/// <summary>
/// Exercises <see cref="ModbusRtuOverTcpTransport"/>'s send/receive orchestration, single-flight,
/// and per-op deadline against an in-memory duplex fake injected through the <c>ForTest</c> seam —
/// no real socket. The fake records the request ADU the transport writes and replays a canned RTU
/// response on read, or blocks forever (honouring cancellation) when <c>stall</c> is set.
/// </summary>
[Trait("Category", "Unit")]
public sealed class ModbusRtuOverTcpTransportTests
{
[Fact]
public async Task SendAsync_writes_rtu_adu_and_returns_deframed_pdu()
{
// Response the fake gateway will emit: FC03, 1 reg = 0x000A.
var response = ModbusCrc.Append(new byte[] { 0x01, 0x03, 0x02, 0x00, 0x0A });
var fake = new CapturingDuplexStream(response);
await using var transport = ModbusRtuOverTcpTransport.ForTest(fake, TimeSpan.FromSeconds(2));
var pdu = await transport.SendAsync(0x01, new byte[] { 0x03, 0x00, 0x00, 0x00, 0x01 },
TestContext.Current.CancellationToken);
pdu.ShouldBe(new byte[] { 0x03, 0x02, 0x00, 0x0A });
// The request went out as an RTU ADU: unit + pdu + CRC, NO MBAP header.
fake.Written.ShouldBe(new byte[] { 0x01, 0x03, 0x00, 0x00, 0x00, 0x01, 0x84, 0x0A });
}
[Fact]
public async Task SendAsync_exception_pdu_surfaces_ModbusException()
{
var response = ModbusCrc.Append(new byte[] { 0x01, 0x83, 0x02 });
var fake = new CapturingDuplexStream(response);
await using var transport = ModbusRtuOverTcpTransport.ForTest(fake, TimeSpan.FromSeconds(2));
await Should.ThrowAsync<ModbusException>(
transport.SendAsync(0x01, new byte[] { 0x03, 0x00, 0x00, 0x00, 0x01 },
TestContext.Current.CancellationToken));
}
[Fact]
public async Task SendAsync_stalled_gateway_hits_per_op_deadline()
{
var fake = new CapturingDuplexStream(respondBytes: Array.Empty<byte>(), stall: true);
await using var transport = ModbusRtuOverTcpTransport.ForTest(fake, TimeSpan.FromMilliseconds(200));
await Should.ThrowAsync<ModbusTransportDesyncException>(
transport.SendAsync(0x01, new byte[] { 0x03, 0x00, 0x00, 0x00, 0x01 },
TestContext.Current.CancellationToken));
}
/// <summary>
/// In-memory duplex <see cref="Stream"/> test double: records everything written and replays
/// <c>respondBytes</c> on read. When <c>stall</c> is set the read blocks until its cancellation
/// token fires (simulating a frozen gateway) so the transport's per-op deadline is exercised.
/// </summary>
private sealed class CapturingDuplexStream : Stream
{
private readonly byte[] _response;
private readonly bool _stall;
private readonly MemoryStream _written = new();
private int _readPos;
public CapturingDuplexStream(byte[] respondBytes, bool stall = false)
{
_response = respondBytes;
_stall = stall;
}
/// <summary>Gets the bytes the transport wrote to this stream (the request ADU).</summary>
public byte[] Written => _written.ToArray();
public override bool CanRead => true;
public override bool CanSeek => false;
public override bool CanWrite => true;
public override long Length => throw new NotSupportedException();
public override long Position { get => throw new NotSupportedException(); set => throw new NotSupportedException(); }
public override async ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken ct = default)
{
if (_stall)
{
// Frozen gateway: never answer. Block until the per-op deadline (or caller) cancels.
var tcs = new TaskCompletionSource();
await using (ct.Register(() => tcs.TrySetCanceled(ct)))
await tcs.Task.ConfigureAwait(false);
return 0; // unreachable — the await above always throws when cancelled
}
var remaining = _response.Length - _readPos;
if (remaining <= 0) return 0;
var n = Math.Min(remaining, buffer.Length);
_response.AsSpan(_readPos, n).CopyTo(buffer.Span);
_readPos += n;
return n;
}
public override async ValueTask WriteAsync(ReadOnlyMemory<byte> buffer, CancellationToken ct = default)
{
await _written.WriteAsync(buffer, ct).ConfigureAwait(false);
}
public override Task FlushAsync(CancellationToken ct) => Task.CompletedTask;
public override void Flush() { }
public override int Read(byte[] buffer, int offset, int count) => throw new NotSupportedException();
public override void Write(byte[] buffer, int offset, int count) => throw new NotSupportedException();
public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException();
public override void SetLength(long value) => throw new NotSupportedException();
}
}
@@ -0,0 +1,33 @@
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests;
/// <summary>
/// Pins the socket-lifecycle surface extracted from <see cref="ModbusTcpTransport"/> into the
/// reusable <see cref="ModbusSocketLifecycle"/> (Task 3). The clamp helper moved with it, and a
/// connect to a dead port must still surface the underlying socket failure.
/// </summary>
[Trait("Category", "Unit")]
public sealed class ModbusSocketLifecycleTests
{
/// <summary>Verifies the whole-seconds clamp rounds sub-second/negative durations up to a minimum of one.</summary>
/// <param name="seconds">The input duration in seconds.</param>
/// <param name="expected">The expected clamped value in whole seconds.</param>
[Theory]
[InlineData(0.4, 1)] // sub-second rounds up to 1 (the int-cast truncation guard)
[InlineData(2.0, 2)]
[InlineData(-5.0, 1)] // negative clamps to 1
public void ClampToWholeSeconds_rounds_up_min_one(double seconds, int expected)
=> ModbusSocketLifecycle.ClampToWholeSeconds(TimeSpan.FromSeconds(seconds)).ShouldBe(expected);
/// <summary>Verifies a connect to a dead port surfaces the underlying socket failure.</summary>
[Fact]
public async Task Connect_to_dead_port_surfaces_socket_failure()
{
var life = new ModbusSocketLifecycle("127.0.0.1", 1, TimeSpan.FromMilliseconds(300),
autoReconnect: false);
await Should.ThrowAsync<System.Net.Sockets.SocketException>(
life.ConnectAsync(TestContext.Current.CancellationToken));
}
}
@@ -0,0 +1,47 @@
using System.Text.Json;
using System.Text.Json.Serialization;
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests;
public sealed class ModbusTransportConfigRoundTripTests
{
// Mirrors the AdminUI ModbusDriverForm serializer: camelCase + string enums.
private static readonly JsonSerializerOptions _adminOpts = new()
{
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
Converters = { new JsonStringEnumConverter() },
};
[Fact]
public void Transport_serializes_as_a_string_not_a_number()
{
var opts = new ModbusDriverOptions { Host = "10.20.0.50", Port = 4001, Transport = ModbusTransportMode.RtuOverTcp };
var json = JsonSerializer.Serialize(opts, _adminOpts);
json.ShouldContain("\"transport\":\"RtuOverTcp\"");
json.ShouldNotContain("\"transport\":1", Case.Sensitive);
}
[Fact]
public void Factory_binds_RtuOverTcp_from_string_config()
{
const string json = """{ "host": "10.20.0.50", "port": 4001, "transport": "RtuOverTcp" }""";
using var driver = ModbusDriverFactoryExtensions.CreateInstance("mb-rtu", json);
var opts = (ModbusDriverOptions)typeof(ModbusDriver)
.GetField("_options", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance)!
.GetValue(driver)!;
opts.Transport.ShouldBe(ModbusTransportMode.RtuOverTcp);
}
[Fact]
public void Factory_defaults_Transport_to_Tcp_when_omitted()
{
const string json = """{ "host": "10.0.0.10" }""";
using var driver = ModbusDriverFactoryExtensions.CreateInstance("mb-tcp", json);
var opts = (ModbusDriverOptions)typeof(ModbusDriver)
.GetField("_options", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance)!
.GetValue(driver)!;
opts.Transport.ShouldBe(ModbusTransportMode.Tcp);
}
}
@@ -0,0 +1,27 @@
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests;
public sealed class ModbusTransportFactoryTests
{
[Fact]
public void Tcp_mode_builds_tcp_transport()
=> ModbusTransportFactory.Create(new ModbusDriverOptions { Transport = ModbusTransportMode.Tcp })
.ShouldBeOfType<ModbusTcpTransport>();
[Fact]
public void RtuOverTcp_mode_builds_rtu_transport()
=> ModbusTransportFactory.Create(new ModbusDriverOptions { Transport = ModbusTransportMode.RtuOverTcp })
.ShouldBeOfType<ModbusRtuOverTcpTransport>();
[Fact]
public void Default_options_build_tcp_transport()
=> ModbusTransportFactory.Create(new ModbusDriverOptions())
.ShouldBeOfType<ModbusTcpTransport>();
[Fact]
public void Unknown_transport_mode_throws()
=> Should.Throw<ArgumentOutOfRangeException>(
() => ModbusTransportFactory.Create(new ModbusDriverOptions { Transport = (ModbusTransportMode)999 }));
}
@@ -0,0 +1,15 @@
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests;
public sealed class ModbusTransportModeTests
{
[Fact]
public void Tcp_is_the_default_zero_member()
=> ((ModbusTransportMode)0).ShouldBe(ModbusTransportMode.Tcp);
[Fact]
public void Exactly_two_members_ship_in_v1()
=> Enum.GetNames<ModbusTransportMode>().ShouldBe(["Tcp", "RtuOverTcp"]);
}
@@ -0,0 +1,33 @@
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests;
/// <summary>
/// Confirms the driver's default transport-factory closure — and, by extension, the
/// probe's transport construction — routes through <see cref="ModbusTransportFactory"/>
/// instead of hardcoding <see cref="ModbusTcpTransport"/>, so an <c>RtuOverTcp</c>-authored
/// config actually gets RTU framing rather than silently running as TCP/MBAP.
/// </summary>
[Trait("Category", "Unit")]
public sealed class ModbusTransportWiringTests
{
private static IModbusTransport BuildDefaultTransport(ModbusDriverOptions opts)
{
using var driver = new ModbusDriver(opts, "wire-test");
var factory = (Func<ModbusDriverOptions, IModbusTransport>)typeof(ModbusDriver)
.GetField("_transportFactory", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance)!
.GetValue(driver)!;
return factory(opts);
}
[Fact]
public void Default_closure_builds_rtu_transport_for_RtuOverTcp()
=> BuildDefaultTransport(new ModbusDriverOptions { Transport = ModbusTransportMode.RtuOverTcp })
.ShouldBeOfType<ModbusRtuOverTcpTransport>();
[Fact]
public void Default_closure_builds_tcp_transport_for_Tcp()
=> BuildDefaultTransport(new ModbusDriverOptions())
.ShouldBeOfType<ModbusTcpTransport>();
}
@@ -61,4 +61,15 @@ public sealed class ModbusDriverFormModelTests
json.ShouldContain("\"family\":\"MELSEC\""); // enum-as-name (not a number), camelCase key
json.ShouldNotContain("\"family\":0", Case.Sensitive); // never the numeric enum form
}
[Fact]
public void Round_trip_preserves_Transport_mode()
{
var form = new ModbusDriverForm.FormModel { Transport = ModbusTransportMode.RtuOverTcp };
var json = JsonSerializer.Serialize(form.ToOptions(), JsonOpts);
var back = ModbusDriverForm.FormModel.FromOptions(
JsonSerializer.Deserialize<ModbusDriverOptions>(json, JsonOpts)!);
back.Transport.ShouldBe(ModbusTransportMode.RtuOverTcp);
json.ShouldContain("\"transport\":\"RtuOverTcp\""); // name string, never a number
}
}
@@ -70,24 +70,80 @@ public sealed class ClusterNodeAddressReconcilerActorTests : ControlPlaneActorTe
});
}
/// <summary>
/// Per-cluster mesh Phase 6: an admin node scoped to its OWN cluster ignores rows in a cluster
/// its mesh can no longer see. Scoped to <c>MAIN</c>, a <c>SITE-A</c> enabled row is filtered
/// out before <see cref="ClusterNodeAddressReconciler.Reconcile"/> ever sees it, so it produces
/// no <see cref="AddressMismatchKind.EnabledRowNotInCluster"/> warning. This is the false
/// positive the split would otherwise print forever.
/// </summary>
[Fact]
public void Row_in_a_foreign_cluster_is_ignored_when_scoped()
{
var dbFactory = NewInMemoryDbFactory();
SeedNode(dbFactory, "site-a-1:4053", enabled: true, clusterId: "SITE-A");
EventFilter.Warning(contains: "site-a-1:4053").Expect(0, TimeSpan.FromSeconds(2), () =>
{
Sys.ActorOf(ClusterNodeAddressReconcilerActor.Props(
dbFactory, ownClusterId: "MAIN", sweepInterval: TimeSpan.FromMilliseconds(400)));
ExpectNoMsg(TimeSpan.FromSeconds(1.5));
});
}
/// <summary>
/// Scoping does not silence the admin node's own cluster. Scoped to <c>MAIN</c>, an enabled
/// <c>MAIN</c> row with no matching driver member is still warned about — proving the filter
/// keeps in-cluster rows (and with them, within-cluster drift detection).
/// </summary>
[Fact]
public void Enabled_row_in_own_cluster_is_still_logged_when_scoped()
{
var dbFactory = NewInMemoryDbFactory();
SeedNode(dbFactory, "main-1:4053", enabled: true, clusterId: "MAIN");
EventFilter.Warning(contains: "main-1:4053").ExpectOne(TimeSpan.FromSeconds(10), () =>
Sys.ActorOf(ClusterNodeAddressReconcilerActor.Props(dbFactory, ownClusterId: "MAIN")));
}
/// <summary>
/// Legacy backward-compat: an admin node with no cluster role (null <c>ownClusterId</c>) still
/// reconciles the WHOLE fleet — such a node genuinely sees every member via gossip. A
/// <c>SITE-A</c> enabled row with no member is warned about, exactly as before the Phase 6 scope.
/// </summary>
[Fact]
public void Foreign_cluster_row_is_still_logged_when_unscoped()
{
var dbFactory = NewInMemoryDbFactory();
SeedNode(dbFactory, "site-a-1:4053", enabled: true, clusterId: "SITE-A");
EventFilter.Warning(contains: "site-a-1:4053").ExpectOne(TimeSpan.FromSeconds(10), () =>
Sys.ActorOf(ClusterNodeAddressReconcilerActor.Props(dbFactory, ownClusterId: null)));
}
private static void SeedNode(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, string nodeId, bool enabled)
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, string nodeId, bool enabled,
string clusterId = "SITE-A")
{
using var db = dbFactory.CreateDbContext();
db.ServerClusters.Add(new ServerCluster
if (!db.ServerClusters.Any(c => c.ClusterId == clusterId))
{
ClusterId = "SITE-A",
Name = "Site A",
Enterprise = "zb",
Site = "site-a",
NodeCount = 2,
RedundancyMode = RedundancyMode.Warm,
CreatedBy = "test",
});
db.ServerClusters.Add(new ServerCluster
{
ClusterId = clusterId,
Name = clusterId,
Enterprise = "zb",
Site = clusterId.ToLowerInvariant(),
NodeCount = 2,
RedundancyMode = RedundancyMode.Warm,
CreatedBy = "test",
});
}
db.ClusterNodes.Add(new ClusterNode
{
NodeId = nodeId,
ClusterId = "SITE-A",
ClusterId = clusterId,
Host = nodeId.Split(':')[0],
ApplicationUri = $"urn:OtOpcUa:{nodeId}",
Enabled = enabled,
@@ -1,3 +1,8 @@
using System.Collections.Immutable;
using System.Reflection;
using Akka.Actor;
using Akka.Cluster;
using Akka.Util;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
@@ -123,4 +128,90 @@ public sealed class ClusterNodeAddressReconcilerTests
[("SITE-A-1", 4053)], [Row("site-a-1", 4053)], ["site-a-1:4053"])
.ShouldBeEmpty();
}
private static int _memberUid;
// Akka's Member.Create factories are internal, so reflect the Up-status overload
// Create(UniqueAddress, int upNumber, MemberStatus, ImmutableHashSet<string> roles, AppVersion).
private static readonly MethodInfo MemberCreate = typeof(Member).GetMethod(
"Create",
BindingFlags.Static | BindingFlags.NonPublic | BindingFlags.Public,
binder: null,
types:
[
typeof(UniqueAddress), typeof(int), typeof(MemberStatus),
typeof(ImmutableHashSet<string>), typeof(AppVersion),
],
modifiers: null)
?? throw new InvalidOperationException("Akka.Cluster.Member.Create(5-arg) not found — Akka API changed");
/// <summary>
/// Builds an Up cluster <see cref="Member"/> at <paramref name="host"/>:<paramref name="port"/>
/// carrying <paramref name="roles"/>, for exercising
/// <see cref="ClusterNodeAddressReconcilerActor.FilterOwnClusterDriverMembers"/> directly.
/// </summary>
private static Member UpMember(string host, int port, params string[] roles) =>
(Member)MemberCreate.Invoke(null,
[
new UniqueAddress(new Address("akka.tcp", "otopcua", host, port), Interlocked.Increment(ref _memberUid)),
1,
MemberStatus.Up,
roles.ToImmutableHashSet(),
AppVersion.Zero,
])!;
/// <summary>
/// Per-cluster mesh Phase 6: scoped to <c>cluster-MAIN</c>, a driver member carrying a FOREIGN
/// cluster role (<c>cluster-SITE-A</c>) — the mixed-mesh window, where central still sees it via
/// gossip while its row is already filtered out by ClusterId — is excluded from observed. Left
/// in, it would hit the Error-level <see cref="AddressMismatchKind.RunningNodeHasNoRow"/> branch.
/// </summary>
[Fact]
public void Foreign_cluster_role_member_is_excluded_when_scoped()
{
var observed = ClusterNodeAddressReconcilerActor.FilterOwnClusterDriverMembers(
[UpMember("site-a-1", 4053, "driver", "cluster-SITE-A")],
ownClusterRole: "cluster-MAIN");
observed.ShouldBeEmpty();
}
/// <summary>Scoped to <c>cluster-MAIN</c>, a member carrying that same role is kept.</summary>
[Fact]
public void Own_cluster_role_member_is_kept_when_scoped()
{
var observed = ClusterNodeAddressReconcilerActor.FilterOwnClusterDriverMembers(
[UpMember("main-1", 4053, "driver", "cluster-MAIN")],
ownClusterRole: "cluster-MAIN");
observed.ShouldHaveSingleItem().ShouldBe(("main-1", 4053));
}
/// <summary>
/// Legacy: a null cluster role keeps every Up driver member regardless of cluster — an admin
/// node with no cluster role genuinely sees the whole fleet via gossip (reconcile-all).
/// </summary>
[Fact]
public void Null_cluster_role_keeps_all_driver_members()
{
var observed = ClusterNodeAddressReconcilerActor.FilterOwnClusterDriverMembers(
[
UpMember("main-1", 4053, "driver", "cluster-MAIN"),
UpMember("site-a-1", 4053, "driver", "cluster-SITE-A"),
],
ownClusterRole: null);
observed.Count.ShouldBe(2);
observed.ShouldContain(("main-1", 4053));
observed.ShouldContain(("site-a-1", 4053));
}
/// <summary>Non-driver members (e.g. admin-only) are excluded whether or not a scope is set.</summary>
[Fact]
public void Non_driver_members_are_always_excluded()
{
ClusterNodeAddressReconcilerActor.FilterOwnClusterDriverMembers(
[UpMember("central-1", 4053, "admin", "cluster-MAIN")], ownClusterRole: "cluster-MAIN")
.ShouldBeEmpty();
}
}
@@ -31,13 +31,20 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
public List<ImmutableHashSet<ActorPath>> Calls { get; } = [];
public List<string> ClusterIds { get; } = [];
public List<TestProbe> Probes { get; } = [];
public IActorRef Create(ActorSystem system, ImmutableHashSet<ActorPath> contacts)
/// <summary>Probe created for a given cluster id, for per-cluster fan-out assertions.</summary>
public Dictionary<string, TestProbe> ProbeByCluster { get; } = [];
public IActorRef Create(ActorSystem system, string clusterId, ImmutableHashSet<ActorPath> contacts)
{
Calls.Add(contacts);
ClusterIds.Add(clusterId);
var probe = _newProbe();
Probes.Add(probe);
ProbeByCluster[clusterId] = probe;
return probe.Ref;
}
}
@@ -138,23 +145,54 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
}
[Fact]
public void Exactly_one_client_is_created_for_a_multi_cluster_fleet()
public void One_client_per_cluster_is_created_for_a_multi_cluster_fleet()
{
// A receptionist serves its whole mesh, so one client per Cluster would fan each command out
// once per cluster while the fleet is still a single mesh — N x duplicate deployments.
// THE Phase 6 inversion. Each application Cluster is now its own Akka mesh, and a ClusterClient
// reaches only the mesh whose receptionist it dialled. Central must therefore build one client
// per cluster, each dialling only that cluster's nodes.
var factory = NewInMemoryDbFactory();
SeedCluster(factory, "C1", ("host-a:4053", "host-a", 4053, true, false));
SeedCluster(factory, "C2", ("host-b:4053", "host-b", 4053, true, false));
var clients = new RecordingClientFactory(() => CreateTestProbe());
Spawn(factory, ClusterClientMode(), clients);
AwaitCondition(() => clients.Calls.Count == 1, TimeSpan.FromSeconds(5));
AwaitCondition(() => clients.Calls.Count == 2, TimeSpan.FromSeconds(5));
// ...and that one client dials BOTH clusters' nodes.
clients.Calls[0].Count.ShouldBe(2);
// Give a second refresh a chance to (wrongly) create another.
// Give a second refresh a chance to (wrongly) create more.
ExpectNoMsg(TimeSpan.FromMilliseconds(300));
clients.Calls.Count.ShouldBe(1);
clients.Calls.Count.ShouldBe(2);
clients.ClusterIds.ShouldBe(new[] { "C1", "C2" }, ignoreOrder: true);
// Each client dials ONLY its own cluster's node.
clients.Calls[clients.ClusterIds.IndexOf("C1")].Single().ToString().ShouldContain("host-a");
clients.Calls[clients.ClusterIds.IndexOf("C2")].Single().ToString().ShouldContain("host-b");
}
[Fact]
public void A_command_fans_out_a_send_to_all_to_every_cluster_client()
{
// The other half of the inversion: one MeshCommand must reach EVERY cluster's mesh, so central
// fans SendToAll across all per-cluster clients. A single Send/one-client shape would leave
// every cluster but one silently on its old configuration.
var factory = NewInMemoryDbFactory();
SeedCluster(factory, "C1", ("host-a:4053", "host-a", 4053, true, false));
SeedCluster(factory, "C2", ("host-b:4053", "host-b", 4053, true, false));
var clients = new RecordingClientFactory(() => CreateTestProbe());
var actor = Spawn(factory, ClusterClientMode(), clients);
AwaitCondition(() => clients.Probes.Count == 2, TimeSpan.FromSeconds(5));
var dispatch = new DispatchDeployment(
new DeploymentId(Guid.NewGuid()), new RevisionHash("abc"), new CorrelationId(Guid.NewGuid()));
actor.Tell(new MeshCommand(DeploymentsTopic, dispatch));
foreach (var (clusterId, probe) in clients.ProbeByCluster)
{
var sent = probe.ExpectMsg<ClusterClient.SendToAll>(
TimeSpan.FromSeconds(5), $"cluster {clusterId} must receive the fanned SendToAll");
sent.Path.ShouldBe(MeshPaths.NodeCommunication);
sent.Message.ShouldBe(dispatch);
}
}
[Fact]
@@ -37,13 +37,13 @@ public class MeshClusterClientFactoryTests : TestKit
{
var factory = new DefaultMeshClusterClientFactory();
var first = factory.Create(Sys, UnreachableContact);
var first = factory.Create(Sys, "C1", UnreachableContact);
Sys.Stop(first);
// Stop is asynchronous: `first`'s name is still reserved right here. Without the generation
// suffix this throws InvalidActorNameException — which is precisely what a contact-set
// change would do in production, where stop-then-recreate happens in one handler.
var second = factory.Create(Sys, UnreachableContact);
var second = factory.Create(Sys, "C1", UnreachableContact);
second.Path.Name.ShouldNotBe(first.Path.Name);
}
@@ -54,12 +54,27 @@ public class MeshClusterClientFactoryTests : TestKit
var factory = new DefaultMeshClusterClientFactory();
var names = Enumerable.Range(0, 5)
.Select(_ => factory.Create(Sys, UnreachableContact).Path.Name)
.Select(_ => factory.Create(Sys, "C1", UnreachableContact).Path.Name)
.ToList();
names.Distinct().Count().ShouldBe(5);
}
[Fact]
public void The_cluster_id_is_woven_into_the_client_actor_name()
{
// Per-cluster clients must be distinguishable in logs and the actor tree, and — more
// importantly — two clusters' clients created on one system must never collide on a name.
var factory = new DefaultMeshClusterClientFactory();
var a = factory.Create(Sys, "site-a", UnreachableContact);
var b = factory.Create(Sys, "site-b", UnreachableContact);
a.Path.Name.ShouldContain("site-a");
b.Path.Name.ShouldContain("site-b");
a.Path.Name.ShouldNotBe(b.Path.Name);
}
// Contact-set propagation is deliberately NOT asserted here: reading initial contacts back
// off a created ClusterClient is not possible, and asserting ClusterClientSettings directly
// would test Akka rather than this factory. It is covered where it can actually fail — the
@@ -122,6 +122,34 @@ public sealed class RedundancyPrimaryElectionTests : IAsyncLifetime
"the fixture is only meaningful if the lowest-addressed node is the YOUNGER one");
}
/// <summary>
/// The generalised selector agrees with the driver-specific one, and is scoped to the role it is
/// asked about.
/// </summary>
/// <remarks>
/// <c>/health/active</c> now answers from <c>SelectOldestUpMemberOfRole</c> while the redundancy
/// snapshot answers from <c>SelectDriverPrimary</c>. If those two ever disagreed, the tier would
/// report a different node as active than the one whose OPC UA ServiceLevel reads 250 — so the
/// parity is the property worth pinning, not either answer alone.
/// </remarks>
[Fact]
public void OldestUpMemberOfRole_agrees_with_SelectDriverPrimary_and_is_role_scoped()
{
var members = Akka.Cluster.Cluster.Get(_oldest!).State.Members;
RedundancyStateActor.SelectOldestUpMemberOfRole(members, "driver")
.ShouldBe(RedundancyStateActor.SelectDriverPrimary(members));
// Both nodes here carry admin AND driver, so the admin-scoped answer is the same oldest node
// — and, critically, still the oldest rather than the lowest-addressed role leader.
RedundancyStateActor.SelectOldestUpMemberOfRole(members, "admin")!.Port
.ShouldBe(OldestPort);
// A role nobody carries has no owner. The health check maps this to Unhealthy: during a cold
// start nobody is active, and nobody may claim to be.
RedundancyStateActor.SelectOldestUpMemberOfRole(members, "no-such-role").ShouldBeNull();
}
/// <summary>
/// The elected Primary is the oldest driver member — the node that hosts the cluster singletons —
/// not the lowest-addressed one.
@@ -0,0 +1,182 @@
using Akka.Actor;
using Akka.Cluster.Hosting;
using Akka.Hosting;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Options;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Configuration;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests;
/// <summary>
/// Guards Phase 6's re-homing of <c>RedundancyStateActor</c> off the fleet-wide admin singleton onto
/// a per-node <c>cluster-{ClusterId}</c> singleton (see
/// <see cref="ServiceCollectionExtensions.WithOtOpcUaClusterRedundancySingleton"/>). After the mesh
/// split a driver-only site pair has NO admin node to elect its Primary, so the singleton must be
/// scoped to the node's own cluster role and spawned on every driver node — one per mesh.
///
/// <para>The role scope + the missing-role guard are asserted against the extracted
/// <see cref="ServiceCollectionExtensions.BuildClusterRedundancySingletonOptions"/> helper directly —
/// the same pattern <c>SplitBrainResolverActivationTests</c> uses for <c>BuildDowningHocon</c> /
/// <c>BuildClusterOptions</c>: it pins the exact <see cref="ClusterSingletonOptions.Role"/> without
/// booting a cluster, which no builder-introspection API can otherwise reach. The admin-removal claim
/// is behavioral (a booted admin node's <see cref="ActorRegistry"/> must no longer carry the
/// redundancy key) so it needs a real host.</para>
/// </summary>
public sealed class RedundancyStateSingletonRehomeTests
{
/// <summary>The singleton options scope to the node's OWN cluster role, not the fixed admin role.</summary>
[Fact]
public void Options_scope_the_singleton_to_the_nodes_cluster_role()
{
var options = ServiceCollectionExtensions.BuildClusterRedundancySingletonOptions(
new AkkaClusterOptions { Roles = new[] { "driver", "cluster-SITE-A" } });
options.Role.ShouldBe("cluster-SITE-A");
}
/// <summary>
/// A node with no <c>cluster-{ClusterId}</c> role (a legacy single-mesh node, or the
/// <c>TwoNodeClusterHarness</c> with roles <c>admin,driver</c>) falls back to the fixed
/// <c>driver</c> role rather than throwing — so legacy/not-yet-migrated deployments keep booting.
/// On a split 2-node mesh the <c>driver</c> role is already pair-local (the mesh IS the pair).
/// </summary>
[Fact]
public void Missing_cluster_role_falls_back_to_the_driver_role()
{
var options = ServiceCollectionExtensions.BuildClusterRedundancySingletonOptions(
new AkkaClusterOptions { Roles = new[] { "admin", "driver" } });
options.Role.ShouldBe(RoleParser.Driver);
}
/// <summary>
/// The admin singleton set no longer registers <c>redundancy-state</c>: a booted admin node's
/// registry carries the OTHER admin singletons (proxy) but NOT the redundancy key.
/// </summary>
[Fact]
public async Task Admin_singletons_no_longer_register_the_redundancy_singleton()
{
var registry = await BootAndGetRegistryAsync(
roles: new[] { "admin" },
configure: ab => ab.WithOtOpcUaControlPlaneSingletons());
registry.TryGet<RedundancyStateActorKey>(out _).ShouldBeFalse(
"the redundancy singleton was re-homed off the admin node onto every driver node (Phase 6)");
registry.TryGet<FleetStatusBroadcasterKey>(out _).ShouldBeTrue(
"the remaining admin singletons must still register");
}
/// <summary>
/// The re-home target: a driver node carrying a <c>cluster-{ClusterId}</c> role registers the
/// redundancy singleton (proxy) via the new extension.
/// </summary>
[Fact]
public async Task Cluster_redundancy_extension_registers_the_singleton_on_a_driver_node()
{
var registry = await BootAndGetRegistryAsync(
roles: new[] { "driver", "cluster-SITE-A" },
configure: ab => ab.WithOtOpcUaClusterRedundancySingleton(
new AkkaClusterOptions { Roles = new[] { "driver", "cluster-SITE-A" } }));
registry.TryGet<RedundancyStateActorKey>(out _).ShouldBeTrue(
"every driver node hosts its own mesh's redundancy singleton (Phase 6)");
}
/// <summary>
/// The driver-role FALLBACK end-to-end: a node carrying only <c>driver</c> (no cluster role — a
/// legacy single-mesh node or the <c>TwoNodeClusterHarness</c>) still boots and registers the
/// redundancy singleton via the new extension. Proves the fallback works at the host/registry
/// level, not just in the pure helper — the boot that the earlier throw-based version would have
/// aborted.
/// </summary>
[Fact]
public async Task Driver_role_fallback_registers_the_singleton_end_to_end()
{
var registry = await BootAndGetRegistryAsync(
roles: new[] { "driver" },
configure: ab => ab.WithOtOpcUaClusterRedundancySingleton(
new AkkaClusterOptions { Roles = new[] { "driver" } }));
registry.TryGet<RedundancyStateActorKey>(out _).ShouldBeTrue(
"a driver node with no cluster role falls back to the driver role and still boots + registers");
}
/// <summary>
/// Boots a real host through the production cluster bootstrap (Port=0, no seeds, so it never forms
/// a cluster and no singleton props factory runs — only the proxies register), applies the
/// supplied singleton registration, and returns the <see cref="ActorRegistry"/> after teardown.
/// Start/stop live here rather than the test bodies to keep the CancellationToken-less calls out
/// of the way (mirrors <c>ClusterRoleInfoTests.ResolveAsync</c>).
/// </summary>
private static async Task<ActorRegistry> BootAndGetRegistryAsync(
string[] roles, Action<AkkaConfigurationBuilder> configure)
{
var options = new AkkaClusterOptions
{
Port = 0,
Hostname = "127.0.0.1",
PublicHostname = "127.0.0.1",
SeedNodes = Array.Empty<string>(),
Roles = roles,
};
var builder = Host.CreateDefaultBuilder();
builder.ConfigureServices(services =>
{
services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(options));
// The admin WithActors block reads IOptions<MeshTransportOptions>.Value at spawn time.
services.AddSingleton<IOptions<MeshTransportOptions>>(Options.Create(new MeshTransportOptions()));
// Akka.Hosting invokes every singleton's props factory eagerly at StartAsync; the admin
// singletons (coordinator/audit/reconciler) read IDbContextFactory<OtOpcUaConfigDbContext>,
// so supply an in-memory one or those factories NRE. The driver-only boots don't touch it.
services.AddDbContextFactory<OtOpcUaConfigDbContext>(
o => o.UseInMemoryDatabase($"rehome-test-{Guid.NewGuid():N}"));
// The admin ClusterNodeAddressReconciler props factory resolves IClusterRoleInfo (for its
// per-cluster reconcile scope). A null cluster role ⇒ full-fleet reconcile, which is all
// this test needs — it never asserts on the reconciler, only on the singleton registry.
services.AddSingleton<IClusterRoleInfo>(new FakeClusterRoleInfo(clusterRole: null, clusterId: null));
services.AddAkka("otopcua-rehome-test", (ab, sp) =>
{
ab.WithOtOpcUaClusterBootstrap(sp);
configure(ab);
});
});
using var host = builder.Build();
await host.StartAsync();
try
{
return host.Services.GetRequiredService<ActorRegistry>();
}
finally
{
await host.StopAsync();
}
}
/// <summary>Minimal <see cref="IClusterRoleInfo"/> stub exposing only the cluster-role identity.</summary>
private sealed class FakeClusterRoleInfo : IClusterRoleInfo
{
public FakeClusterRoleInfo(string? clusterRole, string? clusterId)
{
ClusterRole = clusterRole;
ClusterId = clusterId;
}
public string? ClusterRole { get; }
public string? ClusterId { get; }
public NodeId LocalNode => NodeId.Parse("127.0.0.1:0");
public IReadOnlySet<string> LocalRoles => new HashSet<string>();
public bool HasRole(string role) => false;
public IReadOnlyList<NodeId> MembersWithRole(string role) => Array.Empty<NodeId>();
public NodeId? RoleLeader(string role) => null;
public event EventHandler<RoleLeaderChangedEventArgs>? RoleLeaderChanged { add { } remove { } }
}
}
@@ -3,12 +3,16 @@ using Akka.Actor;
using Akka.Cluster;
using Akka.Cluster.Tools.Client;
using Akka.Configuration;
using Microsoft.EntityFrameworkCore;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
using ZB.MOM.WW.OtOpcUa.Runtime.Communication;
using AkkaCluster = Akka.Cluster.Cluster;
@@ -86,12 +90,12 @@ public sealed class MeshClusterClientBoundaryTests : IAsyncLifetime
ClusterClientReceptionist.Get(_central).RegisterService(centralComm);
// Node: the REAL comm actor, dialling central through the REAL production client factory.
var nodeToCentral = _clientFactory.Create(_node, ReceptionistOf(_central));
var nodeToCentral = _clientFactory.Create(_node, "central", ReceptionistOf(_central));
_nodeComm = _node.ActorOf(
NodeCommunicationActor.Props(nodeToCentral), MeshPaths.NodeCommunicationName);
ClusterClientReceptionist.Get(_node).RegisterService(_nodeComm);
_centralClient = _clientFactory.Create(_central, ReceptionistOf(_node));
_centralClient = _clientFactory.Create(_central, "site-a", ReceptionistOf(_node));
}
/// <inheritdoc />
@@ -179,7 +183,7 @@ public sealed class MeshClusterClientBoundaryTests : IAsyncLifetime
var comm = deaf.ActorOf(NodeCommunicationActor.Props(null), MeshPaths.NodeCommunicationName);
comm.ShouldNotBeNull();
var client = _clientFactory.Create(_central, ReceptionistOf(deaf));
var client = _clientFactory.Create(_central, "deaf", ReceptionistOf(deaf));
var command = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid());
var deadline = DateTime.UtcNow + Silence;
@@ -199,6 +203,146 @@ public sealed class MeshClusterClientBoundaryTests : IAsyncLifetime
}
}
[Fact]
public async Task One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh()
{
// THE Phase 6 assertion at the transport level, driving the REAL CentralCommunicationActor.
// Two genuinely separate site meshes (site-A = _node, site-B created here), each its own
// single-member cluster with its own receptionist. Central must build ONE ClusterClient per
// cluster and fan SendToAll across them, so a single MeshCommand reaches BOTH meshes. A single
// fleet-wide client would reach only the one mesh whose receptionist answered — the exact
// failure this test is red-before-green against.
var siteB = StartSelfSeededSystem(withReceptionist: true);
try
{
await AwaitSingleMemberUpAsync(siteB);
RegisterNodeComm(siteB);
var inboxA = NewInbox();
_node.EventStream.Subscribe(_node.ActorOf(CaptureActor.Props(inboxA)), typeof(RestartDriver));
var inboxB = NewInbox();
siteB.EventStream.Subscribe(siteB.ActorOf(CaptureActor.Props(inboxB)), typeof(RestartDriver));
var dbFactory = SeedTwoClusterDb(SiteAddress(_node), SiteAddress(siteB));
var actor = _central.ActorOf(CentralCommunicationActor.Props(
dbFactory, ClusterClientOptions(), new DefaultMeshClusterClientFactory(), () => null));
var command = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid());
var deadline = DateTime.UtcNow + Arrival;
while (DateTime.UtcNow < deadline && !(inboxA.Task.IsCompleted && inboxB.Task.IsCompleted))
{
actor.Tell(new MeshCommand("deployments", command));
await Task.Delay(250, TestContext.Current.CancellationToken);
}
inboxA.Task.IsCompleted.ShouldBeTrue(
"site-A's mesh must receive the dispatch through its own per-cluster ClusterClient");
inboxB.Task.IsCompleted.ShouldBeTrue(
"site-B's mesh must receive the SAME dispatch through its own per-cluster ClusterClient; "
+ "a single fleet-wide client would have reached only one of the two meshes");
}
finally
{
await ShutdownAsync(siteB);
}
}
[Fact]
public async Task A_client_built_for_one_cluster_does_not_deliver_into_another_clusters_mesh()
{
// The partition proof for the fan-out: a client dialling ONLY site-A's contacts must land in
// site-A's mesh and NEVER cross into site-B's. This is why the per-cluster fan-out is correct
// rather than accidentally N x broadcasting — each client is scoped to its own mesh.
var siteB = StartSelfSeededSystem(withReceptionist: true);
try
{
await AwaitSingleMemberUpAsync(siteB);
RegisterNodeComm(siteB);
var inboxA = NewInbox();
_node.EventStream.Subscribe(_node.ActorOf(CaptureActor.Props(inboxA)), typeof(RestartDriver));
var inboxB = NewInbox();
siteB.EventStream.Subscribe(siteB.ActorOf(CaptureActor.Props(inboxB)), typeof(RestartDriver));
var clientA = _clientFactory.Create(_central, "site-a", ReceptionistOf(_node));
var command = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid());
// Keep sending for the full Silence window so any erroneous cross-delivery into site-B has
// every chance to arrive — a pass means the boundary is shut, not that the clock ran out.
var deadline = DateTime.UtcNow + Silence;
while (DateTime.UtcNow < deadline)
{
clientA.Tell(new ClusterClient.SendToAll(MeshPaths.NodeCommunication, command));
await Task.Delay(250, TestContext.Current.CancellationToken);
}
inboxA.Task.IsCompleted.ShouldBeTrue("site-A's client must deliver into site-A's mesh");
inboxB.Task.IsCompleted.ShouldBeFalse(
"a client built for site-A's contacts must NOT deliver into site-B's mesh — the "
+ "per-cluster fan-out must stay partitioned, one client per mesh");
}
finally
{
await ShutdownAsync(siteB);
}
}
/// <summary>Registers a real node-comm actor with a system's receptionist so central can reach it.</summary>
/// <param name="system">The site system to register into.</param>
private static void RegisterNodeComm(ActorSystem system)
{
var comm = system.ActorOf(NodeCommunicationActor.Props(null), MeshPaths.NodeCommunicationName);
ClusterClientReceptionist.Get(system).RegisterService(comm);
}
private static (string Host, int Port) SiteAddress(ActorSystem system)
{
var addr = AkkaCluster.Get(system).SelfAddress;
return (addr.Host!, addr.Port!.Value);
}
private static MeshTransportOptions ClusterClientOptions() => new()
{
Mode = MeshTransportOptions.ModeClusterClient,
CentralContactPoints = ["akka.tcp://otopcua@127.0.0.1:4053"],
ContactRefreshSeconds = 1,
};
private static IDbContextFactory<OtOpcUaConfigDbContext> SeedTwoClusterDb(
(string Host, int Port) siteA, (string Host, int Port) siteB)
{
var factory = new InMemoryConfigDbFactory(Guid.NewGuid().ToString("N"));
using var db = factory.CreateDbContext();
db.ClusterNodes.Add(NodeRow("SITE-A", "node-a", siteA));
db.ClusterNodes.Add(NodeRow("SITE-B", "node-b", siteB));
db.SaveChanges();
return factory;
}
private static ClusterNode NodeRow(string clusterId, string nodeId, (string Host, int Port) addr) => new()
{
NodeId = nodeId,
ClusterId = clusterId,
Host = addr.Host,
AkkaPort = addr.Port,
ApplicationUri = $"urn:{nodeId}",
Enabled = true,
MaintenanceMode = false,
CreatedBy = "test",
};
private sealed class InMemoryConfigDbFactory(string dbName)
: IDbContextFactory<OtOpcUaConfigDbContext>
{
public OtOpcUaConfigDbContext CreateDbContext()
{
var opts = new DbContextOptionsBuilder<OtOpcUaConfigDbContext>()
.UseInMemoryDatabase(dbName)
.Options;
return new OtOpcUaConfigDbContext(opts);
}
}
/// <summary>
/// Starts a single-member cluster on a dynamic port, optionally without the receptionist.
/// </summary>
@@ -0,0 +1,59 @@
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Diagnostics.HealthChecks;
using Microsoft.Extensions.Options;
using Shouldly;
using Xunit;
using ZB.MOM.WW.Health;
using ZB.MOM.WW.Health.Akka;
using ZB.MOM.WW.OtOpcUa.Host.Health;
namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Health;
/// <summary>
/// Pins OtOpcUa's active-tier wiring. The <i>rule</i> (oldest Up member of a role) now lives in the
/// shared <c>ClusterActiveNode</c> and is tested there against a real two-node cluster; what stays
/// OtOpcUa's own decision — and what regressed in <c>lmxopcua#494</c> — is <b>which check is
/// registered on the active tag, and with which roles</b>.
/// </summary>
public sealed class ActiveTierRegistrationTests
{
private static HealthCheckRegistration ClusterPrimaryRegistration(bool hasAdmin) =>
new ServiceCollection()
.AddOtOpcUaHealth(hasAdmin)
.BuildServiceProvider()
.GetRequiredService<IOptions<HealthCheckServiceOptions>>()
.Value.Registrations
.Where(r => r.Name == "cluster-primary")
.ShouldHaveSingleItem();
[Fact]
public void Active_tier_uses_the_shared_ActiveNodeHealthCheck()
{
// Not a bespoke copy. Two apps in the family independently hand-rolled this check because the
// shared one selected by RoleLeader; since Health 0.3.0 the shared one is the correct rule, so
// a private reimplementation reappearing here would be the regression.
var registration = ClusterPrimaryRegistration(hasAdmin: true);
registration.Factory(new ServiceCollection().BuildServiceProvider())
.ShouldBeOfType<ActiveNodeHealthCheck>();
}
[Theory]
[InlineData(true)]
[InlineData(false)]
public void Active_tier_check_is_registered_on_every_node_role(bool hasAdmin)
{
// Registered on driver-only nodes too, unlike the admin-only configdb probe. A site pair needs
// an active tier just as much as the central pair does — under the old admin-scoped check all
// four driver-only site nodes answered 200 and no consumer could find a site's Primary.
ClusterPrimaryRegistration(hasAdmin).Tags.ShouldContain(ZbHealthTags.Active);
}
[Fact]
public void Active_tier_check_is_not_on_the_ready_tier()
{
// Readiness must not depend on being the active node: a healthy standby is ready to serve, and
// tagging it ready would take the whole pair out of rotation whenever one node is standby.
ClusterPrimaryRegistration(hasAdmin: true).Tags.ShouldNotContain(ZbHealthTags.Ready);
}
}
@@ -193,6 +193,10 @@ public sealed class ServiceCollectionExtensionsTests
public NodeId LocalNode { get; } = NodeId.Parse("test-node");
/// <summary>Gets the local roles.</summary>
public IReadOnlySet<string> LocalRoles { get; } = new HashSet<string>(["driver"]);
/// <summary>Gets the local node's cluster-scoped role. None configured in this fake.</summary>
public string? ClusterRole => null;
/// <summary>Gets the ClusterId extracted from <see cref="ClusterRole"/>. None configured in this fake.</summary>
public string? ClusterId => null;
/// <summary>Determines whether the local node has the specified role.</summary>
/// <param name="role">The role to check.</param>
public bool HasRole(string role) => LocalRoles.Contains(role);