wwtools/mbproxy/docs/design.md

mbproxy — design plan

Architectural design for the mbproxy Modbus TCP proxy service: how it fronts ~54 AutomationDirect DirectLOGIC DL205/DL260 controllers, rewrites BCD tags bidirectionally inline, and recovers from listener and backend failures. Settled in a design Q&A on 2026-05-13.

Status: plan; no code yet. Each decision below is load-bearing — change deliberately, not by drift.

Context (what the service does and why it exists) lives in ../CLAUDE.md under "What this is" and "Purpose: bidirectional BCD rewrite". This file is the how. Device quirks the design depends on live in ../DL260/dl205.md.

Runtime shape: .NET 10 Generic Host worker service registered as a Windows Service via Microsoft.Extensions.Hosting.WindowsServices.

Listener topology — per-PLC port (one port → one PLC)

The host opens one TcpListener per PLC on a distinct port. Upstream clients reach a specific PLC by connecting to its assigned proxy port; no protocol-level routing is needed.

Client A ──┐
Client B ──┼──→ proxy:5020 ──→ PLC #1  (10.0.1.1:502)
           ├──→ proxy:5021 ──→ PLC #2  (10.0.1.2:502)
           │       ...
           └──→ proxy:5073 ──→ PLC #54 (10.0.1.54:502)

Connection model — single backend socket per PLC, multiplexed via MBAP TxId rewriting

Each PLC has one persistent backend TCP socket, owned by a PlcMultiplexer. Many upstream client connections share that single backend socket; the multiplexer distinguishes their in-flight requests by rewriting the MBAP transaction ID on each request and restoring each client's original TxId on the matching response. Implemented in Phase 09; replaced the prior 1:1 per-upstream-client backend-socket model.

Client A ─┐
Client B ─┼─→  proxy:5020  ─[ PlcMultiplexer ]─→  PLC #1 (10.0.1.1:502)
Client C ─┘                     │                  (one persistent socket)
                                ▼
                       CorrelationMap[proxyTxId]
                       TxIdAllocator (16-bit space)

• Upstream → multiplexer: each accepted upstream socket is wrapped in an UpstreamPipe (read loop + bounded response channel). The pipe's read loop hands every parsed MBAP frame to the multiplexer's OnUpstreamFrameAsync, which allocates a free 16-bit proxyTxId, stores an InFlightRequest in a CorrelationMap keyed by that proxyTxId, BCD-rewrites the request payload, overwrites the MBAP header's TxId field with proxyTxId, and enqueues the frame into the per-PLC outbound channel.
• Multiplexer → backend: a single backend writer task drains the outbound channel and sends each frame to the PLC over the shared socket. A single backend reader task reads MBAP frames back, looks each up by proxyTxId in the correlation map, BCD-rewrites the response, restores each interested party's original TxId, and routes the frame to that party's UpstreamPipe._responseChannel. The single-writer / single-reader invariant on the backend socket eliminates the need for socket-level synchronisation.
• Per-request timeout watchdog: a periodic task scans the correlation map at a quarter of Connection.BackendRequestTimeoutMs and times out any in-flight request whose response has not arrived. Timed-out requests get a Modbus exception 0x0B (Gateway Target Device Failed To Respond) delivered to their upstream party and free their allocator slot. Without this watchdog, a single lost or mis-routed response would leak a correlation entry forever and hang the upstream pipe indefinitely.

Operational consequence (replaces the prior 4-client warning). The H2-ECOM100's 4-concurrent-TCP-client cap (see ../DL260/dl205.md → Behavioral Oddities) no longer limits upstream-side connection count — the proxy holds exactly one slot per PLC regardless of how many upstream clients are attached. The wire-rate ceiling is unchanged (the ECOM internally serializes requests at ~2–10 ms per scan); the multiplexer shifts where serialization happens (proxy outbound queue vs PLC accept queue) rather than adding throughput.

⚠ Backend disconnect cascades upstream. When the backend socket dies (PLC reboot, network partition, middlebox idle drop), the multiplexer closes every attached upstream pipe in the same cycle and increments BackendDisconnectCascades by the upstream count. Clients reconnect on their own next request and the multiplexer Polly-reconnects to the backend on the first upstream frame.

⚠ pymodbus 3.13.0 simulator quirk (test-only). The pymodbus simulator's ServerRequestHandler stores a single last_pdu per connection and schedules deferred handlers via asyncio.call_soon. Two MBAP frames arriving in the same recv buffer (as the multiplexer can produce on its shared backend connection) overwrite last_pdu before the first handler runs, and both responses then carry the later request's TxId. The real DL260 ECOM does not suffer this — it echoes per-request TxIds correctly. Multiplexer correctness under truly concurrent backend traffic is therefore proved against a stub backend in PlcMultiplexerTests; the E2E suite paces requests to keep pymodbus in known-good single-PDU mode. The per-request watchdog is the production defence against any backend (real or simulated) that mis-echoes a TxId.

Configuration — single appsettings.json

All configuration lives in one file, loaded via Microsoft.Extensions.Configuration and bound to typed POCOs. No sidecar YAML/CSV.

{
  "Mbproxy": {
    "BcdTags": {
      "Global": [
        { "Address": 1072, "Width": 16 },
        { "Address": 1080, "Width": 32 }
      ]
    },
    "Plcs": [
      {
        "Name": "Line1-Mixer",
        "ListenPort": 5020,
        "Host": "10.0.1.1",
        "BcdTags": {
          "Add":    [ { "Address": 1200, "Width": 32 } ],
          "Remove": [ 1080 ]
        }
      },
      { "Name": "Line1-Conveyor", "ListenPort": 5021, "Host": "10.0.1.2" }
      // ... 54 PLC rows
    ],
    "AdminPort": 8080,
    "Connection": {
      "BackendConnectTimeoutMs": 3000,
      "BackendRequestTimeoutMs": 3000
    },
    "Resilience": {
      "BackendConnect":   { "MaxAttempts": 3, "BackoffMs": [100, 500, 2000] },
      "ListenerRecovery": { "InitialBackoffMs": [1000, 2000, 5000, 15000, 30000], "SteadyStateMs": 30000 }
    }
  }
}

Hybrid tag resolution. For each PLC, the effective BCD tag list is Global ∪ Add − Remove. Remove matches by address; if the same address appears in both Add and Global the Add entry wins (this is how a width override is expressed). Validation at startup must:

• reject duplicate addresses within a single PLC's resolved list
• reject 32-bit entries that would have their high register overlap a separate 16-bit entry
• warn on Remove entries that don't match any global tag (probably stale config)

Configuration hot-reload

Microsoft.Extensions.Configuration loads appsettings.json with reloadOnChange: true, and all consumers read via IOptionsMonitor<MbproxyOptions> so a save to the config file propagates without restarting the service. Each change kind has explicit reconcile semantics:

Change in appsettings	Propagation
BcdTags.Global add/remove/width	Rewriter dereferences the monitor per-PDU. Next PDU sees the new map; in-flight reads/writes are not retroactively touched.
Plcs[i].BcdTags.{Add,Remove}	Same — next-PDU resolution.
New Plcs[i] entry	Listener supervisor binds the new port subject to the same eager-then-auto-recover policy.
Plcs[i] removed	Supervisor stops the listener and closes all upstream client connections for that PLC.
Plcs[i].ListenPort or Host changed	Equivalent to remove + add.
Connection.Backend*TimeoutMs	Next backend connect/request uses the new value. In-flight operations keep their already-applied timeout.
Invalid reload (schema break, duplicate ports, duplicate addresses in a resolved tag list)	Reload is rejected as a whole; current in-memory config stays in effect; mbproxy.config.reload.rejected is logged at Error.

Every accepted reload emits mbproxy.config.reload.applied at Information with a summary of which PLCs were added/removed and the size of the tag-list delta.

BCD tag shape

public sealed record BcdTag(ushort Address, byte Width);   // Width ∈ { 16, 32 }

• 16-bit BCD — one register holds 4 BCD digits (0–9999). Wire value 0x1234 decodes to decimal 1234.
• 32-bit BCD — a CDAB-ordered register pair at Address and Address+1. The register at Address holds the low 4 digits; the register at Address+1 holds the high 4 digits. Decoded decimal = high * 10000 + low. This follows directly from DirectLOGIC's CDAB word order (see ../DL260/dl205.md → Word Order).
• Unsigned only. DL205/DL260 BCD is non-negative in the default ladder pattern; the proxy does not implement signed BCD.
• Holding-register and input-register addresses share the same space. The rewriter applies the configured tag list against both FC03 and FC04 reads.

Read coalescing (Phase 10)

After Phase 10, FC03 / FC04 requests are additionally subject to in-flight read coalescing before they reach the backend. When two or more upstream clients send the same (unitId, fc, startAddress, qty) tuple within the in-flight window of an already-routed request, the multiplexer attaches each late arrival to the existing InFlightRequest.InterestedParties list instead of opening a second backend round-trip. The single backend response is fanned out to every attached party with each party's original MBAP TxId restored individually.

Properties:

• Zero post-response staleness. Coalescing operates entirely between "first request sent to backend" and "response received from backend" (microseconds to ~10 ms typical). Once the response is fanned out, the coalescing entry dies. The proxy is NOT a cache layer — the value each upstream sees is the same value an uncoalesced request would have returned within the PLC's scan-time precision.
• Only FC03 / FC04. Writes (FC06 / FC16) are non-idempotent on BCD tags and never coalesced. Different function codes never share a CoalescingKey even at the same address (FC03 and FC04 read different Modbus tables). Different unitId bytes never coalesce (different PLC personalities behind a shared socket).
• Bounded fan-out via MaxParties (default 32 in Mbproxy.Resilience.ReadCoalescing.MaxParties). Once an entry has MaxParties interested clients, the next arrival opens a fresh entry — bounds the response-fanout cost per entry at O(MaxParties) and shields the backend reader task from pathological pile-on.
• Hot-reloadable on/off. Mbproxy.Resilience.ReadCoalescing.Enabled defaults to true. Flipping it to false at runtime leaves running coalesced entries to drain naturally; subsequent FC03/04 requests take the Phase-9 (one round-trip per upstream request) path.
• Transparency contract preserved. Each upstream client still sees its own original MBAP TxId on the response. The BCD rewriter runs once on the shared response buffer; per-party copies are only made when fan-out has more than one party.

Counter accounting balance (per snapshot): coalescedHitCount + coalescedMissCount equals the total FC03 + FC04 requests seen since the multiplexer was constructed. Both counters increment regardless of whether the coalescing feature is enabled — coalescedHitCount is 0 when disabled, but every read still increments coalescedMissCount.

Rewriter — function code scope

The rewriter inspects and rewrites payloads only for these function codes; every other FC (coils, discrete inputs, diagnostics, exception responses) passes through byte-for-byte:

FC	Direction	Action
03	request + response	FC03 requests may be coalesced with peers before reaching the backend (see Phase-10 section above); response re-encodes covered BCD slots from raw nibbles → binary integer
04	request + response	Same coalescing eligibility as FC03; response re-encoding the same as FC03 (input-register table also surfaces V-memory)
06	request	Re-encode binary integer → BCD nibbles before forwarding
06	response	Decode BCD nibbles → binary integer on the echo (clients validate that the echoed value equals the value they sent; without this, NModbus-style clients throw on the round-trip)
16	request	Per-register over the configured slots, then forward

Partial-overlap policy. A request that touches only ONE register of a configured 32-bit BCD pair (qty=1 at the low addr, or any read/write of the high addr alone) passes through raw with a mbproxy.rewrite.partial_bcd warning. The proxy never synthesises a Modbus exception for a partial-overlap — that response code is reserved for transport failure.

Failure modes — transparent pass-through with Polly-bounded backend connect

• PLC returns a Modbus exception (codes 01–04) → forward verbatim with the original MBAP transaction ID. The client sees the real DL205/DL260 exception.
• Backend connect refused or initial connect timeout → retry under a Polly resilience pipeline: 3 attempts at 100ms / 500ms / 2000ms backoff (tuned via Resilience.BackendConnect). If all attempts fail, the multiplexer closes the upstream client connection that triggered the connect.
• Backend mid-stream broken socket → the multiplexer's reader/writer task throws; the backend tear-down path cancels both tasks, drains the correlation map, and cascades the disconnect by closing every attached upstream pipe. The next upstream request to any pipe triggers a fresh backend connect through the Polly pipeline. BackendDisconnectCascades counter records the upstream-pipe count at each cascade event.
• Backend request timeout → the per-request watchdog times out any correlation entry older than Connection.BackendRequestTimeoutMs, delivers Modbus exception 0x0B (Gateway Target Device Failed To Respond) with the original TxId to the upstream party, and frees the proxy TxId. No mid-request retries — FC06 / FC16 are non-idempotent on BCD tags (a partial-applied multi-register write could leave a 32-bit BCD tag mid-transition), so every in-flight request is one-shot. The client interprets the 0x0B as a transport failure and reconnects through its normal path.
• Partial-BCD overlap → forward raw + warn (see Rewriter section).
• One slow PLC does not stall the rest of the fleet. Each PLC has its own PlcMultiplexer, with its own backend socket, correlation map, and outbound channel; per-PLC failures are local. A slow or dead backend on one PLC only impacts that PLC's clients.

Startup posture — eager, continue on per-port failure

At startup the host attempts to bind all 54 listen sockets up front. Each failure (port already in use, invalid IP, malformed PLC entry) is logged at Error and handed off to the listener supervisor (next section). The service proceeds with whichever PLCs bound on the first attempt; the rest converge in the background. Monitoring should alert on mbproxy.startup.bind.failed so missing PLCs aren't silently dropped, and watch for mbproxy.listener.recovered to confirm late binds eventually succeeded.

Listener auto-recovery (Polly-backed supervisor)

Each PLC's listener runs under a supervisor task that owns its bind lifecycle. If a bind fails at startup, or if a listener faults at runtime (port stolen by another process, transient OS network reset), the supervisor reattempts via a Polly retry pipeline: 5 attempts at 1s / 2s / 5s / 15s / 30s backoff, then steady-state retries every 30s indefinitely (tuned via Resilience.ListenerRecovery). Each attempt logs at Debug; the bind that finally succeeds emits one mbproxy.listener.recovered Information event.

While a supervisor is between attempts, the corresponding PLC is reported as listener.state = recovering on the status page. Hot-reload uses the same supervisor to bring newly-added PLCs online and to tear down removed ones — there is exactly one code path for "bring up a listener" and one for "shut a listener down."

Logging — Serilog, structured, console + rolling file

Serilog wired through the Microsoft.Extensions.Logging bridge:

• Console sink for interactive --console runs.
• Rolling-file sink under %ProgramData%\mbproxy\logs\.
• Default level Information. Per-PLC and per-client scopes via LogContext.PushProperty("Plc", name) / ("Client", remoteEp) so log lines are greppable across the fleet.

Stable event names (keep these stable so log queries don't churn):

Event	Level	Properties
mbproxy.startup.bind	Info	Plc, Port
mbproxy.startup.bind.failed	Error	Plc, Port, Reason
mbproxy.listener.recovered	Info	Plc, Port, AttemptCount
mbproxy.client.connected	Info	Plc, RemoteEp
mbproxy.client.disconnected	Info	Plc, RemoteEp, Reason
mbproxy.backend.failed	Warning	Plc, Reason
mbproxy.rewrite.partial_bcd	Warning	Plc, Address, ClientStart, ClientQty
mbproxy.rewrite.invalid_bcd	Warning	Plc, Address, RawValue, Direction
mbproxy.exception.passthrough	Info	Plc, Fc, ExceptionCode
mbproxy.config.reload.applied	Info	PlcsAdded, PlcsRemoved, TagDelta
mbproxy.config.reload.rejected	Error	Reason
mbproxy.admin.bind.failed	Error	Port, Reason
mbproxy.multiplex.backend.connected	Info	Plc, Host, Port
mbproxy.multiplex.backend.disconnected	Warning	Plc, UpstreamCount, InFlightCount, Reason
mbproxy.multiplex.saturated	Error	Plc, RemoteEp (16-bit TxId space full)
mbproxy.multiplex.request.timeout	Warning	Plc, ProxyTxId, OriginalTxId, Fc, ElapsedMs
mbproxy.coalesce.hit	Debug	Plc, UnitId, Fc, Start, Qty, PartyCount
mbproxy.coalesce.miss	Debug	Plc, UnitId, Fc, Start, Qty
mbproxy.coalesce.dead_upstream	Debug	Plc, UnitId, Fc, Start, Qty

Status page — read-only HTTP endpoint

A separate Kestrel-hosted minimal API runs on Mbproxy.AdminPort (default 8080, distinct from the Modbus listen ports). The endpoint set is intentionally narrow — read-only telemetry; no admin actions (kick client, force reload, restart listener) are exposed:

• GET / — single self-contained HTML page rendering a table of all configured PLCs with their state and live counters. Auto-refreshes every 5s via a meta-refresh tag (no JS bundle, no external assets).
• GET /status.json — the same data as JSON for monitoring scrapers.

Authentication is assumed to live at the network layer (trusted internal segment behind a firewall). Surface that assumption in deployment docs when they exist.

Service-wide fields:

Field	Meaning
service.uptime	Seconds since service start
service.version	Assembly informational version
service.config.lastReloadUtc	Timestamp of last accepted hot-reload (or null)
service.config.reloadCount	Number of reloads accepted since start
service.config.reloadRejectedCount	Number of reloads rejected since start
listeners.bound / listeners.configured	Bound listener count vs configured PLC count

Per-PLC fields (one row per Plcs[i]):

Field	Meaning
name, host, listenPort	Identity from config
listener.state	bound / recovering / stopped
listener.lastBindError	Most recent bind failure message (when recovering)
listener.recoveryAttempts	Polly retry count since last successful bind
clients.connected	Currently connected upstream client count
clients.remoteEndpoints	Array of { remote, connectedAtUtc, pdusForwarded }
pdus.forwarded	Total PDUs (request+response) forwarded since start
pdus.byFc	{ fc03, fc04, fc06, fc16, other } request counts
pdus.rewrittenSlots	Count of register slots BCD-rewritten
pdus.partialBcdWarnings	Count of partial-overlap pass-throughs
backend.connects.success / backend.connects.failed	Polly-final-result counters
backend.exceptions.byCode	{ "01": n, "02": n, "03": n, "04": n }
backend.lastRoundTripMs	EWMA of recent successful round-trip times
backend.coalescedHitCount	FC03/04 requests that attached to an already-in-flight peer (Phase 10)
backend.coalescedMissCount	FC03/04 requests that opened a fresh backend round-trip (Phase 10). Hit + Miss = total FC03/04 requests
backend.coalescedResponseToDeadUpstream	Coalesced fan-out responses skipped because the attached upstream had already disconnected (Phase 10)
bytes.upstreamIn / bytes.upstreamOut	Bytes forwarded each direction

Counters are System.Threading.Interlocked longs read atomically per request; no locking on the read path.

Test simulator — pymodbus DL260/DL205 server

The pymodbus profile at ../DL260/dl205.json already models the DL205/DL260 quirks (BCD nibbles at known addresses, CDAB-ordered 32-bit values, C-relay/Y-output coil mappings, etc.) as concrete register seeds. The test infrastructure wraps it as a managed lifecycle so every integration / e2e test gets a fresh known-good DL-series target without needing real hardware.

Harness shape (lives under tests/sim/):

• Launcher script — tests/sim/run-dl205-sim.ps1 provisions a Python venv under tests/sim/.venv on first run (python -m venv + pip install pymodbus), then launches pymodbus.server with the dl205.json profile on a configurable port. Idempotent: re-runs reuse the venv.
• xUnit fixture — Mbproxy.Tests.Sim.DL205SimulatorFixture : IAsyncLifetime that:
  • InitializeAsync: spawns the simulator subprocess, polls TcpClient.ConnectAsync against the port until success or a 10 s deadline, captures stdout/stderr to test output.
  • DisposeAsync: signals graceful shutdown (Ctrl-C on the process group on Windows), then Process.Kill(entireProcessTree: true) as a safety net.
  • Exposes Host, Port, LogTail (last N lines of sim stderr for diagnosis).
• Test collection — [CollectionDefinition(nameof(DL205SimulatorCollection))] so the fixture is shared across all integration/e2e classes that opt in (cheap startup, expensive process churn).
• Skip policy — if Python or pymodbus isn't available and the auto-provision fails (no network, locked-down CI image, etc.), InitializeAsync records the reason and tests skip via Assert.Skip(sim.SkipReason). CI must have Python 3.10+ available; local devs running only the rewriter unit tests need nothing extra.
• Alternate profiles — additional scenarios (e.g., a profile that seeds a specific partial-overlap test case, or a profile with strict type exception: true to verify the proxy doesn't depend on lax pymodbus behaviour) live alongside dl205.json and are selected via MODBUS_SIM_PROFILE env var, matching the pattern already established by ../DL260/DL205BcdQuirkTests.cs.

The simulator IS the proxy's end-to-end test bed. A standard e2e test does:

1. Start the simulator at 127.0.0.1:<simPort>.
2. Configure the proxy with one PLC entry Host=127.0.0.1, Port=<simPort>, ListenPort=<proxyPort>.
3. Start the proxy (in-process via WebApplicationFactory-style host construction).
4. Drive a plain Modbus TCP client (NModbus or FluentModbus) against 127.0.0.1:<proxyPort>.
5. Assert two directions:
   • Read: client sees the BCD-decoded integer (proxy rewrote the response).
   • Write: simulator's register state shows the BCD-encoded nibbles (proxy rewrote the request).

Testing

• Unit tests — drive the BCD rewriter with synthetic Modbus PDU byte arrays. No network, no simulator. Cover every FC03/04/06/16 × {single 16-bit, full 32-bit pair, partial-overlap low, partial-overlap high, mixed-with-non-BCD} cell.
• Integration tests — drive the proxy end-to-end against the pymodbus simulator described in the previous section, using a plain Modbus TCP client (NModbus or FluentModbus) against proxy:<listenPort> and asserting the decoded value rather than the raw register bytes.
• Auto-recovery tests — bind a TcpListener on a target port BEFORE starting the proxy, assert that the supervisor enters recovering state, release the port, and assert the next supervisor attempt succeeds and mbproxy.listener.recovered fires. Also cover the runtime-fault path by forcing the accept loop to throw and asserting the supervisor reattempts.
• Hot-reload tests — write a temp appsettings.json, start the host, mutate the file (add a PLC, remove a PLC, change a global tag width), and assert: (a) supervisor adds/removes the affected listener, (b) the rewriter on the next PDU reflects the new tag map, (c) a malformed reload is rejected without breaking the running config. Cover both mbproxy.config.reload.applied and mbproxy.config.reload.rejected paths.
• Status page tests — start the host, induce known events (connect 2 clients, force a backend exception, trigger a partial-BCD warning), and assert GET /status.json returns the expected counters. The HTML page is verified separately as a smoke test that the route returns 200 with text/html.