Joseph Doherty
fc08892eb3
Root cause: pymodbus 3.13's `pymodbus.simulator` console script hard-codes asyncio.run(run_main(), debug=True). Debug mode captures a traceback per scheduled callback and logs callbacks slower than 100 ms (the "took 0.146 seconds" lines in #909). Side by side on the shared docker host (closed-loop FC03 clients, 2 s deadline): 100 clients got ~120-170 req/s, p99 1.6-1.9 s with client timeouts in debug mode vs ~2400 req/s, p99 ~135 ms without it. The shared sim also carries the docker-dev rig's 49 Modbus drivers on each of central-1/central-2 (98 sockets, ~220 req/s at idle), which is above the debug-mode capacity, so a test run on top was answered late ("Cancel send, because not connected!") and timed out. - run_simulator.py: same CLI as pymodbus.simulator, debug=False, plus a watchdog thread that exits after 3 failed FC03 probes so restart: unless-stopped replaces the container (docker has no restart-on-unhealthy). - fc3_probe.py: stdlib FC03 read with a deadline (MBAP or RTU framing); the compose healthcheck. - docker-compose.yml: shared anchor for every pymodbus service: new entrypoint, `project: lmxopcua` label, restart: unless-stopped, healthcheck, bounded json-file logs. - README: why, the numbers, and how to diagnose load.
88 lines
3.6 KiB
Python
88 lines
3.6 KiB
Python
#!/usr/bin/env python3
|
|
"""Start the pymodbus simulator WITHOUT asyncio debug mode, under a self-healing watchdog (#909).
|
|
|
|
Takes the same arguments as the stock ``pymodbus.simulator`` console script.
|
|
|
|
Why not run the console script directly: pymodbus 3.13's
|
|
``pymodbus/server/simulator/main.py`` hard-codes ``asyncio.run(run_main(), debug=True)``.
|
|
asyncio debug mode records a traceback for every scheduled callback and times every
|
|
callback, logging any that run over 100 ms. Measured on the shared docker host
|
|
(2026-11-13, same image, side by side, closed-loop FC03 clients with a 2 s deadline):
|
|
20 clients ~170-220 req/s in debug mode against ~1300-1500 req/s without it; 100 clients
|
|
~120-170 req/s (p50 0.5-0.8 s, p99 1.6-1.9 s, client timeouts) against ~2350-2450 req/s
|
|
(p99 ~135 ms, no timeouts). Debug mode made the shared simulator the bottleneck:
|
|
that is the #909 failure: "took 0.146 seconds" warnings, thousands of
|
|
``Cancel send, because not connected!`` lines, and late replies.
|
|
|
|
The watchdog thread runs ``fc3_probe`` against the server every
|
|
``SIM_WATCHDOG_INTERVAL`` seconds (default 10). After ``SIM_WATCHDOG_FAILURES``
|
|
consecutive failures (default 3) it exits the process with status 1, so the compose
|
|
``restart: unless-stopped`` policy starts a fresh container. Docker has no built-in
|
|
"restart when unhealthy", and an autoheal sidecar would need the docker socket.
|
|
Set ``SIM_WATCHDOG_INTERVAL=0`` to disable the watchdog.
|
|
"""
|
|
import asyncio
|
|
import json
|
|
import os
|
|
import sys
|
|
import threading
|
|
import time
|
|
|
|
from pymodbus.server.simulator.main import run_main
|
|
|
|
import fc3_probe
|
|
|
|
|
|
def _arg(name: str, default=None):
|
|
argv = sys.argv[1:]
|
|
if name in argv and argv.index(name) + 1 < len(argv):
|
|
return argv[argv.index(name) + 1]
|
|
return default
|
|
|
|
|
|
def _server_endpoint():
|
|
"""Port and framer of the selected server_list entry, from the profile JSON."""
|
|
try:
|
|
with open(_arg("--json_file"), encoding="utf-8") as fh:
|
|
servers = json.load(fh)["server_list"]
|
|
srv = servers[_arg("--modbus_server", next(iter(servers)))]
|
|
return int(srv.get("port", 5020)), srv.get("framer", "socket")
|
|
except Exception: # noqa: BLE001 - fall back to the standard profile's endpoint
|
|
return 5020, "socket"
|
|
|
|
|
|
def _watchdog(port: int, framer: str, interval: float, limit: int, deadline: float) -> None:
|
|
time.sleep(max(interval, 5.0)) # let the listener come up
|
|
failures = 0
|
|
while True:
|
|
try:
|
|
fc3_probe.probe(port=port, framer=framer, deadline=deadline)
|
|
failures = 0
|
|
except Exception as exc: # noqa: BLE001
|
|
failures += 1
|
|
print(f"watchdog: FC03 probe {failures}/{limit} failed: {type(exc).__name__}: {exc}",
|
|
file=sys.stderr, flush=True)
|
|
if failures >= limit:
|
|
print("watchdog: simulator is not answering in time; exiting so the container restarts",
|
|
file=sys.stderr, flush=True)
|
|
os._exit(1)
|
|
time.sleep(interval)
|
|
|
|
|
|
def main() -> None:
|
|
interval = float(os.environ.get("SIM_WATCHDOG_INTERVAL", "10"))
|
|
if interval > 0:
|
|
port, framer = _server_endpoint()
|
|
threading.Thread(
|
|
target=_watchdog,
|
|
args=(port, framer, interval,
|
|
int(os.environ.get("SIM_WATCHDOG_FAILURES", "3")),
|
|
float(os.environ.get("SIM_WATCHDOG_DEADLINE", "3"))),
|
|
name="fc3-watchdog", daemon=True).start()
|
|
# debug=False is the point of this file: see the module docstring.
|
|
asyncio.run(run_main(sys.argv[1:]), debug=False)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|