feat(mtconnect): refused Agent connections mark values BadNoCommunication after a short confirmation; clamp the raised sample count to the Agent's bufferSize (#846)

- A /sample or re-baseline failure whose chain holds SocketError.ConnectionRefused, unbroken for
  CommsLossConfirmMs (new, default 2000; 0 = first refusal; negative = off), publishes every subscribed
  value BadNoCommunication once per outage; the first delivery afterwards republishes all of them from
  the index (unchanged values included). Timeouts/resets/DNS/pause keep the #527 watchdog contract.
- SampleCountCeiling clamps the adaptive raise to the /probe bufferSize (never below SampleCount).
- Docs: MTConnect.md comms-loss section + config key; drivers.md rule.

Refs #846
This commit is contained in:
Joseph Doherty
2026-11-11 03:48:05 -05:00
parent ddc1f29e3a
commit 19ffb3c902
7 changed files with 522 additions and 10 deletions
+33 -2
View File
@@ -157,6 +157,7 @@ clears, and the Agent's text is logged once per distinct refusal. An answer that
"SampleIntervalMs": 1000,
"SampleCount": 1000,
"HeartbeatMs": 10000,
"CommsLossConfirmMs": 2000,
"Probe": { "Enabled": true, "IntervalMs": 5000, "TimeoutMs": 2000 },
"Reconnect": { "MinBackoffMs": 0, "MaxBackoffMs": 30000, "BackoffMultiplier": 2.0 },
"Write": { "Enabled": false },
@@ -183,6 +184,7 @@ Read off `MTConnectDriverOptions` / `MTConnectDriverConfigDto`
| `SampleIntervalMs` | `1000` | The Agent's `/sample?interval=` query param |
| `SampleCount` | `1000` | The Agent's `/sample?count=` query param. **Caps subscription throughput** at `SampleCount / SampleIntervalMs` — see [Subscribe](#subscribe--the-sample-pump) for the adaptive raise |
| `HeartbeatMs` | `10000` | The Agent's `/sample?heartbeat=` query param — also the watchdog's liveness window |
| `CommsLossConfirmMs` | `2000` | #846: how long the Agent must keep **refusing** connections before every subscribed value goes `BadNoCommunication` (see [Comms loss](#comms-loss-refused-vs-ambiguous-846)). `0` = on the first refusal; negative = fast path off (watchdog only). JSON-only — the AdminUI form preserves it but has no field |
| `Probe.Enabled` / `Probe.IntervalMs` / `Probe.TimeoutMs` | `true` / `5000` / `2000` | Background connectivity-probe knobs (mirrors `ModbusProbeOptions`) |
| `Reconnect.MinBackoffMs` / `MaxBackoffMs` / `BackoffMultiplier` | `0` / `30000` / `2.0` | Geometric backoff after a failed request or a dropped `/sample` stream |
| `Write.Enabled` | `false` | The driver-side half of the write-back opt-in. An absent block means disabled — see [Write-back](#write-back--observation-injection-off-by-default) |
@@ -397,7 +399,10 @@ observations per chunk, one chunk per `SampleIntervalMs`; a machine producing mo
buffer forever, re-baselining every buffer-length (dropping intermediate transitions) with nothing surfacing it.
Each chunk is now measured: `lag = lastSequence + 1 − nextSequence` (unknown when the Agent omits `lastSequence`)
and whether it was **saturated** (carried the full `count`). Three consecutive saturated chunks with a positive (or
unknown) lag reopen the stream at the cursor reached with `count` doubled, up to 16× `SampleCount` (max 65 536). A
unknown) lag reopen the stream at the cursor reached with `count` doubled, up to 16× `SampleCount` (max 65 536) —
**and never above the Agent's `/probe` `bufferSize`** (#846; a count above it is refused `OUT_OF_RANGE`, so a raise past
it only bought a refused stream and a step-back; a configured `SampleCount` already above the buffer is not raised and
is the #825 warning's business). A
raised stream that delivers nothing (an Agent refusing the larger count) steps back to the last count that delivered
and stops raising; at the ceiling a still-behind stream logs one Warning per pump. Every new pump starts from the
configured `SampleCount`. An Agent that refuses the raised count with an `MTConnectError` under HTTP 200 goes through
@@ -724,6 +729,32 @@ deliberately different, kept.
Agent's `bufferSize`, not a genuinely lapped ring buffer; no real machine tool or non-cppagent Agent was exercised.
**Closed by the #823 gate below**, except the real-machine / non-cppagent caveat.
### Comms loss: refused vs ambiguous (#846)
The runtime's comms-loss watchdog (Degraded + `LastSuccessfulRead` frozen 90 s ⇒ re-initialize, host fans
`BadNoCommunication`) is the contract for every **ambiguous** failure, and the #823 gate below measured what it costs:
values Good-but-stale for ~94–118 s. A **refused** connection is not ambiguous — nothing is listening at `AgentUri` — so
the `/sample` pump now handles it itself:
- **Definite = `SocketError.ConnectionRefused`** anywhere in the failure's exception chain (what `HttpClient` throws for
a port nobody listens on), on a `/sample` open or a re-baseline `/current`. A request or stream-idle timeout, a reset or
closed stream, DNS failure, a paused Agent, an `MTConnectError` refusal — all stay with the watchdog, unchanged.
- **Confirmed:** the refusals must continue, unbroken, for `CommsLossConfirmMs` (default 2 s); any answer from the Agent or
any ambiguous failure ends the run. While confirming, the reconnect delay is capped at the time remaining.
- **Then every subscribed value is published `BadNoCommunication`** (value null) **once per outage**, with one Warning.
Health was already Degraded with `LastSuccessfulRead` frozen by the stream failure, so the watchdog still re-initializes
at 90 s exactly as before — this only stops stale values reading Good in between. Reads already fall back to `/current`
on a degraded stream and fail on their own. Native CONDITIONS are untouched (the host annotates them on the
connectivity edge, as before).
- **Recovery republishes everything:** the first delivered chunk (a heartbeat counts) or re-baseline after a marked outage
republishes every subscribed value from the index — a resumed `/sample` carries only what changed, so an unchanged item
would otherwise stay Bad (the (value, status) recovery rule).
- Consequence for an Agent **restart** that refuses for longer than the window (the #823 restart refused for ~1 s, inside
it): values go Bad for the restart and come back on its first chunk / re-baseline. That is the honest answer — the
Agent was down.
Tests: `MTConnectCommsLossTests` (incl. an ambiguous-failure falsification control and the confirmation window).
### End-to-end gate through a deployed server (#823, 2026-11-11, docker-dev)
An MTConnect driver (`SampleCount` 1000, `HeartbeatMs` 10000, `RequestTimeoutMs` 5000) against the shared fixture,
@@ -735,7 +766,7 @@ NodeIds moved together in every leg. Fixture containers were restored healthy af
|---|---|
| Adapter stop | Both NodeIds `BadNoCommunication` within 1 s of the adapter dying (Agent marks items UNAVAILABLE); driver stayed Healthy. Adapter start ⇒ Good 1–2 s later. |
| Agent restart (`docker restart`, ~12 s) | Stream lost and reconnected ~1 s; the stale cursor was refused `OUT_OF_RANGE`, the re-baseline saw the new `instanceId`, **rediscovery raised** — `/hosts` shows the **RE-BROWSE** chip. Values stayed Good (outage shorter than the comms-loss window). |
| Agent stopped ~2 min | Values held Good-but-stale; **Bad at 118 s** (comms-loss watchdog, connection refused). Agent start ⇒ re-init on the 10 s retry, Good ~8 s after the Agent was back. |
| Agent stopped ~2 min | Values held Good-but-stale; **Bad at 118 s** (comms-loss watchdog, connection refused). Agent start ⇒ re-init on the 10 s retry, Good ~8 s after the Agent was back. **Since #846** a refused connection marks the values `BadNoCommunication` after `CommsLossConfirmMs` (2 s) instead — unit-tested, not yet re-gated live. |
| Agent paused (`docker pause`) | Stream idle watchdog at 24 s (2 × 10000 + 5000); **Bad at 94 s** (watchdog); unpause ⇒ Good within ~3 s. |
| Genuine buffer overrun | A stalling TCP proxy froze the driver's transport while 32 000 observations were injected (`AllowPut`) — the Agent's `firstSequence` passed the driver's cursor (75 856 → 77 083). On release the resume was refused `OUT_OF_RANGE`, the driver re-baselined from `/current` within 1 s and resumed; values Good, no restart misdiagnosed. A raised-count stream (count 16, interval 5 s) was **not** lapped by a 266 obs/s burst: chunks are sent back-to-back while data is pending, so `interval` does not cap throughput. |
| `SampleCount` vs `bufferSize` (#825) | Test Connect rendered the amber "OK with warning" chip and a deploy logged the Warning (`/hosts` shows `agentBufferSize`). The same gate measured the Agent's real edge — `count = 16384` streams, `16385` is `OUT_OF_RANGE` — so the first build's "at or above" threshold was corrected to "above" (pinned live by `Test_Connect_warns_when_SampleCount_exceeds_the_live_agents_buffer_size`). |