C1 (critical): a latched "this endpoint cannot stream" verdict could end with a
GREEN driver holding a permanently silent subscription. DriverInstanceActor
handles a tag-set change as Unsubscribe-then-Subscribe with no re-initialize;
the unsubscribe cleared the stream-degraded flag, the resubscribe was refused by
the latch without a word, and one successful read then reported Healthy.
StartSampleStreamCore now re-asserts the degradation and logs a Warning naming
the remedy, and teardown no longer clears the flag while the latch is set.
NOT promoted to Faulted: DriverHealthReport maps any Faulted driver to a /readyz
503, so that would de-ready the whole node over one misconfigured Agent whose
reads are perfectly healthy. Faulted stays for the case that earns it.
I1: teardown waits are bounded (5 s) and honour the caller's token. They were
unbounded and ignored it, so DriverInstanceActor's 5 s budget was inert and
PostStop — which blocks a dispatcher thread on ShutdownAsync while holding the
lifecycle semaphore — could be wedged forever by one blocking subscriber. The
cancellation source is disposed only when the loop is provably gone. Applied to
StopProbeLoopAsync too: Task 13 reproduced the identical shape, and closing the
class in one method while leaving the other as the pattern to copy is worse than
not closing it.
I2: the reconnect ladder was monotonic for the process lifetime, so ~9 unrelated
drops pinned the driver at MaxBackoffMs forever. A stream that delivered before
dropping now starts a fresh ladder (at attempt 1, not 0, so MinBackoffMs is
still honoured).
I3: MaxBackoffMs <= 0 clamped every delay to zero — an operator-authorable
reconnect spin. Treated as unset, like a multiplier that cannot grow.
I4: the session published instanceId without NextSequence, so after a restart it
held the new agent's id beside the dead one's cursor, and a gap/OUT_OF_RANGE
re-baseline (id unchanged) never published the cursor at all. Both move in one
CAS, and the pump publishes its cursor as it exits.
I5: OnDataChange was a multicast Invoke — the first throwing handler aborted the
rest of the list, starving every later subscriber. Now walked by hand with the
catch inside the loop. The test that claimed to cover this registered the
recorder BEFORE the thrower, so it passed regardless; swapped, it was red.
Faults are latched to one Warning per stream generation (Debug thereafter) so a
consistently-throwing consumer cannot flood the log.
Also: the pump restarts after an unexpected fault (IsCompleted, not null); a
device-scope that matches nothing is a Warning, not a Debug tally; a device or
component with neither name nor id is skipped rather than emitting a blank path
segment; DiscoverAsync's remark no longer claims DriverInstanceActor retries
(it does not for a Once driver, and injection is dormant in v3); and two flake
seeds in the fake are gone (Dispose re-read its own counter; _disposeCts was
disposed under a racing SampleAsync).
439/439. Every changed behaviour falsified by mutation.
One shared /sample long poll behind every subscription handle. Subscribe fires
initial data per reference from the primed /current (Part 4 convention) and
returns without waiting on the stream; the pump runs on its own task under its
own token — never the caller's Subscribe token, which is disposed the moment
that call returns.
The pump owns every way the sequence can break:
- sequence gap, measured against the RUNNING cursor (the previous chunk's
nextSequence). Comparing against the opening `from` reports a gap on every
chunk once the buffer rolls — a /current re-baseline storm.
- OUT_OF_RANGE under HTTP 200 (InvalidDataException out of SampleAsync), which
IsSequenceGap cannot see at all — without it a driver recovers from falling a
little behind but not from falling a lot.
- agent restart, checked BEFORE the gap because a restart resets sequences and
so usually trips the gap check too; it clears the index (the held values
describe a device model that no longer exists) and tells the subscribers.
- every chunk advances the cursor, heartbeats included.
Re-baseline calls CurrentAsync directly on the client the pump already holds:
routing it through ReinitializeAsync would take the non-reentrant lifecycle
semaphore from inside a pump a lifecycle method may already be awaiting, and
hang the driver with no exception and no log. StopSampleStreamAsync is filled
in (same call sites) and InitializeAsync now stops the stream before teardown
too, so a re-Initialize on a live instance cannot leave a pump enumerating a
disposed client.
MTConnectStreamEnded/TimeoutException reconnect under a geometric backoff with
a 100 ms growth floor (MinBackoffMs defaults to 0, and 0 x multiplier is still
0). MTConnectStreamNotSupportedException does NOT retry — it latches, reports
loudly, and is cleared only by a re-initialize.
Health precedence is explicit: /current and /sample fail independently, so each
path owns a flag, clears only its own, and Healthy requires both clear. Neither
can paper over the other's failure.
29 new tests, all deterministic — no sleeps, no wall-clock assertions. The fake
grew per-generation sample streams (so a reconnect has somewhere to land),
scripted /current answers, scripted sample failures, and a chunk-consumed
signal that also fires when the consumer abandons the stream. 375/375 green.
Important 1 - the probe-model triple was read outside _lifecycle while every
write happened under it. Packed (Client, ProbeModel, ProbeModelDataItemCount)
into one immutable AgentSession swapped by a single Volatile.Write, matching
what _index already does, rather than locking the readers: both await the
network, so taking _lifecycle would stall a shutdown for a whole
RequestTimeoutMs and would widen the non-reentrancy hazard. GetOrFetch and
Flush re-publish via CompareExchange so a late re-cache cannot undo a
concurrent teardown/re-init, and a client disposed mid-fetch now surfaces as
the same "not connected" InvalidOperationException browse already handles
instead of a raw ObjectDisposedException.
Important 2 - HasConfigBody decided emptiness by matching the literals "{}" /
"[]", so "{ }", "{\n}" and every pretty-printed empty document fell through to
ParseOptions, failed the required-AgentUri check, and turned a semantically
empty config into a cold-start fault. Now parsed: an object/array with no
elements (or a bare null) is empty; malformed text is deliberately NOT empty so
ParseOptions produces the real quoted error rather than silently starting on
stale options.
Important 3 - documented the _lifecycle non-reentrancy hazard in the code, on
both the semaphore and StopSampleStreamAsync, naming Task 11's re-baseline as
the specific path that would deadlock and giving the CurrentAsync-directly
pattern that avoids it.
Important 4 - removed the Initialize/Reinitialize asymmetry rather than
documenting it. Both now share one rule via ResolveIncomingOptions: an
unreadable config document never destroys working state, and faults only a
driver that had none. InitializeAsync previously tore down before parsing, so a
bad document on a live instance destroyed a healthy client - the exact outcome
ReinitializeAsync was written to avoid.
Minors: SafeDispose traces the swallowed disposal fault at Debug (a client that
cannot be released is how a handle leak starts); the RequirePositive test is a
Theory over all four timing knobs, not just RequestTimeoutMs (arch-review
01/S-6); the footprint test no longer overclaims "is zero before initialize".
CannedAgentClient gains a one-shot ProbeGate so a lifecycle change landing
mid-request is deterministic - no timers. 346/346 (329 + 17). Six mutations
verified: fetch-CAS, disposed-translation, flush retired-session guard,
literal HasConfigBody, teardown-before-parse, and two dropped RequirePositive
calls each fail only their own tests.
One deadline-bounded /current per batch, answered out of that one whole-device
document: N references cost one round-trip, one snapshot each, in request order,
duplicates included.
DEVIATION (deliberate, from the plan's implied shared-index write): the response
is indexed into a PER-CALL snapshot index, never into the driver's shared
observation index. That index is written by Task 11's /sample pump and is
last-write-wins with no timestamp arbitration; a read folding its /current into
it would be a second writer racing the first, and /current is frequently the
OLDER document (the Agent composes it while the pump's newer delta is in
flight). Losing that race rolls a subscribed value backwards and leaves it there
until the data item next changes. Both paths coerce through the same
MTConnectObservationIndex logic, so a read and a subscription report a given
observation identically -- the read path is simply a pure function of
(document, authored tags).
Failure posture is the inverse of InitializeAsync: nothing but genuine caller
cancellation crosses the capability boundary. Unreachable Agent / blown deadline
/ unusable answer => every ref codes BadCommunicationError and the driver
degrades (never downgrading Faulted); no client at all (pre-initialize, faulted,
post-shutdown) => BadNotConnected with health untouched. The index's
BadWaitingForInitialData / BadNodeIdUnknown / BadNoCommunication / BadNotSupported
distinctions pass through unmodified.
21 tests pinning arity, order, duplicates, empty batch, one-round-trip, each Bad
code, read-time freshness, the anti-clobber invariant, the deadline, and
cancellation-propagates-but-agent-failure-does-not. CannedAgentClient gains a
cancellation-observing CurrentGate (still no timers or sleeps of its own).
Test csproj takes the AbCip/S7 OTOPCUA0001 NoWarn -- this suite calls the
capability directly by design.
329/329 green.
MTConnectDriver implements IDriver only: connection-free ctor, initialize
(deadline-bounded /probe + /current, caching the device model, the agent
instanceId, and a primed observation index), reinitialize, shutdown, health,
footprint, and cache flush.
Decisions worth knowing:
- The driverConfigJson argument is HONOURED, not ignored. DriverInstanceActor
delivers a config change only via ReinitializeAsync(json) on the live
instance, so a driver reading only its ctor options would turn every MTConnect
config edit into a silent no-op that still sealed the deployment green.
ParseOptions lives on the driver; Task 15's factory must delegate to it rather
than deserialize a second DTO.
- /probe ok but priming /current failing is Faulted, not Healthy-with-no-data:
the index IS the read surface and the /sample cursor comes from /current.
- ReinitializeAsync rebuilds the client whenever ANY option the client reads at
construction changed - AgentUri, DeviceName, RequestTimeoutMs, HeartbeatMs,
SampleIntervalMs, SampleCount - not just the first two. The timing knobs are
baked into the deadlines, the watchdog window, and the /sample query string.
- An unparseable config faults a cold start but leaves a RUNNING driver serving
its previous valid configuration (the deployment fails instead).
- IMTConnectAgentClient now extends IDisposable rather than the driver doing
`as IDisposable`, which would silently no-op against a non-disposable fake and
leak an HttpClient + socket handler per re-deploy. Sync, not async: every
resource behind the seam is sync-disposable, and the async unwind belongs to
the SampleAsync enumerator the pump owns.
- Flush drops the probe/browse cache and keeps the index; every model consumer
goes through GetOrFetchProbeModelAsync so a flush cannot wedge browse.
Tests: CannedAgentClient (shared, deterministic - channel-backed scripted
/sample with no timers, call counts, disposal tracking, failure injection) +
27 lifecycle tests. 308/308 in the project.
The concurrency test asserted a post-Apply post-condition from inside the
race window: dev1_pos is authored, so a reader beating the first Apply
correctly observes BadWaitingForInitialData, which the test's two-member
"legal" set omitted. Consistently red (10/10) rather than intermittent.
Rewritten to assert only properties that hold at every instant — the whole
(status, value, source timestamp) triple against the three legal snapshots,
which is the real torn-snapshot check — plus a read counter guarding a
vacuous pass and one deterministic closing Apply for an exact end state.
The index is unchanged here; BadWaitingForInitialData is a designed state.
Also codes structured (DATA_SET / TABLE) observations BadNotSupported now
that the parser supplies MTConnectObservation.IsStructured, so a two-entry
data set no longer surfaces as the Good string "12".
Code review of c46540ae approved the framing algorithm but moved the risk
onto the contract around it.
C1 (critical). SampleAsync returned normally on three different events:
EOF, the closing --boundary--, and a response that was not multipart at
all (where it yielded one parsed snapshot and finished). The seam
documents the stream as yielding until ct is cancelled, so a Task 11 pump
written to that contract would silently drop its subscription or hot-loop
reconnecting; the non-multipart leg reported a configuration error as a
healthy finished stream, which is the #485 quiet-successful-termination
shape aimed at the very next task. Every non-cancellation end now throws:
MTConnectStreamEndedException (ConnectionClosed / ClosingBoundary -
transient) or MTConnectStreamNotSupportedException (configuration -
reconnecting reproduces it forever), sharing one catchable base. The
non-multipart body is still parsed first so an Agent's own MTConnectError
text wins, but the document is NOT yielded.
I1. IsSequenceGap moved to IMTConnectAgentClient (static) and its first
parameter is now expectedFrom. The old name and doc were wrong for every
chunk after the first - the correct comparand is the PREVIOUS chunk's
NextSequence, and comparing against the opening "from" argument reports a
gap on every chunk once the ring buffer rolls, i.e. an endless /current
re-baseline storm. Both doc blocks also now record that cppagent answers
from < firstSequence with an OUT_OF_RANGE MTConnectError under HTTP 200,
which surfaces as InvalidDataException and NOT as a gap - so Task 11 must
treat that parse failure as a re-baseline trigger too.
I2. Every multipart test served the whole body from one buffer, so every
framing test completed in a single ReadAsync - the split-boundary path,
the split-header path and the multi-fill loop were correct by inspection
only. A ChunkedStream double (N bytes per read, N = 1/3/7) now drives the
fixtures through a split transport, plus a >16 KB document that forces a
buffer resize. Falsifiability: removing either scan overlap, or
collapsing the fill loop, fails ONLY the new chunked tests.
I3. Disposal mid-enumeration surfaces as OperationCanceledException via
an internal dispose-linked token, not a raw ObjectDisposedException that
Task 9's re-init would inflict on an enumerating pump.
I5. HeartbeatMs, SampleIntervalMs and SampleCount join RequestTimeoutMs
in construction-time positive-value validation. All three reach the query
string and an Agent answers heartbeat=0 with an HTTP-200 MTConnectError
that names no config key.
I4/M2. The no-Content-length framing fallback logs a one-shot warning
(the client takes an optional ILogger); a multipart response with no
boundary parameter fails fast instead of degrading into a watchdog
timeout.
M5/M6. Shared element/attribute reading rules extracted to MTConnectXml;
the probe parser documents why it alone does not trim BOM/whitespace. The
literal U+FEFF in source became a '' escape.
Also, from Task 8: MTConnectObservation gained IsStructured (default
false), set from element.HasElements. A DATA_SET/TABLE observation's
<Entry> children concatenate through element.Value to nonsense ("12" for
two entries) that the index would publish as Good, and only the parser
can tell that apart from a legitimate space-bearing Message. False for
CONDITION observations - their value comes from the element name, so
child content cannot corrupt it.
275/275 green in this suite's own files.
Adds the /current and /sample legs to MTConnectAgentClient:
- MTConnectStreamsParser: hand-rolled System.Xml.Linq parse of an
MTConnectStreams document into MTConnectStreamsResult. Observations are
every element child of <Samples>/<Events>/<Condition>, keyed by
dataItemId (the element NAME is the data item's type in PascalCase, so
a fixed name set would drop observations). A CONDITION's value is its
element name, not its text - <Normal/> is empty - and <Unavailable/>
normalizes onto the same UNAVAILABLE sentinel a Sample/Event carries as
text, so Task 8's no-comms mapping sees one token. Timestamps are
normalized to DateTimeKind.Utc explicitly.
- IsSequenceGap(requestedFrom, chunk) => chunk.FirstSequence >
requestedFrom, so Task 11's pump can re-baseline after a ring-buffer
overflow. Equality is NOT a gap.
- SampleAsync: ResponseHeadersRead + a hand-rolled
multipart/x-mixed-replace frame reader (Content-length fast path,
boundary-scan fallback), bounded by a heartbeat watchdog derived from
HeartbeatMs so a silent Agent faults instead of wedging the pump - the
S7 frozen-peer shape (arch-review R2-01). The long poll gets its own
HttpClient with an infinite Timeout so the unary /probe + /current
deadline stays bounded.
Task 0 package decision, option (c): both TrakHound PackageReferences and
their PackageVersions are dropped. MTConnectHttpClientStream takes a URL
and dials its own socket (no handler/stream seam, so it cannot serve a
socket-free unit suite), and it emits already-parsed documents through the
formatter lookup Task 6 proved is missing - framing-only reuse was never
actually on offer. The driver now depends on nothing but the BCL. Recorded
in the plan's Task 0 CORRECTION block.
Also folds in two Task 6 review carry-overs: the stale
IMTConnectAgentClient doc comment claiming a TrakHound-backed
implementation, and a probe test constructing a mixed-namespace vendor
extension (<x:Pump> with default-namespace <DataItems> children) that
backs the parser's ignore-the-namespace claim with a test.
187/187 green.
Joseph Doherty