fix(runtime,adminui): per-tag write ordering, pessimistic init bound, unique generations, PostStop budget (fix round 1)
- Writes: a second write to a tag already inside the driver waits in the queue behind it (_tagsInFlight) while other tags still run under the cap of 4; queued writes are stamped and failed BadTimeout at dequeue once older than the 8 s host Ask budget (Props queuedWriteBudget) instead of being written late. - Initialise attempts are bounded pessimistically (initializeTimeout, default 30 s = 2x Tier C) on top of the invoker's token-only timeout: an overrun is logged as abandoned, reported InitializeFailed so the retry loop continues, and its late result is dropped by generation. Generations come from a process-wide Interlocked counter so a prior incarnation's late result cannot collide (F6). - PostStop shares one shutdownTimeout budget across shutdown + dispose. - TelemetryDialSupervisor cancels the establishment deadline on StreamStopped; FleetDiagnosticsClient builds the ActorSelection inside the try (a malformed node degrades to "unavailable"); the diagnostics page labels the column "Last health publish". - Tests: same-tag serialisation with a distinct-tag positive control; stale queued write BadTimeout; token-ignoring hung init -> later attempt connects; stopwatch assertion demoted to a ceiling. Refs #731 Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
This commit is contained in:
@@ -42,9 +42,11 @@ public sealed class FleetDiagnosticsClient : IFleetDiagnosticsClient
|
||||
/// <inheritdoc />
|
||||
public async Task<NodeDiagnosticsSnapshot> GetDiagnosticsAsync(NodeId nodeId, CancellationToken ct)
|
||||
{
|
||||
var selection = _system.ActorSelection($"akka.tcp://{_systemName}@{nodeId.Value}/user/driver-host");
|
||||
try
|
||||
{
|
||||
// Inside the try: a malformed node value makes ActorSelection throw, and that must degrade to
|
||||
// "unavailable" like a dead node does, not surface through the page's error boundary.
|
||||
var selection = _system.ActorSelection($"akka.tcp://{_systemName}@{nodeId.Value}/user/driver-host");
|
||||
using var linked = CancellationTokenSource.CreateLinkedTokenSource(ct);
|
||||
linked.CancelAfter(AskTimeout);
|
||||
return await selection.Ask<NodeDiagnosticsSnapshot>(
|
||||
|
||||
@@ -120,7 +120,7 @@ else
|
||||
<th>State</th>
|
||||
<th class="numeric">Connected devices</th>
|
||||
<th class="numeric">Faulted devices</th>
|
||||
<th>Last change</th>
|
||||
<th title="When the driver child last published a changed health snapshot (its spawn time until the first)">Last health publish</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
|
||||
@@ -252,6 +252,9 @@ public sealed class TelemetryDialSupervisor : ReceiveActor, IWithTimers
|
||||
return;
|
||||
}
|
||||
|
||||
// The stream is over; a still-armed establishment deadline for it must not fire into the reconnect.
|
||||
Timers.Cancel(DeadlineTimerKey(msg.NodeId));
|
||||
|
||||
if (dialer.Connected)
|
||||
{
|
||||
dialer.Connected = false;
|
||||
|
||||
Reference in New Issue
Block a user