fix(gateway): remove the worker startup-probe retry options that could never retry (#188)

WorkerProcessStartedProbe only checks HasExited and throws a non-retryable
WorkerProcessLaunchException, so StartupProbeRetryAttempts and
StartupProbeRetryDelayMilliseconds configured a Polly retry that no shipped
probe could trigger. Real readiness is WorkerClient's WorkerReady handshake
under the same StartupTimeoutSeconds budget, so the probe now runs once and
the two options, their validation, the Polly pipeline (and the server's now
unused Polly.Core reference) and their docs are removed. A stale value in
configuration is ignored by the binder.

Tests: WorkerProcessLauncherTests.LaunchAsync_WhenStartupProbeFails_DoesNotRetryTheProbe,
WorkerOptions_HasNoStartupProbeRetryOptions (both fail on the old code).
This commit is contained in:
Joseph Doherty
2026-11-06 16:05:42 -05:00
parent 3a3d9e8e06
commit ff397b27be
10 changed files with 48 additions and 105 deletions
+3 -5
View File
@@ -109,7 +109,7 @@ The scalar fields mirror the counters and gauges. The four dictionaries provide
- `CommandFailuresByMethod` keys by gRPC method name.
- `EventsByFamily` keys by event family (the `Family` enum on a worker event).
- `EventsBySession` keys by `sessionId`; entries are removed via `RemoveSessionEvents` when a session closes so the map does not grow without bound.
- `RetryAttemptsByArea` keys by the resilience `area` tag, e.g. `worker_startup`.
- `RetryAttemptsByArea` keys by the resilience `area` tag passed to `RetryAttempted(area)`. No gateway code path records a retry at present (the worker launcher's startup-probe retry was removed in #188), so it is empty unless a future resilience path records one.
`EventsReceived` is read with `Interlocked.Read(ref _eventsReceived)` because `EventReceived` increments it via `Interlocked.Increment` outside the lock to keep the event-ingestion path non-blocking. `CommandsStarted`, `CommandsSucceeded`, `CommandsFailed`, and `CommandFailuresByMethod` are read the same way: the command counters run two-to-three times per gRPC call, so they are recorded with `Interlocked` and a `ConcurrentDictionary` rather than under `_syncRoot` (PERF-02). The two queue depths are pulled from their registered sources before the lock is taken, since those delegates reach into subscriber channels and worker clients.
@@ -152,15 +152,13 @@ _metrics.RemoveSessionEvents(session.SessionId);
### Worker process launcher
`Workers/WorkerProcessLauncher.cs` records process-level kills and startup retries:
`Workers/WorkerProcessLauncher.cs` records process-level kills:
```csharp
_metrics.WorkerKilled(reason);
...
_metrics.RetryAttempted("worker_startup");
```
The `worker_startup` tag is hard-coded so the `RetryAttemptsByArea` snapshot reports launcher retries distinctly from other resilience areas.
The launcher no longer records retries: its startup probe runs once (#188).
### Session worker client factory