# Phase 05 — Listener supervisor + auto-recovery

Wrap each `PlcListener` in a Polly-backed supervisor task. Failed binds (at startup or runtime) are retried per the design's recovery profile. Backend-connect Polly retries that were deferred from phase 03 land here too.

**Depends on:** Phase 03 (PlcListener, PlcConnectionPair).
**Parallel-safe with:** nothing (changes ProxyWorker, listener lifecycle, and connection-pair connect path simultaneously).

## Goal

Eliminate "startup race lost a port, service degraded for hours" as a real failure mode. After this phase, a port temporarily in use at boot will bind once it frees; a backend connect transient failure retries within a tight budget instead of immediately dropping the upstream client.

State per listener: `bound` / `recovering` / `stopped`. Reported on the status page (phase 07) via counters and a state field.

## Outputs

```
src/Mbproxy/Proxy/Supervision/PlcListenerSupervisor.cs   # owns one PlcListener; retry pipeline
src/Mbproxy/Proxy/Supervision/SupervisorState.cs         # enum + state-snapshot record
src/Mbproxy/Proxy/Supervision/PolicyFactory.cs           # builds Polly ResiliencePipelines from ResilienceOptions

tests/Mbproxy.Tests/Proxy/Supervision/SupervisorTests.cs          # port-conflict recovery, runtime-fault recovery
tests/Mbproxy.Tests/Proxy/Supervision/BackendConnectRetryTests.cs # Polly retry on backend connect
tests/Mbproxy.Tests/Proxy/Supervision/PolicyFactoryTests.cs       # unit
```

Modifications:
- `src/Mbproxy/Proxy/ProxyWorker.cs` — owns a `Dictionary<string, PlcListenerSupervisor>` instead of raw `PlcListener` instances. Stop/start of an individual listener now flows through the supervisor.
- `src/Mbproxy/Proxy/PlcConnectionPair.cs` — backend connect now goes through a Polly pipeline built from `ResilienceOptions.BackendConnect`. Remove the `// Phase 05: wrap in Polly` TODO from phase 03.
- `src/Mbproxy/Proxy/ProxyCounters.cs` — add `RecoveryAttempts` counter and `LastBindError` (last failure message, up to 256 chars). Update `CounterSnapshot` to include them.
- `src/Mbproxy/Proxy/RewriterLogEvents.cs` (or a sibling `SupervisorLogEvents.cs`) — add `[LoggerMessage]` definitions for `mbproxy.listener.recovered` (Info, `Plc`, `Port`, `AttemptCount`) and `mbproxy.backend.failed` (Warning, `Plc`, `Reason`). The latter event name already exists in design.md.

## Tasks

1. **`PolicyFactory.cs`** — converts `ResilienceOptions.BackendConnect` and `ResilienceOptions.ListenerRecovery` into `Polly.ResiliencePipeline` instances. Pipelines use `RetryStrategyOptions<T>` with `DelayGenerator` reading from the configured `BackoffMs` arrays. Listener recovery uses a 5-step initial backoff then steady-state at `SteadyStateMs` indefinitely (model as a custom delay generator that returns the steady-state value once the attempt index exceeds the initial array length).
2. **`SupervisorState.cs`** — `enum SupervisorState { Bound, Recovering, Stopped }` and a `record SupervisorSnapshot(SupervisorState State, string? LastBindError, int RecoveryAttempts)`.
3. **`PlcListenerSupervisor.cs`** —
   - Constructor: takes a `PlcOptions`, a `PerPlcContext`, the recovery `ResiliencePipeline`, and an `IPduPipeline`. Internally instantiates `PlcListener` lazily inside the retry loop.
   - `StartAsync(CancellationToken)`: launches a supervisor task. Inside the task: call `_listener.StartAsync()`. On success, transition to `Bound`, log `mbproxy.startup.bind` (first attempt) or `mbproxy.listener.recovered` (subsequent), and `await _listener.RunAsync(ct)` — which returns when the listener accepts loop ends.
   - On exception or normal-but-faulted return from the listener: transition to `Recovering`, log `mbproxy.startup.bind.failed`, increment `RecoveryAttempts`, dispose the failed listener, await Polly's next delay, retry.
   - `StopAsync`: transition to `Stopped`, cancel the supervisor token, await the supervisor task.
   - `Snapshot()`: returns `SupervisorSnapshot` for the status page.
4. **`PlcConnectionPair.cs` backend-connect retry** — wrap `Socket.ConnectAsync(host, port, ct)` in a `ResiliencePipeline.ExecuteAsync` built from `ResilienceOptions.BackendConnect`. After all attempts exhausted, close the upstream socket (as before) and log `mbproxy.backend.failed`. Crucial: backend-connect retries happen ONCE per upstream client connection (not per request); a connect failure terminates the pair.
5. **`ProxyWorker.cs`** — change to owning supervisors instead of raw listeners. Startup creates one supervisor per `PlcOptions`, starts them all in parallel (`await Task.WhenAll(...)` of their start tasks). The "ready" log event now fires after every supervisor has either reached `Bound` or entered `Recovering`. Shutdown stops all supervisors in parallel; clamp the total shutdown time at 5 s.

## Public surface declared in this phase

```csharp
namespace Mbproxy.Proxy.Supervision;

internal sealed class PlcListenerSupervisor : IAsyncDisposable {
    public string PlcName { get; }
    public Task StartAsync(CancellationToken ct);
    public Task StopAsync(CancellationToken ct);
    public SupervisorSnapshot Snapshot();
}

public sealed record SupervisorSnapshot(SupervisorState State, string? LastBindError, int RecoveryAttempts);
public enum SupervisorState { Bound, Recovering, Stopped }

internal static class PolicyFactory {
    public static ResiliencePipeline BuildBackendConnect(RetryProfile profile, ILogger logger);
    public static ResiliencePipeline BuildListenerRecovery(RecoveryProfile profile, ILogger logger);
}
```

`SupervisorSnapshot` is `public` because phase 07 (status page) consumes it. Everything else stays `internal`.

## Tests required

### Unit (`Category = Unit`)

`PolicyFactoryTests` (≥ 4 tests):

1. `BuildBackendConnect_ProducesPipeline_With3Attempts_Default`
2. `BuildBackendConnect_Backoff_MatchesConfig` — fake `TimeProvider`, assert delay sequence.
3. `BuildListenerRecovery_InitialBackoffFollowedBySteadyState` — drive 10 attempts, assert delays match.
4. `BuildBackendConnect_NoRetry_OnNonTransientException` — `SocketException` with WSAECONNREFUSED is retried; `ArgumentException` is not.

### Integration (`Category = Unit`; uses real sockets but no simulator)

`SupervisorTests` (≥ 5 tests):

1. `Supervisor_StartsListener_AndTransitionsToBound`
2. `Supervisor_StartFails_WhenPortInUse_TransitionsToRecovering` — bind a `TcpListener` on a free port first, then start the supervisor on the same port; assert `State == Recovering` and `LastBindError` is populated within 100 ms.
3. `Supervisor_Recovers_WhenPortFrees` — same setup as test 2, then dispose the blocking listener; assert the supervisor transitions to `Bound` and emits `mbproxy.listener.recovered` within `InitialBackoffMs[0] + 500ms`. Use an in-memory Serilog sink to verify the log event.
4. `Supervisor_RuntimeFault_TriggersRecovery` — replace the listener implementation with a faulting fake (or use reflection to force `_listener` to be one) and assert recovery kicks in.
5. `Supervisor_Stop_CleanlyTransitionsTo_Stopped_AndCancelsRetry` — supervisor in `Recovering` state, call `StopAsync`, assert it returns within 1 s without waiting out the next backoff window.

`BackendConnectRetryTests` (≥ 3 tests):

1. `BackendConnect_RetriesPerPipeline_OnConnectionRefused` — point a `PlcConnectionPair` at `127.0.0.1:1`, assert it sees exactly 3 connect attempts with the configured delays.
2. `BackendConnect_Succeeds_OnSecondAttempt_WhenBackendBecomesReachable` — start the pair against a closed port, open a listener on that port mid-backoff, assert connect succeeds and the pair runs.
3. `BackendConnect_AllAttemptsFail_ClosesUpstream` — pair gets a fresh upstream socket, never reaches a backend, the upstream socket is closed within `BackoffMs.Sum() + tolerance`.

### E2E (`Category = E2E`)

`SupervisorE2ETests` (≥ 2 tests, against the simulator):

1. `E2E_Recovery_When_BlockingListenerReleasesPort` — same shape as the unit recovery test, but with the simulator on the backend; confirms the supervisor doesn't disrupt the simulator-facing path during recovery.
2. `E2E_RecoveryAttempts_CounterIncrements_Visible_OnSnapshot` — drives the supervisor into recovery and back, then asserts `counters.RecoveryAttempts > 0`. Phase 07 will surface this on the HTTP endpoint; here we just verify the counter snapshot.

## Phase gate

- [ ] Zero-warnings build.
- [ ] All phase 00–04 tests still green.
- [ ] All new unit + integration tests green.
- [ ] E2E recovery test green when simulator is available.
- [ ] `mbproxy.listener.recovered` event log includes `AttemptCount` field.
- [ ] No deadlocks under StopAsync while supervisor is mid-backoff (verify by the test above).
- [ ] Backend-connect failures from phase 03 are now wrapped in Polly; the TODO comment from phase 03 is gone.
- [ ] [`../design.md`](../design.md) → "Listener auto-recovery" matches implementation. If during implementation the backoff arrays needed tweaking, update design.md in this PR.

## Out of scope

- Hot-reload-driven add/remove of supervisors (phase 06 owns reconcile).
- HTTP exposure of supervisor state (phase 07).
- Restart-from-crash diagnostics, Windows EventLog integration (phase 08).
- Adaptive backoff (e.g., jitter, exponential beyond the configured array). Stick to the configured schedule.

## Notes for the subagent

- Polly v8 (`Polly.Core`) is the target — `ResiliencePipeline` and `RetryStrategyOptions<T>`, not the v7 `Policy.Handle<>()` fluent API. If the package version pinned in phase 00 turns out to be v7, bump it in this phase and note the bump in the csproj comment.
- The supervisor task uses one `CancellationTokenSource` per supervisor instance. Cancelling it must cancel both the Polly delay AND the inner `_listener.RunAsync` cleanly. Polly's `ResiliencePipeline.ExecuteAsync(ct)` honours the token; double-check the listener does too.
- Do not introduce a generic "task supervisor" abstraction. `PlcListenerSupervisor` is the only thing supervising in this codebase; YAGNI on the framework.
- The supervisor must NOT swallow exceptions from `_listener.RunAsync` other than `OperationCanceledException`. Log them at Warning with the exception, then enter the recovery loop. Operators reading logs need to see WHY a listener died, not just that it was restarted.
