lmxopcua/archreview/plans/06-gateway-integrations-plan.md

# Plan 06 — Gateway Integrations (Galaxy Driver + Historian Gateway Driver)

> **Status (2026-07-08):** ✅ **U-1 doc DONE** (CLAUDE.md KNOWN LIMITATION 2 corrected — historized-ref feed
> is wired). Remaining: S-1 (Galaxy write fail-closed — failure-visibility), S-6 (dormant historian health
> surface — built-but-never-wired). See [`STATUS.md`](STATUS.md).

- **Source report:** `archreview/06-gateway-integrations.md`
- **Scope:** `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy` (+ `.Contracts`, `.Browser`), `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway`
- **Commit reviewed against:** current tree (report cut at `9cad9ed0`)
- **Verification method:** every cited file:line opened and checked against the current tree.

## Verification summary

| Finding | Sev | Status | Note |
|---|---|---|---|
| S-1 Galaxy write optimistic-Good | High | **Confirmed** | `GatewayGalaxyDataWriter.cs` 234-243 (advise-fail proceeds), 281-302 (empty statuses → Good), 162-165 (silent-loss comment) all present verbatim. |
| S-2 EventPump drops newest | Medium | **Confirmed** | `EventPump.cs:136-139` `TryWrite` → count drop, newest-dropped by construction. |
| S-3 Read can hang on non-cancellable token | Medium | **Confirmed** | `GalaxyDriver.cs:738-753`: BadTimeout fill only via `cancellationToken.Register`; awaits at 753 block forever with `CancellationToken.None`. |
| S-4 Transport-fault detection is EventPump-only | Medium | **Confirmed** | `ReportTransportFailure` sole prod caller is `OnEventPumpStreamFault` (`GalaxyDriver.cs:943`); `WriteAsync` (line 803) and subscribe path never feed the supervisor. |
| S-5 Outbox `RemoveAsync` prefix truncation silent | Medium | **Confirmed** | `FasterLogHistorizationOutbox.cs:158-182`: removes target + all older live entries, no `_droppedCount` bump on non-target removals. |
| S-6 Historian health flags dormant | Medium | **Confirmed (worse)** | `RefreshConnectionStateAsync` has only test callers; **`GetHealthSnapshot()` itself has zero production callers** — the whole snapshot is currently unconsumed in-process. |
| S-7 Alarm feed reconnect no backoff | Low | **Confirmed** | `GatewayGalaxyAlarmFeed.cs:141` fixed `_reconnectDelay` (default 5 s), no exponential/jitter. |
| S-8 `GalaxyMxSession` unsynchronized state | Low | **Confirmed** | `GalaxyMxSession.cs:28-32` plain fields; `RecreateAsync` (112-117) tears down without a lock. Emergent safety accepted. |
| S-9 Store-and-forward discipline | Positive | Confirmed positive | No action. |
| S-10 TLS / API-key posture | Positive | Confirmed positive | No action. |
| P-1 Read = 3 round-trips | Medium | **Confirmed** | `ReadViaSubscribeOnceAsync` 658-780; MxAccess-forced. Doc + optional cache. |
| P-2 Serial `SendEvent` per event | Medium | Confirmed (accepted) | Gateway-constrained; track batched `SendEvents` on sidecar. |
| P-3 Channel count | Low | Confirmed (accepted) | Documented trade-off; no action. |
| P-4 Indexed fan-in/out | Positive | Confirmed positive | No action. |
| P-5 HistoryRead buffers; `maxEvents<=0` unbounded | Low | **Confirmed** | `GatewayHistorianDataSource.cs:142-168` `hasCap = maxEvents>0`; when false collects unbounded. |
| P-6 Peek holds lock over disk I/O | Low | **Confirmed** | `FasterLogHistorizationOutbox.cs:145-152`. Negligible at 64-batch. |
| C-1 Secret resolver | Positive | Confirmed positive | No action. |
| C-2 Seam quality | Positive | Confirmed positive | No action. |
| C-3 Historian seam leaks wire types | Medium | **Confirmed** | `IHistorianGatewayClient.cs:1-11` proto-typed by design. Doc-guard only. |
| C-4 Options-validation idioms diverge | Low | **Confirmed** | Galaxy throw-on-construct vs Historian warnings-list. Convention note. |
| C-5 Retired Wonderware dirs on disk | Low | **Confirmed** | 3 dirs present, **0** `slnx` references. Delete. |
| C-6 EventPump drops unknown families unmetered | Low | **Confirmed** | `EventPump.cs:199-206` `default: return;` no counter. |
| U-1 CLAUDE.md KNOWN LIMITATION 2 stale | High (doc) | **Confirmed stale** | Feed fully wired: `AddressSpaceApplier.cs:343` `UpdateHistorizedRefs` → `ActorHistorizedTagSubscriptionSink` → recorder `OnUpdateHistorizedRefs` (`ContinuousHistorizationRecorder.cs:186/311`). Doc must be corrected; path never live-verified. |
| U-2 Live-validation gate partially run | Medium | Confirmed | Infra complete; needs a full documented run + CLAUDE.md status. |
| U-3 WriteSecured user id stubbed at 0 | Medium | **Confirmed** | `GatewayGalaxyDataWriter.cs:263-264` `CurrentUserId=0, VerifierUserId=0`. |
| U-4 Dormant paths inventory | Medium | **Confirmed** | `HistorianGatewayClientAdapter.cs:88` drops `maxEvents` on the wire; `GalaxyDriver.cs:314-316` replay note; `FlushOptionalCachesAsync` no-op. |
| U-5 Outbox serializer has no version byte | Low | **Confirmed** | `HistorizationOutboxEntrySerializer.cs:14-16` fixed positional layout, no prefix. |
| U-6 Test-coverage thin spots | — | Confirmed | S-1 failure branch, S-5 out-of-order, S-3 hang, C-6 filter all untested. Folded into each finding's test section. |

**Totals: 28 findings — 23 actionable, 5 positive (no action). Of the 23 actionable, all 23 confirmed against the current tree; 0 stale-as-already-fixed (U-1 is stale *documentation* against correct code).**

Cross-cutting alignment with `00-OVERALL.md`: S-1 is part of OVERALL action #7 (failure visibility); S-6 + U-1 are instances of OVERALL theme #1 ("built-but-never-wired"); U-1 + U-2 are OVERALL theme #5 (doc drift); U-4/C-3 are theme #4 (gateway-gap workarounds accrue client-side).

---

## Priority-ordered plan

Priority key: **P0** ship first (data-integrity / visibility), **P1** next (fault resilience), **P2** hygiene/docs, **P3** nice-to-have.

---

### P0-A — S-1: Galaxy write success is optimistic; a committed-write failure can never surface

**Restatement:** Empty MX status arrays translate to `Good`; a failed supervisory-advise logs a warning and lets the raw `Write` proceed even though the file's own comment says such a write "never reaches the galaxy"; write-outcome self-correction (#5) structurally cannot fire because Galaxy `ExecuteWrite` is fire-and-forget.

**Verification:** Confirmed. `GatewayGalaxyDataWriter.cs`:
- 234-243: on `AdviseSupervisory` protocol failure → `TryRemove` the handle, log warning, **fall through** to `WriteRawAsync`.
- 281-302: `TranslateReply` → protocol-status check, then first MX status row, else `new WriteResult(StatusCodeMap.Good)`.
- 162-165: in-source comment confirms a non-advised raw Write "doesn't throw (reply looks OK) but the value never reaches the galaxy."

**Root cause:** Two independent optimistic defaults. (1) The gateway reply's `Statuses` array reflects *command acceptance at the worker*, not the eventual COM-side `WriteComplete`; an empty array is treated as success. (2) The supervisory-advise precondition is best-effort — when it fails the code still issues a Write it knows won't commit, and reports the Write's (empty/OK) reply as `Good`.

**Design.**

*In-repo (this PR) — three layers, cheap and independent:*

1. **Fail-closed on advise failure.** Change `EnsureSupervisoryAdvisedAsync` to return a bool (advised? true/false). In `WriteOneAsync`, when a non-secured write's advise returns false, **do not issue the raw Write** — return `new WriteResult(StatusCodeMap.BadWaitingForInitialData /* or Uncertain */)`. Rationale: the code already *knows* the value can't commit; sending it and reporting Good is the exact silent-loss the comment describes. Preferred status: `Uncertain_NoCommunicationLastUsableValue` is wrong (implies a read); use `Bad` family so #5 self-correction *could* fire (`BadNotConnected`/`BadWaitingForInitialData`). Decision: return **`StatusCodeMap.BadCommunicationError`** — it is already the writer's "couldn't reach the device" status, so the node-write router / self-correction path treats it consistently with a transport failure. (The report suggests `Uncertain`; `Bad` is stronger and lets self-correction revert the phantom-Good node — align with OVERALL theme #3 "the failure must surface *somewhere*".)

2. **Empty-statuses is not Good for a data write.** `TranslateReply`: when `reply.Statuses.Count == 0` **and** the reply is a data-write reply, return `StatusCodeMap.UncertainNoCommunicationLastUsableValue`-class or, more defensibly, keep `Good` **only** when the gateway contract guarantees a status row on success. This needs a gateway-contract check (see cross-repo). Interim: keep `Good` but **meter** it (layer 3) so the unconfirmed rate is observable, and add an XML-doc note that empty-array-Good is provisional pending the `WriteComplete` correlation.

3. **`galaxy.writes.unconfirmed` counter.** Add a `Counter<long>` on the existing `GalaxyTelemetry`/pump meter (`ZB.MOM.WW.OtOpcUa.Driver.Galaxy`), incremented whenever `TranslateReply` returns Good off an *empty* statuses array, and a `galaxy.writes.advise_failed` counter on the advise-failure branch. Gives ops the signal the report asks for.

*Cross-repo (mxaccessgw) — the real fix, tracked follow-on:* pursue a gateway-side `WriteComplete` correlation so the `MxCommandReply.Statuses` row carries the real COM commit outcome. The `OnWriteComplete` event family already exists in the proto and is currently *filtered out* by `EventPump.Dispatch` (`EventPump.cs:199-206`). Design: gateway correlates the async `WriteComplete` back to the originating command's `ClientCorrelationId` and folds the MX status into the unary reply's `Statuses` before returning (making Write synchronously authoritative), **or** emits `OnWriteComplete` on the event stream keyed by correlation id and the driver awaits it with a deadline. The synchronous-fold option is cleaner for the OPC UA write contract (write is a blocking service call). This is a sibling-repo change; file it on the mxaccessgw backlog and link from the in-source comment at `GatewayGalaxyDataWriter.cs:162-165`.

**Implementation steps (in-repo):**
- `GatewayGalaxyDataWriter.cs`: change `EnsureSupervisoryAdvisedAsync` → `Task<bool>`; branch in `WriteOneAsync` to short-circuit to `BadCommunicationError` when advise failed.
- Add `WriteUnconfirmed` / `WriteAdviseFailed` counters (new static `Counter<long>` fields; reuse `EventPump.MeterName` string or a shared `GalaxyTelemetry` meter — check `GalaxyTelemetry.cs` for the existing meter name to avoid a second Meter).
- `TranslateReply`: increment `WriteUnconfirmed` on the empty-statuses branch; add doc note.

**Tests (unit):**
- New `GatewayGalaxyDataWriterTests` cases (writer already has an internal ctor + `SeedHandleCachesForTest`): advise-fails → assert returned status is `BadCommunicationError` and **no** `WriteRawAsync` was issued (fake session records calls).
- Empty-statuses reply → assert Good returned **and** the unconfirmed counter incremented (via a `MeterListener`).
- Protocol-status non-OK → `BadCommunicationError` (regression guard).

**Tests (live gateway):** extend the skip-gated `GatewayGalaxyLiveReopenAndWriteTests` with a case that forces an advise-failure (or asserts the unconfirmed counter is emitted on a normal write). Recipe:
```
KEY=$(docker exec otopcua-dev-central-1-1 printenv GALAXY_MXGW_API_KEY)
MXGW_ENDPOINT=http://10.100.0.48:5120 GALAXY_MXGW_API_KEY="$KEY" \
  dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests \
  --filter FullyQualifiedName~GatewayGalaxyLiveReopenAndWrite
```

**Effort:** S (in-repo layers 1+3), M (layer 2 + cross-repo correlation). **Risk/blast radius:** Low in-repo — the change makes a currently-silent-loss visible; the only behavioral change is that advise-failed writes now return Bad instead of phantom-Good, which is strictly more correct and feeds #5 revert. Live-verify to confirm normal writes still return Good.

---

### P0-B — S-6: Historian health snapshot's connection flags are dormant (and the snapshot is unconsumed)

**Restatement:** `RefreshConnectionStateAsync` has no production caller, so `ProcessConnectionOpen`/`EventConnectionOpen` are permanently false; deeper: `GetHealthSnapshot()` itself is never called in production.

**Verification:** Confirmed and worse than reported. `RefreshConnectionStateAsync` callers = 3 test lines only. `GetHealthSnapshot()` invocation grep across `src` returns only the interface declaration — no production consumer. So the entire health surface (`GetHealthSnapshot` + the refresh) is dead weight in a deployed host today.

**Root cause:** The "register-AND-pass-into-consumer" trap (same class as PR #423 provisioner, OVERALL theme #1): the component was built and unit-tested, but neither the periodic refresh nor a snapshot consumer was wired.

**Design.** Two decisions:

1. **Is the snapshot meant to be surfaced?** Check `HistorianHealthSnapshot`'s intended consumer (AdminUI historian-status panel / a health endpoint). If yes → wire it. If no (the flags predate a since-abandoned dashboard) → **remove** `ProcessConnectionOpen`/`EventConnectionOpen` + `RefreshConnectionStateAsync` to kill the misleading-false surface. The report offers both; pick based on whether any dashboard is planned. Recommendation: **wire it**, because the other integrations (Galaxy `HostStatusAggregator`) *do* surface connectivity and a historian-down indicator is genuinely useful.

2. **How to drive the refresh.** Add a lightweight `IHostedService` (`HistorianHealthRefreshHostedService`) that, when `ServerHistorian:Enabled`, resolves the registered `IHistorianDataSource`, and if it is a `GatewayHistorianDataSource`, calls `RefreshConnectionStateAsync` on a timer (reuse the existing health-probe cadence if one exists — check `OtOpcUaServerHostedService` for an existing periodic loop to fold into rather than spawning a second timer). Also wire a consumer: project the snapshot into whatever the AdminUI/health endpoint reads.

**Implementation steps:**
- New `HistorianHealthRefreshHostedService` in `ZB.MOM.WW.OtOpcUa.Runtime` (or Host), registered in the historian DI block alongside `AddServerHistorian`. Guard on `Enabled`. Interval from a new `ServerHistorianOptions.HealthRefreshSeconds` (default 15).
- Cast/seam: `RefreshConnectionStateAsync` is on the concrete `GatewayHistorianDataSource`, not `IHistorianDataSource`. Either (a) add an optional `IHistorianConnectionProbe { Task RefreshConnectionStateAsync(ct) }` capability interface the gateway source implements and the hosted service checks for, or (b) resolve the concrete type. Prefer (a) — matches the capability-interface pattern used elsewhere and keeps `NullHistorianDataSource` clean.
- Wire the snapshot consumer (AdminUI panel or `/health`); if none is desired, execute decision-branch (1) removal instead.

**Tests:**
- Unit: hosted service calls `RefreshConnectionStateAsync` on tick against a fake source; disabled → never resolves.
- Unit: `GatewayHealthSnapshotTests` already covers the refresh math — add an assertion that after a successful refresh the snapshot flags are true.
- Live: after wiring, run the historian live suite (U-2) and confirm the snapshot flips to connected against the real sidecar.

**Effort:** S (wire refresh) / M (if adding a consumer surface). **Risk:** Low. **Note:** live-`/run` required per OVERALL theme #1 — do not close on unit tests alone.

---

### P0-C — U-1: CLAUDE.md KNOWN LIMITATION 2 is stale; continuous-historization value-capture never live-verified

**Restatement:** CLAUDE.md says the recorder is spawned with an empty ref-set and "historizes nothing"; the code has closed that gap but the end-to-end path was never live-verified.

**Verification:** Confirmed. The feed is fully wired: `ServiceCollectionExtensions.cs:272` still spawns with `Array.Empty<string>()`, **but** wraps in `ActorHistorizedTagSubscriptionSink` (line 290-292) and hands it to the applier; `AddressSpaceApplier.cs:310-343` `FeedHistorizedRefs` computes the per-deploy add/remove delta and calls `_historizedSubscriptions.UpdateHistorizedRefs`; `ContinuousHistorizationRecorder.cs:186/311` receives `UpdateHistorizedRefs` and re-registers. CLAUDE.md's "Known Limitation 2" text is therefore factually wrong.

**Root cause:** Documentation drift — the feed landed after the CLAUDE.md note was written and the note was never reconciled.

**Design/steps:**
1. **Doc fix (this PR):** rewrite CLAUDE.md "KNOWN LIMITATION 2" to state the ref-feed is wired (applier `FeedHistorizedRefs` → `ActorHistorizedTagSubscriptionSink.UpdateHistorizedRefs` → recorder `OnUpdateHistorizedRefs`), and downgrade it from "records no values" to "wired; pending a live end-to-end verification." Propagate the same correction to the `../scadaproj/CLAUDE.md` OtOpcUa index entry (per the repo's cross-repo propagation rule).
2. **Live-verify the value-capture path (the real close-out):** deploy a config with a historized numeric tag against a real HistorianGateway (VPN to the sidecar / `wonder-sql-vd03`, gateway with `RuntimeDb:Enabled=true`), drive a value change, and confirm `WriteLiveValues` lands (outbox drains, gateway shows the sample). This is the OVERALL theme #1 "prove production wiring" gate.
3. **Add a restart-convergence integration test** (unit-level with a fake gateway value-writer + real outbox): deploy delta adds refs → mux value → outbox append → drain calls writer; assert the writer received the value. This closes the "wired but inert" risk with a repeatable test even before the live run.

**Effort:** S (doc) + M (live-verify + convergence test). **Risk:** Doc-only change is zero-risk; the live run may surface a real inertness bug (exactly what OVERALL warns about) — budget for a fix if it does.

---

### P1-A — S-4: Transport-failure detection is EventPump-only; a write-only/idle driver never degrades

**Restatement:** `ReportTransportFailure` is only called from the EventPump stream-fault callback; a driver that only writes or is idle after discovery keeps `GetHealth()==Healthy` through a gateway restart, and each write fails per-request forever with no reopen/replay.

**Verification:** Confirmed. Sole prod caller `GalaxyDriver.cs:943` (`OnEventPumpStreamFault`). `WriteAsync` (803) delegates straight to `_dataWriter.WriteAsync` with no supervisor hand-off; failed unary RPCs (SubscribeBulk, Write, AcknowledgeAlarm) don't feed the supervisor.

**Root cause:** Fault detection was built around the long-lived StreamEvents pump only; unary RPC failures were treated as per-call and never escalated to the recovery state machine.

**Design.** `ReportTransportFailure` is idempotent by design (the supervisor single-flights recovery), so reporting from more sites is cheap and safe. Add classified transport-failure reporting from the unary paths:
- In `GatewayGalaxyDataWriter.WriteOneAsync`'s catch (currently maps to `BadCommunicationError`), and in the subscribe path, classify the exception (RpcException with transport status codes: Unavailable, DeadlineExceeded, Internal, etc.) and, when transport-fatal, invoke a supplied `Action<Exception>? reportTransportFailure` callback (mirror the pump's `onStreamFault` pattern) that routes to `_supervisor.ReportTransportFailure`.
- Wire the callback from `GalaxyDriver` when it constructs the writer/subscriber, guarded like `OnEventPumpStreamFault` (null supervisor = skeleton path, no-op).
- Reuse the existing transport-classification helper if one exists (grep `StatusCodeMap`/`ReconnectSupervisor` for a classifier); otherwise add a small `IsTransportFatal(Exception)` in the driver.

**Implementation steps:**
- Add a `Func<Exception,bool>`/callback seam to `GatewayGalaxyDataWriter` (optional ctor arg, like `subscribedHandleSource`) or wrap the call at the `GalaxyDriver.WriteAsync` boundary (cleaner — keep the writer pure). Prefer wrapping at `GalaxyDriver`: catch transport-classified failures returned/thrown from the writer and call `supervisor.ReportTransportFailure`.
- Same for `SubscribeAsync`/`ReadViaSubscribeOnceAsync` SubscribeBulk failures and `GatewayGalaxyAlarmAcknowledger`.

**Tests (unit):** fake writer that throws an Unavailable RpcException → assert `GalaxyDriver.WriteAsync` reports the failure to a fake supervisor and returns Bad; non-transport ArgumentException → not reported. Idempotency: two failures → supervisor sees N reports but single-flights (already covered by `ReconnectSupervisorTests` — add a driver-level report test).

**Effort:** M. **Risk:** Low-Medium — must avoid escalating *application-level* write rejections (bad value, secured-tag denial) to transport recovery; the classifier must be conservative (only RpcException transport codes).

---

### P1-B — S-3: `ReadViaSubscribeOnceAsync` can wait forever on a non-cancellable token

**Restatement:** BadTimeout fill is only via `cancellationToken.Register`; a `CancellationToken.None` caller whose tag subscribes OK but never publishes hangs at the await, holding the subscription open.

**Verification:** Confirmed. `GalaxyDriver.cs:738-748` registers the fill on the CT; 750-755 awaits each pending TCS unconditionally. No internal deadline.

**Root cause:** The read deadline is delegated entirely to the caller's token; an infinite/None token has no fallback.

**Design.** Race each pending await against an internal deadline derived from options regardless of the caller's token. Compute `deadline = max(PublishingIntervalMs, DefaultCallTimeoutSeconds)` × a small factor (e.g. 3× publishing interval, floored at the call timeout). Implement by combining tokens: create a linked CTS `CreateLinkedTokenSource(cancellationToken)` + `CancelAfter(deadline)`, and register the BadTimeout fill on the *linked* token instead of the caller's. Then every path (caller cancel, internal deadline) fires the fill. Ensure the linked CTS is disposed in the finally.

**Implementation steps:**
- In `ReadViaSubscribeOnceAsync`: build `using var deadlineCts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); deadlineCts.CancelAfter(TimeSpan.FromMilliseconds(readDeadlineMs));` and change `cancellationToken.Register(...)` → `deadlineCts.Token.Register(...)`.
- Derive `readDeadlineMs` from `_options.MxAccess.PublishingIntervalMs` and `_options.Gateway.DefaultCallTimeoutSeconds` (or equivalent). Add an option `ReadSynthesisTimeoutMs` if a dedicated knob is warranted; default derived.

**Tests (unit):** the driver has injectable seams (`IGalaxySubscriber` fake). Fake that returns a successful SubscribeBulk but never raises OnDataChange → call `ReadViaSubscribeOnceAsync` with `CancellationToken.None` → assert it returns BadTimeout snapshots within the deadline (not hang) and that UnsubscribeBulk was called (subscription not leaked).

**Effort:** S. **Risk:** Low — pure addition of an internal deadline; existing caller-cancel behavior preserved via the linked token.

---

### P1-C — S-5: FasterLog outbox `RemoveAsync` truncates the FIFO prefix; out-of-order acks silently drop unacked entries

**Restatement:** `RemoveAsync(id)` removes the target plus every older live entry and truncates to the target's successor; if a mid-batch id is acked first, older unacked values are durably discarded without incrementing `DroppedCount`.

**Verification:** Confirmed. `FasterLogHistorizationOutbox.cs:158-182`: while-loop removes `_live.First` repeatedly until it hits the target and breaks; `_droppedCount` not touched for the collateral removals. Contract "acks in FIFO order" enforced by comment only.

**Root cause:** FasterLog is a truncate-to-address log; removing an arbitrary id necessarily discards everything before it. The API accepted a single id while the underlying primitive is prefix-truncation, so the invariant lives only in the caller's discipline.

**Design.** Two options:
1. **Count + warn (minimal):** in `RemoveAsync`, when the removal drops entries whose id != the target, increment `_droppedCount` per collateral entry and log a warning naming the count. Makes the silent loss observable. Cheap, no API change.
2. **Make semantics explicit at the call site (preferred):** add `RemoveThroughAsync(Guid throughId)` (or `RemoveBatchAsync(IReadOnlyList<Guid>)`) to `IHistorizationOutbox`, documenting that it truncates the FIFO prefix through the id; keep single-id `RemoveAsync` but have it assert the target is `_live.First` (FIFO) and throw/log if not. This surfaces the invariant at compile/call time.

Recommendation: **do both** — rename the operation to `RemoveThroughAsync` to match reality, and add the collateral-drop counter as a safety net for any future non-FIFO caller.

**Implementation steps:**
- `IHistorizationOutbox`: add `RemoveThroughAsync` (or annotate `RemoveAsync` contract). Update `ContinuousHistorizationRecorder` drain call site to use it.
- `FasterLogHistorizationOutbox.RemoveAsync`: count collateral removals into `_droppedCount` (or a new `_ackPrefixDropped` counter for clarity) and `_logger.LogWarning` when > 0.

**Tests (unit):** append A,B,C; `RemoveAsync(C.Id)` (skipping A,B) → assert A,B are gone, `DroppedCount == 2` (or the new counter), and a warning was logged. Regression: FIFO ack A,B,C in order → `DroppedCount == 0`.

**Effort:** S. **Risk:** Low — additive counter; API rename is mechanical (single recorder caller).

---

### P1-D — S-7: Alarm feed reconnect has no backoff

**Restatement:** `GatewayGalaxyAlarmFeed.RunAsync` re-opens on a fixed 5 s delay forever; `DeployWatcher` and `ReconnectSupervisor` both use capped exponential + jitter.

**Verification:** Confirmed. `GatewayGalaxyAlarmFeed.cs:141` `await Task.Delay(_reconnectDelay, ct)` with a fixed `_reconnectDelay`.

**Root cause:** Inconsistency — the alarm feed predates or diverged from the shared backoff pattern.

**Design.** Replace the fixed delay with capped exponential backoff + jitter, resetting to the floor on a successful stream open (i.e., reset the attempt counter after the first message or after the `snapshot_complete`). Reuse the existing backoff helper `DeployWatcher`/`ReconnectSupervisor` use — grep for a shared `ExponentialBackoff`/`Jitter` type; if none is shared, extract one into the Galaxy `Runtime/` folder and use it in all three (aligns with OVERALL theme #2 "fixes flow to shared templates").

**Implementation steps:**
- Introduce/reuse a `Backoff` primitive (min, max, jitter). Wire into `GatewayGalaxyAlarmFeed` ctor (min/max from `GalaxyReconnectOptions`, defaults 1 s→30 s). Reset on successful open.

**Tests (unit):** the feed uses an injected `_streamFactory` and a settable delay; assert successive faults produce increasing delays capped at max and reset after a successful open (drive via a fake factory + a virtual clock or by asserting the computed delay sequence).

**Effort:** S. **Risk:** Low.

---

### P2-A — C-5: delete retired Wonderware project directories

**Verification:** Confirmed — `Historian.Wonderware`, `.Client`, `.Client.Contracts` dirs present; `grep -c Wonderware slnx == 0`.

**Design/steps:** `git rm -r` the three `src/Drivers/*Wonderware*` dirs and their `tests/` counterparts; confirm no `ProjectReference`/`using` survives (grep). This overlaps OVERALL action #11 and Report 07 hygiene — coordinate so it's deleted once. History preserves the code.

**Effort:** S. **Risk:** Very low (0 references). Verify with a full `dotnet build ZB.MOM.WW.OtOpcUa.slnx` after removal.

---

### P2-B — C-6: EventPump silently drops unknown event families with no metric

**Verification:** Confirmed. `EventPump.cs:199-206` `default: return;` — no counter.

**Design/steps:** add a `galaxy.events.filtered` (or `.unknown_family`) counter incremented in the `default` branch of `Dispatch`, tagged with the family value. Mirrors the alarm feed's decode-drop metering. Keep the filtering behavior — only add observability, so a future gateway family (`OnBufferedDataChange`) surfaces instead of vanishing.

**Tests (unit):** feed the pump an `MxEvent` with a non-`OnDataChange` family via the injected subscriber fake → assert the filtered counter incremented (MeterListener) and no `OnDataChange` fired.

**Effort:** S. **Risk:** Very low.

---

### P2-C — U-3: WriteSecured / VerifiedWrite user identity stubbed at zero

**Verification:** Confirmed. `GatewayGalaxyDataWriter.cs:263-264` hardcodes `CurrentUserId=0, VerifierUserId=0`.

**Root cause:** No mapping from the OPC UA session's LDAP-authenticated principal to an ArchestrA user id; the secured-write path was stubbed to unblock the common (non-secured, supervisory) path.

**Design.** Two-part, and honest about the gap:
1. **Fail-fast interim (this PR):** when a tag classifies `SecuredWrite`/`VerifiedWrite` and no ArchestrA user-id mapping is configured, return a clear Bad status (`BadUserAccessDenied` / `BadNotSupported`) and log, rather than silently sending user 0 (which a real galaxy will reject or mis-attribute). Prevents a confusing "commit succeeded as user 0" or opaque rejection. Also surfaces to the operator that secured-classification tags are unsupported today.
2. **Real fix (cross-cutting, larger):** thread the OPC UA session identity down to the writer. The LDAP principal is available server-side (see `RoleCarryingUserIdentity` used by alarm ack). Add an ArchestrA-user-id mapping (config `GroupToArchestraUserId` or a resolver) and pass `CurrentUserId`/`VerifierUserId` into `InvokeWriteSecuredAsync`. This spans the node-write router → driver `WriteRequest` (needs a user-id/identity field on the request contract) → writer. Coordinate with Report 03 (session identity) and the node-write router. VerifiedWrite additionally needs a *second* verifier identity (dual-authorization) — that is a genuine feature, not a plumbing fix; scope it separately.

**Implementation steps (interim):**
- `WriteOneAsync`: when `NeedsSecuredWrite(classification)` and no user-id source is wired → return `BadNotSupported`/`BadUserAccessDenied`, log, do not send. Document in `docs/` (Galaxy driver / Historian split) that secured/verified writes require the identity-mapping feature.

**Tests (unit):** classification resolver returns SecuredWrite + no identity → assert Bad status and no `InvokeAsync` call.

**Effort:** S (interim), L (real identity threading — cross-report). **Risk:** interim is Low and strictly safer than sending user 0.

---

### P2-D — U-4 / P-5: gateway-gap client-side workarounds; clamp unbounded events read

**Verification:** Confirmed.
- `HistorianGatewayClientAdapter.cs:88` calls `_inner.ReadEventsAsync(startUtc, endUtc, filter, ct)` — `maxEvents` not on the wire (client-side cap only).
- `GatewayHistorianDataSource.cs:142-168` — `maxEvents<=0` collects unbounded (only the gateway's `RuntimeDb:EventReadMaxRows` bounds it).
- Client-side source filter re-applied defensively at 152-157.
- `GalaxyDriver.cs:314-316` replay still per-subscription (batched `ReplaySubscriptionsCommand` note).

**Design/steps:**
1. **In-repo:** clamp `maxEvents<=0` to a server-side default cap (new `ServerHistorianOptions.DefaultMaxEventsPerRead`, default e.g. 10 000) in `GatewayHistorianDataSource.ReadEventsAsync` so a 0 from the SDK can't drain unboundedly. This is the P-5 fix and is fully in-repo.
2. **Cross-repo (sidecar backlog):** file tracked items for (a) a wire-level `maxEvents` on `ReadEventsRequest`, (b) a server-side source filter so the defensive client filter can be removed, (c) a batched `SendEvents` (P-2), and (d) a batched `ReplaySubscriptionsCommand` (mxaccessgw). Link each from the corresponding in-source comment. These are *not* in-repo fixes — they belong on HistorianGateway / mxaccessgw backlogs; the driver's workarounds stay until they land.

**Tests (unit):** `FakeHistorianGatewayClient` streaming > cap events with `maxEvents=0` → assert the data source stops at `DefaultMaxEventsPerRead` and flags truncation.

**Effort:** S (clamp). **Risk:** Low. **Note:** OVERALL theme #4 — track the gateway gaps on the sister repos so they don't rot.

---

### P3-A — S-2: EventPump saturation drops the newest events (inverted staleness bias)

**Verification:** Confirmed. Newest-dropped by construction (`TryWrite` on a full bounded channel). Metered (`galaxy.events.dropped`).

**Root cause:** Bounded channel with `FullMode.Wait` + `TryWrite` preserves the oldest queued events and drops the just-arrived one — wrong for last-value-wins telemetry.

**Design.** Two candidates:
1. **Per-item-handle conflation (best correctness):** keep newest-per-handle. Replace the raw `Channel<MxEvent>` with a small conflating structure: a `ConcurrentDictionary<long,MxEvent>` (handle→latest) + a signal channel; the dispatch loop drains the dict. On overflow the newest value for a handle overwrites the older one — no data staleness, bounded by tag count not event rate. More code; changes the pump's core.
2. **Drop-oldest ring (cheap):** switch the channel to `BoundedChannelFullMode.DropOldest`. But the in-source comment explicitly rejected `DropWrite`/`DropOldest` because they discard *inside* Channel<T> without surfacing on the counter. To keep metering, wrap: on a full channel, `TryRead` one (count it as dropped-oldest) then `TryWrite` the new one. Simpler than conflation, keeps recent data, preserves the counter.

Recommendation: **option 2** (drop-oldest-with-metering) as the low-risk win; option 1 only if soak testing shows a hot-handle storm still loses fresh values. Given P-4 notes the fan-out is already O(1) and the channel default is 50 000, saturation is an edge case — option 2 is proportionate.

**Implementation steps:**
- In `RunAsync`, replace the `if (!TryWrite) count` block with: `while (!TryWrite(ev)) { if (TryRead(out _)) EventsDroppedOldest.Add(1,tag); }` (bounded loop). Add a `galaxy.events.dropped_oldest` counter (or reuse `dropped`). Update the FullMode comment to match.

**Tests (unit):** fill the channel (tiny capacity via ctor), push N more → assert the newest are retained (read the dispatched values) and the dropped counter reflects the discarded oldest.

**Effort:** S-M. **Risk:** Medium — touches the hot pump loop; guard with the existing pump bounded-channel tests + a soak run. Only pursue if S-2 is judged worth the churn (report rates it Medium; it's already metered, so it's a correctness-under-stress refinement).

---

### P3-B — S-8: `GalaxyMxSession` state unsynchronized across reopen (document the invariant)

**Verification:** Confirmed. Plain `_session`/`_connected` fields; `RecreateAsync` tears down without a lock. Report rates it acceptable (emergent safety via single-flight supervisor + per-call try/catch).

**Design/steps:** lowest-effort resolution the report endorses — **document the invariant** on the `Session`/`IsConnected` members: "single-flight recovery (ReconnectSupervisor) is the only writer of session state; concurrent readers tolerate a torn read because every per-call failure maps to BadCommunicationError and caches are invalidated post-reopen." If a defensive change is wanted, mark `_connected` `volatile` and read `_session` into a local once per operation (already the pattern in `WriteAsync`). Do **not** add a lock (would serialize the hot path for no proven benefit).

**Effort:** XS (doc) / S (volatile). **Risk:** None (doc) / Low.

---

### P3-C — U-5: outbox serializer has no version/format byte

**Verification:** Confirmed. `HistorizationOutboxEntrySerializer.cs:14-16` fixed positional layout, no prefix.

**Design/steps:** prepend a single version byte (`0x01`) in `Serialize`; in `Deserialize` read it first and branch (today: only v1). One reserved byte now makes any future field addition a clean migration instead of an undiagnosable recovery failure mid-`RecoverState`. **Migration caveat:** on-disk entries written before this change have no version byte — deploying this reader against a non-empty pre-existing outbox would misread. Since the outbox is drained-and-truncated continuously and is dev-only so far, gate the change on an empty-outbox deploy, or make the reader tolerant (peek: if the first byte isn't a known version AND the buffer length matches the legacy layout, treat as v0). Prefer the tolerant reader for safety.

**Tests (unit):** round-trip v1; legacy-layout bytes (no version) still deserialize under the tolerant reader; unknown version → clear exception.

**Effort:** S. **Risk:** Low-Medium (the on-disk migration is the only sharp edge — the tolerant reader defuses it).

---

### P3-D — C-3, C-4: convention notes (documentation only)

- **C-3:** add a one-line guard note to `IHistorianGatewayClient`'s XML doc: "any consumer beyond `GatewayHistorianDataSource` must sit above the `Mapping/` anti-corruption boundary, never on this proto-typed seam." No code change. **Effort:** XS.
- **C-4:** pick one options-validation idiom for *new* driver-adjacent sections — the report (and I) recommend the Historian `Validate()`-returns-warnings style as the friendlier operational contract. Document the choice in CLAUDE.md / a conventions doc; do not churn the existing Galaxy throw-on-construct code. **Effort:** XS (doc).

---

### Accepted / no-action

- **S-9, S-10, P-4, C-1, C-2** — positives; preserve as-is (don't regress the store-and-forward discipline, the O(1) fan-out indexes, or the secret resolver).
- **P-1** — MxAccess-forced 3-round-trip read; action is documentation ("subscribe, don't poll") + optional read-through cache only if polling clients appear. Fold the doc line into `docs/` Galaxy section. **Effort:** XS.
- **P-2, P-3, P-6** — accepted, gateway-constrained or negligible; P-2 tracked on the sidecar backlog (see U-4).

---

## Suggested execution order (batched)

1. **Batch 1 (P0, failure-visibility):** S-1 (in-repo layers) + S-6 (refresh wiring) + U-1 (doc fix + convergence test). Ship together; all three are OVERALL theme #1/#3. Live-verify S-1 write and U-1 value-capture on the real gateways.
2. **Batch 2 (P1, fault resilience):** S-4 + S-3 + S-5 + S-7. Cohesive "recovery hardening" set, all unit-testable with existing fakes.
3. **Batch 3 (P2, hygiene/interim):** C-5 (delete dirs) + C-6 (metric) + U-3 (fail-fast) + U-4/P-5 (clamp) + C-3/C-4 (doc). Coordinate C-5 with Reports 03/07.
4. **Batch 4 (P3, refinements):** S-2 (drop-oldest) + S-8 (doc/volatile) + U-5 (version byte) + P-1 doc. Optional / as-capacity.

Cross-repo follow-ons to file (not in-repo): mxaccessgw `WriteComplete` correlation (S-1) and batched `ReplaySubscriptionsCommand` (U-4); HistorianGateway wire-level `maxEvents`, server-side source filter, batched `SendEvents` (U-4/P-2). Link each from the existing in-source comment.