lmxopcua/archreview/06-gateway-integrations.md

# Architecture Review 06 — Gateway Integrations (Galaxy Driver + Historian Gateway Driver)

- **Date:** 2026-08-19
- **Commit:** `12b0035a` (master, clean tree)
- **Updates:** this is the **second update** — round 1 ran 2026-07-08 at `9cad9ed0`, round 2 (re-review) 2026-07-12 at `f6eaa267`. Since `f6eaa267`, **25 commits landed in this domain's source** (plus the Runtime-side historian recorder work), including: the R2-04/06/S-1 Galaxy write-confirmation arc (`a537d29f`, `61030673`, `084c6e3c`, `20a816db`, `205cc4a4`); the continuous-historization gate arc #491/#537/#538 (`ad744734`, `e7fbe977`, `a76b4e07`, `384568b4`, `6ff64ca0`, `73205865`, `451273c8`, `7870c91f`); Boolean→Int2 provisioning #441 (`9f62310b`) + StorageRateMs (`4da2d62b`); v3 RawPath re-keying (`636c755b`); replay-gap + sparse-array handling (`4769044f`); BadTypeMismatch for unrepresentable values (`c51c5998`); numeric alarm severity + provisioner channel dispose (`55b12dc2`); skipped-tags reporting #541 (`39db7508`, `dd900f97`); status-constant parity #497 (`b95b99ba`); `GetMemoryFootprint`/`FlushOptionalCachesAsync` deletion #525 (`b8154469`); secrets `secret:` arm (`1424a214`, `772d3a5f`).
- **Scope:**
  - `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy` (+ `.Contracts`, `.Browser`)
  - `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway`
  - Test coverage: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests`, `.Galaxy.Browser.Tests`, `.Historian.Gateway.Tests`
  - Peripheral (read for context): `Runtime/Historian/*` (`ContinuousHistorizationRecorder`, options, `ServerHistorianOptions`), Host `Program.cs` + `Host/Configuration/ServerHistorianOptionsValidator.cs`, `Host/Drivers/DriverFactoryBootstrap.cs`.
- **Dimensions:** Stability, Performance, Conventions, Underdeveloped Areas.

---

## Prior-finding status (f6eaa267 → 12b0035a)

Every verdict below was re-derived from current code, not from remediation claims.

| ID | One-liner | Status @ 12b0035a |
|---|---|---|
| S-1 (High) | Galaxy write success optimistic; committed-write failure could never surface | **CLOSED** (residual metered, documented). Three legs all verified in code: (1) **fail-closed advise** — a failed `AdviseSupervisory` short-circuits to Bad *without* issuing the raw Write (`GatewayGalaxyDataWriter.cs:174-178`, meter `galaxy.writes.advise_failed` `:48-50`), feeding the #5 node revert; (2) **correlated write statuses** — `TranslateReply` consumes the gateway's `OnWriteComplete` outcome rolled into `statuses[0]` (`:308-332`); an EMPTY statuses array is provisional Good and increments `galaxy.writes.unconfirmed` (`:330-331`); (3) **no truncation on the mapping seam** — `StatusCodeMap.FromMxStatus` routes MxStatus detail codes > 0xFF per-code instead of truncating into the quality-byte table (the 1007→239→Good defect: `StatusCodeMap.cs:120-133`), with an explicit invariant that a failure row can never surface Good (`:129-131`) and a per-code `MxStatusDetail` table (1004-1017, `:152-167`). The borrow path is **deleted by design** — the writer never borrows a subscription's plain-advised handle (live-captured rationale in the class remarks, `:21-27`). Pinned offline via the `IMxWriteOps` seam (`GatewayGalaxyDataWriterFailClosedTests`). Residual: empty-statuses ⇒ provisional Good against a pre-correlation gateway — metered, cross-repo (mxaccessgw ≥ `53f69cd`). But see **NEW S-12**: the advise *cache* has an exception-path poisoning bug. |
| S-2 (Med) | EventPump saturation drops the *newest* events | **STILL OPEN** — `TryWrite` drop-newest unchanged (`EventPump.cs:164-170`). Metered; bias still inverted for last-value-wins telemetry. |
| S-3 (Med) | `ReadViaSubscribeOnceAsync` can wait forever on a non-cancellable token | **STILL OPEN** — Bad-fill still only via `cancellationToken.Register` (`GalaxyDriver.cs:899-909`); the awaits at `:914` hang on `CancellationToken.None` + a never-publishing subscribed tag. No internal deadline added; no test covers the case (`GalaxyDriverReadTests` always emits or cancels). |
| S-4 (Med) | Transport-failure detection is EventPump-only | **OPEN, NARROWED** — `ReportTransportFailure` still has exactly one caller (`GalaxyDriver.cs:1196`, the pump fault callback). Improvement: the pump now treats a **server-completed** stream as a fault too (`EventPump.cs:180-188` — closes the "clean termination evades watchdogs" shape for the streaming path). Residual: a subscription-less driver (writes only, or idle) never starts the pump, so a gateway restart still leaves `GetHealth()==Healthy` with per-write `BadCommunicationError` and no reopen — and the #527 comms-loss watchdog needs a non-Healthy state to trip, so it doesn't cover this either. |
| S-5 (Med) | Outbox `RemoveAsync` truncates the FIFO prefix; out-of-order acks silently drop | **OPEN → downgraded Low / accepted-by-design.** The truncate-through semantics are unchanged (`FasterLogHistorizationOutbox.cs:203-227`) but are now (a) **pinned by test** (`FasterLogHistorizationOutboxTests.cs:99` — "acks A and B, leaves C"), (b) **explicitly relied on** by the sole caller (`ContinuousHistorizationRecorder.cs:677-680` documents why the ack stays all-or-nothing), and (c) documented in CLAUDE.md (#538 narrative). The prefix drop still doesn't increment `DroppedCount` — residual observability nit only. |
| S-6 (Med) | Historian health snapshot's connection flags are dormant | **STILL OPEN, and SHARPENED** — `RefreshConnectionStateAsync` (`GatewayHistorianDataSource.cs:219-244`) still has only test callers (`GatewayHealthSnapshotTests.cs:33,59,74`). Worse: grep shows **`GetHealthSnapshot()` itself has no production consumer anywhere** — the whole `HistorianHealthSnapshot` surface (counters + flags) is currently a write-only ledger. Wire a reader (the R2-10 resilience-chip pattern is the template) or delete the flags. |
| S-7 (Low) | Alarm feed reconnect has no backoff | **STILL OPEN** — fixed 5 s (`GatewayGalaxyAlarmFeed.cs:46,141`); `DeployWatcher` and `ReconnectSupervisor` still do capped exponential. |
| S-8 (Low) | `GalaxyMxSession` state unsynchronized across reopen | **STILL OPEN / accepted** — plain fields (`GalaxyMxSession.cs:29-31`); safety remains emergent (single-flight supervisor + per-call catch). The `AttachForTest` path at least documents the `_connected` invariant now (`:129-131`). |
| S-9 / S-10 (Positive) | S&F/outbox crash-safety; TLS/API-key posture | **Stand, strengthened** — batch append is incremental-indexed so a mid-loop fault leaves no untracked prefix (`FasterLogHistorizationOutbox.cs:119-137`); provisioner now disposes its channel (`GatewayTagProvisioner.cs:108-115`); `ApiKey` supports `${secret:}` (`ServerHistorianOptions.cs:43-46`). |
| S-11 (Med) | Misconfigured `ServerHistorian` crash-loops Host with raw `UriFormatException` | **CLOSED** — verified all three layers: fail-fast `ServerHistorianOptionsValidator` wired via `AddValidatedOptions` (`Host/Program.cs:188-196`), consumer-gated so the alarm-only corner is covered, with the alarm-only startup warning (`Program.cs:205-212`); adapter defense-in-depth `Uri.TryCreate` → named `InvalidOperationException` (`HistorianGatewayClientAdapter.cs:53-58`); warning parity for malformed endpoints in `Validate()` (`ServerHistorianOptions.cs:117-123`). |
| P-1 (Med) | Galaxy Read = 3 round-trips + publish wait | **STILL OPEN** — gateway-constrained, unchanged (`GalaxyDriver.cs:805-941`). Accepted; "subscribe, don't poll" remains the pattern. |
| P-2 (Med) | Alarm drain is one unary `SendEvent` per event | **STILL OPEN** — serial per-event loop unchanged (`GatewayAlarmHistorianWriter.cs:66-78`); still no batched `SendEvents` on the client surface. Deliberate poison isolation; accepted ceiling. |
| P-3 (Low) | Four channels to the historian sidecar (deliberate) | **Unchanged** — and the dispose-ownership rationale is now stronger (each path, provisioner included, owns and disposes its channel). |
| P-4 (Positive) | Fan-in/fan-out indexed, bounded, metered | **Stands** — plus new counters (`galaxy.events.replay_gaps`, `galaxy.events.undecodable`, `EventPump.cs:50-55`). |
| P-5 (Low) | HistoryRead buffers whole replies; `maxEvents <= 0` unbounded | **OPEN, NARROWED** — events reads now stop at the cap client-side with a truncation continuation token (`GatewayHistorianDataSource.cs:142-175`); only `maxEvents <= 0` still collects unbounded (`:142-143`), and the adapter still cannot forward the cap on the wire (`HistorianGatewayClientAdapter.cs:100-113` — client signature unchanged). The 03/S3 knobs (`HistoryReadBatchParallelism`/`MaxConcurrentHistoryReadBatches`/`HistoryReadDeadline`, `ServerHistorianOptions.cs:80-97`) bound the server-side fan-out/flood — a real mitigation for the same blast radius. |
| P-6 (Low) | Outbox peek holds the state lock across disk I/O | **STILL OPEN** — `PeekBatchAsync` scans under `_state` (`FasterLogHistorizationOutbox.cs:190-197`); note the exposure grew 16× with `DrainBatchSize` 64→1024 (`ContinuousHistorizationOptions.cs:78`). Still an in-memory-page scan; still acceptable; revisit if drain batches grow again. |
| C-1 / C-2 (Positive) | Secret handling; seam quality | **Stand, strengthened** — `GalaxySecretRef` gained the fail-closed `secret:` arm through the shared `ISecretResolver` (`Galaxy.Contracts/GalaxySecretRef.cs:19-21,93-98`); production wires the real resolver (`DriverFactoryBootstrap.cs:175`) while the `NullSecretResolver` default is confined to the test/standalone convenience overload (`GalaxyDriverFactoryExtensions.cs:45-54` — the ForTest-seam trap checked and clear). New `IMxWriteOps` internal seam (`GatewayGalaxyDataWriter.cs:342-405`) made the sealed-SDK write pipeline unit-testable without widening the public contract. |
| C-3 (Med) | Historian seam leaks wire types by design | **STILL OPEN (Low now)** — `IHistorianGatewayClient.cs:5-11` unchanged, no guard note added. Mitigation exists in practice: the U-7 presence-pinning tests + the 6/6 live gate are the bump-gate the finding asked for. Keep any future consumer above `GatewayHistorianDataSource`. |
| C-4 (Low) | Options-validation styles diverge; warnings-only style crashes | **CLOSED** — the asked-for fail-fast tier landed (S-11's `ServerHistorianOptionsValidator` + `Validate()` parity). Two idioms still coexist (Galaxy throw-on-construct vs. historian warnings+Host validator) but the sharp edge — correct warning followed by opaque crash — is gone. |
| C-5 (Low) | Retired Wonderware project dirs on disk | **CLOSED** — glob `src/Drivers/*Wonderware*` matches nothing (R2-12 established they were untracked husks; they are gone from this checkout). |
| C-6 (Low) | EventPump silently drops unknown event families with no metric | **STILL OPEN (narrowed)** — the default family arm still returns uncounted (`EventPump.cs:297-312`); however undecodable values and replay gaps now have their own counters, and filtered volume is derivable as `received − dispatched − dropped − gaps`. A `galaxy.events.filtered` counter remains the clean fix. |
| C-7 (Low) | Live-value writer populates a documented-dead `Quality` field | **CLOSED** — keep-populating decision verified with the full rationale in the class remarks (`GatewayHistorianValueWriter.cs:27-37`: proto3 unset transmits 0 = Bad, so populating the correct 192 is forward-safe) + inline note at the mapping (`:74-76`) citing 0.2.0 Contracts C-002, + the explicit "do NOT extend the recorder to capture real node quality" warning. |
| U-1 (High, doc-drift) | CLAUDE.md KNOWN LIMITATION 2 stale | **CLOSED** (round 2) — and now doubly moot: the live gate ran and #491 is closed; CLAUDE.md reflects it. |
| U-2 (Med) | Live-validation gate built but not fully run | **CLOSED** — two full documented live runs since: (1) R2-06 T12, 2026-07-13, against a real 0.2.0 gateway + real AVEVA Historian over VPN — **6/6** after finding + fixing the StorageRateMs provisioning bug (`4da2d62b`) and driving the Boolean→Int2 remap (STATUS.md §"R2-06 live gate"); (2) the continuous-historization gate 2026-08-01→04 (#491) — outbox/drain/WriteLiveValues/EnsureTags/read-back all green with distinct non-zero constants (`docs/plans/2026-08-01-continuous-historization-live-gate.md`). The retype-policy question is **answered** (upsert; supported-type retype accepted). |
| U-3 (Med) | WriteSecured / VerifiedWrite user identity stubbed at zero | **STILL OPEN** — `CurrentUserId = 0, VerifierUserId = 0` (`GatewayGalaxyDataWriter.cs:284-285`). Unchanged since round 1: writes to secured/verified-classification tags remain effectively unsupported, with no session-identity→ArchestrA-user mapping and no fail-fast. The correlated statuses (S-1) at least make a rejection *visible* now (Bad instead of phantom Good). |
| U-4 (Med) | Dormant-paths inventory | **PARTIALLY CLOSED** — `GetMemoryFootprint`/`FlushOptionalCachesAsync` **deleted** fleet-wide (#525, `b8154469` — grep-zero in `GalaxyDriver.cs`). Still open: per-subscription replay vs. the gateway's batched `ReplaySubscriptionsCommand` (note survives at `GalaxyDriver.cs:446-447`); `maxEvents` never forwarded on the wire + defensive client-side source filter (`GatewayHistorianDataSource.cs:152-157`) — both still gateway gaps to track against the sidecar repo; `RefreshConnectionStateAsync` (S-6); `IHostConnectivityProbe.GetHostStatuses` still unconsumed (Gitea #521, `GalaxyDriver.cs:724`). |
| U-5 (Low) | Outbox serializer has no version/format byte | **STILL OPEN** — fixed positional layout, no version prefix (`HistorizationOutboxEntrySerializer.cs:14-16,23-40`). The window to add one byte cheaply is closing as deployments accumulate on-disk logs. |
| U-6 | Test-coverage thin spots | **LARGELY CLOSED** — +2,304 test lines across 30 files this period. The advise-failure status path is now covered offline (`GatewayGalaxyDataWriterFailClosedTests` via `IMxWriteOps`); outbox prefix-removal semantics pinned (`FasterLogHistorizationOutboxTests.cs:55-99,288`); pump fault/replay-gap/bounded-channel suites exist; recorder gates all pinned (capture-gate re-read, quarantine falsification control, batch-append prefix accounting). **Remaining thin spots:** the S-3 read-hang case; a C-6 unknown-family filter assertion. |
| U-7 (Med) | 0.2.0 `DataType` presence change silently altered Boolean provisioning | **CLOSED** — and resolved *differently than the round-2 remediation pinned*: the live gate exposed Int1 as server-degenerate, so `Boolean → Int2` (`HistorianTypeMapper.cs:31`, issue #441), retyping cleanly from pre-existing Float tags. `StorageRateMs` is stamped (`GatewayTagProvisioner.cs:72` — its absence had silently failed **every** provision, found live). Live gate 6/6; retype policy answered (accepted-retyped). |

**New findings this pass:** S-12 (Medium), S-13 (Low), C-8 (Low) — see body.

---

## What changed since 2026-07-12 (delta assessment)

1. **The Galaxy write path went from optimistic to correlated** (06/S-1 arc, closed). The shape that shipped is stronger than the round-2 partial: the gateway now correlates the worker's `OnWriteComplete` into every unary write reply, so the driver's reply translation carries the real COM-side commit verdict, including refusals — with an explicit per-code `MxStatusDetail` table and an anti-truncation invariant on the mapping seam (the live gate caught `1007 → low-byte 239 → Good`, i.e. a REFUSED write mapping to Good; `StatusCodeMap.cs:120-133` is the fix). The subscription-handle borrow path was deleted outright after live captures showed a plain-advised handle can never carry a no-login write (SECURITY_ERROR 1008) and cannot be re-advised (E_INVALIDARG).
2. **The continuous-historization pipeline was live-gated and hardened end-to-end** (#491/#537/#538, all verified in code): `Self` captured before the `ConfigureAwait(false)` await (`ContinuousHistorizationRecorder.cs:369-382`); capture gated on the per-cluster Primary via `IHistorizationRoleView` (registered unconditionally on driver nodes, `Runtime/ServiceCollectionExtensions.cs:99` — the #511 registration accident avoided); poison-tag quarantine keyed on "did another tag write this pass" with the falsification control (`:767-806`); shutdown cancellation mid-drain rethrown so it aborts the *pass* instead of counting as a per-tag failure (`:707-716`, `7870c91f`). Perf landed with correctness intact: batch ingest with one group commit per burst (`FasterLogHistorizationOutbox.AppendBatchAsync`, incremental bookkeeping `:119-137`), bounded-parallel per-tag drain (`DrainConcurrency` 8, fold on the single drain flow `:741-758`), `DrainBatchSize` 64→1024.
3. **v3 re-keyed the Galaxy driver to RawPath identity** (`636c755b`): forward/reverse RawPath⇄attributeRef tables built once from the artifact (`GalaxyDriver.cs:247-277`), identity on a miss, with the skipped-tag ledger (#541) accounting for undialable entries. Alarm ConditionId, ack routing, discovery and write security resolution all translate consistently (`:964-972`, `:1298-1310`, `:1354`).
4. **Wire robustness grew**: the replay-gap sentinel is handled ahead of channel pressure with a coalesced, non-blocking re-snapshot (`EventPump.cs:153-162`, `GalaxyDriver.cs:1125-1173` — deliberate contract defence for a path the current configuration cannot hit); sparse arrays decode with gateway-mirrored fill semantics (`MxValueDecoder.cs:96-166`); an unrepresentable value now publishes `BadTypeMismatch` instead of a Good null (`EventPump.cs:336-353` — the "good null is indistinguishable from a real null" fix).

---

## Maturity Ratings

| Dimension | Rating (1-5) | Prior | Justification |
|---|---|---|---|
| Stability | **4.5** | (4) | The two headline stability debts both closed and were *live-gated*: Galaxy writes are fail-closed + correlated (S-1), and the historization pipeline survived a real gate that found and fixed three production-fatal defects (#491/#537/#538). The pump now detects server-completed streams. Docked half a point for the surviving cluster: the S-3 read hang, EventPump-only fault detection for subscription-less drivers (S-4), the dormant health snapshot (S-6), and the new advise-cache poisoning edge (S-12). |
| Performance | **4.5** | (4) | Group-commit batch append (one fsync per burst), bounded-parallel per-tag drain, a 16×-wider drain batch, and the 03/S3 HistoryRead parallelism/limiter/deadline knobs — all with the correctness discipline intact (all-or-nothing ack, outage-vs-poison discrimination). Remaining docks are gateway-constrained and documented: the 3-round-trip read synthesis (P-1) and serial alarm sends (P-2). |
| Conventions | **4.5** | (4) | C-4, C-5, C-7 closed; the `IMxWriteOps` seam and the `secret:` arm follow the house patterns exactly; status-code constants are named + reflection-guarded (#497); in-source rationale quality (the anti-truncation invariant, the keep-populating Quality decision, the replay-gap unreachability note) is exemplary. Docked for the surviving Low trio: seam wire-type leakage note (C-3), the unmetered pump family filter (C-6), and the unlatched quarantine warning (C-8). |
| Underdeveloped areas | **4** | (3) | The live gates that defined this dimension's debt have now *run* — twice — and each found real defects offline tests missed (StorageRateMs, Int1 degeneracy, the ConfigureAwait/`Self` restart, dual-capture, poison-tag pinning); U-2/U-7 closed, dead surface pruned (#525). Remaining: WriteSecured identity still stubbed (U-3), the dormant health-snapshot reader (S-6), the outbox version byte (U-5), and the gateway-gap workarounds tracked cross-repo. |

---

## Findings

### 1. Stability

#### S-12 (Medium, NEW) — A transport failure during `AdviseSupervisory` poisons the supervised-handle cache; subsequent writes skip the advise until the next reconnect

`EnsureSupervisoryAdvisedAsync` marks the handle supervised **before** the round-trip (`_supervisedHandles.TryAdd`, `GatewayGalaxyDataWriter.cs:234`) and removes the mark only on the **non-Ok reply** path (`:251-261`). If `ops.InvokeAsync` **throws** instead — transport blip, gateway restart, cancellation — the exception propagates to `WriteOneAsync`'s catch (`:192-197`, this write correctly returns `BadCommunicationError`) but the handle **stays marked supervised**. Every subsequent write to that tag then takes the "already advised" fast path (`:234`) and issues a raw Write against an item that was never supervisory-advised:

- Against a **correlated** gateway (the production posture), each write is refused with SECURITY_ERROR 1008 → `BadUserAccessDenied` — visible, but **never self-healing**: nothing clears the stale cache entry on a 1008 row, so the tag's writes fail until a session reconnect happens to run `InvalidateHandleCaches` (`:67-71`).
- Against a **pre-correlation** gateway, the reply's statuses are empty → **provisional Good** (`:330-331`) — the exact S-1 phantom-write shape, resurrected for this one edge.

This is the classic one-fault-variant gap: the fail-closed fix was verified against the *reply-failure* variant (which the fail-closed tests cover) but not the *throw* variant. Fix: wrap the invoke in try/catch and `TryRemove` the handle on any failure to complete the advise; optionally also clear the entry when a write's correlated status comes back 1008 (self-heal). One unit test on the throw path via the existing `IMxWriteOps` fake closes it.

#### S-2 (Medium) — EventPump saturation still drops the *newest* events

Unchanged (`EventPump.cs:164-170`): `TryWrite` against the bounded channel preserves 50 000 stale queued events and discards the just-arrived one. For last-value-wins telemetry the bias is inverted. Metered (`galaxy.events.dropped`). Recommendation stands: per-item-handle conflation (keep newest per handle) or drop-oldest ring semantics.

#### S-3 (Medium) — `ReadViaSubscribeOnceAsync` can still wait forever on a non-cancellable token

Unchanged (`GalaxyDriver.cs:899-915`): the Bad-fill for still-pending snapshots runs only from `cancellationToken.Register`. A caller passing `CancellationToken.None` (or a never-firing token) with a successfully-subscribed tag that never publishes its initial event hangs the read and holds the subscription open. `StatusCodeMap.BadTimeout` is now the correct constant (#497) — but nothing arms it without a caller token. Recommendation stands: race each pending TCS against an internal deadline derived from `PublishingIntervalMs`/call-timeout regardless of the caller's token. Still untested (the read suite always emits or cancels).

#### S-4 (Medium) — Transport-failure detection is still EventPump-only; a subscription-less driver never degrades

`ReportTransportFailure` retains its single production caller (`GalaxyDriver.cs:1196`). The pump-side coverage *improved*: a server-completed stream now notifies the supervisor instead of dying silently (`EventPump.cs:180-188` — explicitly written against the #527 watchdog's non-Healthy requirement). But the pump only exists once something subscribes or reads; a write-only or idle-after-discovery driver whose gateway restarts stays `Healthy` with per-request `BadCommunicationError`, and neither the supervisor nor the #527 watchdog (which requires a non-Healthy state) engages. Recommendation stands: report classified transport exceptions from the write/subscribe unary paths into the supervisor (idempotent, cheap).

#### S-5 (Low, was Medium) — Outbox `RemoveAsync` prefix-truncation: now pinned and relied upon; the silent-drop residual is observability only

The truncate-through-the-id semantics (`FasterLogHistorizationOutbox.cs:203-227`) are now an explicit, tested contract (`FasterLogHistorizationOutboxTests.cs:99`) that the recorder's all-or-nothing ack deliberately builds on (`ContinuousHistorizationRecorder.cs:677-680`) — the round-1 worry ("a future recorder change acks mid-batch") is fenced by documentation at both ends plus the #538 design. Residual: entries dropped by a prefix removal still don't count on `DroppedCount`. One counter increment in the prefix loop would make the ledger honest; low priority.

#### S-6 (Medium) — The historian health snapshot is a write-only ledger: no refresh caller, and now provably no snapshot reader either

`RefreshConnectionStateAsync` (`GatewayHistorianDataSource.cs:219-244`) still has zero production callers (only `GatewayHealthSnapshotTests.cs:33,59,74`), so `ProcessConnectionOpen`/`EventConnectionOpen` are permanently `false` in a deployed host. Sharpened this pass: **`GetHealthSnapshot()` itself has no production consumer** — the interface member (`IHistorianDataSource.cs:117`) and the Null impl are the only other references in `src/`. Third review in a row for this built-but-never-wired instance. Either wire a periodic refresh + a reader (the R2-10 resilience-status flow is the exact template: publisher service → DPS/gRPC → AdminUI chip), or delete the snapshot surface. If kept dormant, at least a registration-time guard test (the R2-10 `ResilienceStatusReaderRegistrationTests` pattern) should pin whichever decision is made.

#### S-7 (Low) — Alarm feed reconnect still has no backoff

Fixed 5 s forever (`GatewayGalaxyAlarmFeed.cs:46,141`), still inconsistent with `DeployWatcher`/`ReconnectSupervisor`'s capped exponential. Cosmetic at fleet scale; align when next touched.

#### S-8 (Low) — `GalaxyMxSession` state unsynchronized across reopen (accepted)

`_session`/`_connected` remain plain fields (`GalaxyMxSession.cs:29-31`); the emergent safety (single-flight recovery, per-call catch → `BadCommunicationError`, cache invalidation post-reopen) held through this period's changes, and the batch-scoped `SessionMxWriteOps` binding (`GatewayGalaxyDataWriter.cs:115-122`) keeps a mid-batch reconnect contained to Bad results. Accepted; the invariant is now partially documented (`:46-48`).

#### S-13 (Low, NEW) — Two RawPaths authored to the same Galaxy attributeRef: the second node silently never receives a value

`BuildRawTagTables` is first-wins on the reverse map (`GalaxyDriver.cs:272-273` — comment-only), and `OnPumpDataChange` reverse-translates every event through `ToRawPath` (`:316-317`, `:1394`), so all data-change fan-out for a shared attributeRef surfaces under the **first** RawPath. The second raw tag dials fine (forward map, writes work) but its node never publishes — stuck on `BadWaitingForInitialData` with nothing logged and **no skip-ledger entry** (#541's exact silence, one layer up: the tag is "bound" yet dead). Nothing upstream prevents authoring the duplicate (two folders can each hold a tag dialing the same `tag_name.Attribute`). Minimum fix: log a warning and record the loser in the `SkippedTagLedger` ("duplicate attributeRef — value fan-out surfaces under {winner}"); better: fan the reverse map out to *all* RawPaths sharing the ref (the historian recorder's `_refMap` multi-name pattern, `ContinuousHistorizationRecorder.cs:540-554`, is the in-repo precedent).

#### S-9 (Positive) — Store-and-forward + outbox crash-safety discipline extended under the perf work

The batching arc kept every crash-safety property while removing the per-value fsync: incremental per-entry indexing means a mid-batch fault strands nothing untracked (`FasterLogHistorizationOutbox.cs:119-137` — the follow-up `73205865` fixed exactly the deferred-indexing variant); the commit-residual is documented at the seam (`:145-149`); drop-oldest runs once per batch (`:153-181`). The alarm writer's shutdown short-circuit and never-throw classification stand (`GatewayAlarmHistorianWriter.cs:66-78,125-156`). The recorder's ingest is hardened per-item (a faulting capture-gate delegate cannot drop the burst tail or restart the actor — `ContinuousHistorizationRecorder.cs:385-423`) and the append failure path explicitly refuses to restart-loop under a persistent disk fault (`:446-467`).

#### S-10 (Positive) — TLS / API-key posture unchanged and extended

CA pinning, the documented inverted TLS mapping and the h2c TLS-only-options guard (`HistorianGatewayClientAdapter.cs:60-79`), keys never logged, `${secret:}` support on `ServerHistorian:ApiKey`, and the Galaxy `secret:` arm resolving fail-closed through the shared store with the real resolver wired in production (`DriverFactoryBootstrap.cs:175`).

### 2. Performance

#### P-1 (Medium) — Galaxy Read still costs three gateway round-trips plus a publish wait

Unchanged and gateway-constrained (`GalaxyDriver.cs:805-941`). Correctly batched per request; "subscribe, don't poll" remains the supported pattern.

#### P-2 (Medium) — Alarm-history drain is still one unary `SendEvent` per event

Unchanged (`GatewayAlarmHistorianWriter.cs:66-78`) — deliberate poison-event isolation; still no batched `SendEvents` with per-event status rows on the client surface. Accepted, documented ceiling; adopt a batched RPC when the gateway grows one.

#### P-3 (Low) — Independent channels per path (deliberate)

Unchanged; the trade-off is argued in-source and every owner now disposes its channel (the provisioner joined this period, `GatewayTagProvisioner.cs:108-115`). No action — still noted so nobody "optimizes" it into a shared-singleton dispose bug.

#### P-4 (Positive) — Fan-out paths indexed, bounded, metered — now including the drain

`SubscriptionRegistry` reverse maps, `BuildResultIndex`, and the bounded pump channel stand. New this period: the recorder's per-tag drain groups run under a bounded `SemaphoreSlim` window with the outcome fold deliberately kept on the single logical drain flow so the #538 counters can't be corrupted by concurrent mutation (`ContinuousHistorizationRecorder.cs:696-758`); ingest is one group commit per burst; `Task.WhenAll`'s complete-before-fault semantics are used to make the `using`-scoped semaphore safe on the cancellation path (`:736-739`).

#### P-5 (Low) — Events reads: client-side cap + truncation landed; `maxEvents <= 0` still unbounded; the cap still never reaches the wire

`ReadEventsAsync` now stops draining at the cap and signals truncation with an opaque token (`GatewayHistorianDataSource.cs:142-175`). Remaining: `maxEvents <= 0` disables the cap entirely (`:142-143` — the gateway's remote `RuntimeDb:EventReadMaxRows` is then the only bound), and the adapter still cannot forward the cap on the wire (`HistorianGatewayClientAdapter.cs:100-113`). Clamp `<= 0` to a server-side default; keep tracking the wire-side cap against the sidecar repo. The 03/S3 parallelism/limiter/deadline knobs (`ServerHistorianOptions.cs:80-97`, validated at `:133-138`) meaningfully bound the surrounding fan-out.

#### P-6 (Low) — Outbox peek still holds the state lock across the log scan — and the batch is 16× wider now

`PeekBatchAsync` scans under `_state` (`FasterLogHistorizationOutbox.cs:190-197`); `DrainBatchSize` default rose 64→1024 (`ContinuousHistorizationOptions.cs:78`), so a concurrent `AppendBatchAsync`'s per-entry index updates can now stall behind a 1024-entry scan. Mostly page-cached; fine today. If drain batches grow again, snapshot `_nextScanAddress` under the lock and scan outside it.

### 3. Conventions

#### C-1 (Positive) — Secret handling remains the model

`GalaxySecretRef` now resolves `env:` / `file:` / `dev:` / **`secret:`** / literal, fail-closed on the `secret:` arm through the shared `ZB.MOM.WW.Secrets` resolver (`Galaxy.Contracts/GalaxySecretRef.cs`), shared by driver and browser. `ServerHistorian:ApiKey` documents env-or-`${secret:}` delivery (`ServerHistorianOptions.cs:40-48`). No key value logged on any path.

#### C-2 (Positive) — Seam quality improved again

The `IMxWriteOps` internal seam (`GatewayGalaxyDataWriter.cs:342-405`) is the right-sized answer to a sealed SDK: production binds a zero-translation adapter, tests drive the *real* single-write pipeline (`WriteOneForTestAsync`, `:142-144`) — precisely the design that lets the fail-closed behaviour be pinned offline. `IHistorianGatewayClient` remains the single historian seam with every consumer behind it.

#### C-3 (Low, was Medium) — The historian seam still leaks wire types by design

`IHistorianGatewayClient.cs:5-11` unchanged; no guard note added. Downgraded: the risk the round-2 bump demonstrated is now institutionally covered — presence-semantics pinned by unit tests (`GatewayTagProvisionerTests.Boolean_definition_carries_explicit_Int2_presence`) and the live suite has become a routine bump gate. Keep future consumers above `GatewayHistorianDataSource`; add the one-line guard note when next editing the file.

#### C-6 (Low) — EventPump unknown-family filter still unmetered

`Dispatch`'s default arm returns uncounted (`EventPump.cs:297-312`). Narrowed by the new `replay_gaps`/`undecodable` counters (a sentinel or bad payload is no longer invisible), and filtered volume is derivable from the counter arithmetic — but a first-class `galaxy.events.filtered` counter (mirroring the alarm feed's decode-failure counter) is still the clean close, and no test asserts the filter.

#### C-8 (Low, NEW) — The poison-tag abandonment warning repeats every drain pass with no latch

Once a tag is over its `MaxWriteAttempts` budget, each subsequent pass (2 s default cadence) that captures new entries for it re-abandons and re-logs the full Warning (`ContinuousHistorizationRecorder.cs:808-819` — `_tagWriteFailures` is never reset on abandonment, so `stillBlocking` stays false and the branch re-fires per pass): ~30 warnings/minute per poison tag until it is fixed or de-historized. The recorder already has the right idiom in-file — the ingest-fault Warning-then-Debug latch (`:404-416`). Apply it per tag.

### 4. Underdeveloped Areas

#### U-3 (Medium) — WriteSecured / VerifiedWrite user identity is still stubbed at zero

`GatewayGalaxyDataWriter.cs:284-285` unchanged across all three reviews. Any galaxy that actually classifies tags SecuredWrite/VerifiedWrite will reject or mis-attribute user 0; there is still no mapping from the OPC UA session identity (now a rich, ACL-carrying principal since #520) to an ArchestrA user id, and no fail-fast. The S-1 correlation at least makes the rejection visible as Bad now. Decide: document as unsupported + fail fast with a distinct status, or build the identity mapping. This is the domain's oldest open Medium.

#### U-4 (Medium → Low) — Dormant-paths inventory, pruned

Closed this period: `GetMemoryFootprint`/`FlushOptionalCachesAsync` deleted (#525). Remaining, all tracked or gateway-side: batched `ReplaySubscriptionsCommand` swap (`GalaxyDriver.cs:446-447`), wire-side `maxEvents` + server-side source filter (sidecar backlog), `RefreshConnectionStateAsync`/`GetHealthSnapshot` (S-6), `IHostConnectivityProbe` (Gitea #521).

#### U-5 (Low) — Outbox serializer still has no version/format byte

`HistorizationOutboxEntrySerializer.cs:14-16,23-40` unchanged. Now that `Capacity` defaults non-zero and production deployments accumulate on-disk logs, a layout change would break `RecoverState` mid-scan with an undiagnosable failure. One reserved byte remains free today; a migration later is not. Third listing.

#### U-6 — Test coverage assessment

Substantially improved: +2,304 lines / 30 files across the three gateway test projects since `f6eaa267`. Now covered: the fail-closed advise path and unconfirmed-write metering (offline via `IMxWriteOps`), RawPath keying, replay-gap handling, stream-completion fault, sparse-array/undecodable decode, outbox batch-append prefix accounting + prefix-removal pinning + serializer round-trips, value-writer quality/timestamp mapping, recorder gate/quarantine/falsification controls (Runtime.Tests). Live suites: Galaxy reopen-and-write + alarm feed + write meters (env-gated), historian `LiveIntegration` 6/6. **Remaining thin spots:** S-3's read-hang (no test passes `CancellationToken.None` with a never-publishing subscribed tag); C-6's family filter; and the new S-12 advise-throw path (one `IMxWriteOps` fake test).

---

## Cross-Cutting Themes

1. **The live gate is now this domain's most productive tool — keep running it.** Every closure this period that mattered was live-gate-driven, and each gate found defects that 100+ green unit tests coexisted with: the StorageRateMs=0 silent-total-provisioning-failure, Int1 server-degeneracy, the `ConfigureAwait(false)`/`Self` actor kill, dual-node double-capture, the 1007→Good truncation, and the plain-`Write`/borrow-path gaps in the original S-1 brief. The offline suites are strong, but the pattern is consistent: mapping seams and fire-and-forget boundaries only reveal themselves against the real peer.
2. **"Fixed for the reply-failure variant, not the throw variant" is the residual failure mode.** S-12 is exactly this: the fail-closed advise handles a non-Ok *reply* but not a *thrown* invoke, leaving a poisoned cache. When hardening any request/reply seam here, enumerate both failure shapes (and cancellation) explicitly — the house's one-fault-variant trap.
3. **Built-but-never-wired is down to one instance — but it's on its third review.** The provisioner (PR #423), the sink forwarding (F10b), and the resilience tracker (R2-10) all got readers or guards; `GetHealthSnapshot`/`RefreshConnectionStateAsync` (S-6) is the last holdout, and this pass established the *entire snapshot surface* is unread. Wire it or delete it; either way pin it with a registration guard.
4. **Silent-partial-binding is the next #541.** The skipped-tag ledger closed the "N authored, fewer bound, nothing says why" gap at the artifact boundary — but S-13 shows the same silence one layer deeper (bound-but-never-publishing on a duplicate attributeRef). When a first-wins/last-wins tiebreak decides which of two authored things works, the loser must be logged and ledgered.
5. **Gateway-gap workarounds still accrue client-side** — wire-side `maxEvents`, server-side event source filter, batched replay, batched `SendEvents`, and the WriteComplete correlation's completion-wait timeout (the `galaxy.writes.unconfirmed` residual). All are known and metered/documented here; they belong on the sister repos' backlogs with links from the in-source comments.