lmxopcua/archreview/plans/06-gateway-integrations-plan.md

Plan 06 — Gateway Integrations (Galaxy Driver + Historian Gateway Driver)

Status (2026-07-08): ✅ U-1 doc DONE (CLAUDE.md KNOWN LIMITATION 2 corrected — historized-ref feed is wired). Remaining: S-1 (Galaxy write fail-closed — failure-visibility), S-6 (dormant historian health surface — built-but-never-wired). See STATUS.md.

• Source report: archreview/06-gateway-integrations.md
• Scope: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy (+ .Contracts, .Browser), src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway
• Commit reviewed against: current tree (report cut at 9cad9ed0)
• Verification method: every cited file:line opened and checked against the current tree.

Verification summary

Finding	Sev	Status	Note
S-1 Galaxy write optimistic-Good	High	Confirmed	GatewayGalaxyDataWriter.cs 234-243 (advise-fail proceeds), 281-302 (empty statuses → Good), 162-165 (silent-loss comment) all present verbatim.
S-2 EventPump drops newest	Medium	Confirmed	EventPump.cs:136-139 TryWrite → count drop, newest-dropped by construction.
S-3 Read can hang on non-cancellable token	Medium	Confirmed	GalaxyDriver.cs:738-753: BadTimeout fill only via cancellationToken.Register; awaits at 753 block forever with CancellationToken.None.
S-4 Transport-fault detection is EventPump-only	Medium	Confirmed	ReportTransportFailure sole prod caller is OnEventPumpStreamFault (GalaxyDriver.cs:943); WriteAsync (line 803) and subscribe path never feed the supervisor.
S-5 Outbox RemoveAsync prefix truncation silent	Medium	Confirmed	FasterLogHistorizationOutbox.cs:158-182: removes target + all older live entries, no _droppedCount bump on non-target removals.
S-6 Historian health flags dormant	Medium	Confirmed (worse)	RefreshConnectionStateAsync has only test callers; GetHealthSnapshot() itself has zero production callers — the whole snapshot is currently unconsumed in-process.
S-7 Alarm feed reconnect no backoff	Low	Confirmed	GatewayGalaxyAlarmFeed.cs:141 fixed _reconnectDelay (default 5 s), no exponential/jitter.
S-8 GalaxyMxSession unsynchronized state	Low	Confirmed	GalaxyMxSession.cs:28-32 plain fields; RecreateAsync (112-117) tears down without a lock. Emergent safety accepted.
S-9 Store-and-forward discipline	Positive	Confirmed positive	No action.
S-10 TLS / API-key posture	Positive	Confirmed positive	No action.
P-1 Read = 3 round-trips	Medium	Confirmed	ReadViaSubscribeOnceAsync 658-780; MxAccess-forced. Doc + optional cache.
P-2 Serial SendEvent per event	Medium	Confirmed (accepted)	Gateway-constrained; track batched SendEvents on sidecar.
P-3 Channel count	Low	Confirmed (accepted)	Documented trade-off; no action.
P-4 Indexed fan-in/out	Positive	Confirmed positive	No action.
P-5 HistoryRead buffers; maxEvents<=0 unbounded	Low	Confirmed	GatewayHistorianDataSource.cs:142-168 hasCap = maxEvents>0; when false collects unbounded.
P-6 Peek holds lock over disk I/O	Low	Confirmed	FasterLogHistorizationOutbox.cs:145-152. Negligible at 64-batch.
C-1 Secret resolver	Positive	Confirmed positive	No action.
C-2 Seam quality	Positive	Confirmed positive	No action.
C-3 Historian seam leaks wire types	Medium	Confirmed	IHistorianGatewayClient.cs:1-11 proto-typed by design. Doc-guard only.
C-4 Options-validation idioms diverge	Low	Confirmed	Galaxy throw-on-construct vs Historian warnings-list. Convention note.
C-5 Retired Wonderware dirs on disk	Low	Confirmed	3 dirs present, 0 slnx references. Delete.
C-6 EventPump drops unknown families unmetered	Low	Confirmed	EventPump.cs:199-206 default: return; no counter.
U-1 CLAUDE.md KNOWN LIMITATION 2 stale	High (doc)	Confirmed stale	Feed fully wired: AddressSpaceApplier.cs:343 UpdateHistorizedRefs → ActorHistorizedTagSubscriptionSink → recorder OnUpdateHistorizedRefs (ContinuousHistorizationRecorder.cs:186/311). Doc must be corrected; path never live-verified.
U-2 Live-validation gate partially run	Medium	Confirmed	Infra complete; needs a full documented run + CLAUDE.md status.
U-3 WriteSecured user id stubbed at 0	Medium	Confirmed	GatewayGalaxyDataWriter.cs:263-264 CurrentUserId=0, VerifierUserId=0.
U-4 Dormant paths inventory	Medium	Confirmed	HistorianGatewayClientAdapter.cs:88 drops maxEvents on the wire; GalaxyDriver.cs:314-316 replay note; FlushOptionalCachesAsync no-op.
U-5 Outbox serializer has no version byte	Low	Confirmed	HistorizationOutboxEntrySerializer.cs:14-16 fixed positional layout, no prefix.
U-6 Test-coverage thin spots	—	Confirmed	S-1 failure branch, S-5 out-of-order, S-3 hang, C-6 filter all untested. Folded into each finding's test section.

Totals: 28 findings — 23 actionable, 5 positive (no action). Of the 23 actionable, all 23 confirmed against the current tree; 0 stale-as-already-fixed (U-1 is stale documentation against correct code).

Cross-cutting alignment with 00-OVERALL.md: S-1 is part of OVERALL action #7 (failure visibility); S-6 + U-1 are instances of OVERALL theme #1 ("built-but-never-wired"); U-1 + U-2 are OVERALL theme #5 (doc drift); U-4/C-3 are theme #4 (gateway-gap workarounds accrue client-side).

---

Priority-ordered plan

Priority key: P0 ship first (data-integrity / visibility), P1 next (fault resilience), P2 hygiene/docs, P3 nice-to-have.

---

P0-A — S-1: Galaxy write success is optimistic; a committed-write failure can never surface

Restatement: Empty MX status arrays translate to Good; a failed supervisory-advise logs a warning and lets the raw Write proceed even though the file's own comment says such a write "never reaches the galaxy"; write-outcome self-correction (#5) structurally cannot fire because Galaxy ExecuteWrite is fire-and-forget.

Verification: Confirmed. GatewayGalaxyDataWriter.cs:

• 234-243: on AdviseSupervisory protocol failure → TryRemove the handle, log warning, fall through to WriteRawAsync.
• 281-302: TranslateReply → protocol-status check, then first MX status row, else new WriteResult(StatusCodeMap.Good).
• 162-165: in-source comment confirms a non-advised raw Write "doesn't throw (reply looks OK) but the value never reaches the galaxy."

Root cause: Two independent optimistic defaults. (1) The gateway reply's Statuses array reflects command acceptance at the worker, not the eventual COM-side WriteComplete; an empty array is treated as success. (2) The supervisory-advise precondition is best-effort — when it fails the code still issues a Write it knows won't commit, and reports the Write's (empty/OK) reply as Good.

Design.

In-repo (this PR) — three layers, cheap and independent:

1. Fail-closed on advise failure. Change EnsureSupervisoryAdvisedAsync to return a bool (advised? true/false). In WriteOneAsync, when a non-secured write's advise returns false, do not issue the raw Write — return new WriteResult(StatusCodeMap.BadWaitingForInitialData /* or Uncertain */). Rationale: the code already knows the value can't commit; sending it and reporting Good is the exact silent-loss the comment describes. Preferred status: Uncertain_NoCommunicationLastUsableValue is wrong (implies a read); use Bad family so #5 self-correction could fire (BadNotConnected/BadWaitingForInitialData). Decision: return StatusCodeMap.BadCommunicationError — it is already the writer's "couldn't reach the device" status, so the node-write router / self-correction path treats it consistently with a transport failure. (The report suggests Uncertain; Bad is stronger and lets self-correction revert the phantom-Good node — align with OVERALL theme #3 "the failure must surface somewhere".)

2. Empty-statuses is not Good for a data write. TranslateReply: when reply.Statuses.Count == 0 and the reply is a data-write reply, return StatusCodeMap.UncertainNoCommunicationLastUsableValue-class or, more defensibly, keep Good only when the gateway contract guarantees a status row on success. This needs a gateway-contract check (see cross-repo). Interim: keep Good but meter it (layer 3) so the unconfirmed rate is observable, and add an XML-doc note that empty-array-Good is provisional pending the WriteComplete correlation.

3. galaxy.writes.unconfirmed counter. Add a Counter<long> on the existing GalaxyTelemetry/pump meter (ZB.MOM.WW.OtOpcUa.Driver.Galaxy), incremented whenever TranslateReply returns Good off an empty statuses array, and a galaxy.writes.advise_failed counter on the advise-failure branch. Gives ops the signal the report asks for.

Cross-repo (mxaccessgw) — the real fix, tracked follow-on: pursue a gateway-side WriteComplete correlation so the MxCommandReply.Statuses row carries the real COM commit outcome. The OnWriteComplete event family already exists in the proto and is currently filtered out by EventPump.Dispatch (EventPump.cs:199-206). Design: gateway correlates the async WriteComplete back to the originating command's ClientCorrelationId and folds the MX status into the unary reply's Statuses before returning (making Write synchronously authoritative), or emits OnWriteComplete on the event stream keyed by correlation id and the driver awaits it with a deadline. The synchronous-fold option is cleaner for the OPC UA write contract (write is a blocking service call). This is a sibling-repo change; file it on the mxaccessgw backlog and link from the in-source comment at GatewayGalaxyDataWriter.cs:162-165.

Implementation steps (in-repo):

• GatewayGalaxyDataWriter.cs: change EnsureSupervisoryAdvisedAsync → Task<bool>; branch in WriteOneAsync to short-circuit to BadCommunicationError when advise failed.
• Add WriteUnconfirmed / WriteAdviseFailed counters (new static Counter<long> fields; reuse EventPump.MeterName string or a shared GalaxyTelemetry meter — check GalaxyTelemetry.cs for the existing meter name to avoid a second Meter).
• TranslateReply: increment WriteUnconfirmed on the empty-statuses branch; add doc note.

Tests (unit):

• New GatewayGalaxyDataWriterTests cases (writer already has an internal ctor + SeedHandleCachesForTest): advise-fails → assert returned status is BadCommunicationError and no WriteRawAsync was issued (fake session records calls).
• Empty-statuses reply → assert Good returned and the unconfirmed counter incremented (via a MeterListener).
• Protocol-status non-OK → BadCommunicationError (regression guard).

Tests (live gateway): extend the skip-gated GatewayGalaxyLiveReopenAndWriteTests with a case that forces an advise-failure (or asserts the unconfirmed counter is emitted on a normal write). Recipe:

KEY=$(docker exec otopcua-dev-central-1-1 printenv GALAXY_MXGW_API_KEY)
MXGW_ENDPOINT=http://10.100.0.48:5120 GALAXY_MXGW_API_KEY="$KEY" \
  dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests \
  --filter FullyQualifiedName~GatewayGalaxyLiveReopenAndWrite

Effort: S (in-repo layers 1+3), M (layer 2 + cross-repo correlation). Risk/blast radius: Low in-repo — the change makes a currently-silent-loss visible; the only behavioral change is that advise-failed writes now return Bad instead of phantom-Good, which is strictly more correct and feeds #5 revert. Live-verify to confirm normal writes still return Good.

---

P0-B — S-6: Historian health snapshot's connection flags are dormant (and the snapshot is unconsumed)

Restatement: RefreshConnectionStateAsync has no production caller, so ProcessConnectionOpen/EventConnectionOpen are permanently false; deeper: GetHealthSnapshot() itself is never called in production.

Verification: Confirmed and worse than reported. RefreshConnectionStateAsync callers = 3 test lines only. GetHealthSnapshot() invocation grep across src returns only the interface declaration — no production consumer. So the entire health surface (GetHealthSnapshot + the refresh) is dead weight in a deployed host today.

Root cause: The "register-AND-pass-into-consumer" trap (same class as PR #423 provisioner, OVERALL theme #1): the component was built and unit-tested, but neither the periodic refresh nor a snapshot consumer was wired.

Design. Two decisions:

1. Is the snapshot meant to be surfaced? Check HistorianHealthSnapshot's intended consumer (AdminUI historian-status panel / a health endpoint). If yes → wire it. If no (the flags predate a since-abandoned dashboard) → remove ProcessConnectionOpen/EventConnectionOpen + RefreshConnectionStateAsync to kill the misleading-false surface. The report offers both; pick based on whether any dashboard is planned. Recommendation: wire it, because the other integrations (Galaxy HostStatusAggregator) do surface connectivity and a historian-down indicator is genuinely useful.

2. How to drive the refresh. Add a lightweight IHostedService (HistorianHealthRefreshHostedService) that, when ServerHistorian:Enabled, resolves the registered IHistorianDataSource, and if it is a GatewayHistorianDataSource, calls RefreshConnectionStateAsync on a timer (reuse the existing health-probe cadence if one exists — check OtOpcUaServerHostedService for an existing periodic loop to fold into rather than spawning a second timer). Also wire a consumer: project the snapshot into whatever the AdminUI/health endpoint reads.

Implementation steps:

• New HistorianHealthRefreshHostedService in ZB.MOM.WW.OtOpcUa.Runtime (or Host), registered in the historian DI block alongside AddServerHistorian. Guard on Enabled. Interval from a new ServerHistorianOptions.HealthRefreshSeconds (default 15).
• Cast/seam: RefreshConnectionStateAsync is on the concrete GatewayHistorianDataSource, not IHistorianDataSource. Either (a) add an optional IHistorianConnectionProbe { Task RefreshConnectionStateAsync(ct) } capability interface the gateway source implements and the hosted service checks for, or (b) resolve the concrete type. Prefer (a) — matches the capability-interface pattern used elsewhere and keeps NullHistorianDataSource clean.
• Wire the snapshot consumer (AdminUI panel or /health); if none is desired, execute decision-branch (1) removal instead.

Tests:

• Unit: hosted service calls RefreshConnectionStateAsync on tick against a fake source; disabled → never resolves.
• Unit: GatewayHealthSnapshotTests already covers the refresh math — add an assertion that after a successful refresh the snapshot flags are true.
• Live: after wiring, run the historian live suite (U-2) and confirm the snapshot flips to connected against the real sidecar.

Effort: S (wire refresh) / M (if adding a consumer surface). Risk: Low. Note: live-/run required per OVERALL theme #1 — do not close on unit tests alone.

---

P0-C — U-1: CLAUDE.md KNOWN LIMITATION 2 is stale; continuous-historization value-capture never live-verified

Restatement: CLAUDE.md says the recorder is spawned with an empty ref-set and "historizes nothing"; the code has closed that gap but the end-to-end path was never live-verified.

Verification: Confirmed. The feed is fully wired: ServiceCollectionExtensions.cs:272 still spawns with Array.Empty<string>(), but wraps in ActorHistorizedTagSubscriptionSink (line 290-292) and hands it to the applier; AddressSpaceApplier.cs:310-343 FeedHistorizedRefs computes the per-deploy add/remove delta and calls _historizedSubscriptions.UpdateHistorizedRefs; ContinuousHistorizationRecorder.cs:186/311 receives UpdateHistorizedRefs and re-registers. CLAUDE.md's "Known Limitation 2" text is therefore factually wrong.

Root cause: Documentation drift — the feed landed after the CLAUDE.md note was written and the note was never reconciled.

Design/steps:

1. Doc fix (this PR): rewrite CLAUDE.md "KNOWN LIMITATION 2" to state the ref-feed is wired (applier FeedHistorizedRefs → ActorHistorizedTagSubscriptionSink.UpdateHistorizedRefs → recorder OnUpdateHistorizedRefs), and downgrade it from "records no values" to "wired; pending a live end-to-end verification." Propagate the same correction to the ../scadaproj/CLAUDE.md OtOpcUa index entry (per the repo's cross-repo propagation rule).
2. Live-verify the value-capture path (the real close-out): deploy a config with a historized numeric tag against a real HistorianGateway (VPN to the sidecar / wonder-sql-vd03, gateway with RuntimeDb:Enabled=true), drive a value change, and confirm WriteLiveValues lands (outbox drains, gateway shows the sample). This is the OVERALL theme #1 "prove production wiring" gate.
3. Add a restart-convergence integration test (unit-level with a fake gateway value-writer + real outbox): deploy delta adds refs → mux value → outbox append → drain calls writer; assert the writer received the value. This closes the "wired but inert" risk with a repeatable test even before the live run.

Effort: S (doc) + M (live-verify + convergence test). Risk: Doc-only change is zero-risk; the live run may surface a real inertness bug (exactly what OVERALL warns about) — budget for a fix if it does.

---

P1-A — S-4: Transport-failure detection is EventPump-only; a write-only/idle driver never degrades

Restatement: ReportTransportFailure is only called from the EventPump stream-fault callback; a driver that only writes or is idle after discovery keeps GetHealth()==Healthy through a gateway restart, and each write fails per-request forever with no reopen/replay.

Verification: Confirmed. Sole prod caller GalaxyDriver.cs:943 (OnEventPumpStreamFault). WriteAsync (803) delegates straight to _dataWriter.WriteAsync with no supervisor hand-off; failed unary RPCs (SubscribeBulk, Write, AcknowledgeAlarm) don't feed the supervisor.

Root cause: Fault detection was built around the long-lived StreamEvents pump only; unary RPC failures were treated as per-call and never escalated to the recovery state machine.

Design. ReportTransportFailure is idempotent by design (the supervisor single-flights recovery), so reporting from more sites is cheap and safe. Add classified transport-failure reporting from the unary paths:

• In GatewayGalaxyDataWriter.WriteOneAsync's catch (currently maps to BadCommunicationError), and in the subscribe path, classify the exception (RpcException with transport status codes: Unavailable, DeadlineExceeded, Internal, etc.) and, when transport-fatal, invoke a supplied Action<Exception>? reportTransportFailure callback (mirror the pump's onStreamFault pattern) that routes to _supervisor.ReportTransportFailure.
• Wire the callback from GalaxyDriver when it constructs the writer/subscriber, guarded like OnEventPumpStreamFault (null supervisor = skeleton path, no-op).
• Reuse the existing transport-classification helper if one exists (grep StatusCodeMap/ReconnectSupervisor for a classifier); otherwise add a small IsTransportFatal(Exception) in the driver.

Implementation steps:

• Add a Func<Exception,bool>/callback seam to GatewayGalaxyDataWriter (optional ctor arg, like subscribedHandleSource) or wrap the call at the GalaxyDriver.WriteAsync boundary (cleaner — keep the writer pure). Prefer wrapping at GalaxyDriver: catch transport-classified failures returned/thrown from the writer and call supervisor.ReportTransportFailure.
• Same for SubscribeAsync/ReadViaSubscribeOnceAsync SubscribeBulk failures and GatewayGalaxyAlarmAcknowledger.

Tests (unit): fake writer that throws an Unavailable RpcException → assert GalaxyDriver.WriteAsync reports the failure to a fake supervisor and returns Bad; non-transport ArgumentException → not reported. Idempotency: two failures → supervisor sees N reports but single-flights (already covered by ReconnectSupervisorTests — add a driver-level report test).

Effort: M. Risk: Low-Medium — must avoid escalating application-level write rejections (bad value, secured-tag denial) to transport recovery; the classifier must be conservative (only RpcException transport codes).

---

P1-B — S-3: ReadViaSubscribeOnceAsync can wait forever on a non-cancellable token

Restatement: BadTimeout fill is only via cancellationToken.Register; a CancellationToken.None caller whose tag subscribes OK but never publishes hangs at the await, holding the subscription open.

Verification: Confirmed. GalaxyDriver.cs:738-748 registers the fill on the CT; 750-755 awaits each pending TCS unconditionally. No internal deadline.

Root cause: The read deadline is delegated entirely to the caller's token; an infinite/None token has no fallback.

Design. Race each pending await against an internal deadline derived from options regardless of the caller's token. Compute deadline = max(PublishingIntervalMs, DefaultCallTimeoutSeconds) × a small factor (e.g. 3× publishing interval, floored at the call timeout). Implement by combining tokens: create a linked CTS CreateLinkedTokenSource(cancellationToken) + CancelAfter(deadline), and register the BadTimeout fill on the linked token instead of the caller's. Then every path (caller cancel, internal deadline) fires the fill. Ensure the linked CTS is disposed in the finally.

Implementation steps:

• In ReadViaSubscribeOnceAsync: build using var deadlineCts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); deadlineCts.CancelAfter(TimeSpan.FromMilliseconds(readDeadlineMs)); and change cancellationToken.Register(...) → deadlineCts.Token.Register(...).
• Derive readDeadlineMs from _options.MxAccess.PublishingIntervalMs and _options.Gateway.DefaultCallTimeoutSeconds (or equivalent). Add an option ReadSynthesisTimeoutMs if a dedicated knob is warranted; default derived.

Tests (unit): the driver has injectable seams (IGalaxySubscriber fake). Fake that returns a successful SubscribeBulk but never raises OnDataChange → call ReadViaSubscribeOnceAsync with CancellationToken.None → assert it returns BadTimeout snapshots within the deadline (not hang) and that UnsubscribeBulk was called (subscription not leaked).

Effort: S. Risk: Low — pure addition of an internal deadline; existing caller-cancel behavior preserved via the linked token.

---

P1-C — S-5: FasterLog outbox RemoveAsync truncates the FIFO prefix; out-of-order acks silently drop unacked entries

Restatement: RemoveAsync(id) removes the target plus every older live entry and truncates to the target's successor; if a mid-batch id is acked first, older unacked values are durably discarded without incrementing DroppedCount.

Verification: Confirmed. FasterLogHistorizationOutbox.cs:158-182: while-loop removes _live.First repeatedly until it hits the target and breaks; _droppedCount not touched for the collateral removals. Contract "acks in FIFO order" enforced by comment only.

Root cause: FasterLog is a truncate-to-address log; removing an arbitrary id necessarily discards everything before it. The API accepted a single id while the underlying primitive is prefix-truncation, so the invariant lives only in the caller's discipline.

Design. Two options:

1. Count + warn (minimal): in RemoveAsync, when the removal drops entries whose id != the target, increment _droppedCount per collateral entry and log a warning naming the count. Makes the silent loss observable. Cheap, no API change.
2. Make semantics explicit at the call site (preferred): add RemoveThroughAsync(Guid throughId) (or RemoveBatchAsync(IReadOnlyList<Guid>)) to IHistorizationOutbox, documenting that it truncates the FIFO prefix through the id; keep single-id RemoveAsync but have it assert the target is _live.First (FIFO) and throw/log if not. This surfaces the invariant at compile/call time.

Recommendation: do both — rename the operation to RemoveThroughAsync to match reality, and add the collateral-drop counter as a safety net for any future non-FIFO caller.

Implementation steps:

• IHistorizationOutbox: add RemoveThroughAsync (or annotate RemoveAsync contract). Update ContinuousHistorizationRecorder drain call site to use it.
• FasterLogHistorizationOutbox.RemoveAsync: count collateral removals into _droppedCount (or a new _ackPrefixDropped counter for clarity) and _logger.LogWarning when > 0.

Tests (unit): append A,B,C; RemoveAsync(C.Id) (skipping A,B) → assert A,B are gone, DroppedCount == 2 (or the new counter), and a warning was logged. Regression: FIFO ack A,B,C in order → DroppedCount == 0.

Effort: S. Risk: Low — additive counter; API rename is mechanical (single recorder caller).

---

P1-D — S-7: Alarm feed reconnect has no backoff

Restatement: GatewayGalaxyAlarmFeed.RunAsync re-opens on a fixed 5 s delay forever; DeployWatcher and ReconnectSupervisor both use capped exponential + jitter.

Verification: Confirmed. GatewayGalaxyAlarmFeed.cs:141 await Task.Delay(_reconnectDelay, ct) with a fixed _reconnectDelay.

Root cause: Inconsistency — the alarm feed predates or diverged from the shared backoff pattern.

Design. Replace the fixed delay with capped exponential backoff + jitter, resetting to the floor on a successful stream open (i.e., reset the attempt counter after the first message or after the snapshot_complete). Reuse the existing backoff helper DeployWatcher/ReconnectSupervisor use — grep for a shared ExponentialBackoff/Jitter type; if none is shared, extract one into the Galaxy Runtime/ folder and use it in all three (aligns with OVERALL theme #2 "fixes flow to shared templates").

Implementation steps:

• Introduce/reuse a Backoff primitive (min, max, jitter). Wire into GatewayGalaxyAlarmFeed ctor (min/max from GalaxyReconnectOptions, defaults 1 s→30 s). Reset on successful open.

Tests (unit): the feed uses an injected _streamFactory and a settable delay; assert successive faults produce increasing delays capped at max and reset after a successful open (drive via a fake factory + a virtual clock or by asserting the computed delay sequence).

Effort: S. Risk: Low.

---

P2-A — C-5: delete retired Wonderware project directories

Verification: Confirmed — Historian.Wonderware, .Client, .Client.Contracts dirs present; grep -c Wonderware slnx == 0.

Design/steps: git rm -r the three src/Drivers/*Wonderware* dirs and their tests/ counterparts; confirm no ProjectReference/using survives (grep). This overlaps OVERALL action #11 and Report 07 hygiene — coordinate so it's deleted once. History preserves the code.

Effort: S. Risk: Very low (0 references). Verify with a full dotnet build ZB.MOM.WW.OtOpcUa.slnx after removal.

---

P2-B — C-6: EventPump silently drops unknown event families with no metric

Verification: Confirmed. EventPump.cs:199-206 default: return; — no counter.

Design/steps: add a galaxy.events.filtered (or .unknown_family) counter incremented in the default branch of Dispatch, tagged with the family value. Mirrors the alarm feed's decode-drop metering. Keep the filtering behavior — only add observability, so a future gateway family (OnBufferedDataChange) surfaces instead of vanishing.

Tests (unit): feed the pump an MxEvent with a non-OnDataChange family via the injected subscriber fake → assert the filtered counter incremented (MeterListener) and no OnDataChange fired.

Effort: S. Risk: Very low.

---

P2-C — U-3: WriteSecured / VerifiedWrite user identity stubbed at zero

Verification: Confirmed. GatewayGalaxyDataWriter.cs:263-264 hardcodes CurrentUserId=0, VerifierUserId=0.

Root cause: No mapping from the OPC UA session's LDAP-authenticated principal to an ArchestrA user id; the secured-write path was stubbed to unblock the common (non-secured, supervisory) path.

Design. Two-part, and honest about the gap:

1. Fail-fast interim (this PR): when a tag classifies SecuredWrite/VerifiedWrite and no ArchestrA user-id mapping is configured, return a clear Bad status (BadUserAccessDenied / BadNotSupported) and log, rather than silently sending user 0 (which a real galaxy will reject or mis-attribute). Prevents a confusing "commit succeeded as user 0" or opaque rejection. Also surfaces to the operator that secured-classification tags are unsupported today.
2. Real fix (cross-cutting, larger): thread the OPC UA session identity down to the writer. The LDAP principal is available server-side (see RoleCarryingUserIdentity used by alarm ack). Add an ArchestrA-user-id mapping (config GroupToArchestraUserId or a resolver) and pass CurrentUserId/VerifierUserId into InvokeWriteSecuredAsync. This spans the node-write router → driver WriteRequest (needs a user-id/identity field on the request contract) → writer. Coordinate with Report 03 (session identity) and the node-write router. VerifiedWrite additionally needs a second verifier identity (dual-authorization) — that is a genuine feature, not a plumbing fix; scope it separately.

Implementation steps (interim):

• WriteOneAsync: when NeedsSecuredWrite(classification) and no user-id source is wired → return BadNotSupported/BadUserAccessDenied, log, do not send. Document in docs/ (Galaxy driver / Historian split) that secured/verified writes require the identity-mapping feature.

Tests (unit): classification resolver returns SecuredWrite + no identity → assert Bad status and no InvokeAsync call.

Effort: S (interim), L (real identity threading — cross-report). Risk: interim is Low and strictly safer than sending user 0.

---

P2-D — U-4 / P-5: gateway-gap client-side workarounds; clamp unbounded events read

Verification: Confirmed.

• HistorianGatewayClientAdapter.cs:88 calls _inner.ReadEventsAsync(startUtc, endUtc, filter, ct) — maxEvents not on the wire (client-side cap only).
• GatewayHistorianDataSource.cs:142-168 — maxEvents<=0 collects unbounded (only the gateway's RuntimeDb:EventReadMaxRows bounds it).
• Client-side source filter re-applied defensively at 152-157.
• GalaxyDriver.cs:314-316 replay still per-subscription (batched ReplaySubscriptionsCommand note).

Design/steps:

1. In-repo: clamp maxEvents<=0 to a server-side default cap (new ServerHistorianOptions.DefaultMaxEventsPerRead, default e.g. 10 000) in GatewayHistorianDataSource.ReadEventsAsync so a 0 from the SDK can't drain unboundedly. This is the P-5 fix and is fully in-repo.
2. Cross-repo (sidecar backlog): file tracked items for (a) a wire-level maxEvents on ReadEventsRequest, (b) a server-side source filter so the defensive client filter can be removed, (c) a batched SendEvents (P-2), and (d) a batched ReplaySubscriptionsCommand (mxaccessgw). Link each from the corresponding in-source comment. These are not in-repo fixes — they belong on HistorianGateway / mxaccessgw backlogs; the driver's workarounds stay until they land.

Tests (unit): FakeHistorianGatewayClient streaming > cap events with maxEvents=0 → assert the data source stops at DefaultMaxEventsPerRead and flags truncation.

Effort: S (clamp). Risk: Low. Note: OVERALL theme #4 — track the gateway gaps on the sister repos so they don't rot.

---

P3-A — S-2: EventPump saturation drops the newest events (inverted staleness bias)

Verification: Confirmed. Newest-dropped by construction (TryWrite on a full bounded channel). Metered (galaxy.events.dropped).

Root cause: Bounded channel with FullMode.Wait + TryWrite preserves the oldest queued events and drops the just-arrived one — wrong for last-value-wins telemetry.

Design. Two candidates:

1. Per-item-handle conflation (best correctness): keep newest-per-handle. Replace the raw Channel<MxEvent> with a small conflating structure: a ConcurrentDictionary<long,MxEvent> (handle→latest) + a signal channel; the dispatch loop drains the dict. On overflow the newest value for a handle overwrites the older one — no data staleness, bounded by tag count not event rate. More code; changes the pump's core.
2. Drop-oldest ring (cheap): switch the channel to BoundedChannelFullMode.DropOldest. But the in-source comment explicitly rejected DropWrite/DropOldest because they discard inside Channel without surfacing on the counter. To keep metering, wrap: on a full channel, TryRead one (count it as dropped-oldest) then TryWrite the new one. Simpler than conflation, keeps recent data, preserves the counter.

Recommendation: option 2 (drop-oldest-with-metering) as the low-risk win; option 1 only if soak testing shows a hot-handle storm still loses fresh values. Given P-4 notes the fan-out is already O(1) and the channel default is 50 000, saturation is an edge case — option 2 is proportionate.

Implementation steps:

• In RunAsync, replace the if (!TryWrite) count block with: while (!TryWrite(ev)) { if (TryRead(out _)) EventsDroppedOldest.Add(1,tag); } (bounded loop). Add a galaxy.events.dropped_oldest counter (or reuse dropped). Update the FullMode comment to match.

Tests (unit): fill the channel (tiny capacity via ctor), push N more → assert the newest are retained (read the dispatched values) and the dropped counter reflects the discarded oldest.

Effort: S-M. Risk: Medium — touches the hot pump loop; guard with the existing pump bounded-channel tests + a soak run. Only pursue if S-2 is judged worth the churn (report rates it Medium; it's already metered, so it's a correctness-under-stress refinement).

---

P3-B — S-8: GalaxyMxSession state unsynchronized across reopen (document the invariant)

Verification: Confirmed. Plain _session/_connected fields; RecreateAsync tears down without a lock. Report rates it acceptable (emergent safety via single-flight supervisor + per-call try/catch).

Design/steps: lowest-effort resolution the report endorses — document the invariant on the Session/IsConnected members: "single-flight recovery (ReconnectSupervisor) is the only writer of session state; concurrent readers tolerate a torn read because every per-call failure maps to BadCommunicationError and caches are invalidated post-reopen." If a defensive change is wanted, mark _connected volatile and read _session into a local once per operation (already the pattern in WriteAsync). Do not add a lock (would serialize the hot path for no proven benefit).

Effort: XS (doc) / S (volatile). Risk: None (doc) / Low.

---

P3-C — U-5: outbox serializer has no version/format byte

Verification: Confirmed. HistorizationOutboxEntrySerializer.cs:14-16 fixed positional layout, no prefix.

Design/steps: prepend a single version byte (0x01) in Serialize; in Deserialize read it first and branch (today: only v1). One reserved byte now makes any future field addition a clean migration instead of an undiagnosable recovery failure mid-RecoverState. Migration caveat: on-disk entries written before this change have no version byte — deploying this reader against a non-empty pre-existing outbox would misread. Since the outbox is drained-and-truncated continuously and is dev-only so far, gate the change on an empty-outbox deploy, or make the reader tolerant (peek: if the first byte isn't a known version AND the buffer length matches the legacy layout, treat as v0). Prefer the tolerant reader for safety.

Tests (unit): round-trip v1; legacy-layout bytes (no version) still deserialize under the tolerant reader; unknown version → clear exception.

Effort: S. Risk: Low-Medium (the on-disk migration is the only sharp edge — the tolerant reader defuses it).

---

P3-D — C-3, C-4: convention notes (documentation only)

• C-3: add a one-line guard note to IHistorianGatewayClient's XML doc: "any consumer beyond GatewayHistorianDataSource must sit above the Mapping/ anti-corruption boundary, never on this proto-typed seam." No code change. Effort: XS.
• C-4: pick one options-validation idiom for new driver-adjacent sections — the report (and I) recommend the Historian Validate()-returns-warnings style as the friendlier operational contract. Document the choice in CLAUDE.md / a conventions doc; do not churn the existing Galaxy throw-on-construct code. Effort: XS (doc).

---

Accepted / no-action

• S-9, S-10, P-4, C-1, C-2 — positives; preserve as-is (don't regress the store-and-forward discipline, the O(1) fan-out indexes, or the secret resolver).
• P-1 — MxAccess-forced 3-round-trip read; action is documentation ("subscribe, don't poll") + optional read-through cache only if polling clients appear. Fold the doc line into docs/ Galaxy section. Effort: XS.
• P-2, P-3, P-6 — accepted, gateway-constrained or negligible; P-2 tracked on the sidecar backlog (see U-4).

---

Suggested execution order (batched)

1. Batch 1 (P0, failure-visibility): S-1 (in-repo layers) + S-6 (refresh wiring) + U-1 (doc fix + convergence test). Ship together; all three are OVERALL theme #1/#3. Live-verify S-1 write and U-1 value-capture on the real gateways.
2. Batch 2 (P1, fault resilience): S-4 + S-3 + S-5 + S-7. Cohesive "recovery hardening" set, all unit-testable with existing fakes.
3. Batch 3 (P2, hygiene/interim): C-5 (delete dirs) + C-6 (metric) + U-3 (fail-fast) + U-4/P-5 (clamp) + C-3/C-4 (doc). Coordinate C-5 with Reports 03/07.
4. Batch 4 (P3, refinements): S-2 (drop-oldest) + S-8 (doc/volatile) + U-5 (version byte) + P-1 doc. Optional / as-capacity.

Cross-repo follow-ons to file (not in-repo): mxaccessgw WriteComplete correlation (S-1) and batched ReplaySubscriptionsCommand (U-4); HistorianGateway wire-level maxEvents, server-side source filter, batched SendEvents (U-4/P-2). Link each from the existing in-source comment.