Files
lmxopcua/archreview/06-gateway-integrations.md
T
Joseph Doherty 9fadead6a6 docs(archreview): remediation plans + fix flagged doc drift
Add the 7 per-domain design+implementation plans (archreview/plans/) with
an index, produced from the 2026-07-08 architecture review.

Fix two confirmed doc drifts the review flagged (theme #5):
- CLAUDE.md KNOWN LIMITATION 2: the continuous-historization historized-ref
  feed IS wired (AddressSpaceApplier.FeedHistorizedRefs -> UpdateHistorizedRefs
  -> recorder); rewrite to reflect that value-capture is code-complete and only
  the live end-to-end + restart-convergence verification remains.
- CLAUDE.md ScriptAnalysis gating: endpoints use Roles=Administrator,Designer
  via RequireAuthorization, not the FleetAdmin policy.
2026-07-08 16:14:37 -04:00

26 KiB
Raw Blame History

Architecture Review 06 — Gateway Integrations (Galaxy Driver + Historian Gateway Driver)

  • Date: 2026-07-08
  • Commit: 9cad9ed0 (master)
  • Scope:
    • src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy (+ .Contracts, .Browser)
    • src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway
    • Test coverage: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests, .Galaxy.Browser.Tests, .Historian.Gateway.Tests
    • Peripheral (read for context, not reviewed in depth): Runtime/Historian/* (recorder, options), AddressSpaceApplier provisioning/subscription hooks, OtOpcUaNodeManager tie-cluster paging.
  • Dimensions: Stability, Performance, Conventions, Underdeveloped Areas.

Architecture Overview

Galaxy driver data flow (mxaccessgw)

GalaxyDriver (GalaxyDriver.cs) is a standard Tier-A in-process Equipment-kind driver registered under type name GalaxyMxGateway. All Galaxy access flows over gRPC to the external mxaccessgw gateway (sibling repo), consumed via the Gitea-feed packages ZB.MOM.WW.MxGateway.Client / .Contracts (MxCommand / MxEvent protos). The driver holds three logical gRPC clients:

  1. Worker sessionGalaxyMxSession wraps MxGatewayClient.OpenSessionAsync + MXAccess Register. All data-plane traffic rides it:
    • SubscribeGatewayGalaxySubscriber.SubscribeBulkAsync (session-level SetBufferedUpdateInterval applied first, cached last-applied value), events consumed by the shared EventPump off the bidirectional StreamEvents RPC, fanned out through SubscriptionRegistry's handle→subscription reverse map onto OnDataChange.
    • Read — MxAccess has no one-shot read; ReadViaSubscribeOnceAsync synthesises Read as SubscribeBulk → first-event wait per handle → UnsubscribeBulk.
    • WriteGatewayGalaxyDataWriter: lazy AddItem (or borrow a live handle from the subscription registry), AdviseSupervisory once per handle (required for commit under WriteUserId=0), then raw Write or WriteSecured routed on the discovery-captured per-tag SecurityClassification.
  2. Session-less client (_ownedMxClient) — the always-on central alarm monitor: GatewayGalaxyAlarmFeed (StreamAlarms: active-alarm snapshot → snapshot_complete → live transitions, self-reconnecting) and GatewayGalaxyAlarmAcknowledger (unary AcknowledgeAlarm).
  3. Repository client (GalaxyRepositoryClient) — hierarchy browse for DiscoverAsync (via IGalaxyHierarchySourceGalaxyDiscoverer) and the DeployWatcher deploy-event stream that raises IRediscoverable.OnRediscoveryNeeded.

Recovery is owned by ReconnectSupervisor (Healthy → TransportLost → Reopening → Replaying → Healthy, capped exponential backoff, never gives up): the EventPump stream fault feeds ReportTransportFailure; reopen routes through GalaxyMxSession.RecreateAsync (dispose stale session + client, rebuild) and invalidates the writer's handle/advise caches; replay recreates the EventPump then re-issues SubscribeBulk per tracked subscription and Rebinds the registry with the fresh handles. Host connectivity is surfaced through HostStatusAggregator (transport state from the supervisor + per-platform ScanState probes via PerPlatformProbeWatcher).

Historian gateway data flow (HistorianGateway sidecar)

Driver.Historian.Gateway is the sole historian backend, consuming the ZB.MOM.WW.HistorianGateway.Client package (historian_gateway.v1) behind the proto-typed IHistorianGatewayClient seam (HistorianGatewayClientAdapter is a pure pass-through; channels are lazy — no I/O at construction). Four independent paths, each with its own gRPC channel to the same sidecar (documented deliberate trade-off, GatewayHistorianServiceCollectionExtensions.cs:57-64):

  1. ReadGatewayHistorianDataSource (IHistorianDataSource): OPC UA HistoryRead Raw/Processed/AtTime/Events → ReadRaw / ReadAggregate / ReadAtTime / ReadEvents; at-time replies re-aligned one-snapshot-per-requested-timestamp; health counters under a single lock. Tie-cluster paging (MaxTieClusterOverfetch) lives server-side in OtOpcUaNodeManager (~line 2200), bound from ServerHistorianOptions.
  2. Alarm history writeGatewayAlarmHistorianWriter (SendEvent per event) behind the durable SqliteStoreAndForwardSink; maps every outcome (ack, typed client exceptions, raw RpcException) to exactly one Ack / RetryPlease / PermanentFail per event and never throws.
  3. Continuous historizationContinuousHistorizationRecorder (Runtime actor) taps the dependency-mux value fan-out, appends to the crash-safe FasterLogHistorizationOutbox (PerEntry fsync or Periodic commit; bounded drop-oldest capacity; startup recovery scan), drains through GatewayHistorianValueWriter (WriteLiveValues, non-throwing bool).
  4. Tag provisioningGatewayTagProvisioner (EnsureTags), dispatched fire-and-forget from AddressSpaceApplier.Apply with a tally log; non-historizable data types skipped; can never block or fail a deploy.

Findings

1. Stability

S-1 (High) — Galaxy write success is optimistic; a committed-write failure can never surface

GatewayGalaxyDataWriter.TranslateReply (GatewayGalaxyDataWriter.cs:281-302) honours the protocol status and the first MXAccess status row, defaulting to Good when the statuses array is empty — and the gateway's write execution is effectively fire-and-forget past dispatch (the reply reflects command acceptance, not the eventual COM-side commit). Two concrete consequences:

  • The supervisory-advise failure path (GatewayGalaxyDataWriter.cs:234-243) logs a warning, forgets the handle, and lets the write proceed anyway — but the file's own comment (:163-165) states a raw Write without supervisory advise "doesn't throw (reply looks OK) but the value never reaches the galaxy". That is a silent write loss returning Good to the OPC UA client.
  • The server's write-outcome self-correction (#5, reverts the node on a failed device write) can structurally never trigger for Galaxy, so a lost write leaves a phantom-Good node value indefinitely.

Blast radius: operator writes (WriteOperate-gated) to Galaxy attributes that silently don't commit, with no Bad status, no health degradation, no metric. Recommendation: (a) return Uncertain (not proceed-to-Good) when supervisory advise fails; (b) pursue a gateway-side WriteComplete correlation (the gw backlog's OnWriteComplete event family already exists in the proto — EventPump.cs:200-206 filters it out) so the reply/statuses row carries the real commit outcome; (c) add a galaxy.writes.unconfirmed counter in the interim.

S-2 (Medium) — EventPump saturation drops the newest events (inverted staleness bias)

EventPump.RunAsync (EventPump.cs:128-140) uses TryWrite against a bounded channel: when the fan-out consumer stalls, the just-arrived event is dropped and 50 000 stale queued events are preserved. For last-value-wins OPC UA telemetry this is the wrong bias — after a stall clears, subscribers replay old values and the freshest one may be the dropped one. The drop is at least metered (galaxy.events.dropped). Recommendation: per-item-handle conflation (keep newest per handle) or drop-oldest ring semantics; either preserves the no-backpressure requirement while keeping recent data.

S-3 (Medium) — ReadViaSubscribeOnceAsync can wait forever on a non-cancellable token

The read synthesis (GalaxyDriver.cs:738-755) fills pending snapshots with BadTimeout only via cancellationToken.Register. If a caller passes CancellationToken.None (or a token that never fires) and a successfully subscribed tag never publishes an initial event (gateway hiccup between SubscribeBulk and first push), the awaits at :753 never complete and the read hangs, holding the subscription open. Recommendation: race each pending TCS against an internal deadline derived from DefaultCallTimeoutSeconds / PublishingIntervalMs regardless of the caller's token.

S-4 (Medium) — Transport-failure detection is EventPump-only; a subscription-less driver never degrades

ReconnectSupervisor.ReportTransportFailure has exactly one production caller: the EventPump stream-fault callback (GalaxyDriver.cs:929-949). The pump only starts on the first subscribe/read. A driver doing only writes (or idle after discovery) whose gateway restarts will keep GetHealth() == Healthy; each write then fails per-request with BadCommunicationError, but no reopen/replay/Degraded transition ever runs, and the stale session persists until something subscribes. Failed unary RPCs (SubscribeBulk, Write, AcknowledgeAlarm) do not feed the supervisor either. Recommendation: report classified transport exceptions from the write/subscribe paths into the supervisor (idempotent by design, so this is cheap).

S-5 (Medium) — FasterLog outbox RemoveAsync truncates the FIFO prefix; out-of-order acks silently drop unacked entries

FasterLogHistorizationOutbox.RemoveAsync (FasterLogHistorizationOutbox.cs:158-182) removes the target plus every older live entry and truncates the log to the target's successor. The contract "recorder acks in FIFO order" is enforced only by comment. If the drain ever acks a mid-batch id first (e.g. per-tag partial write success in a future recorder change), all older unacked values are durably discarded without incrementing DroppedCount. Recommendation: count and warn when the prefix removal drops non-target entries, or make the API batch-oriented (RemoveThroughAsync) so the semantics are explicit at the call site.

S-6 (Medium) — Historian health snapshot's connection flags are dormant in production

GatewayHistorianDataSource.RefreshConnectionStateAsync (GatewayHistorianDataSource.cs:219-244) is documented as "intended to be driven by a periodic health hosted-service", but no production caller exists — only GatewayHealthSnapshotTests invokes it. ProcessConnectionOpen / EventConnectionOpen in GetHealthSnapshot are therefore permanently false in a deployed host, which any dashboard consuming the snapshot will read as "historian down" (or, if ignored, the flags are dead weight). Same shape as the memory's "register-AND-pass-into-consumer" trap that bit GatewayTagProvisioner in PR #423. Recommendation: add the periodic refresh hosted-service (or fold a refresh into the existing health probe cadence), or remove the flags from the snapshot.

S-7 (Low) — Alarm feed reconnect has no backoff

GatewayGalaxyAlarmFeed.RunAsync (GatewayGalaxyAlarmFeed.cs:102-148) re-opens on a fixed 5 s delay forever. DeployWatcher (capped exponential + jitter, DeployWatcher.cs:212-233) and ReconnectSupervisor both do this properly. A dead gateway gets hammered every 5 s per driver instance. Cosmetic at this scale, but inconsistent; align on capped exponential.

S-8 (Low) — GalaxyMxSession state is unsynchronized across reopen

_session / _connected (GalaxyMxSession.cs:28-32) are plain fields; RecreateAsync tears down while concurrent writers/subscribers may hold or fetch Session. In practice the supervisor's single-flight recovery plus the per-call try/catch (mapped to BadCommunicationError) contain it, and the caches are invalidated post-reopen — but the safety is emergent, not designed. Acceptable; document the invariant on Session.

S-9 (Positive) — Store-and-forward + outbox crash-safety discipline is strong

  • GatewayAlarmHistorianWriter (GatewayAlarmHistorianWriter.cs:66-118) short-circuits remaining batch entries to RetryPlease on shutdown, classifies auth failures as retryable (an auth blip never dead-letters), and defaults unknowns to PermanentFail so poison events cannot loop.
  • FasterLogHistorizationOutbox PerEntry mode fsyncs before append returns; recovery rebuilds the index from BeginAddress; the capacity-overflow-after-crash convergence case is analysed in-source (FasterLogHistorizationOutbox.cs:200-204).
  • Partial-open recovery in GalaxyMxSession.ConnectAsync (GalaxyMxSession.cs:95-101) tears down half-open state so retry rebuilds cleanly.

S-10 (Positive) — TLS / API-key posture

Both integrations support TLS with CA pinning (CaCertificatePath), the historian's AllowUntrustedServerCertificate inversion is explicitly documented at the mapping site (HistorianGatewayClientAdapter.cs:49-52), and keys are never logged. See C-1 for the resolver.

2. Performance

P-1 (Medium) — Galaxy Read costs three gateway round-trips plus a publish wait per OPC UA Read

ReadViaSubscribeOnceAsync = SubscribeBulk + first-event wait (up to PublishingIntervalMs) + UnsubscribeBulk, with server-side AddItem/RemoveItem churn per read (GalaxyDriver.cs:644-780). This is forced by MxAccess (no one-shot read RPC) and is correctly batched per request, but a client polling via Read instead of subscribing multiplies gateway/COM load ~3×. Recommendation: document "subscribe, don't poll" as the supported pattern; consider a short-lived read-through cache keyed on full reference if polling clients appear.

P-2 (Medium) — Alarm-history drain is one unary SendEvent per event

GatewayAlarmHistorianWriter.WriteBatchAsync (GatewayAlarmHistorianWriter.cs:66-80) serializes the batch — deliberate poison-event isolation, but an alarm storm of N events costs N sequential RPC round-trips out of the SQLite drain worker. If the gateway ever grows a batched SendEvents with per-event status rows, adopt it; until then this is an accepted, well-documented ceiling.

P-3 (Low) — Four channels to the historian sidecar; two-plus to mxaccessgw

Each historian path owns its channel (read / alarm-write / provisioner / value-writer) and the Galaxy driver holds a session client, a session-less client, and a repository client. All are lazy, HTTP/2-multiplexed, and the trade-off (clean independent dispose ownership over channel sharing) is argued in-source (GatewayHistorianServiceCollectionExtensions.cs:57-64). No action; noted so nobody "optimizes" it into a shared-singleton dispose bug.

P-4 (Positive) — Fan-in/fan-out paths are indexed, bounded, and metered

  • SubscriptionRegistry maintains a handle→subscriptions reverse map with a per-entry handle→fullRef index, so ResolveSubscribers is O(subscribers), not O(bindings) (SubscriptionRegistry.cs:97-111), and TryResolveItemHandle lets the writer skip AddItem for subscribed tags with a stale-entry cross-check (:128-142).
  • Subscribe/replay correlation uses a one-pass result index (GalaxyDriver.BuildResultIndex, fixing a former O(n²) on the 50k-tag path).
  • EventPump decouples network read from dispatch via a bounded channel (default 50 000, configurable) with received/dispatched/dropped counters.

P-5 (Low) — HistoryRead buffers whole replies; maxEvents <= 0 is unbounded

ReadRawAsync / ReadProcessedAsync / ReadEventsAsync (GatewayHistorianDataSource.cs:59-182) drain the stream into a list before mapping. Raw reads are capped by maxValuesPerNode, but an events read with maxEvents <= 0 collects without limit (the gateway's RuntimeDb:EventReadMaxRows is the only cap, and it's a remote deployment knob). The tie-cluster paging bound (MaxTieClusterOverfetch, validated > 0 in ServerHistorianOptions.Validate, enforced at OtOpcUaNodeManager.cs:2200-2214) is well-designed. Recommendation: clamp events reads to a server-side default cap when the caller passes 0.

P-6 (Low) — Outbox peek holds the state lock across disk I/O

PeekBatchAsync scans FasterLog from the logical head under _state (FasterLogHistorizationOutbox.cs:140-155), blocking a concurrent AppendAsync's index update for the duration of a (mostly page-cached) disk scan. At the recorder's 64-entry default batch this is negligible; revisit only if drain batches grow.

3. Conventions

C-1 (Positive) — Secret handling is a model for the other drivers

GalaxySecretRef (Galaxy.Contracts/GalaxySecretRef.cs) resolves env: / file: / dev: / literal with fail-fast on unset env vars and a startup warning on unprefixed cleartext; it lives in Contracts so the runtime driver and the AdminUI browser share one implementation (GalaxyDriverBrowser.cs:125). The historian side takes ServerHistorian__ApiKey via env with a "never commit" doc contract and an empty-key Validate() warning. No key value is ever logged on either path.

C-2 (Positive) — Seam quality and driver-family consistency

  • IHistorianGatewayClient is a clean single seam: every consumer (GatewayHistorianDataSource, GatewayAlarmHistorianWriter, GatewayTagProvisioner, GatewayHistorianValueWriter) depends only on it; FakeHistorianGatewayClient backs the offline suite; the adapter is a zero-translation pass-through.
  • The Galaxy capability seams (IGalaxyHierarchySource, IGalaxyDataReader/Writer, IGalaxySubscriber, IGalaxyAlarmFeed/Acknowledger) plus the internal test ctor mirror the protocol-driver pattern; tracing decorators (Traced*) wrap the production seams without an OpenTelemetry dependency.
  • GalaxyDriverFactoryExtensions.CreateInstance throws precise, instance-named errors on missing required fields (GalaxyDriverFactoryExtensions.cs:56-75), and GalaxyDriverProbe uses JsonStringEnumConverter (the FB-9/FB-10 enum-serialization lesson applied).

C-3 (Medium) — The historian seam leaks wire types by design; keep it contained

IHistorianGatewayClient signatures use HistorianGateway.Contracts.Grpc types (HistorianSample, RetrievalMode, WriteAck, ...) — documented as deliberate (IHistorianGatewayClient.cs:5-11). This is fine while the driver is the only consumer, but the pure Mapping/ layer is the real anti-corruption boundary: any future consumer must sit above GatewayHistorianDataSource, never on the seam. Worth a one-line guard note in the interface doc.

C-4 (Low) — Options-validation styles diverge

Galaxy: throw-on-construct for required fields; DataAnnotations attributes on the records are decorative (nothing runs Validator). Historian: Validate() returns operator warnings and never throws (registration logs them). Both are defensible, but the repo now has two idioms for "driver-adjacent options validation". Pick one for new sections (the warnings-list style is the friendlier operational contract).

C-5 (Low) — Retired Wonderware project directories still on disk

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware{,.Client,.Client.Contracts} (and their test dirs) remain in the tree but are absent from ZB.MOM.WW.OtOpcUa.slnx (0 references). Dead directories invite grep noise and accidental resurrection; delete them (git history preserves the code).

C-6 (Low) — EventPump silently drops unknown event families with no metric

Dispatch (EventPump.cs:192-207) returns without counting on any non-OnDataChange family. The rationale comment is good, but a gateway version emitting OnBufferedDataChange (or a new family) would silently discard data with zero observability — the alarm feed counts its decode drops (AlarmTransitionsDecodingFailures); the pump should count filtered/unknown families the same way.

4. Underdeveloped Areas

U-1 (High, documentation-drift) — CLAUDE.md KNOWN LIMITATION 2 is stale: the historized-ref feed is now wired

CLAUDE.md states the recorder is "spawned with an EMPTY historized-ref set … registers interest in nothing and historizes nothing" pending a SetHistorizedRefs-style feed. The code has closed that gap: WithOtOpcUaRuntimeActors still spawns with historizedRefs: Array.Empty<string>() (Runtime/ServiceCollectionExtensions.cs:272) but wraps the recorder in ActorHistorizedTagSubscriptionSink and hands it to the applier (:287-307), AddressSpaceApplier pushes the per-deploy add/remove delta (AddressSpaceApplier.cs:343), and the recorder handles UpdateHistorizedRefs (ContinuousHistorizationRecorder.cs:186) — the in-source comment explicitly says this "clos[es] the T18 ref-feed gap". Actions: (1) update CLAUDE.md; (2) the end-to-end value-capture path (deploy → delta → mux tap → outbox → WriteLiveValues) has, as far as the tree shows, never been live-verified — given this repo's repeated "wired-but-inert in prod" history (F10b DeferredAddressSpaceSink, PR #423 provisioner), a live /run against a real gateway is the required close-out.

U-2 (Medium) — KNOWN LIMITATION 1: the live-validation gate is built but only partially run

The Category=LiveIntegration suite is complete and skip-clean: GatewayLiveFixture env-gates on HISTGW_GATEWAY_ENDPOINT/APIKEY (+ per-test HISTGW_TEST_TAG, HISTGW_WRITE_SANDBOX_TAG, HISTGW_ALARM_SOURCE) with a bounded 3 s TCP probe so a down VPN skips instead of hanging (Live/GatewayLiveFixture.cs). Four live tests cover read, EnsureTags+write, alarm SendEvent→ReadEvents round-trip, and a SendEvent contract check. Recent history (merge 245316d8, "assert FU-1 alarm SendEvent→ReadEvents round-trip (gateway C4 fixed)") shows the alarm leg has been run live at least once, but CLAUDE.md still flags the whole cutover as unvalidated. A full documented run of the suite (and a CLAUDE.md status update) is the remaining work — the infrastructure is not the gap.

U-3 (Medium) — WriteSecured / VerifiedWrite user identity is stubbed at zero

InvokeWriteSecuredAsync hardcodes CurrentUserId = 0, VerifierUserId = 0 (GatewayGalaxyDataWriter.cs:263-264). ArchestrA secured/verified writes exist precisely to attribute and dual-authorize the operation; user 0 will be rejected or unattributed by any galaxy that actually classifies tags SecuredWrite/VerifiedWrite. There is no mapping from the OPC UA session identity (the LDAP-authenticated principal is available server-side) to an ArchestrA user id. Until that lands, writes to secured-classification tags are effectively unsupported — document it, or fail fast with a clear status instead of sending user 0.

U-4 (Medium) — Dormant paths inventory

  • GatewayHistorianDataSource.RefreshConnectionStateAsync — no production caller (see S-6).
  • Replay still fans out per-subscription SubscribeBulk; the gateway's batched ReplaySubscriptionsCommand remains a "PR 6.x can swap this" note (GalaxyDriver.cs:314-316).
  • GalaxyDriver.FlushOptionalCachesAsync is a no-op and GetMemoryFootprint is a constants-based estimate (GalaxyDriver.cs:561-575) — fine, but the server's cache-flush heuristic gets synthetic data from this driver.
  • HistorianGatewayClientAdapter.ReadEventsAsync never forwards maxEvents on the wire (client-side cap only, documented at IHistorianGatewayClient.cs:58-63) and the source filter is re-applied client-side defensively (GatewayHistorianDataSource.cs:152-157) because the gateway-side filter "may not be present" — both are polite workarounds for gateway gaps that should be tracked against the sidecar repo.

U-5 (Low) — Outbox serializer has no version/format byte

HistorizationOutboxEntrySerializer writes a fixed positional layout with no version prefix (HistorizationOutboxEntrySerializer.cs:14-16). Any future field addition breaks recovery of pre-existing on-disk entries with an undiagnosable deserialization failure mid-RecoverState. One reserved byte now is free; a migration later is not.

U-6 — Test coverage assessment

  • Galaxy (~250 test methods across 34 files): strong unit coverage of the hard parts — reconnect orchestration (GalaxyMxSessionReconnectTests, ReconnectSupervisorTests), pump fault/bounded-channel behaviour, registry rebind/handle-resolve, writer caches, alarm feed decode, value encode/decode, status mapping, probe, factory. Three skip-gated live smokes (GatewayGalaxyLiveReopenAndWriteTests — proven 2/2 against 10.100.0.48:5120 per project memory — and GatewayGalaxyAlarmFeedLiveTests) gated on MXGW_ENDPOINT + GALAXY_MXGW_API_KEY.
  • Historian.Gateway (~15 files): mappers fully covered, writer/provisioner/data-source outcome classification covered via FakeHistorianGatewayClient, outbox recovery/capacity covered, plus the 4-test live suite.
  • Thin spots: no test drives GalaxyDriver write-through the supervisory-advise failure branch asserting the returned status (S-1's blast radius is untested); no test exercises out-of-order RemoveAsync on the outbox (S-5); nothing covers the read-hang case in S-3; EventPump unknown-family filtering has no assertion (C-6).

Maturity Ratings

Dimension Rating (1-5) Justification
Stability 4 Layered recovery (supervisor, self-reconnecting feeds, backoff, partial-open teardown) and disciplined never-throw write sinks; docked for the Galaxy silent-write-loss blast radius (S-1), EventPump-only fault detection (S-4), and the dormant health refresh (S-6).
Performance 4 Bounded channels with drop metering, O(1) fan-out indexes, handle borrowing, and a bounded tie-cluster over-fetch; docked for the 3-round-trip read synthesis (P-1) and serial alarm sends (P-2), both documented and gateway-constrained.
Conventions 4 Exemplary seams, shared secret resolver, consistent driver-family shape and tracing decorators; docked for the dual options-validation idiom, on-disk retired Wonderware dirs, and the unmetered pump filter.
Underdeveloped areas 3 Live-validation infrastructure is complete but the gate is only partially run and CLAUDE.md's limitation 2 is stale against code that has closed the gap; WriteSecured identity is stubbed; several polite client-side workarounds paper over gateway gaps.

Cross-Cutting Themes

  1. "Wired but never invoked" is this codebase's recurring failure mode — the provisioner (PR #423), the F10b sink forwarding, and now RefreshConnectionStateAsync (S-6). Any new capability interface or hook needs an explicit production-caller check plus a live /run, not just unit tests.
  2. Optimistic-Good on fire-and-forget writes — Galaxy structurally cannot report a failed commit (S-1); reviewers of the node-write router / write-outcome self-correction should not assume driver parity here.
  3. CLAUDE.md drift — KNOWN LIMITATION 2 is closed in code (U-1) and LIMITATION 1 is partially exercised (U-2); the doc should be reconciled before it misleads the next session.
  4. Gateway-gap workarounds accrue client-side — defensive source filters, client-side event caps, missing batched replay/SendEvents; these belong on the sister repos' backlogs with links from the in-source comments.