diff --git a/docs/AlarmHistorian.md b/docs/AlarmHistorian.md new file mode 100644 index 00000000..d437a2f4 --- /dev/null +++ b/docs/AlarmHistorian.md @@ -0,0 +1,168 @@ +# Alarm Historian — store-and-forward SQLite sink + +Reference for `ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian` +([`src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/)), +the durable local queue that historizes alarm transitions to AVEVA Historian +without ever blocking the alarm engine or operator actions. + +This is the *sink mechanics* doc. For how the three alarm sources converge on +the OPC UA Part 9 surface and which alarms route here, see +[AlarmTracking.md](AlarmTracking.md). For the historian client that drains this +queue, see [DriverLifecycle.md](DriverLifecycle.md#ihistoriandatasource--server-side-historian-read-surface) +and [ServiceHosting.md](ServiceHosting.md). + +--- + +## Why store-and-forward + +Scripted alarms (and any future non-Galaxy `IAlarmSource`, e.g. AB CIP ALMD) +must reach AVEVA Historian, but the historian sidecar can be slow, busy, or +disconnected. The sink decouples the alarm engine from historian reachability: +every qualifying transition is committed to a **local SQLite queue first**, and +a background drain worker forwards rows to the historian on a backoff-aware +cadence. Operator acks and alarm-state transitions are never blocked waiting on +the historian. + +> Galaxy-native alarms with `$Alarm*` extensions reach AVEVA Historian directly +> via System Platform's `HistorizeToAveva` toggle — they do **not** flow through +> this sink. This path is exclusively for non-Galaxy alarm producers. + +--- + +## Contracts + +All in +[`IAlarmHistorianSink.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/IAlarmHistorianSink.cs) +unless noted. + +- **`IAlarmHistorianSink`** — the intake contract. `EnqueueAsync(evt, ct)` + durably enqueues an event and returns as soon as the queue row is committed + (fire-and-forget from the engine's perspective; the sink must not block the + emitting thread). `GetStatus()` returns a `HistorianSinkStatus` snapshot. +- **`NullAlarmHistorianSink`** — the no-op default for tests and deployments + that don't historize alarms. It is the default DI binding (registered in the + Runtime's `AddOtOpcUaRuntime`); production overrides it with + `SqliteStoreAndForwardSink`. +- **`AlarmHistorianEvent`** + ([`AlarmHistorianEvent.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmHistorianEvent.cs)) + — the source-agnostic event record: `AlarmId`, `EquipmentPath` (UNS path, + doubles as Historian's SourceNode), `AlarmName`, `AlarmTypeName` (Part 9 + subtype), `Severity`, `EventKind` (free-form transition string — + "Activated"/"Cleared"/"Acknowledged"/etc.), `Message`, `User`, `Comment`, + `TimestampUtc`. +- **`IAlarmHistorianWriter`** — what the drain worker delegates writes to. + `WriteBatchAsync(batch, ct)` returns one `HistorianWriteOutcome` per event, + in order. Production binds this to `WonderwareHistorianClient` (the AVEVA + Historian sidecar IPC client). +- **`HistorianWriteOutcome`** — per-event drain result: `Ack` (persisted, + remove from queue), `RetryPlease` (transient failure — leave queued, retry + after backoff), `PermanentFail` (malformed/unrecoverable — move to + dead-letter). +- **`HistorianSinkStatus`** — diagnostic snapshot surfaced to the AdminUI and + `/healthz`: `QueueDepth`, `DeadLetterDepth`, `LastDrainUtc`, `LastSuccessUtc`, + `LastError`, `DrainState`, and `EvictedCount`. +- **`HistorianDrainState`** — `Disabled` / `Idle` / `Draining` / `BackingOff`. + +--- + +## SqliteStoreAndForwardSink + +[`SqliteStoreAndForwardSink.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs) +is the production `IAlarmHistorianSink`. Construction takes a SQLite database +path, an `IAlarmHistorianWriter`, a logger, and optional `batchSize` (default +100), `capacity` (default 1,000,000), `deadLetterRetention` (default 30 days), +and a test clock. + +### Queue table + +The sink owns one SQLite table (created on construction, WAL journal mode): + +```sql +CREATE TABLE Queue ( + RowId INTEGER PRIMARY KEY AUTOINCREMENT, + AlarmId TEXT NOT NULL, + EnqueuedUtc TEXT NOT NULL, + PayloadJson TEXT NOT NULL, -- JSON-serialized AlarmHistorianEvent + AttemptCount INTEGER NOT NULL DEFAULT 0, + LastAttemptUtc TEXT NULL, + LastError TEXT NULL, + DeadLettered INTEGER NOT NULL DEFAULT 0 +); +CREATE INDEX IX_Queue_Drain ON Queue (DeadLettered, RowId); +``` + +`EnqueueAsync` does a single `INSERT` on the hot path. To avoid a +`SELECT COUNT(*)` on every enqueue, the sink keeps an in-memory non-dead-lettered +row counter (seeded at startup, kept current by every mutation, and re-synced +from storage every 10,000 enqueues to defend against drift). SQLite writer +contention is handled via `PRAGMA busy_timeout=5000` + WAL so an enqueue/drain +collision waits out the file lock instead of failing fast. + +### Drain worker + +`StartDrainLoop(tickInterval)` starts a **self-rescheduling one-shot +`System.Threading.Timer`** (not started automatically — tests drive +`DrainOnceAsync` deterministically). Each tick: + +1. Purges aged dead-lettered rows past the retention window. +2. Reads up to `batchSize` non-dead-lettered rows in `RowId` order. +3. Rows with un-deserializable payloads are dead-lettered immediately (by their + own `RowId`) so they can't stall the queue head. +4. The remaining batch is handed to `IAlarmHistorianWriter.WriteBatchAsync`, and + each outcome is applied in one transaction: `Ack` deletes the row, + `PermanentFail` flips its `DeadLettered` flag, `RetryPlease` bumps its attempt + count and leaves it queued. +5. The timer re-arms its next due-time to `max(tickInterval, currentBackoff)`. + +**Backoff ladder** (applied to the timer's next due-time, so a historian outage +genuinely slows the drain cadence): 1s → 2s → 5s → 15s → 60s cap. Any +`RetryPlease` outcome — or a writer exception, or a writer cardinality violation +(outcome count ≠ event count) — bumps the backoff and sets `DrainState = +BackingOff`; a clean batch resets it. The async-void timer callback is fully +guarded: a fault is logged and recorded into `GetStatus()` rather than lost as +an unobserved task exception. + +### Durability bound (important) + +**The durability guarantee is bounded by `capacity` (default 1,000,000 rows).** +When the non-dead-lettered queue reaches capacity, `EnqueueAsync` evicts the +oldest non-dead-lettered rows (oldest `RowId` first) to make room, logs a WARN, +and increments `HistorianSinkStatus.EvictedCount`. Under a sustained historian +outage, accepted alarm events can therefore be dropped before delivery. A +non-zero `EvictedCount` is a data-loss signal that requires operator attention — +it surfaces silent loss without log scraping. + +### Dead-letter + operator recovery + +`PermanentFail` and corrupt-payload rows are retained in-place with +`DeadLettered = 1` for the retention window (default 30 days) so operators can +inspect them before the sweeper purges them. `RetryDeadLettered()` is the +operator action (from the AdminUI) that clears the dead-letter flag and attempt +count on every dead-lettered row, returning them to the regular queue with a +fresh backoff. + +--- + +## Runtime wiring + +Production routes alarm transitions through the Akka cluster. The +`HistorianAdapterActor` +([`Runtime/Historian/HistorianAdapterActor.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/HistorianAdapterActor.cs)) +bridges messages from the scripted-alarm actor into the sink's `EnqueueAsync`, +fire-and-forget so the actor loop is never blocked on historian reachability. +The `WonderwareHistorianClient` is the `IAlarmHistorianWriter` the drain worker +delegates to. See [ServiceHosting.md](ServiceHosting.md) for the sidecar setup. + +--- + +## See also + +- [AlarmTracking.md](AlarmTracking.md) — the three alarm sources and the OPC UA + Part 9 surface; which alarms route to this sink. +- [DriverLifecycle.md](DriverLifecycle.md) — `IHistorianDataSource` (the + historian *read* surface; this page covers the *write* path) and the + `WonderwareHistorianClient`. +- [ScriptedAlarms.md](ScriptedAlarms.md) — the scripted-alarm engine that emits + most events into this sink. +- [ServiceHosting.md](ServiceHosting.md) — the optional Wonderware historian + sidecar. diff --git a/docs/DriverLifecycle.md b/docs/DriverLifecycle.md new file mode 100644 index 00000000..e07770b5 --- /dev/null +++ b/docs/DriverLifecycle.md @@ -0,0 +1,295 @@ +# Driver Lifecycle & Server Infrastructure Contracts + +Reference for the server-side infrastructure interfaces that surround a +driver but are **not** driver *capabilities* (read/write/subscribe/etc., +documented in [ReadWriteOperations.md](ReadWriteOperations.md) and the +per-driver pages). These contracts live in +[`src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/) +so they carry no behavior — concrete implementations live in the driver +projects, the Runtime, and the ControlPlane. Each subsection below gives the +purpose, the key members, and where it is implemented/used. + +The capability interfaces a driver opts into (`IReadable`, `IWritable`, +`ITagDiscovery`, `ISubscribable`, `IAlarmSource`, `IHistoryProvider`, +`IHostConnectivityProbe`, `IPerCallHostResolver`, `IRediscoverable`) are +covered elsewhere and discovered by the server via `is`-checks on the +`IDriver` instance. The interfaces here are the *plumbing* the server uses to +**create**, **probe**, **supervise**, **report on**, and **configure** those +drivers, plus the server-side historian read surface. + +--- + +## IDriverFactory — creating drivers from config rows + +[`Core.Abstractions/IDriverFactory.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriverFactory.cs) + +Abstraction over the process-wide driver registry. The Runtime consumes this +instead of the concrete registry so the Runtime project does not pull in +`ZB.MOM.WW.OtOpcUa.Core` (which would drag in Polly + driver hosting). + +Members: + +- `IDriver? TryCreate(string driverType, string driverInstanceId, string driverConfigJson)` + — returns a new driver for the given type, or `null` when no factory is + registered for that type (missing assembly, typo). The `DriverHostActor` + logs and skips the row rather than failing the whole apply. +- `IReadOnlyCollection SupportedTypes` — driver-type names this + factory can materialise; mostly for diagnostics and logs. + +Implementations: + +- `NullDriverFactory` (same file) returns `null` from every `TryCreate` and + exposes zero supported types. Bound when no concrete driver assemblies have + been registered (Mac dev path, smoke tests); the deployment becomes a no-op. +- `DriverFactoryRegistry` + ([`Core/Hosting/DriverFactoryRegistry.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core/Hosting/DriverFactoryRegistry.cs)) + is the real process-singleton registry keyed by `DriverInstance.DriverType` + (case-insensitive). Each driver project ships a `Register(...)` extension; + `Register` records the factory **and** the driver's stability + [`DriverTier`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/DriverTier.cs) + (defaults to Tier A). Registering the same type twice throws. +- `DriverFactoryRegistryAdapter` + ([`Core/Hosting/DriverFactoryRegistryAdapter.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core/Hosting/DriverFactoryRegistryAdapter.cs)) + bridges the registry to the `IDriverFactory` abstraction. + +Wiring: `DriverFactoryBootstrap.AddOtOpcUaDriverFactories` +([`Host/Drivers/DriverFactoryBootstrap.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs)) +registers the singleton registry, runs every driver assembly's `Register` +extension, then binds `IDriverFactory` to the adapter. It must run **before** +`AddAkka` so the Runtime can resolve `IDriverFactory` when spawning the +`DriverHostActor` +([`Runtime/Drivers/DriverHostActor.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs)). +The registry is skipped on admin-only nodes (they never run drivers); the +probe set is the exception — see [IDriverProbe](#idriverprobe--test-connect). + +--- + +## IDriverProbe — Test Connect + +[`Core.Abstractions/IDriverProbe.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriverProbe.cs) + +A cheap test-connect probe for one driver type, backing the AdminUI **Test +Connect** button. An implementation deserializes a driver-config JSON, attempts +a cheap connection (TCP open, OPC UA session, gRPC ping — whatever the driver's +native protocol supports), and reports success/failure with latency. **Probes +must not mutate persistent state**: the AdminUI invokes them against the +transient config in the typed form, not against the persisted `DriverInstance` +row. + +Members: + +- `string DriverType { get; }` — the `DriverInstance.DriverType` string this + probe handles; used for DI lookup. +- `Task ProbeAsync(string configJson, TimeSpan timeout, CancellationToken ct)` + — never throws on connection failure; returns a result with `Ok = false` + and a message instead. +- `DriverProbeResult(bool Ok, string? Message, TimeSpan? Latency)` — outcome + record (`Message` is `null` on success; `Latency` is `null` on failure). + +Implementations: every driver ships a `*DriverProbe` in its driver project +(e.g. +[`Driver.Modbus/ModbusDriverProbe.cs`](../src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusDriverProbe.cs) +does a bare socket open/close), plus the Wonderware historian's +`WonderwareHistorianDriverProbe`. + +Flow: the AdminUI's `AdminProbeService` +([`AdminUI/Clients/AdminProbeService.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Clients/AdminProbeService.cs)) +dispatches a `TestDriverConnect` message through `IAdminOperationsClient` to the +cluster-singleton `AdminOperationsActor` +([`ControlPlane/AdminOperations/AdminOperationsActor.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/AdminOperationsActor.cs)), +which holds the probes keyed by `DriverType` and invokes the matching one +(timeout clamped to `[1, 60]` seconds). Because the admin singleton is +admin-pinned, the probe set must be registered on admin nodes too — `Program.cs` +calls `AddOtOpcUaDriverProbes` in the `hasAdmin` block, and +`AddOtOpcUaDriverFactories` registers it for fused admin+driver nodes. + +--- + +## IDriverSupervisor — Tier C out-of-process recycle + +[`Core.Abstractions/IDriverSupervisor.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriverSupervisor.cs) + +The process-level supervisor contract a **Tier C** (out-of-process) driver's +topology provides. Its concern is restarting the out-of-process Host when a +hard fault is detected (memory breach, wedge, scheduled recycle window). Tier +A/B drivers run in-process and do **not** have a supervisor — recycling them +would kill every OPC UA session and every co-hosted driver. The Core.Stability +layer only invokes this interface after asserting the tier. + +Members: + +- `string DriverInstanceId { get; }` — the driver instance this supervisor + governs. +- `Task RecycleAsync(string reason, CancellationToken cancellationToken)` — + request a terminate+restart of the Host process; implementations are + expected to be idempotent under repeat calls during an in-flight recycle. + +Callers (both in +[`Core/Stability/`](../src/Core/ZB.MOM.WW.OtOpcUa.Core/Stability/)): + +- `ScheduledRecycleScheduler` + ([`Core/Stability/ScheduledRecycleScheduler.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core/Stability/ScheduledRecycleScheduler.cs)) + — opt-in periodic recycle. A `TickAsync` method advanced by the caller's + ambient scheduler decides whether the configured interval has elapsed and, if + so, drives `RecycleAsync`. Its constructor throws unless the tier is C, making + in-process misuse structurally impossible. +- `MemoryRecycle` + ([`Core/Stability/MemoryRecycle.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core/Stability/MemoryRecycle.cs)) + — on a memory hard-breach, calls `RecycleAsync` (when a supervisor is wired). + +--- + +## IDriverHealthPublisher — health pub/sub sink + +[`Core.Abstractions/IDriverHealthPublisher.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriverHealthPublisher.cs) + +A sink for driver-health state-change notifications. Implementations must be +non-blocking and safe to call from any thread. + +Member: + +- `void Publish(string clusterId, string driverInstanceId, DriverHealth health, int errorCount5Min)` + +Implementations: + +- `NullDriverHealthPublisher` (same file) is the drop-in no-op for tests and + dev-stub paths. A `DriverInstanceActor` defaults to it when no publisher is + supplied. +- `AkkaDriverHealthPublisher` + ([`Runtime/Drivers/AkkaDriverHealthPublisher.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/AkkaDriverHealthPublisher.cs)) + is the production binding: it forwards each transition as a + `DriverHealthChanged` message onto the cluster-wide `driver-health` + Akka DistributedPubSub topic. + +Producer: `DriverInstanceActor` +([`Runtime/Drivers/DriverInstanceActor.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs)) +calls `Publish` when a driver's health transitions. The published snapshot is +consumed AdminUI-side and surfaced through the driver-status panel (read +in-process by the AdminUI bridge rather than dialing its own hub). + +--- + +## IDriverConfigEditor — custom AdminUI config editor (plug-point) + +[`Core.Abstractions/IDriverConfigEditor.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriverConfigEditor.cs) + +An **optional** plug-point a driver can implement to provide a custom AdminUI +editor for its `DriverConfig` JSON. Drivers that don't implement it fall back to +the generic JSON editor with schema-driven validation. This is the contract +between the driver and the Admin Blazor app; the Admin app discovers +implementations and slots them into the Driver Detail screen. + +Members: + +- `string DriverType { get; }` — the driver type this editor handles. +- `Type EditorComponentType { get; }` — the Razor component type that renders + the editor (returned as `Type` so `Core.Abstractions` needs no Blazor + reference). + +Status: this is a forward-looking plug-point. No driver ships a concrete +`IDriverConfigEditor` today — every driver uses the generic JSON editor — so +the interface currently has the contract defined but no implementations. + +--- + +## IHistorianDataSource — server-side historian read surface + +[`Core.Abstractions/Historian/IHistorianDataSource.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorianDataSource.cs) + +The server-side historian read surface. Registered with the server's history +router and resolved **per OPC UA namespace**, independent of any driver's +lifecycle. This is distinct from the driver capability `IHistoryProvider`: + +- `IHistoryProvider` is a *driver capability* — the server dispatches to it via + the driver instance. +- `IHistorianDataSource` is a *server registration* — the server resolves it by + namespace and calls it directly, so one historian (e.g. Wonderware) can serve + many drivers' nodes, and drivers can restart without dropping history + availability. + +The interface is `: IDisposable` and declares the full read surface as +**required** members (unlike `IHistoryProvider`, where at-time/event reads are +optional default-impl methods so legacy drivers can stay raw-only): + +- `ReadRawAsync(fullReference, startUtc, endUtc, maxValuesPerNode, ct)` — raw + historical samples over a time range. +- `ReadProcessedAsync(fullReference, startUtc, endUtc, interval, aggregate, ct)` + — interval-bucketed aggregates (average/min/max/count); an empty bucket + returns a `BadNoData` sample. +- `ReadAtTimeAsync(fullReference, timestampsUtc, ct)` — one sample per requested + timestamp (OPC UA HistoryReadAtTime); the returned list matches the requested + length and order, gaps as Bad-quality snapshots. +- `ReadEventsAsync(sourceName, startUtc, endUtc, maxEvents, ct)` — historical + alarm/event records (OPC UA HistoryReadEvents); `sourceName` is `null` to + return all sources. `maxEvents` is a signed `int` so a non-positive value is a + "use the backend's default cap" sentinel. +- `GetHealthSnapshot()` — point-in-time health snapshot for diagnostics and + dashboards; pure observation, never blocks on backend I/O. + +All values use the shared `DataValueSnapshot` / `HistoricalEvent` shapes; +backend-specific quality/type encodings are translated to OPC UA `StatusCode` +uints inside the data source. + +Implementations: + +- `WonderwareHistorianClient` + ([`Driver.Historian.Wonderware.Client/WonderwareHistorianClient.cs`](../src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware.Client/WonderwareHistorianClient.cs)) + — the .NET 10 client that talks to the Wonderware historian sidecar over a + named pipe. It implements both `IHistorianDataSource` (read paths) and + `IAlarmHistorianWriter` (the alarm-event drain target; see + [AlarmHistorian.md](AlarmHistorian.md)). +- `HistorianDataSource` + ([`Driver.Historian.Wonderware/Backend/HistorianDataSource.cs`](../src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware/Backend/HistorianDataSource.cs)) + — the in-process backend implementation behind the sidecar. + +The optional Wonderware historian sidecar setup is described in +[ServiceHosting.md](ServiceHosting.md). + +--- + +## Commons — shared cross-cutting primitives + +[`src/Core/ZB.MOM.WW.OtOpcUa.Commons/`](../src/Core/ZB.MOM.WW.OtOpcUa.Commons/) + +`ZB.MOM.WW.OtOpcUa.Commons` is the low-level shared library that the Runtime, +ControlPlane, AdminUI, and OPC UA server projects all reference. It holds +cross-cutting primitives with no driver- or host-specific behavior, so the +heavier projects can share message contracts and value types without taking a +dependency on each other. It references only `Akka` and the internal +`ZB.MOM.WW.Audit` package. + +Folders: + +- **`Messages/`** — Akka message contracts grouped by concern (`Admin`, + `Alerts`, `Deploy`, `Drivers`, `Fleet`, `Logging`, `Redundancy`). These are + the wire/inter-actor messages — e.g. `Messages/Admin/TestDriverConnect.cs` + (Test Connect request, see [IDriverProbe](#idriverprobe--test-connect)) and + `Messages/Drivers/DriverHealthChanged.cs` (the driver-health pub/sub payload, + see [IDriverHealthPublisher](#idriverhealthpublisher--health-pubsub-sink)). +- **`Interfaces/`** — cluster-facing client contracts such as + `IAdminOperationsClient`, `IClusterRoleInfo`, and `IFleetDiagnosticsClient`. +- **`Types/`** — strongly-typed identifier value types: `CorrelationId`, + `DeploymentId`, `ExecutionId`, `NodeId`, `RevisionHash`. +- **`Browsing/`** — live-browse abstractions (`BrowseNode`, `IBrowseSession`, + `IDriverBrowser`) backing the AdminUI address pickers. +- **`Engines/`** — evaluator seams (`IScriptedAlarmEvaluator`, + `IVirtualTagEvaluator`, `IAlarmActorStateStore`) consumed by the + [VirtualTags](VirtualTags.md) / [ScriptedAlarms](ScriptedAlarms.md) engines. +- **`OpcUa/`** — deferred-publish seams (`IOpcUaAddressSpaceSink`, + `IServiceLevelPublisher` and their `Deferred*` no-op stand-ins) so address-space + and [ServiceLevel](Redundancy.md) writes can be wired late. +- **`Observability/`** — `OtOpcUaTelemetry` (the shared ActivitySource/metrics + surface). + +--- + +## See also + +- [ReadWriteOperations.md](ReadWriteOperations.md) — the driver *capability* + interfaces (read/write/subscribe) and resilience pipeline. +- [ServiceHosting.md](ServiceHosting.md) — role gating, the Akka cluster, and + the optional Wonderware historian sidecar. +- [AlarmHistorian.md](AlarmHistorian.md) — the store-and-forward SQLite alarm + sink that drains to `IAlarmHistorianWriter`. +- [Redundancy.md](Redundancy.md) — driver stability tiers in the redundancy + context.