docs(audit): G1 completeness — driver-lifecycle + alarm-historian reference pages

This commit is contained in:
Joseph Doherty
2026-06-03 15:55:28 -04:00
parent 0a679f2c2a
commit 33912694fd
2 changed files with 463 additions and 0 deletions
+168
View File
@@ -0,0 +1,168 @@
# Alarm Historian — store-and-forward SQLite sink
Reference for `ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian`
([`src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/)),
the durable local queue that historizes alarm transitions to AVEVA Historian
without ever blocking the alarm engine or operator actions.
This is the *sink mechanics* doc. For how the three alarm sources converge on
the OPC UA Part 9 surface and which alarms route here, see
[AlarmTracking.md](AlarmTracking.md). For the historian client that drains this
queue, see [DriverLifecycle.md](DriverLifecycle.md#ihistoriandatasource--server-side-historian-read-surface)
and [ServiceHosting.md](ServiceHosting.md).
---
## Why store-and-forward
Scripted alarms (and any future non-Galaxy `IAlarmSource`, e.g. AB CIP ALMD)
must reach AVEVA Historian, but the historian sidecar can be slow, busy, or
disconnected. The sink decouples the alarm engine from historian reachability:
every qualifying transition is committed to a **local SQLite queue first**, and
a background drain worker forwards rows to the historian on a backoff-aware
cadence. Operator acks and alarm-state transitions are never blocked waiting on
the historian.
> Galaxy-native alarms with `$Alarm*` extensions reach AVEVA Historian directly
> via System Platform's `HistorizeToAveva` toggle — they do **not** flow through
> this sink. This path is exclusively for non-Galaxy alarm producers.
---
## Contracts
All in
[`IAlarmHistorianSink.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/IAlarmHistorianSink.cs)
unless noted.
- **`IAlarmHistorianSink`** — the intake contract. `EnqueueAsync(evt, ct)`
durably enqueues an event and returns as soon as the queue row is committed
(fire-and-forget from the engine's perspective; the sink must not block the
emitting thread). `GetStatus()` returns a `HistorianSinkStatus` snapshot.
- **`NullAlarmHistorianSink`** — the no-op default for tests and deployments
that don't historize alarms. It is the default DI binding (registered in the
Runtime's `AddOtOpcUaRuntime`); production overrides it with
`SqliteStoreAndForwardSink`.
- **`AlarmHistorianEvent`**
([`AlarmHistorianEvent.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmHistorianEvent.cs))
— the source-agnostic event record: `AlarmId`, `EquipmentPath` (UNS path,
doubles as Historian's SourceNode), `AlarmName`, `AlarmTypeName` (Part 9
subtype), `Severity`, `EventKind` (free-form transition string —
"Activated"/"Cleared"/"Acknowledged"/etc.), `Message`, `User`, `Comment`,
`TimestampUtc`.
- **`IAlarmHistorianWriter`** — what the drain worker delegates writes to.
`WriteBatchAsync(batch, ct)` returns one `HistorianWriteOutcome` per event,
in order. Production binds this to `WonderwareHistorianClient` (the AVEVA
Historian sidecar IPC client).
- **`HistorianWriteOutcome`** — per-event drain result: `Ack` (persisted,
remove from queue), `RetryPlease` (transient failure — leave queued, retry
after backoff), `PermanentFail` (malformed/unrecoverable — move to
dead-letter).
- **`HistorianSinkStatus`** — diagnostic snapshot surfaced to the AdminUI and
`/healthz`: `QueueDepth`, `DeadLetterDepth`, `LastDrainUtc`, `LastSuccessUtc`,
`LastError`, `DrainState`, and `EvictedCount`.
- **`HistorianDrainState`** — `Disabled` / `Idle` / `Draining` / `BackingOff`.
---
## SqliteStoreAndForwardSink
[`SqliteStoreAndForwardSink.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs)
is the production `IAlarmHistorianSink`. Construction takes a SQLite database
path, an `IAlarmHistorianWriter`, a logger, and optional `batchSize` (default
100), `capacity` (default 1,000,000), `deadLetterRetention` (default 30 days),
and a test clock.
### Queue table
The sink owns one SQLite table (created on construction, WAL journal mode):
```sql
CREATE TABLE Queue (
RowId INTEGER PRIMARY KEY AUTOINCREMENT,
AlarmId TEXT NOT NULL,
EnqueuedUtc TEXT NOT NULL,
PayloadJson TEXT NOT NULL, -- JSON-serialized AlarmHistorianEvent
AttemptCount INTEGER NOT NULL DEFAULT 0,
LastAttemptUtc TEXT NULL,
LastError TEXT NULL,
DeadLettered INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX IX_Queue_Drain ON Queue (DeadLettered, RowId);
```
`EnqueueAsync` does a single `INSERT` on the hot path. To avoid a
`SELECT COUNT(*)` on every enqueue, the sink keeps an in-memory non-dead-lettered
row counter (seeded at startup, kept current by every mutation, and re-synced
from storage every 10,000 enqueues to defend against drift). SQLite writer
contention is handled via `PRAGMA busy_timeout=5000` + WAL so an enqueue/drain
collision waits out the file lock instead of failing fast.
### Drain worker
`StartDrainLoop(tickInterval)` starts a **self-rescheduling one-shot
`System.Threading.Timer`** (not started automatically — tests drive
`DrainOnceAsync` deterministically). Each tick:
1. Purges aged dead-lettered rows past the retention window.
2. Reads up to `batchSize` non-dead-lettered rows in `RowId` order.
3. Rows with un-deserializable payloads are dead-lettered immediately (by their
own `RowId`) so they can't stall the queue head.
4. The remaining batch is handed to `IAlarmHistorianWriter.WriteBatchAsync`, and
each outcome is applied in one transaction: `Ack` deletes the row,
`PermanentFail` flips its `DeadLettered` flag, `RetryPlease` bumps its attempt
count and leaves it queued.
5. The timer re-arms its next due-time to `max(tickInterval, currentBackoff)`.
**Backoff ladder** (applied to the timer's next due-time, so a historian outage
genuinely slows the drain cadence): 1s → 2s → 5s → 15s → 60s cap. Any
`RetryPlease` outcome — or a writer exception, or a writer cardinality violation
(outcome count ≠ event count) — bumps the backoff and sets `DrainState =
BackingOff`; a clean batch resets it. The async-void timer callback is fully
guarded: a fault is logged and recorded into `GetStatus()` rather than lost as
an unobserved task exception.
### Durability bound (important)
**The durability guarantee is bounded by `capacity` (default 1,000,000 rows).**
When the non-dead-lettered queue reaches capacity, `EnqueueAsync` evicts the
oldest non-dead-lettered rows (oldest `RowId` first) to make room, logs a WARN,
and increments `HistorianSinkStatus.EvictedCount`. Under a sustained historian
outage, accepted alarm events can therefore be dropped before delivery. A
non-zero `EvictedCount` is a data-loss signal that requires operator attention —
it surfaces silent loss without log scraping.
### Dead-letter + operator recovery
`PermanentFail` and corrupt-payload rows are retained in-place with
`DeadLettered = 1` for the retention window (default 30 days) so operators can
inspect them before the sweeper purges them. `RetryDeadLettered()` is the
operator action (from the AdminUI) that clears the dead-letter flag and attempt
count on every dead-lettered row, returning them to the regular queue with a
fresh backoff.
---
## Runtime wiring
Production routes alarm transitions through the Akka cluster. The
`HistorianAdapterActor`
([`Runtime/Historian/HistorianAdapterActor.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/HistorianAdapterActor.cs))
bridges messages from the scripted-alarm actor into the sink's `EnqueueAsync`,
fire-and-forget so the actor loop is never blocked on historian reachability.
The `WonderwareHistorianClient` is the `IAlarmHistorianWriter` the drain worker
delegates to. See [ServiceHosting.md](ServiceHosting.md) for the sidecar setup.
---
## See also
- [AlarmTracking.md](AlarmTracking.md) — the three alarm sources and the OPC UA
Part 9 surface; which alarms route to this sink.
- [DriverLifecycle.md](DriverLifecycle.md) — `IHistorianDataSource` (the
historian *read* surface; this page covers the *write* path) and the
`WonderwareHistorianClient`.
- [ScriptedAlarms.md](ScriptedAlarms.md) — the scripted-alarm engine that emits
most events into this sink.
- [ServiceHosting.md](ServiceHosting.md) — the optional Wonderware historian
sidecar.
+295
View File
@@ -0,0 +1,295 @@
# Driver Lifecycle & Server Infrastructure Contracts
Reference for the server-side infrastructure interfaces that surround a
driver but are **not** driver *capabilities* (read/write/subscribe/etc.,
documented in [ReadWriteOperations.md](ReadWriteOperations.md) and the
per-driver pages). These contracts live in
[`src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/)
so they carry no behavior — concrete implementations live in the driver
projects, the Runtime, and the ControlPlane. Each subsection below gives the
purpose, the key members, and where it is implemented/used.
The capability interfaces a driver opts into (`IReadable`, `IWritable`,
`ITagDiscovery`, `ISubscribable`, `IAlarmSource`, `IHistoryProvider`,
`IHostConnectivityProbe`, `IPerCallHostResolver`, `IRediscoverable`) are
covered elsewhere and discovered by the server via `is`-checks on the
`IDriver` instance. The interfaces here are the *plumbing* the server uses to
**create**, **probe**, **supervise**, **report on**, and **configure** those
drivers, plus the server-side historian read surface.
---
## IDriverFactory — creating drivers from config rows
[`Core.Abstractions/IDriverFactory.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriverFactory.cs)
Abstraction over the process-wide driver registry. The Runtime consumes this
instead of the concrete registry so the Runtime project does not pull in
`ZB.MOM.WW.OtOpcUa.Core` (which would drag in Polly + driver hosting).
Members:
- `IDriver? TryCreate(string driverType, string driverInstanceId, string driverConfigJson)`
— returns a new driver for the given type, or `null` when no factory is
registered for that type (missing assembly, typo). The `DriverHostActor`
logs and skips the row rather than failing the whole apply.
- `IReadOnlyCollection<string> SupportedTypes` — driver-type names this
factory can materialise; mostly for diagnostics and logs.
Implementations:
- `NullDriverFactory` (same file) returns `null` from every `TryCreate` and
exposes zero supported types. Bound when no concrete driver assemblies have
been registered (Mac dev path, smoke tests); the deployment becomes a no-op.
- `DriverFactoryRegistry`
([`Core/Hosting/DriverFactoryRegistry.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core/Hosting/DriverFactoryRegistry.cs))
is the real process-singleton registry keyed by `DriverInstance.DriverType`
(case-insensitive). Each driver project ships a `Register(...)` extension;
`Register` records the factory **and** the driver's stability
[`DriverTier`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/DriverTier.cs)
(defaults to Tier A). Registering the same type twice throws.
- `DriverFactoryRegistryAdapter`
([`Core/Hosting/DriverFactoryRegistryAdapter.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core/Hosting/DriverFactoryRegistryAdapter.cs))
bridges the registry to the `IDriverFactory` abstraction.
Wiring: `DriverFactoryBootstrap.AddOtOpcUaDriverFactories`
([`Host/Drivers/DriverFactoryBootstrap.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs))
registers the singleton registry, runs every driver assembly's `Register`
extension, then binds `IDriverFactory` to the adapter. It must run **before**
`AddAkka` so the Runtime can resolve `IDriverFactory` when spawning the
`DriverHostActor`
([`Runtime/Drivers/DriverHostActor.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs)).
The registry is skipped on admin-only nodes (they never run drivers); the
probe set is the exception — see [IDriverProbe](#idriverprobe--test-connect).
---
## IDriverProbe — Test Connect
[`Core.Abstractions/IDriverProbe.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriverProbe.cs)
A cheap test-connect probe for one driver type, backing the AdminUI **Test
Connect** button. An implementation deserializes a driver-config JSON, attempts
a cheap connection (TCP open, OPC UA session, gRPC ping — whatever the driver's
native protocol supports), and reports success/failure with latency. **Probes
must not mutate persistent state**: the AdminUI invokes them against the
transient config in the typed form, not against the persisted `DriverInstance`
row.
Members:
- `string DriverType { get; }` — the `DriverInstance.DriverType` string this
probe handles; used for DI lookup.
- `Task<DriverProbeResult> ProbeAsync(string configJson, TimeSpan timeout, CancellationToken ct)`
— never throws on connection failure; returns a result with `Ok = false`
and a message instead.
- `DriverProbeResult(bool Ok, string? Message, TimeSpan? Latency)` — outcome
record (`Message` is `null` on success; `Latency` is `null` on failure).
Implementations: every driver ships a `*DriverProbe` in its driver project
(e.g.
[`Driver.Modbus/ModbusDriverProbe.cs`](../src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusDriverProbe.cs)
does a bare socket open/close), plus the Wonderware historian's
`WonderwareHistorianDriverProbe`.
Flow: the AdminUI's `AdminProbeService`
([`AdminUI/Clients/AdminProbeService.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Clients/AdminProbeService.cs))
dispatches a `TestDriverConnect` message through `IAdminOperationsClient` to the
cluster-singleton `AdminOperationsActor`
([`ControlPlane/AdminOperations/AdminOperationsActor.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/AdminOperationsActor.cs)),
which holds the probes keyed by `DriverType` and invokes the matching one
(timeout clamped to `[1, 60]` seconds). Because the admin singleton is
admin-pinned, the probe set must be registered on admin nodes too — `Program.cs`
calls `AddOtOpcUaDriverProbes` in the `hasAdmin` block, and
`AddOtOpcUaDriverFactories` registers it for fused admin+driver nodes.
---
## IDriverSupervisor — Tier C out-of-process recycle
[`Core.Abstractions/IDriverSupervisor.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriverSupervisor.cs)
The process-level supervisor contract a **Tier C** (out-of-process) driver's
topology provides. Its concern is restarting the out-of-process Host when a
hard fault is detected (memory breach, wedge, scheduled recycle window). Tier
A/B drivers run in-process and do **not** have a supervisor — recycling them
would kill every OPC UA session and every co-hosted driver. The Core.Stability
layer only invokes this interface after asserting the tier.
Members:
- `string DriverInstanceId { get; }` — the driver instance this supervisor
governs.
- `Task RecycleAsync(string reason, CancellationToken cancellationToken)`
request a terminate+restart of the Host process; implementations are
expected to be idempotent under repeat calls during an in-flight recycle.
Callers (both in
[`Core/Stability/`](../src/Core/ZB.MOM.WW.OtOpcUa.Core/Stability/)):
- `ScheduledRecycleScheduler`
([`Core/Stability/ScheduledRecycleScheduler.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core/Stability/ScheduledRecycleScheduler.cs))
— opt-in periodic recycle. A `TickAsync` method advanced by the caller's
ambient scheduler decides whether the configured interval has elapsed and, if
so, drives `RecycleAsync`. Its constructor throws unless the tier is C, making
in-process misuse structurally impossible.
- `MemoryRecycle`
([`Core/Stability/MemoryRecycle.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core/Stability/MemoryRecycle.cs))
— on a memory hard-breach, calls `RecycleAsync` (when a supervisor is wired).
---
## IDriverHealthPublisher — health pub/sub sink
[`Core.Abstractions/IDriverHealthPublisher.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriverHealthPublisher.cs)
A sink for driver-health state-change notifications. Implementations must be
non-blocking and safe to call from any thread.
Member:
- `void Publish(string clusterId, string driverInstanceId, DriverHealth health, int errorCount5Min)`
Implementations:
- `NullDriverHealthPublisher` (same file) is the drop-in no-op for tests and
dev-stub paths. A `DriverInstanceActor` defaults to it when no publisher is
supplied.
- `AkkaDriverHealthPublisher`
([`Runtime/Drivers/AkkaDriverHealthPublisher.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/AkkaDriverHealthPublisher.cs))
is the production binding: it forwards each transition as a
`DriverHealthChanged` message onto the cluster-wide `driver-health`
Akka DistributedPubSub topic.
Producer: `DriverInstanceActor`
([`Runtime/Drivers/DriverInstanceActor.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs))
calls `Publish` when a driver's health transitions. The published snapshot is
consumed AdminUI-side and surfaced through the driver-status panel (read
in-process by the AdminUI bridge rather than dialing its own hub).
---
## IDriverConfigEditor — custom AdminUI config editor (plug-point)
[`Core.Abstractions/IDriverConfigEditor.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriverConfigEditor.cs)
An **optional** plug-point a driver can implement to provide a custom AdminUI
editor for its `DriverConfig` JSON. Drivers that don't implement it fall back to
the generic JSON editor with schema-driven validation. This is the contract
between the driver and the Admin Blazor app; the Admin app discovers
implementations and slots them into the Driver Detail screen.
Members:
- `string DriverType { get; }` — the driver type this editor handles.
- `Type EditorComponentType { get; }` — the Razor component type that renders
the editor (returned as `Type` so `Core.Abstractions` needs no Blazor
reference).
Status: this is a forward-looking plug-point. No driver ships a concrete
`IDriverConfigEditor` today — every driver uses the generic JSON editor — so
the interface currently has the contract defined but no implementations.
---
## IHistorianDataSource — server-side historian read surface
[`Core.Abstractions/Historian/IHistorianDataSource.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorianDataSource.cs)
The server-side historian read surface. Registered with the server's history
router and resolved **per OPC UA namespace**, independent of any driver's
lifecycle. This is distinct from the driver capability `IHistoryProvider`:
- `IHistoryProvider` is a *driver capability* — the server dispatches to it via
the driver instance.
- `IHistorianDataSource` is a *server registration* — the server resolves it by
namespace and calls it directly, so one historian (e.g. Wonderware) can serve
many drivers' nodes, and drivers can restart without dropping history
availability.
The interface is `: IDisposable` and declares the full read surface as
**required** members (unlike `IHistoryProvider`, where at-time/event reads are
optional default-impl methods so legacy drivers can stay raw-only):
- `ReadRawAsync(fullReference, startUtc, endUtc, maxValuesPerNode, ct)` — raw
historical samples over a time range.
- `ReadProcessedAsync(fullReference, startUtc, endUtc, interval, aggregate, ct)`
— interval-bucketed aggregates (average/min/max/count); an empty bucket
returns a `BadNoData` sample.
- `ReadAtTimeAsync(fullReference, timestampsUtc, ct)` — one sample per requested
timestamp (OPC UA HistoryReadAtTime); the returned list matches the requested
length and order, gaps as Bad-quality snapshots.
- `ReadEventsAsync(sourceName, startUtc, endUtc, maxEvents, ct)` — historical
alarm/event records (OPC UA HistoryReadEvents); `sourceName` is `null` to
return all sources. `maxEvents` is a signed `int` so a non-positive value is a
"use the backend's default cap" sentinel.
- `GetHealthSnapshot()` — point-in-time health snapshot for diagnostics and
dashboards; pure observation, never blocks on backend I/O.
All values use the shared `DataValueSnapshot` / `HistoricalEvent` shapes;
backend-specific quality/type encodings are translated to OPC UA `StatusCode`
uints inside the data source.
Implementations:
- `WonderwareHistorianClient`
([`Driver.Historian.Wonderware.Client/WonderwareHistorianClient.cs`](../src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware.Client/WonderwareHistorianClient.cs))
— the .NET 10 client that talks to the Wonderware historian sidecar over a
named pipe. It implements both `IHistorianDataSource` (read paths) and
`IAlarmHistorianWriter` (the alarm-event drain target; see
[AlarmHistorian.md](AlarmHistorian.md)).
- `HistorianDataSource`
([`Driver.Historian.Wonderware/Backend/HistorianDataSource.cs`](../src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware/Backend/HistorianDataSource.cs))
— the in-process backend implementation behind the sidecar.
The optional Wonderware historian sidecar setup is described in
[ServiceHosting.md](ServiceHosting.md).
---
## Commons — shared cross-cutting primitives
[`src/Core/ZB.MOM.WW.OtOpcUa.Commons/`](../src/Core/ZB.MOM.WW.OtOpcUa.Commons/)
`ZB.MOM.WW.OtOpcUa.Commons` is the low-level shared library that the Runtime,
ControlPlane, AdminUI, and OPC UA server projects all reference. It holds
cross-cutting primitives with no driver- or host-specific behavior, so the
heavier projects can share message contracts and value types without taking a
dependency on each other. It references only `Akka` and the internal
`ZB.MOM.WW.Audit` package.
Folders:
- **`Messages/`** — Akka message contracts grouped by concern (`Admin`,
`Alerts`, `Deploy`, `Drivers`, `Fleet`, `Logging`, `Redundancy`). These are
the wire/inter-actor messages — e.g. `Messages/Admin/TestDriverConnect.cs`
(Test Connect request, see [IDriverProbe](#idriverprobe--test-connect)) and
`Messages/Drivers/DriverHealthChanged.cs` (the driver-health pub/sub payload,
see [IDriverHealthPublisher](#idriverhealthpublisher--health-pubsub-sink)).
- **`Interfaces/`** — cluster-facing client contracts such as
`IAdminOperationsClient`, `IClusterRoleInfo`, and `IFleetDiagnosticsClient`.
- **`Types/`** — strongly-typed identifier value types: `CorrelationId`,
`DeploymentId`, `ExecutionId`, `NodeId`, `RevisionHash`.
- **`Browsing/`** — live-browse abstractions (`BrowseNode`, `IBrowseSession`,
`IDriverBrowser`) backing the AdminUI address pickers.
- **`Engines/`** — evaluator seams (`IScriptedAlarmEvaluator`,
`IVirtualTagEvaluator`, `IAlarmActorStateStore`) consumed by the
[VirtualTags](VirtualTags.md) / [ScriptedAlarms](ScriptedAlarms.md) engines.
- **`OpcUa/`** — deferred-publish seams (`IOpcUaAddressSpaceSink`,
`IServiceLevelPublisher` and their `Deferred*` no-op stand-ins) so address-space
and [ServiceLevel](Redundancy.md) writes can be wired late.
- **`Observability/`** — `OtOpcUaTelemetry` (the shared ActivitySource/metrics
surface).
---
## See also
- [ReadWriteOperations.md](ReadWriteOperations.md) — the driver *capability*
interfaces (read/write/subscribe) and resilience pipeline.
- [ServiceHosting.md](ServiceHosting.md) — role gating, the Akka cluster, and
the optional Wonderware historian sidecar.
- [AlarmHistorian.md](AlarmHistorian.md) — the store-and-forward SQLite alarm
sink that drains to `IAlarmHistorianWriter`.
- [Redundancy.md](Redundancy.md) — driver stability tiers in the redundancy
context.