docs+chore(localdb): phase-2 rig config + docs

Enables the alarm store-and-forward sink on all four driver nodes of the
docker-dev rig -- the replicating site-a pair and the default-OFF site-b pin
-- so the live gate has a real buffer to watch converge, and can see that
site-b's sink works as a plain node-local queue with no peer traffic.

Two rig details worth stating rather than rediscovering. The endpoint is
deliberately unresolvable: there is no HistorianGateway here, so every drain
attempt fails, which is exactly the historian-outage state the buffer exists
for. But it still has to be a syntactically valid absolute http(s) URI or the
host refuses to start, because ServerHistorianOptionsValidator is
consumer-gated -- AlarmHistorian:Enabled=true makes the endpoint required
even while ServerHistorian:Enabled=false. And MaxAttempts is raised far above
the production default of 10, which against a permanently unreachable gateway
would dead-letter the entire queue about five minutes in, turning a buffering
test into a dead-letter test.

Docs record what an operator now has to know: that a rising queue depth on a
Secondary is correct and on BOTH nodes is not (that shape is a redundancy
snapshot naming neither node -- check node identity before suspecting the
sink), that delivery is at-least-once across a failover by design with no
dedup layer, and that AlarmHistorian:DatabasePath is removed but should be
left in place through the upgrade because the migrator still reads it to find
the file to copy.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-21 04:43:37 -04:00
parent 271fcc4e15
commit f76d1f91e5
9 changed files with 258 additions and 62 deletions
+26 -4
View File
@@ -219,10 +219,13 @@ The server supports configurable OPC UA transport security via the `OpcUa:Enable
The server supports non-transparent warm/hot redundancy via the `Redundancy` section in `appsettings.json`. Two instances share the same Galaxy DB and the same mxaccessgw (under distinct `MxAccess.ClientName` values) but have unique `ApplicationUri` values. Each exposes `RedundancySupport`, `ServerUriArray`, and a dynamic `ServiceLevel` based on role and runtime health. The primary advertises a higher ServiceLevel than the secondary. See `docs/Redundancy.md` for the full guide.
## LocalDb pair-local config cache (Phase 1)
## LocalDb pair-local store (Phases 1 + 2)
Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired
LiteDB `LocalCache` subsystem was deleted as superseded). Phase-1 scope: it caches the
LiteDB `LocalCache` subsystem was deleted as superseded). It holds **three replicated tables**:
`deployment_artifacts` + `deployment_pointer` (Phase 1) and `alarm_sf_events` (Phase 2).
**Phase 1** caches the
**deployed-configuration artifact** (chunked, SHA-256-verified, newest-2 retention per cluster) so a
driver node can **boot from cache when central SQL Server is unreachable**`DriverHostActor` writes
the cache after each successful apply (`IDeploymentArtifactCache``LocalDbDeploymentArtifactCache`,
@@ -234,8 +237,27 @@ library's gRPC sync — **default-OFF, fail-closed bearer auth** (`LocalDbSyncAu
on `LocalDb:SyncListenPort` (a dedicated Kestrel h2c listener) + `LocalDb:Replication:*`. On the
docker-dev rig the **site-a pair** has replication on; central + site-b stay default-OFF (site-b is
the pin). ApiKey via `${secret:}`/env, never committed (the rig's dev key is the committed-dev-secret
exception). See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config
cache), and the runbook `docs/operations/2026-07-20-localdb-pair-replication.md`.
exception).
**Phase 2** moved the **alarm store-and-forward buffer** off its standalone `alarm-historian.db`
and into the same database as `alarm_sf_events`, so a node that dies holding undelivered alarm
history no longer takes it to the grave. `SqliteStoreAndForwardSink` became
`LocalDbStoreAndForwardSink` (`Core.AlarmHistorian`), and the breaking config key
`AlarmHistorian:DatabasePath` was **removed** — it is still read, by `AlarmSfLegacyMigrator`, purely
to find a pre-consolidation file to copy across on first boot (renamed `.migrated` after the copy
commits). Because the buffer replicates, **only the node holding the Primary role drains it**: the
sink takes a `drainGate` delegate, Runtime supplies it from the `IRedundancyRoleView` singleton that
`DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, and a gated-off node reports the new
`HistorianDrainState.NotPrimary`. Delivery is therefore **at-least-once across a failover, by
design** (no dedup layer). Row ids are a SHA-256 of the event payload rather than a GUID, so the
same event accepted on both nodes — which happens in every boot window, since
`HistorianAdapterActor` default-writes while its role is unknown — converges to one row. On the
docker-dev rig the site-a pair and site-b both run `AlarmHistorian__Enabled=true` against a
deliberately unresolvable gateway endpoint, so the buffer is always a live, non-draining queue.
See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config
cache), `docs/AlarmHistorian.md`, and the runbook
`docs/operations/2026-07-20-localdb-pair-replication.md`.
## LDAP Authentication