dohertj2/lmxopcua
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: dohertj2/lmxopcua:v3/batch4-address-space
Branches Tags
dohertj2/lmxopcua:master dohertj2/lmxopcua:fix/issues-503-501-493-499-488 dohertj2/lmxopcua:fix/504-abbreviate-db-strings dohertj2/lmxopcua:fix/487-scripted-alarm-reassert dohertj2/lmxopcua:fix/sql-driver-followups dohertj2/lmxopcua:feat/modbus-rtu-driver dohertj2/lmxopcua:chore/auth-0.1.5 dohertj2/lmxopcua:feat/secrets-0.2.0 dohertj2/lmxopcua:fix/alarm-condition-source-fields dohertj2/lmxopcua:v3/batch4-address-space dohertj2/lmxopcua:v3/batch3-uns-rework dohertj2/lmxopcua:v3/batch2-raw-ui dohertj2/lmxopcua:v3/batch1-schema-identity dohertj2/lmxopcua:bump-historiangateway-0.3.0 dohertj2/lmxopcua:fix/historian-gateway-integration dohertj2/lmxopcua:fix/fu1-alarm-source-roundtrip dohertj2/lmxopcua:feat/historian-gateway-backend dohertj2/lmxopcua:fix/focas-poll-io-serialization dohertj2/lmxopcua:feat/focas-pdu-v3 dohertj2/lmxopcua:chore/fixdocs-xml-doc-comments dohertj2/lmxopcua:feat/adopt-zb-health dohertj2/lmxopcua:docs/native-alarms-verified dohertj2/lmxopcua:fix/adminui-probes-and-live-panels dohertj2/lmxopcua:v2-gap-closeout dohertj2/lmxopcua:v2-akka-fuse dohertj2/lmxopcua:refactor/galaxy-mxgateway-client-rename dohertj2/lmxopcua:auto/driver-gaps-stash dohertj2/lmxopcua:auto/driver-gaps
...
compare: dohertj2/lmxopcua:c6fb8bb0ad9135a85e6f39465ce28b906b28e955
Branches Tags
dohertj2/lmxopcua:master dohertj2/lmxopcua:fix/issues-503-501-493-499-488 dohertj2/lmxopcua:fix/504-abbreviate-db-strings dohertj2/lmxopcua:fix/487-scripted-alarm-reassert dohertj2/lmxopcua:fix/sql-driver-followups dohertj2/lmxopcua:feat/modbus-rtu-driver dohertj2/lmxopcua:chore/auth-0.1.5 dohertj2/lmxopcua:feat/secrets-0.2.0 dohertj2/lmxopcua:fix/alarm-condition-source-fields dohertj2/lmxopcua:v3/batch4-address-space dohertj2/lmxopcua:v3/batch3-uns-rework dohertj2/lmxopcua:v3/batch2-raw-ui dohertj2/lmxopcua:v3/batch1-schema-identity dohertj2/lmxopcua:bump-historiangateway-0.3.0 dohertj2/lmxopcua:fix/historian-gateway-integration dohertj2/lmxopcua:fix/fu1-alarm-source-roundtrip dohertj2/lmxopcua:feat/historian-gateway-backend dohertj2/lmxopcua:fix/focas-poll-io-serialization dohertj2/lmxopcua:feat/focas-pdu-v3 dohertj2/lmxopcua:chore/fixdocs-xml-doc-comments dohertj2/lmxopcua:feat/adopt-zb-health dohertj2/lmxopcua:docs/native-alarms-verified dohertj2/lmxopcua:fix/adminui-probes-and-live-panels dohertj2/lmxopcua:v2-gap-closeout dohertj2/lmxopcua:v2-akka-fuse dohertj2/lmxopcua:refactor/galaxy-mxgateway-client-rename dohertj2/lmxopcua:auto/driver-gaps-stash dohertj2/lmxopcua:auto/driver-gaps

89 Commits
v3/batch4-address-space ... c6fb8bb0ad

Author SHA1 Message Date
Joseph Doherty c6fb8bb0ad docs(design): correct 6.2 — the keep-oldest gap is closed upstream and live here
v2-ci / build (push) Failing after 1m35s
Details
v2-ci / unit-tests (push) Has been skipped
Details 
I reported this gap as still open, citing a requirements doc line and the PR that
made the failover drill honest about it. Both were stale: ScadaBridge closed it
three hours earlier in cf3bd52f by switching to auto-down. My check missed it
because I compared only one direction (main..origin/main) and never asked whether
local main was AHEAD of origin, so two unpushed commits were invisible to me.

The substance matters more than the correction. Their commit message records a
live-proven finding: Akka.NET 1.5.62 KeepOldest.OldestDecision only lets
down-if-alone rescue a side with >= 2 members, so in a 1-vs-1 split the survivor
takes DownReachable and downs ITSELF. OtOpcUa runs precisely that configuration —
akka.conf active-strategy keep-oldest with down-if-alone on, plus the matching
typed KeepOldestOption — which means a two-node pair cannot fail over when the
oldest node crashes. That is a total-outage defect inside the redundancy feature,
independent of the mesh design, and it is now Phase 0a: most urgent, and needing a
crash-the-oldest live gate because the failure only appears 1-vs-1.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 14:16:18 -04:00
Joseph Doherty 8838e92d0a fix(tests): StopNodeBAsync also has to stop the node, not just dispose it
v2-ci / build (push) Failing after 1m36s
Details
v2-ci / unit-tests (push) Has been skipped
Details 
Completes 835fc08c, which fixed the harness's DisposeAsync but left StopNodeBAsync
calling NodeB.DisposeAsync() directly — despite that commit's own message naming
it as carrying the same bug. Caught by re-reading the code rather than trusting
the commit message.

This one is not just about memory. IHost.DisposeAsync never invokes
IHostedService.StopAsync, so CoordinatedShutdown never runs and node B never
performs a graceful Cluster.Leave. Node A instead notices only when the failure
detector times the member out — a slower and materially different path from the
graceful leave the failover tests are written to exercise. The doc comment
asserted the opposite ("shuts down node B via DisposeAsync, which runs
CoordinatedShutdown"); it now says what actually happens and why.

Verified: the three failover suites that call it pass (5/5), and the project is
unchanged at 194 passed with only the AbCip fixture failure, which needs a docker
fixture that is not running.

Also checked the sibling harness while here: LocalDbPairHarness already stops each
host before disposing it, so it needed nothing.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 10:52:57 -04:00
Joseph Doherty 835fc08c3f fix(tests): stop the integration host from ballooning to 30 GB
v2-ci / build (push) Successful in 3m40s
Details
v2-ci / unit-tests (push) Failing after 1m39s
Details 
User-reported: the integration tests leave instances behind and take 18+ GB.
Measured before changing anything — a SINGLE Host.IntegrationTests process peaked
at 30,153 MB RSS across its ~200 tests.

The cause was not a leak and not parallelism. Referencing the Host drags in the
ASP.NET Core framework reference, which turns Server GC on by default: one heap
per core, tuned for throughput and deliberately reluctant to hand memory back.
Correct for a server, wrong for a test host. Workstation GC takes the same suite
from 30,153 MB to 6,717 MB — 78% less — with no change in runtime (45s).

Two hypotheses were measured and rejected on the way, recorded here so nobody
re-runs the experiment: capping xunit maxParallelThreads to 4 cost 29% wall-clock
and saved nothing (30,153 -> 31,115 MB, i.e. noise), and it turns out a single
process was doing all of it. Reverted.

The harness fix is real but was worth only 4% on its own: TwoNodeClusterHarness
disposed each node without stopping it first, and IHost.DisposeAsync only disposes
the service provider — it never invokes IHostedService.StopAsync, which is where
Akka.Hosting terminates the ActorSystem. So every test left two live ActorSystems,
remoting transports and dispatcher pools included, rooted for the process
lifetime. The class doc-comment asserted the opposite ("DisposeAsync, which runs
CoordinatedShutdown"); it was wrong, and StopNodeBAsync inherited the same bug on
a path failover tests depend on.

The payoff is larger than the memory number. Two failures I had classified as
known-flaky baseline noise now pass consistently:
RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed
and ContinuousHistorizationRecorderTests.Retry_after_writer_failure_eventually_acks.
They were never flaky — they were starved. Full solution: peak summed test-process
RSS 15.2 GB, zero new failures, two fewer.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 10:35:35 -04:00
Joseph Doherty 06b5d81bfb Merge feat/localdb-phase2: alarm store-and-forward on LocalDb + live-gate fixes
v2-ci / build (push) Failing after 1m40s
Details
v2-ci / unit-tests (push) Has been skipped
Details 
Phase 2 moves the alarm store-and-forward buffer off its standalone
alarm-historian.db into the consolidated LocalDb as a replicated alarm_sf_events
table, so a node that dies holding undelivered alarm history no longer takes it to
the grave. SqliteStoreAndForwardSink becomes LocalDbStoreAndForwardSink; the
breaking AlarmHistorian:DatabasePath key is removed, surviving only as the path
AlarmSfLegacyMigrator reads to copy a pre-consolidation queue across on first boot.

Because the buffer now replicates, exactly-once delivery could no longer rest on
the enqueue-side gate alone — the Secondary holds a full copy — so the drain gate
had to land in the same commit that registered the table, not as a later
refinement. Delivery is at-least-once across a failover by design; row ids are a
hash of the payload, so an event both nodes accept in a boot window converges to
one row.

The live gate on the docker-dev rig found four production defects, three of which
crash-looped every driver node before its first check could run: an empty
ServerHistorian:ApiKey and a UseTls/scheme mismatch both kill the host (the
validator had classified them as degrading, but the gateway client validates its
own options at construction), and plaintext h2c was unreachable entirely because
the adapter forwarded TLS-only options unconditionally. The fourth was Phase 2's
own: the drain deferred to a cluster-wide elected Primary while the queue is
pair-local, so on a fleet rig every node — including unpaired ones — suspended its
drain and nothing drained anywhere.

Also carries the DoD sweep, which found eight live sites still naming the deleted
sink including user-visible AdminUI text, and the design work that followed: a
per-cluster Akka mesh design mirroring ScadaBridge, and a recorded limitation that
the redundancy election is scoped per Akka cluster rather than per application
Cluster.

Full suite verified twice against a pre-branch baseline worktree: failure sets
identical, zero regressions. The first run showed two extra deploy-test timeouts
that proved to be resource starvation from a leaked test-host process, not a
regression — they pass isolated and did not recur.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 10:17:45 -04:00
Joseph Doherty 4caaa11e9e docs(design): fix two staleness artefacts the settled decisions introduced 
Section 5 still listed "site-local-primary storage" among the things deliberately
NOT copied from ScadaBridge — which §6.1 now reverses, since driver nodes stop
connecting to the ConfigDb and read their configuration from LocalDb. Replaced
with an explicit copied/not-copied split, and narrowed the not-copied item to what
it should have said all along: their transient-only central alarm cache, which
OtOpcUa has no reason to adopt because it persists alarm history through the
historian and Phase 2's replicated buffer already survives a failover.

Also corrected a renumbering artefact — the risk about rewriting the docker-dev
rig cited Phase 4, but the mesh partition moved to Phase 6 when the two data-path
phases were inserted.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 10:09:10 -04:00
Joseph Doherty fe12ed9d34 docs(design): settle the three open questions on the per-cluster mesh 
(1) One central DB, driver nodes disconnected from it. Verified in ScadaBridge
before adopting: AddConfigurationDatabase is called at Program.cs:264, inside the
Central branch (89-478); the Site branch starts at 479, so site nodes never
register the central database at all. There is still exactly one authoritative
configuration database — it is central-only, and sites fetch from central and
cache locally.

The consequence worth naming: this promotes LocalDb from an outage cache to the
driver node's steady-state configuration store. Boot-from-cache stops being the
exceptional path and becomes the normal one. Phase 1's chunking, SHA-256
verification, retention and pair replication all carry over — but a cache defect
that used to be a degraded-mode bug becomes a total-availability one, so the #485
unreadable-artifact class now has a larger blast radius.

Audited the five driver-side ConfigDb consumers that must be re-homed. Two are
more than mechanical: EfAlarmConditionStateStore holds pair-local Part 9 condition
state and should follow the alarm S&F buffer into LocalDb, and DbHealthProbeActor
feeds ServiceLevel — a driver node with no DB to probe changes what a
client-visible value means.

(2) The keep-oldest gap: corrected the status rather than the conclusion. What was
resolved is the documentation and the drill, not the hole. Gitea PR #12 (closed)
carried T1 badc97af, which made the drill "measure the registered keep-oldest
outage instead of pretending recovery", and T2 d5364506, which dropped "the
undeliverable ~25s active-crash promise". The gap itself is still the registered
deferred keep-oldest topology decision on the 2026-07-08 master tracker, with no
open issue. Accepted here with the same posture, recorded as a known risk.

(3) Auth matches ScadaBridge for now — unauthenticated inter-cluster transports on
a trusted network, recorded as an accepted risk with the fail-closed interceptor
already in our tree noted as the template for whenever it is revisited.

Sequencing grows from six phases to eight; the two that change a running system's
data path rather than its wiring each get their own live gate.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 10:07:00 -04:00
Joseph Doherty f862804a35 docs(design): per-cluster Akka mesh — separate 2-node mesh per application Cluster 
Design for review, mirroring the ScadaBridge hub-and-spoke shape. Researched from
the ScadaBridge tree directly rather than from its CLAUDE.md, which understates
the boundary in five ways: there are THREE transports (ClusterClient, gRPC, plus
token-gated HTTP), the gRPC direction is inverted (central dials INTO each site),
active node is the OLDEST Up member and explicitly never the cluster leader, all
clusters share one ActorSystem name, and site nodes carry two roles.

Records a defect worth fixing regardless of the topology decision: OtOpcUa uses
two different node-selection rules at once. SBR is keep-oldest and singletons are
placed on the oldest member, but RedundancyStateActor elects Primary from
RoleLeader("driver") — lowest address. Those diverge permanently after a
restart-and-rejoin, so the node the SBR protects and hosts every singleton on need
not be the node the data-plane gates consider Primary. ScadaBridge forbids that
rule by name, for the reason it gives: both sides claim leadership during a
partition, which is the dual-primary shape archreview 03/S4 exists to prevent.

Three open questions are left explicitly undecided: whether to adopt their
autonomous-site data architecture or keep the shared ConfigDb; what to do about
the registered two-node keep-oldest total-outage gap they acknowledge but have not
closed; and whether to authenticate transports they left unauthenticated.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 06:46:17 -04:00
Joseph Doherty ad385eb70f docs(redundancy): record the per-Akka-cluster role-election limitation 
Corrects two claims made while writing the Phase 2 gate record.

The docker-dev rig is NOT misconfigured: central-1/central-2 carry admin,driver
deliberately — the compose says so — because they are themselves a pair that runs
the MAIN cluster's drivers.

And a redundant pair is not a missing concept. The application Cluster (ClusterId)
already is one: it owns drivers, devices and ClusterNode rows, and Phase 1's
deployment-artifact cache is keyed by it precisely so a pair shares one entry.

So the root cause is wider than the alarm drain. RedundancyStateActor elects one
Primary per AKKA cluster while every real unit of redundancy is an APPLICATION
cluster, and ServiceLevel, the alerts emit gate and the inbound device-write gate
all inherit that mis-scoping. On a fleet of pairs in one Akka cluster, exactly one
driver node cluster-wide services Primary-gated work.

Recommendation revised in the gate doc: fix the election scope (its own change,
its own gate, because it touches the safety-critical write gate), and explicitly
do NOT derive pair identity from LocalDb:Replication:PeerAddress — that would add
a second parallel notion of 'my partner' that the real fix obsoletes, and the
library has no initiator flag, so both halves would have to dial just to obtain a
config value.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 06:18:57 -04:00
Joseph Doherty f9f1b8fcee fix(localdb): phase-2 live gate — 4 production defects found and fixed 
Gate record: docs/plans/2026-07-20-localdb-phase2-live-gate.md.
Checks 1, 2, 5, 6 pass. Checks 3 and 4 are NOT satisfied — the defect they
were meant to confirm turned out to be the opposite of what the plan assumed.

Three defects crash-looped every driver node before check 1 could even run:

1. An empty ServerHistorian:ApiKey kills the host. ServerHistorianOptions-
   Validator exists to turn exactly that class of failure into a named
   OptionsValidationException, but its documented fail tier explicitly excluded
   ApiKey on the reasoning that a keyless client "degrades — the gateway rejects
   calls". It does not: the client validates its own options at construction, so
   the process dies during Akka startup and never makes a call.

2. UseTls disagreeing with the endpoint scheme kills the host too, in both
   directions (both messages confirmed in the shipped client assembly). Moving an
   endpoint from https to http without clearing UseTls is an ordinary migration
   slip.

3. Plaintext h2c was UNREACHABLE. HistorianGatewayClientAdapter forwarded the
   TLS-only options unconditionally, and AllowUntrustedServerCertificate defaults
   to false, so it always sent RequireCertificateValidation=true — which the
   client rejects outright when UseTls=false. Every http:// deployment crashed,
   though the scheme is documented as the supported way to select h2c, and the
   only workaround was to assert a certificate posture for a connection that has
   no certificate.

The fourth was the blocker, and it is Phase 2's own:

4. The drain gate deferred to a Primary that cannot deliver. Redundancy roles are
   elected CLUSTER-WIDE; the alarm queue is PAIR-LOCAL. On the rig the elected
   driver Primary is central-1 — it carries the driver Akka role, replicates
   nobody's LocalDb and does not even run the alarm historian — so every driver
   node logged "Historian drain suspended", including the two site-b nodes that
   have no peer at all. Nothing drained anywhere, where before Phase 2 it drained
   fine. The cost is not a duplicate; it is the buffer growing to the capacity
   wall and evicting the audit trail it exists to protect.

   Fixed in three layers: a separate ShouldDrainAlarmHistory policy (unknown role
   drains; the two gates now deliberately disagree, and a test pins that); peer-
   host matching in DriverHostActor so a node stands down only for a Primary
   holding its rows; and AddAlarmHistorian short-circuiting the gate when
   replication is unconfigured — testing BOTH Replication:PeerAddress and
   SyncListenPort, since only the dialing half sets the former while both halves
   share the queue.

   Every one of these follows from the asymmetry: a false allow costs a duplicate
   row, which at-least-once delivery already accepts and payload-hash ids
   collapse; a false deny loses data silently.

A third vacuous test, caught by the same delete-the-guard discipline: the
role-view tests stayed green with the guard removed, because AwaitAssert polls
until an assertion passes and the assertion was "reads open" — which is the
SEEDED value, satisfied at the first poll before the actor processed anything.
They now assert the sequence of published values through a recording view; the
control then goes red for exactly the cases that matter.

Migration evidence: 11 legacy rows across two deliberately overlapping files
converged to exactly 9 identical rows on both nodes, proving D-6's payload-hash
identity on real nodes rather than in a fixture.

Open design fork, recorded in the gate doc rather than decided here: a pair
cannot currently identify its own Primary, so both halves drain. Safe in every
topology — nothing loses data — but the gate's de-duplication benefit is
unrealised until roles are scoped per pair.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 06:13:01 -04:00
Joseph Doherty 2e4ccf7fe9 chore(localdb): phase-2 DoD sweep 
Build: 0 errors solution-wide, and 0 warnings from every project this branch
touches. The ~816 solution-wide warnings are pre-existing xUnit1051 /
OTOPCUA0001 / CS86xx in untouched driver + client test projects.

Tests: full solution run compared against a full run on a detached worktree at
the pre-branch baseline 2e46d054. The two failure SETS are identical -- all 13
tests, same names, zero new failures. Net +26 tests: +3 Core.AlarmHistorian
(drain gate), +6 Runtime (role view), +17 Host.IntegrationTests (migrator +
convergence). Set comparison rather than counts, because the suite carries
standing environment- and load-dependent failures a count would hide.

The greps found real drift Task 6 missed -- eight live sites still naming the
deleted SqliteStoreAndForwardSink, including the AdminUI /alarms/historian
panel text, which is user-visible, and a <see cref> in HistorianAdapterActor
that resolved to nothing without warning. All repointed at
LocalDbStoreAndForwardSink; CLAUDE.md's alarm-history paragraph now also
records the LocalDb buffer and the primary-gated drain, and drops DatabasePath
from the knob list. docs/AlarmTracking.md still promised an
AlarmHistorianOptions.Validate() startup warning for a relative DatabasePath
and an empty SharedSecret; both branches are gone, so it now says so.

Code references to AlarmHistorian:DatabasePath reduce to exactly two intentional
ones: AlarmSfLegacyMigrator.LegacyPathKey and its test. No `new SqliteConnection`
remains anywhere in Core.AlarmHistorian.

Recon doc gains the durable verification record: guard-deletion evidence for
both vacuous passes, the two exact-set replicated-table pins (both assert set
equality, so an added or a dropped registration fails), and the baseline test
comparison with a per-failure account of why each of the 13 is not this
branch's.

Stops here per the plan. Task 8's live gate needs explicit go-ahead; nothing on
this branch is to be merged.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 05:03:57 -04:00
Joseph Doherty 4480a7d755 docs(localdb): record deviations D-6 and D-7 in the phase-2 recon 
D-6 (payload-hash migrator ids over mig-{node}-{legacyId}) and D-7 (the
plan's Host.Tests project does not exist) were captured only in commit
messages and the tasks file. They belong with the other deviations, where
the next reader looks.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 04:47:35 -04:00
Joseph Doherty f76d1f91e5 docs+chore(localdb): phase-2 rig config + docs 
Enables the alarm store-and-forward sink on all four driver nodes of the
docker-dev rig -- the replicating site-a pair and the default-OFF site-b pin
-- so the live gate has a real buffer to watch converge, and can see that
site-b's sink works as a plain node-local queue with no peer traffic.

Two rig details worth stating rather than rediscovering. The endpoint is
deliberately unresolvable: there is no HistorianGateway here, so every drain
attempt fails, which is exactly the historian-outage state the buffer exists
for. But it still has to be a syntactically valid absolute http(s) URI or the
host refuses to start, because ServerHistorianOptionsValidator is
consumer-gated -- AlarmHistorian:Enabled=true makes the endpoint required
even while ServerHistorian:Enabled=false. And MaxAttempts is raised far above
the production default of 10, which against a permanently unreachable gateway
would dead-letter the entire queue about five minutes in, turning a buffering
test into a dead-letter test.

Docs record what an operator now has to know: that a rising queue depth on a
Secondary is correct and on BOTH nodes is not (that shape is a redundancy
snapshot naming neither node -- check node identity before suspecting the
sink), that delivery is at-least-once across a failover by design with no
dedup layer, and that AlarmHistorian:DatabasePath is removed but should be
left in place through the upgrade because the migrator still reads it to find
the file to copy.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 04:43:37 -04:00
Joseph Doherty 271fcc4e15 test(localdb): alarm S&F convergence scenarios 
Two driver nodes replicating the alarm buffer over the real loopback h2c
transport, through the real fail-closed interceptor. This is the test the
phase exists to pass: schema, registration, sink rewire and drain gate can
all be individually green while a pair still fails to share the undelivered
alarm history that is the whole point of moving the queue.

Rows are written through the production sink rather than hand-rolled
INSERTs, so the id derivation, column set and delete-on-ack semantics under
test are the ones production uses. The scenarios cover a buffered burst
converging with the oplog draining to zero, delivered rows being removed from
BOTH nodes as tombstones (the no-redeliver-after-failover property), writes
during a transport outage surviving the rejoin, and the same event accepted
on both nodes collapsing to one row.

The positive control found a weak assertion. With
RegisterReplicated("alarm_sf_events") commented out, three of the four went
red immediately -- but the same-event-on-both-nodes scenario still PASSED,
because with replication off each node trivially holds its own single copy
and "one row on each" is satisfied by two databases that never spoke. It now
also enqueues a distinct event on B and requires both nodes to hold two rows,
which cannot be satisfied without convergence; it goes red under the control
like the others. Control restored, all four green.

Also updates the second exact-set replicated-tables pin, in LocalDbWiringTests.
The plan predicted these pins would go red here, and that is them working:
one of them is asserted through the full driver DI graph rather than the
OnReady callback, so it catches a registration that exists in the callback
but never reaches a running host.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 04:39:32 -04:00
Joseph Doherty af545efdf5 feat(localdb): one-time alarm-historian.db migrator 
Copies the pre-consolidation store-and-forward queue into the consolidated
database on first boot, then renames the legacy file aside. Rows are in that
file precisely because the historian could not be reached, so dropping them
on upgrade would discard exactly the alarm audit trail the queue exists to
protect.

Runs last in OnReady, after every RegisterReplicated call. It is the only
thing in OnReady that writes rows, and capture is trigger-based: a migration
that ran before registration would recover the backlog locally and never
replicate a line of it, silently and permanently.

Ids are derived from the payload rather than the plan's mig-{node}-{legacyId}
scheme. Node-prefixing solves the collision the legacy AUTOINCREMENT key
would cause -- node A's row 7 and node B's row 7 are different alarms -- but
it preserves a duplication that should be collapsed instead. A warm pair's
two legacy files OVERLAP: HistorianAdapterActor default-writes while its
redundancy role is unknown, so both nodes accepted the same transitions
during every boot window. Prefixed ids would carry those duplicates into the
merged buffer forever; equal-payload ids converge them. The same property
makes a crash between commit and rename harmless under INSERT OR IGNORE.

OnReady now takes IConfiguration rather than offering an overload that skips
the migration. A wiring mistake that silently discarded a node's undelivered
alarm history is not a mistake worth making possible.

The copy is restricted to the columns the legacy table actually has. Naming
a column an older build never wrote throws "no such column", which would
discard every row in the table rather than the one field.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 04:31:10 -04:00
Joseph Doherty 71379816e7 feat(localdb): alarm S&F on LocalDb with primary-gated drain (cutover) 
Moves the alarm store-and-forward buffer out of its own alarm-historian.db
and into the node's consolidated LocalDb, where it replicates to the
redundant pair peer. A node that dies holding undelivered alarm history no
longer takes it to the grave.

Tasks 2 and 3 land together, as the plan anticipated. They are not separable:
the gate is a constructor argument of the rewritten sink, and a commit that
replicated the queue without gating the drain would be a commit in which both
nodes of a pair deliver every alarm event, continuously.

That drain gate is the load-bearing part of the change, and the recon
explains why it is new work rather than a refinement. Exactly-once delivery
across a pair is enforced today on the ENQUEUE side, by
HistorianAdapterActor: only the Primary enqueues, so the Secondary's queue is
empty and its ungated drain has nothing to send. Replicating the table
destroys that invariant.

The gate is a Func<bool> the caller supplies, because the drain runs on a
timer the sink owns rather than on a mailbox, and Core.AlarmHistorian cannot
reference PrimaryGatePolicy in Runtime. Runtime supplies it via a new
IRedundancyRoleView singleton that DriverHostActor publishes its Primary-gate
verdict to -- the same verdict the inbound-write and native-ack gates use, so
there is no second notion of am-I-the-Primary to drift.

Two failure modes are deliberately closed:

- The view is seeded OPEN, matching the policy's own answer for an unknown
  role with no driver peer. A deployment that runs no redundancy never
  publishes to it, and defaulting closed would silently stop its alarm
  history forever.

- A gate that throws is read as not-now, never as permission, and a closed
  gate reports the new HistorianDrainState.NotPrimary rather than Idle. A
  Secondary's rising queue is supposed to look different from a stalled
  drain, and if BOTH nodes report NotPrimary the pair is misconfigured and
  says so instead of quietly filling toward the capacity ceiling.

Row ids are a hash of the payload rather than fresh GUIDs. Both adapters
accept the same fanned transition in the window before the first redundancy
snapshot arrives, and under last-writer-wins an equal key converges those two
accepts into one row instead of duplicating them.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 04:13:00 -04:00
Joseph Doherty 8a9cb40a72 feat(localdb): alarm_sf_events replicated table 
Adds the alarm store-and-forward buffer to the consolidated LocalDb file and
registers it for replication, so a node that dies with undelivered alarm
history no longer takes it to the grave.

The table keeps the legacy queue's shape rather than the status column the
plan sketched. An acknowledged row is deleted, not marked delivered: that
needs no second sweeper to keep the table bounded, leaves the capacity
semantics untouched, and the replication engine carries the delete as a
tombstone so the peer drops its copy anyway -- which is what the status
column was for. last_error is retained because it is the only operator-facing
record of why a row was dead-lettered.

The primary key is app-minted TEXT. The legacy AUTOINCREMENT RowId cannot
replicate under last-writer-wins: two nodes would independently allocate
rowid 7 to different alarms and silently overwrite each other. The drain
index therefore orders by enqueued_at_utc rather than insertion order, with
id as a tiebreak so the ordering is total.

Tables are created unconditionally, independent of AlarmHistorian:Enabled.
An empty registered table costs three triggers; creating it lazily would mean
a node that enables the historian later writes rows before its capture
triggers exist, which is exactly the silent-loss shape OnReady's ordering
comment warns about.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 03:58:50 -04:00
Joseph Doherty 124de57e6f docs(localdb): phase-2 recon findings 
Answers Task 0 with file:line citations against master 2e46d054.

The STOP condition does not fire: PayloadJson is TEXT, no BLOB column, so
alarm_sf_events can be registered for replication as-is.

Two findings reshape the plan's assumptions:

- The drain worker is not a hosted service or an actor. It is a
  self-rescheduling Timer owned by the sink and started from the DI
  factory, so the Primary gate has to live inside DrainOnceAsync as an
  injected delegate -- Core.AlarmHistorian cannot reference
  PrimaryGatePolicy, which lives in Runtime.

- A Primary gate already exists on the enqueue side
  (HistorianAdapterActor.ShouldHistorize), using a different policy than
  the one the plan specifies for the drain. That gate is why today's
  ungated drain is safe: the Secondary never enqueues, so its queue is
  empty. Replicating the table breaks that invariant, which makes the
  drain gate mandatory in the same commit rather than a refinement.

Records deviations D-1 (deterministic content-hash ids so a boot-window
double-enqueue converges instead of duplicating), D-2 (gate denial must be
observable or a snapshot identity mismatch silently stops alarm history),
D-3 (Tasks 2+3 land as one commit) and D-4 (the rig will dead-letter in
~5 minutes at MaxAttempts 10 with no gateway).

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 03:56:06 -04:00
Joseph Doherty 2e46d0544e Merge docs/phase2-stop-condition: Phase 2 Task 0 STOP condition resolved (does not fire)
v2-ci / build (push) Successful in 4m54s
Details
v2-ci / unit-tests (push) Failing after 9m57s
Details 
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 03:46:56 -04:00
Joseph Doherty f2049a31d0 docs(localdb-phase2): pre-answer Task 0's STOP condition — it does not fire 
Phase 2's recon carries a STOP condition: a BLOB payload column in the legacy
alarm store-and-forward table could not be registered for replication and would
have to become base64 TEXT, sized against the chunk guidance. Resolved ahead of
execution so the answer survives into the Phase 2 session:

- No BLOB. All 8 columns are TEXT/INTEGER; the payload is PayloadJson TEXT NOT
  NULL. No base64 conversion, no chunk sizing needed.
- The executed DDL (SqliteStoreAndForwardSink.cs:657-667) matches the class
  doc-comment at :17-26 exactly — checked because a doc comment is not evidence.
- Payload is bounded by shape: a serialized AlarmHistorianEvent is 10 scalar
  fields, no collections or nesting, so realistic worst case is low single-digit
  KB against a 171 KB guidance.
- The PK IS autoincrement, confirming the plan's prediction that the migrator
  must mint deterministic mig-{node}-{legacyId} ids and the new table needs a
  TEXT GUID PK.
- Also captured for reuse: the drain's covering index (DeadLettered, RowId), and
  the exact legacy column list the migrator's pragma_table_info intersection
  check compares against.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 03:46:56 -04:00
Joseph Doherty d218282cd6 Merge docs/track-deferred-items: track deferred work as issues; fix stale limitation
v2-ci / build (push) Successful in 4m38s
Details
v2-ci / unit-tests (push) Failing after 13m22s
Details 
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 03:37:26 -04:00
Joseph Doherty 5ebf9876a5 docs: file Gitea issues for every deferred item; correct a stale limitation 
Deferred work was living in plan docs and a CLAUDE.md banner rather than the
issue tracker, so it was invisible unless you happened to read the right file.
Each item was verified against the current tree before filing — nothing
speculative, nothing already fixed:

- #487 scripted-alarm redeploy recovery (CONFIRMED pre-existing bug). Verified
  still unfixed: ReassertValue exists only for VirtualTags, there is no
  OnAlarmsLoaded equivalent. Carries double-alarm-history risk, hence the
  careful proposed shape (node-only re-assert, never the alerts topic).
- #488 periodic desired-vs-actual subscription reconcile. Records what exists
  today (Connected-entry re-assert + a health-poll tick that never touches
  subscriptions) and why a true reconcile needs a new ISubscribable member
  across 7 drivers, plus the cheap no-interface-change variant to try first.
- #489 raw-tag rename hot-rebind (today: deploy-THEN-recreate).
- #490 absolute-RawPath Monaco tag-path completion.
- #491 continuous-historization live end-to-end + restart-convergence gate.

CLAUDE.md's "KNOWN LIMITATION 1" was STALE and told the reader not to trust a
cutover that has in fact been live-validated (read/write-persist/alarm-send all
green; alarm-readback skipped as a confirmed protocol limitation, not a bug) —
evidence in docs/plans/2026-06-27-otopcua-historian-followups.md. Rewritten as
a passed gate, keeping the recipe for re-running it. Limitation 2 is genuinely
open and now points at #491.

The cross-repo ScadaBridge cutover was already tracked in that repo (its #14
open, #20 closed), so no duplicate was filed; the Batch 4 doc now says so, and
records that the umbrella-index half is done.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 03:37:25 -04:00
Joseph Doherty 7808faa7f6 Merge fix/486: fail the apply when the deployment artifact could not be read
v2-ci / build (push) Successful in 12m43s
Details
v2-ci / unit-tests (push) Failing after 12m43s
Details 
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 03:27:58 -04:00
Joseph Doherty 689fc06e85 docs(486): live-gate the failed-apply fix on docker-dev 
Regression check first, since this changes ACK semantics: a normal deploy still
records Applied on all six nodes. Then the same induction as #485 — the node
now records Failed with a reason while the five nodes that read the real
artifact record Applied (pre-fix all six claimed Applied), and the log shows it
staying on the PREVIOUS revision rather than the dispatched one, which is what
lets a retry land. Values kept flowing throughout (59603, current), and the
next real deploy applied normally.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 03:27:54 -04:00
Joseph Doherty ef41a43102 fix(drivers): fail the apply when the artifact could not be read (#486) 
ApplyAndAck advanced _currentRevision and recorded NodeDeploymentState=Applied
immediately after ReconcileDrivers, which returns null when the artifact could
not be read. So a node that applied NOTHING reported success, claimed a
revision whose configuration it never applied, and — because
HandleDispatchFromSteady short-circuits on a revision match — could never be
healed by re-dispatching that revision. Only a later, different revision
recovered it.

Now a null blob fails the apply: the revision is left where it was,
NodeDeploymentState records Failed with the reason, and the coordinator gets a
Failed ACK. Leaving the revision alone is what makes the retry actually land
instead of being waved through.

This is about telling the truth, not about tearing anything down — the node
keeps serving its last-known-good address space, drivers and subscriptions
throughout (#485).

Tests, RED-first (both verified failing with "should be Failed but was
Applied"): the ACK/revision assertion, plus the one that matters — after a
failed apply, re-dispatching the SAME revision now genuinely applies. Its
recovered artifact adds a second driver precisely so a short-circuited ACK
(which has no side effects) cannot satisfy it.

Four existing tests changed, and both changes are deliberate:

- DriverHostActorTests.SeedDeployment never set ArtifactBlob at all. Its three
  consumers are about the apply/ACK/state machine, not the artifact, and now
  need a genuine apply — so the helper seeds a well-formed artifact declaring
  no drivers. That is what the composer emits for an empty configuration, and
  is precisely what "bytes we could not read" is NOT.
- EmptyArtifact_IsNotCached asserted "the apply still succeeds — an empty
  artifact is a legitimate no-op deployment". That premise is the bug. Flipped
  to Failed; the test's actual subject (nothing is cached) is untouched and now
  holds for two independent reasons.

Closes #486

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 03:22:40 -04:00
Joseph Doherty 5627d325eb Merge docs/485-live-gate: live verification of both #485 fixes
v2-ci / build (push) Successful in 4m1s
Details
v2-ci / unit-tests (push) Failing after 8m56s
Details 
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 03:02:07 -04:00
Joseph Doherty e4397aa84e docs(485): live-gate both #485 fixes on docker-dev 
All three guards observed firing in production, on two nodes, with the live
values proving the subscription half rather than just the log line: the Modbus
tag advanced 56278 -> 56336 -> 56354 straight across the induction.

The RED control was not staged. site-a-1 had hit #485 during the PREVIOUS
gate's SQL flapping and had been serving an empty address space for 34 minutes
(PureRemove, removed=17) on the pre-fix image — the bug in its natural habitat.
Rebuilding on the fixed image restored it, and the induction then failed to
reproduce the teardown.

The original trigger was a race, so the gate drives the empty-bytes branch
instead — byte-for-byte what the code sees when the row cannot be read, and the
branch the driver-side guards key on (their throw paths already returned early
before this work). Pausing the target node opens the window deterministically;
both runs measured <1s, well inside acceptable-heartbeat-pause, so no failover
was triggered. Of the six nodes exactly one took the guard path — the other
five read the real artifact and applied it, which is its own control.

Records one pre-existing defect this work does NOT fix: ApplyAndAck advances
_currentRevision and writes NodeDeploymentState=Applied before knowing whether
anything applied, so a node that skipped reports success for a revision it
never applied — and since dispatch short-circuits on a revision match it cannot
self-heal from a re-dispatch. Data plane is unaffected (the node keeps serving
its last good config); it is a reporting/convergence bug. Left out of these
fixes because correcting it changes deploy ACK semantics fleet-wide.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 03:02:02 -04:00
Joseph Doherty 5aad1e33de Merge fix/485 driver-side: keep drivers + subscriptions when an artifact reads back empty
v2-ci / build (push) Successful in 4m20s
Details
v2-ci / unit-tests (push) Failing after 9m37s
Details 
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 02:40:18 -04:00
Joseph Doherty b0c031f09f fix(drivers): don't tear down field I/O when a deploy's artifact reads back empty (#485) 
The driver-side half of the same mistake the address-space fix corrected.
ReconcileDrivers and PushDesiredSubscriptionsFromArtifact both read the
artifact as `...FirstOrDefault() ?? Array.Empty<byte>()`. Their THROW paths
already returned early, but empty bytes flowed onwards as a real answer: zero
driver specs, so DriverSpawnPlanner planned every running child for StopChild,
and then an empty desired set dropped each surviving driver's live
subscription handle. A node lost its entire field I/O and still ACKed Applied.

Same rule as the address space: no bytes is no answer, not "a configuration
with no drivers". Nothing legitimate produces a zero-length blob — deleting
the last driver still deploys a JSON document with empty arrays. Both sites
now skip and keep what is running. The two guards are independent because
PushDesiredSubscriptions does its OWN ConfigDb read, so the row can go missing
between them.

Also corrects the ReconcileDrivers doc comment, which claimed an empty blob
made it "effectively a no-op" — with children running it was the opposite.

Tests: DriverHostActorUnreadableArtifactTests, RED-first (verified failing —
after the empty dispatch the driver list was empty). A zero-length ArtifactBlob
reproduces byte-for-byte what the missing-row case delivers, so the race does
not have to be constructed.

Two controls, both load-bearing:
- a READABLE driverless artifact still stops the driver, so the guard keys on
  "the read gave us nothing", not on "fewer drivers than before";
- dropping a driver's LAST TAG still clears its subscription. This one was
  added after the absence assertion was caught passing on a race: the
  unsubscribe is an async self-tell, so with the second guard deleted the test
  still went green. The control observes that same unsubscribe arriving well
  inside the settle window, which is what makes the absence meaningful — with
  the guard deleted the suite now correctly goes RED.

SubscribableStubDriver gains an UnsubscribeCount for that observation.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 02:40:14 -04:00
Joseph Doherty c7f5b9cfa3 Merge fix/485: hold the last-known-good address space when a deploy's artifact cannot be read
v2-ci / build (push) Successful in 3m53s
Details
v2-ci / unit-tests (push) Failing after 9m54s
Details 
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 02:29:43 -04:00
Joseph Doherty 4b0be1335e fix(opcua): hold last-known-good when a deploy's artifact cannot be read (#485) 
A transient ConfigDb error while loading a deployment's artifact emptied the
served address space. LoadArtifact caught the exception, logged "rebuild
becomes no-op" and returned zero bytes; those parsed to an EMPTY composition,
which the planner diffed against the live one as a PureRemove and the applier
faithfully executed — 16 nodes gone, while the log claimed nothing happened.

An artifact we could not obtain is not an empty configuration. Nothing
legitimate produces a zero-length blob (an operator who really deletes
everything still deploys a JSON document with empty arrays), so "no bytes" can
only mean "no answer". HandleRebuild now abandons the rebuild on one, leaving
both the materialised nodes and _lastApplied intact so the retry diffs against
what is actually being served. The loader's own log line is now true.

The two cases are logged differently: warn when there is a live address space
being protected, debug when nothing has been deployed to this node yet.

Covered by a RED-first actor test (verified failing: the load error tore down
eq-2's subtree), plus a positive control proving a READABLE artifact that
genuinely drops an equipment still removes it — the guard suppresses teardown
only when the read failed.

Found on the LocalDb Phase 1 follow-up live gate, which induced it by flapping
SQL; that gate's log is the production evidence for the seam.

Closes #485

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 02:29:39 -04:00
Joseph Doherty 5fdb5a5a5e Merge fix/localdb-phase1-followups: close both LocalDb Phase 1 live-gate follow-ups
v2-ci / build (push) Successful in 3m35s
Details
v2-ci / unit-tests (push) Failing after 8m51s
Details 
- Oplog growth on default-OFF nodes, fixed at the source: the artifact cache
  now skips the store entirely when the pointer already names this
  deployment/revision, the SHA matches the bytes, and the expected chunk count
  is present. Both extra conditions are load-bearing — guard tests go RED
  against a naive pointer-only skip.
- Wiped-node back-fill, fixed upstream in ZB.MOM.WW.LocalDb 0.1.2 + 0.1.3 and
  pinned here.

Live-gated on docker-dev (docs/plans/2026-07-21-localdb-followups-live-gate.md).
The gate found a third defect that offline tests could not have found first,
because it lived behind the one being fixed: once back-fill worked, the rebuilt
node's own writes turned out to go nowhere. It also surfaced one unrelated
pre-existing bug, filed as issue #485.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 02:17:51 -04:00
Joseph Doherty 09df440675 docs(localdb): link the live gate's unrelated finding to issue #485 
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 02:17:32 -04:00
Joseph Doherty f587e7972e docs(localdb): live-gate the two Phase 1 follow-ups; pin LocalDb 0.1.3 
The gate closed both follow-ups on the docker-dev rig and found a third defect
that offline tests could not have found first: it lived behind the one being
fixed. Once 0.1.2 made back-fill work, a rebuilt node could be observed writing
for the first time — and its writes went nowhere, because the healthy node kept
the old peer's seq watermark. Library 0.1.1 -> 0.1.2 -> 0.1.3 over the course
of the gate; both consumers now pinned to 0.1.3.

Check 4 (was PARTIAL, now PASS): with SQL stopped and site-a-2's LocalDb volume
destroyed, a-1 logged "Snapshot sent (as_of_seq 0, 4 rows)" — a line that could
not appear on 0.1.1 — and a-2 came back with a row_version dump byte-identical
to a-1's, carrying a-1's origin node ids and its ORIGINAL applied_at_utc. It
then booted from cache and served 17 ns=2 nodes, diff-identical to its peer,
from a configuration it never applied and could not have fetched.

Check 8 (PASS): two restarts with an unchanged artifact left the oplog at 7 and
the pointer timestamp frozen — the re-cache was skipped outright. Positive
control: a real config change still wrote, 7 -> 10 -> 13.

Also records one unrelated pre-existing finding the gate's SQL flapping
surfaced: a transient ConfigDb error during artifact load empties the served
address space while logging "rebuild becomes no-op". The cache layer correctly
refused to store the bad artifact and the node recovered on restart, but the
two logs disagree and it is the same class as the Phase 1 gate's check-3
defect. Untouched by this work; wants its own issue.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 02:14:00 -04:00
Joseph Doherty 232bff5df7 fix(localdb): close both Phase 1 live-gate follow-ups 
1. Wiped-node back-fill (live gate check 4). Fixed upstream in
   ZB.MOM.WW.LocalDb 0.1.2 and pinned here. The gate's diagnosis was slightly
   off: the oplog cap was not the gate. Snapshot detection measured the peer's
   gap against the oldest surviving oplog row and read an empty oplog as "no
   gap possible" — and an empty oplog is the steady state of a converged pair,
   since ack-pruning deletes everything the peer confirmed. The healthy state
   was the one state that could not heal a wiped peer. Now measured against
   last_acked_seq when the oplog is empty.

   Pinned at this level too, not just the library's: the pair harness grows a
   WipePassiveAsync (a NEW database — a rebuilt node comes back with a new node
   id and a zero watermark), and the new scenario asserts the emptied oplog as
   its precondition before wiping, then requires the deployment artifact to
   come back byte-identical with no new deploy. Verified RED against the pinned
   0.1.1 (times out waiting for back-fill) and green on 0.1.2.

2. Oplog growth on default-OFF nodes (live gate check 8), fixed at the source:
   StoreAsync now skips entirely when the pointer already names this
   deployment/revision, the SHA matches the bytes, and the expected chunk count
   is present. Re-caching an artifact the node already holds — every restart's
   boot-from-cache, every RestoreApplied — writes nothing and mints no oplog
   rows, where before it cost a delete plus an insert per chunk plus a pointer
   update, all identical to what was already there.

   Identity is over the bytes and the chunks are counted, both deliberately: a
   re-composed artifact can carry the same ids with different bytes, and a
   pointer can name a deployment whose chunks are missing, where skipping would
   make an unreadable cache permanent. Both guards verified RED against a naive
   pointer-only skip.

   The gate's stated bound was also wrong and is corrected in the doc: growth
   was never headed for the 1M row cap. AddZbLocalDbReplication is registered
   unconditionally, so MaintenanceBackgroundService runs on default-OFF nodes
   and the 7-day MaxOplogAge cap prunes.

Runtime.Tests 412/0/31, Host.IntegrationTests LocalDb 45/45. Pre-existing and
unrelated: AbCip_Green_AgainstSim (needs the AB CIP docker fixture, red on a
clean master too) and a flaky Roslyn race test (green 2/2 in isolation).

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
 2026-07-21 01:27:02 -04:00
Joseph Doherty c957db52e2 Merge feat/localdb-phase1: LocalDb Phase 1 pair-local config cache + boot-from-cache + optional replication
v2-ci / build (push) Successful in 3m39s
Details
v2-ci / unit-tests (push) Failing after 9m26s
Details
2026-07-21 01:02:52 -04:00
Joseph Doherty ebc5fc6742 chore(localdb): record live-gate results in tasks.json 2026-07-20 23:15:21 -04:00
Joseph Doherty 41c36064dd docs(localdb): phase-1 live-gate evidence (8/8 pass; 4 defects fixed, 2 limitations) 
Records the docker-dev live gate: 8/8 checks pass (check 4 with a documented
limitation). The gate caught four real defects every offline test missed — the
NU1101 packageSourceMapping gap, the ASPNETCORE_HTTP_PORTS re-bind, the empty
address space on boot-from-cache, and the cache not repopulating on
RestoreApplied — all fixed on-branch with regression tests. Two limitations
documented as follow-ups: no replication back-fill of a fully-wiped node, and
oplog growth on default-OFF nodes.
 2026-07-20 23:10:44 -04:00
Joseph Doherty c6a9f93a0c fix(localdb): re-cache the artifact on RestoreApplied so a wiped node self-heals 
CacheAppliedArtifact ran only on a fresh apply (ApplyAndAck), never on the
bootstrap RestoreApplied path. So a node whose cache was lost — fresh/wiped
volume, disk failure — recovered its served state from central yet stayed
cache-less, unable to boot-from-cache on the NEXT central outage until some
future new deploy happened to land. Replication does not heal this: a peer's
already-acked cache rows are pruned from its oplog, so a fully-wiped, converged
node is never back-filled. RestoreApplied now writes the served artifact back to
the cache (invariant: the cache holds what the node serves). Found on the
docker-dev live gate wiping a node's LocalDb volume.
 2026-07-20 22:56:23 -04:00
Joseph Doherty 9137cb41eb fix(localdb): materialise the address space from the cached blob on boot-from-cache 
Boot-from-cache told OpcUaPublishActor to RebuildAddressSpace(deploymentId), which
re-loads the artifact from the ConfigDb by id — the very database that is
unreachable during the outage this path exists to survive. The rebuild no-op'd, so
a cache-booted node ran its drivers but served OPC UA clients an EMPTY address
space. RebuildAddressSpace now carries an optional in-hand Artifact blob;
ApplyCachedArtifact passes the cached bytes so materialisation happens from them
directly. Found on the docker-dev live gate: healthy peer served 16 nodes, the
cache-booted node served 1.
 2026-07-20 22:38:54 -04:00
Joseph Doherty ce9fa07ff8 fix(localdb): re-bind ASPNETCORE_HTTP_PORTS surface, not just ASPNETCORE_URLS 
The Kestrel re-bind read only urls/ASPNETCORE_URLS. The aspnet:8.0+ base images
set ASPNETCORE_HTTP_PORTS=8080 (all interfaces) as the CONTAINER default in place
of ASPNETCORE_URLS, so a driver node that never sets URLS fell straight through to
Kestrel's localhost:5000 default — silently moving its health/metrics surface to
loopback when the sync listener took over Kestrel. Now falls through URLS →
HTTP_PORTS/HTTPS_PORTS (wildcard, all-interfaces) → localhost:5000, mirroring
Kestrel's own source precedence. Found on the docker-dev live gate (central nodes
were fine — they set ASPNETCORE_URLS=9000 explicitly).
 2026-07-20 22:25:31 -04:00
Joseph Doherty 4b2f0e6e15 fix(localdb): map ZB.MOM.WW.LocalDb* to the gitea feed in packageSourceMapping 
Task 1 added the package versions but not the packageSourceMapping patterns, so
ZB.MOM.WW.LocalDb / .Replication / .Contracts fell to the '*' → nuget.org rule
and any CLEAN restore (Docker image build, CI, fresh clone) failed NU1101 —
'PackageSourceMapping is enabled, dohertj2-gitea not considered'. Local dev builds
masked it via the warm global NuGet cache. Verified with a temp-packages-dir restore.
 2026-07-20 22:16:16 -04:00
Joseph Doherty c08762db1b chore(localdb): phase-1 DoD sweep 
Offline sweep: full-solution build 0 errors (824 pre-existing OTOPCUA0001
analyzer warnings live in driver test projects, none in LocalDb files);
no LiteDB/LocalCache code references remain (2 explanatory doc-comment lines
allowed); Runtime.Tests 407/0/31 twice (flagged intermittent not reproduced);
Host.IntegrationTests LocalDb subset 40/0/0. Full-solution test suite with
stash-baseline deferred to CI/live gate — infra-gated suites cannot run offline
on macOS. Not merged to master; live gate (Task 16) awaits operator go-ahead.
 2026-07-20 22:11:02 -04:00
Joseph Doherty c834d89ae7 docs(localdb): phase-1 runbook + redundancy/config docs 
New operations runbook (enable/disable, fail-closed ApiKey rule, stop/start-
together, tombstone-retention window, MaxBatchSize row-count-vs-4MB, never
sqlite3-a-live-WAL-DB cp-triplet recipe). Configuration.md gains a LocalDb
section; Redundancy.md gains a pair-local config cache section (what boot-from-
cache does and does not cover); CLAUDE.md gains the Phase 1 scope paragraph.
 2026-07-20 22:06:58 -04:00
Joseph Doherty 7676ab93c4 chore(localdb): rig config — consolidated path everywhere, replication on the site-a pair 
Every driver-role node gets a per-node named volume at /app/data and
LocalDb__Path=/app/data/otopcua-localdb.db so the pair-local config cache
survives container recreates. The site-a pair enables replication (SyncListenPort
9001, a-1 dials a-2, shared dev-only ApiKey, MaxBatchSize 16); the central and
site-b pairs stay default-OFF — site-b is the pin the live gate's check 8
verifies. docker compose config validates; rig not brought up (live gate owns runs).
 2026-07-20 22:04:28 -04:00
Joseph Doherty afa5be713a test(localdb): 2-node convergence harness + scenarios 
Two driver nodes replicate the deployment-artifact cache over a real loopback
h2c transport through the real fail-closed LocalDbSyncAuthInterceptor; both DBs
init via the production LocalDbSetup.OnReady. Scenarios: byte-identical
convergence with matching pointer HLC+origin, retention prune reaching B as
tombstones, writes-while-transport-down surviving rejoin, and wrong-key
non-convergence with a matching-key positive control.

DoD positive control: commenting out both RegisterReplicated calls turns all
four scenarios red (verified locally, restored before commit).
 2026-07-20 22:02:03 -04:00
Joseph Doherty 5a21be0a62 test(localdb): DI pins over the real host composition methods 
Builds a container from the same AddOtOpcUa* methods Program.cs's driver branch
calls (AddOtOpcUaLocalDb / AddGrpc / AddOtOpcUaHealth / AddOtOpcUaObservability) and
resolves through it - DI extensions have shipped inert three times in this family.

Does not boot the real host: WebApplicationFactory<Program> would start hosted
services (Akka, OPC UA server, ValidateOnStart) needing SQL + a mesh, per D-5. The
seam under test is DI resolution, so the container is built and resolved without
starting.

Five pins: ILocalDb singleton + exact replicated-table set; IDeploymentArtifactCache
-> concrete; default-OFF ISyncStatus; localdb-replication health check registered and
constructible; admin-only graph has NO ILocalDb and does not demand LocalDb:Path.
 2026-07-20 21:51:25 -04:00
Joseph Doherty b350fba233 feat(localdb): replication health check + meter export 
LocalDbReplicationHealthCheck reports Healthy when replication is default-OFF or
LocalDb is absent (admin-only graphs) - so a plain node is never degraded by it -
Degraded when configured-but-disconnected, connected-with-unknown-backlog (a failed
oplog poll must not read as zero), or connected-with-backlog past the threshold.
Never Unhealthy: a replication fault does not stop the node serving its address
space. Pure decision core is table-tested; registered unconditionally via a factory
so AddOtOpcUaHealth keeps its no-arg signature and resolves ISyncStatus optionally.

Adds LocalDbMetrics.MeterName to the observability allowlist. o.Meters is a strict
allowlist with no wildcard, so the localdb.sync.* / localdb.oplog.depth series were
otherwise silently absent from /metrics - the omission a ScadaBridge live gate
caught.
 2026-07-20 21:48:47 -04:00
Joseph Doherty abe10dbbd7 docs(localdb): phase-1 execution progress / resume state (tasks 0-9 done) 2026-07-20 21:45:35 -04:00
Joseph Doherty a27eff3298 feat(localdb): boot from the pair-local artifact cache when central SQL is unreachable 
Hooks the Bootstrap catch that previously went straight to Stale. On a cache hit the
node serves its last-known-good configuration through the outage instead of coming
up with an empty address space; on a miss, behaviour is byte-for-byte what it was.

The read is unkeyed because ClusterId is only derivable from an artifact you already
hold or from the unreachable central DB (recon D-1). Newest pointer wins, which is
correct in every real topology - a node belongs to one cluster and its peer
replicates that same row.

Splits PushDesiredSubscriptionsFromArtifact out of PushDesiredSubscriptions: the
cache path runs precisely when the ConfigDb read cannot succeed, so re-reading the
artifact there would fail by definition.

RunningFromCache is surfaced on NodeDiagnosticsSnapshot (optional param, existing
call sites unaffected). A node running from cache looks entirely healthy from
outside - full address space, live values - but its config is frozen and no deploy
can reach it. It clears only on a real apply from central.
 2026-07-20 11:12:19 -04:00
Joseph Doherty 1becf59168 feat(localdb): DriverHostActor stores applied artifacts in the pair-local cache 
New Props/ctor parameter appended LAST at all three positions - the forwarding
list inside Props.Create is positional and compiles into an expression tree, so a
mis-ordered argument is usually type-compatible and binds the wrong dependency at
runtime. All 26 existing test construction sites use named arguments and needed no
change.

ReconcileDrivers now returns the blob it loaded. It catches its own DB failures and
returns without rethrowing, so ApplyAndAck can reach its success path having spawned
zero drivers; caching on 'the apply succeeded' would persist an empty artifact as
last-known-good and boot the node into an empty address space during the next
outage. Verified: weakening the guard to null-only turns EmptyArtifact_IsNotCached
red.

The store runs in its own try/catch because an Applied ACK has already been sent by
then - an escaping exception would land in ApplyAndAck's catch and emit a second,
contradictory Failed ACK for a deployment the fleet believes is live.
 2026-07-20 11:06:57 -04:00
Joseph Doherty a38a52b831 feat(localdb): chunked deployment-artifact cache over ILocalDb 
128 KiB raw chunks base64-encoded into deployment_artifacts, with a per-cluster
pointer carrying a SHA-256 over the raw bytes. Reassembly verifies chunk count
AND the SHA before returning, so a partially replicated artifact reads back as a
clean miss rather than a silently truncated address space.

Adds GetCurrentUnkeyedAsync beyond the plan's interface: at the boot-failure seam
the actor cannot know its ClusterId (it is derivable only from an artifact you
already have, or from the central DB that is unreachable). Recon D-1.

The prune's 'deployment_id <> @DeploymentId' clause makes 'the pointer's target is
always present' structural rather than a side effect of clock resolution. Three
stores inside one timestamp tick fall through to a deployment_id tiebreak, and
since real ids are GUIDs that ordering is random - the row just written could lose,
leaving the pointer naming chunks that no longer exist. Verified red without the
clause.
 2026-07-20 10:58:27 -04:00
Joseph Doherty 2bae1b4c9f refactor(localdb): delete the dormant LiteDB LocalCache (superseded by ZB.MOM.WW.LocalDb) 
Never registered in DI and dead since Phase 6.1. Keeping two unwired cache designs
invites the next reader to wire the wrong one.

Deletes all three test files - LiteDbConfigCacheTests.cs was missing from the plan's
list and would have failed to compile (it calls new LiteDB.LiteDatabase directly).
StaleConfigFlagTests lives inside ResilientConfigReaderTests.cs, so it goes too.

The XML-doc reference in ILdapGroupRoleMappingService is rewritten rather than removed:
it now records that no sign-in fallback exists and that reviving one means an admin-side
cache on LocalDb.
 2026-07-20 10:46:07 -04:00
Joseph Doherty e771a11a30 fix(localdb): rename Runtime.Deployment namespace to Runtime.DeploymentCache 
The Runtime.Deployment namespace shadowed the Deployment EF entity type for
every file under ZB.MOM.WW.OtOpcUa.Runtime.Tests.*: C# name lookup walks the
enclosing namespace chain and finds the namespace before a using-imported type,
so 19 pre-existing call sites failed with CS0118. Caught only by a full-solution
build - the Host and its own tests compiled fine.
 2026-07-20 10:42:22 -04:00
Joseph Doherty b9ddf20edd feat(localdb): dedicated h2c listener + MapZbLocalDbSync gated on LocalDb:SyncListenPort 2026-07-20 10:39:34 -04:00
Joseph Doherty 6aff9a8332 feat(localdb): wire AddOtOpcUaLocalDb into the driver role 2026-07-20 10:35:02 -04:00
Joseph Doherty 3d29be834e feat(localdb): fail-closed bearer interceptor for the sync endpoint 2026-07-20 10:28:39 -04:00
Joseph Doherty 3b65c24bfb feat(localdb): deployment-cache schema + OnReady registration 2026-07-20 10:28:39 -04:00
Joseph Doherty 44255fcb91 build(localdb): reference ZB.MOM.WW.LocalDb 0.1.1 + Grpc.AspNetCore 2026-07-20 10:21:47 -04:00
Joseph Doherty 42f8550716 docs(localdb): phase-1 recon findings 2026-07-20 10:18:01 -04:00
dohertj2 1adaa2fc01 chore(auth): bump ZB.MOM.WW.Auth 0.1.1 -> 0.1.5 (family alignment) (#484)
v2-ci / build (push) Successful in 4m34s
Details
v2-ci / unit-tests (push) Failing after 10m48s
Details 
Functionally inert for OtOpcUa (all intervening changes were in ApiKeys, which it does not consume); build byte-identical to the master baseline.
 2026-07-20 06:06:10 -04:00
Joseph Doherty a5eac3ec78 chore(auth): bump ZB.MOM.WW.Auth 0.1.1 -> 0.1.5
v2-ci / build (pull_request) Successful in 3m56s
Details
v2-ci / unit-tests (pull_request) Failing after 11m26s
Details 
Aligns OtOpcUa with the rest of the family — mxgw, HistorianGateway and
ScadaBridge have all been on 0.1.5; OtOpcUa was four versions behind and the
only outlier.

The bump is functionally inert here, deliberately so. OtOpcUa consumes
Abstractions + Ldap + AspNetCore, NOT ApiKeys, and every intervening change was
in ApiKeys: 0.1.2 stamped its schema version, 0.1.3 added SetScopes/SetEnabled,
0.1.4 added ExpiresUtc. 0.1.5 is the patched-SQLitePCLRaw pin
(GHSA-2m69-gcr7-jv3q), which this repo already carries independently as its own
surgical transitive pin at Directory.Packages.props:105 — so no vulnerability
was open here either.

The value is removing a false signal: a repo sitting four versions behind reads
as "missing something" every time someone audits the family, and the next real
Auth change would have arrived as a four-version jump instead of one.

Verified: build 861 warnings / 0 errors, byte-identical to the master baseline
(measured, not assumed — master was built to compare). Security 84, AdminUI
665, Configuration 121, ControlPlane 78 — all pass, 0 failures.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 06:03:10 -04:00
Joseph Doherty be87ddeb0b chore(secrets): bump to Secrets 0.2.3 - visible delete modal (scadaproj#2)
v2-ci / build (push) Successful in 4m50s
Details
v2-ci / unit-tests (push) Failing after 10m31s
Details 
0.2.3's Secrets.Ui ships ConfirmDeleteModal's own styles under
collision-proof zb-secrets-* class names; on 0.2.2 this host's Bootstrap
.modal{display:none} made the /admin/secrets delete modal permanently
invisible (confirmed live on the docker-dev rig). No other library change
affects this consumer. AdminUI tests 665/665; secrets Host.IntegrationTests 8/8.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-19 01:22:10 -04:00
Joseph Doherty 5f72ff851d fix(adminui): make /admin/secrets interactive - wrap the Secrets.Ui page with a render mode (#483)
v2-ci / build (push) Successful in 4m0s
Details
v2-ci / unit-tests (push) Failing after 10m19s
Details 
The Secrets.Ui RCL's routable page was routed directly, but this host's
router is deliberately static (cookie SignInAsync needs SSR) with per-page
@rendermode opt-in - a directive the RCL page cannot carry, because the
other three family hosts render it under a globally interactive router
where a nested render mode throws. Routing straight to the RCL page
therefore rendered /admin/secrets with no circuit: it displayed, but every
@onclick was silently dead.

Fix: a host-side wrapper page (Pages/SecretsAdmin.razor) now owns
/admin/secrets, carries the standard per-page InteractiveServer render
mode, re-states the RCL page's own authorization policy (the router only
enforces [Authorize] on the routed component, which is now the wrapper),
and renders the RCL page as an ordinary child. The RCL assembly is
de-registered from both AdditionalAssemblies sites (router + endpoint) so
the route is unambiguous.

Guards: SecretsPageWiringTests pins the render mode (the #483 regression),
route parity with the RCL page, and policy parity; the page census in
PageAuthorizationGuardTests classifies the new page and admits the
secrets:manage policy. AdminUI.Tests 665/665.

Live-verified on the rebuilt docker-dev rig with Playwright: /_blazor
circuit negotiated on /admin/secrets and Add secret opens the editor
(before the fix: zero interactive markers, no circuit, dead click).

Closes #483.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-19 00:22:23 -04:00
Joseph Doherty c878fbbd03 feat(secrets): enable Akka clustered secret replication on the docker-dev rig
v2-ci / build (push) Successful in 3m52s
Details
v2-ci / unit-tests (push) Failing after 9m47s
Details 
Turns on Secrets:Replication:Enabled across all six docker-dev host nodes via
one x-secrets-env anchor: replication enabled, a 5s AnnounceInterval so
anti-entropy is observable during dev exercise, and the ONE shared dev-only
KEK every node requires (committed default per the rig's zero-operator-step
pattern, overridable via OTOPCUA_SECRETS_KEK; never reuse outside this rig).

Live-proven on the rig with Secrets 0.2.2: all six nodes start with NO
startup deadlock (the scadaproj#1 signature never appeared), the mesh forms,
and a secret seeded into site-a-1's local store alone converged to all five
peers within 60s — byte-identical ciphertext on all six nodes (same
sha256, kek_id sha256:f3f0c2056e7a), decrypt-verified on central-2 and
site-b-2. site-b-2 restart rejoined cleanly with the row intact; the fleet
also survived a full Docker-engine restart with replication enabled.

Found during verification, tracked separately: the AdminUI /admin/secrets
page renders but is non-interactive (Secrets.Ui RCL components carry no
@rendermode and the host applies render modes per-page), so Add secret does
nothing - pre-existing, unrelated to replication.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-18 16:00:14 -04:00
Joseph Doherty 2254ae3dea fix(secrets): consume Secrets 0.2.2 - clustered-secrets DI deadlock fixed upstream
v2-ci / build (push) Successful in 3m55s
Details
v2-ci / unit-tests (push) Failing after 11m7s
Details 
Bumps the four ZB.MOM.WW.Secrets pins 0.2.1 -> 0.2.2 (closes the OtOpcUa side
of scadaproj#1, tracked here as #482). 0.2.1's Akka replicator deadlocked any
hosted process at startup when Secrets:Replication:Enabled was true: the
package's DI graph closed a circular singleton dependency through factory
lambdas (store decorator -> replicator -> actor provider -> cache invalidator
-> resolver -> store), which MS.DI's StackGuard turns into a silent
cross-thread call-site-lock deadlock. 0.2.2 defers the invalidator edge to
first eviction. The flag stays default-false; enabling remains a
per-environment decision.

The_startup_hook_actually_creates_the_replication_actor is now a real test:
SecretReplicationStarter's docs had promised it since the adoption, and the
upstream fix finally makes a provider-based resolve runnable - container built
exactly as the host does, hook started under a watchdog, replication actor
proven to exist by ActorSelection on a self-joined single-node cluster (no
TestKit needed, which matters because Akka.TestKit.Xunit2 is xunit-v2-only and
this project is on xunit.v3). Also corrects the stale rationale that blamed
the old hang on DistributedPubSub needing a joined cluster - the actor
constructor was never reached; it was the DI cycle.

Verified: SecretsReplicationRegistrationTests 8/8 on the 0.2.2 feed packages;
full slnx build 0 errors; the 2-node Akka live convergence gate re-run against
the published 0.2.2 packages passes 6/6 (write->peer, tombstone propagation
without resurrection, delete visibility through the resolver cache, reverse
direction, wrong-KEK fail-closed).

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-18 15:03:35 -04:00
Joseph Doherty 1ccc237cb6 docs(test): record why these assert descriptors, not a built provider
v2-ci / build (push) Successful in 4m21s
Details
v2-ci / unit-tests (push) Failing after 10m15s
Details 
Akka.TestKit is the family convention for non-end-to-end Akka testing and would
be the right tool here, but Akka.TestKit.Xunit2 is xunit-v2-only and this project
is on xunit.v3 - adding it produces CS0433 type conflicts. Directory.Packages.props
already documents the same constraint for the three projects deliberately held on
xunit v2 for TestKit. Revisit when Akka ships an xunit.v3 TestKit.
 2026-07-18 12:19:50 -04:00
Joseph Doherty b3d1a26f38 fix(secrets): consume Secrets 0.2.1 - Akka replication was inert in 0.2.0 
0.2.0's AddZbSecretsAkkaReplication never bound its own ISecretReplicator: it
called AddZbSecrets first, which TryAdds NoOpSecretReplicator, so the package's
own TryAdd was silently discarded. Replication published into a no-op sink and no
actor was ever spawned - no exception, no log line. Found by the Task 6 wiring
work, which is the first code that ever built a container around that extension.
Fixed upstream in 0.2.1.

Test consequence worth recording: the replication-enabled registration tests
previously passed against a plain ActorSystem only BECAUSE of that bug - the
no-op never touched Akka. With 0.2.1 they resolve a real replicator, which spawns
the actor, whose PreStart needs DistributedPubSub and therefore a joined cluster,
so provider-based assertions hang. Standing a real cluster up inside this test
assembly was attempted and did not work, so these assertions are now made against
the ServiceCollection, which is precisely where the defect lives. Actor creation
and convergence remain covered upstream by the library's TwoNodeClusterReplication
tests against a genuine 2-node cluster.

7 registration tests pass in 23ms. Full solution builds 0 errors; no warnings
originate from these files.
 2026-07-18 12:12:02 -04:00
Joseph Doherty 3336ec08c7 feat(secrets): opt-in Akka cluster secret replication (default OFF; upstream blocker documented) 
Routes the host's secrets registration through a new AddOtOpcUaSecrets extension
that gates the ISecretStore implementation on Secrets:Replication:Enabled.

Opt-in gate (default FALSE)
  This call decides which ISecretStore every node resolves — including driver-role
  nodes with no auth/AdminUI, where a wrong store surfaces as drivers failing to
  open sessions rather than as a failing test. With the flag false the wiring is
  the pre-existing AddZbSecrets(config, "Secrets") call, unchanged, so current
  behavior is byte-identical. With it true, AddZbSecretsAkkaReplication replaces
  that call (it invokes AddZbSecrets internally; calling both would double-register).

  Extracted to a named extension specifically so the registration is testable:
  Program.cs is top-level statements and cannot be exercised by a container test,
  which is how a "registered but never resolvable" defect ships unnoticed.

Serializer HOCON
  AkkaSecretsReplication.SerializationConfig is merged into the ActorSystem config
  inside the AddAkka configurator, conditionally on the same gate — a non-replicating
  node carries no bindings for messages it will never see. Merged via
  AddHocon(..., HoconAddMode.Append), Akka.Hosting's fallback merge and the same mode
  the existing base-config merge uses; a raw Config.WithFallback would fight the
  builder's own assembly.

Lazy-actor mitigation
  The replication actor is created lazily on first ISecretStore resolution, so a node
  that never touches a secret would never announce a manifest and would silently never
  converge. SecretReplicationStarter (IHostedService) resolves the store once at
  startup to make participation unconditional.

KNOWN BLOCKER — replication is currently NON-FUNCTIONAL; do not enable
  ZB.MOM.WW.Secrets.Replicator.AkkaDotNet 0.2.0 never binds its own ISecretReplicator.
  AddZbSecretsAkkaReplication calls AddZbSecrets FIRST, which does
  TryAddSingleton<ISecretReplicator, NoOpSecretReplicator>(); the package's own
  TryAddSingleton<ISecretReplicator>(AkkaSecretReplicator) that follows is therefore
  a no-op. Verified empirically in a built container: with Enabled=true,
  ISecretReplicator resolves to NoOpSecretReplicator, so ReplicatingSecretStore
  publishes into a sink and no actor is ever spawned.

  Consequence: the startup hook cannot create the actor, and the test asserting it
  does is committed Skipped with the evidence. Not worked around here — the fix
  belongs upstream (AddSingleton, or register before calling AddZbSecrets).
  Because the flag defaults false, this commit is inert in production.

Tests: SecretsReplicationRegistrationTests (new) — disabled path resolves plain
SqliteSecretStore and needs no ActorSystem; enabled path resolves
ReplicatingSecretStore AND the undecorated concrete SqliteSecretStore the decorator
is built from (the exact registration gap that shipped once); startup hook registered
only when enabled. Red before wiring (4 assertion failures), green after: 6 pass,
1 skipped (blocker above).

Build: 861 warnings / 0 errors, unchanged from baseline (full --no-incremental A/B).
Host.IntegrationTests: 123 pass, 6 skip, 1 fail — AbCip_Green_AgainstSim, verified
pre-existing on the stashed tree (fixture-gated).

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-18 11:15:03 -04:00
Joseph Doherty e27c19c49d feat(secrets): add ZB.MOM.WW.Secrets.Replicator.AkkaDotNet 0.2.0 package reference 2026-07-18 05:35:12 -04:00
Joseph Doherty f347762350 chore(secrets): bump ZB.MOM.WW.Secrets 0.1.2 -> 0.2.0 
Version hygiene + picks up the G-8 KEK-rotation surface, and is the
precondition for adopting clustered secret replication. NOT a security fix
for this repo: SQLitePCLRaw.bundle_e_sqlite3 was already pinned to 2.1.12.
 2026-07-18 05:20:10 -04:00
dohertj2 2cae4c8f01 Merge pull request 'feat(alarms): scripted condition Quality from worst-of-input tag quality (#478)' (#480) from feat/scripted-alarm-quality-478 into master
v2-ci / build (push) Successful in 5m30s
Details
v2-ci / unit-tests (push) Failing after 12m49s
Details
2026-07-17 16:08:06 -04:00
Joseph Doherty 043e237dba docs(alarms): state #478 coverage boundary + file Layer-4 comms-loss follow-up (#481)
v2-ci / build (pull_request) Successful in 5m42s
Details
v2-ci / unit-tests (pull_request) Failing after 13m11s
Details 
Post-implementation review (HIGH finding) noted #478's mux-delivered
input-quality path does not cover a driver comms-loss: a poll driver
(Modbus/S7) whose device goes unreachable emits only ConnectivityChanged and
goes silent on the value feed, so a scripted alarm keeps the last Good value.
The code as shipped faithfully implements #478's written scope (worst of input
tags' qualities via the dependency mux). The comms-loss bridge for scripted
alarms (symmetric of native #477-L2, plus the null-value/cold-start asymmetry
and its VT-quality ripple) is tracked as #481. Docs updated in
AlarmTracking.md + the design doc.
 2026-07-17 16:07:55 -04:00
Joseph Doherty 8c5e2be92e feat(alarms): scripted condition Quality from worst-of-input tag quality (#478)
v2-ci / build (pull_request) Successful in 3m48s
Details
v2-ci / unit-tests (pull_request) Failing after 11m0s
Details 
Layer 3 of #477: a scripted alarm's condition Quality now reflects the WORST
quality across its input tags, mirroring the native OT semantic (#477 L2).

Plumbing (quality was silently discarded twice on the live path):
- VirtualTagActor.DependencyValueChanged gains Quality (defaulted Good); the
  DependencyMuxActor forwards the published AttributeValuePublished.Quality it
  already carried; ScriptedAlarmHostActor.OnDependencyChanged pushes the real
  quality into the engine (was hardcoded 0u/Good).

Engine (Core.ScriptedAlarms):
- ScriptedAlarmEngine computes worst-of-input quality each eval (skipping
  not-yet-published inputs, which are a readiness concern, not a quality signal)
  and carries it on ScriptedAlarmEvent.WorstInputStatusCode.
- A real transition carries the current worst quality so ToSnapshot's full
  snapshot doesn't clobber quality back to Good (e.g. transition while Uncertain).
- A Bad input freezes the condition (no transition), like a comms-lost native
  driver; a quality-bucket change with no transition emits the new
  EmissionKind.QualityChanged, routed to the existing #477-L2
  AlarmQualityUpdate -> WriteAlarmQuality node path (quality only, no /alerts
  row, no historian write). ScriptedAlarmSource skips QualityChanged so it never
  fabricates a phantom IAlarmSource event.

Host: ToSnapshot maps WorstInputStatusCode -> OpcUaQuality; OnEngineEmission
routes QualityChanged out of band.

Tests (TDD, RED-first): engine worst-carry + Bad/restore QualityChanged +
unchanged-bucket-no-emit; source swallows QualityChanged; mux forwards quality;
host Bad-dep -> AlarmQualityUpdate(no alerts) + transition snapshot carries worst.
Docs: AlarmTracking.md Layer-3 section + design doc.

Closes #478
 2026-07-17 15:56:06 -04:00
dohertj2 6dda0549e2 Merge pull request 'feat(alarms): condition Quality tracks source connectivity (#477)' (#479) from fix/alarm-condition-quality-477 into master
v2-ci / build (push) Successful in 4m53s
Details
v2-ci / unit-tests (push) Failing after 12m49s
Details
2026-07-17 15:18:34 -04:00
Joseph Doherty db751d12a5 feat(alarms): condition Quality tracks source connectivity (#477)
v2-ci / build (pull_request) Successful in 4m49s
Details
v2-ci / unit-tests (pull_request) Failing after 3h11m25s
Details 
Part 9 ConditionType.Quality was never assigned; default(StatusCode)==Good
so every native + scripted condition reported Good unconditionally — a
comms-lost device still showed a healthy, inactive, Good condition (a
wrong-VALUE bug, distinct from the null-value #473/#475). Clients (and HMIs
bucketing on IsGood) could not tell "genuinely inactive" from "lost contact".

Layer 1 — make Quality a real, plumbed field:
- AlarmConditionSnapshot gains OpcUaQuality Quality (default Good).
- MaterialiseAlarmCondition sets it (native BadWaitingForInitialData, scripted Good).
- WriteAlarmCondition projects snapshot.Quality; the delta-gate gains a Quality
  member so a quality-bucket change fires a Part 9 event.

Layer 2 — drive native quality from driver connectivity (a comms-lost driver
emits no alarm transitions, and an alarm-bearing raw tag has no value variable,
so quality can't come from either existing channel):
- DriverInstanceActor Tells parent ConnectivityChanged on Connected/Reconnecting.
- DriverHostActor fans it to every native condition the driver owns as
  OpcUaPublishActor.AlarmQualityUpdate (Good on connect, Bad on disconnect).
- New dedicated IOpcUaAddressSpaceSink.WriteAlarmQuality sets ONLY Quality and
  fires only on a bucket change — never touches Active/Acked/Retain (an active
  alarm that loses comms stays active). Not a full-snapshot re-projection, so it
  can't clobber severity/message and works for a never-fired condition.
  Forwarded through DeferredAddressSpaceSink (F10b trap; auto-verified by the
  reflection forwarding guard). Ungated by redundancy role; no /alerts row.

Scripted conditions stay Good; worst-of-input quality deferred to #478 (Layer 3).

Tests: node-level (materialise/project/no-clobber/unknown-node no-op),
NativeAlarmProjector, DriverInstanceActor connectivity emission, DriverHostActor
fan-out, OpcUaPublishActor routing, and the wire-level guard
(Condition_event_Quality_tracks_source_connectivity_on_the_wire) — RED-verified
against a simulated pre-fix always-Good server. Existing DriverInstanceActor
parent probes ignore the new ConnectivityChanged.

Docs: docs/AlarmTracking.md §"Condition source-data Quality (#477)";
design doc docs/plans/2026-07-17-alarm-condition-quality-477-design.md.
 2026-07-17 15:10:04 -04:00
dohertj2 f6a3c31b60 Merge pull request 'fix(alarms): populate ConditionClassId/ConditionClassName on condition events (#475)' (#476) from fix/alarm-condition-class-fields into master
v2-ci / build (push) Successful in 3m41s
Details
v2-ci / unit-tests (push) Failing after 9m56s
Details
2026-07-17 13:58:39 -04:00
Joseph Doherty e08b6b0e69 fix(alarms): populate ConditionClassId/ConditionClassName on conditions (#475) 
MaterialiseAlarmCondition never assigned the mandatory Part 9 ConditionType
classification fields, so every condition event — native and scripted — shipped
ConditionClassId = NodeId.Null (i=0) and ConditionClassName = empty text. Same
mechanism as #473: Create() builds the mandatory children from the type's
embedded definition but leaves them unset, and nothing downstream synthesises
them (ReportEvent / InstanceStateSnapshot copy children verbatim). An HMI
bucketing alarms by condition class dropped every OtOpcUa alarm as unclassified.

Report BaseConditionClassType — Part 9's "no condition class modelled" value.
This is the honest report: we hold no classification at the materialise seam.
Deliberately NOT ProcessConditionClassType (the SDK sample's pick), which would
assert a classification we cannot back and would be actively wrong for a Galaxy
alarm whose upstream category is Safety/Diagnostics — trading a detectable null
for an undetectable lie. Real per-alarm classification needs the driver's
AlarmCategory carried to this deploy-time seam (it lives only on the runtime
AlarmEventArgs transition today) and is a separate feature.

Guards, both observed RED against the pre-fix server:
- NativeAlarmEventIdentityFieldDeliveryTests: wire-level, its own select clause
  (the #473 test's clause mirrors ScadaBridge's exactly and its indices are
  load-bearing, so it is left untouched). The class fields are declared on
  ConditionType, not BaseEventType, so they are selected against that type.
- NodeManagerAlarmSourceFieldsTests: node-level, native (Raw) + scripted (Uns).

Stacked on #473 (PR #474) — merge after it.
 2026-07-17 00:49:43 -04:00
dohertj2 50426d4790 Merge pull request 'fix(alarms): populate EventType/SourceNode/SourceName on native + scripted conditions (#473)' (#474) from fix/alarm-condition-source-fields into master
v2-ci / build (push) Successful in 3m55s
Details
v2-ci / unit-tests (push) Failing after 10m4s
Details
2026-07-17 00:45:51 -04:00
Joseph Doherty 7339a4af07 fix(alarms): populate EventType/SourceNode/SourceName on conditions (#473)
v2-ci / build (pull_request) Successful in 3m38s
Details
v2-ci / unit-tests (pull_request) Failing after 9m38s
Details 
MaterialiseAlarmCondition never assigned the three mandatory BaseEventType
identity fields, so all three arrived null on every condition event — native
and scripted. The SDK does not synthesise them on this path: Create() builds
the children from the type definition but leaves them unset, the auto-filling
BaseEventState.Initialize overload is only used for transient events, and
ReportEvent / InstanceStateSnapshot copy children verbatim. A conforming
client could not attribute an alarm to its source.

  EventType  = the concrete materialised type (TypeDefinitionId)
  SourceNode = the condition's own NodeId (== ConditionId) — the condition IS
               the source; an alarm-bearing raw tag materialises only the
               condition, with no sibling value variable
  SourceName = the same identifying id string: RawPath (native) /
               ScriptedAlarmId (scripted)

SourceName carries the unique id rather than the leaf name: the leaf collides
across devices (HR200 on two PLCs) and is already carried by ConditionName, so
nothing is lost. Documented in docs/AlarmTracking.md, including that clients
must key on ConditionId and must not compose SourceName with ConditionName,
and that SourceName is NOT a live<->history join key (the alarm-history writer
stamps it with the EquipmentPath — a pre-existing divergence, now called out).

Tests: NativeAlarmEventIdentityFieldDeliveryTests is the wire-level guard —
a real client subscription using the standard [EventType, SourceNode,
SourceName, Time, Message, Severity] select clause, verified to fail against
the pre-fix server. NodeManagerAlarmSourceFieldsTests guards the node across
both realms, the base-type fallback, and the kind-swap re-materialise.

The HistoryRead events projection is a separate path (it projects historian
rows, not node fields) and is unaffected — its EventType => Variant.Null
assertions still hold.
 2026-07-17 00:36:44 -04:00
Joseph Doherty 872cf7e37a test(secrets): register ISecretResolver in driver-factory resilience tests (Task 10 fix)
v2-ci / build (push) Successful in 3m36s
Details
v2-ci / unit-tests (push) Failing after 8m35s
Details 
The full-suite gate caught a real regression: AddOtOpcUaDriverFactories' registration
now resolves ISecretResolver (for Galaxy/OpcUaClient secret: refs, Tasks 7-8), so the
two ResilienceInvokerFactoryRegistrationTests that build a minimal container and resolve
the invoker factory threw 'No service for ISecretResolver'. The real host always
registers it via AddZbSecrets (unconditional, Task 3), and GetRequiredService correctly
fails-fast for a genuinely misconfigured host — so the fix is to complete the test
container with a stub resolver, matching production composition. Test-only.
 2026-07-16 18:27:59 -04:00
Joseph Doherty f1534920de test(secrets): G-2c guard AdminUI DriverConfig secret: ref round-trip (Task 9) 
Verify + regression-guard that the AdminUI driver-config editor persists secret:/
env:/file: refs verbatim and never resolve-then-resaves cleartext (which would
re-leak the secret into the config store, defeating G-2). Confirmed structurally
safe: DriverConfig is persisted as an opaque JSON string via RawTreeService, which
has NO secret-resolver dependency (only the DbContext factory) — resolution is
impossible on the save path. Resolution stays confined to driver-instantiation /
Test-connect (Tasks 7-8); GalaxyDriverBrowser resolves into a connect-scoped local
only, never writing back. Three round-trip tests (OpcUaClient Password/UserCertPassword,
Galaxy ApiKeySecretRef, env:/file: forms) exercise the real RawTreeService seam and
assert byte-identical save→reload. No product change (no leak found). 662 AdminUI tests pass.
 2026-07-16 18:19:24 -04:00
Joseph Doherty 9bb237b794 feat(secrets): G-2b resolve OpcUaClient secret: Password/UserCertificatePassword (Task 8) 
Resolve secret:-prefixed Password + UserCertificatePassword through the shared
ISecretResolver, fail-closed on absent, retiring the cleartext-in-DB path. The
driver-registry factory is synchronous (Func<string,string,IDriver>), so resolution
is done lazily in the async session-open (InitializeAsync, before BuildUserIdentity)
rather than at deserialize — mirroring Task 7's Galaxy pattern and matching its
re-resolve-on-reconnect behavior. Both consumers (username Password and the
certificate-password LoadPkcs12 path via BuildCertificateIdentity) see the resolved
connect-scoped options; _options stays raw (secret: refs intact), no long-lived
plaintext field.

Scope corrections vs the plan (verified against v3): the probe is unauthenticated
GetEndpoints-only and never reads either credential, so it is NOT a resolution site
(comment added, no dead code); OpcUaClientDriverOptions was a sealed class, converted
to sealed record for the with-expression (no positional params → identical JSON; no
reference-equality/dict-key/ToString-log usages → no behavior/leak change).

ISecretResolver threaded via factory Register/CreateInstance + DriverFactoryBootstrap
(real resolver from DI); NullSecretResolver null-object backs test/parse paths only,
fail-closed on secret: refs. TDD: 4 helper tests RED→GREEN; 142 OpcUaClient tests pass.
 2026-07-16 18:14:12 -04:00
Joseph Doherty 1424a21419 feat(secrets): G-2a secret: arm on GalaxySecretRef via ISecretResolver (Task 7) 
Add a secret:NAME arm to GalaxySecretRef.ResolveApiKey that resolves the Galaxy
gateway API key through the shared ISecretResolver — fail-closed if the secret is
absent (never falls through to the cleartext literal arm), retiring the dev:/literal
in-DB path for production. Because GetAsync is async the method becomes
ResolveApiKeyAsync; the await cascade threads ISecretResolver by ctor injection into
GalaxyDriver + GalaxyDriverBrowser and (since GalaxyDriver is built by a static
factory closure, not DI) through GalaxyDriverFactoryExtensions + DriverFactoryBootstrap
(which pulls the real resolver from the service provider — registered unconditionally
in Slice 1). A NullSecretResolver null-object backs the parse-only/test paths only;
the runtime path always gets the real resolver (verified end-to-end).

TDD: 3 new secret:-arm tests (resolve / fail-closed-on-absent / no-literal-warning)
RED without the arm, GREEN with it; 338 Galaxy tests pass; no sync-over-async.
 2026-07-16 17:57:04 -04:00
Joseph Doherty ce383df39a docs(secrets): G-5 clustered master-key posture runbook (Task 6) 
OtOpcUa is Akka-clustered and AddZbSecrets is registered unconditionally, so every
node (admin, driver, fused) resolves secrets and needs the SAME KEK + same store rows.
Ship G-5 as a production-posture runbook rather than hardcoding Source=File + shared
paths into the committed role-overlay appsettings (which dev + TwoNodeClusterHarness
also consume — that would break every dev/CI boot). Base appsettings stays on
Source=Environment. Documents the interim File-KEK + shared-SQLite posture and the
G-7 hand-off (ConfigDb-backed ISecretStore mirroring the existing DP
PersistKeysToDbContext<OtOpcUaConfigDbContext> key-ring sharing). Cross-linked from
docs/security.md. Mirrors the ScadaBridge G-5 resolution.
 2026-07-16 17:30:32 -04:00
Joseph Doherty a0be76b5f0 feat(secrets): mount /admin/secrets UI + secrets authorization (Task 5, G-6) 
Mount the shared ZB.MOM.WW.Secrets.Ui RCL page at /admin/secrets on admin nodes:
- Register secrets:manage/secrets:reveal policies additively via
  Configure<AuthorizationOptions>(o => o.AddSecretsAuthorization()) in AddAdminUI
  (the admin-only composition layer that already references the RCL — avoids forcing
  an RCL dependency into the core Security lib; mirrors HistorianGateway)
- Register the RCL routable assembly in BOTH the SSR endpoint (AddAdditionalAssemblies)
  and the interactive Router (App.razor AdditionalAssemblies) or the route 404s
- Add a Secrets nav item; the page's own [Authorize(Policy=...)] gates access

Claim-type MATCH: AdminRole=Administrator reads the same ClaimTypes.Role as FleetAdmin,
so existing Administrators are authorized with no new role mapping. Full clean boot
verified; interactive reveal deferred to the live gate (shared UI already proven).
 2026-07-16 17:26:54 -04:00
Joseph Doherty 73d8439412 docs(secrets): G-4 ${secret:} config delivery convention + fail-closed proof (Task 4) 
OtOpcUa commits no plaintext secrets — the only secret-shaped committed value is
ServerHistorian:ApiKey='' (empty, disabled by default). Because the pre-host expander
is fail-closed AND section-agnostic, hard-committing a ${secret:} token would break
every dev/CI/integration boot for zero at-rest benefit. Instead document the token
delivery convention for all five config secrets (jwt/ldap/deploy/configdb/historian)
in docs/security.md and note the token option in the ServerHistorian comments.

Mechanism proven live: unseeded token boot fails with SecretNotFoundException at
pre-host expansion; seeded token resolves and boot proceeds. Behavior-neutral
(comment + doc only).
 2026-07-16 17:14:32 -04:00
Joseph Doherty 8843418c54 feat(secrets): register AddZbSecrets unconditionally on the host (Task 3) 
Runtime ISecretResolver must be present on every clustered node regardless of
role: driver-only nodes resolve Layer-B DriverConfig secret: refs but have no
auth/DP/AdminUI (all admin-role-gated). Placed in the unconditional flow next to
AddOtOpcUaHealth(), outside both role blocks. Data Protection left untouched.
 2026-07-16 17:07:41 -04:00
Joseph Doherty 772d3a5f34 feat(secrets): add ZB.MOM.WW.Secrets refs + pre-host ${secret:} expander (Tasks 1-2) 
Adopt the ZB.MOM.WW.Secrets library (0.1.2, Gitea feed) into OtOpcUa under CPM:
- Directory.Packages.props + NuGet.config source mapping for the three packages
- Host/AdminUI/Driver.Galaxy.Contracts/Driver.OpcUaClient package references
- Layer-A pre-host ${secret:} config expander in Host/Program.cs, placed after all
  config providers assemble and before the first config read (mechanism only,
  behavior-neutral no-op until Task 4 introduces tokens)
- Secrets section in appsettings.json (env-var KEK, SQLite store, 30s cache TTL)

Mechanism mirrors the proven HistorianGateway adoption. KEK never committed.
 2026-07-16 17:03:16 -04:00
dohertj2 ec6598ceae Merge pull request 'v3 Batch 4 — dual-namespace address space + raw-only runtime binding (v3.0)' (#472) from v3/batch4-address-space into master
v2-ci / build (push) Successful in 3m59s
Details
v2-ci / unit-tests (push) Failing after 9m25s
Details
2026-07-16 14:38:29 -04:00
165 changed files with 13851 additions and 1868 deletions
Expand all files Collapse all files
CLAUDE.md
+58 -11
View File
@@ -219,6 +219,46 @@ The server supports configurable OPC UA transport security via the `OpcUa:Enable
The server supports non-transparent warm/hot redundancy via the `Redundancy` section in `appsettings.json`. Two instances share the same Galaxy DB and the same mxaccessgw (under distinct `MxAccess.ClientName` values) but have unique `ApplicationUri` values. Each exposes `RedundancySupport`, `ServerUriArray`, and a dynamic `ServiceLevel` based on role and runtime health. The primary advertises a higher ServiceLevel than the secondary. See `docs/Redundancy.md` for the full guide.
## LocalDb pair-local store (Phases 1 + 2)
Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired
LiteDB `LocalCache` subsystem was deleted as superseded). It holds **three replicated tables**:
`deployment_artifacts` + `deployment_pointer` (Phase 1) and `alarm_sf_events` (Phase 2).
**Phase 1** caches the
**deployed-configuration artifact** (chunked, SHA-256-verified, newest-2 retention per cluster) so a
driver node can **boot from cache when central SQL Server is unreachable** — `DriverHostActor` writes
the cache after each successful apply (`IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache`,
in `Runtime/Deployment/`) and reads it as a boot fallback, flagging running-from-cache. Wired by
`AddOtOpcUaLocalDb` in the `hasDriver` branch only (`Host/Configuration/LocalDbRegistration.cs` +
`LocalDbSetup.OnReady`, whose DDL→`RegisterReplicated` order is load-bearing); **admin-only nodes
never register it.** The cache can optionally replicate to the node's redundant pair peer over the
library's gRPC sync — **default-OFF, fail-closed bearer auth** (`LocalDbSyncAuthInterceptor`), gated
on `LocalDb:SyncListenPort` (a dedicated Kestrel h2c listener) + `LocalDb:Replication:*`. On the
docker-dev rig the **site-a pair** has replication on; central + site-b stay default-OFF (site-b is
the pin). ApiKey via `${secret:}`/env, never committed (the rig's dev key is the committed-dev-secret
exception).
**Phase 2** moved the **alarm store-and-forward buffer** off its standalone `alarm-historian.db`
and into the same database as `alarm_sf_events`, so a node that dies holding undelivered alarm
history no longer takes it to the grave. `SqliteStoreAndForwardSink` became
`LocalDbStoreAndForwardSink` (`Core.AlarmHistorian`), and the breaking config key
`AlarmHistorian:DatabasePath` was **removed** — it is still read, by `AlarmSfLegacyMigrator`, purely
to find a pre-consolidation file to copy across on first boot (renamed `.migrated` after the copy
commits). Because the buffer replicates, **only the node holding the Primary role drains it**: the
sink takes a `drainGate` delegate, Runtime supplies it from the `IRedundancyRoleView` singleton that
`DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, and a gated-off node reports the new
`HistorianDrainState.NotPrimary`. Delivery is therefore **at-least-once across a failover, by
design** (no dedup layer). Row ids are a SHA-256 of the event payload rather than a GUID, so the
same event accepted on both nodes — which happens in every boot window, since
`HistorianAdapterActor` default-writes while its role is unknown — converges to one row. On the
docker-dev rig the site-a pair and site-b both run `AlarmHistorian__Enabled=true` against a
deliberately unresolvable gateway endpoint, so the buffer is always a live, non-draining queue.
See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config
cache), `docs/AlarmHistorian.md`, and the runbook
`docs/operations/2026-07-20-localdb-pair-replication.md`.
## LDAP Authentication
The server uses LDAP-based user authentication via the `Security:Ldap` section in `appsettings.json`. When enabled, credentials are validated by LDAP bind against a GLAuth server, and LDAP group membership maps to OPC UA permissions: `ReadOnly` (browse/read), `WriteOperate` (write FreeAccess/Operate attributes), `WriteTune` (write Tune attributes), `WriteConfigure` (write Configure attributes), `AlarmAck` (alarm acknowledgment). `LdapOpcUaUserAuthenticator` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/LdapOpcUaUserAuthenticator.cs`) implements `IOpcUaUserAuthenticator`, delegating the LDAP bind + group lookup to `OtOpcUaLdapAuthService` (`src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/OtOpcUaLdapAuthService.cs`, an `ILdapAuthService`). See `docs/security.md` for the full guide.
@@ -321,11 +361,13 @@ authorization uses the standard `AccessLevels.HistoryRead` bit set at materializ
### Alarm-history path (`AlarmHistorian` section)
Alarm events are written through `GatewayAlarmHistorianWriter` (the gateway **`SendEvent`** path) behind
the durable **`SqliteStoreAndForwardSink`** — `AlarmHistorian:Enabled=true` swaps the `NullAlarmHistorianSink`
default for the SQLite store-and-forward queue, whose drain worker forwards batches to the gateway and uses
per-event outcomes to decide retry vs. dead-letter (never throws). The `AlarmHistorian` section carries
only the `Enabled` gate + the SQLite knobs (`DatabasePath`, `DrainIntervalSeconds`, `Capacity`,
`DeadLetterRetentionDays`, `BatchSize`, `MaxAttempts`) — the downstream gateway connection
the durable **`LocalDbStoreAndForwardSink`** — `AlarmHistorian:Enabled=true` swaps the `NullAlarmHistorianSink`
default for the store-and-forward queue, which buffers into the node's consolidated LocalDb as the
replicated `alarm_sf_events` table (see the LocalDb section) and whose drain worker forwards batches to
the gateway and uses per-event outcomes to decide retry vs. dead-letter (never throws). **Only the node
holding the Primary role drains** (a gated node reports `HistorianDrainState.NotPrimary`). The
`AlarmHistorian` section carries only the `Enabled` gate + the queue knobs (`DrainIntervalSeconds`,
`Capacity`, `DeadLetterRetentionDays`, `BatchSize`, `MaxAttempts`) — the downstream gateway connection
(endpoint/key/TLS) is sourced from the `ServerHistorian` section. **Alarm-history `ReadEvents` requires the
target gateway deployed with `RuntimeDb:EventReadsEnabled=true`** (the C2 SQL event-read workaround).
@@ -397,11 +439,16 @@ The `AlarmHistorian` section's old Wonderware connection keys (`Host`/`Port`/`Us
were pruned — remove them; the SQLite knobs are retained and the downstream connection now comes from
`ServerHistorian`. See `docs/Historian.md` for the full guide.
### KNOWN LIMITATION 1 — live-validation gate (do before merging/trusting the cutover)
### Live-validation gate — PASSED (was "KNOWN LIMITATION 1")
The cutover is code-complete but **must be live-validated against a real gateway** (VPN to
`wonder-sql-vd03`, gateway running the prerequisites above) before it is merged or trusted. Run the
env-gated suite:
**No longer a limitation.** The cutover was live-validated against the real gateway on
`wonder-sql-vd03`: read ✅, write-persist ✅, alarm-send ✅, alarm-readback skipped as a *confirmed
protocol limitation* (the captured `CM_EVENT` wire never carries `SourceName`, so the gateway cannot
populate `Source_Object` — not a fixable bug). Evidence:
`docs/plans/2026-06-27-otopcua-historian-followups.md`.
Kept here because it is the recipe to **re-run** the gate after any gateway or backend change (VPN to
`wonder-sql-vd03`, gateway running the prerequisites above):
```bash
export HISTGW_GATEWAY_ENDPOINT=https://wonder-sql-vd03:5222 # absolute gateway URI; absent ⇒ all live tests skip
@@ -416,7 +463,7 @@ dotnet test --filter "Category=LiveIntegration"
The live suite **skips cleanly** when these env vars are absent (safe to run offline on macOS). It is the
gate the operator runs on the VPN before trusting the cutover.
### KNOWN LIMITATION 2 — continuous-historization value-capture wired; live end-to-end verification still pending
### KNOWN LIMITATION 2 — continuous-historization value-capture wired; live end-to-end verification still pending (issue #491)
The `ContinuousHistorizationRecorder` is fully wired (actor + FasterLog outbox + gateway value-writer +
meters). It is spawned with an **empty *initial* ref set** (`Array.Empty<string>()` in
@@ -430,4 +477,4 @@ deployed set of historized value tags from the first deploy onward (the feed is
path is code-complete.** What remains is the **live gate**: an end-to-end verification against a real gateway
(deploy → dependency-mux value delta → outbox append → `WriteLiveValues`) plus a restart-convergence test
proving the recorder re-registers the deployed set after a process restart. Until that live verification runs,
treat continuous value-capture as unproven-in-production rather than absent.
treat continuous value-capture as unproven-in-production rather than absent. Tracked as **issue #491**.
Directory.Packages.props
+16 -4
View File
@@ -29,10 +29,10 @@
<PackageVersion Include="coverlet.collector" Version="6.0.4" />
<PackageVersion Include="FluentAssertions" Version="8.3.0" />
<PackageVersion Include="Google.Protobuf" Version="3.34.1" />
<PackageVersion Include="Grpc.AspNetCore" Version="2.76.0" />
<PackageVersion Include="Grpc.Core.Api" Version="2.76.0" />
<PackageVersion Include="Grpc.Net.Client" Version="2.76.0" />
<PackageVersion Include="libplctag" Version="1.5.2" />
<PackageVersion Include="LiteDB" Version="5.0.21" />
<PackageVersion Include="MessagePack" Version="2.5.301" />
<PackageVersion Include="Microsoft.AspNetCore.Authorization" Version="10.0.7" />
<PackageVersion Include="Microsoft.AspNetCore.DataProtection" Version="10.0.7" />
@@ -57,6 +57,7 @@
<PackageVersion Include="Microsoft.EntityFrameworkCore.InMemory" Version="10.0.7" />
<PackageVersion Include="Microsoft.EntityFrameworkCore.SqlServer" Version="10.0.7" />
<PackageVersion Include="Microsoft.FASTER.Core" Version="2.6.5" />
<PackageVersion Include="Microsoft.Extensions.Configuration" Version="10.0.7" />
<PackageVersion Include="Microsoft.Extensions.Configuration.Abstractions" Version="10.0.7" />
<PackageVersion Include="Microsoft.Extensions.Configuration.Json" Version="10.0.7" />
<PackageVersion Include="Microsoft.Extensions.DependencyInjection" Version="10.0.7" />
@@ -122,6 +123,13 @@
<PackageVersion Include="xunit.runner.visualstudio" Version="3.0.2" />
<PackageVersion Include="xunit.v3" Version="1.1.0" />
<PackageVersion Include="ZB.MOM.WW.Health" Version="0.1.0" />
<!--
LocalDb: embedded SQLite node-local cache + optional 2-node gRPC replication. The core
package floors the native SQLitePCLRaw.lib.e_sqlite3 at 2.1.12, so consumers inherit the
CVE-2025-6965-fixed native binary without needing the surgical pin below.
-->
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.3" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.3" />
<PackageVersion Include="ZB.MOM.WW.Health.Akka" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Health.EntityFrameworkCore" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Telemetry" Version="0.1.0" />
@@ -129,9 +137,13 @@
<PackageVersion Include="ZB.MOM.WW.MxGateway.Client" Version="0.1.1" />
<PackageVersion Include="ZB.MOM.WW.MxGateway.Contracts" Version="0.1.1" />
<PackageVersion Include="ZB.MOM.WW.Configuration" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Auth.Abstractions" Version="0.1.1" />
<PackageVersion Include="ZB.MOM.WW.Auth.Ldap" Version="0.1.1" />
<PackageVersion Include="ZB.MOM.WW.Auth.AspNetCore" Version="0.1.1" />
<PackageVersion Include="ZB.MOM.WW.Auth.Abstractions" Version="0.1.5" />
<PackageVersion Include="ZB.MOM.WW.Auth.Ldap" Version="0.1.5" />
<PackageVersion Include="ZB.MOM.WW.Auth.AspNetCore" Version="0.1.5" />
<PackageVersion Include="ZB.MOM.WW.Secrets" Version="0.2.3" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.2.3" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.2.3" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.AkkaDotNet" Version="0.2.3" />
<PackageVersion Include="ZB.MOM.WW.Audit" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Theme" Version="0.3.1" />
<PackageVersion Include="ZB.MOM.WW.HistorianGateway.Client" Version="0.3.0" />
NuGet.config
+4
View File
@@ -25,6 +25,10 @@
<package pattern="ZB.MOM.WW.Theme" />
<package pattern="ZB.MOM.WW.HistorianGateway.Contracts" />
<package pattern="ZB.MOM.WW.HistorianGateway.Client" />
<package pattern="ZB.MOM.WW.Secrets" />
<package pattern="ZB.MOM.WW.Secrets.*" />
<package pattern="ZB.MOM.WW.LocalDb" />
<package pattern="ZB.MOM.WW.LocalDb.*" />
</packageSource>
</packageSourceMapping>
</configuration>
docker-dev/docker-compose.yml
+164
View File
@@ -47,6 +47,29 @@
name: otopcua-dev
# ── Shared secret-replication env (all six host nodes) ──────────────────────────
# Akka peer-to-peer clustered secret replication rides the existing `otopcua` Akka
# mesh (DistributedPubSub topic `zb-mom-ww-secrets`) across all six host nodes. Two
# hard requirements are wired here, identically, on every node:
# 1. Secrets__Replication__Enabled=true → SecretsRegistration.AddOtOpcUaSecrets
# switches from plain AddZbSecrets to AddZbSecretsAkkaReplication + the eager
# SecretReplicationStarter hosted service, so every node joins anti-entropy.
# 2. ZB_SECRETS_MASTER_KEY → the ONE shared 32-byte base64 KEK. Every node MUST
# carry the SAME key or peers fail closed decrypting each other's rows
# (kek_id mismatch). appsettings.json binds Secrets:MasterKey:Source=Environment
# / EnvVarName=ZB_SECRETS_MASTER_KEY, so this env var IS the KEK on every node.
# The KEK below is a committed DEV-ONLY default (matches the rig's zero-operator-step
# philosophy, like the committed dev SQL password). Override for a fresh key with:
# OTOPCUA_SECRETS_KEK=$(openssl rand -base64 32) docker compose ... up -d
# NEVER reuse this key outside this local dev rig.
# AnnounceInterval is shortened to 5s (default 30s) so anti-entropy convergence is
# observable quickly during dev exercise. Each node keeps its own local SQLite store
# (Secrets:SqlitePath=otopcua-secrets.db, per-container); replication syncs them.
x-secrets-env: &secrets-env
Secrets__Replication__Enabled: "true"
Secrets__Replication__AnnounceInterval: "00:00:05"
ZB_SECRETS_MASTER_KEY: "${OTOPCUA_SECRETS_KEK:-ZYGhIX0luS/XsevpCB2W18jYHMcqO6AjM9oXy+T6Zp4=}"
services:
sql:
@@ -145,6 +168,7 @@ services:
sql: { condition: service_healthy }
migrator: { condition: service_completed_successfully }
environment:
<<: *secrets-env
OTOPCUA_ROLES: "admin,driver"
ASPNETCORE_URLS: "http://+:9000"
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
@@ -200,8 +224,15 @@ services:
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
# Consolidated pair-local config cache (ZB.MOM.WW.LocalDb). Every driver-role node
# (central is admin,driver) gets one, on a per-node named volume so the cache survives
# container recreates. The central pair leaves replication OFF (default) — only the
# site-a pair replicates, as the enablement demo; site-b is the default-OFF pin.
LocalDb__Path: "/app/data/otopcua-localdb.db"
ports:
- "4840:4840"
volumes:
- otopcua-localdb-central-1:/app/data
central-2:
<<: *otopcua-host
@@ -210,6 +241,7 @@ services:
central-1: { condition: service_started }
migrator: { condition: service_completed_successfully }
environment:
<<: *secrets-env
OTOPCUA_ROLES: "admin,driver"
ASPNETCORE_URLS: "http://+:9000"
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
@@ -258,8 +290,12 @@ services:
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
# Pair-local config cache; central pair leaves replication OFF (see central-1).
LocalDb__Path: "/app/data/otopcua-localdb.db"
ports:
- "4841:4840"
volumes:
- otopcua-localdb-central-2:/app/data
# ── Site A cluster (2-node driver-only) ─────────────────────────────────────
# Driver-only members of the single mesh, scoped to SITE-A by ClusterId. No UI,
@@ -280,6 +316,7 @@ services:
Cluster__PublicHostname: "site-a-1"
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
Cluster__Roles__0: "driver"
<<: *secrets-env
# Quiet EF/AspNetCore SQL flood — see central-1 (Serilog override). mem_limit/
# mem_reservation are inherited from the *otopcua-host anchor.
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
@@ -287,8 +324,49 @@ services:
# Resolved at runtime by GalaxyDriver.ResolveApiKey when a DriverInstance's
# Gateway.ApiKeySecretRef = "env:GALAXY_MXGW_API_KEY".
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
# Pair-local config cache + REPLICATION ON — the site-a pair is the enablement demo.
# site-a-1 is the initiator: it binds the h2c sync listener on 9001 AND dials the peer.
# Setting SyncListenPort makes Program.cs add a dedicated Http2-only Kestrel listener and
# re-bind the primary HTTP port (an explicit Listen* otherwise discards ASPNETCORE_URLS).
# Alarm store-and-forward buffer, Phase 2. Enabled on BOTH halves of the pair so the
# buffer is a live, replicated table rather than an empty one — the live gate needs a real
# queue to watch converge, and needs to see that only the Primary drains it.
#
# There is no HistorianGateway on this rig, so the endpoint below is deliberately
# unresolvable: every drain attempt fails, which is exactly the historian-outage state the
# buffer exists for. Note the endpoint must still be a syntactically valid absolute http(s)
# URI or the host refuses to start — ServerHistorianOptionsValidator is consumer-gated, so
# AlarmHistorian:Enabled=true makes it required even while ServerHistorian:Enabled=false.
#
# MaxAttempts is raised far above the production default of 10. With a permanently
# unreachable gateway the default would dead-letter the whole queue about five minutes into
# the outage, turning a buffering test into a dead-letter test.
AlarmHistorian__Enabled: "true"
AlarmHistorian__MaxAttempts: "1000000"
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
# Required whenever the section is consumed: the gateway client validates its own options at
# construction and throws on an empty key, so a keyless node crash-loops during Akka startup
# rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
# above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
# ServerHistorian__ApiKey from the environment.
ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
# Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
# direction, so an http endpoint with the default UseTls=true crash-loops the host.
ServerHistorian__UseTls: "false"
LocalDb__Path: "/app/data/otopcua-localdb.db"
LocalDb__SyncListenPort: "9001"
LocalDb__Replication__PeerAddress: "http://site-a-2:9001"
# DEV-ONLY committed key (rig philosophy — like the SQL password + secrets KEK above). It
# MUST be byte-identical on both nodes: the interceptor is fail-closed, so any mismatch
# silently stops the pair converging. NEVER reuse this key outside this local dev rig.
LocalDb__Replication__ApiKey: "dev-site-a-localdb-sync-key"
# Row-count batching against gRPC's 4 MB message cap; artifact chunk rows are ≈171 KB, so
# 16 × 171 KB ≈ 2.7 MB stays under the cap with headroom.
LocalDb__Replication__MaxBatchSize: "16"
ports:
- "4842:4840"
volumes:
- otopcua-localdb-site-a-1:/app/data
site-a-2:
<<: *otopcua-host
@@ -304,11 +382,47 @@ services:
Cluster__PublicHostname: "site-a-2"
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
Cluster__Roles__0: "driver"
<<: *secrets-env
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
# site-a-2 is the passive half of the replicating pair: it binds the sync listener on 9001
# but does NOT dial (no PeerAddress). The replication stream is bidirectional, so a-1's
# single dial carries both directions. Same key + MaxBatchSize as a-1 (byte-identical key
# is mandatory — the interceptor fail-closes on a mismatch).
# Alarm store-and-forward buffer, Phase 2. Enabled on BOTH halves of the pair so the
# buffer is a live, replicated table rather than an empty one — the live gate needs a real
# queue to watch converge, and needs to see that only the Primary drains it.
#
# There is no HistorianGateway on this rig, so the endpoint below is deliberately
# unresolvable: every drain attempt fails, which is exactly the historian-outage state the
# buffer exists for. Note the endpoint must still be a syntactically valid absolute http(s)
# URI or the host refuses to start — ServerHistorianOptionsValidator is consumer-gated, so
# AlarmHistorian:Enabled=true makes it required even while ServerHistorian:Enabled=false.
#
# MaxAttempts is raised far above the production default of 10. With a permanently
# unreachable gateway the default would dead-letter the whole queue about five minutes into
# the outage, turning a buffering test into a dead-letter test.
AlarmHistorian__Enabled: "true"
AlarmHistorian__MaxAttempts: "1000000"
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
# Required whenever the section is consumed: the gateway client validates its own options at
# construction and throws on an empty key, so a keyless node crash-loops during Akka startup
# rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
# above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
# ServerHistorian__ApiKey from the environment.
ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
# Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
# direction, so an http endpoint with the default UseTls=true crash-loops the host.
ServerHistorian__UseTls: "false"
LocalDb__Path: "/app/data/otopcua-localdb.db"
LocalDb__SyncListenPort: "9001"
LocalDb__Replication__ApiKey: "dev-site-a-localdb-sync-key"
LocalDb__Replication__MaxBatchSize: "16"
ports:
- "4843:4840"
volumes:
- otopcua-localdb-site-a-2:/app/data
# ── Site B cluster (2-node driver-only) ─────────────────────────────────────
@@ -326,11 +440,32 @@ services:
Cluster__PublicHostname: "site-b-1"
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
Cluster__Roles__0: "driver"
<<: *secrets-env
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
# site-b is the default-OFF pin: it gets the pair-local cache but NO SyncListenPort and NO
# replication config, so no sync listener binds and ISyncStatus stays disconnected/Healthy.
# This is what the live gate's check 8 verifies — the cache works locally with replication off.
# Alarm store-and-forward with replication OFF — the other half of the site-b pin. The sink
# must work as a plain node-local buffer here, with no sync listener and no peer traffic.
AlarmHistorian__Enabled: "true"
AlarmHistorian__MaxAttempts: "1000000"
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
# Required whenever the section is consumed: the gateway client validates its own options at
# construction and throws on an empty key, so a keyless node crash-loops during Akka startup
# rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
# above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
# ServerHistorian__ApiKey from the environment.
ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
# Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
# direction, so an http endpoint with the default UseTls=true crash-loops the host.
ServerHistorian__UseTls: "false"
LocalDb__Path: "/app/data/otopcua-localdb.db"
ports:
- "4844:4840"
volumes:
- otopcua-localdb-site-b-1:/app/data
site-b-2:
<<: *otopcua-host
@@ -346,11 +481,30 @@ services:
Cluster__PublicHostname: "site-b-2"
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
Cluster__Roles__0: "driver"
<<: *secrets-env
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
# site-b default-OFF pin (see site-b-1): cache on, replication off.
# Alarm store-and-forward with replication OFF — the other half of the site-b pin. The sink
# must work as a plain node-local buffer here, with no sync listener and no peer traffic.
AlarmHistorian__Enabled: "true"
AlarmHistorian__MaxAttempts: "1000000"
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
# Required whenever the section is consumed: the gateway client validates its own options at
# construction and throws on an empty key, so a keyless node crash-loops during Akka startup
# rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
# above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
# ServerHistorian__ApiKey from the environment.
ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
# Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
# direction, so an http endpoint with the default UseTls=true crash-loops the host.
ServerHistorian__UseTls: "false"
LocalDb__Path: "/app/data/otopcua-localdb.db"
ports:
- "4845:4840"
volumes:
- otopcua-localdb-site-b-2:/app/data
traefik:
image: traefik:v3.1
@@ -371,3 +525,13 @@ services:
volumes:
# SQL Server data dir — persists the OtOpcUa ConfigDb across container recreates.
otopcua-mssql-data:
# Per-node pair-local config-cache (ZB.MOM.WW.LocalDb) files. One volume per node — the
# cache is node-local, not shared; it persists the cached deployment artifact (and, on the
# site-a pair, the replicated oplog state) across container recreates so a node can boot
# from cache after a restart even with central SQL down.
otopcua-localdb-central-1:
otopcua-localdb-central-2:
otopcua-localdb-site-a-1:
otopcua-localdb-site-a-2:
otopcua-localdb-site-b-1:
otopcua-localdb-site-b-2:
docs/AlarmHistorian.md
+104 -37
View File
@@ -42,7 +42,7 @@ unless noted.
- **`NullAlarmHistorianSink`** — the no-op default for tests and deployments
that don't historize alarms. It is the default DI binding (registered in the
Runtime's `AddOtOpcUaRuntime`); production overrides it with
`SqliteStoreAndForwardSink`.
`LocalDbStoreAndForwardSink`.
- **`AlarmHistorianEvent`**
([`AlarmHistorianEvent.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmHistorianEvent.cs))
— the source-agnostic event record: `AlarmId`, `EquipmentPath` (UNS path,
@@ -61,42 +61,84 @@ unless noted.
- **`HistorianSinkStatus`** — diagnostic snapshot surfaced to the AdminUI and
`/healthz`: `QueueDepth`, `DeadLetterDepth`, `LastDrainUtc`, `LastSuccessUtc`,
`LastError`, `DrainState`, and `EvictedCount`.
- **`HistorianDrainState`** — `Disabled` / `Idle` / `Draining` / `BackingOff`.
- **`HistorianDrainState`** — `Disabled` / `Idle` / `Draining` / `BackingOff` /
**`NotPrimary`** (ticking, but leaving the replicated queue for the node that
holds the Primary role).
---
## SqliteStoreAndForwardSink
## LocalDbStoreAndForwardSink
[`SqliteStoreAndForwardSink.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs)
is the production `IAlarmHistorianSink`. Construction takes a SQLite database
path, an `IAlarmHistorianWriter`, a logger, and optional `batchSize` (default
100), `capacity` (default 1,000,000), `deadLetterRetention` (default 30 days),
and a test clock.
[`LocalDbStoreAndForwardSink.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/LocalDbStoreAndForwardSink.cs)
is the production `IAlarmHistorianSink`. Construction takes the node's
`ILocalDb`, an `IAlarmHistorianWriter`, a logger, and optional `batchSize`
(default 100), `capacity` (default 1,000,000), `deadLetterRetention` (default 30
days), a test clock, and a **`drainGate`**.
### The drain gate
`drainGate` is a `Func<bool>` consulted at the top of every tick; `false` skips
the tick entirely, leaving the queue untouched and reporting
`DrainState = NotPrimary`.
It exists because the queue **replicates**. The Secondary holds a full copy of
the Primary's rows, and an ungated drain there would re-deliver every event,
continuously — not just at a failover. Runtime supplies the gate from
`IRedundancyRoleView`, a singleton `DriverHostActor` publishes its
`PrimaryGatePolicy` verdict to on every redundancy snapshot, so the drain agrees
with the inbound-write and native-ack gates by construction.
Two postures are deliberate:
- **Unset (the default) means drain.** So does an unpublished view. A deployment
that runs no redundancy never publishes a role, and defaulting closed would
silently stop its alarm history forever.
- **A gate that throws means "not now", never permission.** Draining on a gate
fault would put a pair into dual delivery exactly when its redundancy signal is
sick.
### Queue table
The sink owns one SQLite table (created on construction, WAL journal mode):
The sink owns one table in the node's consolidated LocalDb — created and
registered for replication by `LocalDbSetup.OnReady`, before any consumer can
resolve the sink:
```sql
CREATE TABLE Queue (
RowId INTEGER PRIMARY KEY AUTOINCREMENT,
AlarmId TEXT NOT NULL,
EnqueuedUtc TEXT NOT NULL,
PayloadJson TEXT NOT NULL, -- JSON-serialized AlarmHistorianEvent
AttemptCount INTEGER NOT NULL DEFAULT 0,
LastAttemptUtc TEXT NULL,
LastError TEXT NULL,
DeadLettered INTEGER NOT NULL DEFAULT 0
CREATE TABLE alarm_sf_events (
id TEXT NOT NULL PRIMARY KEY, -- SHA-256 of payload_json
alarm_id TEXT NOT NULL,
enqueued_at_utc TEXT NOT NULL,
payload_json TEXT NOT NULL, -- JSON-serialized AlarmHistorianEvent
attempt_count INTEGER NOT NULL DEFAULT 0,
last_attempt_utc TEXT NULL,
last_error TEXT NULL,
dead_lettered INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX IX_Queue_Drain ON Queue (DeadLettered, RowId);
CREATE INDEX ix_alarm_sf_events_drain
ON alarm_sf_events (dead_lettered, enqueued_at_utc, id);
```
`EnqueueAsync` does a single `INSERT` on the hot path. To avoid a
`SELECT COUNT(*)` on every enqueue, the sink keeps an in-memory non-dead-lettered
row counter (seeded at startup, kept current by every mutation, and re-synced
from storage every 10,000 enqueues to defend against drift). SQLite writer
contention is handled via `PRAGMA busy_timeout=5000` + WAL so an enqueue/drain
collision waits out the file lock instead of failing fast.
**The primary key is a hash of the payload, not an autoincrement rowid.** Two
reasons. Replication converges last-writer-wins *per primary key*, so two nodes
independently allocating rowid 7 to different alarms would silently overwrite one
another. And an equal-payload key makes the same event arriving twice collapse
into one row — which happens for real, because `HistorianAdapterActor`
default-writes while its redundancy role is unknown and both nodes of a pair
therefore accept the same fanned transition in every boot window. Two genuinely
distinct events cannot collide: `AlarmHistorianEvent` carries a full-precision
timestamp alongside the alarm id, kind, message and user.
Ordering follows from that: the drain reads in `enqueued_at_utc` order (with `id`
as a tiebreak so the ordering is total), because a hashed key carries no
insertion sequence.
`EnqueueAsync` does a single `INSERT … ON CONFLICT DO NOTHING` on the hot path.
To avoid a `SELECT COUNT(*)` on every enqueue, the sink keeps an in-memory
non-dead-lettered row counter (seeded at startup, kept current by every mutation,
and re-synced from storage every 10,000 enqueues to defend against drift — now
doubly warranted, since replication applies the peer's rows straight into the
table where no local code path can observe them). Pragmas and connection
lifetime belong to LocalDb; the sink no longer manages either.
### Drain worker
@@ -104,13 +146,17 @@ collision waits out the file lock instead of failing fast.
`System.Threading.Timer`** (not started automatically — tests drive
`DrainOnceAsync` deterministically). Each tick:
0. Consults the drain gate. A closed gate returns immediately, leaving the queue
untouched and `DrainState = NotPrimary`.
1. Purges aged dead-lettered rows past the retention window.
2. Reads up to `batchSize` non-dead-lettered rows in `RowId` order.
2. Reads up to `batchSize` non-dead-lettered rows in `enqueued_at_utc, id` order.
3. Rows with un-deserializable payloads are dead-lettered immediately (by their
own `RowId`) so they can't stall the queue head.
own `id`) so they can't stall the queue head.
4. The remaining batch is handed to `IAlarmHistorianWriter.WriteBatchAsync`, and
each outcome is applied in one transaction: `Ack` deletes the row,
`PermanentFail` flips its `DeadLettered` flag, `RetryPlease` bumps its attempt
each outcome is applied in one transaction: `Ack` deletes the row (the delete
replicates as a tombstone, so the peer drops its copy and a promoted standby
cannot re-send it), `PermanentFail` flips its `dead_lettered` flag,
`RetryPlease` bumps its attempt
count and leaves it queued. A row whose `AttemptCount` has reached the configured
**`MaxAttempts`** cap (default 10) is dead-lettered automatically on the next drain
tick rather than retried — this breaks infinite retry loops for poison events whose
@@ -131,7 +177,7 @@ an unobserved task exception.
**The durability guarantee is bounded by `capacity` (default 1,000,000 rows).**
When the non-dead-lettered queue reaches capacity, `EnqueueAsync` evicts the
oldest non-dead-lettered rows (oldest `RowId` first) to make room, logs a WARN,
oldest non-dead-lettered rows (oldest `enqueued_at_utc` first) to make room, logs a WARN,
and increments `HistorianSinkStatus.EvictedCount`. Under a sustained historian
outage, accepted alarm events can therefore be dropped before delivery. A
non-zero `EvictedCount` is a data-loss signal that requires operator attention —
@@ -140,7 +186,7 @@ it surfaces silent loss without log scraping.
### Dead-letter + operator recovery
`PermanentFail` and corrupt-payload rows are retained in-place with
`DeadLettered = 1` for the retention window (default 30 days) so operators can
`dead_lettered = 1` for the retention window (default 30 days) so operators can
inspect them before the sweeper purges them. `RetryDeadLettered()` is the
operator action (from the AdminUI) that clears the dead-letter flag and attempt
count on every dead-lettered row, returning them to the regular queue with a
@@ -173,17 +219,28 @@ the `alerts` topic (future).
The real sink is opt-in via the `AlarmHistorian` section of `appsettings.json`.
When `Enabled` is `false` (the default), `AddAlarmHistorian` registers
`NullAlarmHistorianSink` and the feature is dormant. When `Enabled` is `true`,
`AddAlarmHistorian` constructs `SqliteStoreAndForwardSink` and registers
`AddAlarmHistorian` constructs `LocalDbStoreAndForwardSink` and registers
`GatewayAlarmHistorianWriter` as the `IAlarmHistorianWriter`. This section carries
**only** the `Enabled` gate + the SQLite store-and-forward knobs — the downstream
**only** the `Enabled` gate + the store-and-forward knobs — the downstream
gateway connection (endpoint / key / TLS) is sourced from the `ServerHistorian`
section (see [Historian.md](Historian.md)).
section (see [Historian.md](Historian.md)), and the queue's storage location is the
node's consolidated `LocalDb:Path` database (see
[Configuration.md](Configuration.md)).
> **Where the queue lives (changed).** The buffer is a table — `alarm_sf_events` — in the node's
> consolidated LocalDb, not a standalone `alarm-historian.db`. That is what lets it **replicate to
> the redundant pair peer**, so undelivered alarm history survives losing the node holding it.
> Two consequences follow, both covered in the
> [pair-replication runbook](operations/2026-07-20-localdb-pair-replication.md):
> **only the Primary drains** (the Secondary holds a full replica and would otherwise re-send
> everything), and **delivery is at-least-once across a failover** by design. On first boot after
> the upgrade, `AlarmSfLegacyMigrator` copies any existing `alarm-historian.db` across and renames
> it `.migrated`.
```json
{
"AlarmHistorian": {
"Enabled": true,
"DatabasePath": "C:\\ProgramData\\OtOpcUa\\alarmhistorian.db",
"BatchSize": 100,
"DrainIntervalSeconds": 5,
"Capacity": 1000000,
@@ -194,8 +251,7 @@ section (see [Historian.md](Historian.md)).
| Key | Type | Default | Description |
|---|---|---|---|
| `Enabled` | bool | `false` | Enable the SQLite store-and-forward sink (drains to the HistorianGateway `SendEvent` path). `false` → `NullAlarmHistorianSink`. |
| `DatabasePath` | string | `alarm-historian.db` | Path to the SQLite queue file. Created on first use (WAL mode). Set an **absolute** path in production. |
| `Enabled` | bool | `false` | Enable the durable store-and-forward sink (drains to the HistorianGateway `SendEvent` path). `false` → `NullAlarmHistorianSink`. Requires LocalDb to be registered — an enabled sink with no `ILocalDb` fails at resolution rather than silently discarding events. |
| `BatchSize` | int | `100` | Max rows per drain cycle handed to `IAlarmHistorianWriter.WriteBatchAsync`. |
| `DrainIntervalSeconds` | int | `5` | Seconds between drain-worker ticks. |
| `Capacity` | long | `1000000` | Max queued rows before the sink evicts the oldest (data-loss signal via `EvictedCount`). |
@@ -207,6 +263,15 @@ section (see [Historian.md](Historian.md)).
> `RuntimeDb:EventReadsEnabled=true`. The old Wonderware connection keys (`SharedSecret` /
> `AlarmHistorian:Host`/`Port`/`UseTls`/`ServerCertThumbprint`) were pruned.
> **`DatabasePath` was removed.** The queue no longer owns a file. The key is still *read*, by
> `AlarmSfLegacyMigrator`, purely to locate a pre-consolidation `alarm-historian.db` to migrate —
> so leave it in place through the upgrade, then delete it once every node has an
> `alarm-historian.db.migrated` sidecar. A leftover value configures nothing.
> **`DrainState` gained `NotPrimary`.** A gated-off Secondary reports it instead of `Idle`, so a
> rising queue depth there is distinguishable from a stalled drain. Both nodes reporting it means
> nobody is draining — see the runbook.
> Dev and docker-dev deployments leave `Enabled` unset (defaults to `false`) so alarm transitions historize to nowhere unless a HistorianGateway is configured.
---
@@ -221,3 +286,5 @@ section (see [Historian.md](Historian.md)).
- [ScriptedAlarms.md](ScriptedAlarms.md) — the scripted-alarm engine that emits
most events into this sink.
- [ServiceHosting.md](ServiceHosting.md) — the external HistorianGateway backend.
- [operations/2026-07-20-localdb-pair-replication.md](operations/2026-07-20-localdb-pair-replication.md)
— where the queue lives, the Primary-only drain, and the one-time legacy migration.
docs/AlarmTracking.md
+141 -6
View File
@@ -51,6 +51,136 @@ same condition is wired as an **event notifier of every referencing equipment fo
equipment paths**, and the AdminUI `/alerts` page shows **one row per condition** (primary
identity RawPath + condition NodeId) with the equipment list as display metadata.
## Condition event identity fields (what a client reads on the wire)
Every condition event — native and scripted — carries the mandatory `BaseEventType` identity
fields, assigned at materialize time in `OtOpcUaNodeManager.MaterialiseAlarmCondition`. The SDK
does **not** synthesize them on this path (`Create` builds the children from the type definition
but leaves them unset; `ReportEvent` / `InstanceStateSnapshot` copy children verbatim), so they
are set explicitly. Leaving them unset shipped them as **null** on every event — see issues #473
(the `BaseEventType` trio) and #475 (the `ConditionType` classification pair).
| Field | Value | Notes |
|---|---|---|
| `EventType` | the **concrete** materialized type (`TypeDefinitionId`) | e.g. `OffNormalAlarmType`; falls back to `AlarmConditionType` for an unknown authored type. Readable as a *field*, not only via an `OfType` where-clause |
| `SourceNode` | the condition's **own NodeId** — equal to `ConditionId` | The condition **is** the source: an alarm-bearing raw tag materializes only the condition, with no sibling value variable, so there is no other node to point at |
| `SourceName` | the same identifying id string: **RawPath** (native) / **ScriptedAlarmId** (scripted) | Deliberately the *unique* id, **not** the leaf name |
| `ConditionName` | the leaf / display name (e.g. `HR200`) | Where the short human-readable name lives |
| `ConditionClassId` | always **`BaseConditionClassType`** | Part 9's "no condition class modelled" value. Unset shipped `NodeId.Null` (#475) |
| `ConditionClassName` | always **`"BaseConditionClass"`** | Matches `ConditionClassId`. Unset shipped empty text (#475) |
| `Quality` | the condition's **source-data quality** — native tracks the source's connectivity (`Good` / `Bad`); scripted takes the worst of its input tags' qualities (#478) | A pure annotation; never alters Active/Acked/Retain. Unset shipped the accidentally-Good default (#477) — see below |
**Why `BaseConditionClassType` and not `ProcessConditionClassType`.** We hold no per-alarm
classification at the materialize seam, and `ConditionClassId` is a wire contract clients bucket on.
`BaseConditionClassType` is the honest, spec-conformant report of *"this server does not model
condition classes"* — it fixes the real defect (a null that breaks conformant clients) without
asserting a classification we cannot back. `ProcessConditionClassType` — the SDK sample's pick —
was rejected deliberately: it would be *actively wrong* for a Galaxy alarm whose upstream category is
Safety or Diagnostics, trading a detectable null for an undetectable lie. Real per-alarm
classification is a separate future feature: it needs the driver's alarm category, which today lives
only on the runtime `AlarmEventArgs` transition, carried to the deploy-time authored composition that
`MaterialiseAlarmCondition` sees. Until then the `IAlarmSource` doc comment claiming the category
"maps to `ConditionClassName` downstream" describes an intent, not the implementation.
**Why `SourceName` is the id, not the leaf name.** The leaf is ambiguous across devices (`HR200` on
two PLCs collides) and is already carried by `ConditionName`, so the leaf-name option would add no
identity while costing uniqueness. Carrying the id makes the `SourceNode`/`SourceName`/`ConditionId`
triple mutually consistent and unique. This diverges from the loose OPC UA convention that
`SourceName` mirrors the source node's BrowseName; the divergence is intentional.
**Client guidance:** key on **`ConditionId`** — the condition node's own NodeId, which equals
`SourceNode`, and whose identifier is the RawPath for native alarms and the ScriptedAlarmId for
scripted ones. It is the identity ack/confirm/shelve route on. `SourceName` carries the same
identifier and is unique, so it is safe to key on *by itself* — but do **not** compose it with
`ConditionName` (`$"{SourceName}.{ConditionName}"`), because `SourceName` already ends in the
condition's leaf name and the result stutters (`pymodbus/plc/HR200.HR200`).
Wire-level guard: `NativeAlarmEventIdentityFieldDeliveryTests` asserts the three `BaseEventType`
fields arrive populated on a real subscription using the standard `[EventType, SourceNode,
SourceName, Time, Message, Severity]` select clause, and — in a second test with its own clause —
that `ConditionClassId` / `ConditionClassName` do too. The two class fields are declared on
`ConditionType`, **not** `BaseEventType`, so a client must select them against that type.
`NodeManagerAlarmSourceFieldsTests` guards the node itself across both realms.
> **Do not correlate live events to HistoryRead on `SourceName` — the two paths disagree.**
> The HistoryRead *events* projection (`OtOpcUaNodeManager.ProjectEventField`) returns
> `Variant.Null` for `EventType` / `SourceNode` **by design**: it projects from the historian's
> `HistoricalEvent` rows, which do not carry them. It **does** project `SourceName` — but the
> alarm-history writer stamps that field with the **EquipmentPath**
> (`AlarmEventMapper`: `SourceName = alarm.EquipmentPath`), not the RawPath a live event carries.
> So `SourceName` is the one field populated on both paths **with different values**, and it is not
> a live↔history join key. Correlate on `ConditionId` / the RawPath instead. (Pre-existing; the
> live-path fix above does not change the history path.)
### Condition source-data Quality (#477)
`ConditionType.Quality` reports the quality of the condition's source data. It was never assigned, and
because `StatusCodes.Good == 0x00000000` an unassigned `StatusCode` **is** `Good` — so every condition
reported `Good` unconditionally (a wrong *value*, not a null like #473/#475). A native alarm whose device
went offline still read `Good`, so an operator could not tell *"genuinely inactive"* from *"we have lost
contact and do not know"*.
**How it is driven now (native alarms).** An alarm-bearing raw tag materializes a condition with **no
sibling value variable**, so the value/quality path (`WriteValue`) never touches it, and a comms-lost
driver emits **no alarm transitions** (the feed goes silent). The quality therefore comes from the
**driver's connectivity**, out of band from alarm transitions:
- `DriverInstanceActor` Tells its host `ConnectivityChanged(driverInstanceId, connected)` on every
transition into `Connected` (`true`) / `Reconnecting` (`false`).
- `DriverHostActor.OnDriverConnectivityChanged` fans that out to **every** native condition the driver
owns as an `OpcUaPublishActor.AlarmQualityUpdate` (`Good` on connect, `Bad` on disconnect).
- `OtOpcUaNodeManager.WriteAlarmQuality` sets **only** the condition's `Quality` and fires a Part 9
event **only on a quality-bucket change** — it never touches Active/Acked/Severity/Retain (an active
alarm that loses comms stays active). This is a dedicated path, *not* a full-snapshot re-projection, so
it cannot clobber a condition's severity/message and works for a condition that never fired a transition.
- A freshly materialized native condition starts `BadWaitingForInitialData` (the "no driver data yet"
convention value variables use); the first `Connected` confirms it `Good`.
- The connectivity annotation is **ungated by redundancy role** (a Secondary keeps its condition quality
warm for failover) and publishes **no `/alerts` row** — driver comms health already has its own status
surface (`IDriverHealthPublisher`); a row per condition would be alarm-fatigue.
**Scripted alarms (Layer 3, #478).** A scripted condition's state is computed from one or more input tags,
so its `Quality` is the **worst** quality across those inputs at evaluation time ("can I trust this
condition's state?") — mirroring the native OT semantic:
- The mux now forwards each input's source quality (`DependencyValueChanged.Quality`), and the scripted host
pushes it into the engine's read cache (previously every mux value was treated as `Good`).
- The `ScriptedAlarmEngine` computes the worst input quality each evaluation. A **real transition** carries
it on the emitted event → `ScriptedAlarmHostActor.ToSnapshot` projects it (so a transition fired while an
input is `Uncertain` does not clobber quality back to `Good`).
- A **Bad input freezes the condition** (`AreInputsReady` holds its state — no transition), exactly like a
comms-lost native driver. So a quality-bucket change with no transition is emitted as
`EmissionKind.QualityChanged` and routed to the **same** dedicated `AlarmQualityUpdate → WriteAlarmQuality`
node path native uses (quality only, one Part 9 event on a bucket change, **no `/alerts` row**, no
historian write). `ScriptedAlarmSource` skips `QualityChanged` so it never fabricates a phantom
`IAlarmSource` event.
- An input that has **not been published yet** (cold start) is *not* a quality signal (that is the readiness
guard's job) — it contributes `Good`, so scripted conditions don't flash `Bad` at every deploy. The first
actually-`Bad` published value flips the bucket and annotates.
**Coverage boundary (#478 as shipped).** Scripted quality tracks input tags whose driver **publishes a
data change carrying a Bad/Uncertain `StatusCode`** (e.g. an OpcUaClient input forwarding a server's
per-item Bad). It does **not** yet cover a driver **comms loss**: a poll driver (Modbus/S7) whose device
goes unreachable emits only `ConnectivityChanged` and goes *silent* on the value feed (see
`DriverInstanceActor.Reconnecting`), so the scripted engine keeps the last-known Good value and the
condition stays `Good`. Bridging driver connectivity into scripted inputs — the symmetric of the native
`OnDriverConnectivityChanged` path above, plus resolving the null-value/cold-start asymmetry (a runtime
`Bad` with a null value is currently indistinguishable from cold start and contributes `Good`) — is tracked
as the Layer-4 follow-up (#481).
Guards: `ScriptedAlarmEngineTests` (transition carries `Uncertain`; `Bad` input with no transition emits
`QualityChanged(Bad)`; restore emits `QualityChanged(Good)`; unchanged bucket emits nothing),
`ScriptedAlarmSourceTests.QualityChanged_emission_raises_no_alarm_event`,
`DependencyMuxActorTests.Publish_quality_is_forwarded_on_DependencyValueChanged`, and
`ScriptedAlarmHostActorTests` (`Bad_quality_dependency_publishes_AlarmQualityUpdate_and_no_alerts`,
`Transition_snapshot_carries_worst_input_quality`).
Wire-level guard: `NativeAlarmEventIdentityFieldDeliveryTests.Condition_event_Quality_tracks_source_connectivity_on_the_wire`
subscribes with a `[Quality, Message]` clause (Quality is declared on `ConditionType`) and asserts a
healthy source reports `Good`, a comms-lost source reports non-`Good`, and recovery returns to `Good` — on
a real client subscription. `NodeManagerAlarmSourceFieldsTests` guards the node itself + the no-clobber /
unknown-node-no-op invariants.
## Galaxy driver path (driver-native)
Restored in PR B.2 of the epic. `GalaxyDriver` implements
@@ -235,12 +365,16 @@ AB CIP ALMD) route to AVEVA Historian via the HistorianGateway:
- `IAlarmHistorianSink` is the DI-registered intake contract. The
default binding is `NullAlarmHistorianSink` (registered in
`ServiceCollectionExtensions.AddOtOpcUaRuntime`). Production
deployments override it with `SqliteStoreAndForwardSink` wrapping
deployments override it with `LocalDbStoreAndForwardSink` wrapping
`GatewayAlarmHistorianWriter` (the HistorianGateway `SendEvent` path)
— see [ServiceHosting.md](ServiceHosting.md) for the HistorianGateway setup.
- `SqliteStoreAndForwardSink` queues each transition to a local
SQLite database and drains in the background via an
`IAlarmHistorianWriter`. **The durability guarantee is bounded**: the
- `LocalDbStoreAndForwardSink` queues each transition into the node's
consolidated LocalDb — where it **replicates to the redundant pair peer**,
so undelivered alarm history survives losing the node holding it — and
drains in the background via an `IAlarmHistorianWriter`. **Only the node
holding the Primary role drains**, since the peer holds a full replica;
delivery is consequently at-least-once across a failover, by design. See
[AlarmHistorian.md](AlarmHistorian.md). **The durability guarantee is bounded**: the
queue capacity defaults to 1,000,000 rows; under a sustained
historian outage, older non-dead-lettered rows are evicted (oldest
first) to make room for new events. The `HistorianSinkStatus.EvictedCount`
@@ -249,8 +383,9 @@ AB CIP ALMD) route to AVEVA Historian via the HistorianGateway:
capacity, and dead-letter retention are tunable via the `AlarmHistorian`
config section (`DrainIntervalSeconds`, `Capacity`,
`DeadLetterRetentionDays`); `AlarmHistorianOptions.Validate()` logs a
startup warning for an empty `SharedSecret`, a relative `DatabasePath`,
or a non-positive knob.
startup warning for a non-positive knob. (It no longer warns about
`SharedSecret` or `DatabasePath` — both keys are gone: the connection
comes from `ServerHistorian`, and the queue lives in the LocalDb.)
- `HistorianAdapterActor`
(`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/HistorianAdapterActor.cs`)
subscribes to the cluster `alerts` DPS topic, translates each
docs/Configuration.md
+25 -2
View File
@@ -136,8 +136,9 @@ The historian backend is the external **`ZB.MOM.WW.HistorianGateway`** sidecar,
`ZB.MOM.WW.HistorianGateway.Client` gRPC package (the retired Wonderware TCP sidecar is documented at
[`docs/drivers/Historian.Wonderware.md`](drivers/Historian.Wonderware.md)). The OtOpcUa host reads three
appsettings sections — `ServerHistorian` (read path + gateway connection), `ContinuousHistorization`
(FasterLog outbox + recorder draining to `WriteLiveValues`), and `AlarmHistorian` (SQLite store-and-forward
alarm sink draining to `SendEvent`). The gateway connection (endpoint / key / TLS) lives **only** in
(FasterLog outbox + recorder draining to `WriteLiveValues`), and `AlarmHistorian` (the store-and-forward
alarm sink draining to `SendEvent`; its buffer lives in the node's `LocalDb:Path` database, not a
file of its own). The gateway connection (endpoint / key / TLS) lives **only** in
`ServerHistorian`; the other two sections source it from there.
The gateway API key is supplied via the environment variable **`ServerHistorian__ApiKey`** — never committed
@@ -146,6 +147,28 @@ key must carry the scopes `historian:read`, `historian:write`, `historian:tags:w
[`docs/Historian.md`](Historian.md) for the full key reference, the migration note (old Wonderware keys →
gateway keys), and the deployment prerequisites.
### `LocalDb` (pair-local config cache — driver role only)
Driver-role nodes keep a consolidated **`ZB.MOM.WW.LocalDb`** SQLite database that caches the deployed
configuration artifact (chunked) so the node can **boot from cache when central SQL is unreachable**. The
section is bound by `AddOtOpcUaLocalDb` (`Host/Configuration/LocalDbRegistration.cs`) inside the `hasDriver`
branch — **admin-only nodes never register it, and never require `LocalDb:Path`**. Storage is unconditional;
replication stays inert until the sync port + peer are set.
| Key | Default | Meaning |
|---|---|---|
| `LocalDb:Path` | `./data/otopcua-localdb.db` | SQLite file path. `ValidateOnStart`-required once `AddZbLocalDb` runs (driver nodes only). In containers put it on a durable volume. |
| `LocalDb:SyncListenPort` | `0` | `0` = replication off (no sync listener). A non-zero port binds a dedicated **h2c** sync listener; set the **same** port on both pair nodes. Setting it makes the Host add an explicit Kestrel `Listen*` and re-bind the primary HTTP port (an explicit `Listen*` otherwise discards `ASPNETCORE_URLS`). |
| `LocalDb:Replication:PeerAddress` | `""` | The peer this node **dials** (`http://<peer>:<port>`). Set on the initiator only; leave empty on the passive node. The stream is bidirectional. |
| `LocalDb:Replication:ApiKey` | `""` | Bearer token for the sync stream. **Fail-closed:** no key ⇒ the passive endpoint refuses everything; a mismatch ⇒ the pair silently stops converging. **Must be byte-identical on both nodes.** Supply via `${secret:...}` / env — never a cleartext literal in production. |
| `LocalDb:Replication:MaxBatchSize` | (library default) | Rows per replication batch — **row-count-only** against gRPC's 4 MB cap. `16` for the artifact cache (chunk rows ≈171 KB, so 16 × 171 KB ≈ 2.7 MB). |
Replication is **default-OFF** across the fleet; it is enabled per-pair as an opt-in. See
[`docs/operations/2026-07-20-localdb-pair-replication.md`](operations/2026-07-20-localdb-pair-replication.md)
for the enablement runbook (ApiKey handling, stop/start-together rule, tombstone-retention window, and the
never-`sqlite3`-a-live-WAL-DB inspection rules) and the "pair-local config cache" section of
[`docs/Redundancy.md`](Redundancy.md) for what boot-from-cache does and does not cover.
---
## Environment variables
docs/README.md
+1 -1
View File
@@ -30,7 +30,7 @@ The project was originally called **LmxOpcUa** (a single-driver Galaxy/MXAccess
| [Subscriptions.md](v1/Subscriptions.md) | Monitored items → `ISubscribable` + per-driver subscription refcount (v1 archive) |
| [AlarmTracking.md](AlarmTracking.md) | `IAlarmSource` + `AlarmSurfaceInvoker` + OPC UA alarm conditions — native Galaxy alarms end-to-end (live) |
| [AlarmTracking.md](v1/AlarmTracking.md) | Original alarm-tracking write-up (v1 archive) |
| [AlarmHistorian.md](AlarmHistorian.md) | `Core.AlarmHistorian` store-and-forward SQLite sink — `SqliteStoreAndForwardSink`, `IAlarmHistorianWriter`, dead-letter/retry/eviction |
| [AlarmHistorian.md](AlarmHistorian.md) | `Core.AlarmHistorian` store-and-forward sink — `LocalDbStoreAndForwardSink`, the replicated `alarm_sf_events` buffer + Primary-only drain, `IAlarmHistorianWriter`, dead-letter/retry/eviction |
| [DataTypeMapping.md](v1/DataTypeMapping.md) | Per-driver `DriverAttributeInfo` → OPC UA variable types (v1 archive — live mapping is in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DataTypeMap.cs`) |
| [IncrementalSync.md](IncrementalSync.md) | Address-space rebuild on redeploy + `sp_ComputeGenerationDiff` |
| [HistoricalDataAccess.md](v1/HistoricalDataAccess.md) | `IHistoryProvider` as a per-driver optional capability (v1 archive) |
docs/Redundancy.md
+50
View File
@@ -69,6 +69,23 @@ Roles come from `RedundancyStateActor.BuildSnapshot`: a node with the `driver`
role is `Primary` when it holds the `driver` role-leader lease, otherwise
`Secondary`; a node without the `driver` role is `Detached`.
> **KNOWN LIMITATION — the election is per *Akka* cluster, not per application `Cluster`.**
> `RoleLeader("driver")` yields exactly **one** Primary across the whole Akka cluster. A fleet that
> runs several application clusters (each with its own redundant pair) inside one Akka cluster —
> which is what `docker-dev/docker-compose.yml` models, with MAIN, SITE-A and SITE-B — therefore has
> one Primary among *all* its driver nodes, not one per pair. Every consumer of the role inherits
> this: `ServiceLevel`, the `alerts` emit gate, and the inbound device-write gate below.
>
> The unit of redundancy everywhere else in the product is the application `Cluster` (`ClusterId`) —
> it owns drivers, devices and `ClusterNode` rows, and the LocalDb deployment-artifact cache is
> already keyed by it *so that a pair shares one entry*. Scoping the election the same way is the
> fix; `RedundancyStateActor` is an admin-role singleton in the same assembly as
> `AdminOperationsActor`, which already reads and groups `ClusterNodes` by cluster.
>
> Surfaced by the LocalDb Phase 2 live gate, where it stopped the alarm-history drain on every node
> in the fleet — see `docs/plans/2026-07-20-localdb-phase2-live-gate.md`. The alarm drain no longer
> depends on this (it defers only to a node that shares its queue), but the other three gates still do.
## Data flow
```
@@ -221,11 +238,44 @@ Under warm/hot redundancy both cluster nodes run `ScriptedAlarmHostActor` and ev
- **`alerts` topic emission** — `ScriptedAlarmHostActor` and `DriverHostActor.ForwardNativeAlarm` subscribe to the `redundancy-state` DPS topic and cache the local node's `RedundancyRole`, then gate the cluster `alerts` publish through `PrimaryGatePolicy` (table above). The OPC UA condition-node write and inbound ack/shelve command processing remain **ungated** on both nodes so the secondary is always ready to serve clients after a failover.
- **`HistorianAdapterActor` historization** — likewise Primary-gated so alarm historization is exactly-once across all alarm sources. The actor subscribes to the `alerts` DPS topic and translates each `AlarmTransitionEvent` → `AlarmHistorianEvent` before enqueuing it on the sink; scripted alarms therefore historize exactly once regardless of cluster size.
- **The alarm sink's DRAIN** — a second, independent gate on the same decision. The enqueue gate above is not sufficient any more: since Phase 2 the store-and-forward buffer lives in the replicated LocalDb, so the Secondary holds a full copy of the Primary's queued rows and an ungated drain there would re-deliver every event continuously. `LocalDbStoreAndForwardSink` takes a `drainGate` fed from `IRedundancyRoleView` — a singleton `DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, because the drain runs on a timer and cannot receive `RedundancyStateChanged` itself. A gated-off node reports `HistorianDrainState.NotPrimary`. Delivery is **at-least-once across a failover** by design: rows the old Primary delivered just before dying may not have replicated their deletes yet. See [AlarmHistorian.md](AlarmHistorian.md).
Net effect: each alarm transition appears **once** on `/alerts` and would historize once, not once per node.
See [ScriptedAlarms.md](ScriptedAlarms.md) and [AlarmTracking.md](AlarmTracking.md) for the scripted-alarm engine internals.
## Pair-local store (LocalDb — Phases 1 + 2)
Independently of the ServiceLevel machinery above, every **driver-role** node keeps a consolidated
[`ZB.MOM.WW.LocalDb`](../CLAUDE.md) SQLite database. Phase 2 added the **alarm store-and-forward
buffer** (`alarm_sf_events`) to it alongside the config cache, so a node that dies holding
undelivered alarm history no longer takes it to the grave — see the Primary-only drain note above
and [AlarmHistorian.md](AlarmHistorian.md). Phase 1 caches the **deployed-configuration
artifact** (chunked). Its job is a single failure mode redundancy does not otherwise cover: a driver
node restarting into a **central-SQL-Server outage** would come up with no configuration at all. With
the cache, it **boots from the last artifact it applied** instead, and logs a running-from-cache
signal.
- **What replicates.** The cache can *optionally* replicate to the node's redundant pair peer over
the LocalDb library's gRPC sync (HLC-stamped, last-writer-wins). Two tables replicate:
`deployment_artifacts` and `deployment_pointer`. Replication is **default-OFF and fail-closed** —
it stays inert until `LocalDb:SyncListenPort` + a peer are configured, and a missing/mismatched
`LocalDb:Replication:ApiKey` refuses or silently halts sync. It is enabled per-pair as an opt-in.
- **The replicated-cache payoff.** In a pair with replication on, *either* node can be the one that
applied the latest deploy; the peer holds a byte-identical copy. So a node that restarts during a
central outage can boot the current config **even if it never applied that deploy itself** — its
peer did, and the row replicated.
- **What it does NOT cover.** The cache is a **boot fallback for central-SQL outages only**, not a
standby data path and not a replacement for central. A **new deployment still requires central
SQL**; the cache is read only when the central fetch fails at startup, and the node resumes
fresh-config behavior once central returns. It does not replicate live tag values, alarms, or
historian data — only the config artifact. It is orthogonal to the ServiceLevel/primary-gate
logic: a node running from cache still participates in redundancy normally.
Full detail: the enablement runbook at
[`docs/operations/2026-07-20-localdb-pair-replication.md`](operations/2026-07-20-localdb-pair-replication.md)
and the `LocalDb` section of [`docs/Configuration.md`](Configuration.md).
## Client-side failover
The OtOpcUa Client CLI at `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI` supports `-F` / `--failover-urls` for automatic client-side failover; for long-running subscriptions the CLI monitors session KeepAlive and reconnects to the next available server, recreating the subscription on the new endpoint. See [`Client.CLI.md`](Client.CLI.md).
docs/drivers/Historian.Wonderware.md
+1 -1
View File
@@ -13,7 +13,7 @@ OtOpcUa now consumes the **`ZB.MOM.WW.HistorianGateway`** sidecar through the Gi
- **HistoryRead** → `GatewayHistorianDataSource` over the `ServerHistorian` appsettings section.
- **Alarm history** → `GatewayAlarmHistorianWriter` (the gateway `SendEvent` path) behind the durable
`SqliteStoreAndForwardSink`; alarm-history `ReadEvents` needs the gateway running
`LocalDbStoreAndForwardSink`; alarm-history `ReadEvents` needs the gateway running
`RuntimeDb:EventReadsEnabled=true`.
- **Continuous historization** → a crash-safe FasterLog outbox + `ContinuousHistorizationRecorder`
draining to the gateway's `WriteLiveValues` (`ContinuousHistorization` section); needs the gateway
docs/operations/2026-07-16-secrets-clustered-master-key.md
+177
View File
@@ -0,0 +1,177 @@
# Secrets: Clustered Master-Key Posture (All Roles)
## Purpose
`ZB.MOM.WW.Secrets` resolves `${secret:...}` tokens in `appsettings.*.json` via a
pre-host expander (`SecretReferenceExpander.ExpandConfigurationAsync`, wired in
`src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`) that runs at **every** OtOpcUa node
boot, before the host is built. It reads rows from an envelope-encrypted SQLite
store (`Secrets:SqlitePath`) unwrapped with a key-encryption key (KEK) sourced per
`Secrets:MasterKey:Source`. The runtime `ISecretResolver` (`AddZbSecrets`) is also
registered unconditionally, independent of node role.
OtOpcUa is Akka.NET-clustered (`builder.Services.AddAkka("otopcua", (ab, sp) => { ... })`
in `Program.cs`), with roles parsed by `RoleParser` (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs`):
`admin`, `driver`, `dev`. A node can carry any combination of these roles (e.g. a
fused admin+driver node, or a driver-only node). This runbook covers what a
production deployment needs so that secret resolution behaves identically on
**every node regardless of role** — not just admin nodes. That matters here more
than it might elsewhere: the pre-host expander and the runtime resolver both run
unconditionally on driver-only nodes too, and driver-only nodes are the ones that
will resolve Layer-B `DriverConfig` secret references (coming in Slice 2) at
runtime, not just at boot. It does **not** change any code; it is an
operations/deployment posture, delivered out-of-band from the committed config.
## The two hard requirements
For the pre-host expander (and the runtime resolver) to resolve the same
plaintext secret on every node, no matter its role:
1. **Identical KEK on every node.** All nodes — admin, driver, and any fused
combination — must unwrap the store with the exact same master key. A
per-node KEK (e.g. a per-box DPAPI-protected key) would make each node
decrypt every *other* node's ciphertext rows to garbage.
2. **Identical store rows on every node.** All nodes must read the same SQLite
database (same file, or a replicated/shared copy with the same rows) — not
independently-seeded stores that happen to share a KEK.
`ZB.MOM.WW.Secrets` ships a SQLite-only `ISecretStore` with a `NoOpSecretReplicator`
— there is no built-in cross-node replication today. Meeting both requirements in
production is a deployment concern, covered below.
## Recommended interim posture (G-5)
Until real replication exists (G-7, below), the recommended production posture is:
- **`Secrets:MasterKey:Source = File`**, with `FilePath` pointing at a **read-only
key file that is identical on every node of every role** — a base64-encoded
32-byte key, generated out-of-band (e.g. `openssl rand -base64 32`), distributed
to each node's filesystem/secret-mount by the deployment tooling, and **never
committed** to the repo. Treat it with the same discipline as any other
production secret (restrictive file ACLs, no logging, rotated via a future
KEK-rotation runbook — not yet built).
- **`Secrets:SqlitePath`** pointing at a **single shared or replicated volume**
that every node mounts (admin, driver, and dev/fused alike), so every node's
migrator opens and reads the same rows at boot.
Writes to the store are rare and human-driven — an operator using the
`/admin/secrets` UI (admin nodes only) or the `ZB.MOM.WW.Secrets` CLI — while
reads happen on every node's boot and on the `ResolveCacheTtl` refresh cycle,
regardless of role. The access pattern is read-mostly / effectively
single-writer, which is what makes a shared SQLite volume viable as an interim
posture (see caveat below).
## How it's delivered (do NOT commit these values)
The File-KEK + shared-store posture is supplied per-node at deployment time —
**never** by editing the committed `appsettings.json` or the role-overlay files
(`appsettings.admin.json`, `appsettings.driver.json`, `appsettings.admin-driver.json`).
Those committed overlays are also consumed by local dev and the
`TwoNodeClusterHarness` integration tests, so hardcoding a `Source=File` path
into them would break every dev/test/CI boot. Two acceptable delivery
mechanisms instead:
**Option A — environment variable overrides** (Windows Service / NSSM env block,
container `env_file`, etc.), applied identically on every node regardless of role:
```
# production deployment — do not commit to the dev appsettings
Secrets__MasterKey__Source=File
Secrets__MasterKey__FilePath=/run/secrets/otopcua-master.key
Secrets__SqlitePath=/shared/secrets/otopcua-secrets.db
```
**Option B — a production-only config layer** that is *not* the committed dev
base (e.g. an untracked `appsettings.Production.json` deployed alongside the
binaries, or an orchestrator-injected config mount):
```jsonc
// production deployment — do not commit to the dev appsettings
{
"Secrets": {
"MasterKey": {
"Source": "File",
"FilePath": "/run/secrets/otopcua-master.key"
},
"SqlitePath": "/shared/secrets/otopcua-secrets.db"
}
}
```
Either way, the file/path referenced must exist and be identical on every node
**before** that node boots — the pre-host expander runs unconditionally on every
role and will throw (`SecretNotFoundException` / migration failure) if the store
or key is missing.
## Caveat: SQLite over a shared volume is not real replication
SQLite's file-locking model does not tolerate concurrent multi-writer access well
over network filesystems (SMB/NFS locking is unreliable, and even on a clustered
block volume only one writer should be active at a time). The interim posture
above is acceptable because:
- Reads dominate (every node's boot + cache-refresh cycle, across every role).
- Writes are rare, human-initiated, and effectively single-writer in practice
(an operator runs the CLI/UI against one admin node at a time).
It is **not** a substitute for real replication, and it is not safe if multiple
nodes attempt concurrent writes. Do not build automation that writes secrets
from more than one node simultaneously.
## Data Protection is independent — do not touch it here
OtOpcUa's cookie/session protection already has its own clustered-key story:
`AddDataProtection().PersistKeysToDbContext<OtOpcUaConfigDbContext>()`
(`src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs:73-75`),
which shares the Data Protection key ring across every node via the existing
ConfigDb. That mechanism is unrelated to `ZB.MOM.WW.Secrets`' envelope encryption
(KEK + SQLite store) and must **not** be reconfigured as part of secrets-adoption
work — doing so risks invalidating active sessions for an unrelated reason.
It is, however, the model for where the *next* iteration of secret storage
should go — see the G-7 hand-off below.
## The G-7 hand-off
The posture above is an interim, ops-only workaround. The long-term shape,
tracked as **G-7** in `scadaproj/components/secrets/GAPS.md`, is a
ConfigDb-backed `ISecretStore` that mirrors the pattern OtOpcUa already uses for
the Data Protection key ring:
```csharp
services.AddDataProtection()
.PersistKeysToDbContext<OtOpcUaConfigDbContext>()
.SetApplicationName("OtOpcUa");
```
(`src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs:73-75`).
`OtOpcUaConfigDbContext` already gives every node — admin, driver, and dev/fused
alike — a single MS SQL-backed source of truth for the Data Protection key ring;
the secret store is the natural next tenant of that same shared database instead
of a shared SQLite file. Building this requires new `ZB.MOM.WW.Secrets` library
code (a ConfigDb-backed `ISecretStore` implementation) that does not exist yet,
overlaps the G-7 tracking item, and is explicitly **deferred there** — it is not
built as part of this cut. This runbook's shared-SQLite-volume posture is the
bridge until G-7 lands.
## Dev/test/default posture (unchanged)
The committed default in `appsettings.json` is:
```json
"Secrets": {
"SqlitePath": "otopcua-secrets.db",
"MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" },
"RunMigrationsOnStartup": true
}
```
This is dev-safe: `Source=Environment` needs no filesystem key, and the SQLite
path is relative to the working directory, so local dev, the role-overlay
appsettings (`appsettings.admin.json`, `appsettings.driver.json`,
`appsettings.admin-driver.json`), and the `TwoNodeClusterHarness` integration
tests all boot cleanly with no external mount. The File-KEK + shared-volume
posture in this runbook applies only to real clustered production deployments —
it must never be baked into the committed dev/role-overlay base, because the
expander runs unconditionally at every node boot (any role) and would break
dev/CI if pointed at a nonexistent `/shared` mount.
docs/operations/2026-07-20-localdb-pair-replication.md
+191
View File
@@ -0,0 +1,191 @@
# LocalDb pair replication — operations runbook
> **Scope.** Every driver-role OtOpcUa node keeps a consolidated
> [`ZB.MOM.WW.LocalDb`](../../CLAUDE.md) SQLite database. It holds two things: the
> **deployed-configuration artifact** (chunked, Phase 1) so a node can **boot from cache when
> central SQL Server is unreachable**, and the **alarm store-and-forward buffer** (Phase 2) so a
> node that dies holding undelivered alarm history does not take it to the grave. Both can
> *optionally* replicate to the node's redundant pair peer over the library's gRPC sync.
> **Replication is default-OFF and fail-closed.** This runbook covers enabling it, the operational
> rules that keep a pair converging, and the DB-inspection safety rules.
## What replicates, and what it buys you
- **Replicated tables:** `deployment_artifacts` (the artifact, split into base64 chunks),
`deployment_pointer` (one current-deployment pointer per cluster), and `alarm_sf_events` (the
alarm store-and-forward buffer). Registered — in this order, after the DDL — by
`LocalDbSetup.OnReady`.
- **The payoff:** in a redundant driver pair, *either* node can be the one that applied the latest
deploy and wrote the cache. With replication on, the peer holds a byte-identical copy. So a node
that restarts into a central-SQL outage can boot the current config **even if it never applied
that deploy itself** — its peer did, and the row replicated.
- **Convergence model:** last-writer-wins per primary key, HLC-stamped. Both nodes also store
locally after every apply, so the pair converges to identical content regardless of which node
deployed. Retention (newest-2 deployments per cluster) prunes on one node and the deletes
replicate as tombstones.
## Alarm store-and-forward — what replication changes
Phase 2 moved the alarm buffer out of its own `alarm-historian.db` and into this database. Three
consequences an operator needs to know:
- **Only the Primary drains.** The buffer replicates, so the Secondary holds a full copy of the
Primary's queue. Its drain worker is gated on the same Primary decision the inbound-write and
native-ack gates use, and reports `DrainState = NotPrimary`. **A rising queue depth on a
Secondary is correct.** A rising queue depth on *both* nodes is not — see below.
- **Delivery is at-least-once across a failover, by design.** Rows the old Primary delivered
moments before it died may not have had their deletes replicated yet, so the new Primary will
re-send them. This is accepted, not a defect: a duplicate alarm-history row is recoverable, a
missing one is not. There is deliberately no dedup layer.
- **The buffer is bounded.** `AlarmHistorian:Capacity` (1,000,000 by default) still evicts the
oldest undelivered rows when the historian has been unreachable long enough. Watch
`EvictedCount` — non-zero means accepted alarm events were dropped before reaching the historian.
### If BOTH nodes report `NotPrimary`
Nobody is draining, and the queue is filling toward the capacity ceiling on both. The gate
default-DENIES while the redundancy role is unknown *and* a driver peer exists, so this is what a
redundancy snapshot that never names either node looks like — the identity-mismatch shape
`DriverHostActor` also warns about once ("redundancy snapshot omitted this node"). Check node
identity (`Cluster:PublicHostname` / `Cluster:Port` skew) before suspecting the sink.
### One-time migration from `alarm-historian.db`
On first boot after the upgrade, `AlarmSfLegacyMigrator` copies any pre-existing
`alarm-historian.db` into `alarm_sf_events` and renames the file to `alarm-historian.db.migrated`.
- **Both nodes of a pair migrate independently**, and their files overlap — row ids are derived
from the event payload, so the same event on both nodes converges to one row rather than
duplicating. The new Primary drains the union.
- **The rename happens only after the copy commits.** A failure fails host startup with the legacy
file untouched; the `.migrated` sidecar is what makes a later boot a no-op.
- **A file whose shape is unrecognised is left alone** — not renamed, not copied — so it stays
available for inspection.
- The `AlarmHistorian:DatabasePath` key is **removed**. It is still *read* by the migrator, to
find the legacy file; it no longer configures anything. Delete it from appsettings once the
`.migrated` sidecar exists on every node.
## Enabling replication on a pair
Replication has two knobs, both under `LocalDb`:
| Key | Role | Notes |
|---|---|---|
| `LocalDb:SyncListenPort` | binds the dedicated **h2c** sync listener | `0` (default) = no listener, replication off. Set the **same** non-zero port on **both** nodes of the pair. |
| `LocalDb:Replication:PeerAddress` | the address this node **dials** | Set on **one** node (the initiator); leave empty on the other (passive). The stream is bidirectional — one dial carries both directions. |
| `LocalDb:Replication:ApiKey` | bearer token for the sync stream | **Must be byte-identical on both nodes.** See fail-closed rule below. Supply via `${secret:...}` / env — never a cleartext literal in production config. |
| `LocalDb:Replication:MaxBatchSize` | rows per replication batch | `16` for the artifact cache. Batching is **row-count-only** against gRPC's 4 MB message cap; artifact chunk rows are ≈171 KB, so 16 × 171 KB ≈ 2.7 MB stays under the cap. Raising it risks tripping the cap. |
Example (site-a-1 initiator, site-a-2 passive):
```
site-a-1: LocalDb__SyncListenPort=9001
LocalDb__Replication__PeerAddress=http://site-a-2:9001
LocalDb__Replication__ApiKey=${secret:site-a-localdb-sync-key}
LocalDb__Replication__MaxBatchSize=16
site-a-2: LocalDb__SyncListenPort=9001
LocalDb__Replication__ApiKey=${secret:site-a-localdb-sync-key} # no PeerAddress
LocalDb__Replication__MaxBatchSize=16
```
> **Kestrel note.** Setting `SyncListenPort` makes the Host add an explicit `Listen*` for the h2c
> sync listener. An explicit `Listen*` makes Kestrel **ignore `ASPNETCORE_URLS` entirely**, so the
> Host also re-binds the primary HTTP port in the same block. If you change the primary port,
> verify both `:<httpPort>` and `:<syncPort>` appear in the startup "Now listening on" lines.
### The ApiKey rule (fail-closed)
The replication library's passive endpoint verifies **no** authentication — the host
`LocalDbSyncAuthInterceptor` is the only gate, and it is **fail-closed**:
- **No key configured ⇒ every sync call is refused** (`PermissionDenied`). "No key" is never
"no auth required".
- **A key mismatch ⇒ the pair silently stops converging.** The initiator's stream is rejected at
the peer; nothing errors loudly. A typo in one node's key looks exactly like "replication is
broken". If a pair is not converging, **check the keys match first.**
## Operational rules
- **Stop / start the pair together where you can.** Each node keeps working (and caching locally)
while its peer is down; the outage is not a data-loss event — the surviving node accumulates
writes and the peer catches up on rejoin. But a long-lived solo node drifts further from its
peer, so avoid leaving a pair split for extended periods.
- **Tombstone-retention resurrection window.** Retention prunes to the newest 2 deployments and
replicates the prune as tombstones. Tombstones are themselves retained only for a bounded window.
If a node is offline **longer than the tombstone-retention window**, a delete that happened during
its outage may no longer be expressible as a tombstone on rejoin — a pruned deployment could
briefly reappear until the next deploy re-prunes it. Keep pair outages well inside that window.
- **A rebuilt node is back-filled by its peer.** If a node loses its LocalDb file — a wiped volume,
a re-imaged host, a fresh container — it rejoins with an empty database and its peer snapshots the
cached configuration back to it, with no new deploy and no central SQL. Two things are worth
knowing: this needs **`ZB.MOM.WW.LocalDb` ≥ 0.1.3**, and it heals the *cache*, so the node regains
boot-from-cache for future outages. (On `0.1.1` a converged pair has pruned every oplog row on ack
and the wiped node was never snapshotted — it stayed empty until the next deploy. On `0.1.2` it is
back-filled but its OWN writes are silently dropped until its restarted seq counter climbs past the
peer's stale watermark, so the pair looks converged right up until the moment only the rebuilt node
applies a deploy.) A node
wiped **during** a central outage is still repopulated by its peer under 0.1.2; a node with **no**
peer replication configured self-heals only from central on its next successful apply.
- **What boot-from-cache does NOT cover.** The cache is a *fallback for central-SQL outages at
boot*, not a replacement for central. A **new deployment still requires central SQL** — the cache
is only read when the central fetch fails at startup. When central returns, the node resumes
fresh-config behavior. Boot-from-cache logs a running-from-cache signal; treat a node that stays
on it as a central-connectivity incident, not steady state.
## Inspecting the LocalDb file — safety rules
The database runs in **WAL mode**. These rules exist because violating them corrupted a live DB in
the 2026-07-20 ScadaBridge incident:
- **Never run `sqlite3` on the live file** (host-side, against a bind-mounted or container path).
Opening a live WAL DB from a second process across virtiofs poisons the WAL. **Always copy the
triplet first** and query the copy:
```bash
docker cp <node>:/app/data/otopcua-localdb.db /tmp/localdb.db
docker cp <node>:/app/data/otopcua-localdb.db-wal /tmp/localdb.db-wal
docker cp <node>:/app/data/otopcua-localdb.db-shm /tmp/localdb.db-shm
sqlite3 /tmp/localdb.db 'SELECT cluster_id, deployment_id FROM deployment_pointer;'
```
- **Metrics** come from the container, not a host curl (`aspnet:10.0` has no `curl`):
```bash
docker run --rm --network container:<node> curlimages/curl:latest -s localhost:<httpPort>/metrics | grep localdb_
```
- **If an anomaly appears within seconds of your own measurement, suspect the measurement.**
Restart both nodes and re-observe untouched before blaming the library.
## Useful queries (on a copied triplet)
```sql
-- What each node thinks the current deployment is
SELECT cluster_id, deployment_id, revision_hash, applied_at_utc FROM deployment_pointer;
-- Convergence check: the pointer's origin stamp should match on both nodes
SELECT pk_json, hlc, node_id, is_tombstone
FROM __localdb_row_version WHERE table_name = 'deployment_pointer' ORDER BY pk_json;
-- Replication backlog (should drain to 0 when a pair is caught up)
SELECT COUNT(*) FROM __localdb_oplog;
-- Undelivered alarm history, and how much of it has given up
SELECT dead_lettered, COUNT(*) FROM alarm_sf_events GROUP BY dead_lettered;
-- Why the dead-lettered rows died
SELECT alarm_id, attempt_count, last_attempt_utc, last_error
FROM alarm_sf_events WHERE dead_lettered = 1 ORDER BY last_attempt_utc DESC LIMIT 20;
-- Convergence check for the buffer: identical dumps mean the peer holds the ORIGIN-stamped rows
SELECT pk_json, hlc, node_id, is_tombstone
FROM __localdb_row_version WHERE table_name = 'alarm_sf_events' ORDER BY pk_json;
```
## See also
- [`docs/Redundancy.md`](../Redundancy.md) — the pair-local config cache section.
- [`docs/Configuration.md`](../Configuration.md) — the `LocalDb` appsettings section.
- [`docs/AlarmHistorian.md`](../AlarmHistorian.md) — the alarm sink, its knobs, and the drain.
- The two-node convergence harness: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/`.
docs/plans/2026-07-16-v3-batch4-PR.md
+6 -5
View File
@@ -94,7 +94,7 @@ dual-namespace + event-delivery), Runtime 377 (+ VT reassert regression), AdminU
## Documented follow-ups (non-blocking)
- **Scripted-alarm redeploy recovery (CONFIRMED pre-existing bug, deferred — deserves its own careful fix).**
- **Scripted-alarm redeploy recovery (CONFIRMED pre-existing bug, deferred — deserves its own careful fix). Filed as issue #487.**
The scripted-alarm Part 9 condition node has the same redeploy-reset race the VT `ReassertValue` fix closed:
`RebuildAddressSpace` clears `_alarmConditions` and `MaterialiseScriptedAlarms` recreates each condition
fresh/normal, but `ScriptedAlarmEngine.LoadAsync` reloads from the persisted state and yields
@@ -106,10 +106,11 @@ dual-namespace + event-delivery), Runtime 377 (+ VT reassert regression), AdminU
historian issue). Proposed shape: a host-driven, node-only `AlarmStateUpdate` re-assert in `OnAlarmsLoaded`
(same safe post-materialise ordering as the VT fix), reusing `_engine.GetState(alarmId)` + `LoadedAlarmIds`,
**never touching the `alerts` topic**.
- Raw-tag rename hot-rebind of the renamed tag's live value without a recreate (today: deploy-THEN-recreate).
- Absolute-path Monaco completion → RawPath (from Batch 3; the `{{equip}}/` completion works).
- Raw-tag rename hot-rebind of the renamed tag's live value without a recreate (today: deploy-THEN-recreate). Filed as issue #489.
- Absolute-path Monaco completion → RawPath (from Batch 3; the `{{equip}}/` completion works). Filed as issue #490.
- Cross-repo (post-merge): ScadaBridge Data-Connection-Layer NodeId/namespace cutover (raw `s=<RawPath>` /
UNS `s=<Area>/<Line>/<Equipment>/<EffectiveName>`, retired `EquipmentNodeIds`) + the umbrella index
`../scadaproj/CLAUDE.md` OtOpcUa entry.
UNS `s=<Area>/<Line>/<Equipment>/<EffectiveName>`, retired `EquipmentNodeIds`) — tracked in the
ScadaBridge repo as its issue #14 (its #20 landed the `nsu=` binding + native-alarm routing). The umbrella
index `../scadaproj/CLAUDE.md` OtOpcUa entry is **DONE** (scadaproj `ede5275`).
https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
docs/plans/2026-07-17-alarm-condition-quality-477-design.md
+187
View File
@@ -0,0 +1,187 @@
# Alarm condition Quality (issue #477) — design
**Status:** implemented (L1+L2) · **Date:** 2026-07-17 · **Issue:** #477 (follow-up chain #473 → #475 → #477)
**Scope decision:** Layer 1 + Layer 2, Bad-direct, annotate-only. Layer 3 (scripted worst-of-input) deferred → **#478**.
## Problem
`AlarmConditionState.Quality` is never assigned anywhere in `src/` — neither by
`OtOpcUaNodeManager.MaterialiseAlarmCondition` nor by the `WriteAlarmCondition` transition path.
Because `StatusCodes.Good == 0x00000000`, `default(StatusCode)` **is** `Good`, so the field is
*accidentally valid* — clients parse it, but it reports **`Good` unconditionally regardless of the
backing tag's real quality**.
This is a wrong-*value* bug, not the null-value bug class of #473/#475. Part 9 defines
`ConditionType.Quality` as "the quality of the Condition's source data". OT impact: when a native
alarm's device goes offline (comms lost) the condition still reports `Quality = Good`, so an operator
(or an HMI bucketing on `IsGood`) cannot distinguish *"genuinely not active"* from *"we have lost
contact and do not know"*.
## Why it isn't a 2-line default (confirmed by code)
1. **Alarm-bearing raw tags have no value variable.** `AddressSpaceApplier` materialises a raw tag as
*either* a condition node (`tag.Alarm is not null`) *or* a value variable (`else`) — never both,
since they'd share the same `s=<RawPath>` NodeId. So `WriteValue` (the only path carrying
`OpcUaQuality`) is never invoked for an alarm node. Quality has nowhere to land today.
2. **The alarm channel is quality-blind.** `AlarmEventArgs` (driver → host) and `AlarmConditionSnapshot`
(host → SDK sink) both carry no quality field.
3. **On comms-loss the alarm feed goes silent.** `DriverInstanceActor` on `DisconnectObserved` detaches
the alarm subscription and re-enters `Reconnecting` — no transition event ever arrives to carry Bad.
So the "device offline" signal must come from **driver connectivity**, independently of alarm
transitions.
## Decisions (the issue's open questions)
| # | Question | Decision | Rationale |
|---|----------|----------|-----------|
| 1 | Does an alarm tag get quality today? | No | Confirmed above — new plumbing required. |
| 2 | Direct status code vs. policy map | **Direct Bad** on comms-loss; Good on reconnect | Matches how a value variable would read; unambiguous for `IsGood` bucketing. |
| 3 | Does Bad also suppress transitions / touch Retain? | **No — annotate only** | A comms-lost *active* condition must stay active + retained. Silently clearing an active alarm on comms-loss is the unsafe direction. Quality is a pure annotation; the Active/Ack/Retain state machine is untouched. |
| 4 | Scripted alarms: worst-of-inputs quality? | **Deferred (Layer 3)** | Scripted conditions stay `Good`. Filed as a follow-up issue. |
## Architecture — reuse the existing publish path, add no sink method
The key move: **do not add a new `IOpcUaAddressSpaceSink` method.** A new sink-interface surface would
have to be forwarded through `DeferredAddressSpaceSink` or it is inert on driver hosts (the F10b
prod-inertness trap). Instead the `NativeAlarmProjector` becomes the single owner of per-condition
state *and* quality, and a connectivity change re-projects the *last* snapshot with a swapped quality
through the **existing** `AlarmStateUpdate → OpcUaPublishActor → WriteAlarmCondition` path.
### Layer 1 — make Quality a real, plumbed field
- `AlarmConditionSnapshot` (Commons) gains `OpcUaQuality Quality` (last positional param, default
`OpcUaQuality.Good` so scripted callers and existing tests keep compiling; Commons already knows
`OpcUaQuality` via `IOpcUaAddressSpaceSink`).
- `MaterialiseAlarmCondition` sets `alarm.Quality.Value` at build time:
**native → `BadWaitingForInitialData`** (honest until connectivity confirms Good, matching the
value-variable "waiting for initial data" convention), **scripted → `Good`** (script-computed, always
live in this scope).
- `WriteAlarmCondition` projects `StatusFromQuality(state.Quality)` onto `condition.Quality.Value`
(+ `SourceTimestamp`).
- The delta-gate (`AlarmConditionDelta` / `ReadConditionDelta` / `ToConditionDelta`) gains a `Quality`
member, so a Good→Bad bucket change is a genuine delta and **fires a Part 9 condition event**.
### Layer 2 — drive native quality from driver connectivity
- `DriverInstanceActor`: new `public sealed record ConnectivityChanged(string DriverInstanceId, bool Connected)`.
`Context.Parent.Tell` it on `Become(Connected)` entry (`true`) and on the transitions into
`Reconnecting` (`DisconnectObserved` / `ForceReconnect`) (`false`). Fire-and-forget, mirrors
`DeltaApplied`.
- `NativeAlarmProjector`: per-node state becomes `(bool Active, bool Acked, OpcUaQuality Quality)`.
`Project(transition)` preserves the current quality; new `ProjectQuality(nodeId, quality)` preserves
Active/Acked and swaps only the quality, returning a full snapshot.
- `DriverHostActor`: `Receive<ConnectivityChanged>` iterates `_alarmNodeIdByDriverRef` for that driver
instance and Tells one `AlarmStateUpdate` per condition with the re-projected snapshot
(`connected ? Good : Bad`). **Ungated** — both redundancy nodes track their own driver's comms, matching
the existing "condition write stays ungated (Secondary keeps its address space warm)" rule.
**No `/alerts` row** for a quality-only change — driver health already has its own status/alerts surface
via `IDriverHealthPublisher`; a row here would be alarm-fatigue.
Scripted alarms are unaffected: they are not driver instances, receive no `ConnectivityChanged`, and
their snapshot quality stays `Good`.
## Files
**Layer 1**
- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/AlarmConditionSnapshot.cs`
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs` (`MaterialiseAlarmCondition`,
`WriteAlarmCondition`, `AlarmConditionDelta`/`ReadConditionDelta`/`ToConditionDelta`)
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmHostActor.cs` (`ToSnapshot` — Quality=Good, or rely on default)
**Layer 2**
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs`
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/NativeAlarmProjector.cs`
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs`
## Tests (TDD, RED-first)
1. **Wire-level (the issue's suggested guard)** — extend `NativeAlarmEventIdentityFieldDeliveryTests`
(OpcUaServer.IntegrationTests): active alarm → event `Quality.IsGood`; driver disconnect → condition
event `Quality.IsGood == false`; reconnect → Good. Verify RED against pre-fix.
2. **Node-level** — `NodeManagerAlarmSourceFieldsTests`: materialise sets Quality (native
`BadWaitingForInitialData`, scripted `Good`); `WriteAlarmCondition` projects snapshot quality and
fires on a quality-bucket change only.
3. **`NativeAlarmProjector`** unit: `ProjectQuality` keeps Active/Acked + swaps quality; `Project`
preserves quality.
4. **`DriverInstanceActor`**: `Connected` entry Tells `ConnectivityChanged(true)`; `DisconnectObserved`
Tells `ConnectivityChanged(false)`.
5. **`DriverHostActor`**: `ConnectivityChanged(false)` pushes a Bad-quality `AlarmStateUpdate` to every
condition of that driver instance.
## Deferred / notes
- **Layer 3** (scripted worst-of-input quality) → **Gitea #478**.
- **Implementation note:** L2 uses a **dedicated `IOpcUaAddressSpaceSink.WriteAlarmQuality`** path (not a
full-snapshot re-projection). Rationale: a connectivity change must set *only* Quality; re-projecting a full
snapshot would clobber a cold condition's severity/message and can't annotate a condition that never fired a
transition. The new sink method is forwarded through `DeferredAddressSpaceSink` (the F10b inertness trap) —
auto-verified by `DeferredSinkForwardingReflectionTests` (reflection guard) + its realm-discriminator guard.
- **Test-harness note:** the new `DriverInstanceActor → parent` `ConnectivityChanged` Tell polluted existing
parent-`TestProbe` assertions in 3 `DriverInstanceActor*Tests` files; those tests now
`parent.IgnoreMessages(m => m is ConnectivityChanged)` since they assert on data/alarm/discovery forwards,
not connectivity.
- `Bad_NoCommunication` vs generic `Bad`: v1 maps `OpcUaQuality.Bad → StatusCodes.Bad`; refining
`StatusFromQuality` to emit `BadNoCommunication` for the comms-loss case is a one-line nicety, noted in
the issue.
- `docs/AlarmTracking.md` §"Condition event identity fields" gains a Quality subsection (Good/Bad
semantics, annotation-not-state-change, quality-bucket change fires an event).
## Layer 3 — scripted worst-of-input quality (Gitea #478, implemented 2026-07-17)
**Problem.** A scripted alarm is computed from one or more input tags. Its condition should report the
**worst** quality of those inputs ("can I trust this condition's state?"), not the hardcoded `Good` Layer 1
left at `ScriptedAlarmHostActor.ToSnapshot`.
**Two blockers discovered in the live path (both silently discard quality):**
1. `DependencyMuxActor.OnAttributeValuePublished` builds `VirtualTagActor.DependencyValueChanged` **without**
the `AttributeValuePublished.Quality` it already carries.
2. `ScriptedAlarmHostActor.OnDependencyChanged` pushes each mux value into the engine's upstream with a
**hardcoded `0u` (Good)** StatusCode.
So even a `Bad` driver value reached the scripted engine as Good — Layer 3 has to plumb quality first.
**Design (mirrors Layer 2's native OT semantic through the scripted channel):**
- **Plumb quality end-to-end.** `DependencyValueChanged` gains `OpcUaQuality Quality` (defaulted `Good`, so the
virtual-tag engine's calls are unchanged); the mux forwards `msg.Quality`; `OnDependencyChanged` maps it to a
StatusCode (`Good→0`, `Uncertain→0x40000000`, `Bad→0x80000000`) on the pushed `DataValueSnapshot`.
- **Engine computes the worst input quality** each evaluation (over the refilled read cache, **before** the
`AreInputsReady` short-circuit so a `Bad` input is still observed) and carries it as
`ScriptedAlarmEvent.WorstInputStatusCode` (a raw `uint` StatusCode — `Core.ScriptedAlarms` doesn't reference
Commons, so it stays in the engine's existing StatusCode vocabulary; the host maps it to `OpcUaQuality`).
- **Transitions carry the current worst quality** → `ToSnapshot` projects it (no clobber-back-to-Good when a
transition fires while an input is `Uncertain`).
- **Quality-only changes emit out of band.** A `Bad` input freezes the condition (`AreInputsReady` returns
false → no transition), exactly like a comms-lost native driver — so quality can't ride a transition. The
engine tracks the last worst-quality **bucket** per alarm and, when the bucket changes with **no** transition
emission, emits a new `EmissionKind.QualityChanged` event. The host routes that to the **existing** Layer 2
`OpcUaPublishActor.AlarmQualityUpdate → IOpcUaAddressSpaceSink.WriteAlarmQuality` path (sets ONLY Quality,
one Part 9 event on a bucket change, **no `/alerts` row**, no historian write). No new sink surface.
- **`ScriptedAlarmSource` (the `IAlarmSource` fan-out adapter) skips `QualityChanged`** — quality is delivered
through the dedicated node path, never as a phantom `AlarmEventArgs` (which would materialize/historize a
native condition).
**Files (Layer 3):**
- `src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/Part9StateMachine.cs` — `EmissionKind.QualityChanged`.
- `src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/ScriptedAlarmEngine.cs` — worst-of-input, bucket tracking,
`WorstInputStatusCode` on the event, quality-only emission.
- `src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/ScriptedAlarmSource.cs` — skip `QualityChanged`.
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagActor.cs` — `DependencyValueChanged.Quality`.
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyMuxActor.cs` — forward `msg.Quality`.
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmHostActor.cs` — push real quality;
`ToSnapshot` maps `WorstInputStatusCode`; `OnEngineEmission` routes `QualityChanged → AlarmQualityUpdate`.
**Tests (RED-first):** engine — transition carries `Uncertain` worst; `Bad` input with no transition emits
`QualityChanged(Bad)`; restore emits `QualityChanged(Good)`; no spurious emit when the bucket is unchanged.
`ScriptedAlarmSource` — `QualityChanged` raises no `OnAlarmEvent`. Mux — `DependencyValueChanged` carries the
published quality. Host — `Bad` dependency → `AlarmQualityUpdate(Bad)`, no `/alerts` publish; `ToSnapshot`
maps the event's worst quality.
**Coverage boundary → Layer 4 (#481).** L3 covers inputs whose driver **publishes a Bad/Uncertain-status
data change** (the mux quality path). It does **not** cover a driver **comms loss**: a poll driver
(Modbus/S7) whose device goes unreachable emits only `ConnectivityChanged` and goes silent on the value feed
(`DriverInstanceActor.Reconnecting`), so the scripted engine keeps the last-known Good value and the
condition stays Good — the same silent-feed problem native solved in L2, but native's `OnDriverConnectivityChanged`
bridge fans only to **native** condition nodes (`_alarmNodeIdByDriverRef`), not into the mux the scripted
engine reads. Bridging connectivity into scripted inputs — plus resolving the null-value/cold-start
asymmetry (a runtime `Bad` with a null value is currently indistinguishable from cold start and contributes
`Good`) and its ripple into virtual-tag quality — is **Gitea #481 (Layer 4)**. Found by the post-implementation
code review; the code as shipped faithfully implements #478's written scope (mux-delivered input quality).
docs/plans/2026-07-20-localdb-adoption-phase1.md
+706
View File
@@ -0,0 +1,706 @@
# OtOpcUa LocalDb Adoption — Phase 1 Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
>
> **Execution model:** This plan is optimized for **Claude Opus** agents (`claude --model opus`).
> Dispatch implementer subagents on Opus for every task classified `high-risk`; `small`/`trivial`
> tasks may use Sonnet. Work on branch **`feat/localdb-phase1`** in this repo (`~/Desktop/OtOpcUa`,
> remote `lmxopcua`). Do NOT merge to `master` as part of this plan — stop at the DoD task and
> report.
>
> **Design authority:** `~/Desktop/scadaproj/docs/plans/2026-07-20-otopcua-localdb-design.md`.
> Read it before Task 0. The reference adoption is ScadaBridge (merged PR #23) — when in doubt,
> mirror `~/Desktop/ScadaBridge` (files named per task below).
**Goal:** Give every driver-role OtOpcUa node a consolidated `ZB.MOM.WW.LocalDb` database that caches
the deployed-configuration artifact (chunked), boots from that cache when central SQL Server is
unreachable, and optionally replicates it to the node's redundant pair peer over the library's gRPC
sync (default-OFF, fail-closed bearer auth).
**Architecture:** `AddZbLocalDb`/`AddZbLocalDbReplication` wired in a new `LocalDbRegistration`
(mirroring `SecretsRegistration`), DDL + `RegisterReplicated` in `LocalDbSetup.OnReady`, a dedicated
Kestrel h2c listener for `MapZbLocalDbSync` gated on `LocalDb:SyncListenPort`, a consumer-supplied
fail-closed bearer interceptor, and an `IDeploymentArtifactCache` seam written by `DriverHostActor`
after each successful apply and read as a boot fallback. The dormant LiteDB LocalCache subsystem is
deleted as superseded.
**Tech stack:** .NET 10, `ZB.MOM.WW.LocalDb`/`.Replication`/`.Contracts` **0.1.1** (Gitea feed),
`Grpc.AspNetCore` 2.76.0, Akka.NET (existing), xunit.v3 (Host.IntegrationTests) + xunit2 TestKit
(actor tests).
**Hard rules (from the ScadaBridge adoption — violating any of these is a defect):**
1. Order inside `OnReady` is load-bearing: **DDL → `RegisterReplicated` → writes**. Rows written
before registration are never captured or snapshotted — silently, forever.
2. No autoincrement PKs, no BLOB columns in replicated tables.
3. The library's passive sync endpoint verifies **no** ApiKey — the host interceptor is the only
auth, and it must be fail-closed (no key configured ⇒ deny everything).
4. `ILocalDb.CreateConnection()` returns an **already-open**, pragma-configured connection with the
`zb_hlc_next()` UDF. Never call `Open()` on it. There is **no in-memory mode** — tests use temp
files + `SqliteConnection.ClearAllPools()` before delete.
5. `Grpc.Core.Testing` does not exist on grpc-dotnet — hand-roll fake `ServerCallContext`s.
6. Explicit Kestrel `Listen*` calls make Kestrel **ignore `ASPNETCORE_URLS`** — when adding the sync
listener you must re-bind the existing HTTP port in the same block, and verify it on the rig.
7. Never run host `sqlite3` against a live bind-mounted WAL DB (copy the `db`/`-wal`/`-shm` triplet
with `cp` instead). `aspnet:10.0` containers have no `curl` — use a
`curlimages/curl` sidecar with `--network container:<name>`.
---
### Task 0: Preflight + recon (produces `docs/plans/2026-07-20-localdb-phase1-recon.md`)
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none (everything downstream consumes its findings)
**Files:**
- Create: `docs/plans/2026-07-20-localdb-phase1-recon.md`
- Read-only: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs`,
`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DeploymentArtifact.cs`,
`src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`,
`src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/SecretsRegistration.cs`,
`Directory.Packages.props`, `docker-dev/docker-compose.yml`,
the observability registration (`AddOtOpcUaObservability` implementation),
`src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/*`
**Step 1: Verify the feed packages restore.**
Run:
```bash
cd ~/Desktop/OtOpcUa && dotnet package search ZB.MOM.WW.LocalDb --source dohertj2-gitea --format json | head -50
```
Expected: `ZB.MOM.WW.LocalDb`, `.Replication`, `.Contracts` at `0.1.1`. If absent → STOP, report.
**Step 2: Map and record, with `file:line` citations, each of:**
1. **Deploy fetch path:** where `DriverHostActor` loads `Deployment.ArtifactBlob` (the
`_dbFactory.CreateDbContext()` sites, ~L1472/1543), the artifact's runtime type
(`DeploymentArtifact` — how it's deserialized from the blob), the type/format of
`DeploymentId` and `RevisionHash`, and where a successful apply completes (the point after
which the cache write belongs).
2. **Cold-boot path:** what the actor does at startup before any `DispatchDeployment` arrives —
does it query central SQL for the current deployment? Where does the failure path land
(the `Stale` state, `DbHealthProbeActor` interaction, L1345 dispatch-ignore)? Identify the
exact seam where "central fetch failed at boot" is known — that is where the cache fallback goes.
3. **Cluster identity:** how a driver node knows its `ClusterId` (config key / `ClusterNode` row /
`IClusterRoleInfo`) — the cache is keyed by it.
4. **DriverHostActor construction:** how its dependencies are injected
(`WithOtOpcUaRuntimeActors`, `Props` wiring) so `IDeploymentArtifactCache` can be threaded in.
⚠ `Props.Create` builds an expression tree — adding a parameter silently rebinds
out-of-position named args at call sites; list every construction site.
5. **Telemetry allowlist:** does `AddOtOpcUaObservability` restrict meters
(`ZbTelemetryOptions.Meters` or equivalent)? If yes, record where the list lives.
6. **Kestrel/URLs:** confirm the Host binds via `ASPNETCORE_URLS=http://+:9000` with no
`ConfigureKestrel` calls; record any existing `builder.WebHost` usage.
7. **LiteDB LocalCache:** confirm (grep) that nothing outside
`src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/` and its tests references
`ILocalConfigCache`/`GenerationSealedCache`/`ResilientConfigReader`/`LiteDB`.
8. **docker-dev volumes:** whether nodes have a writable data volume; record what `LocalDb:Path`
should be per node and how env vars are passed (`Cluster__*` style double-underscore).
**Step 3: STOP conditions — halt the plan and report instead of improvising if:**
- the artifact apply path cannot be given a post-apply hook without restructuring the actor;
- `DeploymentId`/cluster identity are not stable strings/GUIDs;
- LiteDB LocalCache turns out to be referenced by live code.
**Step 4: Commit.**
```bash
git add docs/plans/2026-07-20-localdb-phase1-recon.md
git commit -m "docs(localdb): phase-1 recon findings"
```
---
### Task 1: Package references and pins
**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** none (all code tasks build on it)
**Files:**
- Modify: `Directory.Packages.props`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ZB.MOM.WW.OtOpcUa.Runtime.csproj`
**Step 1:** Add to `Directory.Packages.props` (versions exact):
```xml
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.1" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.1" />
<PackageVersion Include="Grpc.AspNetCore" Version="2.76.0" />
```
Check the existing `Grpc.Net.Client`/`Grpc.Core.Api` are already ≥ 2.76.0 and
`Google.Protobuf` ≥ 3.34.1 (the LocalDb.Replication nuspec floors); raise if not. Confirm a
`SQLitePCLRaw` pin ≥ 2.1.12 exists for the transitive `Microsoft.Data.Sqlite` chain (the family
advisory GHSA-2m69-gcr7-jv3q); Core.AlarmHistorian already pins `bundle_e_sqlite3` 2.1.12 — add
`SQLitePCLRaw.lib.e_sqlite3` 2.1.12 as an explicit reference in the Host if the audit flags it.
**Step 2:** Host csproj: `ZB.MOM.WW.LocalDb`, `ZB.MOM.WW.LocalDb.Replication`, `Grpc.AspNetCore`.
Runtime csproj: `ZB.MOM.WW.LocalDb` only (it needs `ILocalDb`, not the sync engine).
**Step 3:** `dotnet build ZB.MOM.WW.OtOpcUa.slnx` → 0 warnings/errors (repo builds warnings-as-errors).
**Step 4: Commit.** `git commit -m "build(localdb): reference ZB.MOM.WW.LocalDb 0.1.1 + Grpc.AspNetCore"`
---
### Task 2: Schema + `LocalDbSetup.OnReady` + registration tests
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 3
**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/DeploymentCacheSchema.cs`
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/LocalDbSetupTests.cs` (create the test project
reference layout to match neighboring Host unit tests; if Host has no unit-test project, place in
the closest existing one and note the deviation)
**Step 1: Write the failing tests first** — using a real temp-file `ILocalDb` (build it via
`new ServiceCollection().AddZbLocalDb(config, LocalDbSetup.OnReady)` with `LocalDb:Path` pointed at
a temp file; dispose + `SqliteConnection.ClearAllPools()` in cleanup):
- `OnReady_RegistersExactlyTheTwoDeploymentTables` — `db.ReplicatedTables.Keys` ordinal-sorted
equals `["deployment_artifacts", "deployment_pointer"]` (assert BOTH directions: no fewer, no more).
- `DeploymentArtifacts_PkIsDeploymentIdPlusChunkIndex` and `DeploymentPointer_PkIsClusterId`
(pin `ReplicatedTable.PkColumns`).
- `RowsWrittenAfterOnReady_EnterTheOplog` — insert a row, assert `SELECT COUNT(*) FROM __localdb_oplog` ≥ 1
(pins the DDL→register ordering; this is the assertion that catches a silently-broken CDC).
**Step 2: Run tests, watch them fail** (types don't exist yet).
**Step 3: Implement.** `DeploymentCacheSchema` depends only on `Microsoft.Data.Sqlite` (the
ScadaBridge `*Schema.Apply` pattern — self-sufficient for direct construction in tests):
```csharp
namespace ZB.MOM.WW.OtOpcUa.Runtime.Deployment;
public static class DeploymentCacheSchema
{
public static void Apply(SqliteConnection connection)
{
using var cmd = connection.CreateCommand();
cmd.CommandText = """
CREATE TABLE IF NOT EXISTS deployment_artifacts (
deployment_id TEXT NOT NULL,
chunk_index INTEGER NOT NULL,
cluster_id TEXT NOT NULL,
revision_hash TEXT NOT NULL,
chunk_count INTEGER NOT NULL,
chunk_base64 TEXT NOT NULL,
cached_at_utc TEXT NOT NULL,
PRIMARY KEY (deployment_id, chunk_index)
);
CREATE TABLE IF NOT EXISTS deployment_pointer (
cluster_id TEXT NOT NULL PRIMARY KEY,
deployment_id TEXT NOT NULL,
revision_hash TEXT NOT NULL,
artifact_sha256 TEXT NOT NULL,
applied_at_utc TEXT NOT NULL
);
""";
cmd.ExecuteNonQuery();
}
}
```
`LocalDbSetup` (Host):
```csharp
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
internal static class LocalDbSetup
{
// ORDER IS LOAD-BEARING: DDL, then RegisterReplicated, then (nothing else in Phase 1).
// Rows written before RegisterReplicated are invisible to the peer forever, silently.
public static void OnReady(ILocalDb db)
{
using var connection = db.CreateConnection(); // already open — do NOT call Open()
DeploymentCacheSchema.Apply(connection);
db.RegisterReplicated("deployment_artifacts");
db.RegisterReplicated("deployment_pointer");
}
}
```
**Step 4:** `dotnet test --filter "FullyQualifiedName~LocalDbSetupTests"` → PASS.
**Step 5: Commit.** `git commit -m "feat(localdb): deployment-cache schema + OnReady registration"`
---
### Task 3: Fail-closed sync auth interceptor + tests
**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 2
**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSyncAuthInterceptor.cs`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/LocalDbSyncAuthInterceptorTests.cs`
Port ScadaBridge's `src/ZB.MOM.WW.ScadaBridge.Host/LocalDbSyncAuthInterceptor.cs` (read it first;
keep behavior identical, adjust namespace only). Semantics to preserve exactly:
- `ServicePrefix = "/localdb_sync.v1.LocalDbSync/"`; any other method path passes through untouched.
- Expected token from `IOptions<ReplicationOptions>.Value.ApiKey`.
- **Fail-closed:** null/empty configured key ⇒ `RpcException(new Status(StatusCode.PermissionDenied, ...))`
for every sync call. Missing/mismatched bearer ⇒ same.
- `CryptographicOperations.FixedTimeEquals` over UTF-8 bytes; header `authorization`,
case-insensitive, `"Bearer "` prefix. Override all four server handler kinds.
**Tests** (hand-rolled fake `ServerCallContext` — `Grpc.Core.Testing` does not exist on grpc-dotnet;
copy ScadaBridge's `LocalDbSyncAuthInterceptorTests.cs` fake): non-sync method passes with no key;
sync + no configured key → PermissionDenied; wrong bearer → PermissionDenied; correct bearer → passes.
Write tests first (fail), implement, `dotnet test --filter "FullyQualifiedName~LocalDbSyncAuthInterceptor"`,
commit `feat(localdb): fail-closed bearer interceptor for the sync endpoint`.
---
### Task 4: `LocalDbRegistration` + Program.cs DI wiring + config defaults
**Classification:** high-risk (touches Program.cs / role gating)
**Estimated implement time:** ~5 min
**Parallelizable with:** none (Task 5 edits the same file)
**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbRegistration.cs`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (inside the `hasDriver` branch)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json`
**Step 1:** `LocalDbRegistration`, mirroring `SecretsRegistration`'s shape:
```csharp
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
internal static class LocalDbRegistration
{
/// Driver-role nodes only. Storage is unconditional; replication stays inert until
/// LocalDb:Replication:PeerAddress (initiator) / LocalDb:SyncListenPort (listener) are set.
public static IServiceCollection AddOtOpcUaLocalDb(
this IServiceCollection services, IConfiguration configuration)
{
services.AddZbLocalDb(configuration, LocalDbSetup.OnReady);
services.AddZbLocalDbReplication(configuration);
return services;
}
public static int SyncListenPort(IConfiguration configuration) =>
configuration.GetValue<int>("LocalDb:SyncListenPort");
}
```
**Step 2:** Program.cs, in the `hasDriver` block (near `AddAlarmHistorian`):
`builder.Services.AddOtOpcUaLocalDb(builder.Configuration);` and
`builder.Services.AddGrpc(o => o.Interceptors.Add<LocalDbSyncAuthInterceptor>());`
(AddGrpc only under `hasDriver` — admin-only nodes expose no sync surface).
**Step 3:** appsettings.json:
```json
"LocalDb": {
"Path": "./data/otopcua-localdb.db",
"SyncListenPort": 0,
"Replication": {}
}
```
`LocalDb:Path` is `ValidateOnStart`-required once `AddZbLocalDb` runs — since registration is
driver-gated, admin-only configs need nothing. Check the role-overlay files
(`appsettings.driver.json` / `appsettings.admin-driver.json`) and any deploy templates for
conflicting `LocalDb` keys.
**Step 4:** Build + full Host unit tests. Verify an admin-only graph doesn't require the key: this
is pinned properly in Task 10's integration tests, but do a quick
`OTOPCUA_ROLES=admin dotnet run` smoke only if cheap; otherwise rely on Task 10.
**Step 5: Commit.** `feat(localdb): wire AddOtOpcUaLocalDb into the driver role`
---
### Task 5: Dedicated h2c sync listener + endpoint mapping (THE Kestrel task)
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** none (Program.cs)
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`
**Why high-risk:** the Host today binds exclusively via `ASPNETCORE_URLS=http://+:9000`. Any
explicit `ConfigureKestrel(... Listen*)` makes Kestrel **ignore URLs entirely** (it logs
"Overriding address(es)"), which would silently kill the AdminUI/deploy API behind Traefik.
**Step 1:** Add, before `builder.Build()`:
```csharp
var syncPort = LocalDbRegistration.SyncListenPort(builder.Configuration);
if (hasDriver && syncPort > 0)
{
// Explicit Listen* replaces ASPNETCORE_URLS wholesale, so re-bind the primary HTTP
// endpoint here too. Parse it from the configured URLs rather than hard-coding 9000.
var urls = builder.Configuration["ASPNETCORE_URLS"] ?? builder.Configuration["urls"] ?? "http://+:9000";
var httpPort = new Uri(urls.Split(';')[0].Replace("+", "localhost").Replace("*", "localhost")).Port;
builder.WebHost.ConfigureKestrel(k =>
{
k.ListenAnyIP(httpPort); // HTTP/1.1 (existing surface)
k.ListenAnyIP(syncPort, o => o.Protocols = HttpProtocols.Http2); // h2c, prior-knowledge gRPC
});
}
```
When `syncPort == 0` (the default) nothing changes — URLs binding stays untouched. Cleartext
`Http1AndHttp2` cannot serve prior-knowledge h2c, hence the dedicated `Http2`-only listener.
**Step 2:** After the app pipeline's other `Map*` calls, driver-gated:
```csharp
if (hasDriver && syncPort > 0)
{
app.MapZbLocalDbSync();
}
```
(Mapping is harmless when unauthenticated — the interceptor fail-closes — but gating on the port
keeps admin-only and default-OFF graphs entirely free of the endpoint.)
**Step 3: Verify both surfaces.**
```bash
dotnet build ZB.MOM.WW.OtOpcUa.slnx
```
Then a local smoke with the port on:
`ASPNETCORE_URLS=http://+:9000 OTOPCUA_ROLES=driver LocalDb__SyncListenPort=9001 dotnet run --project src/Server/ZB.MOM.WW.OtOpcUa.Host ...`
— expect startup logs to show BOTH `:9000` and `:9001` bound, and `curl -s localhost:9000/healthz`
(or the mapped health route) still answering. If the app needs SQL to boot, defer the smoke to the
Task 12 rig check but say so explicitly in the task notes.
**Step 4: Commit.** `feat(localdb): dedicated h2c listener + MapZbLocalDbSync gated on LocalDb:SyncListenPort`
---
### Task 6: `IDeploymentArtifactCache` + chunked LocalDb implementation + tests
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 3, Task 5
**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/IDeploymentArtifactCache.cs`
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/LocalDbDeploymentArtifactCache.cs`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Deployment/LocalDbDeploymentArtifactCacheTests.cs`
**Step 1: Failing tests** (real temp-file `ILocalDb` with `LocalDbSetup.OnReady` applied):
- round-trip: store a 300 KiB random artifact → `GetCurrentAsync` returns byte-identical payload
(forces ≥ 3 chunks at 128 KiB);
- retention: store 3 deployments for one cluster → only the newest 2 remain in
`deployment_artifacts`; pointer names the newest;
- integrity: corrupt one chunk row via SQL → `GetCurrentAsync` returns null (miss), never a
truncated artifact;
- missing pointer → null.
**Step 2: Implement:**
```csharp
public interface IDeploymentArtifactCache
{
Task StoreAsync(string clusterId, string deploymentId, string revisionHash,
byte[] artifact, CancellationToken ct = default);
Task<CachedDeploymentArtifact?> GetCurrentAsync(string clusterId, CancellationToken ct = default);
}
public sealed record CachedDeploymentArtifact(
string DeploymentId, string RevisionHash, byte[] Artifact, DateTimeOffset AppliedAtUtc);
```
`LocalDbDeploymentArtifactCache(ILocalDb db)`:
- `StoreAsync`: one `ILocalDbTransaction` — delete any existing chunks for this `deployment_id`
(idempotent re-store), insert chunks (raw 128 * 1024 bytes per chunk, `Convert.ToBase64String`),
upsert the pointer (`INSERT ... ON CONFLICT(cluster_id) DO UPDATE`), then prune: delete
`deployment_artifacts` rows whose `deployment_id` is not among the newest 2 `cached_at_utc` for
this `cluster_id`. SHA-256 over the raw artifact into the pointer. ISO-8601 UTC timestamps.
- `GetCurrentAsync`: read pointer; read chunks `ORDER BY chunk_index`; verify count ==
`chunk_count` and SHA-256 matches; return null on any mismatch (log a warning naming which check
failed).
- Use `db.ExecuteAsync`/`QueryAsync` with anonymous-object parameters (`@Name` markers). Remember
a `Dictionary<string,string>` parameter **throws** — use anonymous objects.
**Step 3:** run tests → PASS. **Step 4: Commit.**
`feat(localdb): chunked deployment-artifact cache over ILocalDb`
---
### Task 7: Cache write path in `DriverHostActor`
**Classification:** high-risk (actor model, `Props` expression-tree trap)
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:** (exact edit sites come from the Task 0 recon doc — cite it in the commit)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs`
- Modify: the actor-registration site (`WithOtOpcUaRuntimeActors` / DI extension) to provide
`IDeploymentArtifactCache`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` or `LocalDbRegistration` — register
`IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache` singleton (driver branch)
- Test: the **xunit2** TestKit project used for existing DriverHostActor tests (recon names it)
**Steps:**
1. **Failing test:** after a successful artifact apply, the cache received `StoreAsync` with the
applied deployment's id/hash/bytes (inject a recording fake `IDeploymentArtifactCache`).
Also: a cache that throws on `StoreAsync` does NOT fail the apply (apply result unchanged, error
logged).
2. Thread the dependency through the actor's constructor. ⚠ `Props.Create` is an expression tree:
after adding the parameter, re-check **every** construction site for positional/named-arg
rebinding (the recon lists them) — do not rely on the compiler.
3. Invoke `StoreAsync` fire-and-forget (`PipeTo`-style or a guarded `Task.Run` per the actor's
existing async conventions — match whatever pattern the actor already uses for side-effect IO)
at the post-apply point identified in recon. Cluster id from the recon-identified source.
4. Run the actor test suite for this project. Commit:
`feat(localdb): DriverHostActor stores applied artifacts in the pair-local cache`
---
### Task 8: Boot-from-cache read path + running-from-cache signal
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs`
- Test: same xunit2 TestKit project as Task 7
**Steps:**
1. **Failing tests:**
- central fetch fails at startup AND cache has an artifact → the actor applies the cached
artifact (assert via whatever observable the apply already exposes) and logs/flags
running-from-cache;
- central fetch fails AND cache empty → today's behavior exactly (Stale, no apply);
- central fetch **succeeds** → cache is NOT consulted (fresh config wins; assert the fake
cache's `GetCurrentAsync` was never called on the happy path).
2. Implement at the recon-identified boot-failure seam. The signal: a log warning at minimum plus,
if the actor already publishes health/status (recon says how), a `RunningFromCache` marker on it.
Do NOT touch `DispatchDeployment` handling — a new deployment still requires central.
3. Run tests; commit `feat(localdb): boot from the pair-local artifact cache when central SQL is unreachable`.
---
### Task 9: Delete the dormant LiteDB LocalCache subsystem
**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 6, Task 10, Task 11
**Files:**
- Delete: `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/` (entire directory:
`ILocalConfigCache`, `LiteDbConfigCache`, `GenerationSealedCache`, `ResilientConfigReader`,
`GenerationSnapshot`, `StaleConfigFlag`, exceptions)
- Delete: `tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests/GenerationSealedCacheTests.cs`,
`ResilientConfigReaderTests.cs`
- Modify: `Directory.Packages.props` + the Configuration csproj — remove the `LiteDB` package if
nothing else references it (grep first)
**Steps:** Confirm the Task 0 recon's "nothing references it" finding still holds
(`grep -rn "ILocalConfigCache\|GenerationSealedCache\|ResilientConfigReader\|LiteDB" src/ tests/`),
delete, build the full solution, run the Configuration test project. DoD phrasing: **no references
from code** — leave any explanatory prose/comments that point readers at LocalDb instead. Add one
line to the recon doc noting LiteDB's removal. Commit:
`refactor(localdb): delete the dormant LiteDB LocalCache (superseded by ZB.MOM.WW.LocalDb)`
---
### Task 10: Health check + telemetry meter allowlist
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 9, Task 11
**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/LocalDbReplicationHealthCheck.cs` (or the
directory `AddOtOpcUaHealth` uses — recon names it)
- Modify: the `AddOtOpcUaHealth` registration (driver branch)
- Modify: the observability meter list IF Task 0 found an allowlist
**Steps:**
1. Failing unit tests: replication unconfigured (no peer, no listener) → `Healthy` (default-OFF
must not degrade a plain node); peer configured + `ISyncStatus.Connected == false` → `Degraded`;
connected + backlog `null` → `Degraded` (unknown backlog is not healthy); connected + backlog
small → `Healthy`.
2. Implement against `ISyncStatus` + `IOptions<ReplicationOptions>` (peer-configured = non-empty
`PeerAddress` OR `SyncListenPort > 0` — pass the latter in via options/config).
3. **Meter allowlist:** if `AddOtOpcUaObservability` restricts meters, add
`"ZB.MOM.WW.LocalDb.Replication"` (use `LocalDbMetrics.MeterName`) — this is a silent allowlist;
the ScadaBridge live gate is the proof it bites. If there is no allowlist, record that in the
recon doc and skip.
4. Commit `feat(localdb): replication health check + meter export`.
---
### Task 11: DI-pin integration tests over the real Program.cs
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 9, Task 10
**Files:**
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDbWiringTests.cs` (xunit.v3,
`WebApplicationFactory<Program>` — the project already builds the real Program.cs; set
`OTOPCUA_ROLES` per test case the way neighboring tests do)
**Pins (each its own test):**
1. Driver graph: `ILocalDb` resolves, is a singleton, `ReplicatedTables` ordinal-sorted ==
`["deployment_artifacts", "deployment_pointer"]` — exact set, both directions load-bearing.
2. Driver graph: `IDeploymentArtifactCache` resolves to `LocalDbDeploymentArtifactCache`.
3. Default-OFF pin: `ISyncStatus` resolves with `Connected == false`, `PeerNodeId == null`.
4. Admin-only graph: `ILocalDb` is NOT registered (`GetService<ILocalDb>()` is null) and boot does
not demand `LocalDb:Path`.
5. Health: the LocalDb health check is registered in the driver graph.
Point `LocalDb:Path` at a per-test temp file via the factory's config overrides;
`ClearAllPools` + delete in cleanup. These tests exist because DI extensions without a
container-built test have shipped inert three times in this family (Secrets 0.2.0/0.2.2,
ScadaBridge#22). Commit `test(localdb): DI pins over the real host graph`.
---
### Task 12: Two-node convergence harness + tests
**Classification:** high-risk
**Estimated implement time:** ~5 min (harness) + ~4 min (scenarios) — split the commit if needed
**Parallelizable with:** none (depends on Tasks 2, 3, 5, 6)
**Files:**
- Create: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/LocalDbPairHarness.cs`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/LocalDbPairConvergenceTests.cs`
**Harness:** copy the pattern from
`~/Desktop/ScadaBridge/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs` and
the library's `ConvergenceFixture` (`~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/tests/.../Convergence/ConvergenceFixture.cs`):
- Two full stacks over a real loopback Kestrel h2c socket (port 0), **through the real
`LocalDbSyncAuthInterceptor`** with a shared test key; node A initiator, node B passive.
- Both initialized via the **production `LocalDbSetup.OnReady`** — a hand-written schema proves
only that the test agrees with itself.
- DBs owned by the harness, registered as pre-constructed singletons (so MS.DI won't dispose them
on host teardown); `KillTransportAsync`/`RestartTransportAsync`; tight `FlushInterval` (50 ms),
bounded backoff (2 s); temp files, `ClearAllPools` cleanup; a collection definition to serialize
these tests.
**Scenarios:**
1. `ArtifactStoredOnA_ConvergesToB_ByteIdentical` — store a multi-chunk artifact via
`LocalDbDeploymentArtifactCache` on A; B's cache returns byte-identical bytes; both nodes'
`__localdb_row_version` rows for the pointer carry the **same HLC and origin node id** (B holds
A's row, not a re-derived one).
2. `RetentionPruneOnA_TombstonesReachB` — third deployment on A prunes the first; B's chunk rows
for the pruned deployment disappear and `__localdb_row_version WHERE is_tombstone=1` rows exist
on B.
3. `WritesWhileTransportDown_SurviveRejoin` — kill transport, store on A, restart, converge.
4. `WrongApiKey_NeverConverges` — harness with mismatched keys: assert **no** convergence after a
bounded wait AND (positive control) that the same scenario with matching keys converges — an
absence assertion without a positive control passed vacuously in ScadaBridge.
**DoD within this task:** temporarily comment out the two `RegisterReplicated` calls and confirm
scenarios 1–3 go red (run locally, do not commit the red state); restore. Record the red/green
evidence in the task notes. Commit `test(localdb): 2-node convergence harness + scenarios`.
---
### Task 13: docker-dev rig configuration
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 14
**Files:**
- Modify: `docker-dev/docker-compose.yml`
**Steps** (adjust to the recon's findings on volumes/env style):
1. Every driver node: a named volume (or existing data mount) backing `/app/data`, and
`LocalDb__Path=/app/data/otopcua-localdb.db`.
2. **site-a pair only** (mirror the ScadaBridge default-OFF posture — site-b stays unreplicated as
the pin):
- site-a-1: `LocalDb__SyncListenPort=9001`, `LocalDb__Replication__PeerAddress=http://site-a-2:9001`,
`LocalDb__Replication__ApiKey=dev-site-a-localdb-sync-key`, `LocalDb__Replication__MaxBatchSize=16`
- site-a-2: `LocalDb__SyncListenPort=9001`, same `ApiKey`, same `MaxBatchSize`, **no PeerAddress**
(passive; the stream is bidirectional).
The key must be byte-identical on both — the interceptor fail-closes on any mismatch and the
pair silently stops converging.
3. `MaxBatchSize=16` because batching is row-count-only against gRPC's 4 MB cap and artifact chunk
rows are ≈ 171 KB (16 × 171 KB ≈ 2.7 MB worst case).
4. `docker compose config` to validate; do NOT bring the rig up in this task (the live gate task
owns rig runs).
5. Commit `chore(localdb): rig config — consolidated path everywhere, replication on the site-a pair`.
---
### Task 14: Documentation
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 13
**Files:**
- Create: `docs/operations/2026-07-20-localdb-pair-replication.md` (runbook: enable/disable,
ApiKey handling via `${secret:}`, stop/start-together rule, tombstone-retention resurrection
window, MaxBatchSize row-count-vs-4MB, never host-`sqlite3` a live WAL DB / cp-triplet recipe)
- Modify: `docs/Redundancy.md` (a short "pair-local config cache" section: what replicates, what
boot-from-cache does and does not cover)
- Modify: `CLAUDE.md` (this repo): LocalDb adoption row/paragraph — state Phase 1 scope, default-OFF,
rig-only enablement
- Modify: `docs/Configuration.md` if it documents config keys (add the `LocalDb` section)
Commit `docs(localdb): phase-1 runbook + redundancy/config docs`.
---
### Task 15: DoD sweep (offline)
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none (final offline task)
**Steps:**
1. `dotnet build ZB.MOM.WW.OtOpcUa.slnx` → **0 warnings** (warnings-as-errors).
2. `dotnet test ZB.MOM.WW.OtOpcUa.slnx` → full suite green; record counts. Baseline any
pre-existing failures BEFORE this branch (`git stash` → run → unstash) rather than assuming;
report deltas only.
3. Greps (phrase as "no references from code"; explanatory comments may remain):
`grep -rn "LiteDB\|ILocalConfigCache\|GenerationSealedCache" src/ tests/` → no code references.
4. Confirm the positive-control evidence from Task 12 is recorded.
5. Update `docs/plans/2026-07-20-localdb-adoption-phase1.md.tasks.json` statuses.
6. Commit `chore(localdb): phase-1 DoD sweep` and STOP. Report: branch name, commit list, test
counts, and that the live gate (Task 16) needs the rig + operator go-ahead.
---
### Task 16: Live gate on the docker-dev rig (run only with explicit user go-ahead)
**Classification:** high-risk (rig)
**Estimated implement time:** ~30 min wall-clock (mostly waiting)
**Parallelizable with:** none
**Preamble:** All DB inspection via `cp` of the `db`/`-wal`/`-shm` triplet out of the container
(`docker cp`) and querying the copy — never host `sqlite3` on the live file (it poisons the WAL
across virtiofs; root cause of the 2026-07-20 ScadaBridge incident). Metrics via
`docker run --rm --network container:<node> curlimages/curl:latest -s localhost:<port>/metrics`.
If an anomaly appears within seconds of one of your own measurements, suspect the measurement
first — restart both nodes and re-run untouched before blaming the library.
**Checks (all must PASS; record evidence in `docs/plans/2026-07-20-localdb-phase1-live-gate.md`):**
1. Rig up, site-a pair healthy, `/metrics` on both shows `localdb_` series (proves the meter
allowlist work).
2. Deploy a config through the normal deploy API → both site-a nodes' `deployment_pointer` +
`deployment_artifacts` rows are byte-identical with **identical HLC + origin node id** (one
node's row replicated, not two independent derivations — both nodes also locally store after
apply, so expect LWW-converged identical content either way; assert convergence, and note which
origin won).
3. **Boot-from-cache:** stop the SQL container; restart site-a-2; it comes up serving the cached
config with the running-from-cache signal in its log; start SQL again; node recovers fresh
behavior.
4. **Replicated-cache payoff:** wipe site-a-2's local DB file (container stopped), restart it with
SQL **up**, let replication repopulate the cache from site-a-1, then repeat check 3's SQL-down
restart — it boots from a cache it never wrote itself.
5. Transport kill (pause site-a-2 container): store/deploy on a-1, oplog depth rises; unpause;
drains to 0; converged.
6. Retention: deploy a 3rd config; pruned deployment's chunks gone on BOTH nodes with tombstone
rows present on both.
7. Both-nodes-together restart: clean rejoin, identical counts, zero
`disk I/O error`/`SQLITE_IOERR`/`corrupt` in either log.
8. site-b pair (default-OFF pin): LocalDb file exists and works locally, `ISyncStatus` health
Healthy, no sync connections, no listener on 9001.
Record PASS/FAIL per check with the actual evidence (row counts, md5s, log lines). Commit the gate
doc. Do not merge — report.
---
## Task persistence
Tasks file: `docs/plans/2026-07-20-localdb-adoption-phase1.md.tasks.json` (same directory).
Update statuses as tasks complete; any deviation from this plan gets a `deviation` note on the task
(the ScadaBridge tasks.json convention — the record of *why* is what makes the plan auditable).
docs/plans/2026-07-20-localdb-adoption-phase1.md.tasks.json
+153
View File
@@ -0,0 +1,153 @@
{
"planPath": "docs/plans/2026-07-20-localdb-adoption-phase1.md",
"tasks": [
{
"id": 0,
"subject": "Task 0: Preflight + recon (findings doc, STOP conditions)",
"status": "completed",
"notes": "Feed verified: LocalDb/.Contracts/.Replication all 0.1.1. All 3 STOP conditions clear. Findings: docs/plans/2026-07-20-localdb-phase1-recon.md. 8 deviations recorded (D-1..D-8); D-1/D-3/D-5/D-6 change downstream work materially.",
"deviation": "D-1 cache read cannot be keyed by ClusterId at the boot seam (unkeyed newest-pointer read instead); D-3 a third LiteDB test file must be deleted; D-5 WebApplicationFactory<Program> is deliberately unused in this repo (use TwoNodeClusterHarness); D-6 driver-only nodes have no ASPNETCORE_URLS so the Kestrel re-bind fallback is wrong. See recon doc \u00a79."
},
{
"id": 1,
"subject": "Task 1: Package references and pins (LocalDb 0.1.1, Grpc.AspNetCore, SQLitePCLRaw)",
"status": "completed",
"blockedBy": [
0
]
},
{
"id": 2,
"subject": "Task 2: Schema + LocalDbSetup.OnReady + registration tests",
"status": "completed",
"blockedBy": [
1
]
},
{
"id": 3,
"subject": "Task 3: Fail-closed sync auth interceptor + tests",
"status": "completed",
"blockedBy": [
1
]
},
{
"id": 4,
"subject": "Task 4: LocalDbRegistration + Program.cs DI wiring + config defaults",
"status": "completed",
"blockedBy": [
2,
3
]
},
{
"id": 5,
"subject": "Task 5: Dedicated h2c sync listener + MapZbLocalDbSync (Kestrel URLs-override risk)",
"status": "completed",
"blockedBy": [
4
]
},
{
"id": 6,
"subject": "Task 6: IDeploymentArtifactCache + chunked LocalDb implementation + tests",
"status": "completed",
"blockedBy": [
2
]
},
{
"id": 7,
"subject": "Task 7: Cache write path in DriverHostActor (Props expression-tree trap)",
"status": "completed",
"blockedBy": [
0,
6
]
},
{
"id": 8,
"subject": "Task 8: Boot-from-cache read path + running-from-cache signal",
"status": "completed",
"blockedBy": [
7
]
},
{
"id": 9,
"subject": "Task 9: Delete the dormant LiteDB LocalCache subsystem",
"status": "completed",
"blockedBy": [
0
]
},
{
"id": 10,
"subject": "Task 10: Health check + telemetry meter allowlist",
"status": "completed",
"blockedBy": [
4
]
},
{
"id": 11,
"subject": "Task 11: DI-pin integration tests over the real Program.cs",
"status": "completed",
"blockedBy": [
5,
10
]
},
{
"id": 12,
"subject": "Task 12: Two-node convergence harness + scenarios (+ positive control)",
"status": "completed",
"blockedBy": [
5,
6
]
},
{
"id": 13,
"subject": "Task 13: docker-dev rig configuration (site-a pair on, site-b pin off)",
"status": "completed",
"blockedBy": [
5
]
},
{
"id": 14,
"subject": "Task 14: Documentation (runbook, Redundancy.md, CLAUDE.md)",
"status": "completed",
"blockedBy": [
8
]
},
{
"id": 15,
"subject": "Task 15: DoD sweep (offline) \u2014 STOP and report after this",
"status": "completed",
"blockedBy": [
7,
8,
9,
11,
12,
13,
14
],
"notes": "Offline DoD sweep: full-solution build 0 errors (824 pre-existing OTOPCUA0001 analyzer warnings in driver *test* projects; none reference any LocalDb/DeploymentCache file). Grep: no LiteDB/ILocalConfigCache/GenerationSealedCache code refs (2 explanatory doc-comment lines in ILdapGroupRoleMappingService remain, allowed). Runtime.Tests 407/0/31 twice (the previously-flagged intermittent did NOT reproduce). Host.IntegrationTests LocalDb subset 40/0/0. Full-solution `dotnet test` with stash-baseline NOT run: many suites are infra-gated (driver fixtures, full Akka mesh, LDAP real-bind, shared-SQL DB tests) and cannot run offline on macOS \u2014 deferred to Task 16 / CI. Task 12 positive-control evidence recorded in commit afa5be71. Did NOT merge to master \u2014 stopped here per plan."
},
{
"id": 16,
"subject": "Task 16: Live gate on the docker-dev rig (needs explicit user go-ahead)",
"status": "completed",
"blockedBy": [
15
],
"notes": "Live gate RAN on the docker-dev rig with explicit user go-ahead. 8/8 checks pass (check 4 with a documented limitation). Caught + fixed FOUR real defects offline tests missed (4b2f0e6e NU1101 packageSourceMapping, ce9fa07f ASPNETCORE_HTTP_PORTS re-bind, 9137cb41 empty-address-space-on-cache-boot, c6a9f93a cache-not-repopulated-on-RestoreApplied), each with a regression test. Two documented limitations (follow-ups): no replication back-fill of a fully-wiped node; oplog growth on default-OFF nodes. Post-fix: full build 0 errors, Runtime.Tests 409/0, all LocalDb Host.IntegrationTests green; only consistent failure is the infra-gated AbCip_Green_AgainstSim (sim not up). Evidence: 2026-07-20-localdb-phase1-live-gate.md. NOT merged."
}
],
"lastUpdated": "2026-07-20T00:00:00Z"
}
docs/plans/2026-07-20-localdb-adoption-phase2.md
+316
View File
@@ -0,0 +1,316 @@
# OtOpcUa LocalDb Adoption — Phase 2 Implementation Plan (alarm-historian store-and-forward)
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
>
> **Execution model:** Optimized for **Claude Opus** agents (`claude --model opus`); dispatch
> `high-risk` tasks on Opus. Branch **`feat/localdb-phase2`** in `~/Desktop/OtOpcUa` (remote
> `lmxopcua`). **Prerequisite: Phase 1 (`docs/plans/2026-07-20-localdb-adoption-phase1.md`) is
> merged (or this branch is stacked on it) and its live gate passed.** Do not merge as part of this
> plan; stop at the DoD task.
>
> **Design authority:** `~/Desktop/scadaproj/docs/plans/2026-07-20-otopcua-localdb-design.md` §3.4,
> D8, D9. Reference implementation for "replace a bespoke store" phasing: ScadaBridge Phase 2
> (`~/Desktop/ScadaBridge/docs/plans/2026-07-19-localdb-adoption-phase2.md` + its live gate doc).
**Goal:** Move the alarm-historian store-and-forward buffer (today the standalone
`alarm-historian.db` owned by `SqliteStoreAndForwardSink`) into the consolidated LocalDb file as a
replicated table with a primary-gated drain, so a redundant pair no longer loses buffered alarm
history when a node dies — and delete the sink's bespoke file/connection management outright.
**Architecture:** `alarm_sf_events` (TEXT GUID PK) registered in `LocalDbSetup.OnReady`; the sink
rewired onto `ILocalDb` behind its unchanged public seam; drain gated on the delivered-snapshot
Primary role via `PrimaryGatePolicy` (at-least-once across failover, accepted and documented); a
one-time idempotent migrator from the legacy file, running **after** registration.
**Risk framing (from ScadaBridge):** Phase 2 replaces a *working* mechanism — a harder risk class
than Phase 1's "add where none existed." Cutover (delete + rewire) lands in **one commit**; there is
no dual-mechanism period, and the cutover is the test.
**Hard rules:** identical to Phase 1's list (OnReady ordering, no autoincrement/BLOB, fail-closed
auth already in place, no in-memory SQLite in tests, cp-triplet-only rig inspection), plus:
- **Legacy-copy column lists must INTERSECT** with what the legacy file actually has
(`pragma_table_info` probe) — a missing column throws and readers silently discard every row.
- Absence assertions need a positive control.
- DoD greps phrased as "no references from code" (explanatory comments may survive).
---
### Task 0: Recon (produces `docs/plans/2026-07-20-localdb-phase2-recon.md`)
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Create: `docs/plans/2026-07-20-localdb-phase2-recon.md`
- Read-only: `src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs` (and the
whole `Core.AlarmHistorian` project), `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/AlarmHistorianOptions.cs`,
the drain worker (whatever forwards batches to `GatewayHistorian`/`SendEvent`),
`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/PrimaryGatePolicy.cs`,
`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` (how `_localRole` +
driver-member-count reach the gate today)
**Record, with `file:line` citations:**
1. The sink's exact table schema: name, columns, PK type (**autoincrement?** — decides whether the
migrator needs deterministic `mig-{node}-{legacyId}` ids), indices, and any BLOB columns
(**STOP condition:** a BLOB payload column cannot be registered — it must become base64 TEXT in
the new schema, and the recon must size the largest realistic payload against the 171 KB-ish
chunk guidance; alarm events are small JSON, so expect this to be fine, but verify).
> **PRE-ANSWERED 2026-07-21 (verified against `master` `d218282c`) — the STOP condition does NOT
> fire.** Re-verify cheaply during recon, but do not expect a surprise:
>
> - **Executed DDL:** `SqliteStoreAndForwardSink.cs:657-667`. It matches the class doc-comment at
> `:17-26` exactly — no drift between the documented and executed schema.
> - **No BLOB.** All 8 columns are TEXT/INTEGER; the payload column is **`PayloadJson TEXT NOT
> NULL`**. So no base64 conversion is needed and the chunk-size sizing is moot.
> - **Payload is small and bounded by shape.** `PayloadJson` is a serialized
> `AlarmHistorianEvent` — 10 scalar fields (`AlarmId`, `EquipmentPath`, `AlarmName`,
> `AlarmTypeName`, `Severity`, `EventKind`, `Message`, `User`, `Comment?`, `TimestampUtc`).
> No collections, no nesting. Realistic worst case is low single-digit KB (operator `Comment`
> is the only unbounded-ish field), far under the 171 KB chunk guidance.
> - **PK IS autoincrement** — `RowId INTEGER PRIMARY KEY AUTOINCREMENT`. This confirms the
> plan's own prediction: LocalDb cannot replicate an autoincrement key, so the migrator
> **must** mint deterministic `mig-{node}-{legacyId}` ids, and the new table needs a TEXT
> GUID PK as already specified in the Architecture note.
> - **One index to carry across:** `IX_Queue_Drain ON Queue (DeadLettered, RowId)` (`:667`) —
> the drain's covering index. The `alarm_sf_events` equivalent wants the same shape over
> (dead-lettered flag, insertion order) so the drain query stays index-covered.
> - **Legacy column list for the migrator's `pragma_table_info` intersection check:**
> `RowId, AlarmId, EnqueuedUtc, PayloadJson, AttemptCount, LastAttemptUtc, LastError,
> DeadLettered`.
2. The public seam: the interface the drain worker and producers use (e.g. `IAlarmHistorianSink` /
enqueue+dequeue+markDelivered+deadLetter methods), so the rewire can keep it byte-compatible.
3. Semantics to preserve: `Capacity` (1,000,000) enforcement, `MaxAttempts` (10), dead-letter
retention (30 d), `BatchSize` (100), `DrainIntervalSeconds` (5) — where each lives.
4. The drain worker's lifecycle: hosted service or actor? Where a Primary-role check can be
injected, and how the delivered-snapshot role (`RedundancyStateChanged` cache) is accessible
from it (via `DriverHostActor`, a shared status service, or a message). If the role is only
available inside `DriverHostActor`, note the cleanest bridge (e.g. an `IRedundancyRoleView`
singleton the actor updates) — that becomes Task 4's shape.
5. Whether the sink is constructed per-node config path (`AlarmHistorian:DatabasePath`) anywhere
else (tests, tooling).
6. How `AlarmHistorian:Enabled=false` short-circuits (NullAlarmHistorianSink) — the rewire must
keep the disabled path allocating no LocalDb tables? No: tables are created unconditionally in
`OnReady` (cheap, empty); only the sink/drain stay Null. Note this in the doc.
Commit: `docs(localdb): phase-2 recon findings`.
---
### Task 1: `alarm_sf_events` schema + registration (+ tests)
**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** none (Task 2 depends on it)
**Files:**
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmSfSchema.cs` (depends only on
`Microsoft.Data.Sqlite`)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs`
- Test: extend `LocalDbSetupTests`
Schema shape (adjust column names to the recon's findings — preserve today's semantics):
```sql
CREATE TABLE IF NOT EXISTS alarm_sf_events (
id TEXT NOT NULL PRIMARY KEY, -- app-minted GUID (never autoincrement)
payload_json TEXT NOT NULL,
enqueued_at_utc TEXT NOT NULL,
attempts INTEGER NOT NULL DEFAULT 0,
status TEXT NOT NULL DEFAULT 'pending', -- pending | delivered | dead
last_attempt_utc TEXT NULL,
dead_at_utc TEXT NULL
);
CREATE INDEX IF NOT EXISTS ix_alarm_sf_events_status ON alarm_sf_events(status, enqueued_at_utc);
```
`OnReady` order becomes: Phase-1 DDL → `AlarmSfSchema.Apply` → the two Phase-1
`RegisterReplicated` calls → `RegisterReplicated("alarm_sf_events")` → **migrator (Task 5) last**.
(All DDL may run before all registrations; the invariant is registration-before-writes.)
TDD: failing test first — exact replicated set becomes
`["alarm_sf_events", "deployment_artifacts", "deployment_pointer"]` (ordinal-sorted; update the
Phase-1 exact-set pins in the same commit — they are *supposed* to go red here, that's them
working). Oplog-capture test for an `alarm_sf_events` insert. Commit
`feat(localdb): alarm_sf_events replicated table`.
---
### Task 2: Rewire the sink onto `ILocalDb` + delete bespoke file management (the cutover commit, part 1 of 2 — see Task 3)
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:**
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs` (or replace
with `LocalDbStoreAndForwardSink.cs` — keep the public seam identical either way)
- Modify: its registration (`AddAlarmHistorian`) to inject `ILocalDb`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/AlarmHistorianOptions.cs` — remove
`DatabasePath` (a breaking config key removal: note it in the runbook/CHANGELOG task)
- Test: the sink's existing unit tests, rewired to a temp-file `ILocalDb` via `TestLocalDb`-style
helper (create `tests/.../TestSupport` helper if none exists — real DB, never a stub: a stubbed
bare `SqliteConnection` lacks `zb_hlc_next()` and fails closed on registered tables)
**Steps:**
1. Write/port failing tests for the seam's semantics: enqueue, drain batch of `BatchSize`,
`MaxAttempts` → dead-letter, capacity enforcement, dead-letter retention purge.
2. Implement over `ILocalDb.ExecuteAsync/QueryAsync` (anonymous-object params). Delete the private
connection/pragma/file-open code and any `PRAGMA journal_mode` calls (LocalDb owns pragmas).
GUIDs minted at enqueue (`Guid.NewGuid().ToString("N")`).
3. `Capacity` enforcement: count-based insert guard (preserve today's overflow behavior per recon).
4. Core.AlarmHistorian gains a package ref on core `ZB.MOM.WW.LocalDb` (interface only).
5. Build + project tests green. **Commit together with Task 3** if the drain gate can't compile
separately (the ScadaBridge tasks-14/15/16 circular-dependency landmine — check before assuming
they're independent commits).
---
### Task 3: Primary-gated drain
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Files:** (exact shape from recon item 4)
- Modify: the drain worker
- Create (if recon says so): `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Redundancy/IRedundancyRoleView.cs`
— a singleton snapshot (`RedundancyRole? LocalRole`, `int DriverMemberCount`) updated by
`DriverHostActor` where it already caches `_localRole`
- Test: drain-worker tests + an actor test pinning that `DriverHostActor` publishes role changes to
the view
**Semantics:**
- Drain runs only when `PrimaryGatePolicy.ShouldServiceAsPrimary(localRole, driverMemberCount)` is
true — same policy, same boot-window posture (unknown role drains only when the node is alone).
- Delivered/dead-letter marks are row UPDATEs → they replicate, so the standby's copy tracks drain
progress and does not re-deliver already-marked rows after failover.
- **At-least-once across failover is accepted:** rows delivered on the old primary whose
`delivered` mark hadn't replicated yet will be re-sent by the new primary. Document in the
runbook (Task 7); do NOT build dedup.
- When replication is OFF (default), the gate still applies but `driverMemberCount` for a solo
node keeps today's behavior — verify with a test: single-node, role unknown → drains (no
regression for unpaired deployments).
TDD: failing tests — secondary role does not drain; primary drains; unknown+alone drains;
unknown+paired does not. Commit (with Task 2 if coupled):
`feat(localdb): alarm S&F on LocalDb with primary-gated drain (cutover)`.
---
### Task 4: One-time legacy migrator (`alarm-historian.db` → consolidated)
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 6
**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/AlarmSfLegacyMigrator.cs`
- Modify: `LocalDbSetup.OnReady` — call it **last**, after all `RegisterReplicated` calls
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/AlarmSfLegacyMigratorTests.cs`
Mirror `~/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs`:
- Source path from the pre-removal `AlarmHistorian:DatabasePath` default (`alarm-historian.db`) —
read the raw config key even though the option property is gone.
- `ShouldMigrate`: skip if file missing or `<file>.migrated` exists. `:memory:`/`file:` sources → no-op.
- **If the legacy PK is autoincrement (recon):** deterministic ids `mig-{NodeName}-{legacyId}`
(node name from the recon-identified config key) — rerunnable without duplicates under
`INSERT OR IGNORE`, and no cross-node collision. If already GUIDs, copy as-is.
- One transaction for the whole copy; `File.Move(path, path + ".migrated")` only after commit;
failure throws out of `OnReady` → boot fails, legacy untouched.
- **Column intersection** via `pragma_table_info` on the legacy table; required-PK guard.
- Both pair nodes migrate independently; their rows have distinct ids (node-prefixed), so the
merged buffer is the union — expected; note that the new primary will drain the standby's
migrated rows too.
TDD: failing tests — happy path row counts; idempotent re-run; failure leaves legacy file
untouched; migrated rows **enter the oplog** (assert `__localdb_oplog` — the registration-order
pin); older legacy file missing a column still migrates (intersection). Commit
`feat(localdb): one-time alarm-historian.db migrator`.
---
### Task 5: Convergence + failover scenarios in the pair harness
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 4
**Files:**
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/AlarmSfConvergenceTests.cs`
(reuses the Phase-1 `LocalDbPairHarness`)
**Scenarios:**
1. `AlarmBurstOnA_ConvergesToB_AndOplogDrains` — enqueue N events on A; identical rowset on B;
oplog → 0.
2. `DeliveredMarksReplicate` — mark rows delivered on A; B's copies show delivered (the
no-redeliver-after-failover property, asserted at the data layer).
3. `WritesWhileTransportDown_SurviveRejoin`.
4. Positive control: with `RegisterReplicated("alarm_sf_events")` commented out, scenarios 1–3 go
red (run locally, record, restore — do not commit red).
Commit `test(localdb): alarm S&F convergence scenarios`.
---
### Task 6: Rig config + docs
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 4
**Files:**
- Modify: `docker-dev/docker-compose.yml` — enable `AlarmHistorian__Enabled=true` on the site-a
pair if not already (the gate needs a live buffer); remove any `AlarmHistorian__DatabasePath`
env vars (key deleted).
- Modify: `docs/operations/2026-07-20-localdb-pair-replication.md` — add: alarm S&F replication
semantics, at-least-once-across-failover statement, migrator behavior (`.migrated` sidecar),
`DatabasePath` key removal.
- Modify: `docs/AlarmHistorian.md` + `CLAUDE.md` — sink now lives in the consolidated LocalDb;
drain is primary-gated.
Commit `docs+chore(localdb): phase-2 rig config + docs`.
---
### Task 7: DoD sweep (offline)
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none
1. Full solution build → 0 warnings; full test suite green (deltas vs pre-branch baseline only).
2. Greps, phrased as "no references from **code**": the old bespoke connection management
(`AlarmHistorian:DatabasePath`, direct `new SqliteConnection` inside Core.AlarmHistorian except
via schema helpers/tests) — explanatory comments may remain.
3. Positive-control evidence from Task 5 recorded.
4. Exact-set replicated-tables pin = 3 tables, both directions.
5. Update `…phase2.md.tasks.json`; commit `chore(localdb): phase-2 DoD sweep`; STOP and report.
---
### Task 8: Live gate on the docker-dev rig (run only with explicit user go-ahead)
**Classification:** high-risk (rig)
**Estimated implement time:** ~30 min wall-clock
**Parallelizable with:** none
Same inspection rules as Phase 1's gate (cp-triplet, curl sidecar, observer-suspicion rule).
Record in `docs/plans/2026-07-20-localdb-phase2-live-gate.md`:
1. Migration ran: `.migrated` sidecar present on both site-a nodes; row counts match legacy.
2. Alarm burst (drive a real driver alarm or the historian-gateway-unreachable path) converges:
identical rowsets, oplog drains to 0, dead letters 0.
3. Only the primary drains: stop the historian gateway egress, buffer builds on BOTH nodes'
tables (replicated), but only the primary's drain worker logs attempts.
4. Failover: stop the primary; the standby (new primary) resumes draining the shared buffer;
count of double-delivered events observed and recorded (at-least-once evidence, not a failure).
5. Both-nodes-together restart clean; zero `disk I/O error`/`SQLITE_IOERR` in logs.
6. site-b (default-OFF pin): sink works locally, no sync traffic.
Commit the gate doc; report; do not merge.
---
## Task persistence
Tasks file: `docs/plans/2026-07-20-localdb-adoption-phase2.md.tasks.json`. Record deviations per
task — especially if Tasks 2/3 had to land as one commit (the expected outcome).
docs/plans/2026-07-20-localdb-adoption-phase2.md.tasks.json
+86
View File
@@ -0,0 +1,86 @@
{
"planPath": "docs/plans/2026-07-20-localdb-adoption-phase2.md",
"tasks": [
{
"id": 0,
"subject": "Task 0: Recon \u2014 sink schema, seam, drain lifecycle, role-view bridge (STOP conditions)",
"status": "completed",
"note": "STOP condition does NOT fire (no BLOB; PayloadJson TEXT). Recon doc: docs/plans/2026-07-20-localdb-phase2-recon.md. Key finding: the drain worker is an internal Timer inside the sink (not a hosted service/actor), so the gate is a Func<bool> ctor param; and HistorianAdapterActor already primary-gates ENQUEUE with a different policy (ShouldHistorize) - which is why today's ungated drain is safe and why replication breaks it. Deviations D-1..D-4 recorded."
},
{
"id": 1,
"subject": "Task 1: alarm_sf_events schema + registration (+ exact-set pin update)",
"status": "completed",
"blockedBy": [
0
],
"note": "alarm_sf_events created in AlarmSfSchema (Core.AlarmHistorian) + registered third in LocalDbSetup.OnReady. Exact-set pin updated to 3 tables. DEVIATION D-5: kept the legacy delete-on-ack + dead_lettered flag + last_error rather than the plan's status column (no sweeper for 'delivered' rows; last_error is the only record of why a row died). Drain ORDER BY moves to (enqueued_at_utc, id)."
},
{
"id": 2,
"subject": "Task 2: Rewire sink onto ILocalDb + delete bespoke file management (cutover 1/2)",
"status": "completed",
"blockedBy": [
1
],
"note": "Sink rewritten as LocalDbStoreAndForwardSink over ILocalDb; bespoke file/pragma/schema management deleted with the old class. AlarmHistorian:DatabasePath removed (breaking config key). Ids are a deterministic payload hash (D-1), not GUIDs."
},
{
"id": 3,
"subject": "Task 3: Primary-gated drain via PrimaryGatePolicy (cutover 2/2 \u2014 may co-commit with Task 2)",
"status": "completed",
"blockedBy": [
2
],
"note": "Drain gated on IRedundancyRoleView, a singleton DriverHostActor publishes PrimaryGatePolicy's verdict to on every snapshot. Fails closed on a throwing gate; seeded OPEN so a non-redundant deployment is never silently stopped. New HistorianDrainState.NotPrimary + transition-logged (D-2). Landed with Task 2 as one commit (D-3)."
},
{
"id": 4,
"subject": "Task 4: One-time alarm-historian.db legacy migrator",
"status": "completed",
"blockedBy": [
3
],
"note": "AlarmSfLegacyMigrator runs LAST in OnReady (which now takes IConfiguration - no skip-migration overload exists). DEVIATION D-6: ids are the payload hash (AlarmSfSchema.DeriveId, lifted out of the sink) rather than mig-{node}-{legacyId} - a warm pair's two legacy files OVERLAP, and node-prefixing would carry that duplication forward forever. DEVIATION: tests live in Host.IntegrationTests; the plan's Host.Tests project does not exist."
},
{
"id": 5,
"subject": "Task 5: Convergence + failover scenarios in the pair harness (+ positive control)",
"status": "completed",
"blockedBy": [
3
],
"note": "4 scenarios green. POSITIVE CONTROL RUN (RegisterReplicated(alarm_sf_events) commented out): 3/4 went red immediately; the 4th (same-event-on-both-nodes) passed VACUOUSLY - with replication off each node trivially held its own single row. Strengthened by enqueuing a second distinct event on B and asserting BOTH nodes hold 2; it then went red under the control too. Control restored, all 4 green. Also fixed a SECOND exact-set pin the plan predicted: LocalDbWiringTests."
},
{
"id": 6,
"subject": "Task 6: Rig config + docs",
"status": "completed",
"blockedBy": [
3
],
"note": "Rig: AlarmHistorian__Enabled=true on site-a-1/2 AND site-b-1/2, with MaxAttempts raised to 1e6 (D-4: the default 10 would dead-letter the queue ~5min into the rig's permanent gateway outage) and a deliberately unresolvable ServerHistorian__Endpoint. NOTE: the endpoint is REQUIRED - ServerHistorianOptionsValidator is consumer-gated on AlarmHistorian:Enabled, so an empty one fails host start. Docs: runbook, AlarmHistorian.md, AlarmTracking.md, Configuration.md, Redundancy.md, docs/README.md, CLAUDE.md."
},
{
"id": 7,
"subject": "Task 7: DoD sweep (offline) \u2014 STOP and report after this",
"status": "completed",
"blockedBy": [
4,
5,
6
],
"note": "Build: 0 errors solution-wide; 0 warnings from every project this branch touches (the ~816 solution-wide warnings are pre-existing xUnit1051/OTOPCUA0001/CS86xx in untouched driver + client test projects). GREPS FOUND REAL DRIFT Task 6 missed: 8 live sites still named the deleted SqliteStoreAndForwardSink - CLAUDE.md, the AdminUI /alarms/historian panel text (user-visible), HistorianAdapterActor (a <see cref> that did NOT warn), Runtime + Host + 2 Driver.Historian.Gateway doc comments, 1 gateway test comment, docs/drivers/Historian.Wonderware.md; plus docs/AlarmTracking.md still claimed a Validate() startup warning for a relative DatabasePath (that branch is gone). All fixed. Code refs to AlarmHistorian:DatabasePath now reduce to exactly two intentional ones: AlarmSfLegacyMigrator.LegacyPathKey and its test. No 'new SqliteConnection' in Core.AlarmHistorian. Both exact-set pins assert the 3-table set, both directions. Guard-deletion evidence (both vacuous passes + the pin inventory) consolidated into the recon doc."
},
{
"id": 8,
"subject": "Task 8: Live gate on the docker-dev rig (needs explicit user go-ahead)",
"status": "completed",
"blockedBy": [
7
],
"note": "Gate doc: docs/plans/2026-07-20-localdb-phase2-live-gate.md. Checks 1/2/5/6 PASS; checks 3+4 NOT SATISFIED (see below). FOUR production defects found + fixed, three of which crash-looped every driver node before check 1 could run: (1) empty ServerHistorian:ApiKey crashes the host - the validator had explicitly classified it as 'degrades', which is false because the gateway client validates its own options at construction; (2) UseTls vs endpoint scheme mismatch crashes likewise, both directions; (3) plaintext h2c was UNREACHABLE - the adapter forwarded TLS-only options unconditionally so every documented http:// deployment crashed; (4) THE BLOCKER - the drain gate deferred to a CLUSTER-WIDE elected Primary while the queue is PAIR-LOCAL. On the rig that Primary is central-1 (carries the driver Akka role, replicates nobody, runs no historian), so ALL FOUR driver nodes suspended their drains including unpaired site-b: silent permanent loss where pre-Phase-2 drained fine. Fixed in 3 layers (separate ShouldDrainAlarmHistory policy; peer-host matching in DriverHostActor; gate short-circuited entirely when replication is unconfigured, testing BOTH PeerAddress and SyncListenPort since only the dialer sets the former). Also fixed a THIRD vacuous test: AwaitAssert on the seeded-open value passed with the guard deleted; now asserts the sequence of PUBLISHED values via a recording view. Migration evidence: 11 legacy rows across two overlapping files -> exactly 9 identical rows on both nodes, proving D-6's payload-hash identity live. OPEN DESIGN FORK (in the gate doc): a pair cannot identify its own Primary, so both halves drain - safe (no loss, duplicates only) but the gate's de-dup benefit is unrealised."
}
],
"lastUpdated": "2026-07-21T00:00:00Z"
}
docs/plans/2026-07-20-localdb-phase1-live-gate.md
+93
View File
@@ -0,0 +1,93 @@
# LocalDb Phase 1 — live gate (docker-dev rig)
> Task 16 of `2026-07-20-localdb-adoption-phase1.md`. Evidence log for the 8 live-rig checks.
> Rig: local `docker-dev/docker-compose.yml` (6 host nodes, one shared SQL). site-a pair replicates;
> central + site-b are default-OFF.
>
> **Safety rules followed:** all DB inspection via `docker cp` of the `db`/`-wal`/`-shm` triplet out
> of the container, querying the copy — never host `sqlite3` on the live WAL file. Metrics via a
> `curlimages/curl` sidecar with `--network container:<node>` (`aspnet:10.0` has no curl).
## Rig build/up
Two real defects surfaced by the rebuild (both fixed on-branch, re-verified):
1. **`NU1101` — packageSourceMapping gap** (`4b2f0e6e`). `ZB.MOM.WW.LocalDb*` was not mapped to the
`dohertj2-gitea` feed, so any *clean* restore (Docker image build, CI, fresh clone) failed. Local
dev builds masked it via the warm global NuGet cache. Fixed by adding the two patterns; verified
with a temp-packages-dir restore.
2. **`ASPNETCORE_HTTP_PORTS` re-bind gap** (`ce9fa07f`). The Kestrel re-bind read only
`urls`/`ASPNETCORE_URLS`; the aspnet:8.0+ base image sets `ASPNETCORE_HTTP_PORTS=8080` (all
interfaces) as the container default instead. A driver node that never sets `URLS` fell through to
`localhost:5000`, silently moving its health/metrics surface to loopback. Central nodes were fine
(they set `ASPNETCORE_URLS=9000` explicitly). Fixed to fall through URLS → HTTP_PORTS/HTTPS_PORTS →
5000; 4 new unit tests. After the fix, driver nodes bind `[::]:8080` + `[::]:9001` (site-a) /
`[::]:8080` only (site-b).
Rig: 6 nodes rebuilt from branch `feat/localdb-phase1` @ `ce9fa07f`, all up.
## Checks
| # | Check | Result | Evidence |
|---|---|---|---|
| 1 | Rig up, site-a pair healthy, `/metrics` shows `localdb_` series | ✅ PASS | Both site-a nodes `/healthz` 200 Healthy; `/metrics` shows `localdb_oplog_depth` + `localdb_sync_reconnects_total` on meter `ZB.MOM.WW.LocalDb.Replication` (allowlist works); site-a-2 served `/localdb_sync.v1.LocalDbSync/Sync` with routing `match_status=success` (interceptor passed, session established); both oplogs depth 0 |
| 2 | Deploy → both site-a nodes' pointer + artifacts byte-identical, same HLC + origin | ✅ PASS | Deploy `6e687451…` (rev `efb04c79…`) via `POST :9200/api/deployments` → both site-a-1 and site-a-2 `deployment_pointer` = `SITE-A / 6e687451… / efb04c79… / sha EFB04C79…` (identical); artifact 1 chunk each; **pointer `__localdb_row_version` identical on both: hlc `116955620225646592`, origin node_id `a0576df6-…`, tombstone 0** — one row replicated, not two derivations; both oplogs depth 0 |
| 3 | Boot-from-cache: SQL down, restart site-a-2, serves cached config w/ signal | ✅ PASS (found + fixed a defect) | With SQL stopped, site-a-2 restart logged `RUNNING FROM CACHE — … booted deployment 6e687451…`; **address space materialised from the cached blob**: `AddressSpaceApplier: applied plan (added=18)` / `OpcUaPublish: applied rebuild (added=18)`. Browse confirms **16 nodes on site-a-2 = 16 on site-a-1**. **Defect the gate caught (`9137cb41`):** pre-fix the rebuild re-read the artifact from the down ConfigDb and no-op'd, so the cache-booted node served **1** node vs the healthy peer's 16 — it logged success but browsed empty. Fixed by passing the in-hand cached blob to `RebuildAddressSpace`; regression test added |
| 4 | Replicated-cache payoff: wipe a-2 DB, repopulate, then SQL-down boot | ⚠️ PARTIAL — 1 defect fixed; the limitation is since **CLOSED** (LocalDb 0.1.2 — see Summary) | Wiped a-2's LocalDb volume, restarted with **SQL down**: sync session re-established (a-2 served a `Sync` call) but a-2's cache stayed **0/0/0** and a-1's oplog was **0** — the deploy's rows had been acked by the old a-2 and pruned, so there was no delta to send and **no snapshot-resync fired**. ⇒ **Limitation: pure replication does not back-fill a fully-wiped, already-converged node** (library delta-replication; snapshot-resync is gated on the oplog cap, which the running engine never hits). Separately found a **defect (`c6a9f93a`)**: recovery via `RestoreApplied` never wrote the cache (only fresh `ApplyAndAck` did), so a wiped node stayed cache-less. Fixed: `RestoreApplied` now re-caches. Verified — after the fix, wiped a-2 restarted with SQL up: `restored served state … on bootstrap` → cache repopulated `SITE-A / 6e687451`, chunks=1. **Net: a wiped node self-heals its cache from central on next boot (regaining boot-from-cache for future outages); it is not back-filled by its peer during the same outage.** |
| 5 | Transport kill: pause a-2, oplog rises; unpause, drains to 0 | ✅ PASS | `docker pause` on a-2 severed the sync transport; during the window a-2 accumulated **oplog=2** unacked local writes (its re-cache) while a-1 held 0. `docker unpause` → a-2's oplog **drained to 0 within 5s**, both converged on the same pointer. Note: the rig's redeploys carry an identical revision, so a fresh cache write on a-1 short-circuits (no-op) — the store-during-partition variant is covered by the automated harness scenario `WritesWhileTransportDown_SurviveRejoin` (proven red without `RegisterReplicated`) |
| 6 | Retention: 3rd deploy prunes oldest; chunks gone + tombstones on BOTH nodes | ✅ PASS | Cached 3 distinct deployment-ids (via deploy + restart re-cache). Both nodes retain exactly the newest 2 (`475df87a`, `962962dc`); the oldest `6e687451`'s chunks are **0 on both** (pruned); **1 artifact tombstone on both** — the prune replicated as a tombstone, not left as a live chunk |
| 7 | Both-nodes-together restart: clean rejoin, identical counts, no I/O errors | ✅ PASS | Restarted site-a-1 + site-a-2 together: **0** `disk I/O error`/`SQLITE_IOERR`/`corrupt` lines in either log; both re-cached and converged to **identical** counts (ptr `962962dc…`, chunks=2, rowver=3). Byte-identical convergence preserved across a simultaneous restart |
| 8 | site-b default-OFF pin: cache works locally, Healthy, no sync listener on 9001 | ✅ PASS (+1 finding, since **CLOSED** by the skip-if-unchanged cache write — see Summary) | site-b-1 cache works locally (`SITE-B / 6e687451`, chunks=1, from its own apply); listeners **`[::]:8080` only — no 9001**; port 9001 **not bound** (`/proc/net/tcp`); `/healthz` **200**; no sync/reconnect metric series. **Finding (follow-up, not a blocker):** `localdb_oplog_depth=5` — because `OnReady` registers replication unconditionally, a default-OFF node captures cache writes into its oplog but has no peer to ack/prune them, so the oplog grows slowly (≈2 rows/deploy, amplified by the re-cache-on-restore fix). Bounded by the library `MaxOplogRows` cap (1M); worth a periodic pruner or skip-if-unchanged in a follow-up |
## Summary
**8/8 checks pass** (check 4 with a documented limitation). The gate exercised the real image built
from the branch, the real interceptor over a real loopback+cross-container h2c transport, and real
`docker cp`-triplet DB inspection — and caught **four real defects that every offline test had
missed**, each now fixed on-branch with a regression test:
1. **`NU1101` packageSourceMapping gap** (`4b2f0e6e`) — `ZB.MOM.WW.LocalDb*` unmapped to the gitea
feed; every clean restore (Docker, CI, fresh clone) failed. Masked locally by the warm NuGet cache.
2. **`ASPNETCORE_HTTP_PORTS` re-bind gap** (`ce9fa07f`) — the sync-listener re-bind ignored the
aspnet:8.0+ container-default port var, silently moving driver health/metrics to loopback:5000.
3. **Empty address space on boot-from-cache** (`9137cb41`) — the rebuild re-read the artifact from the
down ConfigDb and no-op'd, so a cache-booted node logged success but served clients **1 node vs the
healthy peer's 16**. The single most important find: boot-from-cache was half-working.
4. **Cache not repopulated on `RestoreApplied`** (`c6a9f93a`) — a wiped/fresh-volume node recovered its
served state from central but stayed cache-less, unable to boot-from-cache on the next outage.
**Two documented limitations (follow-ups, not blockers) — both since CLOSED:**
- **No replication back-fill of a fully-wiped node** (check 4): a peer's already-acked cache rows are
pruned from its oplog and snapshot-resync is gated on the (never-hit) oplog cap, so a wiped node is
not healed by its peer — it self-heals from central on next boot instead.
→ **FIXED in `ZB.MOM.WW.LocalDb` 0.1.2 + 0.1.3** (scadaproj `cad3bcb` + `3b7489a`), live-gated in
`2026-07-21-localdb-followups-live-gate.md`. The diagnosis in this doc was
slightly off: the cap was not the gate. `ComputeSnapshotRequiredAsync` measured the peer's gap
against the *oldest surviving oplog row* and read an empty oplog as "no gap possible" — and an
empty oplog is the **steady state of a converged pair**, since ack-pruning deletes everything the
peer has confirmed. So the healthy state was the one state that could not heal a wiped peer. With
an empty oplog the gap is now measured against `last_acked_seq`. Covered at both levels:
`SnapshotResyncTests.WipedPeer_AfterEverythingWasAckedAndPruned_StillGetsSnapshot` in the library,
and `LocalDbPairConvergenceTests.WipedNode_IsBackFilledByItsPeer_WithoutAnyNewDeploy` here —
the latter verified RED against the pinned 0.1.1 and green on 0.1.2. **0.1.3 closes a second,
deeper half that only became reachable once back-fill worked:** the rebuilt node's OWN writes were
silently dropped, because `last_applied_remote_seq` is a watermark in the *peer's* seq space and a
rebuilt peer numbers from 1 again. The follow-up live gate caught that one — offline tests could
not have, since the scenario had no test while it was still a documented limitation.
- **Oplog growth on default-OFF nodes** (check 8): `OnReady` registers replication unconditionally, so
a node with no peer accumulates unacked/unpruned oplog rows (~2/deploy). Slow, bounded by the 1M cap;
a periodic pruner or a skip-if-unchanged cache write would close it.
→ **FIXED at the source**: `LocalDbDeploymentArtifactCache.StoreAsync` now **skips entirely** when
the pointer already names this deployment/revision, the SHA matches the bytes, and the expected
chunk count is present — so re-caching an artifact the node already holds (every restart's
boot-from-cache, every `RestoreApplied`) writes nothing and mints no oplog rows. Two guard tests
pin what the skip must NOT do (stale bytes under an unchanged revision; a matching pointer over
missing chunks), both verified RED against a naive pointer-only skip.
Also correcting this row's bound: the growth was never headed for the 1M cap. `AddZbLocalDbReplication`
is registered unconditionally, so `MaintenanceBackgroundService` runs on default-OFF nodes too and
the **7-day `MaxOplogAge`** cap prunes — the true bound was ~7 days of writes, not 1M rows.
Merged to `master` as `c957db52` (2026-07-21) and pushed. Both follow-ups closed on
`fix/localdb-phase1-followups` (2026-07-21) and live-gated — see
`2026-07-21-localdb-followups-live-gate.md`, which found a third defect in the process.
docs/plans/2026-07-20-localdb-phase1-progress.md
+72
View File
@@ -0,0 +1,72 @@
# LocalDb Phase 1 — Execution Progress (resume state)
**Branch:** `feat/localdb-phase1` (off `master`). Working tree CLEAN, all work committed, nothing pushed.
**Plan:** `docs/plans/2026-07-20-localdb-adoption-phase1.md`
**Recon:** `docs/plans/2026-07-20-localdb-phase1-recon.md` (READ THIS — it has the 8 deviations D-1..D-8 that override the plan text).
**Skill in use:** `superpowers-extended-cc:executing-plans` — batch, verify, commit per task, report between batches.
**Do NOT merge to master** — plan stops at Task 15 (DoD) and reports; Task 16 is a rig live-gate needing explicit user go-ahead.
## Status: Tasks 0–9 DONE (10 commits). Tasks 10–16 REMAIN.
| # | Task | State | Commit |
|---|---|---|---|
| 0 | Preflight + recon | ✅ | `42f85507` |
| 1 | Package refs (LocalDb 0.1.1, Grpc.AspNetCore 2.76.0) | ✅ | `44255fcb` |
| 2 | Schema + LocalDbSetup.OnReady + tests | ✅ | `3b65c24b` |
| 3 | Fail-closed sync auth interceptor + tests | ✅ | `3d29be83` |
| 4 | LocalDbRegistration + Program.cs DI + appsettings | ✅ | `6aff9a83` |
| 5 | h2c sync listener + MapZbLocalDbSync (Kestrel) | ✅ | `b9ddf20e` |
| (fix) | rename Runtime.Deployment → Runtime.DeploymentCache | ✅ | `e771a11a` |
| 6 | IDeploymentArtifactCache + chunked impl + tests | ✅ | `a38a52b8` |
| 9 | Delete LiteDB LocalCache | ✅ | `2bae1b4c` |
| 7 | Cache write path in DriverHostActor | ✅ | `1becf591` |
| 8 | Boot-from-cache + RunningFromCache signal | ✅ | `a27eff32` |
| 10 | Health check + telemetry meter allowlist | ⬜ TODO | |
| 11 | DI-pin integration tests over real host graph | ⬜ TODO | |
| 12 | 2-node convergence harness + scenarios | ⬜ TODO | |
| 13 | docker-dev rig config | ⬜ TODO | |
| 14 | Documentation | ⬜ TODO | |
| 15 | DoD sweep (offline) — STOP + report | ⬜ TODO | |
| 16 | Live gate on docker-dev rig — NEEDS USER GO-AHEAD | ⬜ TODO | |
Native task list (TaskCreate ids 1–17 map to plan tasks 0–16) tracks the same; tasks.json in plan dir mirrors it.
## What exists now (key files)
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/DeploymentCacheSchema.cs` — DDL, namespace `Runtime.DeploymentCache`.
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/IDeploymentArtifactCache.cs` — `StoreAsync` / `GetCurrentAsync` / **`GetCurrentUnkeyedAsync`** (D-1 addition) + `CachedDeploymentArtifact` record.
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/LocalDbDeploymentArtifactCache.cs` — chunked impl. NOTE the prune guard `AND deployment_id <> @DeploymentId` (bug I found: pointer-target could be pruned under same-tick GUIDs → permanent silent miss).
- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs` — `OnReady` (DDL→RegisterReplicated). PUBLIC (Host has no InternalsVisibleTo).
- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbRegistration.cs` — `AddOtOpcUaLocalDb`, `SyncListenPort(config)`. Registers `IDeploymentArtifactCache → LocalDbDeploymentArtifactCache` singleton.
- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSyncAuthInterceptor.cs` — fail-closed bearer.
- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/KestrelHttpBinding.cs` — URL parse/re-bind helper (D-6).
- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` — LocalDb wired in `hasDriver` branch (~L176 after AddAlarmHistorian: `AddOtOpcUaLocalDb` + `AddGrpc(interceptor)`); Kestrel re-bind block before `builder.Build()`; `MapZbLocalDbSync()` gated `hasDriver && syncListenPort>0` before `MapOtOpcUaHealth()`.
- `src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json` — `LocalDb` section (Path, SyncListenPort:0, Replication{}), between AlarmHistorian and Deployment.
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` — cache dep threaded LAST in Props/ctor/forwarding; `CacheAppliedArtifact` (write, after PushDesiredSubscriptions, empty-blob skip, own try/catch); `TryBootFromCache` + `ApplyCachedArtifact` (in Bootstrap catch); `PushDesiredSubscriptionsFromArtifact` split out; `_isRunningFromCache` field; `SingleClusterCacheKey="__single"`; `ReconcileDrivers` now returns `byte[]?`.
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs` — resolves `IDeploymentArtifactCache` (optional, nullable) and passes into DriverHostActor.Props.
- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Interfaces/NodeDiagnosticsSnapshot.cs` — added `bool RunningFromCache = false`.
- Tests (all green): `Host.IntegrationTests/LocalDbSetupTests.cs`, `LocalDbSyncAuthInterceptorTests.cs`, `LocalDbSyncListenerTests.cs`; `Runtime.Tests/Deployment/LocalDbDeploymentArtifactCacheTests.cs`, `Runtime.Tests/Drivers/DriverHostActorArtifactCacheTests.cs`, `DriverHostActorBootFromCacheTests.cs`.
## Load-bearing facts for the remaining tasks
- **Namespace trap (recurs!):** any `Deployment` child namespace under `Runtime` OR `Runtime.Tests` shadows the `Deployment` EF entity → CS0118 across the assembly. Use `...DeploymentCache`. Only a FULL-SOLUTION build catches it (per-project builds pass).
- **xUnit1051 is an error** in these test projects: every `ILocalDb`/DB call in a test must pass `TestContext.Current.CancellationToken` (xunit.v3 Host.IntegrationTests) — but `Runtime.Tests` is **xunit v2** (no `TestContext.Current`).
- **Test project reality (D-5):** NO `Host.Tests` project; NO `WebApplicationFactory<Program>` (deliberately — `TwoNodeClusterHarness.cs:48` explains why). Host unit tests live in `Host.IntegrationTests` (xunit.v3). Actor tests in `Runtime.Tests` (xunit v2 + Akka.TestKit.Xunit2).
- **Task 10 (D-7):** `AddOtOpcUaHealth()` takes NO args, called unconditionally at `Program.cs:365`. DON'T change its signature — register the LocalDb check unconditionally and return `Healthy` when LocalDb absent (ActiveNodeHealthCheck precedent). Health check classes: there are ZERO in-repo today; `Host/Health/` has only `HealthEndpoints.cs`. Meter allowlist is REAL and REQUIRED: `Host/Observability/ObservabilityExtensions.cs:30` `o.Meters = [OtOpcUaTelemetry.MeterName]` — add `LocalDbMetrics.MeterName` (from `ZB.MOM.WW.LocalDb.Replication`). Health check reads `ISyncStatus` + `IOptions<ReplicationOptions>`.
- **Task 11:** copy the `TwoNodeClusterHarness` direct-host-build pattern, NOT WebApplicationFactory. Pins: `ILocalDb` singleton + `ReplicatedTables` == `["deployment_artifacts","deployment_pointer"]`; `IDeploymentArtifactCache`→concrete; `ISyncStatus` default-OFF (`Connected==false`); admin-only graph has NO `ILocalDb` + doesn't demand `LocalDb:Path`; health check registered. Set `OTOPCUA_ROLES` per case; `LocalDb:Path`→temp file, `ClearAllPools`+delete triplet in cleanup.
- **Task 12:** harness copied from `~/Desktop/ScadaBridge/tests/.../LocalDbSitePairHarness.cs` + `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/tests/.../Convergence/ConvergenceFixture.cs`. Both nodes init via PRODUCTION `LocalDbSetup.OnReady`. Real loopback Kestrel h2c through the REAL interceptor. DBs as pre-constructed singletons. Positive control: comment out both `RegisterReplicated`, confirm scenarios 1–3 red, restore (don't commit red).
- **Task 13 (D-8):** NO host node has a writable volume today — must ADD named volumes. `docker-dev/docker-compose.yml` services: `central-1/2` (admin,driver, MAIN), `site-a-1/2` (driver, SITE-A), `site-b-1/2` (driver, SITE-B). Env style `Section__Key`. `9001` free container-internal. Enable replication on **site-a pair only** (a-1 initiator w/ PeerAddress, a-2 passive, byte-identical ApiKey, MaxBatchSize=16). site-b stays OFF = default-OFF pin. Validate `docker compose config`; DON'T bring rig up.
- **Task 15:** baseline pre-existing test failures FIRST (`git stash` → run full `dotnet test` → unstash), report deltas only. **KNOWN intermittent:** one `Runtime.Tests` failure seen ~3× across runs (mine + subagent's), never reproduced under trx logger, NOT one of the new tests — investigate/baseline here.
- **Warnings-as-errors** across src + most test projects. Pre-existing CS0618/xUnit1051 warnings exist ONLY in `Client.Shared*` (which don't set TreatWarningsAsErrors) — not mine, ignore.
- **SQLitePCLRaw:** no pin needed — LocalDb 0.1.1 nuspec floors `lib.e_sqlite3` at 2.1.12 (CVE-fixed). Confirmed.
## Follow-ups logged (NOT in scope, don't action without asking)
- `Polly.Core` in `Configuration.csproj` now orphaned (ResilientConfigReader was its only real consumer). Left in place — recorded in recon §7.3b.
- Pre-existing: `TryRecoverFromStale` marks Applied w/o reconciling drivers (recon §3.5); `Install-Services.ps1` gives driver-only nodes no `ASPNETCORE_URLS` (recon §6.4, worked around by D-6).
## Verify-anything-quick
Full build: `dotnet build ZB.MOM.WW.OtOpcUa.slnx` (expect 0 errors). LocalDb tests:
`dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LocalDb"` and
`dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~ArtifactCache|FullyQualifiedName~BootFromCache"`.
docs/plans/2026-07-20-localdb-phase1-recon.md
+617
View File
@@ -0,0 +1,617 @@
# LocalDb Phase 1 — Task 0 Recon Findings
**Date:** 2026-07-20
**Branch:** `feat/localdb-phase1` (based on `master`)
**Plan:** `docs/plans/2026-07-20-localdb-adoption-phase1.md`
**Design authority:** `~/Desktop/scadaproj/docs/plans/2026-07-20-otopcua-localdb-design.md`
All `file:line` citations are against the branch base commit.
---
## Step 1 — Feed verification: PASS
```
$ dotnet package search ZB.MOM.WW.LocalDb --source dohertj2-gitea --format json
ZB.MOM.WW.LocalDb 0.1.1
ZB.MOM.WW.LocalDb.Contracts 0.1.1
ZB.MOM.WW.LocalDb.Replication 0.1.1
```
All three present at **0.1.1**. No STOP. (Note: the local NuGet cache holds both `0.1.0` and `0.1.1`
— pin exactly `0.1.1` so a stale cache cannot win.)
---
## Step 3 — STOP conditions: NONE TRIGGERED
| STOP condition | Verdict |
|---|---|
| Artifact apply path cannot take a post-apply hook without restructuring the actor | **Clear** — `ApplyAndAck` has a clean insertion point (§2.2) |
| `DeploymentId` / cluster identity are not stable strings/GUIDs | **Clear** — both are stable strings, but cluster identity is **not resolvable at boot**; see D-1 |
| LiteDB LocalCache is referenced by live code | **Clear** — zero live-code references (§7) |
Three STOP conditions clear. **Eight deviations** from the plan's stated assumptions are recorded in
§9 — four of them (D-1, D-3, D-5, D-6) change the work materially.
---
## 1. Deploy fetch path
### 1.1 Blob-load sites — there are **five** `CreateDbContext()` calls, not two
| Line | Method | Purpose |
|---|---|---|
| `DriverHostActor.cs:513` | `Bootstrap()` | `NodeDeploymentStates` + `Deployments` at PreStart |
| `DriverHostActor.cs:1472` | `ReconcileDrivers(DeploymentId)` | **loads `ArtifactBlob`** |
| `DriverHostActor.cs:1543` | `PushDesiredSubscriptions(DeploymentId)` | **loads `ArtifactBlob` again** |
| `DriverHostActor.cs:2000` | `TryRecoverFromStale()` | latest `Sealed` deployment (id + hash only, no blob) |
| `DriverHostActor.cs:2029` | `UpsertNodeDeploymentState(...)` | writes `NodeDeploymentState` |
`DriverHostActor.cs:1472-1476`:
```csharp
using var db = _dbFactory.CreateDbContext();
blob = db.Deployments.AsNoTracking()
.Where(d => d.DeploymentId == deploymentId.Value)
.Select(d => d.ArtifactBlob)
.FirstOrDefault() ?? Array.Empty<byte>();
```
`:1543-1547` is byte-identical apart from the local. **The same blob is read twice per apply.**
### 1.2 Artifact runtime type — there is no instance type
`DeploymentArtifact` (`DeploymentArtifact.cs:48`) is a **static parser class** over
`ReadOnlySpan<byte>`. The blob is never materialised into an object graph.
- Storage: `byte[]` — `Deployment.cs:29` `public byte[] ArtifactBlob { get; init; } = Array.Empty<byte>();`
- Format: **UTF-8 JSON**, `JsonDocument.Parse` (`DeploymentArtifact.cs:66`, `:490`). Not MessagePack.
- Leniency contract: empty/malformed blob → empty result, **never throws** (`:62`, `:537-541`).
**Consequence for the cache:** we cache raw `byte[]`. No serializer coupling. Good.
### 1.3 Identity types
| Type | Definition | Shape |
|---|---|---|
| `DeploymentId` | `Commons/Types/DeploymentId.cs:3` — `readonly record struct DeploymentId(Guid Value)` | `ToString()` → `Value.ToString("N")`, 32 hex chars, **no hyphens** (`:10`) |
| `RevisionHash` | `Commons/Types/RevisionHash.cs:8` — `readonly record struct RevisionHash(string Value)` | SHA-256 hex, **lowercase 64 chars**, no `0x`; empty invalid (`:3-6`) |
Both are stable, filesystem/SQL-safe strings. Use `DeploymentId.ToString()` (the "N" form) as the
`deployment_id` TEXT column value.
---
## 2. Post-apply point
### 2.1 `ApplyAndAck` control flow — `DriverHostActor.cs:1413-1458`
```
1415 _applyingDeploymentId = deploymentId;
1416 Become(Applying);
1425 UpsertNodeDeploymentState(..., Applying, null)
1427 try {
1429 ReconcileDrivers(deploymentId); // blob fetch #1 — SWALLOWS DB errors
1430 _currentRevision = revision;
1431 UpsertNodeDeploymentState(..., Applied, null);
1432 SendAck(..., ApplyAckOutcome.Applied, null, correlation);
1436 _opcUaPublishActor?.Tell(RebuildAddressSpace(...));
1439 PushDesiredSubscriptions(deploymentId); // blob fetch #2 — SWALLOWS DB errors
1440 OtOpcUaTelemetry.DeploymentApplied.Add(1, "outcome"="ack");
1441 _log.Info("applied deployment ...");
1443 }
1444 catch (Exception ex) { ...Failed ACK... }
1452 finally { _applyingDeploymentId = null; Become(Steady); }
```
### 2.2 Where the cache write belongs
**Immediately after `DriverHostActor.cs:1439`**, before `:1440` — but **wrapped in its own
try/catch**. Reason: `:1432` already sent an `Applied` ACK to the coordinator. An unguarded cache
write that throws would fall into the `catch` at `:1444` and send a *second*, contradictory
`Failed` ACK for a deployment the fleet already believes is applied.
### 2.3 Paths that must NOT write to the cache
1. **`catch` at `:1444`** — any apply failure.
2. **Idempotent short-circuit at `:1402-1409`** — `HandleDispatchFromSteady` returns early with an
`Applied` ACK when `_currentRevision == msg.RevisionHash`. `ApplyAndAck` is never entered and no
blob is read.
3. **⚠ THE SILENT-DEGRADATION TRAP.** `ReconcileDrivers:1478-1483` and
`PushDesiredSubscriptions:1549-1553` catch DB failures, log a **warning**, and `return` **without
rethrowing**. `ApplyAndAck` therefore reaches `:1440` and declares success having applied **zero
drivers** when the blob load failed. Caching "whatever we read" would **persist an empty artifact
as a successful apply** — poisoning the boot-from-cache path with a config that starts no drivers.
**Mitigation (load-bearing):** the cache write must be gated on the blob having actually loaded
non-empty. Have `ReconcileDrivers` surface the loaded blob upward (a field or an out-param)
rather than re-reading it a third time, and skip the cache write when it is
`Array.Empty<byte>()`. This also removes the duplicate read noted in §1.1.
4. **`RestoreApplied` (`:1514-1528`)** — the bootstrap replay path. Restoring from central; caching
here is harmless and arguably desirable, but it is a distinct seam. **Phase 1 scope: skip it.**
5. **`TryRecoverFromStale` (`:1996-2022`)** — never reads the artifact at all (see §3.4).
---
## 3. Cold-boot path
### 3.1 PreStart — `DriverHostActor.cs:415-436`
Ctor already did `Become(Steady)` at `:411`. PreStart subscribes to topics, spawns the virtual-tag
and scripted-alarm hosts, then calls `Bootstrap()` at `:435`.
### 3.2 `Bootstrap()` — `DriverHostActor.cs:506-565` — **yes, it queries central SQL at boot**
Two queries inside one `try`: latest `NodeDeploymentState` for self (`:514-518`), then the matching
`Deployment` row (`:527-532`). Four outcomes: no prior deployments → Steady; `Applied` →
`RestoreApplied` (`:544`); `Applying` orphan → `ApplyAndAck` replay (`:549`); `Failed` → Steady.
### 3.3 **THE SEAM** — `DriverHostActor.cs:560-564`
```csharp
catch (Exception ex)
{
_log.Warning(ex, "DriverHost {Node}: ConfigDb unreachable on bootstrap; entering Stale", _localNode);
Become(Stale);
}
```
**This is the exact point where "central fetch failed at boot" is known.** The cache fallback goes
between `:562` and `:563`: attempt a local-cache load, and only `Become(Stale)` if the cache is also
a miss.
⚠ The catch also fires if the *second* query throws after the first succeeded, so `latest` is not in
scope-valid state from the catch. **The cache must be self-describing** — it must carry its own
`DeploymentId` + `RevisionHash`, because at this seam the actor has no idea what it should be
restoring. The planned `deployment_pointer` shape already satisfies this.
### 3.4 The `Stale` state — `DriverHostActor.cs:1343-1379`
`ReconnectInterval = 30s` (`:55`), retry timer started at `:1378`.
The dispatch-ignore at **`:1345-1348`** discards `DispatchDeployment` entirely — no ACK, no
buffering (contrast `Applying:599` which does `Self.Forward(msg)`). A Stale node looks like a
timeout to the coordinator. Phase 1 does **not** change this (per design §3.3: a *new* deployment
still requires central).
`RouteNodeWrite` fast-fails at `:1359-1360` with `"driver host stale (config DB unreachable)"`.
### 3.5 Pre-existing gap worth recording (not Phase 1 scope)
`TryRecoverFromStale` (`:1996-2022`) sets `_currentRevision` and marks `Applied` **without calling
`ReconcileDrivers` or `PushDesiredSubscriptions`** — documented at `:1692-1695` and `:1706`. A
Stale-recovered node reports Applied while running **zero drivers**, and
`HandleDispatchFromSteady:1402` then short-circuits on revision match. Boot-from-cache largely
neutralises this in practice, but the bookkeeping remains optimistic. **Logged as a follow-up, not
fixed here.**
### 3.6 `DbHealthProbeActor`
**No interaction with `DriverHostActor` whatsoever.** Spawned as a sibling
(`ServiceCollectionExtensions.cs:243-246`), consumed only by `OpcUaPublishActor`
(`OpcUaPublishActor.cs:105`, `:256`, `:528`, `:543-544`). `DriverHostActor` runs its own independent
30s `retry-db` timer. If running-from-cache should later influence OPC UA ServiceLevel,
`OpcUaPublishActor`'s DB-health seam is the precedent to mirror.
---
## 4. Cluster identity — **the one real design problem (see D-1)**
**There is no `ClusterId` field on `DriverHostActor` and no accessor for it.** The actor knows only
`_localNode` (`DriverHostActor.cs:61`, type `Commons.Types.NodeId`, aliased at `:28`).
`ClusterId` is resolved **per-artifact**, by `DeploymentArtifact.ResolveClusterScope(blob, nodeId)`
(`DeploymentArtifact.cs:485-516`) — it scans the artifact's `Nodes[]` array for a matching `NodeId`
and reads that element's `ClusterId`. Types: `ClusterScope` (`:46`), `ClusterFilterMode` (`:28`).
`IClusterRoleInfo` (`Commons/Interfaces/IClusterRoleInfo.cs:12`) exposes `LocalNode`, `LocalRoles`,
`HasRole`, `MembersWithRole`, `RoleLeader`, `RoleLeaderChanged` — **no `ClusterId`**.
`docker-dev/docker-compose.yml` has `Cluster__Hostname`/`Port`/`Roles`/`SeedNodes` — **no
`Cluster__ClusterId`**. The `ClusterNode` entity
(`Configuration/Entities/ClusterNode.cs:7,10`) carries both, but lives in central SQL — exactly
what is unreachable at the seam that needs it.
> **The bind:** the design keys the cache by `ClusterId` so a replicated pair shares one cache
> entry. But `ClusterId` is only discoverable *from a blob you already have* or *from central SQL*.
> At the §3.3 boot-failure seam we have neither.
**Resolution adopted (D-1):** keep `cluster_id` as the `deployment_pointer` PK — pair sharing is the
whole point of replication and must not be given up. Resolve it at each end as follows:
- **Write path** (`ApplyAndAck`, §2.2): call `DeploymentArtifact.ResolveClusterScope(blob, nodeId)`
on the blob just applied. `ClusterFilterMode.ScopeTo` → use its `ClusterId`. `None` (single-cluster
artifact) or `Suppress` → fall back to the literal `"__single"` sentinel, matching the actor's own
existing degenerate-case convention at `:1827-1830`.
- **Read path** (boot seam, §3.3): we cannot compute the key, so **do not key the read**. Select the
single `deployment_pointer` row, ordered by `applied_at_utc DESC`, `LIMIT 1`. A node belongs to
exactly one cluster and its pair peer replicates the same cluster's row, so the table holds one
row in every real topology. If more than one row is present (a node was re-homed between
clusters), take the newest and **log a warning naming both cluster ids** — an operator-visible
signal rather than a silent wrong-config boot.
- Optional escape hatch: honour a `Cluster:ClusterId` config key when present. **Deferred** — not
needed for the rig, and adding a required key contradicts the design's "storage ships
unconditionally" posture.
This preserves the replication payoff (design §3.3) and the schema exactly as specified, and
confines the change to the read query's `WHERE` clause.
---
## 5. `DriverHostActor` construction — the `Props` expression tree
### 5.1 The factory — `DriverHostActor.cs:326-346`
`static Props(...)` takes **16 parameters**, 14 optional, and forwards them **fully positionally**
into `Akka.Actor.Props.Create(() => new DriverHostActor(...))` at **`:343-346`**. Ctor at
`:375-412`, assignments `:393-408`.
> ⚠ Six `IActorRef?` parameters and three interface-typed parameters mean many mis-bindings are
> **type-compatible and therefore compile clean**. Callers use *named* arguments (safe); the
> internal forwarding at `:343-346` is *positional* (unsafe).
>
> **Rule for Task 7: append the new parameter LAST in the `Props` signature, LAST in the ctor
> signature, and LAST in the positional forwarding list at `:346`. Change nothing else.**
### 5.2 Construction sites — **27 total** (1 production + 26 test)
**Production (1):** `Runtime/ServiceCollectionExtensions.cs:332`, inside `WithOtOpcUaRuntimeActors`
(`:193`); registered as `DriverHostActorKey` (`:344`). All args named except the first three. Does
not pass `virtualTagHostOverride`, `scriptedAlarmHostOverride`, or `driverMemberCountProvider`.
`WithOtOpcUaRuntimeActors` callers: `Host/Program.cs:331`,
`Host.IntegrationTests/TwoNodeClusterHarness.cs:398`, `ServiceCollectionExtensionsTests.cs:42,144`.
**Tests (26)**, all in `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/`:
| File | Lines |
|---|---|
| `Drivers/DriverHostActorTests.cs` | 28, 60, 81, 118, 143, 189 |
| `Drivers/DriverHostActorReconcileTests.cs` | 37, 59, 90, 119, 156, 193 |
| `Drivers/DriverHostActorDiscoveryTests.cs` | 385, 671, 748, 1023 |
| `Drivers/DriverHostActorWriteRoutingTests.cs` | 148, 208 |
| `Drivers/DriverHostActorPrimaryGateTests.cs` | 228, 247 |
| `Drivers/DriverHostActorVirtualTagTests.cs` | 44, 76 |
| `Drivers/DriverHostActorNativeAlarmTests.cs` | 350 |
| `Drivers/DriverHostActorNativeAlarmAckRoutingTests.cs` | 126 |
| `Drivers/DriverHostActorProbeResultDropTests.cs` | 66 |
| `Drivers/DriverHostActorHistoryWriterTests.cs` | 59 |
| `Drivers/DiscoveryInjectionEndToEndTests.cs` | 219 |
| `VirtualTags/DependencyMuxActorTests.cs` | 139 |
No site calls `Props.Create<DriverHostActor>()` or `new DriverHostActor(...)` directly — the only
`new DriverHostActor` in the tree is inside the factory at `:343`. Because the new parameter is
optional and appended last, **no test site should need editing.**
### 5.3 Status object for the running-from-cache signal
`NodeDiagnosticsSnapshot` (`Commons/Interfaces/NodeDiagnosticsSnapshot.cs:17-22`) — built in
`HandleGetDiagnostics` (`DriverHostActor.cs:1381-1398`), a positional record of
`(NodeId, RevisionHash? CurrentRevision, IReadOnlyList<DriverInstanceDiagnostics> Drivers, DateTime AsOfUtc)`.
`GetDiagnostics` is handled in **all three states** (`Steady:570`, `Applying:601`, `Stale:1349`), so
a flag here is observable regardless of state. **This is the natural home for `RunningFromCache`**
(append last to the record). Consumer: `IFleetDiagnosticsClient` → AdminUI.
`IDriverHealthPublisher` (`:73`, defaulted `:402`) is **not** a fit — the host never calls it; it
only passes it down to `DriverInstanceActor` children (`:1845`, `:1859`). Per-driver, not per-node.
### 5.4 Async / side-effect IO convention
The actor is **overwhelmingly synchronous-blocking on DB IO** — all five `CreateDbContext()` sites
use `using var db = ...` with synchronous LINQ on the actor thread. No `async`/`await` in the file,
no `Task.Run`.
The single async pattern is `ContinueWith(..., TaskScheduler.Default).PipeTo(replyTo)` in
`HandleRouteNodeWrite` (`:1188-1200`), with an explicit `Sender` capture before the continuation.
**Recommendation for Task 7:** a **synchronous** cache write at the §2.2 point is *consistent with
the file's existing style* and is correct there — the actor is mid-apply, holds `Applying`, and
every other DB call on that path already blocks. A fire-and-forget `PipeTo(Self)` would need a
handler registered in `Steady`, `Applying` **and** `Stale`, or it dead-letters after the
`Become(Steady)` at `:1456`. **Prefer synchronous + try/catch.** (This deviates from the plan's
"fire-and-forget"; see D-4.)
`IWithTimers` is already implemented (`:50`, `:282`); `"retry-db"` key in use at `:1378`/`:2009`.
---
## 6. Host composition root
### 6.1 Role flag — `Program.cs:47-50`
```csharp
var roles = RoleParser.Parse(Environment.GetEnvironmentVariable("OTOPCUA_ROLES"));
var hasAdmin = roles.Contains("admin");
var hasDriver = roles.Contains("driver");
```
Allowed roles: `admin`, `driver`, `dev` (`Cluster/RoleParser.cs:5-8`); unknown throws (`:25-27`).
### 6.2 `hasDriver` branch — `Program.cs:131-310`
Anchor for the new call: **`Program.cs:173-175`** `builder.Services.AddAlarmHistorian(...)`, inside
the config-gated cluster at `:154-195`. Must land **before** `AddAkka` at `:314`.
### 6.3 Pipeline — `Program.cs:368-404`
`:383` `MapStaticAssets` · `:385-396` `if (hasAdmin) { ... }` · **`:398` `MapOtOpcUaHealth()`** ·
`:399` `MapOtOpcUaMetrics()` · `:404` `RunAsync()`.
⚠ **There is no `hasDriver` block in the pipeline half** — only `hasAdmin` is branched on. A
driver-gated `app.Map*` is a new construct; it goes between `:396` and `:398`.
`Program.cs:406-410` re-exports `public partial class Program`.
### 6.4 Kestrel / URLs — **CONFIRMED: no Kestrel configuration exists**
- `Program.cs:57` `builder.WebHost.UseStaticWebAssets();` — the **only** `builder.WebHost` call.
- **Zero** `ConfigureKestrel` / `UseUrls` / `ListenAnyIP` / `UseKestrel` anywhere in `src/`.
- Binding is exclusively via `ASPNETCORE_URLS`: `docker-compose.yml:173` (central-1) and `:240`
(central-2) set `http://+:9000`; `launchSettings.json:7` sets `http://localhost:9000`.
Two traps for Task 5:
1. ⚠ **`scripts/install/Install-Services.ps1:186`** (`$hostEnv += "ASPNETCORE_URLS=..."`) is inside
`if ($hasAdmin)` at `:185`. **A driver-only Windows-service node gets no `ASPNETCORE_URLS` at
all** and binds the ASP.NET default. The plan's parse-URLs-then-rebind snippet would re-bind
`http://+:9000` there — a port that node never listened on. The `?? "http://+:9000"` fallback in
the plan's snippet is therefore **wrong for driver-only nodes**; see D-6.
2. ⚠ **`Host.IntegrationTests/TwoNodeClusterHarness.cs:331`** already calls
`builder.WebHost.UseKestrel(o => o.Listen(IPAddress.Parse(LoopbackHost), 0));`. Adding a
production `ConfigureKestrel` gives two competing explicit configurations in integration tests —
last-one-wins. The `syncPort > 0` gate keeps this dormant by default (harness sets no sync port),
but any harness that *does* set one will fight the port-0 binding.
**Health route for smoke tests:** `/healthz` — liveness, runs no checks, always 200 while the
process is up, `AllowAnonymous`. (`Health/HealthEndpoints.cs:48` → `MapZbHealth()`; routes from the
`ZB.MOM.WW.Health` package.) Also `/health/ready`, `/health/active`, and `/metrics` (`Program.cs:399`).
### 6.5 `SecretsRegistration` template — `Host/Configuration/SecretsRegistration.cs`
Shape for `LocalDbRegistration` to mirror: `public static class` in
`ZB.MOM.WW.OtOpcUa.Host.Configuration`; `public const string` section/key path constants; a
`public static bool IsXEnabled(IConfiguration)` predicate using
`GetValue(key, defaultValue: false)` (**default-deny**) that `Program.cs` can also call;
`AddOtOpcUaX(this IServiceCollection, IConfiguration)` returning `IServiceCollection`, both args
`ArgumentNullException.ThrowIfNull`-guarded; XML docs with `<remarks><para>` explaining *why* the
gate is default-deny.
Call-site precedent: `Program.cs:363` `AddOtOpcUaSecrets(...)`; predicate reused inside the `AddAkka`
lambda at `Program.cs:323`.
### 6.6 Telemetry meter allowlist — **EXISTS. Task 10 step 3 is REQUIRED, not conditional.**
`Host/Observability/ObservabilityExtensions.cs:25-38`:
```csharp
return services.AddZbTelemetry(o =>
{
o.ServiceName = "otopcua";
o.Meters = [OtOpcUaTelemetry.MeterName]; // <-- ObservabilityExtensions.cs:30
o.ActivitySources = [OtOpcUaTelemetry.ActivitySourceName];
```
`o.Meters` is a **strict allowlist** — `ZbTelemetryExtensions.cs:51-54` iterates it calling
`metrics.AddMeter(name)`; **no wildcard support**. Anything unlisted is silently not exported.
Single element today: `OtOpcUaTelemetry.MeterName` = `"ZB.MOM.WW.OtOpcUa"`
(`Commons/Observability/OtOpcUaTelemetry.cs:19`).
**Action:** add `LocalDbMetrics.MeterName` at `ObservabilityExtensions.cs:30`. This is exactly the
omission the ScadaBridge live gate caught.
### 6.7 Health registration — `Host/Health/HealthEndpoints.cs:19-41`
`AddOtOpcUaHealth()` **takes no parameters** and is called **unconditionally** at `Program.cs:365`,
outside both role blocks. Three `.AddTypeActivatedCheck<>` calls (`configdb`, `akka`,
`admin-leader`); **no registration-time role gating exists** — `ActiveNodeHealthCheck` does runtime
role scoping instead (returns `Healthy` when the node lacks the role).
⚠ **There are zero `IHealthCheck` implementations in this repo** — all three come from the
`ZB.MOM.WW.Health.*` packages (`Directory.Packages.props:124-126`, v0.1.0). A LocalDb check would be
the **first health-check class in the tree**; `Host/Health/` holds only `HealthEndpoints.cs` today.
**Consequence for Task 10:** driver-gating the new check requires changing
`AddOtOpcUaHealth()`'s signature (add `bool hasDriver` or `IConfiguration`) and its `Program.cs:365`
call site. **Preferred alternative:** follow the `ActiveNodeHealthCheck` precedent — register
unconditionally and have the check return `Healthy` when LocalDb is not registered. That keeps the
signature and matches the "default-OFF must not degrade a plain node" requirement for free. See D-7.
### 6.8 Config files
| File | Top-level sections |
|---|---|
| `appsettings.json` | `Serilog`, `Security`, `Secrets`, `ServerHistorian`, `ContinuousHistorization`, `AlarmHistorian`, `Deployment` |
| `appsettings.driver.json` / `.admin.json` / `.admin-driver.json` / `.Development.json` | `Serilog`, `Security` only |
Overlay selection: `Program.cs:67` sorts roles ordinally and joins with `-` → `admin,driver`
becomes `appsettings.admin-driver.json`, added `optional: true` at `:70`; env vars (`:77`) and CLI
args (`:78`) are re-appended after, so deployment overrides outrank the overlay.
**`LocalDb` key search: CONFIRMED ABSENT.** Case-insensitive `localdb` across all `*.json`,
`*.yml`, `*.ps1`, `*.csproj`, `*.props` matches only the two Phase-1/Phase-2 planning artifacts. No
conflicting keys in any overlay or in `Install-Services.ps1`.
**Placement convention:** each feature section opens with a `"_comment"` string then `"Enabled":
false`; per-field caveats use sibling `"_<Field>Comment"` keys (`appsettings.json:34`, `:45`). Place
`LocalDb` between `AlarmHistorian` (ends `:61`) and `Deployment` (`:62`).
---
## 7. LiteDB LocalCache — dormant, safe to delete
### 7.1 Files (6) under `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/`
`ILocalConfigCache.cs`, `GenerationSnapshot.cs`, `StaleConfigFlag.cs`, `LiteDbConfigCache.cs`
(also declares `LocalConfigCacheCorruptException` at `:128`), `GenerationSealedCache.cs`,
`ResilientConfigReader.cs`. Single namespace `...Configuration.LocalCache`. The only two
`using LiteDB;` statements in `src/` are `LiteDbConfigCache.cs:1` and `GenerationSealedCache.cs:1`.
### 7.2 Reference classification — **no live-code references. NO STOP.**
- **(c) live code elsewhere: ZERO.** Also confirms full dormancy: `SealedBootstrap` (referenced by
`docs/v2/v2-release-readiness.md:46`) has **zero hits** in `src/` and `tests/` — it no longer exists.
- Two benign near-misses:
- `Configuration/Services/ILdapGroupRoleMappingService.cs:14` — the string `ResilientConfigReader`
inside an **XML doc comment**. Prose only. Needs a comment edit or the doc goes stale.
- `Configuration/ZB.MOM.WW.OtOpcUa.Configuration.csproj:29` — `<PackageReference Include="LiteDB"/>`,
the only one in the repo.
- Dozens of `LocalConfigCacheCorruptException` hits under `src/**/bin/**/*.xml` are **generated doc
build artifacts** (`GenerateDocumentationFile=true`), not source.
### 7.3 ⚠ Test-file drift — the plan's delete list is incomplete
Plan `…phase1.md:484-485` names only `GenerationSealedCacheTests.cs` and
`ResilientConfigReaderTests.cs`. Actual test files in
`tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests/`:
| File | Status |
|---|---|
| `GenerationSealedCacheTests.cs` | in plan ✅ |
| `ResilientConfigReaderTests.cs` | in plan ✅ — note `StaleConfigFlagTests` is co-located here at `:323` |
| **`LiteDbConfigCacheTests.cs`** | **MISSING from the plan** — will fail to compile if the source is deleted (`:125` calls `new LiteDB.LiteDatabase(...)` directly) |
See D-3.
### 7.3b Outcome (Task 9, executed 2026-07-20)
Deleted: the 6 `LocalCache/` sources and **all three** test files (including
`LiteDbConfigCacheTests.cs`, which the plan omitted — D-3). `LiteDB` removed from both
`Configuration.csproj` and `Directory.Packages.props`. The stale XML-doc reference at
`ILdapGroupRoleMappingService.cs:14` was rewritten to state the present truth (no fallback exists;
reviving it means an admin-side cache on LocalDb, not restoring the old pipeline) rather than
deleted, per the DoD's "explanatory prose may remain". Configuration tests: 92/92 green.
**Follow-up found while executing:** `Polly.Core` in `Configuration.csproj` is now orphaned too —
`ResilientConfigReader` was its only real consumer in that project (remaining `Polly` mentions are
comments). Left in place deliberately: removing it is outside Task 9's scope and could break a
consumer relying on the transitive flow. Worth a separate cleanup.
### 7.4 Package removal
- `PackageReference`: `Configuration.csproj:29` — only consumer.
- `PackageVersion`: `Directory.Packages.props:35` — `LiteDB` `5.0.21`.
Both safe to delete; no transitive consumer.
---
## 8. Test projects, rig, packages
### 8.1 Test project layout
| Question | Answer |
|---|---|
| `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/` | **DOES NOT EXIST** — there is no Host *unit*-test project at all |
| Closest Host project | `Host.IntegrationTests` — despite the name it carries plenty of pure unit tests (`LdapOptionsValidatorTests`, `ServerHistorianOptionsValidatorTests`, `ResilienceInvokerFactoryRegistrationTests`, …) |
| `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/` | **EXISTS** |
| DriverHostActor tests | `Runtime.Tests/Drivers/` — 11 `DriverHostActor*Tests.cs` files. **xunit v2** (`xunit` 2.9.3), references **`Akka.TestKit.Xunit2`** ✅ |
| `Host.IntegrationTests` xunit version | **xunit.v3** 1.1.0; SDK `Microsoft.NET.Sdk.Web` |
`Directory.Packages.props` documents that AdminUI.Tests / ControlPlane.Tests / **Runtime.Tests** are
the three xunit-v2 holdouts, held there by `Akka.TestKit.Xunit2` (verified 2026-07-13: no xunit.v3
Akka TestKit exists). This matches the design's "actor tests live in an xunit2 TestKit project".
### 8.2 ⚠ `WebApplicationFactory<Program>` is **deliberately not used** — Task 11 must be rewritten
The only mention of the type in the entire `tests/` tree is a comment explaining why it is avoided:
`Host.IntegrationTests/TwoNodeClusterHarness.cs:48`
```
/// <para>Why not <c>WebApplicationFactory&lt;Program&gt;</c>? Program.cs reads <c>OTOPCUA_ROLES</c> ...
```
The actual pattern builds the host directly — `TwoNodeClusterHarness.cs:329`:
```csharp
var builder = WebApplication.CreateBuilder(new WebApplicationOptions { Args = [] });
```
`Microsoft.AspNetCore.Mvc.Testing` is still referenced (csproj `:15`) but only for transitive
test-host plumbing. **Copy `TwoNodeClusterHarness`, not `WebApplicationFactory`.** See D-5.
`Host.IntegrationTests` also has its own `docker-compose.yml` (sql `14331`, ldap `3894`) gated by
`DockerFixtureAvailability.cs`.
### 8.3 docker-dev rig topology — plan's site-a/site-b assumption is **correct**
Single Akka mesh (`otopcua`), seeded by `central-1`. Tenant separation is by `ServerCluster.ClusterId`
rows (MAIN / SITE-A / SITE-B) inside **one shared ConfigDb** — not separate meshes.
| Service | Roles | Cluster |
|---|---|---|
| `central-1`, `central-2` | `admin,driver` | MAIN |
| `site-a-1`, `site-a-2` | `driver` | SITE-A |
| `site-b-1`, `site-b-2` | `driver` | SITE-B |
| `sql`, `migrator`, `cluster-seed`, `traefik` | — | infra |
**There is no admin-only service** — the "admin" nodes are fused `admin,driver`. All six host nodes
share the `&otopcua-host` YAML anchor.
⚠ **NO HOST NODE HAS A WRITABLE VOLUME.** The only `volumes:` in the file are `sql` →
`otopcua-mssql-data`, plus two read-only binds (`./seed:ro`, `./traefik-dynamic.yml:ro`). All six
nodes write to the container's ephemeral layer, destroyed on recreate. This is already an accepted
pattern (per-container `Secrets:SqlitePath=otopcua-secrets.db`, replicated over Akka rather than
persisted). **Task 13 must add six named volumes** — there is nothing to piggyback on. See D-8.
**Env style: confirmed double-underscore `Section__Key`**, with array indices and dictionary keys as
a third segment (`Cluster__SeedNodes__0`, `Security__Ldap__GroupToRole__ReadOnly`). Exceptions that
are plain SCREAMING_SNAKE (read directly, not config-bound): `OTOPCUA_ROLES`,
`ZB_SECRETS_MASTER_KEY`, `GALAXY_MXGW_API_KEY`, `OTOPCUA_CONFIG_CONNECTION`, `ASPNETCORE_URLS`.
**Ports.** Host-published: `14330`→sql, `4840`-`4845`→the six nodes' OPC UA, `9200`→traefik web,
`8089`→traefik dashboard. Container-internal on every node: `4053` (Akka remoting), `9000`
(Kestrel), `4840` (OPC UA). Also reserved on this machine: `14331`/`3894` (Host.IntegrationTests
compose), `80`/`8080` (sibling scadabridge/scadalink stack), `10.100.0.35:3893` (shared GLAuth).
**`9001` (the plan's choice) is free** container-internal and need not be published. ✅
### 8.4 Package versions — `Directory.Packages.props`
| Package | Version | Line |
|---|---|---|
| `Grpc.Core.Api` | 2.76.0 | 32 |
| `Grpc.Net.Client` | 2.76.0 | 33 |
| **`Grpc.AspNetCore`** | **NOT PRESENT** | — |
| `Google.Protobuf` | 3.34.1 | 31 |
| `Microsoft.Data.Sqlite` | 10.0.7 | ~55 |
| `SQLitePCLRaw.bundle_e_sqlite3` | 2.1.12 | ~108 |
| `LiteDB` | 5.0.21 | 35 |
Both Grpc floors and the Protobuf floor are already satisfied — **no bumps needed**, only the new
`Grpc.AspNetCore` entry (use 2.76.0 to match the train; watch for NU1605/NU1608 if it wants a
Protobuf newer than the pinned 3.34.1).
⚠ **`SQLitePCLRaw` is a surgical direct pin, not a transitive one.**
`CentralPackageTransitivePinningEnabled` is deliberately **off** (it breaks the Roslyn version
split). The 2.1.12 pin is promoted by a **direct `PackageReference` in `Core.AlarmHistorian`**
(`Core.AlarmHistorian.csproj:15` + `:21`) — today the only Sqlite consumer in `src/`.
> **Any new project that pulls `Microsoft.Data.Sqlite` MUST also take a direct
> `<PackageReference Include="SQLitePCLRaw.bundle_e_sqlite3"/>`,** or it silently resolves the
> vulnerable 2.1.11 native bundle. This applies to **Host** and **Runtime** (both gain `ZB.MOM.WW.LocalDb`,
> which depends on `Microsoft.Data.Sqlite`) and to every test project that builds a real `ILocalDb`.
> Copy the `Core.AlarmHistorian.csproj:15,21` pair exactly. The plan's Task 1 mentions
> `SQLitePCLRaw.lib.e_sqlite3` "if the audit flags it" — the correct package is
> **`bundle_e_sqlite3`**, and it is **required, not conditional**.
---
## 9. Deviations from the plan
Recorded per the tasks.json `deviation` convention. **D-1, D-3, D-5, D-6 change the work
materially.**
| # | Plan says | Reality | Resolution |
|---|---|---|---|
| **D-1** | Cache keyed by `ClusterId`; "cluster identity" is a recon item with a STOP condition | `ClusterId` is stable, but **not resolvable at the boot seam** — it lives inside the artifact or in the unreachable central DB (§4) | Keep `cluster_id` as PK (pair sharing is the point). **Write:** resolve via `DeploymentArtifact.ResolveClusterScope`, `"__single"` sentinel for the degenerate case. **Read:** unkeyed `ORDER BY applied_at_utc DESC LIMIT 1`, warn if >1 row. No STOP. |
| **D-2** | Cache write "fire-and-forget (`PipeTo`-style or guarded `Task.Run`)" | The actor has **no async DB IO at all**; a `PipeTo(Self)` needs handlers in all three states or it dead-letters after `Become(Steady)` (§5.4) | Synchronous write in its own try/catch at `:1439`. Matches the file's style; failure still cannot fail the apply. |
| **D-3** | Task 9 deletes 2 test files | **3** test files reference the subsystem — `LiteDbConfigCacheTests.cs` is missing from the list (§7.3) | Delete all three. Also fix the stale XML-doc mention at `ILdapGroupRoleMappingService.cs:14`. |
| **D-4** | Cache the artifact after a successful apply | `ReconcileDrivers`/`PushDesiredSubscriptions` **swallow** DB errors, so a "successful" apply can have loaded an **empty blob** and started zero drivers (§2.3) | Gate the write on a non-empty blob; surface the blob from `ReconcileDrivers` instead of a third read. **Load-bearing — without it the cache can be poisoned with an empty config.** |
| **D-5** | Task 11 uses `WebApplicationFactory<Program>` | Deliberately **not used** in this repo; `TwoNodeClusterHarness.cs:48` documents why (Program.cs reads `OTOPCUA_ROLES` from the environment) (§8.2) | Rewrite Task 11 against the `TwoNodeClusterHarness` direct-host-build pattern. |
| **D-6** | Task 5 re-binds `ASPNETCORE_URLS ?? "http://+:9000"` | `Install-Services.ps1:185-186` sets `ASPNETCORE_URLS` **only when `hasAdmin`** — the fallback would bind a driver-only node to a port it never served (§6.4) | Re-bind only when a URL is actually configured; when absent, add **only** the sync listener. Also note `TwoNodeClusterHarness.cs:331` already calls `UseKestrel(Listen(…, 0))`. |
| **D-7** | Task 10 driver-gates the health check | `AddOtOpcUaHealth()` takes no args and is called unconditionally at `Program.cs:365`; no registration-time gating exists anywhere (§6.7) | Register unconditionally; return `Healthy` when LocalDb is absent — the `ActiveNodeHealthCheck` precedent. Avoids a signature change and satisfies "default-OFF must not degrade a plain node". |
| **D-8** | Task 13 "existing data mount" | **No host node has any writable volume** (§8.3) | Add six named volumes. Note: `9001` is free container-internal. Meter allowlist (§6.6) is **required**, not conditional. |
### Follow-ups logged, not fixed here
1. **`TryRecoverFromStale` marks `Applied` without reconciling drivers** (§3.5) — pre-existing,
documented in-code at `:1692-1695`. Out of Phase 1 scope.
2. **Duplicate blob read per apply** (`:1472` and `:1543`, §1.1) — D-4's fix removes one of them as a
side effect.
3. **`Install-Services.ps1` gives driver-only nodes no `ASPNETCORE_URLS`** (§6.4) — pre-existing;
D-6 works around it rather than fixing it.
docs/plans/2026-07-20-localdb-phase2-live-gate.md
+180
View File
@@ -0,0 +1,180 @@
# LocalDb Phase 2 — live gate (docker-dev rig)
> Task 8 of `2026-07-20-localdb-adoption-phase2.md`. Evidence log.
> Rig: local `docker-dev/docker-compose.yml` — 2 central + site-a pair (replication ON) + site-b pair
> (default-OFF), one Akka cluster, one shared SQL.
>
> **Safety rules followed** (same as the Phase 1 gate): all DB inspection via `docker cp` of the
> `db`/`-wal`/`-shm` triplet out of the container, querying the copy — never host `sqlite3` on the
> live WAL file.
## Headline
**The gate did not get past its own first boot before finding real defects.** Four production bugs,
all fixed on-branch with regression tests, plus a fifth finding in the test suite itself. Three of
the four crash-looped every driver node; the fourth silently stopped alarm history everywhere.
Checks 1, 2, 5 and 6 pass. **Checks 3 and 4 are NOT satisfied** — see "The blocker" below; they
cannot be run as written on a rig that has no per-pair redundancy roles, and the defect they were
meant to confirm turned out to be the opposite of what the plan assumed.
## Seeding the migration
The rig had no pre-consolidation `alarm-historian.db`, so one was synthesised per node and `docker
cp`-ed to `/app/alarm-historian.db` (the WORKDIR-relative default) into created-but-not-started
containers — the exact upgrade shape.
Deliberately overlapping content, to test D-6 in production: **6 rows on site-a-1, 5 on site-a-2,
2 of them byte-identical payloads** (the warm-pair duplication `HistorianAdapterActor` produces in
every boot window). 11 legacy rows, **9 distinct**.
## Defects found and fixed
### 1. Empty `ServerHistorian:ApiKey` crash-loops the host (was classified "degrades")
`AlarmHistorian:Enabled=true` requires the gateway alarm writer, which requires a key.
`HistorianGatewayClientOptions.Validate()` throws `The gateway API key must not be empty` inside the
client factory during Akka startup, so every driver node crash-looped.
`ServerHistorianOptionsValidator` exists precisely to convert that class of failure into a named
`OptionsValidationException` — but its documented fail tier explicitly excluded `ApiKey`, reasoning
that a keyless client "degrades rather than crashes (the gateway rejects calls)". **That premise is
false**: the client validates its own options at construction, so the process dies before any call.
Fixed; the key is never echoed in the message.
### 2. `UseTls` / endpoint-scheme mismatch crash-loops the host
Immediately after fixing #1, the rig crash-looped again: `UseTls` defaults to `true`, the rig
endpoint is `http://`, and the client throws `UseTls requires an https gateway endpoint`. The
inverse (`https://` + `UseTls=false`) throws too — both strings confirmed in the shipped client
assembly. Setting an `http` endpoint without clearing `UseTls` is an ordinary migration slip that
takes the host down. Now validated in both directions.
### 3. Plaintext h2c was *unreachable* — every `http://` deployment crashed
Third crash-loop, and a genuine product bug rather than operator config.
`HistorianGatewayClientAdapter.Create` forwarded the TLS-only options unconditionally, and
`AllowUntrustedServerCertificate` defaults to `false`, so it always sent
`RequireCertificateValidation=true` — which the client rejects outright when `UseTls=false`
(`RequireCertificateValidation is a TLS-only option and requires UseTls=true`).
CLAUDE.md and `docs/Historian.md` both document `http://` as the supported way to select h2c, but
**no h2c configuration could start**, and the only workaround was to claim
`AllowUntrustedServerCertificate=true` — a certificate posture for a connection with no certificate.
Fixed: TLS-only options stay at their defaults when `UseTls=false`.
### 4. THE BLOCKER — the drain gate deferred to a Primary that cannot deliver
With the rig finally up, **every driver node logged `Historian drain suspended`. Nothing drained
anywhere** — including the two site-b nodes, which have no redundancy peer and no replication at all.
Before Phase 2 this deployment drained fine, because the drain was ungated.
Root cause, established from the rig rather than guessed:
- `RedundancyStateActor` derives roles from Akka's **cluster-wide** `RoleLeader("driver")`.
- `central-1`/`central-2` carry **both** `admin` and `driver` Akka roles (`OTOPCUA_ROLES=admin,driver`).
- So the elected driver Primary is **central-1** — a node that replicates nobody's LocalDb and does
not even run the alarm historian.
- Every site node was `Secondary`, saw that a Primary existed, and stood down in its favour.
**The redundancy role is a cluster-wide election; the alarm queue is pair-local.** Deferring on
"somebody is Primary" is therefore wrong in principle: in a multi-pair fleet the Primary is usually
in another pair and holds none of this node's rows. The consequence is not a duplicate — it is the
buffer growing on every node until the capacity wall evicts the oldest events, i.e. silent permanent
loss of the audit trail the queue exists to protect.
Fixed in three layers, each with a positive control:
1. `PrimaryGatePolicy.ShouldDrainAlarmHistory(role, queueSharingPeerIsPrimary)` — a **separate**
policy from `ShouldServiceAsPrimary`. Unknown role ⇒ drain. A node stands down only for a Primary
that holds its rows. The two gates now deliberately disagree, and a test pins that disagreement so
nobody "harmonises" them back.
2. `DriverHostActor` publishes that verdict, matching the snapshot's Primary against this node's
configured LocalDb replication peer host.
3. `AddAlarmHistorian` short-circuits the gate entirely when replication is not configured — a node
whose rows exist nowhere else must never defer. The predicate tests **both** `Replication:PeerAddress`
and `SyncListenPort`, because only the dialing half sets the former and both halves share the queue.
The asymmetry of the error costs is what drives every one of these: a false allow costs a duplicate
history row, which the design already accepts (*at-least-once across a failover, by design*, and
payload-hash ids collapse identical events); a false deny loses data silently.
### 5. A third vacuous test — `AwaitAssert` on the seeded value
Deleting the guard (publishing the old write-gate verdict) left `DriverHostActorRoleViewTests`
**green**. `AwaitAssert` polls until an assertion passes, and the assertion was "the view reads
open" — which is also the value the view is *seeded* with, so it succeeded at the first poll, before
the actor had processed anything. It could not distinguish a correct publish from no publish at all.
Fixed by asserting on the **sequence of published values** via a recording view, never on current
state. Re-run under the control: red, for exactly the cases that matter. This is the same
"absence-assertion passes by race" trap recorded for #485/#486, in a new disguise.
## Checks
| # | Check | Result | Evidence |
|---|---|---|---|
| 1 | Migration ran; `.migrated` sidecar on both site-a nodes; row counts match | ✅ PASS | `alarm-historian.db.migrated` present on both. 11 legacy rows across the two files → **exactly 9 rows on each node**, id sets byte-identical — the 2 shared payloads collapsed, proving D-6's payload-hash identity in production. Each node holds the *other's* rows (`legacy/a1-only-*` present on a-2 and vice versa). Legacy `attempt_count` distinction survived the copy (the 3-attempt row still leads the 0-attempt rows), `last_error` carried, 0 dead-lettered |
| 2 | Alarm burst converges: identical rowsets, oplog drains to 0, dead letters 0 | ✅ PASS | Identical id sets on both nodes; `__localdb_oplog` depth **0** on both; **0** dead-lettered. Drain visibly live against the deliberately unresolvable gateway — `attempt_count` climbing 33 → 42 → 168 with `last_error='retry-please'`, which is exactly the buffering behaviour the queue exists for |
| 3 | Only the primary drains | ❌ **NOT SATISFIED** — see defect 4 | As written the check assumes a per-pair Primary exists. On this rig the elected Primary is a central node outside both pairs, so the honest post-fix behaviour is that **all four nodes drain**. No node can currently identify a queue-sharing Primary: site-b is unpaired, and of the site-a pair only the dialing half knows its peer. Safe (no loss, duplicates only), but the check's intent is unproven |
| 4 | Failover: standby resumes draining; count double-delivered events | ⛔ **NOT RUN** | Superseded by defect 4. A meaningful failover test needs a rig where the pair itself elects a Primary; measuring "double delivery" is meaningless while both halves drain unconditionally |
| 5 | Both nodes restarted together: clean rejoin, identical counts, no I/O errors | ✅ PASS | `docker compose restart site-a-1 site-a-2` → **0** `disk I/O error` / `SQLITE_IOERR` / `corrupt` lines and 0 fatals in either log; both back to 9 rows / 0 dead / oplog 0, still identical |
| 6 | site-b default-OFF pin: sink works locally, no sync traffic | ✅ PASS | `alarm_sf_events` created and registered on both site-b nodes; port **9001 not bound** (`/proc/net/tcp`), no sync listener, no replication config; both drain their own queue (defect 4's third fix) with 0 fatals |
## The open design question
The gate leaves one question that is a design fork, not a bug to patch:
**A pair cannot currently identify its own Primary.** Redundancy roles are elected cluster-wide, but
the queue is pair-local. Only the dialing half of a pair knows its peer (`Replication:PeerAddress`);
the listening half sets `SyncListenPort` only. So the drain gate can never fully engage, and both
halves of a replicated pair drain — safe, but duplicated.
### Follow-up analysis — the root cause is wider than the drain gate
Two things assumed while writing the section above turned out to be wrong, and correcting them
changes the recommendation.
**The rig is not misconfigured.** `central-1`/`central-2` carry `admin,driver` deliberately — the
compose says so in as many words ("central is admin,driver"), because they are themselves a
redundant pair that also runs the MAIN cluster's drivers. Stripping the driver role would change what
the rig models, not fix anything.
**A "pair" is not a missing concept — it already exists, as the application Cluster.** OtOpcUa's
`Cluster`/`ClusterId` (the config-DB entity that owns drivers, devices and `ClusterNode` rows) is
exactly the unit a redundant pair belongs to. Phase 1 already relies on this: the deployment-artifact
cache is *keyed by ClusterId so a pair shares one entry*, and the rig's three application clusters
(MAIN, SITE-A, SITE-B) are precisely its three pairs.
So the real defect is that **`RedundancyStateActor` elects one Primary per *Akka* cluster, while
every meaningful unit of redundancy is an *application* cluster.** The rig runs six driver nodes
across three application clusters in one Akka cluster — a topology the product otherwise supports
throughout — and exactly one of those six can ever be Primary.
That mis-scoping is not confined to the alarm drain. Every consumer of the role inherits it:
`ServiceLevel`, the `alerts` emit gate, and the inbound device-write gate. On a multi-cluster fleet
today, only one driver node cluster-wide will service Primary-gated work at all.
### Revised recommendation
**(C), as a separate change with its own live gate — and explicitly *not* (A).**
- (C) is no longer "invent a pair concept"; it is "elect per `ClusterId` instead of per Akka cluster",
using a first-class entity that already exists. It is feasible where it needs to be:
`RedundancyStateActor` has no DB access today, but it is an admin-role singleton in the same
assembly as `AdminOperationsActor`, which already reads and groups `ClusterNodes` by cluster.
- **(A) is now actively undesirable.** Deriving pair identity from `LocalDb:Replication:PeerAddress`
would introduce a *second*, parallel notion of "who is my partner" that (C) immediately obsoletes —
and it cannot be done cleanly anyway: the library exposes no initiator/passive flag, so making both
halves declare a peer means both halves *dial*, opening two duplex sessions purely to obtain a
configuration value.
- (C) touches the inbound device-write gate, which is safety-critical (the archreview 03/S4
dual-primary fix). It therefore belongs in its own change, with its own live gate, rather than
bolted onto Phase 2.
Until then this branch is safe in every topology — no configuration loses data — so what remains is
reclaiming the de-duplication benefit, not an outstanding hazard.
## Not merged
Per the plan, nothing on this branch is merged.
docs/plans/2026-07-20-localdb-phase2-recon.md
+371
View File
@@ -0,0 +1,371 @@
# LocalDb Phase 2 — Recon findings
**Date:** 2026-07-21 · **Branch:** `feat/localdb-phase2` · **Base:** `master` `2e46d054`
Answers Task 0 of `2026-07-20-localdb-adoption-phase2.md`. Every claim carries a `file:line`
citation against the base commit. Read the **Deviations** section last — two of them change the
shape of Tasks 2/3 from what the plan assumed.
---
## 1. The sink's table schema
**STOP condition does NOT fire** — confirmed by re-reading the executed DDL, as the plan's
pre-answer instructed.
Executed DDL: `SqliteStoreAndForwardSink.cs:652-670` (`InitializeSchema`). It matches the class
doc-comment at `:17-26` exactly — no drift between documented and executed schema.
```sql
CREATE TABLE IF NOT EXISTS Queue (
RowId INTEGER PRIMARY KEY AUTOINCREMENT,
AlarmId TEXT NOT NULL,
EnqueuedUtc TEXT NOT NULL,
PayloadJson TEXT NOT NULL,
AttemptCount INTEGER NOT NULL DEFAULT 0,
LastAttemptUtc TEXT NULL,
LastError TEXT NULL,
DeadLettered INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX IF NOT EXISTS IX_Queue_Drain ON Queue (DeadLettered, RowId);
```
| Question | Answer |
|---|---|
| BLOB columns? | **None.** All 8 columns are TEXT/INTEGER. Payload is `PayloadJson TEXT NOT NULL`. No base64 conversion needed; the 171 KB chunk sizing is moot. |
| PK autoincrement? | **Yes** — `RowId INTEGER PRIMARY KEY AUTOINCREMENT`. The migrator must mint deterministic `mig-{node}-{legacyId}` ids and the new table needs a TEXT PK, as the plan specified. |
| Indices to carry | One: `IX_Queue_Drain (DeadLettered, RowId)` — the drain's covering index. `alarm_sf_events` wants the same shape over (status, insertion order). |
| Largest realistic payload | `PayloadJson` is a serialized `AlarmHistorianEvent` — 10 scalar fields (`AlarmHistorianEvent.cs`), no collections, no nesting. Low single-digit KB worst case; `Comment` is the only loosely-bounded field. |
**Legacy column list** for the migrator's `pragma_table_info` intersection check:
`RowId, AlarmId, EnqueuedUtc, PayloadJson, AttemptCount, LastAttemptUtc, LastError, DeadLettered`.
`RegisterReplicated` will accept this table: no BLOB, and the PK will be app-minted TEXT.
## 2. The public seam
`IAlarmHistorianSink` (`IAlarmHistorianSink.cs:23-34`) is **two members**:
```csharp
Task EnqueueAsync(AlarmHistorianEvent evt, CancellationToken cancellationToken);
HistorianSinkStatus GetStatus();
```
The concrete `SqliteStoreAndForwardSink` additionally exposes, and these are all consumed
somewhere, so the rewire must keep them:
| Member | Cited | Consumer |
|---|---|---|
| `StartDrainLoop(TimeSpan)` | `:189` | `ServiceCollectionExtensions.cs:105` (DI factory) |
| `DrainOnceAsync(CancellationToken)` | `:320` | tests only — the deterministic drive |
| `RetryDeadLettered()` | `:511` | operator action (AdminUI) |
| `CurrentBackoff` | `:676` | tests |
| `DebugCapacityProbeCount` | `:99` | tests |
| `DefaultCapacity` / `DefaultMaxAttempts` consts | `:49`,`:53` | `AlarmHistorianOptions.cs:38,45` |
| `IDisposable` | `:679` | DI + tests; **disposes the injected writer too** |
There is **no separate enqueue/dequeue/markDelivered/deadLetter interface** — the plan's item 2
guessed at one. Dequeue, delivered-marking and dead-lettering are all *private* to the sink
(`ReadBatch :532`, `DeleteRow :564`, `DeadLetterRow :573`, `BumpAttempt :587`). That is good news:
the rewire is entirely internal to one class and the public seam is untouched.
## 3. Semantics to preserve
All defaults live in **two** places that must stay in agreement:
| Semantic | Sink default | Options default | Enforced at |
|---|---|---|---|
| `Capacity` 1,000,000 | `:49` | `AlarmHistorianOptions.cs:38` | `EnforceCapacityFastPathAsync :271` → `EnforceCapacityAsync :602` |
| `MaxAttempts` 10 | `:53` | `:45` | drain loop `:429` |
| Dead-letter retention 30 d | `:50` | `:41` | `PurgeAgedDeadLetters :638`, run once per drain tick |
| `BatchSize` 100 | ctor arg `:116` | `:31` | `ReadBatch :542` |
| `DrainIntervalSeconds` 5 | — | `:34` | `StartDrainLoop` via `ServiceCollectionExtensions.cs:105` |
Behaviours that are **not** in the plan's list but are load-bearing, and which the cutover must
not silently drop:
- **Backoff ladder** 1s→2s→5s→15s→60s (`:57-64`), applied to the timer's *next due time*
(`RescheduleDrain :224`) so an outage genuinely slows the cadence.
- **Overflow evicts oldest non-dead-lettered rows** with a WARN and a lifetime
`EvictedCount` counter (`:630-635`) — the durability guarantee is explicitly bounded.
- **Corrupt-payload rows dead-letter immediately** for their own RowId so they cannot stall the
queue head (`:344-361`).
- **Cardinality guard**: a writer returning ≠1 outcome per event is treated as a batch retry
(`:393-404`).
- **In-memory row counter with periodic resync** every 10,000 enqueues (`:89-96`, `:271-290`) —
a perf optimisation that avoids `COUNT(*)` on the hot enqueue path.
- **Post-dispose throwing contract**: `GetStatus`/`RetryDeadLettered`/`EnqueueAsync`/`StartDrainLoop`
throw `ObjectDisposedException`; `DrainOnceAsync` returns quietly (`:322`). Pinned by a
regression test (`SqliteStoreAndForwardSinkTests.cs:854-865`).
## 4. Drain-worker lifecycle — **the finding that reshapes Task 3**
> The plan asked "hosted service or actor?". **Neither.**
The drain worker is a **self-rescheduling one-shot `System.Threading.Timer` owned by the sink
itself** (`_drainTimer`, `:76`; started by `StartDrainLoop :189`; callback `DrainTimerCallback :199`;
re-armed by `RescheduleDrain :224`). It is started from inside the DI singleton factory at
`ServiceCollectionExtensions.cs:105`, so it begins ticking the moment the first consumer resolves
`IAlarmHistorianSink`.
Consequences for the Primary gate:
1. **The gate has to live inside `DrainOnceAsync`**, not in a wrapper. There is no external
scheduler to gate.
2. **`Core.AlarmHistorian` cannot see `PrimaryGatePolicy`.** The policy is in
`Runtime/Drivers/PrimaryGatePolicy.cs`, and Runtime → Core is the reference direction.
`Core.AlarmHistorian` references only `Core.Abstractions` (`.csproj`). So the sink takes a
**`Func<bool>` gate delegate** (default: always-drain) and Runtime supplies the lambda that
calls `PrimaryGatePolicy`. No new project reference, no new interface in Core.
3. **A role view is still needed** because the timer fires outside any actor. See the next point
for who should own it.
### There is already a Primary gate — on the *enqueue* side, with a different policy
`HistorianAdapterActor` (`Historian/HistorianAdapterActor.cs`) already subscribes to
`RedundancyStateChanged` (`:86`, handler `:145`), caches `_localRole` (`:42`), and gates the
enqueue on `ShouldHistorize()` (`:97`):
```csharp
private bool ShouldHistorize() => _localRole is not (RedundancyRole.Secondary or RedundancyRole.Detached);
```
This is **not** `PrimaryGatePolicy` — it ignores driver-member count and defaults to *write* while
the role is unknown. Its comment (`:68-71`) explains why it exists: DPS fans the Primary's single
`alerts` publish to **every** node's adapter, so without the gate both nodes of a pair would
double-write.
**This is why today's ungated drain is safe, and why Phase 2 breaks that.** Today the Secondary
never enqueues, so its queue is empty and an ungated drain has nothing to send. Once
`alarm_sf_events` replicates, the Primary's rows land in the Secondary's table — and an ungated
drain there would double-deliver *every* event, continuously, not just at failover. **The drain
gate is not a nicety; it is required by the same commit that registers the table.**
### Who owns the role view
`HistorianAdapterActor`, not `DriverHostActor`. Both are spawned under
`WithOtOpcUaRuntimeActors` (`ServiceCollectionExtensions.cs:353`) which `Program.cs:133` calls only
under `hasDriver` — as are `AddAlarmHistorian` (`:175`) and `AddOtOpcUaLocalDb` (`:185`). So sink,
LocalDb and adapter are co-located by construction. The adapter wins over `DriverHostActor`
because it **already** holds the role for exactly this purpose; using it adds no new subscription
and no new way for the view to go stale while the sink keeps draining.
`DriverMemberCount` is read off Akka cluster state in `DriverHostActor.DriverMemberCount() :1446`
(try/catch → 0 on a non-cluster provider ⇒ single-node posture). Task 3 will lift that into a
small shared helper rather than copy it.
## 5. Other `DatabasePath` construction sites
- `ServiceCollectionExtensions.cs:98` — the only production site.
- `appsettings.json:57` — `"DatabasePath": "alarm-historian.db"`.
- `AlarmHistorianOptions.Validate() :54-55` — warns when the path is relative. **This warning
disappears with the key**; the validator loses one of its five checks.
- `SqliteStoreAndForwardSinkTests.cs` — ~30 construction sites, all passing the per-test temp
`_dbPath` seeded at `:25` and deleted in `Dispose :32`. These are the tests Task 2 rewires onto
a temp-file `ILocalDb`.
- **`docker-dev/docker-compose.yml` has no `AlarmHistorian` keys at all** — the rig runs with
`Enabled=false` today. Task 6 is therefore an *addition*, not an edit.
No tooling or CLI constructs the sink.
## 6. How `Enabled=false` short-circuits
`AddAlarmHistorian` returns before registering anything when the section is absent or
`Enabled != true` (`ServiceCollectionExtensions.cs:87`), leaving the `NullAlarmHistorianSink`
seeded by `AddOtOpcUaRuntime :49`. The Null sink is a pure no-op (`IAlarmHistorianSink.cs:37-52`).
Confirming the plan's own answer: **the tables are created unconditionally in `OnReady`** (cheap,
empty) because `AddOtOpcUaLocalDb` is independent of the `AlarmHistorian` gate. Only the
sink/drain stay Null. This is also required for correctness — `RegisterReplicated` must run on
both pair members regardless of their historian config, or a node that later enables the sink
would write rows that are never captured (see the CDC note below).
## 7. Library constraints re-confirmed (README, `ZB.MOM.WW.LocalDb` 0.1.3)
- **BLOB columns are rejected at registration.** Not a problem here (§1).
- **CDC semantics** — rows that existed *before* `RegisterReplicated` are neither captured nor
snapshotted. This is the mechanical reason the migrator must run **after** registration, and
the reason `LocalDbSetup.OnReady`'s ordering comment is load-bearing (`LocalDbSetup.cs:23-31`).
- **LWW keyed on the PK** — two nodes minting the same PK for *different* rows silently overwrite
each other; two nodes minting the same PK for the *same* row converge. Decision D-1 below turns
that second property into a feature.
- `ILocalDb` surface in use: `ExecuteAsync` / `QueryAsync(sql, rowMapper, params, ct)` /
`BeginTransactionAsync` / `CreateConnection` / `RegisterReplicated`
(`Deployment/LocalDbDeploymentArtifactCache.cs:77,82,205`).
---
## Deviations and decisions
Recorded here rather than raised as questions, per the plan's follow-up convention.
### D-1 — Enqueue ids are a deterministic content hash, not `Guid.NewGuid()`
**The plan says** GUIDs minted at enqueue (Task 2 step 2).
**The problem.** The enqueue gate (`ShouldHistorize`, default-**write** on unknown role) and the
drain gate the plan specifies (`PrimaryGatePolicy`, default-**deny** on unknown role when paired)
disagree during the boot window. On a two-node pair before the first redundancy snapshot arrives,
**both** adapters enqueue the same DPS-fanned transition. With random GUIDs those are two distinct
rows that replicate into each other's tables — the buffer *duplicates* instead of converging, and
whichever node becomes Primary later delivers both copies.
**The decision.** Mint `id` as a hash over the serialized payload. Two nodes independently
enqueuing the same fanned event produce the same PK, so LWW converges them to one row for free.
Two genuinely distinct events cannot collide: `AlarmHistorianEvent` carries a full-precision
`TimestampUtc` plus alarm id, kind, message and user, so an identical hash means an identical
event.
This costs one `SHA256.HashData` per enqueue and removes a whole duplicate-row class. It also
slightly *improves* today's behaviour, where the same boot-window fan-out is already
double-delivered because the drain is ungated on both nodes.
`ShouldHistorize` itself is left unchanged — out of scope, and with D-1 in place the two gates
disagreeing is no longer harmful.
### D-2 — Gate denial must be observable
`PrimaryGatePolicy` defaults to **deny** when the role is unknown on a multi-driver cluster. That
is the correct safety posture, but it means a redundancy-snapshot identity mismatch — the
documented 03/S5 shape that `DriverHostActor.cs:1485-1491` already warns about once — would
silently stop alarm history on *both* nodes, degrading only as retry/eviction growth minutes later.
Task 3 will therefore: log the skip once per gate-state transition (not per tick — the drain runs
every 5 s), and surface it in `HistorianSinkStatus` so `/healthz` and the AdminUI can see
"buffering, not draining, because not Primary" rather than an unexplained rising queue depth.
Adding a `HistorianDrainState` member is additive; Task 3 will check every `switch` over it.
### D-3 — Tasks 2 and 3 will land as one commit
The plan anticipated this. Confirmed necessary: the gate delegate is a constructor parameter of
the rewritten sink, so the sink cannot compile in its final shape without it, and an intermediate
commit with a rewired-but-ungated sink would be a commit in which a replicated table is drained by
both nodes. Not a state worth having in history.
### D-5 — The table keeps the legacy delete-on-ack shape, not a `status` column
**The plan proposes** `status TEXT NOT NULL DEFAULT 'pending'` with values `pending | delivered |
dead`, and drops the legacy `LastError` column.
**Two problems.** A `delivered` status means acknowledged rows accumulate in the table forever —
the plan specifies no sweeper for them, and they would also have to be excluded from the capacity
count, changing a semantic §3 says to preserve. And `LastError` is not dead weight: `DeadLetterRow`
writes the failure reason into it and `RetryDeadLettered` clears it, so it is the only operator-
facing record of *why* a row was dead-lettered.
**The decision.** Keep the legacy shape: DELETE on acknowledgement, `dead_lettered` as the same 0/1
flag, and retain `last_error`. Deleting keeps the table bounded with no second sweeper, and the
replication engine carries the delete as a tombstone (pruned after `TombstoneRetention`, 7 d), so
the peer drops its copy too — which is the property the plan wanted the `delivered` status for.
Columns then map 1:1 onto the legacy table, making the Task-4 migrator a straight copy.
One genuine change: the drain's `ORDER BY` moves from `RowId ASC` to `enqueued_at_utc, id`, because
a hashed TEXT id carries no insertion order. Round-trip ("O") timestamps sort lexicographically in
chronological order, and `id` makes the ordering total; the index covers both.
### D-6 — The migrator keys on the payload hash, not `mig-{node}-{legacyId}`
**The plan says** deterministic ids of the form `mig-{NodeName}-{legacyId}`, mirroring ScadaBridge.
**Why that is not enough here.** Node-prefixing does solve the collision the legacy AUTOINCREMENT
key would otherwise cause — node A's `RowId 7` and node B's `RowId 7` are different alarms — but it
*preserves* a duplication that ought to be collapsed. A warm pair's two legacy files **overlap**:
`HistorianAdapterActor` default-writes while its redundancy role is unknown, so both nodes accepted
the same fanned transitions during every boot window they ever had. Prefixed ids would carry every
one of those duplicates into the merged buffer permanently, for the eventual Primary to deliver
twice.
**The decision.** Reuse D-1's `AlarmSfSchema.DeriveId` (lifted out of the sink so both call sites
share one identity rule). Distinct events cannot collide; identical ones converge. It also removes
the migrator's only need for node identity from configuration, and makes a crash between commit and
rename harmless under `INSERT OR IGNORE` — which the plan's own idempotency requirement wanted
anyway.
### D-7 — The migrator's tests live in `Host.IntegrationTests`
The plan names `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests`. **That project does not exist.** The
tests sit beside `LocalDbSetupTests` in `Host.IntegrationTests`, which is where every other
LocalDb-facing test already lives. They are offline — a temp file and no external services.
### D-4 — The live gate needs `MaxAttempts` raised on the rig
Not actionable until Task 8, recorded now so it is not rediscovered there. The rig has no
HistorianGateway, so every drain attempt fails. With `MaxAttempts: 10` and the backoff ladder
capping at 60 s, buffered rows dead-letter roughly five minutes into the outage — which would make
the gate's "dead letters 0" check (step 2) fail for reasons that have nothing to do with
replication. Task 6 should set a high `MaxAttempts` on the rig, or Task 8 should assert on
dead-letter *timing* instead.
Related: `Program.cs:170-173` shows an `AlarmHistorian:Enabled=true` /
`ServerHistorian:Enabled=false` deployment is explicitly supported and warned about — which is
exactly the rig's shape. Task 6 must confirm `ServerHistorianOptionsValidator` does not fail host
start for a disabled section with an empty `Endpoint`.
---
## Guard-deletion evidence (recorded at the Task 7 DoD sweep)
Every guard this phase adds was proved by deleting it and watching the tests go red. Two of those
runs found tests that would have passed over a broken implementation. Both are recorded here because
the *shape* of the vacuity is reusable, not just the fix.
### Control 1 — remove `IRedundancyRoleView.Publish` from `DriverHostActor`
**3 of 6** `DriverHostActorRoleViewTests` went red. The other three assert
`ShouldServiceAsPrimary == true`, which is exactly the value the view is **seeded** with — so they
cannot distinguish "published true" from "never published at all". That split is expected and
acceptable *given* the three that do go red pin the false cases, which only a real publish can
produce. Worth knowing before someone reads a future 3/6 result as a regression.
### Control 2 — remove `RegisterReplicated("alarm_sf_events")` from `LocalDbSetup.OnReady`
**3 of 4** `AlarmSfConvergenceTests` went red immediately. The fourth,
`TheSameEventOnBothNodes_ConvergesToOneRow`, **passed vacuously**: with replication switched off
each node trivially held its own single row, and "count == 1 on both nodes" was satisfied by two
completely unconverged databases. The assertion could not tell convergence from isolation.
**Fix:** enqueue a second, *distinct* event on node B and require **both** nodes to hold two rows.
Isolation now yields 1 and 2; only convergence yields 2 and 2. Re-run under the control: red.
Control restored: all 4 green.
The general trap — an assertion whose expected value is also what a *disabled* system produces —
is the same one recorded in the `#485`/`#486` unreadable-artifact defect class.
### Exact-set replicated-table pins
Two, not one, and both assert set equality (ordered `ShouldBe` against the literal three-table list,
so an added *or* dropped registration fails):
- `LocalDbSetupTests` — drives the production `OnReady` callback directly.
- `LocalDbWiringTests` — resolves `ILocalDb` from the full driver DI graph, which is what catches a
registration that never reaches a running host.
## Test-suite evidence (Task 7 DoD sweep)
Full solution run on the branch, compared against a **full solution run on a detached worktree at
the pre-branch baseline `2e46d054`**. Comparing failure *sets*, not counts, because the suite has a
standing set of environment- and load-dependent failures that a raw count would hide.
**The two failure sets are identical — all 13 tests, same names, both runs. Zero new failures.**
| Assembly | Baseline | Branch | Delta |
|---|---|---|---|
| `Core.AlarmHistorian.Tests` | 0 failed / 29 passed | 0 failed / **32** passed | **+3** (the drain-gate tests; 29 ported from the old sink's suite) |
| `Runtime.Tests` | 1 failed / 418 passed | 1 failed / **424** passed | **+6** (`DriverHostActorRoleViewTests`) |
| `Host.IntegrationTests` | 2 failed / 169 passed | 2 failed / **186** passed | **+17** (migrator + convergence + pin updates) |
The 13 standing failures, and why each is not this branch's:
- **3 × `DriverTypeNamesGuardTests`** — `ArgumentNullException (secretResolver)` out of
`GalaxyDriverFactoryExtensions.Register`, reached by reflection. A real pre-existing defect in a
guard test, reproduced exactly on the baseline worktree. Unrelated to this phase.
- **4 × `AbLegacy.IntegrationTests`**, **3 × `OpcUaClient.IntegrationTests`**,
**1 × `DriverProbeHandshakeE2eTests.AbCip_Green_AgainstSim`** — driver fixtures on the
`10.100.0.35` docker host are not running.
- **1 × `RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed`**
— times out under full-suite load; passes in isolation.
- **1 × `ContinuousHistorizationRecorderTests.Retry_after_writer_failure_eventually_acks`** — fails
under full-suite load ("writer must have been called at least twice"), and `Runtime.Tests` passes
**425/425** when run alone. Also fails on the baseline. Worth noting for anyone reading a future
run: this one is *newly observed* here only because the previous sweep's log was truncated, not
because the branch introduced it.
docs/plans/2026-07-21-issue-485-live-gate.md
+76
View File
@@ -0,0 +1,76 @@
# Issue #485 — live gate (docker-dev rig)
> Verifies the two fixes for #485 — `OpcUaPublishActor` (master `c7f5b9cf`) and `DriverHostActor`
> (master `5aad1e33`) — on the local `docker-dev` rig: 6 nodes, shared SQL, real Modbus fixture on
> `10.100.0.35:5020`.
>
> **Safety rules followed**: all DB inspection + mutation via `sqlcmd` inside the SQL container; the
> rig's Modbus fixture is read-only for this gate (no writes to the PLC sim).
## Result
**Both fixes verified, all three guards observed firing in production, with a live RED control that
was not staged.** The gate also confirmed a **pre-existing** defect the fixes do not address — the node
records `Applied` for a deployment it skipped (see "Finding" below).
| # | Check | Result | Evidence |
|---|---|---|---|
| 1 | RED control — the bug in its natural habitat, unforced | ✅ | site-a-1 hit #485 during the *previous* gate's SQL flapping and had been serving an **empty address space for 34 minutes**: `failed to load artifact … rebuild becomes no-op` followed immediately by `applied plan (kind=PureRemove, added=0, removed=17, rebuild=True)`. Its 17 `ns=2` nodes were gone; a browse returned only the root `OtOpcUa` folder. Nothing was staged to produce this — it was already true on the pre-fix image. |
| 2 | Rebuilt on the fixed image, the node recovers | ✅ | After `docker compose build` + `--force-recreate`, site-a-1 restored to `raw subtree materialised (containers=16, tags=1)` — 18 `ns=2` nodes. |
| 3 | Address-space guard fires; nodes survive | ✅ | site-a-1, empty artifact: `OpcUaPublish: no usable artifact for deployment f4b9dd63… — KEEPING the currently served address space rather than tearing it down`. **No `PureRemove`.** Browse before vs after: 18 vs 18, `diff` **identical**. |
| 4 | Reconcile guard fires; drivers survive | ✅ | `artifact for f4b9dd63… read back empty; skipping reconcile and keeping the 1 running driver(s)`, and the apply still reported `children=1`. Repeated on central-2 with a bigger blast radius: `keeping the 5 running driver(s)`, `children=5`. |
| 5 | SubscribeBulk guard fires; **live values keep flowing** | ✅ | central-2, `artifact for 8c36a673… read back empty; skipping SubscribeBulk and keeping the live subscriptions`. The decisive proof is the value, not the log: `ns=2;s=pymodbus/plc/HR200X` read **56278 @ 06:59:51** before the induction, then **56336 @ 07:00:21** and **56354 @ 07:00:30** after it (induction at 07:00:07). The subscription was still delivering fresh Modbus data across the event. |
| 6 | Positive control — a readable artifact still applies | ✅ | Immediately after the guards fired, a genuine deploy applied normally on both nodes: site-a-1 `AddRemoveMix, added=1, removed=1` + `SubscribeBulk pushed 1 references`; central-2 `AddRemoveMix, added=1, removed=1` + `SubscribeBulk pushed 5 references across 4 driver(s)`. The guards suppress the teardown, **not** deployment. |
| 7 | Control — only the node that saw empty bytes took the guard path | ✅ | Of the six nodes, exactly **one** logged `read back empty` (the paused one); the other five read the real artifact before it was blanked and applied it normally. The induction window is what produced the condition, not a global change. |
## How the failure was induced deterministically
The original incident was a race (a transient ConfigDb blip landing between the apply and the artifact
read). Racing it again would be unreliable, so the gate drives the **empty-bytes** branch instead — which
is byte-for-byte what the code sees when the row cannot be read, and is the branch the driver-side guards
key on (their *throw* paths already returned early before this work):
1. `docker pause` the target node, freezing it mid-cluster.
2. `POST /api/deployments` — the deployment seals and dispatches while the node cannot process it.
3. `UPDATE Deployment SET ArtifactBlob = 0x WHERE DeploymentId = <new>`.
4. `docker unpause` — the node now processes the dispatch and reads an empty artifact.
The pause measured **<1s** on both runs, well inside `acceptable-heartbeat-pause = 10s` and SBR's
`stable-after = 15s`, so the node was never flagged unreachable and no failover was triggered.
## Finding — the node records `Applied` for a deployment it skipped (pre-existing, NOT fixed here)
Both guards leave the node running its previous configuration, which is the intent. But `ApplyAndAck`
advances `_currentRevision = revision` and writes `NodeDeploymentState = Applied` **before** it knows
whether anything was applied — that assignment sits immediately after `ReconcileDrivers`, which returns
`null` on both the (pre-existing) throw path and the new empty path:
```
[07:00:07 WRN] DriverHost central-2:4053: artifact … read back empty; skipping reconcile …
[07:00:07 INF] DriverHost central-2:4053: applied deployment 8c36a673… (rev 03b33904…, children=5)
```
`NodeDeploymentState` for that deployment reads `Status = Applied` for central-2. The node now claims a
revision whose configuration it never applied — and because `HandleDispatchFromSteady` short-circuits on
a revision match (`dispatch matches current rev; immediate ACK`), **re-dispatching that same revision is
a no-op**, so the node cannot self-heal. Only a later deployment with a *different* revision recovers it
(check 6 confirms that path works).
This is pre-existing — the throw path has always done it — and is strictly a reporting/convergence bug,
not a data-plane one: the node keeps serving its last good configuration throughout. The honest fix is to
ACK `Failed` and leave `_currentRevision` alone when the artifact could not be read, so the normal deploy
retry heals it. Tracked separately rather than folded in here, because it changes deploy ACK semantics
fleet-wide.
## Addendum — issue #486 fix, live-gated (2026-07-21)
The finding above is now fixed (`DriverHostActor.ApplyAndAck` fails the apply when `ReconcileDrivers`
returns null) and gated on the same rig with the same induction.
| # | Check | Result | Evidence |
|---|---|---|---|
| 8 | Regression — a normal deploy still ACKs Applied fleet-wide | ✅ | The change alters ACK semantics, so this is the risk that matters. A genuine deploy recorded `Status = Applied` on **all six** nodes. |
| 9 | An unreadable artifact now ACKs Failed, with a reason | ✅ | central-2 recorded `Status = Failed` + `the deployment artifact could not be read (ConfigDb unreachable, or the row is missing/empty); nothing was applied`. The five nodes that read the real artifact recorded Applied — so the fleet view now distinguishes them. Pre-fix, all six claimed Applied. |
| 10 | The revision is NOT advanced | ✅ | `apply of a855dd9e… FAILED — … Staying on revision 5bc0d9f3…` — the **previous** revision, not the dispatched `583122b7…`. This is what stops `HandleDispatchFromSteady` from short-circuiting a retry of that revision. |
| 11 | The node keeps serving throughout (#485 still holds) | ✅ | The #485 guard fired first (`keeping the 5 running driver(s)`), and `ns=2;s=pymodbus/plc/HR200X` read **59603 @ 07:27:28**, live and current, after the Failed apply. Failing the apply reports the truth; it does not tear anything down. |
| 12 | The node recovers on the next real deploy | ✅ | The following deployment applied on central-2 (`children=5`) and recorded Applied on all six nodes. |
docs/plans/2026-07-21-localdb-followups-live-gate.md
+86
View File
@@ -0,0 +1,86 @@
# LocalDb Phase 1 follow-ups — live gate (docker-dev rig)
> Closes the two limitations left open by `2026-07-20-localdb-phase1-live-gate.md` (checks 4 and 8).
> Rig: local `docker-dev/docker-compose.yml`, 6 nodes rebuilt from `fix/localdb-phase1-followups`.
> site-a pair replicates; central + site-b stay default-OFF.
>
> **Safety rules followed** (same as the Phase 1 gate): all DB inspection via `docker cp` of the
> `db`/`-wal`/`-shm` triplet, querying the copy — never host `sqlite3` on the live WAL file.
## Result
**Both follow-ups closed, and the gate found a third defect** — one that was *unreachable* until the
first fix landed. Library `ZB.MOM.WW.LocalDb` went `0.1.1` → `0.1.2` → `0.1.3` in the course of this
gate; both consumers (OtOpcUa, ScadaBridge) are pinned to `0.1.3`.
| # | Check | Result | Evidence |
|---|---|---|---|
| 8 | Oplog stops growing on a default-OFF node | ✅ PASS | site-b-1 restarted twice with an unchanged cached artifact: `oplog_depth` **7 → 7 → 7**, `deployment_pointer.applied_at_utc` frozen at `05:34:58` across both — the re-cache was skipped outright. Repeated on the `0.1.3` build after a real deploy: **13 → 13**. Pre-fix this grew ~2 rows per restart forever, because a default-OFF node has no peer to ack and prune them. |
| 8b | …but a genuinely new deployment still writes (positive control) | ✅ PASS | A real config change (renamed tag `SITEA-tag`) produced rev `822bdf34`, then `002c16b3`. site-b-1's oplog **7 → 10 → 13** and its pointer advanced each time. The skip suppresses *identical* re-writes only; it is not a dead cache-write path. |
| 4 | A wiped node is back-filled by its peer, no new deploy, central down | ✅ PASS (was ⚠️ PARTIAL) | Precondition asserted first — site-a-1's oplog **0**, `needs_snapshot 0`: the exact state that used to make back-fill impossible. Then SQL stopped, site-a-2's LocalDb **volume destroyed**, container recreated. a-1 logged `Snapshot sent (as_of_seq 0, 4 rows)` — a line that could not appear on `0.1.1`. a-2 came back with pointer `SITE-A / 475df87a @ 03:09:12` (a-1's **original** timestamp, not a re-derived one), chunks 2, and a `__localdb_row_version` dump **byte-identical to a-1's**, carrying origin node ids `6812ca9c` and `935659a5` — the rebuilt node's own id appears nowhere, so the rows were replicated, not recreated. |
| 4b | …and the back-filled node then boots from cache | ✅ PASS | With SQL still down, site-a-2 logged `RUNNING FROM CACHE — … booted deployment 475df87a… cached at 2026-07-21T03:09:12`, `raw subtree materialised (containers=16, tags=1)`. OPC UA browse of both nodes: **17 ns=2 nodes each, diff-identical** — the rebuilt node serves exactly what its healthy peer does, from a configuration it never applied itself and could not have fetched. |
| 4c | The rebuilt node's OWN writes reach its peer | ✅ PASS **after a third fix** (see below) | On `0.1.2` this failed: a-1 held `last_applied_remote = 10` while the rebuilt a-2's oplog was seqs **1,2,3** and its `last_acked` stayed **0** — never accepted. On `0.1.3`, a-1 logs `Replication peer identity changed (ebb10854 -> c85d8b1d): the peer was rebuilt … Resetting the inbound watermark`, and after the next deploy a-1 shows `last_applied_remote = 3` / a-2 shows `last_acked = 3` with a drained oplog and identical row_versions. |
## The third defect (the reason this gate was worth running)
`0.1.2` made back-fill work, which for the first time let a rebuilt node be observed *writing*. It
turned out its writes went nowhere.
`last_applied_remote_seq` is a watermark in the **peer's** seq space, and a rebuilt peer is a new
database whose oplog numbers from 1. Two independent bugs stacked, and fixing either alone leaves the
data stuck:
1. The healthy node kept advertising the **old** peer's watermark. The *sending* side seeds its pump
directly from that number (`_sentThruSeq`), so the rebuilt node skipped its entire oplog.
→ the watermark (and the observed peer clock) is now reset when the peer's node id changes.
`last_acked_seq` is deliberately **not** reset: it describes our own oplog's pruned horizon and is
exactly what `0.1.2` keys the snapshot decision off — clearing it would switch back-fill back off.
2. A peer claiming to have applied more of our stream than we have ever produced can only be
remembering a previous incarnation of *us*. → that claim is now rejected in favour of pumping from
the start; LWW makes the re-send harmless. This mirrors the clamp `RecordPeerAckAsync` already
applies to acks.
Both were reproduced offline as a RED test before fixing
(`SnapshotResyncTests.RebuiltPeer_OwnWritesReachTheHealthyNode_DespiteTheOldPeersWatermark`).
**Why offline tests could not have found it first:** the wiped-peer scenario had no test because the
scenario was believed impossible to recover from — it was a *documented limitation*, not a bug. The
gate is what turned the limitation into a reproducible case, and the second defect only existed
behind the first.
## Unrelated finding (NOT a follow-up of this work — pre-existing; `lmxopcua` issue #485, since FIXED)
While SQL was being stopped and started to drive check 4, site-b-1 took a deploy whose artifact load
hit a transient ConfigDb error, and **emptied its served address space**:
```
[06:11:20 ERR] An error occurred using the connection to database 'OtOpcUa' on server 'sql,1433'.
[06:11:20 WRN] OpcUaPublish: failed to load artifact for deployment 43d54397…; rebuild becomes no-op
[06:11:20 INF] AddressSpaceApplier: applied plan (kind=PureRemove, added=0, removed=16, rebuild=True)
```
The log says "rebuild becomes no-op" and the applier then removed all 16 nodes — those two disagree.
The cache layer behaved correctly and refused to store the bad artifact ("not caching deployment … its
artifact was empty or could not be loaded, so it is not a usable last-known-good configuration"), and
the node recovered fully on restart (`containers=16`). But a transient ConfigDb blip emptying the
served address space instead of holding last-known-good is the same failure *class* as the Phase 1
gate's check-3 defect. Untouched by this work — site-b-1 has replication off entirely — and induced by
this gate's own SQL flapping.
**Fixed** (issue #485): `OpcUaPublishActor.HandleRebuild` now treats an artifact it could not obtain as
*no answer* rather than *an empty configuration*, and abandons the rebuild with the served address space
(and `_lastApplied`) intact. Nothing legitimate produces a zero-length blob — an operator who really
deletes everything still deploys a JSON document with empty arrays — so "no bytes" can only mean the read
failed. The log excerpt above is now impossible: the `PureRemove` never runs, and the loader's own
"the rebuild is abandoned" line is finally true. Covered by
`OpcUaPublishActorRebuildTests.Rebuild_keeps_the_last_known_good_address_space_when_the_artifact_load_fails`,
with a positive control proving a *readable* artifact that genuinely drops an equipment still tears its
subtree down.
The **driver-side half** of the same mistake was fixed alongside it: `DriverHostActor.ReconcileDrivers`
and `PushDesiredSubscriptionsFromArtifact` read the artifact the same way, so empty bytes meant zero
driver specs — `DriverSpawnPlanner` planned every running child for `StopChild` — and then an empty
desired set dropped each surviving driver's live subscription handle. A node lost its entire field I/O
and still ACKed Applied. Both now skip on empty. Covered by `DriverHostActorUnreadableArtifactTests`,
whose absence assertion is calibrated by a control that observes the very same unsubscribe *arriving*
inside the settle window (without it the assertion passed on a race).
docs/plans/2026-07-21-per-cluster-mesh-design.md
+283
View File
@@ -0,0 +1,283 @@
# Per-Cluster Akka Mesh — design
> **Status:** design for review. No implementation planned yet.
> **Decisions settled 2026-07-21** — see §6: one central DB with driver nodes disconnected from it,
> the two-node keep-oldest gap CLOSED upstream and still live here (§6.2 — most urgent item), and
> their auth posture matched for now.
## 1. What this changes
Today OtOpcUa runs **one Akka mesh containing every node of every application Cluster**. That was a
deliberate choice — `docs/plans/2026-06-07-per-cluster-scoping.md`, "Per-ClusterId Scoping
(hub-and-spoke **single mesh**)" — so the deploy channel could stay in-mesh: AdminUI →
`admin-operations` singleton → `deployments` topic → every node filters to its own `ClusterId`.
The proposal is to move to the sister project's shape: **one Akka mesh per application Cluster,
two nodes maximum**, with central and clusters joined by explicit transports instead of cluster
gossip.
The motivation is that redundancy becomes correct *by construction*. Every Primary-gated decision
(inbound device writes, native alarm acks, fleet alerts, ServiceLevel, and — since Phase 2 — the
alarm-history drain) currently asks "am I the driver Primary?" of a **cluster-wide** election, while
every resource being gated is **pair-local**. Splitting the mesh collapses the two scopes into one
and the question stops being ambiguous.
## 2. What ScadaBridge actually does
Researched directly from `/Users/dohertj2/Desktop/ScadaBridge`. Five things differ from the
summary in its own CLAUDE.md, and they matter:
**Three transports cross the boundary, not two.**
| Transport | Direction | Carries |
|---|---|---|
| Akka **ClusterClient** | central → site (+ replies) | command/control: deploy notify, lifecycle, queries, subscribe handshakes |
| **gRPC** server-streaming | **central dials into the site** | real-time attribute + alarm events |
| Plain **HTTP**, token-gated | site → central | the deployment config itself (notify-and-fetch) |
**The gRPC direction is inverted from the obvious guess.** Data flows site→central, but *each site
node hosts the gRPC server* (Kestrel h2c :8083) and *central is the client*. There is no gRPC server
on central at all.
**Active node = OLDEST Up member, explicitly not the cluster leader.**
`ActiveNodeEvaluator.SelfIsOldestUp` is documented as *"THE single definition of 'active node'"*:
> *"Cluster LEADERSHIP (lowest address) is an Akka-internal concept that diverges from singleton
> placement permanently once the original first node restarts and rejoins; every product-level
> active/standby decision must use this evaluator, never `cluster.State.Leader`."*
The equivalence **oldest-Up == where `ClusterSingletonManager` places singletons** *is* the design.
**All clusters share one ActorSystem name** (`"scadabridge"`, hardcoded). They are separate clusters
only by seed-node partitioning — required, because Akka.Remote address matching means ClusterClient
could not reach a differently-named system.
**Site nodes carry two roles:** `"Site"` and `"site-{SiteId}"`. Singletons scope to the
**site-specific** role.
Other load-bearing details:
- **Central discovers sites from the database** (a `Site` entity with `NodeAAddress`/`NodeBAddress`
+ `GrpcNodeAAddress`/`GrpcNodeBAddress`), refreshed every 60 s and on admin change. Sites discover
central from **appsettings** — static, restart required. The asymmetry is deliberate.
- **Exactly one actor is exposed per side** via `ClusterClientReceptionist.RegisterService` —
`/user/central-communication` and `/user/site-communication` — registered **per node, not as a
singleton**, so contact rotation reaches whichever node answers.
- **No central buffering when a site is unreachable.** The send is dropped with a warning and the
caller's Ask times out. *"It keeps the central coordinator stateless with respect to site
availability."* A `ConnectionStateChanged` mechanism was built for this and **deleted as dead
code**.
- **Every unhandled message replies with a typed failure rather than dropping**, so a central Ask
can never stall on a missing handler.
- **Sender preservation is the whole Ask idiom**: `_client.Tell(new ClusterClient.Send(path, msg),
Sender)` routes the reply straight back to the waiting Ask, bypassing the comm actor.
### The registered outage gap — read before copying
> `Component-ClusterInfrastructure.md:113`: *"Only the FIRST seed listed in `Cluster:SeedNodes` may
> self-join to form a *new* cluster… a lone restarted non-first-seed node (with the first seed still
> down) loops on `InitJoin` forever — never `Up`, never routable. This is why the two-node
> keep-oldest **oldest/active-node crash is a total-outage gap**: after the oldest dies the younger
> survivor self-downs, and it cannot re-bootstrap alone. Recovery is operator-driven."*
Their failover drill has a mode that exists *"to make the registered gap observable — not to pretend
it is covered."* **A two-node keep-oldest mesh has an acknowledged total-outage hole.** If OtOpcUa
adopts 2-node meshes, this must be decided deliberately, not inherited.
### Both inter-cluster transports are unauthenticated
Akka remoting has no TLS, no secure cookie, no `trusted-selection-paths`. The gRPC listener is h2c
with no auth covering `SiteStreamService` — including two Pull RPCs that return audit rows. The only
authenticated pieces are the HTTP fetch token and `LocalDbSyncAuthInterceptor`, which gates *only*
`/localdb_sync.v1.LocalDbSync/`. Their boundary assumes a trusted network.
**OtOpcUa already ships that same fail-closed interceptor** (LocalDb Phase 1) — it is a ready-made
template if we choose to authenticate more than they did.
## 3. What OtOpcUa has today
**Nine cross-node DPS topics plus a singleton**, all currently relying on one mesh:
| Channel | Direction | Purpose |
|---|---|---|
| `deployments` / `deployment-acks` | central ↔ nodes | deploy dispatch + acks |
| `driver-control` | central → nodes | AdminUI Reconnect/Restart |
| `redundancy-state` | central → nodes | roles + peer probe results |
| `alerts` | nodes → central | fleet alarms → `/alerts` |
| `driver-health`, `driver-resilience-status`, `fleet-status`, `script-logs` | nodes → central | AdminUI live panels |
| `admin-operations` singleton | central | operator ack/shelve into engines |
Three facts make the split cheaper than that table suggests:
1. **The deploy path is already notify-and-fetch.** `DispatchDeployment` carries only
`DeploymentId` + `RevisionHash` + `CorrelationId`; the node fetches the artifact itself. We
independently arrived at the pattern ScadaBridge had to retrofit, and we do **not** carry its
128 KB Akka frame exposure on this path.
2. **Deploy state already has a DB substrate.** `ConfigPublishCoordinator` persists per-node ACKs to
`NodeDeploymentState` so a singleton failover recovers in-flight state from the DB.
3. **`ClusterNode` rows already enumerate every node per application Cluster** — the same table
`AdminOperationsActor` groups by cluster. This replaces the coordinator's one genuinely
mesh-bound dependency: it derives its expected-ack set from `Akka.Cluster.State.Members` filtered
by role.
So the deploy channel needs only a small notify + ack transport. **The nine live-telemetry channels
are the real work**, and they are exactly what ScadaBridge built ClusterClient + gRPC for.
## 4. A defect to fix regardless of this design
OtOpcUa currently uses **two different node-selection rules at once**:
- Split-brain resolution is **keep-oldest** (`Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:49`).
- `ClusterSingletonManager` places singletons on the **oldest** member.
- But `RedundancyStateActor.BuildSnapshot` elects Primary from **`RoleLeader("driver")`** — lowest
address among driver members.
Those select the same node only by coincidence. After a restart-and-rejoin the role leader and the
oldest member diverge permanently, which means the node the SBR protects and hosts every singleton
on need not be the node the data-plane gates consider Primary. This is precisely the rule ScadaBridge
forbids by name, and for the reason it gives: *"both sides claim leadership during a partition"* —
the dual-primary shape archreview 03/S4 exists to prevent.
**Switching the role derivation to oldest-Up-with-role is small, is correct under either topology,
and should not wait for this design.** It also happens to be exactly what the separate-mesh model
needs, so it is not throwaway work.
## 5. Target architecture
Mirror ScadaBridge's shape, adapted to OtOpcUa's existing substrate:
- **One Akka mesh per application `Cluster`, two nodes max.** Same ActorSystem name (`otopcua`)
everywhere; separation by seed-node partitioning only.
- **Roles per node:** `driver` plus a cluster-specific `cluster-{ClusterId}`, with singletons scoped
to the cluster-specific role.
- **Active node = oldest Up member with role** (§4), one definition, used by the data-plane gates,
ServiceLevel, and the alarm drain alike.
- **Central ↔ cluster command/control over ClusterClient**, with exactly one receptionist-registered
actor per side and central discovering cluster node addresses from `ClusterNode` rows (extended
with Akka + gRPC addresses, mirroring their `Site` entity).
- **Deploy stays notify-and-fetch, but the fetch changes source.** The notify crosses via
ClusterClient; the artifact then comes **from central**, not from the ConfigDb the driver node no
longer connects to (§6.1), and is cached in LocalDb. The coordinator's expected-ack set comes from
`ClusterNode` rows instead of cluster membership.
- **Driver nodes hold no ConfigDb connection.** LocalDb becomes their steady-state configuration
store; five current DB consumers are re-homed or re-sourced per §6.1.
- **Live telemetry over gRPC**, central dialling into each cluster node, replacing the seven
observability topics with one stream contract carrying a `oneof` event — additive-only field
evolution, contract-locked by test.
**Copied:** their node-local configuration store fed by fetch-and-cache (§6.1), their oldest-Up
active-node rule (§4), their two-node keep-oldest posture including its acknowledged outage gap
(§6.2), and their unauthenticated inter-cluster transports (§6.3).
**Deliberately not copied:** their *transient-only* central alarm cache. OtOpcUa persists alarm
history centrally through the historian, and Phase 2's replicated store-and-forward buffer already
guarantees delivery across a failover — so there is no reason to adopt a design whose stated
trade-off is that a new active node re-seeds alarm state from scratch.
## 6. Decisions (settled 2026-07-21)
### 6.1 DECIDED — one central DB, and driver nodes never connect to it
**Verified in ScadaBridge before adopting.** `Program.cs` calls
`AddConfigurationDatabase(configDbConnectionString)` at line 264 — inside the Central branch
(89–478). The Site branch begins at line 479. **Site nodes never register the central configuration
database at all**; they receive config from central (notify → token-gated HTTP fetch) and persist it
locally. There is still exactly one authoritative configuration database — it is simply
central-only.
OtOpcUa adopts the same shape: **the ConfigDb stays the single source of truth, and driver nodes stop
connecting to it.** They obtain their configuration from the central nodes and cache it locally in
LocalDb.
**This promotes LocalDb from an outage cache to the driver node's steady-state configuration store.**
That is a reframing of the Phase 1 work rather than a contradiction of it: boot-from-cache stops
being the exceptional path and becomes the normal one, and "central SQL unreachable" stops being a
degraded mode for driver nodes because they never held a connection to lose. The chunking,
SHA-256 verification, newest-2 retention and pair replication all carry over unchanged.
**Scope — five driver-side ConfigDb consumers must be re-homed or re-sourced.** Audited:
| Consumer | Current use | Disposition |
|---|---|---|
| `DriverHostActor` | artifact fetch + ack writes (5 `CreateDbContext` sites) | artifact via fetch-and-cache; acks via the ack transport |
| `EfAlarmConditionStateStore` | Part 9 alarm condition state | **move to LocalDb** — it is pair-local state, the same journey Phase 2 made for the alarm S&F buffer |
| `DbHealthProbeActor` | DB reachability feeding ServiceLevel tiering | **semantics change** — a driver node has no DB to probe; decide what health input replaces it |
| `OpcUaPublishActor` | (audit required) | TBD |
| `ServiceCollectionExtensions` | registration | follows the above |
`DbHealthProbeActor` deserves particular care: DB health is currently a ServiceLevel input, and
under this model it stops existing for driver nodes. That is a behaviour change on a client-visible
value, not just an internal refactor.
### 6.2 CORRECTED — the gap was closed in ScadaBridge, and OtOpcUa still has it
**An earlier revision of this section was wrong.** It reported the two-node keep-oldest outage gap as
still open, citing `Component-ClusterInfrastructure.md:113` and Gitea PR
[#12](https://gitea.dohertylan.com/dohertj2/ScadaBridge/pulls/12) (which made the failover drill
*honest* about the gap rather than closing it). That doc line was stale: ScadaBridge closed the gap
in `cf3bd52f`, *"feat(cluster): auto-down downing strategy — either-node crash now fails over (owner
decision 2026-07-21: availability over partition-safety)"*.
Its commit message carries the analysis, live-proven on their rig:
> *"Two-node keep-oldest could NEVER survive a crash of the oldest/active node: Akka.NET 1.5.62
> `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side with >= 2 members, so the 1-vs-1
> survivor takes `DownReachable` and downs ITSELF — proven live on the rig ('SBR took decision …
> including myself') before this change. `static-quorum(1)` is worse (`IsTooManyMembers` -> `DownAll`);
> `keep-majority` just re-keys the fatal crash to the lowest address."*
**OtOpcUa is running that exact configuration today** —
`Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:49-53`: `active-strategy = "keep-oldest"`,
`keep-oldest { down-if-alone = on }`, `stable-after = 15s`, with
`BuildClusterOptions` setting the matching typed `KeepOldestOption { DownIfAlone = true }`.
So a two-node OtOpcUa pair **cannot fail over when the oldest node crashes**: the survivor downs
itself. That is a total-outage defect in the redundancy feature itself, it is independent of this
design, and the sister project has already done the analysis and shipped the fix. **It should be
Phase 0 alongside the oldest-Up role derivation**, and it needs its own live gate — a
crash-the-oldest drill, since the failure only appears in the 1-vs-1 case.
### 6.3 DECIDED — match ScadaBridge's auth posture for now
Inter-cluster transports stay unauthenticated and unencrypted, as theirs are: the boundary assumes a
trusted network. Recorded as an accepted risk rather than an oversight, with two notes for whenever
it is revisited: `LocalDbSyncAuthInterceptor` is already in our tree as a fail-closed template, and
the surface most worth gating first is any Pull-style RPC that returns historical rows.
## 7. Sequencing sketch
Deliberately not a task plan — per-phase plans follow, one at a time.
| Phase | Content | Independent of the split? |
|---|---|---|
| 0a | Downing strategy: keep-oldest cannot survive an oldest-node crash in a 2-node cluster (§6.2) | **Yes** — most urgent |
| 0b | Oldest-Up role derivation (§4) | **Yes** |
| 1 | `ClusterNode` gains Akka + gRPC address columns; coordinator sources its expected-ack set from the DB | Yes |
| 2 | Comm actors + receptionist registration; ClusterClient transport; deploy notify + acks across the boundary | No |
| 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No |
| 4 | Cut the driver-side ConfigDb connection: re-home `EfAlarmConditionStateStore`, resolve the `DbHealthProbeActor` ServiceLevel input, audit `OpcUaPublishActor` | No |
| 5 | gRPC stream contract; migrate the seven observability topics | No |
| 6 | Mesh partition: per-cluster seed nodes, cluster-scoped roles + singletons; rig models the real topology | No |
| 7 | Failover drill (both directions, per §6.2); live gate | No |
Phases 3 and 4 are the ones that change a running system's data path rather than its wiring, and each
deserves its own live gate.
## 8. Risks
- **LocalDb becomes load-bearing for configuration, not just resilient.** Phase 1 built it as a
fallback; §6.1 makes it the driver node's only config store. A cache defect that was previously a
degraded-mode bug becomes a total-availability bug — the #485 unreadable-artifact class, with the
blast radius raised.
- **`DbHealthProbeActor` feeds a client-visible value.** Removing the driver-side DB connection
changes what ServiceLevel means on those nodes; that is an interop-visible change, not internal.
- **The rig currently models the topology that would be abolished.** Phase 6 rewrites
`docker-dev/docker-compose.yml` substantially, and every live gate that depends on it.
- **Losing gossip loses free fleet-wide observability.** Seven AdminUI panels are fed by DPS today;
each needs an explicit stream and a reconnect story.
- **Akka frame size.** We are clean on the deploy path, but anything new that carries payload over
ClusterClient inherits the 128 KB default with `log-frame-size-exceeding` off — a silent
single-message drop that leaves the association healthy. If we send payload at all, set both.
- **Do not stack application-level last-write-wins on LocalDb's HLC.** Their `SiteStorageService`
carries this warning in code. Verified: our `deployment_pointer` upsert is unconditional and the
alarm sink is idempotent on a payload hash, so we are currently clean — keep it that way.
docs/security.md
+49
View File
@@ -380,6 +380,55 @@ polling the node.
---
## Config secrets (`${secret:}` delivery)
OtOpcUa never commits secret values to `appsettings*.json`. Every real secret is
supplied at deploy time — either as a plain environment variable, or as a
`${secret:<name>}` token backed by the shared **`ZB.MOM.WW.Secrets`** encrypted store.
A pre-host expander (`SecretReferenceExpander.ExpandConfigurationAsync`, wired in
`OtOpcUa.Host/Program.cs`) walks the assembled configuration and rewrites every
`${secret:<name>}` token into its resolved plaintext **before** the host is built —
so every downstream binder/validator (`AddZbSerilog`, `AddOtOpcUaConfigDb`, the first
`ValidateOnStart`) sees resolved values. The store's key-encryption key (KEK) comes from
the `ZB_SECRETS_MASTER_KEY` environment variable (`Secrets:MasterKey:Source=Environment`);
the encrypted SQLite store lives at `Secrets:SqlitePath`.
The expander is **fail-closed and section-agnostic**: a `${secret:<name>}` token whose
secret is absent throws `SecretNotFoundException` at startup, regardless of which feature
owns the key or whether that feature is enabled. Keys prefixed with `_` (comment keys) are
skipped, so a `${secret:...}` example inside a `_…Comment` value is never resolved.
The five config secrets and their canonical secret names:
| Config key | Owner | Secret name |
|---|---|---|
| `Security:Jwt:SigningKey` | `JwtOptions` | `otopcua/jwt/signing-key` |
| `Security:Ldap:ServiceAccountPassword` | `LdapOptions` | `otopcua/ldap/service-account-password` |
| `Security:DeployApiKey` | `DeployApiEndpoints` | `otopcua/deploy/api-key` |
| `ConnectionStrings:ConfigDb` | `AddOtOpcUaConfigDb` | `otopcua/sql/configdb-connstr` |
| `ServerHistorian:ApiKey` | `ServerHistorianOptions` | `otopcua/historian/api-key` |
**Delivery.** By default these are delivered as plain environment variables (e.g.
`ServerHistorian__ApiKey=histgw_…`), never committed. To deliver one from the encrypted
store instead, seed it once with the `secret` CLI (from `ZB.MOM.WW.Secrets`), then supply
the token in place of the literal — via env var or a deployment appsettings overlay:
```
secret set otopcua/historian/api-key <value> # seed the encrypted store first
ServerHistorian__ApiKey='${secret:otopcua/historian/api-key}'
```
Only switch a value to a `${secret:}` token **after** the secret is seeded — an unseeded
token fails the boot fail-closed. Do not commit the KEK (`ZB_SECRETS_MASTER_KEY`) or any
secret value.
For the production posture needed so every clustered node (admin, driver, and any
fused role) resolves the same secrets from the same store, see
[`docs/operations/2026-07-16-secrets-clustered-master-key.md`](operations/2026-07-16-secrets-clustered-master-key.md).
---
## Troubleshooting
### Certificate trust failure
src/Core/ZB.MOM.WW.OtOpcUa.Commons/Interfaces/NodeDiagnosticsSnapshot.cs
+9 -1
View File
@@ -14,8 +14,16 @@ public sealed record DriverInstanceDiagnostics(
/// Per-node diagnostics returned by <c>IFleetDiagnosticsClient</c>. Populated by the node's
/// local <c>DriverHostActor</c> via a request/response over Akka.
/// </summary>
/// <param name="RunningFromCache">
/// True when the node booted its configuration from the node-local artifact cache because the
/// central ConfigDb was unreachable. Such a node looks entirely healthy — full address space,
/// live values — but its configuration is frozen and no deployment can reach it, so the state
/// needs to be explicitly visible rather than inferred. Defaults to false, which is also what
/// every node that booted normally reports.
/// </param>
public sealed record NodeDiagnosticsSnapshot(
NodeId NodeId,
RevisionHash? CurrentRevision,
IReadOnlyList<DriverInstanceDiagnostics> Drivers,
DateTime AsOfUtc);
DateTime AsOfUtc,
bool RunningFromCache = false);
src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/AlarmConditionSnapshot.cs
+8 -1
View File
@@ -16,6 +16,12 @@ namespace ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
/// <param name="Shelving">The shelving mode (ShelvingState): unshelved, one-shot, or timed.</param>
/// <param name="Severity">OPC UA severity on the 1..1000 scale (the SDK <c>SetSeverity</c> input).</param>
/// <param name="Message">The human-readable condition message (LocalizedText payload).</param>
/// <param name="Quality">
/// Quality of the condition's source data (Part 9 <c>ConditionType.Quality</c>). Carried so a
/// comms-lost native source can report a non-<c>Good</c> condition instead of the accidentally-Good
/// default (issue #477). It is a pure annotation — it never alters Active/Acked/Retain. Defaults to
/// <see cref="OpcUaQuality.Good"/> so scripted-alarm callers (which stay Good in v1) need not supply it.
/// </param>
public sealed record AlarmConditionSnapshot(
bool Active,
bool Acknowledged,
@@ -23,7 +29,8 @@ public sealed record AlarmConditionSnapshot(
bool Enabled,
AlarmShelvingKind Shelving,
ushort Severity,
string Message);
string Message,
OpcUaQuality Quality = OpcUaQuality.Good);
/// <summary>
/// Commons-local mirror of the Core <c>ShelvingKind</c> enum so this assembly carries no
src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/DeferredAddressSpaceSink.cs
+3
View File
@@ -30,6 +30,9 @@ public sealed class DeferredAddressSpaceSink : IOpcUaAddressSpaceSink, ISurgical
public void WriteAlarmCondition(string alarmNodeId, AlarmConditionSnapshot state, DateTime sourceTimestampUtc, AddressSpaceRealm realm)
=> _inner.WriteAlarmCondition(alarmNodeId, state, sourceTimestampUtc, realm);
public void WriteAlarmQuality(string alarmNodeId, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm)
=> _inner.WriteAlarmQuality(alarmNodeId, quality, sourceTimestampUtc, realm);
/// <inheritdoc />
public void MaterialiseAlarmCondition(string alarmNodeId, string equipmentNodeId, string displayName, string alarmType, int severity, AddressSpaceRealm realm, bool isNative = false)
=> _inner.MaterialiseAlarmCondition(alarmNodeId, equipmentNodeId, displayName, alarmType, severity, realm, isNative);
src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/IOpcUaAddressSpaceSink.cs
+12
View File
@@ -33,6 +33,17 @@ public interface IOpcUaAddressSpaceSink
/// the condition was materialised under).</param>
void WriteAlarmCondition(string alarmNodeId, AlarmConditionSnapshot state, DateTime sourceTimestampUtc, AddressSpaceRealm realm);
/// <summary>#477 — annotate a materialised condition's source-data quality OUT OF BAND from any alarm
/// transition (used by the driver-connectivity path: comms lost → <see cref="OpcUaQuality.Bad"/>,
/// restored → <see cref="OpcUaQuality.Good"/>). Sets ONLY the condition's Quality — never
/// Active/Acked/Severity/Retain (a comms-lost active alarm stays active) — and fires one Part 9 event
/// only on a quality-bucket change. A no-op for an unmaterialised / non-condition node.</summary>
/// <param name="alarmNodeId">The condition node id (RawPath for a native alarm).</param>
/// <param name="quality">The source-data quality to annotate.</param>
/// <param name="sourceTimestampUtc">The connectivity transition timestamp in UTC.</param>
/// <param name="realm">The namespace realm the condition was materialised under.</param>
void WriteAlarmQuality(string alarmNodeId, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm);
/// <summary>
/// Materialise a real OPC UA Part 9 alarm-condition node under its equipment folder so clients
/// can browse it as a proper condition (with basic Active/Ack state). The node id equals the
@@ -165,6 +176,7 @@ public sealed class NullOpcUaAddressSpaceSink : IOpcUaAddressSpaceSink
/// <inheritdoc />
public void WriteAlarmCondition(string alarmNodeId, AlarmConditionSnapshot state, DateTime sourceTimestampUtc, AddressSpaceRealm realm) { }
public void WriteAlarmQuality(string alarmNodeId, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm) { }
/// <inheritdoc />
public void MaterialiseAlarmCondition(string alarmNodeId, string equipmentNodeId, string displayName, string alarmType, int severity, AddressSpaceRealm realm, bool isNative = false) { }
src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/GenerationSealedCache.cs
-197
View File
@@ -1,197 +0,0 @@
using LiteDB;
namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
/// <summary>
/// Generation-sealed LiteDB cache per <c>docs/v2/plan.md</c> and Phase 6.1
/// Stream D.1. Each published generation writes one <b>read-only</b> LiteDB file under
/// <c>&lt;cache-root&gt;/&lt;clusterId&gt;/&lt;generationId&gt;.db</c>. A per-cluster
/// <c>CURRENT</c> text file holds the currently-active generation id; it is updated
/// atomically (temp file + <see cref="File.Replace(string, string, string?)"/>) only after
/// the sealed file is fully written.
/// </summary>
/// <remarks>
/// <para>Mixed-generation reads are impossible: any read opens the single file pointed to
/// by <c>CURRENT</c>, which is a coherent snapshot. Corruption of the CURRENT file or the
/// sealed file surfaces as <see cref="GenerationCacheUnavailableException"/> — the reader
/// fails closed rather than silently falling back to an older generation. Recovery path
/// is to re-fetch from the central DB (and the Phase 6.1 Stream C <c>UsingStaleConfig</c>
/// flag goes true until that succeeds).</para>
///
/// <para>This cache is the read-path fallback when the central DB is unreachable. The
/// write path (draft edits, publish) bypasses the cache and fails hard on DB outage per
/// Stream D.2 — inconsistent writes are worse than a temporary inability to edit.</para>
/// </remarks>
public sealed class GenerationSealedCache
{
private const string CollectionName = "generation";
private const string CurrentPointerFileName = "CURRENT";
private readonly string _cacheRoot;
// Private per-database BsonMapper with the entity pre-registered. LiteDB's default
// BsonMapper.Global is a process-wide singleton whose lazy per-type member registration is
// not thread-safe across concurrently-constructed LiteDatabase instances; a seal racing a
// read (or this cache racing LiteDbConfigCache) corrupts the global mapper, surfacing as
// "Member … not found on BsonMapper" or a bogus duplicate-_id insert.
private static BsonMapper BuildMapper()
{
var mapper = new BsonMapper();
mapper.Entity<GenerationSnapshot>();
return mapper;
}
/// <summary>Root directory for all clusters' sealed caches.</summary>
public string CacheRoot => _cacheRoot;
/// <summary>Initializes a new instance of the GenerationSealedCache class.</summary>
/// <param name="cacheRoot">The root directory for the cache.</param>
public GenerationSealedCache(string cacheRoot)
{
ArgumentException.ThrowIfNullOrWhiteSpace(cacheRoot);
_cacheRoot = cacheRoot;
Directory.CreateDirectory(_cacheRoot);
}
/// <summary>
/// Seal a generation: write the snapshot to <c>&lt;cluster&gt;/&lt;generationId&gt;.db</c>,
/// mark the file read-only, then atomically publish the <c>CURRENT</c> pointer. Existing
/// sealed files for prior generations are preserved (prune separately).
/// </summary>
/// <param name="snapshot">The generation snapshot to seal.</param>
/// <param name="ct">The cancellation token.</param>
/// <returns>A task representing the asynchronous operation.</returns>
public async Task SealAsync(GenerationSnapshot snapshot, CancellationToken ct = default)
{
ArgumentNullException.ThrowIfNull(snapshot);
ct.ThrowIfCancellationRequested();
var clusterDir = Path.Combine(_cacheRoot, snapshot.ClusterId);
Directory.CreateDirectory(clusterDir);
var sealedPath = Path.Combine(clusterDir, $"{snapshot.GenerationId}.db");
if (File.Exists(sealedPath))
{
// Already sealed — idempotent. Treat as no-op + update pointer in case an earlier
// seal succeeded but the pointer update failed (crash recovery).
WritePointerAtomically(clusterDir, snapshot.GenerationId);
return;
}
var tmpPath = sealedPath + ".tmp";
try
{
using (var db = new LiteDatabase(new ConnectionString { Filename = tmpPath, Upgrade = false }, BuildMapper()))
{
var col = db.GetCollection<GenerationSnapshot>(CollectionName);
col.Insert(snapshot);
}
File.Move(tmpPath, sealedPath);
File.SetAttributes(sealedPath, File.GetAttributes(sealedPath) | FileAttributes.ReadOnly);
WritePointerAtomically(clusterDir, snapshot.GenerationId);
}
catch
{
try { if (File.Exists(tmpPath)) File.Delete(tmpPath); } catch { /* best-effort */ }
throw;
}
await Task.CompletedTask;
}
/// <summary>
/// Read the current sealed snapshot for <paramref name="clusterId"/>. Throws
/// <see cref="GenerationCacheUnavailableException"/> when the pointer is missing
/// (first-boot-no-snapshot case) or when the sealed file is corrupt. Never silently
/// falls back to a prior generation.
/// </summary>
/// <param name="clusterId">The cluster ID to read the snapshot for.</param>
/// <param name="ct">The cancellation token.</param>
/// <returns>A task representing the asynchronous operation containing the generation snapshot.</returns>
public Task<GenerationSnapshot> ReadCurrentAsync(string clusterId, CancellationToken ct = default)
{
ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
ct.ThrowIfCancellationRequested();
var clusterDir = Path.Combine(_cacheRoot, clusterId);
var pointerPath = Path.Combine(clusterDir, CurrentPointerFileName);
if (!File.Exists(pointerPath))
throw new GenerationCacheUnavailableException(
$"No sealed generation for cluster '{clusterId}' at '{clusterDir}'. First-boot case: the central DB must be reachable at least once before cache fallback is possible.");
long generationId;
try
{
var text = File.ReadAllText(pointerPath).Trim();
generationId = long.Parse(text, System.Globalization.CultureInfo.InvariantCulture);
}
catch (Exception ex)
{
throw new GenerationCacheUnavailableException(
$"CURRENT pointer at '{pointerPath}' is corrupt or unreadable.", ex);
}
var sealedPath = Path.Combine(clusterDir, $"{generationId}.db");
if (!File.Exists(sealedPath))
throw new GenerationCacheUnavailableException(
$"CURRENT points at generation {generationId} but '{sealedPath}' is missing — fails closed rather than serving an older generation.");
try
{
using var db = new LiteDatabase(new ConnectionString { Filename = sealedPath, ReadOnly = true }, BuildMapper());
var col = db.GetCollection<GenerationSnapshot>(CollectionName);
var snapshot = col.FindAll().FirstOrDefault()
?? throw new GenerationCacheUnavailableException(
$"Sealed file '{sealedPath}' contains no snapshot row — file is corrupt.");
return Task.FromResult(snapshot);
}
catch (GenerationCacheUnavailableException) { throw; }
catch (Exception ex) when (ex is LiteException or InvalidDataException or IOException
or NotSupportedException or FormatException)
{
throw new GenerationCacheUnavailableException(
$"Sealed file '{sealedPath}' is corrupt or unreadable — fails closed rather than falling back to an older generation.", ex);
}
}
/// <summary>Return the generation id the <c>CURRENT</c> pointer points at, or null if no pointer exists.</summary>
/// <param name="clusterId">The cluster ID to get the current generation ID for.</param>
/// <returns>The generation ID, or null if no pointer exists.</returns>
public long? TryGetCurrentGenerationId(string clusterId)
{
ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
var pointerPath = Path.Combine(_cacheRoot, clusterId, CurrentPointerFileName);
if (!File.Exists(pointerPath)) return null;
try
{
return long.Parse(File.ReadAllText(pointerPath).Trim(), System.Globalization.CultureInfo.InvariantCulture);
}
catch
{
return null;
}
}
private static void WritePointerAtomically(string clusterDir, long generationId)
{
var pointerPath = Path.Combine(clusterDir, CurrentPointerFileName);
var tmpPath = pointerPath + ".tmp";
File.WriteAllText(tmpPath, generationId.ToString(System.Globalization.CultureInfo.InvariantCulture));
if (File.Exists(pointerPath))
File.Replace(tmpPath, pointerPath, destinationBackupFileName: null);
else
File.Move(tmpPath, pointerPath);
}
}
/// <summary>Sealed cache is unreachable — caller must fail closed.</summary>
public sealed class GenerationCacheUnavailableException : Exception
{
/// <summary>Initializes a new instance of the GenerationCacheUnavailableException class.</summary>
/// <param name="message">The error message.</param>
public GenerationCacheUnavailableException(string message) : base(message) { }
/// <summary>Initializes a new instance of the GenerationCacheUnavailableException class with an inner exception.</summary>
/// <param name="message">The error message.</param>
/// <param name="inner">The inner exception.</param>
public GenerationCacheUnavailableException(string message, Exception inner) : base(message, inner) { }
}
src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/GenerationSnapshot.cs
-20
View File
@@ -1,20 +0,0 @@
namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
/// <summary>
/// A self-contained snapshot of one generation — enough to rebuild the address space on a node
/// that has lost DB connectivity. The payload is the JSON-serialized <c>sp_GetGenerationContent</c>
/// result; the local cache doesn't inspect the shape, it just round-trips bytes.
/// </summary>
public sealed class GenerationSnapshot
{
/// <summary>Gets or sets the auto-generated LiteDB ID.</summary>
public int Id { get; set; } // LiteDB auto-ID
/// <summary>Gets or sets the cluster identifier.</summary>
public required string ClusterId { get; set; }
/// <summary>Gets or sets the generation identifier.</summary>
public required long GenerationId { get; set; }
/// <summary>Gets or sets the time this snapshot was cached.</summary>
public required DateTime CachedAt { get; set; }
/// <summary>Gets or sets the JSON-serialized payload content.</summary>
public required string PayloadJson { get; set; }
}
src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/ILocalConfigCache.cs
-32
View File
@@ -1,32 +0,0 @@
namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
/// <summary>
/// Per-node local cache of the most-recently-applied generation(s). Used to bootstrap the
/// address space when the central DB is unreachable (degraded-but-running).
/// </summary>
/// <remarks>
/// <para><b>Concurrency contract:</b> implementations must serialize writes — specifically,
/// <see cref="PutAsync"/> for the same <c>(ClusterId, GenerationId)</c> from concurrent
/// callers must not produce duplicate rows. Reads may run concurrently with reads and writes.
/// The <see cref="LiteDbConfigCache"/> implementation enforces this via an instance-level
/// <see cref="SemaphoreSlim"/> around the find-then-insert/update window.</para>
/// </remarks>
public interface ILocalConfigCache
{
/// <summary>Retrieves the most recent generation snapshot for the specified cluster.</summary>
/// <param name="clusterId">The cluster identifier.</param>
/// <param name="ct">The cancellation token.</param>
/// <returns>The most recent generation snapshot, or null if none exists.</returns>
Task<GenerationSnapshot?> GetMostRecentAsync(string clusterId, CancellationToken ct = default);
/// <summary>Stores a generation snapshot in the local cache.</summary>
/// <param name="snapshot">The generation snapshot to store.</param>
/// <param name="ct">The cancellation token.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
Task PutAsync(GenerationSnapshot snapshot, CancellationToken ct = default);
/// <summary>Removes old generations, keeping only the most recent N.</summary>
/// <param name="clusterId">The cluster identifier.</param>
/// <param name="keepLatest">The number of latest generations to keep.</param>
/// <param name="ct">The cancellation token.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
Task PruneOldGenerationsAsync(string clusterId, int keepLatest = 10, CancellationToken ct = default);
}
src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/LiteDbConfigCache.cs
-129
View File
@@ -1,129 +0,0 @@
using LiteDB;
namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
/// <summary>
/// LiteDB-backed <see cref="ILocalConfigCache"/>. One file per node (default
/// <c>config_cache.db</c>), one collection per snapshot. Corruption surfaces as
/// <see cref="LocalConfigCacheCorruptException"/> on construction or read — callers should
/// delete and re-fetch from the central DB.
/// </summary>
public sealed class LiteDbConfigCache : ILocalConfigCache, IDisposable
{
private const string CollectionName = "generations";
// LiteDB's default BsonMapper.Global is a process-wide singleton whose per-type member
// registration is lazy and NOT thread-safe across concurrently-constructed LiteDatabase
// instances. When several caches (this one + GenerationSealedCache) initialise in parallel
// the global mapper races, surfacing as "Member ClusterId not found on BsonMapper" or a
// bogus "duplicate key _id = 0" (the int auto-id mapping was lost so Insert writes a literal
// 0 twice). Give each database a private, pre-registered mapper so member
// resolution happens once, single-threaded, at construction and never touches the global.
private static BsonMapper BuildMapper()
{
var mapper = new BsonMapper();
mapper.Entity<GenerationSnapshot>();
return mapper;
}
private readonly LiteDatabase _db;
private readonly ILiteCollection<GenerationSnapshot> _col;
// PutAsync is a find-then-insert/update; without serialization, two concurrent puts for the
// same (ClusterId, GenerationId) can both observe `existing is null` and both Insert,
// producing duplicate rows. Serialize writes through this semaphore so
// the read-modify-write block is atomic for a given instance. LiteDB itself only locks the
// page-level write, not the find-then-insert window.
private readonly SemaphoreSlim _writeGate = new(initialCount: 1, maxCount: 1);
/// <summary>Initializes a new instance of the <see cref="LiteDbConfigCache"/> class.</summary>
/// <param name="dbPath">Path to the LiteDB database file.</param>
public LiteDbConfigCache(string dbPath)
{
// LiteDB can be tolerant of header-only corruption at construction time (it may overwrite
// the header and "recover"), so we force a write + read probe to fail fast on real corruption.
try
{
_db = new LiteDatabase(new ConnectionString { Filename = dbPath, Upgrade = true }, BuildMapper());
_col = _db.GetCollection<GenerationSnapshot>(CollectionName);
_col.EnsureIndex(s => s.ClusterId);
_col.EnsureIndex(s => s.GenerationId);
_ = _col.Count();
}
catch (Exception ex) when (ex is LiteException or InvalidDataException or IOException
or NotSupportedException or UnauthorizedAccessException
or ArgumentOutOfRangeException or FormatException)
{
_db?.Dispose();
throw new LocalConfigCacheCorruptException(
$"LiteDB cache at '{dbPath}' is corrupt or unreadable — delete the file and refetch from the central DB.",
ex);
}
}
/// <inheritdoc />
public Task<GenerationSnapshot?> GetMostRecentAsync(string clusterId, CancellationToken ct = default)
{
ct.ThrowIfCancellationRequested();
var snapshot = _col
.Find(s => s.ClusterId == clusterId)
.OrderByDescending(s => s.GenerationId)
.FirstOrDefault();
return Task.FromResult<GenerationSnapshot?>(snapshot);
}
/// <inheritdoc />
public async Task PutAsync(GenerationSnapshot snapshot, CancellationToken ct = default)
{
ct.ThrowIfCancellationRequested();
// Serialize the find-then-insert/update so concurrent callers do not observe a stale
// `existing is null` and both Insert. LiteDB's per-call lock is not enough — the
// read and the write are independent calls.
await _writeGate.WaitAsync(ct).ConfigureAwait(false);
try
{
// upsert by (ClusterId, GenerationId) — replace in place if already cached
var existing = _col
.Find(s => s.ClusterId == snapshot.ClusterId && s.GenerationId == snapshot.GenerationId)
.FirstOrDefault();
if (existing is null)
_col.Insert(snapshot);
else
{
snapshot.Id = existing.Id;
_col.Update(snapshot);
}
}
finally
{
_writeGate.Release();
}
}
/// <inheritdoc />
public Task PruneOldGenerationsAsync(string clusterId, int keepLatest = 10, CancellationToken ct = default)
{
ct.ThrowIfCancellationRequested();
var doomed = _col
.Find(s => s.ClusterId == clusterId)
.OrderByDescending(s => s.GenerationId)
.Skip(keepLatest)
.Select(s => s.Id)
.ToList();
foreach (var id in doomed)
_col.Delete(id);
return Task.CompletedTask;
}
/// <summary>Releases all resources used by the cache.</summary>
public void Dispose()
{
_writeGate.Dispose();
_db.Dispose();
}
}
public sealed class LocalConfigCacheCorruptException(string message, Exception inner)
: Exception(message, inner);
src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/ResilientConfigReader.cs
-140
View File
@@ -1,140 +0,0 @@
using System.Text.RegularExpressions;
using Microsoft.Extensions.Logging;
using Polly;
using Polly.Retry;
using Polly.Timeout;
namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
/// <summary>
/// Wraps a central-DB fetch function with Phase 6.1 Stream D.2 resilience:
/// <b>timeout 2 s → retry 3× jittered → fallback to sealed cache</b>. Maintains the
/// <see cref="StaleConfigFlag"/> — fresh on central-DB success, stale on cache fallback.
/// </summary>
/// <remarks>
/// <para>Read-path only per plan. The write path (draft save, publish) bypasses this
/// wrapper entirely and fails hard on DB outage so inconsistent writes never land.</para>
///
/// <para>Fallback is triggered by <b>any exception</b> the fetch raises (central-DB
/// unreachable, SqlException, timeout). If the sealed cache also fails (no pointer,
/// corrupt file, etc.), <see cref="GenerationCacheUnavailableException"/> surfaces — caller
/// must fail the current request (InitializeAsync for a driver, etc.).</para>
/// </remarks>
public sealed class ResilientConfigReader
{
private readonly GenerationSealedCache _cache;
private readonly StaleConfigFlag _staleFlag;
private readonly ResiliencePipeline _pipeline;
private readonly ILogger<ResilientConfigReader> _logger;
/// <summary>Initializes a resilient config reader with the given cache and options.</summary>
/// <param name="cache">The sealed cache for fallback.</param>
/// <param name="staleFlag">The stale config flag to manage.</param>
/// <param name="logger">The logger instance.</param>
/// <param name="timeout">The timeout for central fetch (default 2s).</param>
/// <param name="retryCount">The number of retries (default 3).</param>
public ResilientConfigReader(
GenerationSealedCache cache,
StaleConfigFlag staleFlag,
ILogger<ResilientConfigReader> logger,
TimeSpan? timeout = null,
int retryCount = 3)
{
_cache = cache;
_staleFlag = staleFlag;
_logger = logger;
var builder = new ResiliencePipelineBuilder()
.AddTimeout(new TimeoutStrategyOptions { Timeout = timeout ?? TimeSpan.FromSeconds(2) });
if (retryCount > 0)
{
builder.AddRetry(new RetryStrategyOptions
{
MaxRetryAttempts = retryCount,
BackoffType = DelayBackoffType.Exponential,
UseJitter = true,
Delay = TimeSpan.FromMilliseconds(100),
MaxDelay = TimeSpan.FromSeconds(1),
// Handle ALL exceptions including OperationCanceledException. A SQL command-level
// timeout surfaces as TaskCanceledException (derives from OperationCanceledException)
// when the caller's token is NOT cancelled, and must be retried just like any other
// transient error. Polly itself checks the cancellation token between retries and
// stops with OperationCanceledException on genuine caller cancellation regardless of
// this predicate.
ShouldHandle = new PredicateBuilder().Handle<Exception>(),
});
}
_pipeline = builder.Build();
}
/// <summary>
/// Redacts connection-string fragments (Password, User Id, Pwd, etc.)
/// that a caller's exception message could carry. Conservative regex pass — anything
/// matching <c>Key=Value</c> with a known credential key gets its value replaced.
/// </summary>
private static readonly Regex SecretsRegex = new(
@"(?ix)\b(Password|Pwd|User\s*Id|Uid|AccessToken|Authorization|Api[-_]?Key)\s*=\s*[^;,)\s]*",
RegexOptions.Compiled);
/// <summary>Redacts sensitive credential information from a message.</summary>
/// <param name="message">The message to scrub.</param>
/// <returns>The message with redacted credentials.</returns>
internal static string ScrubSecrets(string? message)
{
if (string.IsNullOrEmpty(message)) return message ?? string.Empty;
// Replace the entire matched fragment (key + value) with a redaction marker so the
// key name itself doesn't leak — log scrapers grep for "Password=" too.
return SecretsRegex.Replace(message, "[redacted credential]");
}
/// <summary>
/// Executes a central fetch through the resilience pipeline. On full failure
/// (post-retry), reads the sealed cache and extracts the requested shape.
/// </summary>
/// <typeparam name="T">The type of configuration to read.</typeparam>
/// <param name="clusterId">The cluster ID to fetch for.</param>
/// <param name="centralFetch">Function to fetch from central DB.</param>
/// <param name="fromSnapshot">Function to extract the config from a snapshot.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The configuration of type T.</returns>
public async ValueTask<T> ReadAsync<T>(
string clusterId,
Func<CancellationToken, ValueTask<T>> centralFetch,
Func<GenerationSnapshot, T> fromSnapshot,
CancellationToken cancellationToken)
{
ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
ArgumentNullException.ThrowIfNull(centralFetch);
ArgumentNullException.ThrowIfNull(fromSnapshot);
try
{
var result = await _pipeline.ExecuteAsync(centralFetch, cancellationToken).ConfigureAwait(false);
_staleFlag.MarkFresh();
return result;
}
// Catch all exceptions that are NOT genuine caller cancellations. A SQL command-level
// timeout surfaces as TaskCanceledException (derives from OperationCanceledException)
// but the caller's token is NOT cancelled — we must fall back to the sealed cache for
// that case, not propagate. Only rethrow if the caller actually requested cancellation.
catch (Exception ex) when (ex is not OperationCanceledException || !cancellationToken.IsCancellationRequested)
{
// Do NOT pass the raw exception object — it carries the stack
// and inner-exception chain, and SqlException/wrapping delegates can surface
// connection-string fragments (Password=…, User Id=…) embedded in messages.
// Log only the exception type and a scrubbed message so secrets stay out of logs.
_logger.LogWarning(
"Central-DB read failed after retries ({ExceptionType}: {SanitizedMessage}); falling back to sealed cache for cluster {ClusterId}",
ex.GetType().Name,
ScrubSecrets(ex.Message),
clusterId);
// GenerationCacheUnavailableException surfaces intentionally — fails the caller's
// operation. StaleConfigFlag stays unchanged; the flag only flips when we actually
// served a cache snapshot.
var snapshot = await _cache.ReadCurrentAsync(clusterId, cancellationToken).ConfigureAwait(false);
_staleFlag.MarkStale();
return fromSnapshot(snapshot);
}
}
}
src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/StaleConfigFlag.cs
-20
View File
@@ -1,20 +0,0 @@
namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
/// <summary>
/// Thread-safe <c>UsingStaleConfig</c> signal per Phase 6.1 Stream D.3. Flips true whenever
/// a read falls back to a sealed cache snapshot; flips false on the next successful central-DB
/// round-trip. Surfaced on <c>/healthz</c> body and on the Admin <c>/hosts</c> page.
/// </summary>
public sealed class StaleConfigFlag
{
private int _stale;
/// <summary>True when the last config read was served from the sealed cache, not the central DB.</summary>
public bool IsStale => Volatile.Read(ref _stale) != 0;
/// <summary>Mark the current config as stale (a read fell back to the cache).</summary>
public void MarkStale() => Volatile.Write(ref _stale, 1);
/// <summary>Mark the current config as fresh (a central-DB read succeeded).</summary>
public void MarkFresh() => Volatile.Write(ref _stale, 0);
}
src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Services/ILdapGroupRoleMappingService.cs
+12 -4
View File
@@ -10,10 +10,18 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Services;
/// Phase 6.2 compliance check on control/data-plane separation).
/// </summary>
/// <remarks>
/// Per Phase 6.2 Stream A.2 this service is expected to run behind the Phase 6.1
/// <c>ResilientConfigReader</c> pipeline (timeout → retry → fallback-to-cache) so a
/// transient DB outage during sign-in falls back to the sealed snapshot rather than
/// denying every login.
/// <para>
/// This service has no local-cache fallback: a DB outage during sign-in denies logins.
/// </para>
/// <para>
/// The Phase 6.1 <c>ResilientConfigReader</c> pipeline (timeout → retry →
/// fallback-to-sealed-snapshot) this was once expected to run behind was never wired to
/// anything and was deleted along with the rest of the dormant LiteDB local cache, which
/// <c>ZB.MOM.WW.LocalDb</c> supersedes. LocalDb caches the deployed-configuration artifact
/// for driver-role nodes; it does not currently cover admin-plane reads like this one.
/// Reviving the fallback means adding an admin-side cache on LocalDb, not restoring the
/// old pipeline.
/// </para>
/// </remarks>
public interface ILdapGroupRoleMappingService
{
src/Core/ZB.MOM.WW.OtOpcUa.Configuration/ZB.MOM.WW.OtOpcUa.Configuration.csproj
-1
View File
@@ -26,7 +26,6 @@
<PackageReference Include="Microsoft.AspNetCore.DataProtection.EntityFrameworkCore"/>
<PackageReference Include="Microsoft.Extensions.Configuration.Abstractions"/>
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions"/>
<PackageReference Include="LiteDB"/>
<PackageReference Include="Polly.Core"/>
</ItemGroup>
src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmSfSchema.cs
+105
View File
@@ -0,0 +1,105 @@
using System.Security.Cryptography;
using System.Text;
using Microsoft.Data.Sqlite;
namespace ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
/// <summary>
/// DDL for the alarm store-and-forward buffer: one row per alarm event awaiting delivery to
/// the historian gateway.
/// </summary>
/// <remarks>
/// <para>
/// Replaces the standalone <c>alarm-historian.db</c> the sink used to own outright. Living
/// in the consolidated LocalDb file is what lets the buffer replicate to the redundant pair
/// peer, so a node that dies with undelivered alarm history no longer takes it to the grave.
/// </para>
/// <para>
/// Deliberately depends on nothing but <see cref="SqliteConnection"/> so it can be applied
/// to any connection — the host's <c>LocalDbSetup.OnReady</c> in production, and a bare
/// connection in a test — without dragging the DI graph along. Mirrors
/// <c>DeploymentCacheSchema</c>.
/// </para>
/// <para>
/// <b>Why <see cref="IdColumn"/> is TEXT and app-minted.</b> The legacy queue keyed on
/// <c>RowId INTEGER PRIMARY KEY AUTOINCREMENT</c>. Convergence is last-writer-wins over the
/// primary key, so two nodes independently allocating rowid 7 for different alarms would
/// silently overwrite one another. The sink mints the id from a hash of the payload, which
/// additionally makes the same event converge to one row when both nodes of a pair enqueue
/// it — as they legitimately do in the window before the first redundancy snapshot arrives.
/// </para>
/// </remarks>
public static class AlarmSfSchema
{
/// <summary>Table holding queued alarm events awaiting delivery.</summary>
public const string EventsTable = "alarm_sf_events";
/// <summary>The single primary-key column.</summary>
public const string IdColumn = "id";
/// <summary>
/// Derives a row's primary key from its serialized payload.
/// </summary>
/// <remarks>
/// <para>
/// Deterministic rather than a fresh GUID, so that the same event arriving twice
/// converges to one row under last-writer-wins instead of duplicating. That happens for
/// real in two places: <c>HistorianAdapterActor</c> default-writes while its redundancy
/// role is unknown, so both nodes of a pair accept the same fanned transition in the
/// window before the first snapshot; and both nodes independently migrate their own
/// legacy queue file, which for a warm pair holds overlapping history.
/// </para>
/// <para>
/// Two genuinely distinct events cannot collide. <c>AlarmHistorianEvent</c> carries a
/// full-precision timestamp alongside the alarm id, transition kind, message and user,
/// so an equal hash means an equal event.
/// </para>
/// </remarks>
/// <param name="payloadJson">The serialized <c>AlarmHistorianEvent</c>.</param>
/// <returns>The row's primary key.</returns>
public static string DeriveId(string payloadJson)
{
ArgumentNullException.ThrowIfNull(payloadJson);
return Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(payloadJson)));
}
/// <summary>
/// Creates the buffer table if it does not already exist. Idempotent.
/// </summary>
/// <param name="connection">
/// An already-open connection. <c>ILocalDb.CreateConnection()</c> hands out open,
/// pragma-configured connections — do not call <c>Open()</c> on one.
/// </param>
public static void Apply(SqliteConnection connection)
{
ArgumentNullException.ThrowIfNull(connection);
using var cmd = connection.CreateCommand();
// Columns map 1:1 onto the legacy Queue table so the one-time migrator is a straight copy
// and the drain keeps its existing semantics: dead_lettered is the same 0/1 flag, and an
// acknowledged row is DELETEd rather than marked. Deleting keeps the table bounded without
// a second sweeper, and the replication engine carries the delete as a tombstone, so the
// peer drops its copy too.
//
// The drain index orders by enqueued_at_utc because a hashed TEXT id carries no insertion
// order the way the legacy AUTOINCREMENT RowId did. Timestamps are round-trip ("O") format,
// so lexicographic ordering is chronological; id breaks ties so the order is total and the
// index covers the drain's read.
cmd.CommandText = """
CREATE TABLE IF NOT EXISTS alarm_sf_events (
id TEXT NOT NULL PRIMARY KEY,
alarm_id TEXT NOT NULL,
enqueued_at_utc TEXT NOT NULL,
payload_json TEXT NOT NULL,
attempt_count INTEGER NOT NULL DEFAULT 0,
last_attempt_utc TEXT NULL,
last_error TEXT NULL,
dead_lettered INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX IF NOT EXISTS ix_alarm_sf_events_drain
ON alarm_sf_events (dead_lettered, enqueued_at_utc, id);
""";
cmd.ExecuteNonQuery();
}
}
src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/IAlarmHistorianSink.cs
+14 -2
View File
@@ -4,13 +4,13 @@ namespace ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
/// The historian sink contract — where qualifying alarm events land. Ingestion routes
/// through the HistorianGateway alarm writer (the gateway's <c>SendEvent</c> gRPC path)
/// behind the durable store-and-forward queue. Tests use an in-memory fake; production uses
/// <see cref="SqliteStoreAndForwardSink"/>.
/// <see cref="LocalDbStoreAndForwardSink"/>.
/// </summary>
/// <remarks>
/// <para>
/// <see cref="EnqueueAsync"/> is fire-and-forget from the engine's perspective —
/// the sink MUST NOT block the emitting thread. Production implementations
/// (<see cref="SqliteStoreAndForwardSink"/>) persist to a local SQLite queue
/// (<see cref="LocalDbStoreAndForwardSink"/>) persist to the node's local SQLite queue
/// first, then drain asynchronously to the actual historian. Per the Phase 7 plan,
/// failed downstream writes replay with exponential backoff;
/// operator actions are never blocked waiting on the historian.
@@ -79,6 +79,18 @@ public enum HistorianDrainState
Idle,
Draining,
BackingOff,
/// <summary>
/// Ticking, but not draining: this node does not hold the Primary role, so it leaves the
/// (replicated) queue for the node that does.
/// </summary>
/// <remarks>
/// Distinct from <see cref="Idle"/> so a rising queue depth on a Secondary reads as the
/// designed behaviour rather than a stalled drain. It is also how a misconfigured pair
/// becomes diagnosable: if BOTH nodes report this, no one is draining and alarm history is
/// silently accumulating toward the capacity ceiling.
/// </remarks>
NotPrimary,
}
/// <summary>Returned by the historian alarm writer per event — drain worker uses this to decide retry cadence.</summary>
src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs → src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/LocalDbStoreAndForwardSink.cs
+262 -243
View File
@@ -1,53 +1,54 @@
using System.Text.Json;
using Microsoft.Data.Sqlite;
using Serilog;
using ZB.MOM.WW.LocalDb;
namespace ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
/// <summary>
/// Durable SQLite queue on the node absorbs every qualifying alarm event, a drain
/// worker batches rows to the Wonderware historian sidecar via
/// <see cref="IAlarmHistorianWriter"/> on an exponential-backoff cadence, and
/// operator acks never block on the historian being reachable.
/// Durable queue in the node's consolidated LocalDb absorbs every qualifying alarm event, a
/// drain worker batches rows to the historian gateway via <see cref="IAlarmHistorianWriter"/>
/// on an exponential-backoff cadence, and operator acks never block on the historian being
/// reachable.
/// </summary>
/// <remarks>
/// <para>
/// Queue schema:
/// <code>
/// CREATE TABLE Queue (
/// RowId INTEGER PRIMARY KEY AUTOINCREMENT,
/// AlarmId TEXT NOT NULL,
/// EnqueuedUtc TEXT NOT NULL,
/// PayloadJson TEXT NOT NULL,
/// AttemptCount INTEGER NOT NULL DEFAULT 0,
/// LastAttemptUtc TEXT NULL,
/// LastError TEXT NULL,
/// DeadLettered INTEGER NOT NULL DEFAULT 0
/// );
/// </code>
/// Dead-lettered rows stay in place for the configured retention window (default
/// 30 days) so operators can inspect + manually
/// retry before the sweeper purges them. Regular queue capacity is bounded —
/// overflow evicts the oldest non-dead-lettered rows with a WARN log. The
/// durability guarantee is therefore bounded by <see cref="DefaultCapacity"/>:
/// under a sustained historian outage, accepted events may be evicted before
/// delivery. The <see cref="HistorianSinkStatus.EvictedCount"/> counter makes
/// overflow visible to operators without requiring the WARN log to be scraped.
/// Rows live in <c>alarm_sf_events</c> (see <see cref="AlarmSfSchema"/>), which the host
/// registers for replication. That is the point of this type living on
/// <see cref="ILocalDb"/> rather than owning its own file: the buffer mirrors to the
/// redundant pair peer, so a node that dies holding undelivered alarm history no longer
/// takes it to the grave.
/// </para>
/// <para>
/// <b>Only one node of a pair may drain.</b> Replication puts the Primary's queued rows in
/// the Secondary's table too; an ungated drain there would re-deliver every event,
/// continuously. The <c>drainGate</c> constructor argument is how the caller scopes
/// draining to the Primary. Enqueue is gated separately and upstream, by
/// <c>HistorianAdapterActor</c>.
/// </para>
/// <para>
/// Dead-lettered rows stay in place for the configured retention window (default 30 days)
/// so operators can inspect + manually retry before the sweeper purges them. Regular queue
/// capacity is bounded — overflow evicts the oldest non-dead-lettered rows with a WARN log.
/// The durability guarantee is therefore bounded by <see cref="DefaultCapacity"/>: under a
/// sustained historian outage, accepted events may be evicted before delivery. The
/// <see cref="HistorianSinkStatus.EvictedCount"/> counter makes overflow visible to
/// operators without requiring the WARN log to be scraped.
/// </para>
/// <para>
/// Drain runs on a self-rescheduling one-shot <see cref="System.Threading.Timer"/>.
/// Exponential backoff on <see cref="HistorianWriteOutcome.RetryPlease"/>:
/// 1s → 2s → 5s → 15s → 60s cap — the backoff is applied to the timer's next
/// due-time, so a historian outage genuinely slows the drain cadence.
/// <see cref="HistorianWriteOutcome.PermanentFail"/> rows flip
/// the <c>DeadLettered</c> flag on the individual row; neighbors in the batch
/// still retry on their own cadence.
/// 1s → 2s → 5s → 15s → 60s cap — the backoff is applied to the timer's next due-time, so a
/// historian outage genuinely slows the drain cadence.
/// <see cref="HistorianWriteOutcome.PermanentFail"/> rows flip the <c>dead_lettered</c> flag
/// on the individual row; neighbors in the batch still retry on their own cadence.
/// </para>
/// </remarks>
public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
public sealed class LocalDbStoreAndForwardSink : IAlarmHistorianSink, IDisposable
{
/// <summary>Default queue capacity — oldest non-dead-lettered rows evicted past this.</summary>
public const long DefaultCapacity = 1_000_000;
/// <summary>Default window dead-lettered rows are retained for before the sweeper purges them.</summary>
public static readonly TimeSpan DefaultDeadLetterRetention = TimeSpan.FromDays(30);
/// <summary>Default max delivery attempts before a perpetually-retrying (poison) row is dead-lettered.</summary>
@@ -62,7 +63,7 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
TimeSpan.FromSeconds(60),
];
private readonly string _connectionString;
private readonly ILocalDb _db;
private readonly IAlarmHistorianWriter _writer;
private readonly ILogger _logger;
private readonly int _batchSize;
@@ -70,8 +71,9 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
private readonly TimeSpan _deadLetterRetention;
private readonly int _maxAttempts;
private readonly Func<DateTime> _clock;
private readonly Func<bool> _drainGate;
private readonly SemaphoreSlim _drainGate = new(1, 1);
private readonly SemaphoreSlim _drainGateLock = new(1, 1);
private Timer? _drainTimer;
private TimeSpan _tickInterval;
private volatile int _backoffIndex;
@@ -85,13 +87,16 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
private string? _lastError;
private HistorianDrainState _drainState = HistorianDrainState.Idle;
private long _evictedCount;
// Tracks the last gate decision so a denial is logged on the transition rather than on every
// tick. Null until the first drain runs.
private bool? _lastGateDecision;
private long _queuedRowCount;
// Probe counter — incremented every time we actually issue a real COUNT(*) for
// capacity enforcement. Public for test instrumentation only.
private long _capacityProbeCount;
// After every Nth enqueue we resync the in-memory counter from storage to defend
// against silent drift (e.g. an external process editing the DB).
// against silent drift (e.g. the peer replicating rows in underneath us).
private const long ResyncEnqueueInterval = 10_000;
private long _enqueuesSinceResync;
@@ -99,9 +104,13 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
public long DebugCapacityProbeCount => Interlocked.Read(ref _capacityProbeCount);
/// <summary>
/// Initializes a new instance of the <see cref="SqliteStoreAndForwardSink"/> class with the specified configuration.
/// Initializes a new instance of the <see cref="LocalDbStoreAndForwardSink"/> class.
/// </summary>
/// <param name="databasePath">The filesystem path to the SQLite database file.</param>
/// <param name="db">
/// The node's local database. Its <c>alarm_sf_events</c> table must already exist and be
/// registered for replication — the host does both in <c>LocalDbSetup.OnReady</c>, before
/// any consumer can resolve this sink.
/// </param>
/// <param name="writer">The alarm historian writer to handle batch forwarding.</param>
/// <param name="logger">The logger for diagnostic output.</param>
/// <param name="batchSize">The maximum number of rows to forward in a single batch. Defaults to 100.</param>
@@ -109,18 +118,24 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
/// <param name="deadLetterRetention">The timespan to retain dead-lettered rows before purging. Defaults to 30 days.</param>
/// <param name="maxAttempts">The maximum number of delivery attempts before a perpetually-retrying (poison) row is dead-lettered. Defaults to 10.</param>
/// <param name="clock">Optional clock function for testing; defaults to <see cref="DateTime.UtcNow"/>.</param>
public SqliteStoreAndForwardSink(
string databasePath,
/// <param name="drainGate">
/// Consulted at the top of every drain tick; <c>false</c> skips the tick entirely, leaving
/// the queue intact. Callers running a redundant pair MUST supply a gate that is true on at
/// most one node, because the queue replicates — see the class remarks. The default
/// (<c>null</c> ⇒ always drain) is the correct single-node and test posture.
/// </param>
public LocalDbStoreAndForwardSink(
ILocalDb db,
IAlarmHistorianWriter writer,
ILogger logger,
int batchSize = 100,
long capacity = DefaultCapacity,
TimeSpan? deadLetterRetention = null,
int maxAttempts = DefaultMaxAttempts,
Func<DateTime>? clock = null)
Func<DateTime>? clock = null,
Func<bool>? drainGate = null)
{
if (string.IsNullOrWhiteSpace(databasePath))
throw new ArgumentException("Database path required.", nameof(databasePath));
_db = db ?? throw new ArgumentNullException(nameof(db));
_writer = writer ?? throw new ArgumentNullException(nameof(writer));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_batchSize = batchSize > 0 ? batchSize : throw new ArgumentOutOfRangeException(nameof(batchSize));
@@ -128,50 +143,11 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
_deadLetterRetention = deadLetterRetention ?? DefaultDeadLetterRetention;
_maxAttempts = maxAttempts > 0 ? maxAttempts : throw new ArgumentOutOfRangeException(nameof(maxAttempts));
_clock = clock ?? (() => DateTime.UtcNow);
// DefaultTimeout gives ADO.NET command-level retry; the PRAGMA busy_timeout
// applied in OpenConnection backs it with SQLite's own busy-handler so an
// enqueue/drain collision waits out the file lock instead of throwing
// SQLITE_BUSY immediately.
_connectionString = new SqliteConnectionStringBuilder
{
DataSource = databasePath,
DefaultTimeout = 5,
}.ToString();
_drainGate = drainGate ?? (static () => true);
InitializeSchema();
_queuedRowCount = ProbeQueuedRowCount();
}
/// <summary>
/// Open a connection with the busy timeout + WAL journal applied. SQLite
/// serializes writers with a file lock; the busy_timeout lets a writer wait
/// out a competing lock (default is 0 — fail fast), and WAL lets readers and
/// the single writer proceed without blocking each other.
/// </summary>
private SqliteConnection OpenConnection()
{
var conn = new SqliteConnection(_connectionString);
conn.Open();
ApplyPragmas(conn);
return conn;
}
/// <summary>Apply busy_timeout + WAL pragmas to an already-open connection (sync).</summary>
private static void ApplyPragmas(SqliteConnection conn)
{
using var pragma = conn.CreateCommand();
pragma.CommandText = "PRAGMA busy_timeout=5000; PRAGMA journal_mode=WAL;";
pragma.ExecuteNonQuery();
}
/// <summary>Apply busy_timeout + WAL pragmas to an already-open connection (async).</summary>
private static async Task ApplyPragmasAsync(SqliteConnection conn, CancellationToken ct)
{
using var pragma = conn.CreateCommand();
pragma.CommandText = "PRAGMA busy_timeout=5000; PRAGMA journal_mode=WAL;";
await pragma.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
}
/// <summary>
/// Start the background drain worker. Not started automatically so tests can
/// drive <see cref="DrainOnceAsync"/> deterministically.
@@ -188,7 +164,7 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
/// <param name="tickInterval">The base interval between drain attempts.</param>
public void StartDrainLoop(TimeSpan tickInterval)
{
if (_disposed) throw new ObjectDisposedException(nameof(SqliteStoreAndForwardSink));
if (_disposed) throw new ObjectDisposedException(nameof(LocalDbStoreAndForwardSink));
_tickInterval = tickInterval;
_drainTimer?.Dispose();
// One-shot: dueTime = tickInterval, period = Infinite. RescheduleDrain re-arms
@@ -239,25 +215,31 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
public async Task EnqueueAsync(AlarmHistorianEvent evt, CancellationToken cancellationToken)
{
if (evt is null) throw new ArgumentNullException(nameof(evt));
if (_disposed) throw new ObjectDisposedException(nameof(SqliteStoreAndForwardSink));
if (_disposed) throw new ObjectDisposedException(nameof(LocalDbStoreAndForwardSink));
using var conn = new SqliteConnection(_connectionString);
await conn.OpenAsync(cancellationToken).ConfigureAwait(false);
await ApplyPragmasAsync(conn, cancellationToken).ConfigureAwait(false);
await EnforceCapacityFastPathAsync(cancellationToken).ConfigureAwait(false);
await EnforceCapacityFastPathAsync(conn, cancellationToken).ConfigureAwait(false);
var payload = JsonSerializer.Serialize(evt);
using var cmd = conn.CreateCommand();
cmd.CommandText = """
INSERT INTO Queue (AlarmId, EnqueuedUtc, PayloadJson, AttemptCount)
VALUES ($alarmId, $enqueued, $payload, 0);
""";
cmd.Parameters.AddWithValue("$alarmId", evt.AlarmId);
cmd.Parameters.AddWithValue("$enqueued", _clock().ToString("O"));
cmd.Parameters.AddWithValue("$payload", JsonSerializer.Serialize(evt));
await cmd.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false);
// ON CONFLICT DO NOTHING: re-enqueuing an identical event (the pair's boot-window
// double-accept) must not resurrect a row the drain already bumped or dead-lettered.
var inserted = await _db.ExecuteAsync(
"""
INSERT INTO alarm_sf_events
(id, alarm_id, enqueued_at_utc, payload_json, attempt_count)
VALUES (@Id, @AlarmId, @EnqueuedAtUtc, @PayloadJson, 0)
ON CONFLICT(id) DO NOTHING
""",
new
{
Id = AlarmSfSchema.DeriveId(payload),
AlarmId = evt.AlarmId,
EnqueuedAtUtc = _clock().ToString("O"),
PayloadJson = payload,
},
cancellationToken).ConfigureAwait(false);
Interlocked.Increment(ref _queuedRowCount);
if (inserted > 0) Interlocked.Increment(ref _queuedRowCount);
}
/// <summary>
@@ -268,15 +250,17 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
/// through <see cref="EnforceCapacityAsync"/> which still runs a precise
/// COUNT to compute the exact number of rows to evict.
/// </summary>
private async Task EnforceCapacityFastPathAsync(SqliteConnection conn, CancellationToken ct)
private async Task EnforceCapacityFastPathAsync(CancellationToken ct)
{
var enqueuesSinceResync = Interlocked.Increment(ref _enqueuesSinceResync);
var cached = Interlocked.Read(ref _queuedRowCount);
// Periodic resync — bounded amount of drift even under exotic conditions.
// Periodic resync — bounded amount of drift even under exotic conditions. Now doubly
// warranted: replication applies the peer's rows straight into the table, so the counter
// has a second way to fall behind that no local code path can observe.
if (enqueuesSinceResync >= ResyncEnqueueInterval)
{
await ResyncQueuedRowCountAsync(conn, ct).ConfigureAwait(false);
await ResyncQueuedRowCountAsync(ct).ConfigureAwait(false);
cached = Interlocked.Read(ref _queuedRowCount);
Interlocked.Exchange(ref _enqueuesSinceResync, 0);
}
@@ -286,55 +270,70 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
// Cached counter says we're at or above the capacity wall — fall back to the
// precise path which probes COUNT(*) and evicts whatever's needed.
await EnforceCapacityAsync(conn, ct).ConfigureAwait(false);
await EnforceCapacityAsync(ct).ConfigureAwait(false);
}
/// <summary>Synchronously query <c>COUNT(*)</c> of non-dead-lettered rows. Used at startup.</summary>
private long ProbeQueuedRowCount()
{
Interlocked.Increment(ref _capacityProbeCount);
using var conn = OpenConnection();
using var conn = _db.CreateConnection();
using var cmd = conn.CreateCommand();
cmd.CommandText = "SELECT COUNT(*) FROM Queue WHERE DeadLettered = 0";
cmd.CommandText = "SELECT COUNT(*) FROM alarm_sf_events WHERE dead_lettered = 0";
return (long)(cmd.ExecuteScalar() ?? 0L);
}
/// <summary>Re-sync the in-memory counter from storage (async path).</summary>
private async Task ResyncQueuedRowCountAsync(SqliteConnection conn, CancellationToken ct)
private async Task ResyncQueuedRowCountAsync(CancellationToken ct)
{
Interlocked.Increment(ref _capacityProbeCount);
using var cmd = conn.CreateCommand();
cmd.CommandText = "SELECT COUNT(*) FROM Queue WHERE DeadLettered = 0";
var live = (long)(await cmd.ExecuteScalarAsync(ct).ConfigureAwait(false) ?? 0L);
var live = await CountAsync("dead_lettered = 0", ct).ConfigureAwait(false);
Interlocked.Exchange(ref _queuedRowCount, live);
}
private async Task<long> CountAsync(string predicate, CancellationToken ct)
{
var rows = await _db.QueryAsync(
$"SELECT COUNT(*) FROM alarm_sf_events WHERE {predicate}",
r => r.GetInt64(0),
parameters: null,
ct).ConfigureAwait(false);
return rows.Count > 0 ? rows[0] : 0L;
}
/// <summary>
/// Read up to <see cref="_batchSize"/> queued rows, forward through the writer,
/// remove Ack'd rows, dead-letter PermanentFail rows, and extend the backoff
/// on RetryPlease. Safe to call from multiple threads; the semaphore enforces
/// serial execution.
/// </summary>
/// <remarks>
/// Returns without touching the queue when the drain gate is closed. On a redundant pair
/// that is the Secondary's steady state: its table fills by replication and stays put until
/// it is promoted.
/// </remarks>
/// <param name="ct">Cancellation token for the operation.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task DrainOnceAsync(CancellationToken ct)
{
if (_disposed) return;
if (!await _drainGate.WaitAsync(0, ct).ConfigureAwait(false)) return;
if (!await _drainGateLock.WaitAsync(0, ct).ConfigureAwait(false)) return;
try
{
if (!EvaluateDrainGate())
{
lock (_statusLock) { _drainState = HistorianDrainState.NotPrimary; }
return;
}
lock (_statusLock)
{
_drainState = HistorianDrainState.Draining;
_lastDrainUtc = _clock();
}
// One connection per drain tick — used by purge, read, corrupt-dead-letter,
// and the outcome-applying transaction.
using var conn = OpenConnection();
PurgeAgedDeadLetters(conn);
var batch = ReadBatch(conn);
await PurgeAgedDeadLettersAsync(ct).ConfigureAwait(false);
var batch = await ReadBatchAsync(ct).ConfigureAwait(false);
if (batch.Count == 0)
{
lock (_statusLock) { _drainState = HistorianDrainState.Idle; }
@@ -342,22 +341,24 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
}
// A null/un-deserializable payload can never succeed — dead-letter it
// immediately for its own RowId so it cannot stall the queue head, and
// immediately for its own id so it cannot stall the queue head, and
// exclude it from the batch handed to the writer.
var corruptRowIds = batch.Where(r => r.Event is null).Select(r => r.RowId).ToList();
var corruptIds = batch.Where(r => r.Event is null).Select(r => r.Id).ToList();
var liveRows = batch.Where(r => r.Event is not null).ToList();
var events = liveRows.Select(r => r.Event!).ToList();
if (corruptRowIds.Count > 0)
if (corruptIds.Count > 0)
{
using var corruptTx = conn.BeginTransaction();
foreach (var rowId in corruptRowIds)
DeadLetterRow(conn, corruptTx, rowId, $"corrupt payload at {_clock():O}");
corruptTx.Commit();
Interlocked.Add(ref _queuedRowCount, -corruptRowIds.Count);
await using (var corruptTx = await _db.BeginTransactionAsync(ct).ConfigureAwait(false))
{
foreach (var id in corruptIds)
await DeadLetterRowAsync(corruptTx, id, $"corrupt payload at {_clock():O}", ct).ConfigureAwait(false);
await corruptTx.CommitAsync(ct).ConfigureAwait(false);
}
Interlocked.Add(ref _queuedRowCount, -corruptIds.Count);
_logger.Warning(
"Dead-lettered {Count} historian queue row(s) with un-deserializable payload",
corruptRowIds.Count);
corruptIds.Count);
}
if (events.Count == 0)
@@ -404,41 +405,44 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
}
int rowsLeavingQueue = 0;
using (var tx = conn.BeginTransaction())
await using (var tx = await _db.BeginTransactionAsync(ct).ConfigureAwait(false))
{
for (var i = 0; i < outcomes.Count; i++)
{
var outcome = outcomes[i];
var rowId = liveRows[i].RowId;
var id = liveRows[i].Id;
switch (outcome)
{
case HistorianWriteOutcome.Ack:
DeleteRow(conn, tx, rowId);
// Deleted, not marked delivered. The replication engine carries the
// delete as a tombstone, so the peer drops its copy and cannot
// re-deliver the row after a failover.
await DeleteRowAsync(tx, id, ct).ConfigureAwait(false);
rowsLeavingQueue++;
break;
case HistorianWriteOutcome.PermanentFail:
DeadLetterRow(conn, tx, rowId, $"permanent fail at {_clock():O}");
await DeadLetterRowAsync(tx, id, $"permanent fail at {_clock():O}", ct).ConfigureAwait(false);
rowsLeavingQueue++;
break;
case HistorianWriteOutcome.RetryPlease:
// finding 002: cap retries so a perpetually-RetryPlease (poison)
// row cannot retry forever at the 60s backoff floor. The incoming
// AttemptCount is the count BEFORE this attempt; +1 accounts for the
// attempt count is the count BEFORE this attempt; +1 accounts for the
// bump this drain represents. At the cap, dead-letter instead of
// bumping — and count it as leaving the live queue like PermanentFail.
if (liveRows[i].AttemptCount + 1 >= _maxAttempts)
{
DeadLetterRow(conn, tx, rowId, $"max attempts ({_maxAttempts}) exceeded");
await DeadLetterRowAsync(tx, id, $"max attempts ({_maxAttempts}) exceeded", ct).ConfigureAwait(false);
rowsLeavingQueue++;
}
else
{
BumpAttempt(conn, tx, rowId, "retry-please");
await BumpAttemptAsync(tx, id, "retry-please", ct).ConfigureAwait(false);
}
break;
}
}
tx.Commit();
await tx.CommitAsync(ct).ConfigureAwait(false);
}
// Ack-deleted + PermanentFail-dead-lettered rows both leave the
// non-dead-lettered queue, as do RetryPlease rows that hit the max-attempts
@@ -464,22 +468,66 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
}
finally
{
_drainGate.Release();
_drainGateLock.Release();
}
}
/// <summary>
/// Consults the injected gate, logging only when the decision changes.
/// </summary>
/// <remarks>
/// Logged at all because the closed state is indistinguishable from a healthy idle queue
/// from the outside, and one way to reach it is a redundancy snapshot that never names this
/// node — the documented identity-mismatch shape. Without this line, that misconfiguration
/// would present as alarm history quietly ceasing on both nodes of a pair. Logged on the
/// transition rather than per tick because the drain runs every few seconds.
/// </remarks>
/// <returns><c>true</c> when this tick may proceed.</returns>
private bool EvaluateDrainGate()
{
bool open;
try
{
open = _drainGate();
}
catch (Exception ex)
{
// A throwing gate must not wedge the queue silently, and must not be treated as
// permission either: skip this tick, say why, and re-ask on the next one.
_logger.Error(ex, "Historian drain gate threw; skipping this tick");
return false;
}
bool? previous;
lock (_statusLock)
{
previous = _lastGateDecision;
_lastGateDecision = open;
}
if (previous == open) return open;
if (open)
_logger.Information("Historian drain resumed — this node now services the alarm queue");
else
_logger.Information(
"Historian drain suspended — this node is not the Primary. Queued alarm events are retained and will drain from whichever node holds the Primary role");
return open;
}
/// <inheritdoc />
public HistorianSinkStatus GetStatus()
{
if (_disposed) throw new ObjectDisposedException(nameof(SqliteStoreAndForwardSink));
if (_disposed) throw new ObjectDisposedException(nameof(LocalDbStoreAndForwardSink));
var queued = Interlocked.Read(ref _queuedRowCount);
if (queued < 0) queued = 0;
long deadlettered;
using (var conn = OpenConnection())
using (var conn = _db.CreateConnection())
using (var cmd = conn.CreateCommand())
{
cmd.CommandText = "SELECT COUNT(*) FROM Queue WHERE DeadLettered = 1";
cmd.CommandText = "SELECT COUNT(*) FROM alarm_sf_events WHERE dead_lettered = 1";
deadlettered = (long)(cmd.ExecuteScalar() ?? 0L);
}
@@ -510,10 +558,13 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
/// <returns>The number of rows revived from the dead-letter state.</returns>
public int RetryDeadLettered()
{
if (_disposed) throw new ObjectDisposedException(nameof(SqliteStoreAndForwardSink));
using var conn = OpenConnection();
if (_disposed) throw new ObjectDisposedException(nameof(LocalDbStoreAndForwardSink));
// Through a LocalDb connection, not a private one: the capture triggers are what make this
// revival replicate to the peer, and they only exist on the registered table.
using var conn = _db.CreateConnection();
using var cmd = conn.CreateCommand();
cmd.CommandText = "UPDATE Queue SET DeadLettered = 0, AttemptCount = 0, LastError = NULL WHERE DeadLettered = 1";
cmd.CommandText =
"UPDATE alarm_sf_events SET dead_lettered = 0, attempt_count = 0, last_error = NULL WHERE dead_lettered = 1";
var revived = cmd.ExecuteNonQuery();
// Dead-lettered rows rejoin the non-dead-lettered queue — keep the in-memory
// counter aligned.
@@ -523,29 +574,31 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
/// <summary>
/// One queued row paired with its deserialized event. <see cref="Event"/> is
/// <c>null</c> when the row's <c>PayloadJson</c> is corrupt or un-deserializable —
/// the <see cref="RowId"/> always stays bound to its own row so outcomes can
/// <c>null</c> when the row's <c>payload_json</c> is corrupt or un-deserializable —
/// the <see cref="Id"/> always stays bound to its own row so outcomes can
/// never be mapped to the wrong row.
/// </summary>
private readonly record struct QueueRow(long RowId, AlarmHistorianEvent? Event, long AttemptCount);
private readonly record struct QueueRow(string Id, AlarmHistorianEvent? Event, long AttemptCount);
private List<QueueRow> ReadBatch(SqliteConnection conn)
private async Task<List<QueueRow>> ReadBatchAsync(CancellationToken ct)
{
var rows = new List<QueueRow>();
using var cmd = conn.CreateCommand();
cmd.CommandText = """
SELECT RowId, PayloadJson, AttemptCount FROM Queue
WHERE DeadLettered = 0
ORDER BY RowId ASC
LIMIT $limit
""";
cmd.Parameters.AddWithValue("$limit", _batchSize);
using var reader = cmd.ExecuteReader();
while (reader.Read())
// Ordered by enqueue time rather than insertion order: a hashed TEXT primary key carries
// no sequence the way the legacy AUTOINCREMENT rowid did. Round-trip ("O") timestamps sort
// lexicographically in chronological order; id makes the ordering total.
var raw = await _db.QueryAsync(
"""
SELECT id, payload_json, attempt_count FROM alarm_sf_events
WHERE dead_lettered = 0
ORDER BY enqueued_at_utc ASC, id ASC
LIMIT @Limit
""",
r => (Id: r.GetString(0), Payload: r.GetString(1), AttemptCount: r.GetInt64(2)),
new { Limit = _batchSize },
ct).ConfigureAwait(false);
var rows = new List<QueueRow>(raw.Count);
foreach (var (id, payload, attemptCount) in raw)
{
var rowId = reader.GetInt64(0);
var payload = reader.GetString(1);
var attemptCount = reader.GetInt64(2);
AlarmHistorianEvent? evt;
try
{
@@ -556,77 +609,58 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
// Malformed JSON — carry a null event so the caller dead-letters this row.
evt = null;
}
rows.Add(new QueueRow(rowId, evt, attemptCount));
rows.Add(new QueueRow(id, evt, attemptCount));
}
return rows;
}
private static void DeleteRow(SqliteConnection conn, SqliteTransaction tx, long rowId)
{
using var cmd = conn.CreateCommand();
cmd.Transaction = tx;
cmd.CommandText = "DELETE FROM Queue WHERE RowId = $id";
cmd.Parameters.AddWithValue("$id", rowId);
cmd.ExecuteNonQuery();
}
private static Task DeleteRowAsync(ILocalDbTransaction tx, string id, CancellationToken ct) =>
tx.ExecuteAsync("DELETE FROM alarm_sf_events WHERE id = @Id", new { Id = id }, ct);
private void DeadLetterRow(SqliteConnection conn, SqliteTransaction tx, long rowId, string reason)
{
using var cmd = conn.CreateCommand();
cmd.Transaction = tx;
cmd.CommandText = """
UPDATE Queue SET DeadLettered = 1, LastAttemptUtc = $now, LastError = $err, AttemptCount = AttemptCount + 1
WHERE RowId = $id
""";
cmd.Parameters.AddWithValue("$now", _clock().ToString("O"));
cmd.Parameters.AddWithValue("$err", reason);
cmd.Parameters.AddWithValue("$id", rowId);
cmd.ExecuteNonQuery();
}
private Task DeadLetterRowAsync(ILocalDbTransaction tx, string id, string reason, CancellationToken ct) =>
tx.ExecuteAsync(
"""
UPDATE alarm_sf_events
SET dead_lettered = 1, last_attempt_utc = @Now, last_error = @Err,
attempt_count = attempt_count + 1
WHERE id = @Id
""",
new { Now = _clock().ToString("O"), Err = reason, Id = id },
ct);
private void BumpAttempt(SqliteConnection conn, SqliteTransaction tx, long rowId, string reason)
{
using var cmd = conn.CreateCommand();
cmd.Transaction = tx;
cmd.CommandText = """
UPDATE Queue SET LastAttemptUtc = $now, LastError = $err, AttemptCount = AttemptCount + 1
WHERE RowId = $id
""";
cmd.Parameters.AddWithValue("$now", _clock().ToString("O"));
cmd.Parameters.AddWithValue("$err", reason);
cmd.Parameters.AddWithValue("$id", rowId);
cmd.ExecuteNonQuery();
}
private Task BumpAttemptAsync(ILocalDbTransaction tx, string id, string reason, CancellationToken ct) =>
tx.ExecuteAsync(
"""
UPDATE alarm_sf_events
SET last_attempt_utc = @Now, last_error = @Err, attempt_count = attempt_count + 1
WHERE id = @Id
""",
new { Now = _clock().ToString("O"), Err = reason, Id = id },
ct);
// Async variant used by EnqueueAsync.
private async Task EnforceCapacityAsync(SqliteConnection conn, CancellationToken ct)
private async Task EnforceCapacityAsync(CancellationToken ct)
{
Interlocked.Increment(ref _capacityProbeCount);
long count;
using (var cmd = conn.CreateCommand())
{
cmd.CommandText = "SELECT COUNT(*) FROM Queue WHERE DeadLettered = 0";
count = (long)(await cmd.ExecuteScalarAsync(ct).ConfigureAwait(false) ?? 0L);
}
var count = await CountAsync("dead_lettered = 0", ct).ConfigureAwait(false);
// Resync the in-memory counter while we have a fresh number.
Interlocked.Exchange(ref _queuedRowCount, count);
if (count < _capacity) return;
var toEvict = count - _capacity + 1;
using (var cmd = conn.CreateCommand())
{
cmd.CommandText = """
DELETE FROM Queue
WHERE RowId IN (
SELECT RowId FROM Queue
WHERE DeadLettered = 0
ORDER BY RowId ASC
LIMIT $n
)
""";
cmd.Parameters.AddWithValue("$n", toEvict);
await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
}
await _db.ExecuteAsync(
"""
DELETE FROM alarm_sf_events
WHERE id IN (
SELECT id FROM alarm_sf_events
WHERE dead_lettered = 0
ORDER BY enqueued_at_utc ASC, id ASC
LIMIT @N
)
""",
new { N = toEvict },
ct).ConfigureAwait(false);
Interlocked.Add(ref _queuedRowCount, -toEvict);
long lifetimeEvicted;
lock (_statusLock) { _evictedCount += toEvict; lifetimeEvicted = _evictedCount; }
@@ -635,40 +669,21 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
_capacity, toEvict, lifetimeEvicted);
}
private void PurgeAgedDeadLetters(SqliteConnection conn)
private async Task PurgeAgedDeadLettersAsync(CancellationToken ct)
{
var cutoff = (_clock() - _deadLetterRetention).ToString("O");
using var cmd = conn.CreateCommand();
cmd.CommandText = """
DELETE FROM Queue
WHERE DeadLettered = 1 AND LastAttemptUtc IS NOT NULL AND LastAttemptUtc < $cutoff
""";
cmd.Parameters.AddWithValue("$cutoff", cutoff);
var purged = cmd.ExecuteNonQuery();
var purged = await _db.ExecuteAsync(
"""
DELETE FROM alarm_sf_events
WHERE dead_lettered = 1 AND last_attempt_utc IS NOT NULL AND last_attempt_utc < @Cutoff
""",
new { Cutoff = cutoff },
ct).ConfigureAwait(false);
if (purged > 0)
_logger.Information("Purged {Count} dead-lettered row(s) past retention window", purged);
}
private void InitializeSchema()
{
using var conn = OpenConnection();
using var cmd = conn.CreateCommand();
cmd.CommandText = """
CREATE TABLE IF NOT EXISTS Queue (
RowId INTEGER PRIMARY KEY AUTOINCREMENT,
AlarmId TEXT NOT NULL,
EnqueuedUtc TEXT NOT NULL,
PayloadJson TEXT NOT NULL,
AttemptCount INTEGER NOT NULL DEFAULT 0,
LastAttemptUtc TEXT NULL,
LastError TEXT NULL,
DeadLettered INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX IF NOT EXISTS IX_Queue_Drain ON Queue (DeadLettered, RowId);
""";
cmd.ExecuteNonQuery();
}
private void BumpBackoff() => _backoffIndex = Math.Min(_backoffIndex + 1, BackoffLadder.Length - 1);
private void ResetBackoff() => _backoffIndex = 0;
@@ -676,12 +691,16 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
public TimeSpan CurrentBackoff => BackoffLadder[_backoffIndex];
/// <summary>Disposes the sink and releases all held resources including the drain timer and the writer.</summary>
/// <remarks>
/// The <see cref="ILocalDb"/> is NOT disposed here — it is a shared singleton owned by the
/// host and used by the deployment cache too.
/// </remarks>
public void Dispose()
{
if (_disposed) return;
_disposed = true;
_drainTimer?.Dispose();
_drainGate.Dispose();
_drainGateLock.Dispose();
if (_writer is IDisposable writerDisposable) writerDisposable.Dispose();
}
}
src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian.csproj
+6
View File
@@ -20,6 +20,12 @@
-->
<PackageReference Include="SQLitePCLRaw.bundle_e_sqlite3"/>
<PackageReference Include="Serilog"/>
<!--
The store-and-forward buffer lives in the node's consolidated LocalDb file so it
replicates to the redundant pair peer. Only the ILocalDb SQL surface is used here; the
Host owns the DI registration and the replication package.
-->
<PackageReference Include="ZB.MOM.WW.LocalDb"/>
</ItemGroup>
<ItemGroup>
src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/Part9StateMachine.cs
+5
View File
@@ -377,4 +377,9 @@ public enum EmissionKind
Enabled,
Disabled,
CommentAdded,
/// <summary>#478 — the worst-of-input source quality changed bucket (Good/Uncertain/Bad) with no
/// accompanying Part 9 state transition. Delivered out of band via the dedicated
/// <c>WriteAlarmQuality</c> node path, never through the <c>IAlarmSource</c> fan-out (it is not a
/// state change and must not materialize or historize a condition).</summary>
QualityChanged,
}
src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/ScriptedAlarmEngine.cs
+130 -9
View File
@@ -62,6 +62,17 @@ public sealed class ScriptedAlarmEngine : IDisposable
private readonly ConcurrentDictionary<string, AlarmScratch> _scratchByAlarmId =
new(StringComparer.Ordinal);
/// <summary>
/// #478 — last emitted worst-of-input quality bucket per alarm (0 = Good, 1 = Uncertain,
/// 2 = Bad), computed each evaluation over the refilled read cache. A change in this bucket
/// with no accompanying Part 9 state transition drives a standalone
/// <see cref="EmissionKind.QualityChanged"/> emission (a Bad input freezes the condition — no
/// transition — so quality can't ride one, exactly like a comms-lost native driver). Only ever
/// mutated under <c>_evalGate</c>; cleared alongside <see cref="_alarms"/> on load/dispose.
/// </summary>
private readonly ConcurrentDictionary<string, int> _lastQualityBucketByAlarmId =
new(StringComparer.Ordinal);
/// <summary>
/// Compile cache for every alarm predicate. Routes <see cref="LoadAsync"/>'s
/// <see cref="ScriptEvaluator{TContext, TResult}.Compile"/> calls through the
@@ -203,6 +214,7 @@ public sealed class ScriptedAlarmEngine : IDisposable
// have changed (different Inputs, different Logger), so any reuse would be
// unsafe.
_scratchByAlarmId.Clear();
_lastQualityBucketByAlarmId.Clear();
// Dispose every compiled-predicate ALC from the prior generation BEFORE we
// recompile this one. Skipping this is what made the earlier fix a
// no-op in production.
@@ -412,7 +424,7 @@ public sealed class ScriptedAlarmEngine : IDisposable
// OnEvent dispatch until after Release() so a slow subscriber or a
// subscriber that re-enters the engine doesn't block / deadlock.
if (result.Emission != EmissionKind.None)
pending = BuildEmission(state, result.State, result.Emission);
pending = BuildEmission(state, result.State, result.Emission, LastWorstStatus(alarmId));
else if (result.NoOpReason is { } reason)
{
// The Part9StateMachine remarks promise a diagnostic log line for
@@ -513,13 +525,27 @@ public sealed class ScriptedAlarmEngine : IDisposable
_ => new AlarmScratch(state.Inputs, state.Logger, _clock));
RefillReadCache(scratch.ReadCache, state.Inputs);
// #478 — worst OPC UA quality across the alarm's inputs, computed BEFORE the readiness
// short-circuit so an outright-Bad input is still observed. A bucket change with no state
// transition is delivered out of band as a QualityChanged emission (see below).
var worstStatus = WorstInputStatus(scratch.ReadCache);
var qualityBucketChanged = TrackQualityBucket(state.Definition.AlarmId, worstStatus);
// Cold-start guard — skip the predicate when any referenced upstream tag has no
// cached value yet (the upstream subscription hasn't delivered its first push).
// Without this, predicates that cast `(double)ctx.GetTag(path).Value` throw NRE on
// every tick until the cache fills, spamming the log with identical stack traces.
// Bad quality is treated the same: the input isn't available at the predicate's
// expected type, so the only defensible move is to hold the prior condition state.
if (!AreInputsReady(scratch.ReadCache)) return seed;
if (!AreInputsReady(scratch.ReadCache))
{
// The condition is frozen (can't trust its state), but its source quality just changed
// bucket — annotate it out of band so a comms-lost / Bad-input scripted condition reports
// Bad, mirroring the native OT path.
if (qualityBucketChanged)
pendingEmissions.Add(BuildQualityEmission(state, seed, worstStatus));
return seed;
}
var context = scratch.Context;
@@ -544,10 +570,19 @@ public sealed class ScriptedAlarmEngine : IDisposable
}
var result = Part9StateMachine.ApplyPredicate(seed, predicateTrue, nowUtc);
if (result.Emission != EmissionKind.None)
var transition = result.Emission != EmissionKind.None
? BuildEmission(state, result.State, result.Emission, worstStatus)
: null;
if (transition is not null)
{
var evt = BuildEmission(state, result.State, result.Emission);
if (evt is not null) pendingEmissions.Add(evt);
// A real transition carries the current worst quality so the projected full-snapshot
// write doesn't clobber quality back to Good (e.g. a transition while an input is Uncertain).
pendingEmissions.Add(transition);
}
else if (qualityBucketChanged)
{
// No transition (or a Suppressed one) but the quality bucket moved — annotate out of band.
pendingEmissions.Add(BuildQualityEmission(state, result.State, worstStatus));
}
return result.State;
}
@@ -599,7 +634,8 @@ public sealed class ScriptedAlarmEngine : IDisposable
/// done by <see cref="FireEvent(ScriptedAlarmEvent)"/> AFTER the gate is
/// released.
/// </summary>
private ScriptedAlarmEvent? BuildEmission(AlarmState state, AlarmConditionState condition, EmissionKind kind)
private ScriptedAlarmEvent? BuildEmission(
AlarmState state, AlarmConditionState condition, EmissionKind kind, uint worstInputStatus)
{
// Suppressed kind means shelving ate the emission — we don't fire for subscribers
// but the state record still advanced so startup recovery reflects reality.
@@ -629,9 +665,89 @@ public sealed class ScriptedAlarmEngine : IDisposable
// Carry the per-alarm durable-historization opt-out through to subscribers. The historian
// adapter honors it to suppress ONLY the durable sink write; the live alerts fan-out is
// unaffected (it is not gated on this flag).
HistorizeToAveva: state.Definition.HistorizeToAveva);
HistorizeToAveva: state.Definition.HistorizeToAveva,
// #478 — the worst input quality at evaluation time rides the transition so the projected
// full snapshot keeps quality consistent (no clobber-to-Good).
WorstInputStatusCode: worstInputStatus);
}
/// <summary>
/// #478 — build a standalone <see cref="EmissionKind.QualityChanged"/> event carrying the new
/// worst-of-input quality. Emitted when the quality bucket moved but no Part 9 transition fired
/// (a Bad input freezes the condition; a Suppressed/None transition also leaves state unchanged).
/// The host routes it to the dedicated <c>WriteAlarmQuality</c> node path (annotate quality only,
/// no <c>/alerts</c> row, no historian write); the <see cref="IAlarmSource"/> fan-out skips it.
/// </summary>
private ScriptedAlarmEvent BuildQualityEmission(
AlarmState state, AlarmConditionState condition, uint worstInputStatus)
=> new(
AlarmId: state.Definition.AlarmId,
EquipmentPath: state.Definition.EquipmentPath,
AlarmName: state.Definition.AlarmName,
Kind: state.Definition.Kind,
Severity: state.Definition.Severity,
Message: MessageTemplate.Resolve(state.Definition.MessageTemplate, TryLookup),
Condition: condition,
Emission: EmissionKind.QualityChanged,
TimestampUtc: _clock(),
HistorizeToAveva: state.Definition.HistorizeToAveva,
WorstInputStatusCode: worstInputStatus);
/// <summary>Worst OPC UA StatusCode across a refilled read cache — the entry with the highest severity
/// bits (top 2). An input with no value yet (null snapshot/value — the cold-start placeholder, or a
/// not-yet-published upstream) is NOT a quality signal: it means "no data", which the
/// <see cref="AreInputsReady"/> guard already handles by holding the condition. Counting it as Bad here
/// would flash every scripted condition Bad at deploy until the first push and would flood the quality
/// path with load-time annotations, so unread inputs are skipped (contribute Good). Empty / all-unread
/// cache ⇒ Good (0).</summary>
private static uint WorstInputStatus(IReadOnlyDictionary<string, DataValueSnapshot> cache)
{
uint worst = 0u;
var worstSeverity = 0u;
foreach (var kv in cache)
{
if (kv.Value is null || kv.Value.Value is null) continue; // no data yet — not a quality signal
var status = kv.Value.StatusCode;
var severity = status >> 30;
if (severity > worstSeverity)
{
worstSeverity = severity;
worst = status;
}
}
return worst;
}
/// <summary>Update the tracked worst-quality bucket for an alarm; return true iff the 3-state bucket
/// (0 = Good, 1 = Uncertain, 2 = Bad) changed from the last observed value. Only called under
/// <c>_evalGate</c>.</summary>
private bool TrackQualityBucket(string alarmId, uint worstStatus)
{
var bucket = QualityBucket(worstStatus);
var prior = _lastQualityBucketByAlarmId.TryGetValue(alarmId, out var b) ? b : 0; // default Good
_lastQualityBucketByAlarmId[alarmId] = bucket;
return bucket != prior;
}
/// <summary>Collapse an OPC UA StatusCode's 2 severity bits (00/01/10/11) to a 3-state quality bucket
/// (0 = Good, 1 = Uncertain, 2 = Bad).</summary>
private static int QualityBucket(uint statusCode)
{
var severity = statusCode >> 30;
return severity >= 2 ? 2 : (int)severity;
}
/// <summary>#478 — a canonical worst StatusCode for an alarm's last-observed quality bucket, used by
/// the operator-command + shelving-timer emission paths (which don't re-read inputs) so an ack /
/// shelve while an input is Bad still carries Bad rather than resetting the condition to Good.</summary>
private uint LastWorstStatus(string alarmId)
=> (_lastQualityBucketByAlarmId.TryGetValue(alarmId, out var bucket) ? bucket : 0) switch
{
2 => 0x80000000u, // Bad
1 => 0x40000000u, // Uncertain
_ => 0u, // Good
};
/// <summary>
/// Invoke the <see cref="OnEvent"/> handler for a built emission. Must be
/// called OUTSIDE <c>_evalGate</c>: a slow subscriber would otherwise
@@ -708,7 +824,7 @@ public sealed class ScriptedAlarmEngine : IDisposable
_alarms[id] = state with { Condition = result.State };
if (result.Emission != EmissionKind.None)
{
var evt = BuildEmission(state, result.State, result.Emission);
var evt = BuildEmission(state, result.State, result.Emission, LastWorstStatus(id));
if (evt is not null) pending.Add(evt);
}
}
@@ -780,6 +896,7 @@ public sealed class ScriptedAlarmEngine : IDisposable
_alarms.Clear();
_alarmsReferencing.Clear();
_scratchByAlarmId.Clear();
_lastQualityBucketByAlarmId.Clear();
// Dispose every compiled-predicate ALC so the engine's shutdown actually
// releases the emitted assemblies. The drain above ensures no evaluator is
// mid-call; CompiledScriptCache.Dispose internally guards against use-after-
@@ -851,7 +968,11 @@ public sealed record ScriptedAlarmEvent(
EmissionKind Emission,
DateTime TimestampUtc,
string? Comment = null,
bool HistorizeToAveva = true);
bool HistorizeToAveva = true,
// #478 — the worst OPC UA StatusCode across the alarm's input tags at evaluation time. A raw uint
// (Core.ScriptedAlarms does not reference Commons/OpcUaQuality); the host maps it to OpcUaQuality by
// the top-2 severity bits. Default 0u == Good keeps every existing constructor call unchanged.
uint WorstInputStatusCode = 0u);
/// <summary>
/// Upstream source abstraction — intentionally identical shape to the virtual-tag
src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/ScriptedAlarmSource.cs
+5
View File
@@ -81,6 +81,11 @@ public sealed class ScriptedAlarmSource : IAlarmSource, IDisposable
{
if (_disposed) return;
// #478 — QualityChanged is a source-quality annotation, not a Part 9 state change. It is delivered
// out of band via the dedicated WriteAlarmQuality node path; surfacing it here would fabricate a
// phantom AlarmEventArgs that materializes / historizes a condition. Swallow it.
if (evt.Emission == EmissionKind.QualityChanged) return;
foreach (var sub in _subscriptions.Values)
{
if (!Matches(sub, evt)) continue;
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Browser/GalaxyDriverBrowser.cs
+27 -14
View File
@@ -5,6 +5,7 @@ using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.MxGateway.Client;
using ZB.MOM.WW.OtOpcUa.Commons.Browsing;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Browser;
@@ -34,11 +35,14 @@ public sealed class GalaxyDriverBrowser : IDriverBrowser
};
private readonly ILogger<GalaxyDriverBrowser> _logger;
private readonly ISecretResolver _secretResolver;
/// <summary>Creates a new browser. Logger defaults to <see cref="NullLogger{T}"/>.</summary>
/// <param name="secretResolver">Resolves the <c>secret:</c> arm of <c>Gateway.ApiKeySecretRef</c> (DI-injected).</param>
/// <param name="logger">Optional logger; null is allowed for unit-test construction.</param>
public GalaxyDriverBrowser(ILogger<GalaxyDriverBrowser>? logger = null)
public GalaxyDriverBrowser(ISecretResolver secretResolver, ILogger<GalaxyDriverBrowser>? logger = null)
{
_secretResolver = secretResolver ?? throw new ArgumentNullException(nameof(secretResolver));
_logger = logger ?? NullLogger<GalaxyDriverBrowser>.Instance;
}
@@ -68,7 +72,7 @@ public sealed class GalaxyDriverBrowser : IDriverBrowser
if (string.IsNullOrWhiteSpace(opts.MxAccess.ClientName))
throw new InvalidOperationException("Galaxy browser requires MxAccess.ClientName.");
var clientOpts = BuildClientOptions(opts.Gateway);
var clientOpts = await BuildClientOptionsAsync(opts.Gateway, cancellationToken).ConfigureAwait(false);
// 30s wall-clock budget for the connect phase, linked to the caller's token so
// an AdminUI cancel still wins early.
@@ -116,19 +120,28 @@ public sealed class GalaxyDriverBrowser : IDriverBrowser
/// Build the gateway client options from the form's Gateway section. Mirrors the
/// runtime driver's <c>GalaxyDriver.BuildClientOptions</c> field-for-field so the
/// gateway sees an identical option shape. The API-key reference is resolved via
/// the shared <see cref="GalaxySecretRef.ResolveApiKey"/> in Driver.Galaxy.Contracts
/// the shared <see cref="GalaxySecretRef.ResolveApiKeyAsync"/> in Driver.Galaxy.Contracts
/// (the same resolver the runtime driver uses), so browse and runtime stay in lock-step.
/// The <c>secret:</c> arm is async, so the key is resolved into a local before the
/// object initializer (you can't await inside one).
/// </summary>
private MxGatewayClientOptions BuildClientOptions(GalaxyGatewayOptions gw) => new()
private async Task<MxGatewayClientOptions> BuildClientOptionsAsync(
GalaxyGatewayOptions gw, CancellationToken cancellationToken)
{
Endpoint = new Uri(gw.Endpoint, UriKind.Absolute),
ApiKey = GalaxySecretRef.ResolveApiKey(gw.ApiKeySecretRef, _logger),
UseTls = gw.UseTls,
CaCertificatePath = gw.CaCertificatePath,
ConnectTimeout = TimeSpan.FromSeconds(gw.ConnectTimeoutSeconds),
DefaultCallTimeout = TimeSpan.FromSeconds(gw.DefaultCallTimeoutSeconds),
StreamTimeout = gw.StreamTimeoutSeconds > 0
? TimeSpan.FromSeconds(gw.StreamTimeoutSeconds)
: null,
};
var apiKey = await GalaxySecretRef
.ResolveApiKeyAsync(gw.ApiKeySecretRef, _secretResolver, _logger, cancellationToken)
.ConfigureAwait(false);
return new MxGatewayClientOptions
{
Endpoint = new Uri(gw.Endpoint, UriKind.Absolute),
ApiKey = apiKey,
UseTls = gw.UseTls,
CaCertificatePath = gw.CaCertificatePath,
ConnectTimeout = TimeSpan.FromSeconds(gw.ConnectTimeoutSeconds),
DefaultCallTimeout = TimeSpan.FromSeconds(gw.DefaultCallTimeoutSeconds),
StreamTimeout = gw.StreamTimeoutSeconds > 0
? TimeSpan.FromSeconds(gw.StreamTimeoutSeconds)
: null,
};
}
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Contracts/GalaxyDriverOptions.cs
+3 -1
View File
@@ -43,12 +43,14 @@ public sealed record GalaxyDriverOptions(
/// <summary>
/// Connection details for the MxAccess gateway. <see cref="ApiKeySecretRef"/> is
/// resolved by <see cref="GalaxySecretRef.ResolveApiKey"/> at InitializeAsync time. Four forms
/// resolved by <see cref="GalaxySecretRef.ResolveApiKeyAsync"/> at InitializeAsync time. Five forms
/// supported, in priority order:
/// <list type="bullet">
/// <item><c>env:NAME</c> — read from an environment variable (recommended for
/// production; the central config DB holds only the indirection, not the key).</item>
/// <item><c>file:PATH</c> — read from an ACL'd file outside the repo.</item>
/// <item><c>secret:NAME</c> — resolved through the shared <c>ZB.MOM.WW.Secrets</c>
/// encrypted store; fail-closed if absent (the production path).</item>
/// <item><c>dev:KEY</c> — explicit cleartext opt-in for dev rigs / parity tests;
/// no startup warning.</item>
/// <item>Anything else — treated as a literal cleartext API key for back-compat.
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Contracts/GalaxySecretRef.cs
+35 -8
View File
@@ -1,9 +1,10 @@
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
/// <summary>
/// Resolves <c>Gateway.ApiKeySecretRef</c> to the actual API-key string. Four
/// Resolves <c>Gateway.ApiKeySecretRef</c> to the actual API-key string. Five
/// forms supported, evaluated in order:
/// <list type="number">
/// <item><c>env:NAME</c> — reads <c>Environment.GetEnvironmentVariable(NAME)</c>.
@@ -15,12 +16,17 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
/// <item><c>dev:KEY</c> — explicit cleartext literal. The <c>dev:</c> prefix
/// is a deliberate opt-in signal (dev box, parity rig) so the resolver
/// doesn't emit a warning; production should never use this arm.</item>
/// <item><c>secret:NAME</c> — resolves NAME through the shared
/// <c>ZB.MOM.WW.Secrets</c> <see cref="ISecretResolver"/> (the encrypted-at-rest
/// store). Fail-closed: a <c>secret:</c> ref whose secret is absent/tombstoned
/// throws rather than falling through to the literal arm — the production path
/// that retires the cleartext <c>dev:</c>/literal-in-DB model.</item>
/// <item>Anything else — used as the literal API key for back-compat with
/// configs that pre-date this resolver. When a logger is supplied the
/// resolver emits a startup warning so an operator who accidentally
/// committed a cleartext key sees it.</item>
/// </list>
/// A future PR can swap any of these arms for a DPAPI-backed lookup without
/// A future PR can swap any of these arms for a different backing store without
/// changing the call site.
/// </summary>
/// <remarks>
@@ -31,18 +37,26 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
public static class GalaxySecretRef
{
/// <summary>
/// Resolves the supplied secret reference. When the ref falls through to the
/// back-compat literal arm (an unprefixed cleartext API key in
/// <c>DriverConfig</c> JSON) and a <paramref name="logger"/> is supplied, emits
/// a <see cref="LogLevel.Warning"/>. The <c>dev:</c> prefix is the explicit
/// opt-in path that doesn't warn.
/// Resolves the supplied secret reference. The <c>secret:NAME</c> arm resolves
/// through <paramref name="resolver"/> and is fail-closed (throws when the secret
/// is absent). When the ref falls through to the back-compat literal arm (an
/// unprefixed cleartext API key in <c>DriverConfig</c> JSON) and a
/// <paramref name="logger"/> is supplied, emits a <see cref="LogLevel.Warning"/>.
/// The <c>dev:</c> prefix is the explicit opt-in path that doesn't warn.
/// </summary>
/// <param name="secretRef">The secret reference string to resolve.</param>
/// <param name="resolver">The shared secret resolver used by the <c>secret:</c> arm.</param>
/// <param name="logger">Optional logger for warning on cleartext keys.</param>
/// <param name="ct">Cancellation token for the async <c>secret:</c> resolution.</param>
/// <returns>The resolved API-key string.</returns>
public static string ResolveApiKey(string secretRef, ILogger? logger = null)
public static async Task<string> ResolveApiKeyAsync(
string secretRef,
ISecretResolver resolver,
ILogger? logger = null,
CancellationToken ct = default)
{
ArgumentException.ThrowIfNullOrEmpty(secretRef);
ArgumentNullException.ThrowIfNull(resolver);
if (secretRef.StartsWith("env:", StringComparison.OrdinalIgnoreCase))
{
@@ -76,6 +90,19 @@ public static class GalaxySecretRef
return secretRef[4..];
}
if (secretRef.StartsWith("secret:", StringComparison.OrdinalIgnoreCase))
{
// Production path: resolve the name through the shared encrypted secret store.
// Fail-closed — an absent/tombstoned secret throws rather than falling through
// to the literal arm (which would silently treat the ref string as the key).
var name = secretRef["secret:".Length..];
var value = await resolver.GetAsync(new SecretName(name), ct).ConfigureAwait(false);
return !string.IsNullOrEmpty(value)
? value
: throw new InvalidOperationException(
$"Galaxy.Gateway.ApiKeySecretRef='{secretRef}' resolves secret '{name}', but it is absent from the store (fail-closed).");
}
// Back-compat literal arm. An unprefixed string is treated as the literal
// API key — but emit a warning so an operator who accidentally committed a
// cleartext key into DriverConfig sees it. Use the dev: prefix to suppress
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Contracts/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Contracts.csproj
+1
View File
@@ -13,5 +13,6 @@
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.Abstractions\ZB.MOM.WW.OtOpcUa.Core.Abstractions.csproj" />
<!-- Logging abstraction needed by GalaxySecretRef.ResolveApiKey's optional warning logger. -->
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" />
<PackageReference Include="ZB.MOM.WW.Secrets.Abstractions" />
</ItemGroup>
</Project>
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/GalaxyDriver.cs
+57 -27
View File
@@ -8,6 +8,7 @@ using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Browse;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Health;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy;
@@ -43,6 +44,12 @@ public sealed class GalaxyDriver
private readonly GalaxyDriverOptions _options;
private readonly ILogger<GalaxyDriver> _logger;
// Resolves the Gateway.ApiKeySecretRef secret: arm through the shared encrypted store.
// Injected via ctor (the production factory pulls it from DI). The internal test ctor
// defaults it to a null-object resolver so the 45 seam-injecting test call sites keep
// compiling; those tests never use a secret: ref (they inject seams or use env/file/literal).
private readonly ISecretResolver _secretResolver;
// PR 4.1 — IGalaxyHierarchySource is the test seam for browse. When null, the driver
// lazily builds a GatewayGalaxyHierarchySource around a GalaxyRepositoryClient on
// first DiscoverAsync. Tests inject a fake source via the internal ctor to exercise
@@ -147,14 +154,17 @@ public sealed class GalaxyDriver
/// <summary>Initializes a new instance of the <see cref="GalaxyDriver"/> class.</summary>
/// <param name="driverInstanceId">The unique identifier for this driver instance.</param>
/// <param name="options">The Galaxy driver configuration options.</param>
/// <param name="secretResolver">Resolves the <c>secret:</c> arm of <c>Gateway.ApiKeySecretRef</c>.</param>
/// <param name="logger">Optional logger instance for diagnostics.</param>
public GalaxyDriver(
string driverInstanceId,
GalaxyDriverOptions options,
ISecretResolver secretResolver,
ILogger<GalaxyDriver>? logger = null)
: this(driverInstanceId, options,
hierarchySource: null, dataReader: null, dataWriter: null, subscriber: null,
alarmAcknowledger: null, alarmFeed: null, logger)
alarmAcknowledger: null, alarmFeed: null, logger,
secretResolver: secretResolver ?? throw new ArgumentNullException(nameof(secretResolver)))
{
}
@@ -173,6 +183,11 @@ public sealed class GalaxyDriver
/// <param name="alarmAcknowledger">Optional custom alarm acknowledger for testing.</param>
/// <param name="alarmFeed">Optional custom alarm feed for testing.</param>
/// <param name="logger">Optional logger instance for diagnostics.</param>
/// <param name="secretResolver">
/// Optional secret resolver for the <c>secret:</c> API-key arm. Defaults to a
/// null-object resolver (returns null for every name) so seam-injecting tests that
/// don't exercise a <c>secret:</c> ref need not supply one.
/// </param>
internal GalaxyDriver(
string driverInstanceId,
GalaxyDriverOptions options,
@@ -182,13 +197,15 @@ public sealed class GalaxyDriver
IGalaxySubscriber? subscriber = null,
IGalaxyAlarmAcknowledger? alarmAcknowledger = null,
IGalaxyAlarmFeed? alarmFeed = null,
ILogger<GalaxyDriver>? logger = null)
ILogger<GalaxyDriver>? logger = null,
ISecretResolver? secretResolver = null)
{
_driverInstanceId = !string.IsNullOrWhiteSpace(driverInstanceId)
? driverInstanceId
: throw new ArgumentException("Driver instance id required.", nameof(driverInstanceId));
_options = options ?? throw new ArgumentNullException(nameof(options));
_logger = logger ?? NullLogger<GalaxyDriver>.Instance;
_secretResolver = secretResolver ?? NullSecretResolver.Instance;
_hierarchySource = hierarchySource;
_dataReader = dataReader;
_dataWriter = dataWriter;
@@ -316,7 +333,7 @@ public sealed class GalaxyDriver
_driverInstanceId);
}
StartDeployWatcher();
await StartDeployWatcherAsync(cancellationToken).ConfigureAwait(false);
_logger.LogInformation(
"GalaxyDriver {InstanceId} initialized — endpoint={Endpoint} clientName={ClientName}",
_driverInstanceId, _options.Gateway.Endpoint, _options.MxAccess.ClientName);
@@ -331,7 +348,7 @@ public sealed class GalaxyDriver
/// </summary>
private async Task BuildProductionRuntimeAsync(CancellationToken cancellationToken)
{
var clientOptions = BuildClientOptions(_options.Gateway);
var clientOptions = await BuildClientOptionsAsync(_options.Gateway, cancellationToken).ConfigureAwait(false);
_ownedMxClient = MxGatewayClient.Create(clientOptions);
_ownedMxSession = new GalaxyMxSession(_options.MxAccess, _logger);
await _ownedMxSession.ConnectAsync(clientOptions, cancellationToken).ConfigureAwait(false);
@@ -386,7 +403,7 @@ public sealed class GalaxyDriver
private async Task ReopenAsync(CancellationToken cancellationToken)
{
if (_ownedMxSession is null) return;
var clientOptions = BuildClientOptions(_options.Gateway);
var clientOptions = await BuildClientOptionsAsync(_options.Gateway, cancellationToken).ConfigureAwait(false);
await _ownedMxSession.RecreateAsync(clientOptions, cancellationToken).ConfigureAwait(false);
// The recreated session invalidates every prior gw item handle; drop the writer's handle/advise
// caches so the next write re-AddItems + re-AdviseSupervisory against the fresh session.
@@ -527,32 +544,42 @@ public sealed class GalaxyDriver
}
}
private MxGatewayClientOptions BuildClientOptions(GalaxyGatewayOptions gw) => new()
private async Task<MxGatewayClientOptions> BuildClientOptionsAsync(
GalaxyGatewayOptions gw, CancellationToken cancellationToken)
{
Endpoint = new Uri(gw.Endpoint, UriKind.Absolute),
// Pass the logger so the literal-arm cleartext fallback surfaces a startup
// warning rather than silently shipping the key. The resolver lives in
// Driver.Galaxy.Contracts (GalaxySecretRef) so the runtime driver and the
// AdminUI browser share one implementation.
ApiKey = GalaxySecretRef.ResolveApiKey(gw.ApiKeySecretRef, _logger),
UseTls = gw.UseTls,
CaCertificatePath = gw.CaCertificatePath,
ConnectTimeout = TimeSpan.FromSeconds(gw.ConnectTimeoutSeconds),
DefaultCallTimeout = TimeSpan.FromSeconds(gw.DefaultCallTimeoutSeconds),
StreamTimeout = gw.StreamTimeoutSeconds > 0 ? TimeSpan.FromSeconds(gw.StreamTimeoutSeconds) : null,
};
// Resolve the API-key ref BEFORE the object initializer — the secret: arm is
// async and you can't await inside an initializer. Pass the logger so the
// literal-arm cleartext fallback surfaces a startup warning rather than
// silently shipping the key. The resolver lives in Driver.Galaxy.Contracts
// (GalaxySecretRef) so the runtime driver and the AdminUI browser share one
// implementation; the secret: arm resolves through the shared ISecretResolver.
var apiKey = await GalaxySecretRef
.ResolveApiKeyAsync(gw.ApiKeySecretRef, _secretResolver, _logger, cancellationToken)
.ConfigureAwait(false);
return new MxGatewayClientOptions
{
Endpoint = new Uri(gw.Endpoint, UriKind.Absolute),
ApiKey = apiKey,
UseTls = gw.UseTls,
CaCertificatePath = gw.CaCertificatePath,
ConnectTimeout = TimeSpan.FromSeconds(gw.ConnectTimeoutSeconds),
DefaultCallTimeout = TimeSpan.FromSeconds(gw.DefaultCallTimeoutSeconds),
StreamTimeout = gw.StreamTimeoutSeconds > 0 ? TimeSpan.FromSeconds(gw.StreamTimeoutSeconds) : null,
};
}
private void StartDeployWatcher()
private async Task StartDeployWatcherAsync(CancellationToken cancellationToken)
{
if (!_options.Repository.WatchDeployEvents) return;
if (_ownedRepositoryClient is null && _hierarchySource is null) return;
// Reuse the lazily-built repository client (DiscoverAsync constructs it on demand).
// If discovery hasn't run yet, build the client here so the watcher has a target.
// Guard with ??= so if BuildDefaultHierarchySource later runs it reuses this client
// rather than overwriting the field and leaking the first instance.
// Build under a null-check (not ??=) so if BuildDefaultHierarchySourceAsync later
// runs it reuses this client rather than overwriting the field and leaking the
// first instance — the client-options build is now async (secret: arm).
_ownedRepositoryClient ??= ZB.MOM.WW.MxGateway.Client.GalaxyRepositoryClient.Create(
BuildClientOptions(_options.Gateway));
await BuildClientOptionsAsync(_options.Gateway, cancellationToken).ConfigureAwait(false));
var source = new GatewayGalaxyDeployWatchSource(_ownedRepositoryClient);
_deployWatcher = new DeployWatcher(source, _logger);
@@ -691,7 +718,8 @@ public sealed class GalaxyDriver
// After discovery, SyncPlatformsAsync refreshes the probe watcher's membership so
// newly added $WinPlatform / $AppEngine objects start advising their ScanState attribute.
var capturingBuilder = new SecurityCapturingBuilder(builder, _securityByFullRef);
var source = _hierarchySource ??= BuildDefaultHierarchySource();
var source = _hierarchySource ??=
await BuildDefaultHierarchySourceAsync(cancellationToken).ConfigureAwait(false);
// Thread the v3 reverse resolver so the discoverer emits each authored attribute's RawPath as
// its node reference (FullName) — the security map SecurityCapturingBuilder captures is then
// keyed by RawPath, matching what WriteAsync resolves against.
@@ -1292,12 +1320,14 @@ public sealed class GalaxyDriver
/// <see cref="Dispose"/>. Tests bypass this by injecting their own source via the
/// internal ctor.
/// </summary>
private IGalaxyHierarchySource BuildDefaultHierarchySource()
private async Task<IGalaxyHierarchySource> BuildDefaultHierarchySourceAsync(CancellationToken cancellationToken)
{
// Reuse a client that StartDeployWatcher may have already created (??=) rather
// Reuse a client that StartDeployWatcherAsync may have already created (??=) rather
// than always overwriting the field and leaking the first instance. Both paths
// produce equivalent clients from the same options.
_ownedRepositoryClient ??= GalaxyRepositoryClient.Create(BuildClientOptions(_options.Gateway));
// produce equivalent clients from the same options. The client-options build is
// async now (secret: arm resolves through the shared ISecretResolver).
_ownedRepositoryClient ??= GalaxyRepositoryClient.Create(
await BuildClientOptionsAsync(_options.Gateway, cancellationToken).ConfigureAwait(false));
return new TracedGalaxyHierarchySource(
new GatewayGalaxyHierarchySource(_ownedRepositoryClient), _options.MxAccess.ClientName);
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/GalaxyDriverFactoryExtensions.cs
+24 -8
View File
@@ -4,6 +4,7 @@ using Microsoft.Extensions.Logging;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Hosting;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy;
@@ -23,30 +24,45 @@ public static class GalaxyDriverFactoryExtensions
{
public const string DriverTypeName = "GalaxyMxGateway";
/// <summary>Registers the Galaxy driver factory with the given registry and optional logger factory.</summary>
/// <summary>Registers the Galaxy driver factory with the given registry, secret resolver, and optional logger factory.</summary>
/// <param name="registry">The driver factory registry to register with.</param>
/// <param name="secretResolver">
/// The shared secret resolver (from DI) used to resolve the <c>secret:</c> arm of
/// each instance's <c>Gateway.ApiKeySecretRef</c>.
/// </param>
/// <param name="loggerFactory">The optional logger factory for creating drivers.</param>
public static void Register(DriverFactoryRegistry registry, ILoggerFactory? loggerFactory = null)
public static void Register(
DriverFactoryRegistry registry,
ISecretResolver secretResolver,
ILoggerFactory? loggerFactory = null)
{
ArgumentNullException.ThrowIfNull(registry);
registry.Register(DriverTypeName, (id, json) => CreateInstance(id, json, loggerFactory));
ArgumentNullException.ThrowIfNull(secretResolver);
registry.Register(DriverTypeName, (id, json) => CreateInstance(id, json, loggerFactory, secretResolver));
}
/// <summary>Convenience for tests + standalone callers.</summary>
/// <summary>
/// Convenience for tests + standalone callers. Uses the <see cref="NullSecretResolver"/>
/// null-object, so the <c>secret:</c> API-key arm resolves fail-closed (absent) — callers
/// that need a working <c>secret:</c> ref must use the <see cref="Register"/> path that
/// threads the DI resolver.
/// </summary>
/// <param name="driverInstanceId">The unique identifier for the driver instance.</param>
/// <param name="driverConfigJson">The driver configuration in JSON format.</param>
/// <returns>The constructed Galaxy driver instance.</returns>
public static GalaxyDriver CreateInstance(string driverInstanceId, string driverConfigJson)
=> CreateInstance(driverInstanceId, driverConfigJson, loggerFactory: null);
=> CreateInstance(driverInstanceId, driverConfigJson, loggerFactory: null, NullSecretResolver.Instance);
/// <summary>Creates a Galaxy driver instance from configuration JSON with optional logger factory.</summary>
/// <summary>Creates a Galaxy driver instance from configuration JSON with optional logger factory and a secret resolver.</summary>
/// <param name="driverInstanceId">The unique identifier for the driver instance.</param>
/// <param name="driverConfigJson">The driver configuration in JSON format.</param>
/// <param name="loggerFactory">The optional logger factory for creating drivers.</param>
/// <param name="secretResolver">The secret resolver for the driver's <c>secret:</c> API-key arm.</param>
/// <returns>The constructed Galaxy driver instance.</returns>
public static GalaxyDriver CreateInstance(
string driverInstanceId, string driverConfigJson, ILoggerFactory? loggerFactory)
string driverInstanceId, string driverConfigJson, ILoggerFactory? loggerFactory, ISecretResolver secretResolver)
{
ArgumentNullException.ThrowIfNull(secretResolver);
ArgumentException.ThrowIfNullOrWhiteSpace(driverInstanceId);
ArgumentException.ThrowIfNullOrWhiteSpace(driverConfigJson);
@@ -88,7 +104,7 @@ public static class GalaxyDriverFactoryExtensions
RawTags = dto.RawTags ?? [],
};
return new GalaxyDriver(driverInstanceId, options, loggerFactory?.CreateLogger<GalaxyDriver>());
return new GalaxyDriver(driverInstanceId, options, secretResolver, loggerFactory?.CreateLogger<GalaxyDriver>());
}
private static readonly JsonSerializerOptions JsonOptions = new()
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/NullSecretResolver.cs
+26
View File
@@ -0,0 +1,26 @@
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy;
/// <summary>
/// A null-object <see cref="ISecretResolver"/> that resolves every name to <c>null</c>
/// (absent). Used as the default for the internal test ctor and the parse-only
/// <c>CreateInstance</c> path so callers that never exercise a <c>secret:</c> API-key
/// ref need not thread a real resolver. A production <c>GalaxyDriver</c> always receives
/// the DI-registered resolver via the factory; a <c>secret:</c> ref resolved against this
/// null object throws fail-closed (the secret is reported absent), which is the correct
/// behaviour for a mis-wired deployment.
/// </summary>
internal sealed class NullSecretResolver : ISecretResolver
{
/// <summary>The shared singleton instance.</summary>
public static readonly NullSecretResolver Instance = new();
private NullSecretResolver()
{
}
/// <inheritdoc />
public Task<string?> GetAsync(SecretName name, CancellationToken ct) =>
Task.FromResult<string?>(null);
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/GatewayAlarmHistorianWriter.cs
+1 -1
View File
@@ -9,7 +9,7 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway;
/// <summary>
/// <see cref="IAlarmHistorianWriter"/> backed by the HistorianGateway <c>SendEvent</c> path. The
/// drain worker behind <c>SqliteStoreAndForwardSink</c> calls
/// drain worker behind <c>LocalDbStoreAndForwardSink</c> calls
/// <see cref="WriteBatchAsync"/> and uses the returned per-event
/// <see cref="HistorianWriteOutcome"/> to decide retry vs. dead-letter, so this writer maps every
/// gateway result — success ack, the published client's typed exception hierarchy, raw
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/GatewayHistorianServiceCollectionExtensions.cs
+1 -1
View File
@@ -47,7 +47,7 @@ public static class GatewayHistorian
/// <see cref="ServerHistorianOptions"/> — the <b>same single gateway</b> the read path
/// (<see cref="CreateDataSource"/>) targets. The Host's <c>AddAlarmHistorian</c> wiring supplies
/// this as the concrete <see cref="IAlarmHistorianWriter"/> the durable
/// <c>SqliteStoreAndForwardSink</c> drain worker delegates to, sourcing the connection from the
/// <c>LocalDbStoreAndForwardSink</c> drain worker delegates to, sourcing the connection from the
/// <c>ServerHistorian</c> section (endpoint/key/TLS) rather than the legacy Wonderware-shaped
/// <c>AlarmHistorian</c> host/port. Resolves an <see cref="ILoggerFactory"/> and the writer's
/// <see cref="ILogger{TCategoryName}"/> from <paramref name="services"/>, falling back to the null
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/HistorianGatewayClientAdapter.cs
+12 -5
View File
@@ -57,16 +57,23 @@ public sealed class HistorianGatewayClientAdapter : IHistorianGatewayClient, IDi
$"ServerHistorian:Endpoint must be an absolute http(s) URI (e.g. https://host:5222); got '{options.Endpoint}'.");
}
// TLS-only options must stay at their defaults on a plaintext h2c connection: the client rejects
// each of them outright when UseTls=false ("<name> is a TLS-only option and requires
// UseTls=true"). Forwarding them unconditionally made h2c UNREACHABLE — AllowUntrustedServer-
// Certificate defaults to false, so RequireCertificateValidation was always sent as true and
// every http:// deployment crashed at startup, even though the scheme is documented as the
// supported way to select h2c. There is no certificate to have a posture about here.
var clientOptions = new HistorianGatewayClientOptions
{
Endpoint = endpointUri,
ApiKey = options.ApiKey,
UseTls = options.UseTls,
CaCertificatePath = options.CaCertificatePath,
// INVERTED mapping: ServerHistorianOptions.AllowUntrustedServerCertificate (opt-in to accept
// a self-signed cert) is the negation of the client's RequireCertificateValidation. Allowing
// an untrusted cert == not requiring validation; a pinned CaCertificatePath always verifies.
RequireCertificateValidation = !options.AllowUntrustedServerCertificate,
CaCertificatePath = options.UseTls ? options.CaCertificatePath : null,
// INVERTED mapping (TLS only): ServerHistorianOptions.AllowUntrustedServerCertificate
// (opt-in to accept a self-signed cert) is the negation of the client's
// RequireCertificateValidation. Allowing an untrusted cert == not requiring validation; a
// pinned CaCertificatePath always verifies.
RequireCertificateValidation = options.UseTls && !options.AllowUntrustedServerCertificate,
DefaultCallTimeout = options.CallTimeout,
LoggerFactory = loggerFactory,
};
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Contracts/OpcUaClientDriverOptions.cs
+3 -1
View File
@@ -14,7 +14,9 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient;
/// exposes PLC data" flow. Tier A (pure managed, OPC Foundation reference SDK); universal
/// protections cover it.
/// </remarks>
public sealed class OpcUaClientDriverOptions
// A record (not a plain class) so G-2b's OpcUaClientSecretResolution can produce a
// credential-resolved copy with a `with` expression — every property keeps its init setter.
public sealed record OpcUaClientDriverOptions
{
/// <summary>
/// Remote OPC UA endpoint URL, e.g. <c>opc.tcp://plc.internal:4840</c>. Convenience
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/NullSecretResolver.cs
+26
View File
@@ -0,0 +1,26 @@
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient;
/// <summary>
/// A null-object <see cref="ISecretResolver"/> that resolves every name to <c>null</c>
/// (absent). Backs the driver's default (test/parse-only) construction path so callers that
/// never author a <c>secret:</c> credential ref need not thread a real resolver. A production
/// <c>OpcUaClientDriver</c> always receives the DI-registered resolver via the factory; a
/// <c>secret:</c> ref resolved against this null object throws fail-closed (the secret is
/// reported absent), which is the correct behaviour for a mis-wired deployment — the
/// <c>secret:</c> literal is never sent verbatim as a password.
/// </summary>
internal sealed class NullSecretResolver : ISecretResolver
{
/// <summary>The shared singleton instance.</summary>
public static readonly NullSecretResolver Instance = new();
private NullSecretResolver()
{
}
/// <inheritdoc />
public Task<string?> GetAsync(SecretName name, CancellationToken ct) =>
Task.FromResult<string?>(null);
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/OpcUaClientDriver.cs
+24 -2
View File
@@ -4,6 +4,7 @@ using Opc.Ua;
using Opc.Ua.Client;
using Opc.Ua.Configuration;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient;
@@ -36,15 +37,25 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
/// <param name="options">Driver configuration.</param>
/// <param name="driverInstanceId">Stable logical ID from the config DB.</param>
/// <param name="logger">Optional logger; defaults to NullLogger when not supplied.</param>
/// <param name="secretResolver">
/// Optional shared secret resolver used to resolve <c>secret:</c>-prefixed
/// <see cref="OpcUaClientDriverOptions.Password"/> /
/// <see cref="OpcUaClientDriverOptions.UserCertificatePassword"/> at session-open.
/// Defaults to the <see cref="NullSecretResolver"/> null-object (which reports every
/// secret absent) for the test/parse-only path; the production factory threads the
/// DI-registered resolver. A <c>secret:</c> ref against the null-object fails closed.
/// </param>
public OpcUaClientDriver(OpcUaClientDriverOptions options, string driverInstanceId,
ILogger<OpcUaClientDriver>? logger = null)
ILogger<OpcUaClientDriver>? logger = null, ISecretResolver? secretResolver = null)
{
_options = options;
_driverInstanceId = driverInstanceId;
_logger = logger ?? NullLogger<OpcUaClientDriver>.Instance;
_secretResolver = secretResolver ?? NullSecretResolver.Instance;
}
private readonly OpcUaClientDriverOptions _options;
private readonly ISecretResolver _secretResolver;
private readonly string _driverInstanceId;
// ---- IAlarmSource state ----
@@ -159,7 +170,18 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
var appConfig = await BuildApplicationConfigurationAsync(cancellationToken).ConfigureAwait(false);
var candidates = ResolveEndpointCandidates(_options);
var identity = BuildUserIdentity(_options);
// G-2b: resolve any secret:-prefixed Password / UserCertificatePassword through the
// shared secret store ONCE, here in the async connect flow, just before the credentials
// are consumed. _options stays the raw config (with secret: refs) — only this
// connect-scoped local carries plaintext, and only for the duration of the connect.
// Resolving on every InitializeAsync (a full reconnect calls it) picks up rotations.
// Both credential consumers — the Username password below and the Certificate password
// in BuildCertificateIdentity — flow through BuildUserIdentity(resolvedOptions), so a
// single resolve covers both paths.
var resolvedOptions = await OpcUaClientSecretResolution
.ResolveSecretRefsAsync(_options, _secretResolver, cancellationToken).ConfigureAwait(false);
var identity = BuildUserIdentity(resolvedOptions);
// Failover sweep: try each endpoint in order, return the session from the first
// one that successfully connects. Per-endpoint failures are captured so the final
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/OpcUaClientDriverFactoryExtensions.cs
+23 -5
View File
@@ -2,6 +2,7 @@ using System.Text.Json;
using System.Text.Json.Serialization;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.OtOpcUa.Core.Hosting;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient;
@@ -31,22 +32,37 @@ public static class OpcUaClientDriverFactoryExtensions
Converters = { new JsonStringEnumConverter() },
};
/// <summary>Register the OpcUaClient factory with the driver registry.</summary>
/// <summary>Register the OpcUaClient factory with the driver registry, threading the shared secret resolver.</summary>
/// <param name="registry">The driver factory registry to register with.</param>
/// <param name="loggerFactory">Optional logger factory used to create per-instance loggers.</param>
public static void Register(DriverFactoryRegistry registry, ILoggerFactory? loggerFactory = null)
/// <param name="secretResolver">
/// The shared secret resolver (from DI) used to resolve the <c>secret:</c> arm of each
/// instance's <c>Password</c> / <c>UserCertificatePassword</c> at session-open. Defaults to
/// the <see cref="NullSecretResolver"/> null-object for the test/standalone path — callers
/// that need a working <c>secret:</c> credential ref must thread the DI resolver.
/// </param>
public static void Register(
DriverFactoryRegistry registry,
ILoggerFactory? loggerFactory = null,
ISecretResolver? secretResolver = null)
{
ArgumentNullException.ThrowIfNull(registry);
registry.Register(DriverTypeName, (id, json) => CreateInstance(id, json, loggerFactory));
var resolver = secretResolver ?? NullSecretResolver.Instance;
registry.Register(DriverTypeName, (id, json) => CreateInstance(id, json, loggerFactory, resolver));
}
/// <summary>Public for the Server-side bootstrapper + test consumers.</summary>
/// <param name="driverInstanceId">The unique identifier for the driver instance.</param>
/// <param name="driverConfigJson">The JSON configuration string for the driver.</param>
/// <param name="loggerFactory">Optional logger factory for the per-instance logger.</param>
/// <param name="secretResolver">
/// Optional shared secret resolver for the driver's <c>secret:</c> credential arm; defaults
/// to the <see cref="NullSecretResolver"/> null-object (fail-closed on a <c>secret:</c> ref).
/// </param>
/// <returns>A configured <see cref="OpcUaClientDriver"/>.</returns>
public static OpcUaClientDriver CreateInstance(
string driverInstanceId, string driverConfigJson, ILoggerFactory? loggerFactory = null)
string driverInstanceId, string driverConfigJson, ILoggerFactory? loggerFactory = null,
ISecretResolver? secretResolver = null)
{
ArgumentException.ThrowIfNullOrWhiteSpace(driverInstanceId);
ArgumentException.ThrowIfNullOrWhiteSpace(driverConfigJson);
@@ -55,6 +71,8 @@ public static class OpcUaClientDriverFactoryExtensions
?? throw new InvalidOperationException(
$"OpcUaClient driver config for '{driverInstanceId}' deserialised to null");
return new OpcUaClientDriver(options, driverInstanceId, loggerFactory?.CreateLogger<OpcUaClientDriver>());
return new OpcUaClientDriver(
options, driverInstanceId, loggerFactory?.CreateLogger<OpcUaClientDriver>(),
secretResolver ?? NullSecretResolver.Instance);
}
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/OpcUaClientDriverProbe.cs
+6
View File
@@ -34,6 +34,12 @@ public sealed class OpcUaClientDriverProbe : IDriverProbe
/// <inheritdoc />
public async Task<DriverProbeResult> ProbeAsync(string configJson, TimeSpan timeout, CancellationToken ct)
{
// G-2b note: the probe does an UNAUTHENTICATED GetEndpoints discovery preflight
// (BuildMinimalAppConfig sets no user identity) and never reads opts.Password /
// opts.UserCertificatePassword. Those credentials — including any secret: refs — are
// therefore intentionally NOT resolved here; resolving them would be dead code. Secret
// resolution happens lazily in OpcUaClientDriver's async session-open path, where the
// credentials are actually consumed.
OpcUaClientDriverOptions? opts;
try { opts = JsonSerializer.Deserialize<OpcUaClientDriverOptions>(configJson, _opts); }
catch (Exception ex) { return new(false, $"Config JSON is invalid: {ex.Message}", null); }
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/OpcUaClientSecretResolution.cs
+90
View File
@@ -0,0 +1,90 @@
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient;
/// <summary>
/// Layer-B (G-2b) secret resolution for the OPC UA Client driver's credential fields. Resolves
/// <c>secret:</c>-prefixed <see cref="OpcUaClientDriverOptions.Password"/> and
/// <see cref="OpcUaClientDriverOptions.UserCertificatePassword"/> through the shared
/// <see cref="ISecretResolver"/> (the encrypted-at-rest store) just before the async
/// session-open path consumes them — retiring the cleartext password-in-DB model for
/// production.
/// </summary>
/// <remarks>
/// Resolution is lazy (called from the connect flow, not at config deserialization) so a
/// full reconnect via <c>InitializeAsync</c> re-resolves and picks up secret rotations,
/// mirroring the Galaxy driver's re-resolve-on-reconnect behaviour. Only <c>secret:</c>-prefixed
/// values are resolved; a null/empty or non-<c>secret:</c> value passes through verbatim so the
/// literal-password back-compat path is preserved. The <c>secret:</c> arm is <b>fail-closed</b>:
/// an absent/tombstoned secret throws rather than leaving the <c>secret:</c> literal in place,
/// which would otherwise be sent verbatim to the remote server as the password.
/// </remarks>
internal static class OpcUaClientSecretResolution
{
private const string SecretPrefix = "secret:";
/// <summary>
/// Return a copy of <paramref name="options"/> with any <c>secret:</c>-prefixed
/// <see cref="OpcUaClientDriverOptions.Password"/> /
/// <see cref="OpcUaClientDriverOptions.UserCertificatePassword"/> resolved to their
/// plaintext via <paramref name="resolver"/>. Non-secret / null / empty fields are
/// returned unchanged.
/// </summary>
/// <param name="options">The raw driver options (credential fields may carry <c>secret:</c> refs).</param>
/// <param name="resolver">The shared secret resolver used by the <c>secret:</c> arm.</param>
/// <param name="ct">Cancellation token for the async secret resolution.</param>
/// <returns>An options copy whose credential fields carry resolved plaintext.</returns>
/// <exception cref="InvalidOperationException">
/// A <c>secret:</c> ref names a secret that is absent/tombstoned in the store (fail-closed).
/// </exception>
internal static async Task<OpcUaClientDriverOptions> ResolveSecretRefsAsync(
OpcUaClientDriverOptions options, ISecretResolver resolver, CancellationToken ct)
{
ArgumentNullException.ThrowIfNull(options);
ArgumentNullException.ThrowIfNull(resolver);
var password = await ResolveFieldAsync(options.Password, nameof(options.Password), resolver, ct)
.ConfigureAwait(false);
var certPassword = await ResolveFieldAsync(
options.UserCertificatePassword, nameof(options.UserCertificatePassword), resolver, ct)
.ConfigureAwait(false);
// Only re-materialize when something actually changed — a `with` on the reference-equal
// strings is harmless, but skipping it keeps the common (no-secret) case allocation-free.
if (ReferenceEquals(password, options.Password)
&& ReferenceEquals(certPassword, options.UserCertificatePassword))
{
return options;
}
return options with { Password = password, UserCertificatePassword = certPassword };
}
/// <summary>
/// Resolve a single credential field. Null/empty or non-<c>secret:</c> values pass through
/// unchanged (the reference-equal original is returned). A <c>secret:NAME</c> value is
/// resolved through <paramref name="resolver"/> and is fail-closed when the secret is absent.
/// </summary>
/// <param name="value">The raw field value (may be a <c>secret:</c> ref).</param>
/// <param name="fieldName">The field name, used in the fail-closed exception message.</param>
/// <param name="resolver">The shared secret resolver.</param>
/// <param name="ct">Cancellation token for the async resolution.</param>
/// <returns>The resolved plaintext, or the original value when it is not a <c>secret:</c> ref.</returns>
private static async Task<string?> ResolveFieldAsync(
string? value, string fieldName, ISecretResolver resolver, CancellationToken ct)
{
if (string.IsNullOrEmpty(value)
|| !value.StartsWith(SecretPrefix, StringComparison.OrdinalIgnoreCase))
{
return value;
}
var name = value[SecretPrefix.Length..];
var resolved = await resolver.GetAsync(new SecretName(name), ct).ConfigureAwait(false);
return !string.IsNullOrEmpty(resolved)
? resolved
: throw new InvalidOperationException(
$"OpcUaClientDriverOptions.{fieldName}='{value}' resolves secret '{name}', but it is " +
"absent from the store (fail-closed).");
}
}
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.csproj
+1
View File
@@ -21,6 +21,7 @@
<ItemGroup>
<PackageReference Include="OPCFoundation.NetStandard.Opc.Ua.Client"/>
<PackageReference Include="OPCFoundation.NetStandard.Opc.Ua.Configuration"/>
<PackageReference Include="ZB.MOM.WW.Secrets.Abstractions"/>
</ItemGroup>
<ItemGroup>
src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/App.razor
+5 -1
View File
@@ -19,7 +19,11 @@
<HeadOutlet/>
</head>
<body>
<Routes/>
@* No AdditionalAssemblies: the Secrets.Ui RCL's routable page is NOT routed directly in this
host — Pages/Secrets.razor wraps it so the route can carry this host's per-page
@rendermode (lmxopcua#483); registering the RCL routes too would make /admin/secrets
ambiguous. *@
<Routes />
<script src="_content/ZB.MOM.WW.OtOpcUa.AdminUI/lib/bootstrap/js/bootstrap.bundle.min.js"></script>
<ThemeScripts />
<script src="_content/ZB.MOM.WW.OtOpcUa.AdminUI/js/monaco-init.js"></script>
src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Layout/MainLayout.razor
+1
View File
@@ -20,6 +20,7 @@
<NavRailItem Href="/reservations" Text="Reservations" />
<NavRailItem Href="/certificates" Text="Certificates" />
<NavRailItem Href="/role-grants" Text="Role grants" />
<NavRailItem Href="/admin/secrets" Text="Secrets" />
</NavRailSection>
<NavRailSection Title="Scripting" Key="scripting">
<NavRailItem Href="/scripts" Text="Scripts" />
src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/AlarmsHistorian.razor
+3 -2
View File
@@ -18,8 +18,9 @@
<section class="panel notice rise" style="animation-delay:.02s">
Snapshot from the local node's <span class="mono">HistorianAdapterActor</span>. Default sink
is a no-op (<span class="mono">NullAlarmHistorianSink</span>); production wires
<span class="mono">SqliteStoreAndForwardSink</span> draining to the HistorianGateway
(<span class="mono">SendEvent</span>) behind it. Polling every @PollSeconds s.
<span class="mono">LocalDbStoreAndForwardSink</span> — buffering into this node's LocalDb, and
draining to the HistorianGateway (<span class="mono">SendEvent</span>) only while this node holds
the Primary role. Polling every @PollSeconds s.
</section>
@if (_status is null)
src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/SecretsAdmin.razor
+20
View File
@@ -0,0 +1,20 @@
@page "/admin/secrets"
@attribute [Authorize(Policy = ZB.MOM.WW.Secrets.Ui.SecretsAuthorization.ManagePolicy)]
@rendermode RenderMode.InteractiveServer
@using ZB.MOM.WW.Secrets.Ui
@* Host-side wrapper for the ZB.MOM.WW.Secrets.Ui secrets page (lmxopcua#483).
This host's <Routes/> is deliberately static (cookie SignInAsync needs SSR), so every
interactive page opts in per-page with @rendermode — a directive the RCL's own routable
SecretsPage cannot carry, because the other three family hosts render it under a globally
interactive router where a nested render mode throws. Routing straight to the RCL page
therefore produced a statically-rendered, dead page here: no circuit, @onclick inert.
So this wrapper owns the /admin/secrets route in THIS host (the RCL assembly is no longer
passed to either AdditionalAssemblies registration — doing both would make the route
ambiguous), carries the standard per-page render mode, re-states the RCL page's own
authorization policy (the router only enforces [Authorize] on the routed page itself,
which is now this one), and renders the RCL page as an ordinary child component. *@
<SecretsPage />
src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/EndpointRouteBuilderExtensions.cs
+17
View File
@@ -1,3 +1,4 @@
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Components;
using Microsoft.AspNetCore.Components.Web;
@@ -10,6 +11,7 @@ using ZB.MOM.WW.OtOpcUa.Commons.Browsing;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Browser;
using ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Browser;
using ZB.MOM.WW.Secrets.Ui;
namespace ZB.MOM.WW.OtOpcUa.AdminUI;
@@ -28,6 +30,12 @@ public static class EndpointRouteBuilderExtensions
// Razor class library static assets (_content/ZB.MOM.WW.OtOpcUa.AdminUI/**) are
// served via the Host's app.UseStaticFiles() middleware which must run BEFORE
// UseAuthentication() — see Program.cs.
// The Secrets.Ui RCL's routable page is deliberately NOT registered here (lmxopcua#483):
// this host's router is static with per-page @rendermode opt-in, so routing straight to
// the RCL page rendered it without a circuit — it displayed but every @onclick was dead.
// Pages/Secrets.razor in THIS assembly owns /admin/secrets instead (wrapping the RCL
// page with the render mode); registering the RCL routes too would make the route
// ambiguous.
app.MapRazorComponents<TApp>()
.AddInteractiveServerRenderMode();
return app;
@@ -43,6 +51,15 @@ public static class EndpointRouteBuilderExtensions
services.AddRazorComponents().AddInteractiveServerComponents();
services.AddOtOpcUaDriverStatusServices();
// Secrets-management UI (ZB.MOM.WW.Secrets.Ui RCL, mounted at /admin/secrets). Register its
// "secrets:manage"/"secrets:reveal" authorization policies additively onto the AuthorizationOptions
// that AddOtOpcUaAuth (called just before AddAdminUI on admin nodes) sets up — Configure<T> stacks,
// so this ADDS the two policies without disturbing FleetAdmin/DriverOperator/ConfigEditor. The
// policies' AdminRole = "Administrator" reads the same ClaimTypes.Role claim as FleetAdmin, so an
// existing Administrator satisfies them with no new group→role mapping. Registered here (the AdminUI
// composition layer that already references the RCL) rather than in the core Security lib.
services.Configure<AuthorizationOptions>(o => o.AddSecretsAuthorization());
// Browse pipeline — see docs/plans/2026-05-28-driver-browsers-design.md
services.AddSingleton<Browsing.BrowseSessionRegistry>();
services.AddHostedService<Browsing.BrowseSessionReaper>();
src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/ZB.MOM.WW.OtOpcUa.AdminUI.csproj
+1
View File
@@ -10,6 +10,7 @@
<FrameworkReference Include="Microsoft.AspNetCore.App"/>
<PackageReference Include="Microsoft.AspNetCore.SignalR.Client"/>
<PackageReference Include="ZB.MOM.WW.Theme"/>
<PackageReference Include="ZB.MOM.WW.Secrets.Ui"/>
</ItemGroup>
<ItemGroup>
src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/AlarmSfLegacyMigrator.cs
+217
View File
@@ -0,0 +1,217 @@
using Microsoft.Data.Sqlite;
using Microsoft.Extensions.Configuration;
using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
/// <summary>
/// One-time copy of the pre-consolidation <c>alarm-historian.db</c> store-and-forward queue
/// into the consolidated <c>ZB.MOM.WW.LocalDb</c> database.
/// </summary>
/// <remarks>
/// <para>
/// <b>What is at stake.</b> Rows sit in that file precisely because the historian could not
/// be reached. Discarding them on upgrade would throw away exactly the alarm audit trail
/// the store-and-forward queue exists to protect — so the copy tolerates an older column
/// set rather than bailing out, and refuses to rename a file it did not actually read.
/// </para>
/// <para>
/// <b>Runs after <c>RegisterReplicated</c>, deliberately.</b> Capture is trigger-based, so
/// rows inserted before registration never enter the oplog and never reach the peer —
/// silently, and permanently, because nothing revisits history. Migrating after
/// registration means the recovered backlog replicates like any other write.
/// </para>
/// <para>
/// <b>Ids come from the payload, not from the legacy row id.</b> The legacy primary key was
/// <c>RowId INTEGER PRIMARY KEY AUTOINCREMENT</c>, which cannot survive into a replicated
/// table: each node allocated its own sequence, so node A's row 7 and node B's row 7 are
/// different alarms that would silently overwrite one another under last-writer-wins.
/// Hashing the payload (<see cref="AlarmSfSchema.DeriveId"/>) solves that and one more
/// problem besides — a warm pair's two legacy files <i>overlap</i>, because
/// <c>HistorianAdapterActor</c> default-writes while its redundancy role is unknown and
/// both nodes therefore accepted the same transitions during every boot window. A
/// node-prefixed id would carry that duplication into the merged buffer forever; an
/// equal-payload id collapses it. It is also what makes a re-run harmless under
/// <c>INSERT OR IGNORE</c>.
/// </para>
/// <para>
/// <b>All-or-nothing.</b> The copy runs in one transaction and the legacy file is renamed
/// to <c>&lt;name&gt;.migrated</c> only after the commit. A failure throws out of
/// <c>OnReady</c>, failing host startup with the legacy file untouched — no half-migrated
/// state to reason about, and the operator still holds the original. A later boot sees the
/// renamed file and no-ops.
/// </para>
/// </remarks>
public static class AlarmSfLegacyMigrator
{
private const string MigratedSuffix = ".migrated";
/// <summary>The legacy queue table.</summary>
private const string LegacyTable = "Queue";
/// <summary>
/// The configuration key that used to carry the standalone queue file's path. Read as a raw
/// key because the corresponding <c>AlarmHistorianOptions</c> property was removed with the
/// bespoke file management it configured.
/// </summary>
public const string LegacyPathKey = "AlarmHistorian:DatabasePath";
/// <summary>The pre-removal default for <see cref="LegacyPathKey"/>.</summary>
private const string DefaultLegacyPath = "alarm-historian.db";
/// <summary>
/// The legacy column whose absence means this is not a queue file we can read. Without the
/// payload there is no event and no id to derive from one.
/// </summary>
private const string RequiredColumn = "PayloadJson";
/// <summary>
/// Every legacy column, mapped to its consolidated counterpart. Columns absent from an
/// older file are dropped from the copy rather than failing it — see
/// <see cref="PresentColumns"/>.
/// </summary>
private static readonly (string Legacy, string Current)[] ColumnMap =
[
("AlarmId", "alarm_id"),
("EnqueuedUtc", "enqueued_at_utc"),
("PayloadJson", "payload_json"),
("AttemptCount", "attempt_count"),
("LastAttemptUtc", "last_attempt_utc"),
("LastError", "last_error"),
("DeadLettered", "dead_lettered"),
];
/// <summary>
/// Copies any legacy alarm queue into <paramref name="db"/>, then renames the legacy file.
/// </summary>
/// <param name="db">The consolidated database, with <c>alarm_sf_events</c> already registered.</param>
/// <param name="config">Configuration supplying the legacy queue's path.</param>
public static void Migrate(ILocalDb db, IConfiguration config)
{
ArgumentNullException.ThrowIfNull(db);
ArgumentNullException.ThrowIfNull(config);
var legacyPath = ResolveLegacyPath(config);
if (!ShouldMigrate(legacyPath)) return;
using (var legacy = OpenLegacyReadOnly(legacyPath))
{
var present = PresentColumns(legacy);
// An absent table probes as zero columns, so this one guard covers both "no queue
// table" and "a shape we do not recognise". Returning without renaming leaves the file
// for an operator to inspect.
if (!present.Contains(RequiredColumn)) return;
using var connection = db.CreateConnection();
using var transaction = connection.BeginTransaction();
CopyRows(legacy, connection, transaction, present);
transaction.Commit();
}
// Only after the commit. A crash between the two leaves the file in place and the next
// boot copies it again, which the payload-derived ids make harmless.
File.Move(legacyPath, legacyPath + MigratedSuffix, overwrite: true);
}
/// <summary>
/// Resolves the legacy queue path from the removed configuration key, falling back to the
/// code default it used to carry.
/// </summary>
/// <remarks>
/// Relative paths resolve against the process working directory — not the LocalDb
/// directory — because that is where the old code actually put them.
/// </remarks>
/// <param name="config">The application configuration.</param>
/// <returns>An absolute path, or empty when there is nothing durable to migrate from.</returns>
internal static string ResolveLegacyPath(IConfiguration config)
{
var path = config[LegacyPathKey] ?? DefaultLegacyPath;
// In-memory and URI-form sources (test / dev configurations) have nothing durable behind
// them, and Path.GetFullPath on them would produce nonsense.
if (string.IsNullOrWhiteSpace(path) ||
path.Equals(":memory:", StringComparison.OrdinalIgnoreCase) ||
path.StartsWith("file:", StringComparison.OrdinalIgnoreCase))
{
return string.Empty;
}
return Path.GetFullPath(path);
}
/// <summary>Whether there is a legacy file worth opening.</summary>
private static bool ShouldMigrate(string legacyPath) =>
!string.IsNullOrEmpty(legacyPath)
&& File.Exists(legacyPath)
&& !File.Exists(legacyPath + MigratedSuffix);
/// <summary>Opens the legacy file read-only, so a failed migration cannot damage it.</summary>
private static SqliteConnection OpenLegacyReadOnly(string path)
{
var connection = new SqliteConnection(new SqliteConnectionStringBuilder
{
DataSource = path,
Mode = SqliteOpenMode.ReadOnly,
}.ToString());
connection.Open();
return connection;
}
/// <summary>
/// Which of the columns we know about the legacy table actually has.
/// </summary>
/// <remarks>
/// A file written by an older build predates some columns, and naming a missing one in the
/// SELECT throws "no such column" — which discards every row in the table rather than the
/// one field. Intersecting first means an old file migrates its data and simply leaves the
/// newer columns at their schema defaults.
/// </remarks>
private static HashSet<string> PresentColumns(SqliteConnection legacy)
{
var present = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
using var cmd = legacy.CreateCommand();
cmd.CommandText = $"SELECT name FROM pragma_table_info('{LegacyTable}')";
using var reader = cmd.ExecuteReader();
while (reader.Read())
present.Add(reader.GetString(0));
return present;
}
/// <summary>Copies every legacy row into the consolidated table, keyed by payload hash.</summary>
private static void CopyRows(
SqliteConnection legacy,
SqliteConnection target,
SqliteTransaction transaction,
HashSet<string> present)
{
var columns = ColumnMap.Where(c => present.Contains(c.Legacy)).ToArray();
var payloadOrdinal = Array.FindIndex(columns, c => c.Legacy == RequiredColumn);
using var read = legacy.CreateCommand();
read.CommandText =
$"SELECT {string.Join(", ", columns.Select(c => c.Legacy))} FROM {LegacyTable}";
using var reader = read.ExecuteReader();
while (reader.Read())
{
using var insert = target.CreateCommand();
insert.Transaction = transaction;
insert.CommandText = $"""
INSERT OR IGNORE INTO {AlarmSfSchema.EventsTable}
({AlarmSfSchema.IdColumn}, {string.Join(", ", columns.Select(c => c.Current))})
VALUES
($id, {string.Join(", ", columns.Select((_, i) => $"$c{i}"))})
""";
insert.Parameters.AddWithValue("$id", AlarmSfSchema.DeriveId(reader.GetString(payloadOrdinal)));
for (var i = 0; i < columns.Length; i++)
insert.Parameters.AddWithValue($"$c{i}", reader.GetValue(i) ?? DBNull.Value);
insert.ExecuteNonQuery();
}
}
}
src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/KestrelHttpBinding.cs
+102
View File
@@ -0,0 +1,102 @@
using System.Net;
using Microsoft.AspNetCore.Server.Kestrel.Core;
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
/// <summary>
/// One HTTP endpoint the host was asked to serve, parsed out of the configured URL list so it
/// can be re-bound explicitly.
/// </summary>
/// <param name="Host">The host component as written — <c>+</c>, <c>*</c>, a hostname, or an IP.</param>
/// <param name="Port">The port.</param>
/// <param name="IsSecure">True when the URL used the <c>https</c> scheme.</param>
public sealed record KestrelHttpBinding(string Host, int Port, bool IsSecure)
{
/// <summary>
/// Parses a semicolon-separated URL list (the <c>ASPNETCORE_URLS</c> / <c>urls</c> format)
/// into bindings. Unparseable entries are skipped rather than throwing — a malformed URL
/// must not take the process down at startup.
/// </summary>
/// <param name="urls">The configured URL list, or null/empty when none is configured.</param>
/// <returns>The parsed bindings, in the order given; empty when nothing is configured.</returns>
public static IReadOnlyList<KestrelHttpBinding> Parse(string? urls)
{
if (string.IsNullOrWhiteSpace(urls))
return [];
var result = new List<KestrelHttpBinding>();
foreach (var raw in urls.Split(';', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries))
{
// Uri cannot parse the "+"/"*" wildcard hosts Kestrel accepts, so substitute a
// placeholder host purely to get scheme/port out, then keep the original host token.
var isWildcard = raw.Contains("://+", StringComparison.Ordinal)
|| raw.Contains("://*", StringComparison.Ordinal);
var probe = isWildcard
? raw.Replace("://+", "://placeholder", StringComparison.Ordinal)
.Replace("://*", "://placeholder", StringComparison.Ordinal)
: raw;
if (!Uri.TryCreate(probe, UriKind.Absolute, out var uri))
continue;
var host = isWildcard
? raw.Contains("://+", StringComparison.Ordinal) ? "+" : "*"
: uri.Host;
result.Add(new KestrelHttpBinding(host, uri.Port, uri.Scheme == Uri.UriSchemeHttps));
}
return result;
}
/// <summary>
/// Parses the bare-port list format of <c>ASPNETCORE_HTTP_PORTS</c> / <c>HTTP_PORTS</c> (and
/// the <c>HTTPS</c> variants) into wildcard (<c>+</c>, all-interfaces) bindings — the shape
/// Kestrel itself gives those variables. Modern .NET base images (aspnet:8.0+) set
/// <c>ASPNETCORE_HTTP_PORTS=8080</c> as the container default <b>instead of</b>
/// <c>ASPNETCORE_URLS</c>, so a node that never sets <c>URLS</c> explicitly still has a real
/// configured surface here that must be re-bound, not replaced with Kestrel's localhost:5000.
/// </summary>
/// <param name="ports">A <c>;</c>- or <c>,</c>-separated list of bare ports, or null/empty.</param>
/// <param name="isSecure">True to mark the parsed bindings <c>https</c> (the <c>HTTPS_PORTS</c> vars).</param>
/// <returns>One all-interfaces binding per parseable port; empty when nothing is configured.</returns>
public static IReadOnlyList<KestrelHttpBinding> ParsePorts(string? ports, bool isSecure)
{
if (string.IsNullOrWhiteSpace(ports))
return [];
var result = new List<KestrelHttpBinding>();
foreach (var token in ports.Split([';', ','], StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries))
{
// Skip a malformed port rather than throwing — a bad env var must not down startup.
if (int.TryParse(token, out var port) && port is > 0 and <= 65535)
result.Add(new KestrelHttpBinding("+", port, isSecure));
}
return result;
}
/// <summary>
/// Applies this binding to Kestrel, choosing the narrowest listen call the host component
/// allows.
/// </summary>
/// <remarks>
/// A hostname that is neither a literal IP nor <c>localhost</c> cannot be reliably mapped to
/// a local interface, so it falls back to <see cref="KestrelServerOptions.ListenAnyIP"/> —
/// the safe superset. Narrowing it and guessing wrong would silently stop serving.
/// </remarks>
/// <param name="options">The Kestrel options being configured.</param>
public void Apply(KestrelServerOptions options)
{
ArgumentNullException.ThrowIfNull(options);
if (Host is "+" or "*")
options.ListenAnyIP(Port);
else if (IPAddress.TryParse(Host, out var address))
options.Listen(address, Port);
else if (string.Equals(Host, "localhost", StringComparison.OrdinalIgnoreCase))
options.ListenLocalhost(Port);
else
options.ListenAnyIP(Port);
}
}
src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbRegistration.cs
+101
View File
@@ -0,0 +1,101 @@
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.LocalDb.Replication;
using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
/// <summary>
/// Single registration point for the node-local LocalDb subsystem: the embedded SQLite store
/// that caches the deployed-configuration artifact, plus the optional gRPC replication that
/// mirrors it to the node's redundant pair peer.
/// </summary>
/// <remarks>
/// <para>
/// Exists as a named extension rather than inline <c>AddZbLocalDb</c> calls in
/// <c>Program.cs</c> so the wiring is covered by a DI resolution test — <c>Program.cs</c> is
/// top-level statements and cannot be exercised directly, which is exactly how a
/// "registered but never resolvable" defect ships unnoticed. This family has shipped that
/// defect three times (Secrets 0.2.0, Secrets 0.2.2, ScadaBridge #22).
/// </para>
/// <para>
/// <b>Driver-role nodes only.</b> Admin-only nodes have no deployed configuration to cache,
/// and registering here would impose the <c>LocalDb:Path</c>-required-or-no-boot constraint
/// on them for nothing.
/// </para>
/// <para>
/// <b>Storage is unconditional; replication is default-OFF.</b> A node with no
/// <c>PeerAddress</c> and no <c>SyncListenPort</c> is simply a fast local SQLite file —
/// <c>AddZbLocalDbReplication</c> registers the engine but it stays idle. Replication is
/// opt-in per pair because there is no production distribution story for the sync
/// <c>ApiKey</c> yet (the same open question the Secrets KEK has).
/// </para>
/// </remarks>
public static class LocalDbRegistration
{
/// <summary>Configuration section holding <c>LocalDbOptions</c>.</summary>
public const string LocalDbSectionPath = "LocalDb";
/// <summary>Configuration section holding <c>ReplicationOptions</c>.</summary>
public const string ReplicationSectionPath = "LocalDb:Replication";
/// <summary>
/// Configuration key carrying the dedicated h2c sync listener's port. Zero or absent means
/// no listener is bound and the passive sync endpoint is not mapped.
/// </summary>
public const string SyncListenPortKey = "LocalDb:SyncListenPort";
/// <summary>
/// The port the dedicated cleartext-HTTP/2 sync listener should bind, or <c>0</c> when the
/// node should not listen at all. Defaults to <c>0</c> — absent configuration must mean
/// "off".
/// </summary>
/// <param name="configuration">The application configuration.</param>
/// <returns>The configured sync port, or <c>0</c>.</returns>
public static int SyncListenPort(IConfiguration configuration)
{
ArgumentNullException.ThrowIfNull(configuration);
return configuration.GetValue(SyncListenPortKey, defaultValue: 0);
}
/// <summary>
/// Registers the local database (with the deployment-cache schema applied and registered for
/// replication) and the replication engine.
/// </summary>
/// <remarks>
/// <para>
/// <see cref="LocalDbSetup.OnReady"/> runs inside <c>AddZbLocalDb</c>'s singleton factory,
/// before the first caller receives the <c>ILocalDb</c> — so every consumer is guaranteed
/// to see the tables created and their capture triggers installed.
/// </para>
/// <para>
/// <c>AddZbLocalDbReplication</c> is called unconditionally rather than behind an
/// <c>Enabled</c> flag because the library already models "off" as "no
/// <c>PeerAddress</c>": the initiator background service idles and the passive endpoint
/// is only reachable if <c>Program.cs</c> maps it, which it does only when
/// <see cref="SyncListenPortKey"/> is set. Adding a second gate on top would give two
/// ways to express the same thing and a way for them to disagree.
/// </para>
/// </remarks>
/// <param name="services">The service collection to add to.</param>
/// <param name="configuration">The application configuration.</param>
/// <returns>The same <paramref name="services"/> instance, for chaining.</returns>
public static IServiceCollection AddOtOpcUaLocalDb(
this IServiceCollection services,
IConfiguration configuration)
{
ArgumentNullException.ThrowIfNull(services);
ArgumentNullException.ThrowIfNull(configuration);
services.AddZbLocalDb(configuration, db => LocalDbSetup.OnReady(db, configuration));
services.AddZbLocalDbReplication(configuration);
// The consumer-facing seam. WithOtOpcUaRuntimeActors resolves this optionally and threads it
// into DriverHostActor; registering the store without this leaves the cache tables present,
// replicating, and never written to.
services.AddSingleton<IDeploymentArtifactCache, LocalDbDeploymentArtifactCache>();
return services;
}
}
src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs
+68
View File
@@ -0,0 +1,68 @@
using Microsoft.Extensions.Configuration;
using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
/// <summary>
/// The <c>onReady</c> callback handed to <c>AddZbLocalDb</c>: creates the deployment-cache and
/// alarm store-and-forward tables and opts them into replication.
/// </summary>
/// <remarks>
/// <para>
/// Public rather than internal only so <c>LocalDbSetupTests</c> can drive the production
/// callback directly. Initialising a test database from a hand-written copy of this schema
/// would prove only that the test agrees with itself — the whole value of those tests is
/// that they exercise <b>this</b> method.
/// </para>
/// </remarks>
public static class LocalDbSetup
{
/// <summary>
/// Initialises the local database: DDL first, then replication registration.
/// </summary>
/// <remarks>
/// <para>
/// <b>THE ORDER IS LOAD-BEARING: DDL → RegisterReplicated → writes.</b>
/// <c>RegisterReplicated</c> is what installs the three AFTER triggers that capture
/// changes into the oplog. Any row written before that call is never captured, so it
/// never reaches the peer — silently, and permanently, because nothing ever revisits
/// history. <see cref="AlarmSfLegacyMigrator"/> therefore runs <b>last</b>, after every
/// registration — it writes rows, and they must be captured like any other write.
/// </para>
/// <para>
/// The alarm buffer's tables are created unconditionally, regardless of whether this
/// node has <c>AlarmHistorian:Enabled</c> set. An empty registered table costs three
/// triggers and nothing else, whereas creating it lazily when the sink first appears
/// would mean a node that enables the historian later writes rows before its triggers
/// exist — which is precisely the silent-loss shape above.
/// </para>
/// </remarks>
/// <param name="db">The freshly constructed local database.</param>
/// <param name="configuration">
/// Application configuration, read by the legacy migrator for the pre-consolidation queue's
/// path. Required rather than optional: an overload that silently skipped the migration
/// would be one wiring mistake away from discarding a node's undelivered alarm history.
/// </param>
public static void OnReady(ILocalDb db, IConfiguration configuration)
{
ArgumentNullException.ThrowIfNull(db);
ArgumentNullException.ThrowIfNull(configuration);
// CreateConnection() hands back an already-open, pragma-configured connection carrying the
// zb_hlc_next() UDF the capture triggers need. Calling Open() on it would throw.
using (var connection = db.CreateConnection())
{
DeploymentCacheSchema.Apply(connection);
AlarmSfSchema.Apply(connection);
}
db.RegisterReplicated(DeploymentCacheSchema.ArtifactsTable);
db.RegisterReplicated(DeploymentCacheSchema.PointerTable);
db.RegisterReplicated(AlarmSfSchema.EventsTable);
// LAST, and only here. This is the one call in OnReady that writes rows.
AlarmSfLegacyMigrator.Migrate(db, configuration);
}
}
src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSyncAuthInterceptor.cs
+162
View File
@@ -0,0 +1,162 @@
using System.Security.Cryptography;
using System.Text;
using Grpc.Core;
using Grpc.Core.Interceptors;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.LocalDb.Replication;
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
/// <summary>
/// Gates the LocalDb passive sync endpoint. The replication library deliberately leaves inbound
/// authentication to the host — its <c>LocalDbSyncService</c> verifies nothing — so without this
/// interceptor anything that can reach a driver node's sync port could stream arbitrary rows
/// into that node's deployment-artifact cache.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why that matters here specifically.</b> The cached artifact is what a driver node
/// boots from when central SQL is unreachable. An attacker able to write it could choose
/// the configuration a node comes up on during exactly the outage nobody is watching.
/// </para>
/// <para>
/// <b>Scoped by method path.</b> Only calls under <c>/localdb_sync.v1.LocalDbSync/</c> are
/// gated; every other method on the shared gRPC pipeline passes through untouched.
/// </para>
/// <para>
/// <b>Fail-closed.</b> With no <c>LocalDb:Replication:ApiKey</c> configured, NO sync stream
/// is accepted, authenticated or not. That is the deliberate choice: the alternative —
/// treating "no key" as "no auth required" — would silently expose the endpoint on exactly
/// the default configuration every node ships with. An operator enabling replication must
/// set the same key on both nodes of the pair, which is already required for the initiator
/// to dial out (<c>SyncBackgroundService</c> sends <c>Authorization: Bearer &lt;key&gt;</c>).
/// A key typo therefore stops convergence outright rather than degrading to unauthenticated.
/// </para>
/// <para>
/// Comparison is <see cref="CryptographicOperations.FixedTimeEquals"/> over UTF-8 bytes, so
/// a wrong key cannot be recovered byte-by-byte from response timing. Length differences are
/// unavoidably observable and are not sensitive.
/// </para>
/// </remarks>
public sealed class LocalDbSyncAuthInterceptor : Interceptor
{
private const string ServicePrefix = "/localdb_sync.v1.LocalDbSync/";
private const string AuthorizationHeader = "authorization";
private const string BearerPrefix = "Bearer ";
private readonly IOptions<ReplicationOptions> _options;
private readonly ILogger<LocalDbSyncAuthInterceptor> _logger;
/// <summary>Creates the interceptor.</summary>
/// <param name="options">Replication options; <c>ApiKey</c> is the expected bearer token.</param>
/// <param name="logger">Logger for denial diagnostics.</param>
public LocalDbSyncAuthInterceptor(
IOptions<ReplicationOptions> options,
ILogger<LocalDbSyncAuthInterceptor> logger)
{
ArgumentNullException.ThrowIfNull(options);
ArgumentNullException.ThrowIfNull(logger);
_options = options;
_logger = logger;
}
/// <inheritdoc />
public override Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
TRequest request,
ServerCallContext context,
UnaryServerMethod<TRequest, TResponse> continuation)
{
Authorize(context);
return continuation(request, context);
}
/// <inheritdoc />
public override Task DuplexStreamingServerHandler<TRequest, TResponse>(
IAsyncStreamReader<TRequest> requestStream,
IServerStreamWriter<TResponse> responseStream,
ServerCallContext context,
DuplexStreamingServerMethod<TRequest, TResponse> continuation)
{
Authorize(context);
return continuation(requestStream, responseStream, context);
}
/// <inheritdoc />
public override Task<TResponse> ClientStreamingServerHandler<TRequest, TResponse>(
IAsyncStreamReader<TRequest> requestStream,
ServerCallContext context,
ClientStreamingServerMethod<TRequest, TResponse> continuation)
{
Authorize(context);
return continuation(requestStream, context);
}
/// <inheritdoc />
public override Task ServerStreamingServerHandler<TRequest, TResponse>(
TRequest request,
IServerStreamWriter<TResponse> responseStream,
ServerCallContext context,
ServerStreamingServerMethod<TRequest, TResponse> continuation)
{
Authorize(context);
return continuation(request, responseStream, context);
}
/// <summary>
/// Throws <see cref="RpcException"/> with <see cref="StatusCode.PermissionDenied"/> if this
/// is a sync call that does not carry the configured bearer token. Non-sync calls return
/// immediately.
/// </summary>
private void Authorize(ServerCallContext context)
{
if (!context.Method.StartsWith(ServicePrefix, StringComparison.Ordinal))
return;
var expected = _options.Value.ApiKey;
if (string.IsNullOrEmpty(expected))
{
_logger.LogWarning(
"Rejected a LocalDb sync call to {Method}: no LocalDb:Replication:ApiKey is configured, " +
"so the passive sync endpoint is closed. Configure the same key on both nodes of the pair.",
context.Method);
throw new RpcException(new Status(
StatusCode.PermissionDenied,
"LocalDb sync is not accepting connections: no API key is configured on this node."));
}
var presented = ExtractBearerToken(context.RequestHeaders);
if (presented is null || !FixedTimeEquals(presented, expected))
{
_logger.LogWarning(
"Rejected a LocalDb sync call to {Method}: {Reason}.",
context.Method,
presented is null ? "no bearer token presented" : "bearer token did not match");
throw new RpcException(new Status(
StatusCode.PermissionDenied,
"LocalDb sync authentication failed."));
}
}
private static string? ExtractBearerToken(Metadata headers)
{
// gRPC lowercases header keys on the wire; compare case-insensitively anyway so a
// hand-built Metadata in a test behaves the same as a real request.
foreach (var entry in headers)
{
if (!string.Equals(entry.Key, AuthorizationHeader, StringComparison.OrdinalIgnoreCase))
continue;
var value = entry.Value;
if (value is not null && value.StartsWith(BearerPrefix, StringComparison.OrdinalIgnoreCase))
return value[BearerPrefix.Length..];
}
return null;
}
private static bool FixedTimeEquals(string presented, string expected)
=> CryptographicOperations.FixedTimeEquals(
Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expected));
}
src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/SecretReplicationStarter.cs
+55
View File
@@ -0,0 +1,55 @@
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
/// <summary>
/// Forces this node's secret-replication actor into existence at startup.
/// </summary>
/// <remarks>
/// <para>
/// The replication actor is created <b>lazily</b>, on the first resolution of
/// <see cref="ISecretStore"/> — resolving the store builds <c>ReplicatingSecretStore</c>,
/// which takes <c>ISecretReplicator</c>, whose factory touches
/// <c>SecretReplicationActorProvider.ActorRef</c> and thereby spawns the actor.
/// </para>
/// <para>
/// Laziness is correct for the library (the <c>ActorSystem</c> is usually registered after
/// the replication call), but it makes participation in anti-entropy accidental: a node that
/// happens never to read or write a secret — a plausible steady state for a driver-role node
/// whose driver configs carry no <c>${secret:}</c> references — would never create the actor,
/// never announce its manifest, and never converge. Nothing would fail; it would just
/// silently not replicate. Resolving the store once at startup makes participation
/// unconditional.
/// </para>
/// <para>
/// <b>History — this hook has met two library defects, both fixed.</b> Against 0.2.0 it was
/// inert: the package never bound its own <c>ISecretReplicator</c> (TryAdd registration
/// order), so this resolve built a <c>ReplicatingSecretStore</c> around a no-op sink and
/// spawned no actor. Against 0.2.1 it was worse: the resolve <b>deadlocked the host at
/// startup</b> — the package's DI graph had a circular singleton dependency, invisible to
/// the container through factory lambdas (scadaproj#1). Fixed in 0.2.2 by deferring the
/// cycle-closing invalidator edge;
/// <c>SecretsReplicationRegistrationTests.The_startup_hook_actually_creates_the_replication_actor</c>
/// exercises this hook against a built provider on a real single-node cluster.
/// </para>
/// </remarks>
/// <param name="services">Root provider used to resolve the (decorated) secret store exactly once.</param>
public sealed class SecretReplicationStarter(IServiceProvider services) : IHostedService
{
/// <summary>Resolves <see cref="ISecretStore"/>, which spawns the replication actor.</summary>
/// <param name="cancellationToken">Unused — resolution is synchronous and non-blocking.</param>
/// <returns>A completed task.</returns>
public Task StartAsync(CancellationToken cancellationToken)
{
// The resolution itself is the side effect; the instance is deliberately unused.
_ = services.GetRequiredService<ISecretStore>();
return Task.CompletedTask;
}
/// <summary>No-op: the actor's lifetime is the actor system's.</summary>
/// <param name="cancellationToken">Unused.</param>
/// <returns>A completed task.</returns>
public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask;
}
src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/SecretsRegistration.cs
+82
View File
@@ -0,0 +1,82 @@
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using ZB.MOM.WW.Secrets.DependencyInjection;
using ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.DependencyInjection;
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
/// <summary>
/// Single registration point for the secrets subsystem on every OtOpcUa node, admin- and
/// driver-role alike. Exists as a named extension rather than an inline <c>AddZbSecrets</c> call
/// in <c>Program.cs</c> so the choice of <c>ISecretStore</c> implementation is covered by a DI
/// resolution test — <c>Program.cs</c> is top-level statements and cannot be exercised directly,
/// which is exactly how a "registered but never resolvable" wiring defect ships unnoticed.
/// </summary>
public static class SecretsRegistration
{
/// <summary>Configuration key gating cluster secret replication. Absent or false = off.</summary>
public const string ReplicationEnabledKey = "Secrets:Replication:Enabled";
/// <summary>Configuration section holding <c>AkkaSecretsReplicationOptions</c>.</summary>
public const string ReplicationSectionPath = "Secrets:Replication";
/// <summary>Configuration section holding the core secrets options.</summary>
public const string SecretsSectionPath = "Secrets";
/// <summary>
/// True when cluster secret replication is switched on in configuration. Defaults to false —
/// absent configuration must mean "off".
/// </summary>
/// <param name="configuration">The application configuration.</param>
/// <returns><c>true</c> when replication is enabled.</returns>
public static bool IsReplicationEnabled(IConfiguration configuration)
{
ArgumentNullException.ThrowIfNull(configuration);
return configuration.GetValue(ReplicationEnabledKey, defaultValue: false);
}
/// <summary>
/// Registers the secrets subsystem. Replication is <b>opt-in</b>: unless
/// <see cref="ReplicationEnabledKey"/> is true this is exactly the plain local SQLite wiring
/// the host has always used.
/// </summary>
/// <remarks>
/// <para>
/// The gate is deliberately default-deny. This call decides which <c>ISecretStore</c> the
/// container hands to <b>every</b> node, including driver-role nodes with no auth or
/// AdminUI surface, where a bad store surfaces as drivers failing to open sessions rather
/// than as a failing test. Enabling replication by default would change secret resolution
/// across a production fleet with nobody having asked for it.
/// </para>
/// <para>
/// <c>AddZbSecretsAkkaReplication</c> calls <c>AddZbSecrets</c> internally, so the two
/// branches are alternatives, never additive — calling both would double-register.
/// </para>
/// </remarks>
/// <param name="services">The service collection to add to.</param>
/// <param name="configuration">The application configuration.</param>
/// <returns>The same <paramref name="services"/> instance, for chaining.</returns>
public static IServiceCollection AddOtOpcUaSecrets(
this IServiceCollection services,
IConfiguration configuration)
{
ArgumentNullException.ThrowIfNull(services);
ArgumentNullException.ThrowIfNull(configuration);
if (!IsReplicationEnabled(configuration))
{
services.AddZbSecrets(configuration, SecretsSectionPath);
return services;
}
services.AddZbSecretsAkkaReplication(configuration, SecretsSectionPath, ReplicationSectionPath);
// Anti-entropy participation must not hinge on this node happening to touch a secret — see
// SecretReplicationStarter for why the library's lazy actor creation is insufficient here.
services.AddSingleton<SecretReplicationStarter>();
services.AddHostedService(sp => sp.GetRequiredService<SecretReplicationStarter>());
return services;
}
}
src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ServerHistorianOptionsValidator.cs
+39 -5
View File
@@ -24,12 +24,25 @@ namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
/// <see cref="ServerHistorianOptions.Enabled"/> in the Host, so <c>Enabled</c> covers it.
/// </para>
/// <para>
/// <b>Fail tier = provably-crashing configs only.</b> Only an empty / non-absolute / non-http(s)
/// <c>Endpoint</c> fails here (it throws in the factory). Empty <c>ApiKey</c> and non-positive
/// <c>MaxTieClusterOverfetch</c> degrade rather than crash (the gateway rejects calls / the node
/// manager surfaces a Bad read), so they stay operator warnings in
/// <b>Fail tier = provably-crashing configs only.</b> Three settings qualify, and they all fail
/// the same way — <c>HistorianGatewayClientOptions.Validate()</c> throws inside the client
/// factory before any call is attempted, so the host dies at startup:
/// an empty / non-absolute / non-http(s) <c>Endpoint</c>; an empty <c>ApiKey</c> ("The gateway
/// API key must not be empty"); and a <c>UseTls</c> that disagrees with the endpoint scheme
/// ("UseTls requires an https gateway endpoint" / "An https gateway endpoint requires UseTls").
/// A non-positive <c>MaxTieClusterOverfetch</c> genuinely degrades rather than crashes (the node
/// manager surfaces a Bad read), so it stays an operator warning in
/// <see cref="ServerHistorianOptions.Validate"/>. The <c>Endpoint</c> value is not a secret and
/// is echoed to make the error actionable; the <c>ApiKey</c> is never surfaced.
/// is echoed to make each error actionable; the <c>ApiKey</c> is never surfaced.
/// </para>
/// <para>
/// <b>The <c>ApiKey</c> and <c>UseTls</c> checks were added after two consecutive live-rig
/// crash-loops</b> (LocalDb Phase 2 gate). Both had been classified as degrading, on the
/// assumption that a bad client configuration would connect and be rejected by the gateway — but
/// the client validates its own options at construction, so the process dies during Akka startup
/// instead, which is precisely the failure this validator exists to convert into a named,
/// aggregated error. The lesson generalises: "degrades" is only true of settings the client
/// does not itself validate.
/// </para>
/// </remarks>
public sealed class ServerHistorianOptionsValidator : OptionsValidatorBase<ServerHistorianOptions>
@@ -62,5 +75,26 @@ public sealed class ServerHistorianOptionsValidator : OptionsValidatorBase<Serve
builder.RequireThat(
endpointValid,
$"ServerHistorian:Endpoint is empty or not an absolute http(s) URI ('{options.Endpoint}') — {reason}.");
// Deliberately not echoed: unlike the endpoint, the key is a secret.
builder.RequireThat(
!string.IsNullOrWhiteSpace(options.ApiKey),
$"ServerHistorian:ApiKey is empty — {reason}. The gateway client rejects a keyless "
+ "configuration at construction, so the host would crash on startup rather than degrade. "
+ "Supply it via the environment variable ServerHistorian__ApiKey.");
// The flag and the scheme must agree in BOTH directions — the client throws either way
// ("UseTls requires an https gateway endpoint" / "An https gateway endpoint requires UseTls").
// Only meaningful once the endpoint parsed; a malformed one already failed above.
if (endpointValid)
{
var https = uri!.Scheme == Uri.UriSchemeHttps;
builder.RequireThat(
options.UseTls == https,
$"ServerHistorian:UseTls is {options.UseTls.ToString().ToLowerInvariant()} but the "
+ $"endpoint is '{options.Endpoint}' — {reason}. The flag must match the scheme "
+ "(https ⇒ UseTls=true, http ⇒ UseTls=false); the gateway client rejects a mismatch "
+ "at construction, crashing the host on startup.");
}
}
}
src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs
+8 -3
View File
@@ -4,6 +4,7 @@ using Microsoft.Extensions.Logging;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Hosting;
using ZB.MOM.WW.OtOpcUa.Core.Resilience;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Host.Drivers;
@@ -48,7 +49,10 @@ public static class DriverFactoryBootstrap
// The calc driver needs the root script logger so a script failure fans out onto the
// script-logs topic; resolve it here (null on nodes without the script pipeline wired).
var scriptRoot = sp.GetService<ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptRootLogger>();
Register(registry, loggerFactory, scriptRoot);
// The Galaxy driver resolves its Gateway.ApiKeySecretRef secret: arm through the
// shared ISecretResolver (registered unconditionally by AddZbSecrets on every node).
var secretResolver = sp.GetRequiredService<ISecretResolver>();
Register(registry, loggerFactory, secretResolver, scriptRoot);
return registry;
});
services.AddSingleton<IDriverFactory>(sp =>
@@ -131,15 +135,16 @@ public static class DriverFactoryBootstrap
private static void Register(
DriverFactoryRegistry registry,
ILoggerFactory? loggerFactory,
ISecretResolver secretResolver,
ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptRootLogger? scriptRoot = null)
{
Driver.AbCip.AbCipDriverFactoryExtensions.Register(registry);
Driver.AbLegacy.AbLegacyDriverFactoryExtensions.Register(registry, loggerFactory);
Driver.Calculation.CalculationDriverFactoryExtensions.Register(registry, loggerFactory, scriptRoot);
Driver.FOCAS.FocasDriverFactoryExtensions.Register(registry);
Driver.Galaxy.GalaxyDriverFactoryExtensions.Register(registry, loggerFactory);
Driver.Galaxy.GalaxyDriverFactoryExtensions.Register(registry, secretResolver, loggerFactory);
Driver.Modbus.ModbusDriverFactoryExtensions.Register(registry, loggerFactory);
Driver.OpcUaClient.OpcUaClientDriverFactoryExtensions.Register(registry, loggerFactory);
Driver.OpcUaClient.OpcUaClientDriverFactoryExtensions.Register(registry, loggerFactory, secretResolver);
Driver.S7.S7DriverFactoryExtensions.Register(registry);
Driver.TwinCAT.TwinCATDriverFactoryExtensions.Register(registry);
}
src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/HealthEndpoints.cs
+19 -1
View File
@@ -1,10 +1,15 @@
using Microsoft.AspNetCore.Routing;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Diagnostics.HealthChecks;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.Health;
using ZB.MOM.WW.Health.Akka;
using ZB.MOM.WW.Health.EntityFrameworkCore;
using ZB.MOM.WW.LocalDb.Replication;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Host.Configuration;
namespace ZB.MOM.WW.OtOpcUa.Host.Health;
@@ -36,7 +41,20 @@ public static class HealthEndpoints
"admin-leader",
failureStatus: null,
tags: new[] { ZbHealthTags.Active },
args: "admin");
args: "admin")
// Registered unconditionally, not driver-gated. AddOtOpcUaHealth takes no role argument
// and runs on every node; the check itself resolves ISyncStatus optionally and reports
// Healthy when LocalDb is absent (admin-only graphs) or replication is default-OFF, so a
// plain node is never degraded by it. A factory registration keeps this no-arg signature
// while still reading ISyncStatus + options + the sync port from the container.
.Add(new HealthCheckRegistration(
"localdb-replication",
sp => new LocalDbReplicationHealthCheck(
sp.GetService<ISyncStatus>(),
sp.GetRequiredService<IOptions<ReplicationOptions>>(),
LocalDbRegistration.SyncListenPort(sp.GetRequiredService<IConfiguration>())),
failureStatus: null,
tags: new[] { ZbHealthTags.Active }));
return services;
}
src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/LocalDbReplicationHealthCheck.cs
+126
View File
@@ -0,0 +1,126 @@
using Microsoft.Extensions.Diagnostics.HealthChecks;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.LocalDb.Replication;
namespace ZB.MOM.WW.OtOpcUa.Host.Health;
/// <summary>
/// Pure decision function for the LocalDb replication probe, factored out of
/// <see cref="LocalDbReplicationHealthCheck"/> so the whole matrix is table-testable without
/// standing up a replication engine.
/// </summary>
public static class LocalDbReplicationDecision
{
/// <summary>
/// Maps the resolved replication facts to a health status.
/// </summary>
/// <param name="peerConfigured">
/// True when this node participates in replication at all — it has a peer address to dial, or
/// a sync listener bound. False means default-OFF, and default-OFF must never degrade a node.
/// </param>
/// <param name="connected">Whether a sync session is currently running.</param>
/// <param name="oplogBacklog">
/// The unacked oplog backlog, or <see langword="null"/> when it could not be read. Null is
/// deliberately not treated as zero: a failed poll is "unknown", not "caught up".
/// </param>
/// <param name="degradedThreshold">
/// Backlog at or above which a connected pair is judged to be falling behind.
/// </param>
/// <returns>The status plus a human-readable reason.</returns>
public static (HealthStatus Status, string Description) Evaluate(
bool peerConfigured, bool connected, long? oplogBacklog, long degradedThreshold)
{
if (!peerConfigured)
return (HealthStatus.Healthy, "LocalDb replication is not configured on this node (default-OFF).");
if (!connected)
return (HealthStatus.Degraded, "LocalDb replication is configured but no sync session is connected.");
if (oplogBacklog is null)
return (HealthStatus.Degraded, "LocalDb replication is connected but its oplog backlog is unknown (the poll failed).");
if (oplogBacklog.Value >= degradedThreshold)
return (HealthStatus.Degraded,
$"LocalDb replication is connected but its oplog backlog ({oplogBacklog.Value}) is at or above the degraded threshold ({degradedThreshold}).");
return (HealthStatus.Healthy, $"LocalDb replication is connected; oplog backlog {oplogBacklog.Value}.");
}
}
/// <summary>
/// Reports the health of this node's LocalDb replication link. Registered unconditionally with
/// the shared health pipeline; when LocalDb is not present at all (admin-only graphs) it reports
/// <see cref="HealthStatus.Healthy"/> rather than degrading — matching the
/// <c>ActiveNodeHealthCheck</c> precedent of "not applicable ⇒ Healthy".
/// </summary>
/// <remarks>
/// A node running LocalDb with replication switched off (no peer, no listener) is also Healthy:
/// that is the default posture for most of the fleet, and it must not look degraded. Only a node
/// that is <i>meant</i> to be replicating and is not — or is connected but cannot confirm it is
/// draining — is surfaced as Degraded. It never reports Unhealthy: a replication problem does not
/// stop the node serving its address space, so it must not fail a readiness gate.
/// </remarks>
public sealed class LocalDbReplicationHealthCheck : IHealthCheck
{
/// <summary>
/// Backlog at or above which a connected pair is reported Degraded. Well below the library's
/// <c>MaxOplogRows</c> default (1,000,000, where a snapshot resync is forced), so the probe
/// flags a pair that is falling behind long before the engine itself intervenes.
/// </summary>
public const long DefaultBacklogDegradedThreshold = 100_000;
private readonly ISyncStatus? _syncStatus;
private readonly ReplicationOptions _options;
private readonly int _syncListenPort;
private readonly long _degradedThreshold;
/// <summary>Creates the health check.</summary>
/// <param name="syncStatus">
/// The replication status singleton, or <see langword="null"/> when the replication engine is
/// not registered (admin-only nodes) — in which case the check is a Healthy no-op.
/// </param>
/// <param name="options">The bound replication options (peer address, etc.).</param>
/// <param name="syncListenPort">
/// The configured sync listener port; a value greater than zero counts as "replication
/// configured" even when this node is the passive side with no peer address.
/// </param>
/// <param name="degradedThreshold">Backlog degraded threshold; defaults to <see cref="DefaultBacklogDegradedThreshold"/>.</param>
public LocalDbReplicationHealthCheck(
ISyncStatus? syncStatus,
IOptions<ReplicationOptions> options,
int syncListenPort,
long degradedThreshold = DefaultBacklogDegradedThreshold)
{
ArgumentNullException.ThrowIfNull(options);
_syncStatus = syncStatus;
_options = options.Value;
_syncListenPort = syncListenPort;
_degradedThreshold = degradedThreshold;
}
/// <inheritdoc />
public Task<HealthCheckResult> CheckHealthAsync(
HealthCheckContext context, CancellationToken cancellationToken = default)
{
// No replication engine registered at all → not applicable → Healthy. Admin-only nodes and
// any graph that did not call AddOtOpcUaLocalDb land here.
if (_syncStatus is null)
return Task.FromResult(HealthCheckResult.Healthy(
"LocalDb replication is not present on this node."));
var peerConfigured =
!string.IsNullOrWhiteSpace(_options.PeerAddress) || _syncListenPort > 0;
var (status, description) = LocalDbReplicationDecision.Evaluate(
peerConfigured, _syncStatus.Connected, _syncStatus.OplogBacklog, _degradedThreshold);
var result = status switch
{
HealthStatus.Healthy => HealthCheckResult.Healthy(description),
_ => HealthCheckResult.Degraded(description),
};
return Task.FromResult(result);
}
}
src/Server/ZB.MOM.WW.OtOpcUa.Host/Observability/ObservabilityExtensions.cs
+6 -1
View File
@@ -1,4 +1,5 @@
using Microsoft.Extensions.Configuration;
using ZB.MOM.WW.LocalDb.Replication;
using ZB.MOM.WW.OtOpcUa.Commons.Observability;
using ZB.MOM.WW.Telemetry;
@@ -27,7 +28,11 @@ public static class ObservabilityExtensions
return services.AddZbTelemetry(o =>
{
o.ServiceName = "otopcua";
o.Meters = [OtOpcUaTelemetry.MeterName];
// o.Meters is a STRICT allowlist — ZbTelemetry adds only the named meters, with no
// wildcard. The LocalDb replication engine publishes under its own meter name, so
// without this entry its localdb.sync.* / localdb.oplog.depth series are silently absent
// from /metrics on driver nodes (the exact omission a ScadaBridge live gate caught).
o.Meters = [OtOpcUaTelemetry.MeterName, LocalDbMetrics.MeterName];
o.ActivitySources = [OtOpcUaTelemetry.ActivitySourceName];
if (Enum.TryParse<ZbExporter>(configuration["OtOpcUa:Telemetry:Exporter"], ignoreCase: true, out var exporter))
o.Exporter = exporter;
src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs
+151 -1
View File
@@ -1,7 +1,9 @@
using Akka.Actor;
using Akka.Hosting;
using Microsoft.AspNetCore.Server.Kestrel.Core;
using Microsoft.Extensions.DependencyInjection.Extensions;
using Serilog;
using ZB.MOM.WW.LocalDb.Replication;
using Serilog.Events;
using ZB.MOM.WW.OtOpcUa.AdminUI;
using ZB.MOM.WW.OtOpcUa.AdminUI.Clients;
@@ -37,6 +39,12 @@ using ZB.MOM.WW.OtOpcUa.Security.Ldap;
using ZB.MOM.WW.Configuration;
using ZB.MOM.WW.Telemetry.Serilog;
using ZB.MOM.WW.OtOpcUa.AdminUI.Api;
using Microsoft.Extensions.DependencyInjection;
using ZB.MOM.WW.Secrets.Abstractions;
using ZB.MOM.WW.Secrets.Configuration;
using ZB.MOM.WW.Secrets.DependencyInjection;
using ZB.MOM.WW.Secrets.Replicator.AkkaDotNet;
using ZB.MOM.WW.Secrets.Sqlite;
// Roles drive the entire conditional wiring below — see ZB.MOM.WW.OtOpcUa.Cluster.RoleParser.
var roles = RoleParser.Parse(Environment.GetEnvironmentVariable("OTOPCUA_ROLES"));
@@ -72,6 +80,27 @@ if (roleSuffix is not null)
builder.Configuration.AddCommandLine(args);
}
// Pre-host ${secret:} expansion: rewrites ${secret:name} tokens in the assembled configuration
// BEFORE the host is built, so every downstream binder/validator sees resolved plaintext.
// Placement matters: this runs AFTER all config providers are assembled (base appsettings, the
// role overlay, and the env-vars + command-line re-append above) but BEFORE anything READS a
// config value (AddZbSerilog's ReadFrom.Configuration, AddOtOpcUaConfigDb, and the first
// ValidateOnStart bind all follow). With no ${secret:} tokens present this is a no-op pass;
// it also migrates the secrets SQLite store on startup.
// ASP0000: DELIBERATE throwaway container — expansion must run before builder.Build(), so it
// cannot use the app's DI. Fully disposed here; shares no singletons with the host container.
#pragma warning disable ASP0000
await using (var secretsProvider = new ServiceCollection()
.AddZbSecrets(builder.Configuration, "Secrets")
.BuildServiceProvider())
#pragma warning restore ASP0000
{
await secretsProvider.GetRequiredService<SqliteSecretsStoreMigrator>().MigrateAsync(default);
var resolver = secretsProvider.GetRequiredService<ISecretResolver>();
await new SecretReferenceExpander(resolver)
.ExpandConfigurationAsync((IConfigurationRoot)builder.Configuration, default);
}
// Anchor the process working directory to the install directory (AppContext.BaseDirectory) so every
// relative runtime path resolves under the install dir rather than the service's startup CWD. A Windows
// service starts with CWD=C:\Windows\System32, which otherwise scattered the Serilog rolling-file sink
@@ -107,7 +136,7 @@ if (hasDriver)
// Config-gated durable alarm-historian sink. When the AlarmHistorian section is enabled this
// overrides the NullAlarmHistorianSink default from AddOtOpcUaRuntime (last registration wins)
// with a SqliteStoreAndForwardSink draining to the gateway SendEvent writer. The alarm-write path
// with a LocalDbStoreAndForwardSink draining to the gateway SendEvent writer. The alarm-write path
// targets the SAME single gateway as the read path, so its connection (endpoint/key/TLS) is sourced
// from the ServerHistorian section. AlarmHistorianOptions supplies only the Enabled gate + the
// SQLite store-and-forward knobs (consumed inside AddAlarmHistorian) — it carries no connection fields.
@@ -147,6 +176,20 @@ if (hasDriver)
builder.Configuration,
(_, sp) => GatewayHistorian.CreateAlarmWriter(serverHistorianOptions, sp));
// Node-local LocalDb: caches the deployed-configuration artifact so this node can boot from
// its last-known-good config when central SQL is unreachable, and optionally replicates that
// cache to its redundant pair peer. Driver-role only — admin-only nodes have nothing to cache,
// and registering here would impose LocalDb:Path-required-or-no-boot on them for nothing.
// Storage ships unconditionally; replication stays inert until LocalDb:Replication:PeerAddress
// (initiator) / LocalDb:SyncListenPort (listener) are set. See LocalDbRegistration.
builder.Services.AddOtOpcUaLocalDb(builder.Configuration);
// gRPC server plumbing for the passive sync endpoint mapped below. Registered only under
// hasDriver so admin-only nodes expose no sync surface at all. The interceptor is the ONLY
// inbound auth on that endpoint — the replication library's LocalDbSyncService verifies
// nothing — and it fail-closes when no ApiKey is configured.
builder.Services.AddGrpc(o => o.Interceptors.Add<LocalDbSyncAuthInterceptor>());
// Config-gated server-side HistoryRead backend. When the ServerHistorian section is enabled this
// overrides the NullHistorianDataSource default from AddOtOpcUaRuntime (last registration wins) with
// a read-only HistorianGateway-backed data source the node manager's HistoryRead overrides
@@ -287,6 +330,14 @@ if (hasDriver)
builder.Services.AddAkka("otopcua", (ab, sp) =>
{
ab.WithOtOpcUaClusterBootstrap(sp);
// Secret-replication wire protocol → its own serializer, merged only when replication is on so a
// non-replicating node carries no serializer bindings for messages it will never see. Appended
// (Akka.Hosting's fallback merge — the same mode WithOtOpcUaClusterBootstrap uses for the base
// config) rather than a raw Config.WithFallback, which would fight the builder's own assembly.
// Without this the protocol DTOs would round-trip on Akka's default JSON serializer, leaving the
// serializer that carries secret ciphertext an inherited default rather than an explicit choice.
if (SecretsRegistration.IsReplicationEnabled(builder.Configuration))
ab.AddHocon(AkkaSecretsReplication.SerializationConfig, HoconAddMode.Append);
if (hasAdmin)
{
ab.WithOtOpcUaControlPlaneSingletons();
@@ -323,9 +374,99 @@ if (hasAdmin)
builder.Services.AddOtOpcUaAdminClients();
}
// Registered unconditionally: driver-role nodes resolve Layer-B DriverConfig secrets and have no auth/DP/AdminUI.
// Cluster replication is opt-in behind Secrets:Replication:Enabled (default false) — see SecretsRegistration.
builder.Services.AddOtOpcUaSecrets(builder.Configuration);
builder.Services.AddOtOpcUaHealth();
builder.Services.AddOtOpcUaObservability(builder.Configuration);
// ---------------------------------------------------------------------------------------------
// LocalDb sync listener (default-OFF).
//
// DANGER: any explicit Kestrel Listen* call makes Kestrel IGNORE ASPNETCORE_URLS/urls ENTIRELY
// (it logs "Overriding address(es)"). The host has no ConfigureKestrel today and binds solely via
// that configuration, so adding a listener naively would silently unbind the AdminUI + deploy API
// behind Traefik. Everything the host was already asked to serve is therefore re-bound explicitly
// in the same block.
//
// The listener is HTTP/2-ONLY on purpose: the sync client speaks prior-knowledge h2c, which a
// cleartext Http1AndHttp2 endpoint cannot negotiate (there is no ALPN without TLS). Hence a
// dedicated port rather than multiplexing onto the main one.
//
// When LocalDb:SyncListenPort is 0 (the default) none of this runs and URL binding is untouched.
// ---------------------------------------------------------------------------------------------
var syncListenPort = LocalDbRegistration.SyncListenPort(builder.Configuration);
if (hasDriver && syncListenPort > 0)
{
// Mirror Kestrel's own source precedence: URLS wins; else the HTTP_PORTS/HTTPS_PORTS bare-port
// vars; else Kestrel's localhost:5000 default. Missing the HTTP_PORTS leg is not academic — the
// aspnet:8.0+ base images set ASPNETCORE_HTTP_PORTS=8080 as the CONTAINER default in place of
// ASPNETCORE_URLS, so a driver node that never sets URLS is really serving :8080 on all
// interfaces. Falling straight through to localhost:5000 would silently move its health/metrics
// surface to loopback (found on the docker-dev live gate).
var configuredUrls = builder.Configuration["urls"]
?? builder.Configuration["ASPNETCORE_URLS"];
var existingBindings = KestrelHttpBinding.Parse(configuredUrls);
if (existingBindings.Count == 0)
{
var httpPorts = builder.Configuration["ASPNETCORE_HTTP_PORTS"] ?? builder.Configuration["HTTP_PORTS"];
var httpsPorts = builder.Configuration["ASPNETCORE_HTTPS_PORTS"] ?? builder.Configuration["HTTPS_PORTS"];
existingBindings =
[
.. KestrelHttpBinding.ParsePorts(httpPorts, isSecure: false),
.. KestrelHttpBinding.ParsePorts(httpsPorts, isSecure: true),
];
if (existingBindings.Count > 0)
Log.Information(
"LocalDb sync listener enabled; re-binding the HTTP_PORTS surface ({HttpPorts}/{HttpsPorts}) " +
"alongside the sync port.", httpPorts, httpsPorts);
}
// A driver-only Windows-service node gets NO URLS and NO *_PORTS (Install-Services.ps1 sets URLS
// only under $hasAdmin), so "nothing configured" is a real, supported state — not an error.
// Kestrel's own default is localhost:5000; re-state it explicitly, because taking over
// configuration means taking over the default too. Health probes hit this port.
if (existingBindings.Count == 0)
{
existingBindings = [new KestrelHttpBinding("localhost", 5000, IsSecure: false)];
Log.Information(
"LocalDb sync listener enabled with no URLs configured; re-binding Kestrel's default " +
"http://localhost:5000 alongside the sync port.");
}
// Re-binding an https endpoint would need its certificate configuration replayed too, which
// this host does not model (TLS is terminated at Traefik). Rather than silently serve it
// without TLS — or drop it — refuse to take over Kestrel at all and leave the existing
// surface exactly as it was. The operator gets a loud, actionable error instead of an
// AdminUI that stopped answering.
if (existingBindings.Any(b => b.IsSecure))
{
Log.Error(
"LocalDb:SyncListenPort is set to {Port} but this host serves HTTPS endpoint(s) ({Urls}). " +
"Binding the sync listener requires re-binding every existing endpoint explicitly, and the " +
"HTTPS certificate configuration cannot be replayed safely. The sync listener is DISABLED; " +
"terminate TLS upstream (as the docker-dev rig does) or leave replication off on this node.",
syncListenPort, configuredUrls);
syncListenPort = 0;
}
else
{
builder.WebHost.ConfigureKestrel(kestrel =>
{
foreach (var binding in existingBindings)
binding.Apply(kestrel);
kestrel.ListenAnyIP(syncListenPort, o => o.Protocols = HttpProtocols.Http2);
});
Log.Information(
"LocalDb sync listener bound on :{SyncPort} (h2c); re-bound existing endpoint(s) {Urls}.",
syncListenPort, configuredUrls ?? "http://localhost:5000 (Kestrel default)");
}
}
var app = builder.Build();
// AddZbSerilog registers Serilog as the MEL logging provider but does NOT assign the static
@@ -356,6 +497,15 @@ if (hasAdmin)
app.MapOtOpcUaDeployApi(app.Configuration);
}
// Passive LocalDb sync endpoint. Gated on the same port check that bound the listener, so a
// default-OFF or admin-only graph carries no sync surface whatsoever. Mapping it would be safe
// even unauthenticated — LocalDbSyncAuthInterceptor fail-closes with no ApiKey configured — but
// not mapping it at all is the stronger default.
if (hasDriver && syncListenPort > 0)
{
app.MapZbLocalDbSync();
}
app.MapOtOpcUaHealth();
app.MapOtOpcUaMetrics();
src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj
+10
View File
@@ -33,6 +33,16 @@
<PackageReference Include="ZB.MOM.WW.Telemetry" />
<PackageReference Include="ZB.MOM.WW.Telemetry.Serilog" />
<PackageReference Include="ZB.MOM.WW.Configuration" />
<!-- LocalDb: node-local SQLite cache (AddZbLocalDb) plus the replication sync engine and
its passive gRPC endpoint (AddZbLocalDbReplication / MapZbLocalDbSync). Grpc.AspNetCore
is required because the sync endpoint is served by this host's Kestrel, not dialled by
it — Grpc.Net.Client (already pinned) only covers the initiator side. -->
<PackageReference Include="Grpc.AspNetCore" />
<PackageReference Include="ZB.MOM.WW.LocalDb" />
<PackageReference Include="ZB.MOM.WW.LocalDb.Replication" />
<PackageReference Include="ZB.MOM.WW.Secrets" />
<PackageReference Include="ZB.MOM.WW.Secrets.Abstractions" />
<PackageReference Include="ZB.MOM.WW.Secrets.Replicator.AkkaDotNet" />
</ItemGroup>
<ItemGroup>
src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json
+27 -3
View File
@@ -11,12 +11,27 @@
"DisableLogin": false
}
},
"Secrets": {
"SqlitePath": "otopcua-secrets.db",
"MasterKey": {
"Source": "Environment",
"EnvVarName": "ZB_SECRETS_MASTER_KEY"
},
"RunMigrationsOnStartup": true,
"ResolveCacheTtl": "00:00:30",
"Replication": {
"_comment": "Peer-to-peer secret replication over the Akka cluster. OPT-IN: Enabled=false keeps the plain local SQLite store on every node. KNOWN NON-FUNCTIONAL against ZB.MOM.WW.Secrets.Replicator.AkkaDotNet 0.2.0 — that version's ISecretReplicator registration is shadowed by the NoOp one AddZbSecrets registers first, so enabling this decorates the store but replicates nothing and starts no actor. Do not enable until the library is fixed. All nodes must also share the same KEK.",
"Enabled": false,
"AnnounceInterval": "00:00:30",
"ActorName": "zb-secret-replication"
}
},
"ServerHistorian": {
"_comment": "Server-side HistoryRead backend (the ZB.MOM.WW.HistorianGateway gRPC client). Disabled => NullHistorianDataSource (historized nodes return GoodNoData). The gateway must run RuntimeDb:EventReadsEnabled=true for alarm-history ReadEvents, and the API key must carry historian:read + historian:write + historian:tags:write scopes.",
"Enabled": false,
"Endpoint": "",
"ApiKey": "",
"_ApiKeyComment": "NEVER commit a real key. Supply via the environment variable ServerHistorian__ApiKey.",
"_ApiKeyComment": "NEVER commit a real key. Supply via env var ServerHistorian__ApiKey, either as a literal or as a ${secret:otopcua/historian/api-key} token resolved fail-closed by the pre-host secrets expander.",
"UseTls": true,
"AllowUntrustedServerCertificate": false,
"CaCertificatePath": null,
@@ -37,13 +52,22 @@
"MaxBackoffSeconds": 30
},
"AlarmHistorian": {
"_comment": "Durable SQLite store-and-forward alarm sink. Drains alarm events to the ServerHistorian gateway's SendEvent path; the downstream connection (endpoint/key/TLS) is sourced from the ServerHistorian section.",
"_comment": "Durable store-and-forward alarm sink. Buffers into the node's consolidated LocalDb (see the LocalDb section) so the queue replicates to the redundant pair peer, and drains alarm events to the ServerHistorian gateway's SendEvent path from whichever node holds the Primary role. The downstream connection (endpoint/key/TLS) is sourced from the ServerHistorian section.",
"Enabled": false,
"DatabasePath": "alarm-historian.db",
"DrainIntervalSeconds": 5,
"Capacity": 1000000,
"DeadLetterRetentionDays": 30
},
"LocalDb": {
"_comment": "Node-local SQLite store caching the deployed-configuration artifact, so a driver node can boot from its last-known-good config when central SQL is unreachable. Registered on driver-role nodes only (see LocalDbRegistration). Storage is unconditional; replication to the redundant pair peer is default-OFF.",
"Path": "./data/otopcua-localdb.db",
"_SyncListenPortComment": "Port for the dedicated cleartext-HTTP/2 (h2c) listener serving the passive sync endpoint. 0 = no listener bound and the endpoint is not mapped. Must be a port distinct from the main HTTP listener: prior-knowledge h2c cannot share a cleartext Http1AndHttp2 port.",
"SyncListenPort": 0,
"Replication": {
"_comment": "Empty = passive/off. Set PeerAddress on exactly ONE node of the pair (the stream is bidirectional, so the other side still pushes its own deltas back). ApiKey must be BYTE-IDENTICAL on both nodes — the host interceptor fail-closes, so a typo silently stops convergence rather than degrading to unauthenticated. Supply it via ${secret:...} in production, never committed.",
"_MaxBatchSizeComment": "Row count, NOT bytes, against gRPC's 4 MB default message cap. Artifact chunk rows are ~171 KB base64, so keep this at 16 on the pair (16 x 171 KB ~= 2.7 MB worst case)."
}
},
"Deployment": {
"_comment": "R2-11 (05/CONV-2): deploy-gate TagConfig strictness. Warn (default) = non-blocking warnings logged + appended to the deployment result message; Error = a config with a typo'd enum or unparseable TagConfig is rejected at the draft gate. Running servers are untouched; the gate only sees re-deploys.",
"TagConfigValidationMode": "Warn"
src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs
+109 -3
View File
@@ -497,6 +497,16 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
condition.SetSeverity(SystemContext, MapSeverity(state.Severity));
condition.Message.Value = new LocalizedText(state.Message);
// #477 — project the source-data quality. A Good→Bad bucket change is a genuine condition
// change (it's in the delta-gate above), so it fires a Part 9 event carrying the new Quality;
// it never touches Active/Acked/Retain (annotation only). Quality is an optional Part 9 child —
// null-guard it like Confirmed/Shelving in case a leaner SDK child set omits it.
if (condition.Quality is not null)
{
condition.Quality.Value = StatusFromQuality(state.Quality);
if (condition.Quality.SourceTimestamp is not null) condition.Quality.SourceTimestamp.Value = sourceTimestampUtc;
}
// Part 9: retain the condition while it is active OR unacknowledged so a client's
// ConditionRefresh replays it. The event firing below also depends on this Retain being
// correct (a non-retained inactive+acked condition still fires its transition event, but
@@ -535,6 +545,50 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
}
}
/// <summary>
/// #477 Layer 2 — apply a source-data quality annotation to a materialised condition, OUT OF BAND
/// from any alarm transition. Used by the driver-connectivity path (comms lost → Bad, restored →
/// Good) so a native condition whose device is unreachable stops reporting the accidentally-Good
/// default. This sets ONLY <see cref="ConditionState.Quality"/> — it never touches
/// Active / Acked / Severity / Retain (a comms-lost active alarm must stay active). A change in the
/// quality bucket is a genuine Part 9 condition change, so it fires one condition event carrying the
/// new Quality; an unchanged bucket suppresses (no spurious event). Unknown / unmaterialised node ⇒
/// safe no-op (a mid-rebuild race must not fault a connectivity update), mirroring the other writes.
/// </summary>
/// <param name="alarmNodeId">The condition node id (RawPath for a native alarm).</param>
/// <param name="quality">The source-data quality to annotate.</param>
/// <param name="sourceTimestampUtc">Timestamp of the connectivity transition in UTC.</param>
/// <param name="realm">The condition's address-space realm.</param>
public void WriteAlarmQuality(string alarmNodeId, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm)
{
ArgumentException.ThrowIfNullOrEmpty(alarmNodeId);
EnsureAddressSpaceCreated();
var key = MapKey(realm, alarmNodeId);
lock (Lock)
{
// Only materialised conditions carry a Quality child; a bare value variable or a missing node is
// a no-op (the connectivity fan-out visits every condition the driver owns, some of which a
// concurrent rebuild may have just cleared).
if (!_alarmConditions.TryGetValue(key, out var condition) || condition.Quality is null)
return;
var newCode = StatusFromQuality(quality);
var changed = condition.Quality.Value.Code != newCode.Code;
condition.Quality.Value = newCode;
if (condition.Quality.SourceTimestamp is not null) condition.Quality.SourceTimestamp.Value = sourceTimestampUtc;
// Fire ONLY on a real bucket change so a steady-state connectivity re-assert doesn't spam events.
if (changed)
{
condition.Time.Value = sourceTimestampUtc;
condition.ReceiveTime.Value = sourceTimestampUtc;
ReportConditionEvent(condition, sourceTimestampUtc);
}
}
}
/// <summary>
/// Fire a real OPC UA Part 9 condition event for one engine-driven state transition on a
/// materialised <see cref="AlarmConditionState"/>. The caller MUST already hold <c>Lock</c> and
@@ -650,7 +704,11 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
bool Enabled,
AlarmShelvingKind Shelving,
ushort MappedSeverity,
string Message);
string Message,
// #477 — the source-data quality bucket. Included so a quality-only transition (e.g. a device
// going comms-lost: Good→Bad with no state change) is a genuine delta and fires a Part 9 event.
// StatusCode has value equality, so the record struct's == still holds.
StatusCode Quality);
/// <summary>Decide whether a <see cref="WriteAlarmCondition"/> projection is a genuine state change
/// (and so should fire a Part 9 condition event) by comparing the node's pre-projection state to the
@@ -676,7 +734,10 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
Enabled: condition.EnabledState?.Id?.Value ?? true,
Shelving: ReadShelvingKind(condition),
MappedSeverity: condition.Severity?.Value ?? (ushort)0,
Message: condition.Message?.Value?.Text ?? string.Empty);
Message: condition.Message?.Value?.Text ?? string.Empty,
// Optional Part 9 child; a node without it reads as Good (matching the accidentally-Good default
// and ToConditionDelta's fold), so a snapshot Quality can't create a phantom delta against it.
Quality: condition.Quality?.Value ?? StatusCodes.Good);
/// <summary>Build the gate-relevant slice from the incoming snapshot, normalising the two fields that
/// the node stores in a derived form: Severity is run through <see cref="MapSeverity"/> so it matches
@@ -694,7 +755,10 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// node's read-back default (Unshelved).
Shelving: condition.ShelvingState is not null ? state.Shelving : AlarmShelvingKind.Unshelved,
MappedSeverity: (ushort)MapSeverity(state.Severity),
Message: state.Message ?? string.Empty);
Message: state.Message ?? string.Empty,
// If the node has no Quality child, WriteAlarmCondition's projection is a no-op there; fold to the
// node's read-back default (Good) so a snapshot Quality can't register a spurious delta.
Quality: condition.Quality is not null ? StatusFromQuality(state.Quality) : StatusCodes.Good);
/// <summary>Map the live shelving state machine's CurrentState back to our 3-way
/// <see cref="AlarmShelvingKind"/> by matching its well-known Part 9 state object id. Any node without
@@ -799,6 +863,48 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// (Real-server finding from the T14 integration test — not obvious from the SDK notes.)
if (alarm.BranchId is not null) alarm.BranchId.Value = NodeId.Null;
// #473 — the mandatory BaseEventType identity fields. Create() builds these three children from
// the type's embedded definition but leaves them UNSET (the init string declares them mandatory
// with NO default), and nothing downstream fills them in: the auto-filling
// BaseEventState.Initialize(context, source, severity, message) overload is only used for
// transient events, and ReportEvent / InstanceStateSnapshot copy the children verbatim. Unset
// here ⇒ null on the wire on EVERY condition event, so a client cannot attribute the alarm.
// SourceNode — the condition's OWN NodeId (== ConditionId): the condition IS the source. An
// alarm-bearing raw tag materialises ONLY this condition, with no sibling value
// variable (see AddressSpaceApplier's alarm branch), so there is no other node.
// SourceName — the same identifying id string (RawPath native / ScriptedAlarmId scripted).
// Deliberately the UNIQUE id, NOT the leaf: the leaf collides across devices and
// is already carried by ConditionName below. See docs/AlarmTracking.md.
alarm.EventType.Value = alarm.TypeDefinitionId;
alarm.SourceNode.Value = alarm.NodeId; // Create() assigned this above; do not rebuild it
alarm.SourceName.Value = alarmNodeId;
// #475 — the mandatory ConditionType classification fields, unset by Create() for the same reason as
// the fields above (mandatory, no default, nothing downstream synthesises them) ⇒ NodeId.Null + empty
// text on the wire, which buckets every alarm as unclassified in a Part 9 HMI.
// BaseConditionClassType is Part 9's "no class modelled" value and is the honest report: we hold no
// classification at this seam. Deliberately NOT ProcessConditionClassType (the SDK sample's pick) — it
// would assert a classification we cannot back, and would be actively wrong for a Galaxy alarm whose
// upstream category is Safety/Diagnostics. Real per-alarm classification needs the driver's
// AlarmCategory, which today exists only on the runtime AlarmEventArgs transition and not on the
// authored composition this deploy-time seam sees — a separate feature, not a default picked here.
if (alarm.ConditionClassId is not null) alarm.ConditionClassId.Value = ObjectTypeIds.BaseConditionClassType;
if (alarm.ConditionClassName is not null) alarm.ConditionClassName.Value = new LocalizedText("BaseConditionClass");
// #477 — ConditionType.Quality (the quality of the condition's source data). Create() leaves it
// UNSET, and default(StatusCode) == StatusCodes.Good (0x0), so an unassigned Quality reports Good
// unconditionally — a wrong VALUE (not a null), which hides a comms-lost source. A NATIVE condition
// has no data yet at materialise (its driver hasn't confirmed connectivity), so it starts
// BadWaitingForInitialData — the same "no driver data yet" convention value variables use — and is
// driven Good by the driver-connectivity path (DriverHostActor → ProjectQuality) once Connected. A
// SCRIPTED condition is script-computed and always live in v1, so it starts Good. Quality is a pure
// annotation: it NEVER alters Active/Acked/Retain (a comms-lost active alarm must stay active).
if (alarm.Quality is not null)
{
alarm.Quality.Value = isNative ? StatusCodes.BadWaitingForInitialData : StatusCodes.Good;
if (alarm.Quality.SourceTimestamp is not null) alarm.Quality.SourceTimestamp.Value = DateTime.UtcNow;
}
// Initial state via the SDK setters (T14: basic state only, NO event firing).
alarm.SetEnableState(SystemContext, true);
alarm.SetActiveState(SystemContext, false);
src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/SdkAddressSpaceSink.cs
+3
View File
@@ -28,6 +28,9 @@ public sealed class SdkAddressSpaceSink : IOpcUaAddressSpaceSink, ISurgicalAddre
public void WriteAlarmCondition(string alarmNodeId, AlarmConditionSnapshot state, DateTime sourceTimestampUtc, AddressSpaceRealm realm)
=> _nodeManager.WriteAlarmCondition(alarmNodeId, state, sourceTimestampUtc, realm);
public void WriteAlarmQuality(string alarmNodeId, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm)
=> _nodeManager.WriteAlarmQuality(alarmNodeId, quality, sourceTimestampUtc, realm);
/// <inheritdoc />
public void MaterialiseAlarmCondition(string alarmNodeId, string equipmentNodeId, string displayName, string alarmType, int severity, AddressSpaceRealm realm, bool isNative = false)
=> _nodeManager.MaterialiseAlarmCondition(alarmNodeId, equipmentNodeId, displayName, alarmType, severity, realm, isNative);
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/DeploymentCacheSchema.cs
+70
View File
@@ -0,0 +1,70 @@
using Microsoft.Data.Sqlite;
namespace ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
/// <summary>
/// DDL for the node-local deployment-artifact cache: the chunked artifact table plus the
/// per-cluster current-deployment pointer.
/// </summary>
/// <remarks>
/// <para>
/// Deliberately depends on nothing but <see cref="SqliteConnection"/> so it can be applied
/// to any connection — the host's <c>LocalDbSetup.OnReady</c> in production, and a bare
/// connection in a test — without dragging the DI graph along.
/// </para>
/// <para>
/// <b>Why the artifact is chunked base64 TEXT and not one BLOB row.</b> Two independent
/// constraints force it. <c>RegisterReplicated</c> rejects BLOB columns outright; and the
/// replication engine batches by <i>row count</i> against gRPC's 4 MB default message cap,
/// so a single multi-megabyte artifact row would simply never be deliverable — at any
/// batch size. A 128 KiB raw chunk is ≈ 171 KB once base64-encoded, which keeps a
/// worst-case batch comfortably inside the cap.
/// </para>
/// <para>
/// <b>Why no autoincrement PKs.</b> Convergence is last-writer-wins keyed on the primary
/// key. Two nodes independently allocating rowid 7 for different rows would silently
/// overwrite one another. Every key here is TEXT or a composite of caller-supplied values.
/// </para>
/// </remarks>
public static class DeploymentCacheSchema
{
/// <summary>Table holding the artifact bytes, split into base64 chunks.</summary>
public const string ArtifactsTable = "deployment_artifacts";
/// <summary>Table holding one current-deployment pointer per cluster.</summary>
public const string PointerTable = "deployment_pointer";
/// <summary>
/// Creates both cache tables if they do not already exist. Idempotent.
/// </summary>
/// <param name="connection">
/// An already-open connection. <c>ILocalDb.CreateConnection()</c> hands out open,
/// pragma-configured connections — do not call <c>Open()</c> on one.
/// </param>
public static void Apply(SqliteConnection connection)
{
ArgumentNullException.ThrowIfNull(connection);
using var cmd = connection.CreateCommand();
cmd.CommandText = """
CREATE TABLE IF NOT EXISTS deployment_artifacts (
deployment_id TEXT NOT NULL,
chunk_index INTEGER NOT NULL,
cluster_id TEXT NOT NULL,
revision_hash TEXT NOT NULL,
chunk_count INTEGER NOT NULL,
chunk_base64 TEXT NOT NULL,
cached_at_utc TEXT NOT NULL,
PRIMARY KEY (deployment_id, chunk_index)
);
CREATE TABLE IF NOT EXISTS deployment_pointer (
cluster_id TEXT NOT NULL PRIMARY KEY,
deployment_id TEXT NOT NULL,
revision_hash TEXT NOT NULL,
artifact_sha256 TEXT NOT NULL,
applied_at_utc TEXT NOT NULL
);
""";
cmd.ExecuteNonQuery();
}
}
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/IDeploymentArtifactCache.cs
+68
View File
@@ -0,0 +1,68 @@
namespace ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
/// <summary>
/// Node-local durable cache of the deployment artifact this node last applied.
/// </summary>
/// <remarks>
/// <para>
/// Exists so a driver node can boot into its last-known-good address space when the central
/// configuration database is unreachable. Without it, a control-plane outage that outlives a
/// node restart leaves that node with no address space at all — the plant loses visibility
/// for a reason that has nothing to do with the plant.
/// </para>
/// </remarks>
public interface IDeploymentArtifactCache
{
/// <summary>
/// Persists <paramref name="artifact"/> as the cluster's current deployment.
/// </summary>
/// <remarks>
/// <para>
/// Idempotent per <paramref name="deploymentId"/>: re-storing the same deployment
/// replaces its chunks rather than accumulating orphans. Retention is bounded to the two
/// newest deployments per cluster, so a long-lived node cannot grow its local database
/// without limit.
/// </para>
/// </remarks>
Task StoreAsync(string clusterId, string deploymentId, string revisionHash,
byte[] artifact, CancellationToken ct = default);
/// <summary>
/// Reads the cached artifact for <paramref name="clusterId"/>, or <see langword="null"/>
/// when nothing is cached or the cached bytes fail their integrity check.
/// </summary>
/// <remarks>
/// <para>
/// A failed integrity check is deliberately indistinguishable from a miss. Booting a
/// plant from a truncated address space is strictly worse than booting from none: the
/// missing half looks like a deliberate configuration rather than an error.
/// </para>
/// </remarks>
Task<CachedDeploymentArtifact?> GetCurrentAsync(string clusterId, CancellationToken ct = default);
/// <summary>
/// Reads the single cached pointer without knowing the cluster id.
/// </summary>
/// <remarks>
/// <para>
/// The boot seam needs this. A node's cluster id is only derivable from an artifact it
/// has already loaded, or from the central database — which is precisely what is
/// unreachable in the scenario this cache exists to survive. So the cold path reads the
/// newest pointer unkeyed.
/// </para>
/// <para>
/// More than one pointer row means the node was re-homed between clusters. The newest
/// wins, but the choice is logged as a warning naming both clusters — a node silently
/// booting a neighbouring cluster's configuration is exactly the failure that must never
/// be quiet.
/// </para>
/// </remarks>
Task<CachedDeploymentArtifact?> GetCurrentUnkeyedAsync(CancellationToken ct = default);
}
/// <summary>
/// A deployment artifact recovered from the node-local cache, with the metadata needed to decide
/// whether it is still current once the control plane is reachable again.
/// </summary>
public sealed record CachedDeploymentArtifact(
string DeploymentId, string RevisionHash, byte[] Artifact, DateTimeOffset AppliedAtUtc);
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/LocalDbDeploymentArtifactCache.cs
+400
View File
@@ -0,0 +1,400 @@
using System.Globalization;
using System.Security.Cryptography;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.LocalDb;
namespace ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
/// <summary>
/// <see cref="IDeploymentArtifactCache"/> backed by the replicated LocalDb deployment-cache
/// tables.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why the artifact is stored as base64 chunks.</b> See
/// <see cref="DeploymentCacheSchema"/> — replication rejects BLOB columns and batches by row
/// count against gRPC's message cap, so a whole-artifact row could never be delivered.
/// Everything in this class that looks like ceremony (chunk indices, a chunk count, a
/// separate SHA over the raw bytes) exists to make that split safely reversible.
/// </para>
/// </remarks>
public sealed class LocalDbDeploymentArtifactCache : IDeploymentArtifactCache
{
/// <summary>
/// Raw bytes per chunk, before base64 expansion.
/// </summary>
/// <remarks>
/// 128 KiB raw is ≈ 171 KB encoded, which keeps a worst-case replication batch inside gRPC's
/// 4 MB default cap with room to spare.
/// </remarks>
private const int ChunkSize = 128 * 1024;
/// <summary>
/// How many deployments per cluster survive a store.
/// </summary>
/// <remarks>
/// Two, not one: the previous artifact is what a rollback would need, and keeping it costs
/// one artifact's worth of disk. Three would only add a generation nobody rolls back to.
/// </remarks>
private const int RetainedDeployments = 2;
private readonly ILocalDb _db;
private readonly ILogger<LocalDbDeploymentArtifactCache> _logger;
public LocalDbDeploymentArtifactCache(ILocalDb db, ILogger<LocalDbDeploymentArtifactCache> logger)
{
ArgumentNullException.ThrowIfNull(db);
ArgumentNullException.ThrowIfNull(logger);
_db = db;
_logger = logger;
}
/// <inheritdoc/>
public async Task StoreAsync(string clusterId, string deploymentId, string revisionHash,
byte[] artifact, CancellationToken ct = default)
{
ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
ArgumentException.ThrowIfNullOrWhiteSpace(deploymentId);
ArgumentException.ThrowIfNullOrWhiteSpace(revisionHash);
ArgumentNullException.ThrowIfNull(artifact);
var sha = Convert.ToHexString(SHA256.HashData(artifact));
var cachedAtUtc = FormatTimestamp(DateTimeOffset.UtcNow);
var chunkCount = (artifact.Length + ChunkSize - 1) / ChunkSize;
if (await IsAlreadyCachedAsync(clusterId, deploymentId, revisionHash, sha, chunkCount, ct))
{
_logger.LogDebug(
"Deployment {DeploymentId} (revision {RevisionHash}) is already cached intact for cluster {ClusterId}; skipping the re-store.",
deploymentId, revisionHash, clusterId);
return;
}
// One transaction for the whole store. A pointer that commits without its chunks — or
// chunks that commit without the pointer — is a cache that reads back as a corrupt hit
// rather than a clean miss, which is the one outcome this type exists to prevent.
await using var tx = await _db.BeginTransactionAsync(ct);
// Delete-then-insert rather than upsert-per-chunk: a re-store with FEWER chunks than last
// time would otherwise leave the old tail behind, and those orphans read back as a
// chunk-count mismatch forever.
await tx.ExecuteAsync(
"DELETE FROM deployment_artifacts WHERE deployment_id = @DeploymentId",
new { DeploymentId = deploymentId },
ct);
for (var index = 0; index < chunkCount; index++)
{
var offset = index * ChunkSize;
var length = Math.Min(ChunkSize, artifact.Length - offset);
await tx.ExecuteAsync(
"""
INSERT INTO deployment_artifacts
(deployment_id, chunk_index, cluster_id, revision_hash, chunk_count,
chunk_base64, cached_at_utc)
VALUES
(@DeploymentId, @ChunkIndex, @ClusterId, @RevisionHash, @ChunkCount,
@ChunkBase64, @CachedAtUtc)
""",
new
{
DeploymentId = deploymentId,
ChunkIndex = index,
ClusterId = clusterId,
RevisionHash = revisionHash,
ChunkCount = chunkCount,
ChunkBase64 = Convert.ToBase64String(artifact, offset, length),
CachedAtUtc = cachedAtUtc,
},
ct);
}
await tx.ExecuteAsync(
"""
INSERT INTO deployment_pointer
(cluster_id, deployment_id, revision_hash, artifact_sha256, applied_at_utc)
VALUES
(@ClusterId, @DeploymentId, @RevisionHash, @Sha, @AppliedAtUtc)
ON CONFLICT(cluster_id) DO UPDATE SET
deployment_id = excluded.deployment_id,
revision_hash = excluded.revision_hash,
artifact_sha256 = excluded.artifact_sha256,
applied_at_utc = excluded.applied_at_utc
""",
new
{
ClusterId = clusterId,
DeploymentId = deploymentId,
RevisionHash = revisionHash,
Sha = sha,
AppliedAtUtc = cachedAtUtc,
},
ct);
// Prune inside the same transaction so the cache is never briefly unbounded.
//
// The `deployment_id <> @DeploymentId` clause is load-bearing, not belt-and-braces: it makes
// "the deployment the pointer names is always present" a structural invariant instead of a
// consequence of clock resolution. Without it, three stores landing inside one timestamp
// tick fall through to the `deployment_id DESC` tiebreak — and since real deployment ids are
// GUIDs, that ordering is effectively random, so the row just written can lose. The pointer
// would then name chunks that no longer exist, which reads back as a cache miss on every
// subsequent boot until the next deploy overwrites it. Silent and permanent: exactly the
// failure this cache exists to prevent.
await tx.ExecuteAsync(
$"""
DELETE FROM deployment_artifacts
WHERE cluster_id = @ClusterId
AND deployment_id <> @DeploymentId
AND deployment_id NOT IN (
SELECT deployment_id
FROM deployment_artifacts
WHERE cluster_id = @ClusterId
GROUP BY deployment_id
ORDER BY MAX(cached_at_utc) DESC, deployment_id DESC
LIMIT {RetainedDeployments}
)
""",
new { ClusterId = clusterId, DeploymentId = deploymentId },
ct);
await tx.CommitAsync(ct);
}
/// <summary>
/// Whether this exact artifact is already cached <b>and readable</b> for the cluster, so
/// storing it again would rewrite every row to its current value.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why this exists.</b> Every write to a replicated table fires a capture trigger and
/// mints an oplog row. Re-caching an artifact the node already holds — which is what a
/// restart's boot-from-cache and every <c>RestoreApplied</c> recovery does — would
/// therefore cost a delete plus an insert per chunk plus a pointer update, all carrying
/// values identical to what is already there. On a replicating pair that is pure churn;
/// on an unpaired (default-OFF) node there is no peer to ack those rows, so they
/// accumulate in the oplog until the library's age cap prunes them. The Phase 1 live gate
/// saw exactly that (check 8).
/// </para>
/// <para>
/// <b>Identity is over the bytes, not the ids.</b> The pointer must match on
/// <c>artifact_sha256</c> as well as the deployment id and revision hash: a re-composed
/// artifact can legitimately carry the same ids with different bytes, and keeping the
/// stale copy would serve a boot-from-cache node the wrong address space while every
/// id-shaped signal claimed it was right.
/// </para>
/// <para>
/// <b>The chunk count is load-bearing.</b> A pointer can name a deployment whose chunks
/// are missing or partial — a half-replicated artifact, or a prune that raced the
/// pointer. Skipping on the pointer alone would make that state permanent, because the
/// re-store that repairs it is the call being skipped.
/// </para>
/// <para>
/// <b>Accepted consequence:</b> a skipped store leaves <c>applied_at_utc</c> at the
/// moment the artifact was <i>first</i> cached rather than last confirmed. Refreshing it
/// would mint the very oplog row this check exists to avoid. Nothing reads it as a
/// liveness signal; retention ranks distinct deployments, which are unaffected.
/// </para>
/// </remarks>
private async Task<bool> IsAlreadyCachedAsync(
string clusterId, string deploymentId, string revisionHash, string sha, int chunkCount,
CancellationToken ct)
{
var matches = await _db.QueryAsync(
"""
SELECT (
SELECT COUNT(*)
FROM deployment_artifacts
WHERE cluster_id = @ClusterId
AND deployment_id = @DeploymentId
AND chunk_count = @ChunkCount
)
FROM deployment_pointer
WHERE cluster_id = @ClusterId
AND deployment_id = @DeploymentId
AND revision_hash = @RevisionHash
AND artifact_sha256 = @Sha
""",
r => r.GetInt32(0),
new
{
ClusterId = clusterId,
DeploymentId = deploymentId,
RevisionHash = revisionHash,
Sha = sha,
ChunkCount = chunkCount,
},
ct);
return matches.Count == 1 && matches[0] == chunkCount;
}
/// <inheritdoc/>
public async Task<CachedDeploymentArtifact?> GetCurrentAsync(
string clusterId, CancellationToken ct = default)
{
ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
var pointers = await _db.QueryAsync(
"""
SELECT deployment_id, revision_hash, artifact_sha256, applied_at_utc
FROM deployment_pointer
WHERE cluster_id = @ClusterId
""",
ReadPointer,
new { ClusterId = clusterId },
ct);
if (pointers.Count == 0)
{
_logger.LogDebug("No cached deployment pointer for cluster {ClusterId}.", clusterId);
return null;
}
return await ReassembleAsync(clusterId, pointers[0], ct);
}
/// <inheritdoc/>
public async Task<CachedDeploymentArtifact?> GetCurrentUnkeyedAsync(CancellationToken ct = default)
{
// applied_at_utc is round-trip ISO-8601 UTC, so ordering it as TEXT is chronological —
// that is the reason the format is pinned rather than left to the current culture.
var pointers = await _db.QueryAsync(
"""
SELECT cluster_id, deployment_id, revision_hash, artifact_sha256, applied_at_utc
FROM deployment_pointer
ORDER BY applied_at_utc DESC
""",
r => (ClusterId: r.GetString(0), Pointer: new PointerRow(
DeploymentId: r.GetString(1),
RevisionHash: r.GetString(2),
ArtifactSha256: r.GetString(3),
AppliedAtUtc: r.GetString(4))),
parameters: null,
ct);
if (pointers.Count == 0)
{
_logger.LogDebug("No cached deployment pointer of any cluster.");
return null;
}
if (pointers.Count > 1)
{
// The node was re-homed between clusters. Booting the newest is the only defensible
// choice, but it must never be a silent one — a wrong-cluster address space presents as
// a plausible configuration, so the operator needs the cluster names to spot it.
_logger.LogWarning(
"Local deployment cache holds pointers for {PointerCount} clusters ({ClusterIds}); " +
"booting the newest ({ChosenClusterId}). This node appears to have been re-homed — " +
"clear the stale cache if that is unexpected.",
pointers.Count,
string.Join(", ", pointers.Select(p => p.ClusterId)),
pointers[0].ClusterId);
}
return await ReassembleAsync(pointers[0].ClusterId, pointers[0].Pointer, ct);
}
/// <summary>
/// Rebuilds the artifact the pointer names, returning <see langword="null"/> unless every
/// integrity check passes.
/// </summary>
private async Task<CachedDeploymentArtifact?> ReassembleAsync(
string clusterId, PointerRow pointer, CancellationToken ct)
{
var chunks = await _db.QueryAsync(
"""
SELECT chunk_count, chunk_base64
FROM deployment_artifacts
WHERE deployment_id = @DeploymentId
ORDER BY chunk_index
""",
r => (ChunkCount: r.GetInt32(0), Base64: r.GetString(1)),
new { pointer.DeploymentId },
ct);
// The chunk_count carried on every row is what makes a partial replica detectable. Without
// it a missing tail chunk is indistinguishable from a shorter artifact.
var expectedChunkCount = chunks.Count > 0 ? chunks[0].ChunkCount : 0;
if (chunks.Count != expectedChunkCount)
{
_logger.LogWarning(
"Cached deployment {DeploymentId} for cluster {ClusterId} is incomplete: found " +
"{FoundChunks} of {ExpectedChunks} chunks. Treating as a cache miss.",
pointer.DeploymentId, clusterId, chunks.Count, expectedChunkCount);
return null;
}
byte[] artifact;
try
{
artifact = Decode(chunks.Select(c => c.Base64));
}
catch (FormatException ex)
{
_logger.LogWarning(
ex,
"Cached deployment {DeploymentId} for cluster {ClusterId} has a chunk that is not " +
"valid base64. Treating as a cache miss.",
pointer.DeploymentId, clusterId);
return null;
}
var actualSha = Convert.ToHexString(SHA256.HashData(artifact));
if (!string.Equals(actualSha, pointer.ArtifactSha256, StringComparison.OrdinalIgnoreCase))
{
_logger.LogWarning(
"Cached deployment {DeploymentId} for cluster {ClusterId} failed its SHA-256 check " +
"(expected {ExpectedSha}, computed {ActualSha}). Treating as a cache miss.",
pointer.DeploymentId, clusterId, pointer.ArtifactSha256, actualSha);
return null;
}
if (!DateTimeOffset.TryParse(pointer.AppliedAtUtc, CultureInfo.InvariantCulture,
DateTimeStyles.RoundtripKind, out var appliedAtUtc))
{
_logger.LogWarning(
"Cached deployment {DeploymentId} for cluster {ClusterId} has an unparseable " +
"applied_at_utc ({AppliedAtUtc}). Treating as a cache miss.",
pointer.DeploymentId, clusterId, pointer.AppliedAtUtc);
return null;
}
return new CachedDeploymentArtifact(
pointer.DeploymentId, pointer.RevisionHash, artifact, appliedAtUtc);
}
private static byte[] Decode(IEnumerable<string> chunksInOrder)
{
using var buffer = new MemoryStream();
foreach (var chunk in chunksInOrder)
{
var bytes = Convert.FromBase64String(chunk);
buffer.Write(bytes, 0, bytes.Length);
}
return buffer.ToArray();
}
private static PointerRow ReadPointer(Microsoft.Data.Sqlite.SqliteDataReader reader)
=> new(
DeploymentId: reader.GetString(0),
RevisionHash: reader.GetString(1),
ArtifactSha256: reader.GetString(2),
AppliedAtUtc: reader.GetString(3));
/// <summary>
/// Round-trip ("O") UTC, so lexicographic ordering of the stored TEXT is chronological
/// ordering — the retention query sorts on it directly.
/// </summary>
private static string FormatTimestamp(DateTimeOffset value)
=> value.ToUniversalTime().ToString("O", CultureInfo.InvariantCulture);
private sealed record PointerRow(
string DeploymentId, string RevisionHash, string ArtifactSha256, string AppliedAtUtc);
}
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs
+419 -21
View File
@@ -23,6 +23,8 @@ using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
using ZB.MOM.WW.OtOpcUa.Core.VirtualTags;
using ZB.MOM.WW.OtOpcUa.OpcUaServer;
using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
using ZB.MOM.WW.OtOpcUa.Runtime.Redundancy;
using ZB.MOM.WW.OtOpcUa.Runtime.ScriptedAlarms;
using ZB.MOM.WW.OtOpcUa.Runtime.VirtualTags;
using CommonsNodeId = ZB.MOM.WW.OtOpcUa.Commons.Types.NodeId;
@@ -57,6 +59,31 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
/// <summary>Publishing interval handed to each driver's SubscribeBulk pass after an apply.</summary>
private static readonly TimeSpan SubscriptionPublishingInterval = TimeSpan.FromSeconds(1);
/// <summary>
/// Cache key used when an artifact carries no cluster scoping (single-cluster or unscoped
/// deployments). A literal rather than an empty string so the row is visibly deliberate when
/// someone reads the table during an incident.
/// </summary>
private const string SingleClusterCacheKey = "__single";
/// <summary>
/// Node-local cache of applied deployment artifacts, or null when this node has none
/// (admin-only graphs, and tests that do not exercise it).
/// </summary>
private readonly IDeploymentArtifactCache? _deploymentArtifactCache;
/// <summary>
/// True once this node has booted a cached artifact because central SQL was unreachable.
/// </summary>
/// <remarks>
/// Surfaced on <see cref="NodeDiagnosticsSnapshot"/> because a node running from cache looks
/// completely healthy from the outside — it serves a full address space with live values.
/// The difference is that its configuration is frozen at whatever the cache held, and no
/// deployment can reach it. Without an explicit signal that is invisible until someone
/// wonders why a deploy "succeeded" everywhere but did not take effect here.
/// </remarks>
private bool _isRunningFromCache;
private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
private readonly CommonsNodeId _localNode;
private readonly IActorRef? _coordinatorOverride;
@@ -226,6 +253,17 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
/// non-cluster ActorRefProvider). See <see cref="DriverMemberCount"/>.</summary>
private readonly Func<int>? _driverMemberCountProvider;
/// <summary>Where the Primary-gate decision is published for consumers that live outside a mailbox —
/// today, the alarm store-and-forward drain. Null on nodes that do not wire one.</summary>
private readonly IRedundancyRoleView? _redundancyRoleView;
/// <summary>
/// Host of this node's LocalDb replication partner — the only node that holds a copy of this
/// node's alarm queue, and therefore the only node it may ever stand down in favour of.
/// <c>null</c> when this node dials nobody (unpaired, or the listening half of a pair).
/// </summary>
private readonly string? _replicationPeerHost;
/// <summary>Debounces the S5 "snapshot omitted this node" Warning to once per process — a persistent
/// identity mismatch (03/S5) would otherwise log on every snapshot.</summary>
private bool _warnedSnapshotMissingLocalNode;
@@ -339,11 +377,20 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
ScriptRootLogger? scriptRootLogger = null,
IActorRef? scriptedAlarmHostOverride = null,
IDriverCapabilityInvokerFactory? invokerFactory = null,
Func<int>? driverMemberCountProvider = null) =>
Func<int>? driverMemberCountProvider = null,
IDeploymentArtifactCache? deploymentArtifactCache = null,
IRedundancyRoleView? redundancyRoleView = null,
string? replicationPeerHost = null) =>
// WARNING: this forwarding list is POSITIONAL, and Props.Create compiles it into an
// expression tree. Six IActorRef? parameters and several interface-typed ones mean a
// mis-ordered argument is usually type-compatible and therefore compiles clean, then binds
// the wrong dependency at runtime. New parameters go LAST in all three places — this
// signature, the constructor's, and this call — and nothing else moves.
Akka.Actor.Props.Create(() => new DriverHostActor(
dbFactory, localNode, coordinator, driverFactory, localRoles, dependencyMux, opcUaPublishActor,
healthPublisher, virtualTagEvaluator, historyWriter, virtualTagHostOverride,
loggerFactory, scriptRootLogger, scriptedAlarmHostOverride, invokerFactory, driverMemberCountProvider));
loggerFactory, scriptRootLogger, scriptedAlarmHostOverride, invokerFactory, driverMemberCountProvider,
deploymentArtifactCache, redundancyRoleView, replicationPeerHost));
/// <summary>Initializes a new DriverHostActor with the specified dependencies.</summary>
/// <param name="dbFactory">Database context factory for configuration database access.</param>
@@ -372,6 +419,10 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
/// <param name="driverMemberCountProvider">Test seam (archreview 03/S4): overrides the count of Up
/// <c>driver</c>-role cluster members the Primary gate reads while the role is unknown. When null the
/// default reads <c>Cluster.Get(Context.System).State.Members</c> (0 on a non-cluster ActorRefProvider).</param>
/// <param name="deploymentArtifactCache">Optional node-local cache of applied deployment artifacts.
/// When supplied, each successful apply stores its artifact so the node can boot from its
/// last-known-good configuration while central SQL is unreachable. Null on admin-only nodes and in
/// tests that do not exercise the cache — caching is then simply skipped.</param>
public DriverHostActor(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
CommonsNodeId localNode,
@@ -388,8 +439,14 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
ScriptRootLogger? scriptRootLogger = null,
IActorRef? scriptedAlarmHostOverride = null,
IDriverCapabilityInvokerFactory? invokerFactory = null,
Func<int>? driverMemberCountProvider = null)
Func<int>? driverMemberCountProvider = null,
IDeploymentArtifactCache? deploymentArtifactCache = null,
IRedundancyRoleView? redundancyRoleView = null,
string? replicationPeerHost = null)
{
_deploymentArtifactCache = deploymentArtifactCache;
_redundancyRoleView = redundancyRoleView;
_replicationPeerHost = replicationPeerHost;
_dbFactory = dbFactory;
_localNode = localNode;
_coordinatorOverride = coordinator;
@@ -541,7 +598,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
// space were lost on restart. Re-spawn + re-materialise + re-subscribe from the
// applied deployment so a restarted/rebuilt node restores its served state instead
// of waiting for a config change (whose identical-config revision would no-op).
RestoreApplied(new DeploymentId(latest.DeploymentId));
RestoreApplied(new DeploymentId(latest.DeploymentId), revision);
break;
case NodeDeploymentStatus.Applying:
_log.Warning("DriverHost {Node}: found orphan Applying row for deployment {Id}; replaying",
@@ -560,16 +617,129 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
catch (Exception ex)
{
_log.Warning(ex, "DriverHost {Node}: ConfigDb unreachable on bootstrap; entering Stale", _localNode);
Become(Stale);
// Central is unreachable, so try the node-local cache before giving up. On a hit this
// node serves its last-known-good configuration through the outage instead of coming up
// with an empty address space. On a miss, behaviour is exactly what it always was.
if (!TryBootFromCache())
Become(Stale);
}
}
/// <summary>
/// Last-resort boot path: apply the artifact cached by a previous successful deploy (or
/// replicated from this node's pair peer) when central SQL cannot be reached.
/// </summary>
/// <returns>
/// <see langword="true"/> when a cached artifact was applied and the actor is now Steady;
/// <see langword="false"/> when the caller should fall through to <c>Stale</c>.
/// </returns>
/// <remarks>
/// <para>
/// <b>The read is unkeyed.</b> The cache is keyed by ClusterId so a pair shares one
/// entry, but ClusterId is only derivable from an artifact you already hold or from the
/// central DB — and at this seam we have neither. So the newest pointer row wins, which
/// is correct in every real topology because a node belongs to one cluster and its peer
/// replicates that same cluster's row. A node re-homed between clusters is the only
/// ambiguous case, and the cache logs a warning naming both.
/// </para>
/// <para>
/// <b>This does not make the node current.</b> <c>_currentRevision</c> is set from the
/// cached artifact, so a subsequent dispatch of that same revision correctly no-ops,
/// while any NEW revision still requires central — the cache holds the past, not the
/// future. Dispatch handling is unchanged.
/// </para>
/// <para>
/// Never throws: a fault here must degrade to Stale, which is exactly where the node
/// would have been without a cache at all.
/// </para>
/// </remarks>
private bool TryBootFromCache()
{
if (_deploymentArtifactCache is null)
return false;
try
{
var cached = _deploymentArtifactCache.GetCurrentUnkeyedAsync().GetAwaiter().GetResult();
if (cached is null)
{
_log.Info(
"DriverHost {Node}: no cached deployment available; entering Stale with no configuration.",
_localNode);
return false;
}
var deploymentId = DeploymentId.Parse(cached.DeploymentId);
var revision = RevisionHash.Parse(cached.RevisionHash);
// Steady, not Stale: this node is serving a real configuration. The retry-db timer that
// Stale would have started does not run here, so recovery rides on the next dispatch —
// matching how a normally-booted node behaves.
_currentRevision = revision;
Become(Steady);
ApplyCachedArtifact(deploymentId, cached.Artifact);
_isRunningFromCache = true;
_log.Warning(
"DriverHost {Node}: RUNNING FROM CACHE — central ConfigDb is unreachable, so this node " +
"booted deployment {Id} (rev {Rev}) cached at {CachedAtUtc:o} from its local database. " +
"Configuration changes cannot be applied until the ConfigDb is reachable again.",
_localNode, deploymentId, revision, cached.AppliedAtUtc);
return true;
}
catch (Exception ex)
{
_log.Error(ex,
"DriverHost {Node}: failed to boot from the local deployment cache; falling back to Stale.",
_localNode);
return false;
}
}
/// <summary>
/// Applies an artifact already in hand — no ConfigDb read, no ACK.
/// </summary>
/// <remarks>
/// Mirrors <see cref="RestoreApplied"/>, but sources the artifact from the cache rather than
/// re-reading it from a database that is by definition unreachable here. It also skips
/// <c>UpsertNodeDeploymentState</c> and <c>SendAck</c> for the same reason: both write to or
/// depend on central.
/// </remarks>
private void ApplyCachedArtifact(DeploymentId deploymentId, byte[] blob)
{
var correlation = CorrelationId.NewId();
var specs = DeploymentArtifact.ParseDriverInstances(blob, _localNode.Value);
var snapshots = _children.ToDictionary(
kv => kv.Key,
kv => new DriverChildSnapshot(kv.Value.DriverType, kv.Value.LastConfigJson, kv.Value.ResilienceConfig),
StringComparer.Ordinal);
var plan = DriverSpawnPlanner.Compute(snapshots, specs);
foreach (var id in plan.ToStop) StopChild(id);
foreach (var spec in plan.ToApplyDelta) ApplyChildDelta(spec);
foreach (var spec in plan.ToSpawn) SpawnChild(spec);
// Hand the cached blob to the rebuild so it materialises the client-facing address space from
// it directly. Passing only the deploymentId would drive a ConfigDb read — unreachable here by
// definition — and the rebuild would no-op, leaving OPC UA clients browsing an empty server
// while the drivers below poll happily. (Found on the docker-dev live gate.)
_opcUaPublishActor?.Tell(
new ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.RebuildAddressSpace(correlation, deploymentId, blob));
PushDesiredSubscriptionsFromArtifact(deploymentId, blob);
}
private void Steady()
{
Receive<DispatchDeployment>(HandleDispatchFromSteady);
Receive<GetDiagnostics>(HandleGetDiagnostics);
Receive<DriverInstanceActor.AttributeValuePublished>(ForwardToMux);
Receive<DriverInstanceActor.AttributeAlarmPublished>(ForwardNativeAlarm);
Receive<DriverInstanceActor.ConnectivityChanged>(OnDriverConnectivityChanged);
Receive<DriverInstanceActor.DiscoveredNodesReady>(HandleDiscoveredNodes);
Receive<DriverInstanceActor.DeltaApplied>(HandleDeltaApplied);
Receive<RestartDriver>(HandleRestartDriver);
@@ -600,6 +770,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
Receive<GetDiagnostics>(HandleGetDiagnostics);
Receive<DriverInstanceActor.AttributeValuePublished>(ForwardToMux);
Receive<DriverInstanceActor.AttributeAlarmPublished>(ForwardNativeAlarm);
Receive<DriverInstanceActor.ConnectivityChanged>(OnDriverConnectivityChanged);
Receive<DriverInstanceActor.DiscoveredNodesReady>(HandleDiscoveredNodes);
Receive<DriverInstanceActor.DeltaApplied>(HandleDeltaApplied);
Receive<RestartDriver>(HandleRestartDriver);
@@ -1010,6 +1181,46 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
/// signal the inbound-write gate uses — only the Primary publishes the single fleet-wide copy.
/// </para>
/// </summary>
/// <summary>
/// #477 — a child driver's connectivity transition. Annotates the source-data Quality of EVERY native
/// alarm condition the driver owns: comms lost → <see cref="OpcUaQuality.Bad"/>, restored →
/// <see cref="OpcUaQuality.Good"/>. This is the ONLY signal for a comms-lost native source, because a
/// disconnected driver emits no alarm transitions — the alarm feed goes silent, so without this a
/// comms-lost condition would keep reporting the accidentally-Good default forever.
/// <para>
/// UNGATED by redundancy role (like the condition write in <see cref="ForwardNativeAlarm"/>): a
/// Secondary keeps its address space — including condition quality — warm for failover. Quality is
/// a pure annotation: <c>WriteAlarmQuality</c> touches ONLY the condition's Quality, never its
/// Active/Acked/Retain, and fires a Part 9 event only on a real quality-bucket change. No cluster
/// <c>alerts</c> row is published here — driver comms health has its own status surface
/// (<see cref="IDriverHealthPublisher"/>); a row per condition would be alarm-fatigue.
/// </para>
/// </summary>
private void OnDriverConnectivityChanged(DriverInstanceActor.ConnectivityChanged msg)
{
if (_opcUaPublishActor is null) return;
var quality = msg.Connected ? OpcUaQuality.Good : OpcUaQuality.Bad;
var ts = DateTime.UtcNow;
var annotated = 0;
// Fan out to every condition this driver owns. _alarmNodeIdByDriverRef is keyed by
// (DriverInstanceId, RawPath); one driver ref can back several condition NodeIds (identical machines).
foreach (var ((driverId, _), nodeIds) in _alarmNodeIdByDriverRef)
{
if (driverId != msg.DriverInstanceId) continue;
foreach (var n in nodeIds)
{
_opcUaPublishActor.Tell(new ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.AlarmQualityUpdate(
n.NodeId, quality, ts, n.Realm));
annotated++;
}
}
if (annotated > 0)
_log.Debug("DriverHost {Node}: driver {Driver} {State} — annotated {Count} native condition(s) {Quality}",
_localNode, msg.DriverInstanceId, msg.Connected ? "connected" : "disconnected", annotated, quality);
}
private void ForwardNativeAlarm(DriverInstanceActor.AttributeAlarmPublished msg)
{
if (_opcUaPublishActor is null) return;
@@ -1264,6 +1475,14 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
}
}
/// <summary>Host part of a <c>host:port</c> node id.</summary>
private static string HostOf(NodeId nodeId)
{
var text = nodeId.ToString();
var colon = text.LastIndexOf(':');
return colon < 0 ? text : text[..colon];
}
/// <summary>The Primary-gate deny reason tag for the denial meter (<c>secondary|detached|role-unknown</c>).</summary>
private string PrimaryGateDenyReason() => _localRole switch
{
@@ -1283,19 +1502,35 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
if (local is not null)
{
_localRole = local.Role;
return;
}
// S5 diagnostic: the snapshot never mentioned this node. On a multi-driver cluster that means the
// Primary gate stays default-DENY indefinitely for an unknown role — surface it once so a silent
// identity mismatch (e.g. an ApplicationUri/NodeId skew) is diagnosable.
if (!_warnedSnapshotMissingLocalNode && DriverMemberCount() > 1)
else if (!_warnedSnapshotMissingLocalNode && DriverMemberCount() > 1)
{
// S5 diagnostic: the snapshot never mentioned this node. On a multi-driver cluster that means the
// Primary gate stays default-DENY indefinitely for an unknown role — surface it once so a silent
// identity mismatch (e.g. an ApplicationUri/NodeId skew) is diagnosable.
_warnedSnapshotMissingLocalNode = true;
_log.Warning(
"DriverHost {Node}: redundancy snapshot omitted this node (snapshotNodes={SnapshotNodes}); Primary data-plane gate stays default-DENY while role is unknown on a multi-driver cluster",
_localNode, string.Join(",", msg.Nodes.Select(n => n.NodeId)));
}
// Publish on EVERY snapshot, including ones that leave the cached role untouched, so a role
// that changes without this node being named still reaches the drain within one heartbeat.
//
// NOTE the different policy: the drain gate keys on the role ALONE and opens when it is
// unknown, where the data-plane gate above resolves an unknown role by member count and
// denies. Publishing ShouldServiceAsPrimary() here suspended the drain on every node of a
// multi-driver cluster that had no redundancy configured — see ShouldDrainAlarmHistory.
// Defer ONLY to the node that holds a copy of our rows. The redundancy election is
// cluster-wide and the queue is pair-local, so "somebody is Primary" is not a licence to stop
// draining — that Primary is usually in another pair entirely, and cannot deliver our events.
var peerIsPrimary =
_replicationPeerHost is not null
&& msg.Nodes.Any(n => n.Role == RedundancyRole.Primary
&& HostOf(n.NodeId) == _replicationPeerHost);
_redundancyRoleView?.Publish(
PrimaryGatePolicy.ShouldDrainAlarmHistory(_localRole, queueSharingPeerIsPrimary: peerIsPrimary));
}
private void Stale()
@@ -1326,6 +1561,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
// applied yet, so the equipment can't be resolved). Drop it — the re-discovery loop re-sends it
// and the post-recovery re-apply self-heals it once an apply runs (matches the no-op drops above).
Receive<DriverInstanceActor.DiscoveredNodesReady>(_ => { });
// A child connectivity transition while the host is Stale has no live address space to annotate — drop
// it (the post-recovery rebuild re-materialises conditions, and the child re-announces on its next entry).
Receive<DriverInstanceActor.ConnectivityChanged>(_ => { });
// A late DeltaApplied (an apply completed just before the DB went Stale) — re-register the driver's
// mux adapter anyway; it simply re-reads the driver's current refs (harmless, no DB access).
Receive<DriverInstanceActor.DeltaApplied>(HandleDeltaApplied);
@@ -1348,7 +1586,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
NodeId: _localNode,
CurrentRevision: _currentRevision,
Drivers: drivers,
AsOfUtc: DateTime.UtcNow);
AsOfUtc: DateTime.UtcNow,
RunningFromCache: _isRunningFromCache);
Sender.Tell(snapshot);
}
@@ -1381,7 +1620,30 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
try
{
ReconcileDrivers(deploymentId);
var appliedBlob = ReconcileDrivers(deploymentId);
// Issue #486 — a null blob means the artifact could not be read (unreachable ConfigDb, or the
// empty bytes a missing row yields), so NOTHING was applied. Reporting Applied here used to
// advance _currentRevision too, and HandleDispatchFromSteady short-circuits on a revision
// match — so the node claimed a configuration it never applied AND could never be healed by
// re-dispatching that revision. Failing the apply keeps the revision where it was, which is
// what makes the retry land instead of being waved through. The node keeps serving its
// last-known-good address space, drivers and subscriptions throughout (issue #485) — this is
// about telling the truth, not about tearing anything down.
if (appliedBlob is null)
{
const string reason =
"the deployment artifact could not be read (ConfigDb unreachable, or the row is missing/empty); nothing was applied";
UpsertNodeDeploymentState(deploymentId, NodeDeploymentStatus.Failed, reason);
SendAck(deploymentId, ApplyAckOutcome.Failed, reason, correlation);
OtOpcUaTelemetry.DeploymentApplied.Add(1, new KeyValuePair<string, object?>("outcome", "reject"));
span?.SetStatus(ActivityStatusCode.Error, reason);
_log.Error(
"DriverHost {Node}: apply of {Id} FAILED — {Reason}. Staying on revision {Rev} and continuing to serve it; re-dispatch once the ConfigDb is readable",
_localNode, deploymentId, reason, _currentRevision);
return;
}
_currentRevision = revision;
UpsertNodeDeploymentState(deploymentId, NodeDeploymentStatus.Applied, failureReason: null);
SendAck(deploymentId, ApplyAckOutcome.Applied, failureReason: null, correlation);
@@ -1392,6 +1654,12 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
// SubscribeBulk pass: hand each driver its desired tag references so live values flow into
// the just-rebuilt address space instead of staying BadWaitingForInitialData.
PushDesiredSubscriptions(deploymentId);
CacheAppliedArtifact(deploymentId, revision, appliedBlob);
// Reaching here means the artifact came from central, so the node is no longer serving a
// cache-sourced configuration. Note this only clears on a real apply: a dispatch of the
// revision already booted from cache short-circuits in HandleDispatchFromSteady without
// touching the ConfigDb, and the flag correctly stays set.
_isRunningFromCache = false;
OtOpcUaTelemetry.DeploymentApplied.Add(1, new KeyValuePair<string, object?>("outcome", "ack"));
_log.Info("DriverHost {Node}: applied deployment {Id} (rev {Rev}, children={Count})",
_localNode, deploymentId, revision, _children.Count);
@@ -1415,11 +1683,24 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
/// <summary>
/// Read the deployment artifact + reconcile the set of running <see cref="DriverInstanceActor"/>
/// children. Spawn missing, ApplyDelta on config change, stop removed/disabled drivers.
/// When the artifact blob is empty (legacy ControlPlane tests, smoke fixtures) or the
/// configured <see cref="IDriverFactory"/> can't materialise any of the requested
/// types, this is effectively a no-op.
/// When the artifact blob is empty (legacy ControlPlane tests, smoke fixtures) the reconcile is
/// SKIPPED outright — see the issue #485 guard below — and when the configured
/// <see cref="IDriverFactory"/> can't materialise any of the requested types this is a no-op.
/// </summary>
private void ReconcileDrivers(DeploymentId deploymentId)
/// <returns>
/// The artifact blob that was reconciled, or <see langword="null"/> when it could not be
/// loaded at all.
/// </returns>
/// <remarks>
/// Returning the blob rather than swallowing it is load-bearing for the artifact cache. This
/// method catches its own DB failures, logs a warning and returns WITHOUT rethrowing — so
/// <see cref="ApplyAndAck"/> proceeds to its success path, ACKs Applied and logs success
/// having spawned zero drivers. A cache write that trusted "the apply succeeded" would then
/// persist an empty artifact as this node's last-known-good configuration, and the node would
/// later boot from it into an empty address space. The caller distinguishes the two cases by
/// this return value.
/// </remarks>
private byte[]? ReconcileDrivers(DeploymentId deploymentId)
{
byte[] blob;
try
@@ -1434,7 +1715,21 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
{
_log.Warning(ex, "DriverHost {Node}: failed to load artifact for {Id}; skipping reconcile",
_localNode, deploymentId);
return;
return null;
}
// Issue #485 — no bytes is no answer, not "a configuration with no drivers". Parsing zero bytes
// yields zero specs, and DriverSpawnPlanner then plans EVERY running child for StopChild: a missing
// deployment row (or a half-written one) silently takes this node's entire field I/O down while the
// apply still ACKs Applied. Nothing legitimate produces a zero-length blob — deleting the last
// driver still deploys a JSON document with empty arrays — so treat it exactly like the load
// failure above and leave the running children alone.
if (blob.Length == 0)
{
_log.Warning(
"DriverHost {Node}: artifact for {Id} read back empty; skipping reconcile and keeping the {Count} running driver(s)",
_localNode, deploymentId, _children.Count);
return null;
}
var specs = DeploymentArtifact.ParseDriverInstances(blob, _localNode.Value);
@@ -1455,6 +1750,74 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
// sequenced AFTER ReinitializeAsync rebuilds DependencyRefs — a synchronous re-register HERE would
// re-read the STALE ref set (ApplyDelta runs asynchronously on the child), so it is deliberately NOT
// done inline.
return blob;
}
/// <summary>
/// Store a successfully applied artifact in the node-local cache, so this node can boot from
/// it while central SQL is unreachable.
/// </summary>
/// <remarks>
/// <para>
/// <b>Never throws.</b> By the time this runs the deployment is already recorded Applied
/// in central SQL and an Applied ACK has been sent to the coordinator. An exception
/// escaping here would unwind into <see cref="ApplyAndAck"/>'s catch and send a second,
/// contradictory Failed ACK for a deployment the fleet already believes is live.
/// </para>
/// <para>
/// <b>An empty or unloadable blob is skipped, not cached.</b> See
/// <see cref="ReconcileDrivers"/>: a DB failure there degrades silently, so "the apply
/// succeeded" does not imply "a real configuration was applied". Caching an empty
/// artifact would make it this node's last-known-good and boot it into an empty address
/// space during the next outage — strictly worse than having no cache at all.
/// </para>
/// <para>
/// Runs synchronously on the actor thread. That matches every other DB call on this path
/// (all of which already block), and the alternative — piping the result back as a
/// self-message — would need a handler registered in Steady, Applying AND Stale or it
/// dead-letters after the <c>Become(Steady)</c> in the enclosing finally.
/// </para>
/// </remarks>
private void CacheAppliedArtifact(DeploymentId deploymentId, RevisionHash revision, byte[]? blob)
{
if (_deploymentArtifactCache is null)
return;
if (blob is null || blob.Length == 0)
{
_log.Warning(
"DriverHost {Node}: not caching deployment {Id} — its artifact was empty or could not " +
"be loaded, so it is not a usable last-known-good configuration.",
_localNode, deploymentId);
return;
}
try
{
// The node's ClusterId is carried inside the artifact (ClusterNode rows), not in config
// or on IClusterRoleInfo. A single-cluster or unscoped artifact has none, so it shares
// one sentinel key — the pair still converges on a single pointer row either way.
var scope = DeploymentArtifact.ResolveClusterScope(blob, _localNode.Value);
var clusterId = scope.Mode == ClusterFilterMode.ScopeTo && !string.IsNullOrWhiteSpace(scope.ClusterId)
? scope.ClusterId
: SingleClusterCacheKey;
_deploymentArtifactCache
.StoreAsync(clusterId, deploymentId.ToString(), revision.Value, blob)
.GetAwaiter()
.GetResult();
_log.Debug("DriverHost {Node}: cached deployment {Id} (rev {Rev}, cluster {ClusterId}, {Bytes} bytes)",
_localNode, deploymentId, revision, clusterId, blob.Length);
}
catch (Exception ex)
{
_log.Error(ex,
"DriverHost {Node}: failed to cache applied deployment {Id}. The apply itself succeeded " +
"and is unaffected; this node simply has no local fallback for that revision.",
_localNode, deploymentId);
}
}
/// <summary>
@@ -1466,14 +1829,24 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
/// drivers, rebuilds the address space from the applied artifact, and re-pushes SubscribeBulk.
/// No re-ack: the deployment is already Applied.
/// </summary>
private void RestoreApplied(DeploymentId deploymentId)
private void RestoreApplied(DeploymentId deploymentId, RevisionHash? revision)
{
var correlation = CorrelationId.NewId();
try
{
ReconcileDrivers(deploymentId);
_opcUaPublishActor?.Tell(new ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.RebuildAddressSpace(correlation, deploymentId));
var appliedBlob = ReconcileDrivers(deploymentId);
_opcUaPublishActor?.Tell(new ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.RebuildAddressSpace(correlation, deploymentId, appliedBlob));
PushDesiredSubscriptions(deploymentId);
// Populate the node-local cache from the artifact this restored node is now serving. The
// cache invariant is "holds what the node currently serves"; without this only a FRESH
// apply (ApplyAndAck) ever wrote it, so a node whose cache was lost — a wiped/fresh volume,
// a disk failure — would recover its served state here yet stay cache-less, unable to
// boot-from-cache on the NEXT central outage until some future new deploy happened to land.
// Replication does not heal that gap either: a peer's already-acked rows are pruned from its
// oplog, so a fully-wiped node is never back-filled. Re-caching on restore is what closes
// it. (Found on the docker-dev live gate.)
if (revision is { } rev)
CacheAppliedArtifact(deploymentId, rev, appliedBlob);
_log.Info("DriverHost {Node}: restored served state for applied deployment {Id} on bootstrap", _localNode, deploymentId);
}
catch (Exception ex)
@@ -1507,6 +1880,31 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
return;
}
PushDesiredSubscriptionsFromArtifact(deploymentId, blob);
}
/// <summary>
/// The artifact-processing half of <see cref="PushDesiredSubscriptions"/>, split out so the
/// boot-from-cache path can drive it with bytes already in hand.
/// </summary>
/// <remarks>
/// Separated because the cache path runs precisely when the ConfigDb read above cannot
/// succeed — re-reading the artifact there would fail by definition.
/// </remarks>
private void PushDesiredSubscriptionsFromArtifact(DeploymentId deploymentId, byte[] blob)
{
// Issue #485 — the same "no bytes is no answer" rule as ReconcileDrivers. An empty artifact parses
// to a composition with no raw tags, which hands every child an EMPTY desired set — and an empty set
// drops the live subscription handle rather than re-subscribing. Reachable independently of the
// reconcile guard: this method does its OWN ConfigDb read, so the row can go missing between the two.
if (blob.Length == 0)
{
_log.Warning(
"DriverHost {Node}: artifact for {Id} read back empty; skipping SubscribeBulk and keeping the live subscriptions",
_localNode, deploymentId);
return;
}
AddressSpaceComposition composition;
try
{
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs
+15
View File
@@ -46,6 +46,12 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
public sealed record InitializeSucceeded(int Generation);
public sealed record InitializeFailed(string Reason, int Generation);
public sealed record DisconnectObserved(string Reason);
/// <summary>#477 — sent to the parent (<see cref="DriverHostActor"/>) on every connectivity transition:
/// <c>Connected=true</c> on entering the Connected state, <c>false</c> on entering Reconnecting. The host
/// annotates this driver's native alarm conditions' source-data Quality from it (comms lost → Bad,
/// restored → Good) — independently of alarm transitions, since a comms-lost driver emits no alarm
/// events. Fire-and-forget, mirroring <see cref="DeltaApplied"/>.</summary>
public sealed record ConnectivityChanged(string DriverInstanceId, bool Connected);
public sealed record ApplyDelta(string DriverConfigJson, CorrelationId Correlation);
public sealed record ApplyResult(bool Success, string? Reason, CorrelationId Correlation);
/// <summary>
@@ -413,6 +419,11 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
private void Connected()
{
// #477 — announce connectivity to the host so it can clear any comms-lost Quality annotation on this
// driver's native alarm conditions (Bad → Good). Fire-and-forget; the host defaults conditions to a
// non-Good "waiting for initial data" quality at materialise, and this is what confirms them Good.
Context.Parent.Tell(new ConnectivityChanged(_driverInstanceId, Connected: true));
ReceiveAsync<ApplyDelta>(HandleApplyDeltaAsync);
Receive<DisconnectObserved>(msg =>
{
@@ -483,6 +494,10 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
private void Reconnecting()
{
// #477 — announce comms loss to the host so it annotates this driver's native alarm conditions Bad
// (a comms-lost driver emits no alarm events, so this is the ONLY signal that the source is unreachable).
Context.Parent.Tell(new ConnectivityChanged(_driverInstanceId, Connected: false));
Receive<RetryConnect>(_ => InitializeAsync(_currentConfigJson ?? "{}"));
// Fast-fail writes while reconnecting (same reason as Connecting — avoids the 8s host Ask
// timeout on an inbound write to a transiently-down driver). Synchronous Receive.
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/PrimaryGatePolicy.cs
+46
View File
@@ -25,4 +25,50 @@ public static class PrimaryGatePolicy
RedundancyRole.Secondary or RedundancyRole.Detached => false,
_ => driverMemberCount <= 1, // unknown: allow only when no driver peer exists
};
/// <summary>
/// Decide whether this node should drain the replicated alarm store-and-forward queue.
/// </summary>
/// <remarks>
/// <para>
/// <b>Deliberately not <see cref="ShouldServiceAsPrimary"/>.</b> That gate protects a shared
/// field device, where two nodes acting at once is dangerous and irreversible, so an unknown
/// role with a peer present must deny. This gate protects an append-only historian, and the
/// delivery contract is already <i>at-least-once across a failover, by design</i> — identical
/// events even collapse to a single row, because ids hash the payload.
/// </para>
/// <para>
/// The two error costs are therefore asymmetric: a false allow costs a duplicate history row;
/// a false deny silently stops the alarm audit trail and eventually evicts it at the capacity
/// wall. So this decision keys on the role <b>alone</b> and opens when the role is unknown —
/// no membership tie-break, because cluster-wide driver count says nothing about whether
/// <i>this</i> node has a redundant partner.
/// </para>
/// <para>
/// Found by the LocalDb Phase 2 live gate: on a rig of four driver nodes in one cluster with
/// no <c>Redundancy</c> section, borrowing the write gate's verdict suspended the drain on
/// every node — including unpaired ones — so nothing drained anywhere.
/// </para>
/// </remarks>
/// <param name="localRole">This node's last-known redundancy role, or <c>null</c> when unknown.</param>
/// <param name="queueSharingPeerIsPrimary">
/// Whether the node holding <see cref="RedundancyRole.Primary"/> in the same snapshot is a node
/// that <b>shares this node's queue</b> — i.e. its LocalDb replication partner.
/// <para>
/// Merely knowing that <i>a</i> Primary exists is not enough, because the redundancy role is
/// a CLUSTER-WIDE election while the alarm queue is PAIR-LOCAL. A fleet runs many pairs in
/// one cluster, so the elected driver Primary is usually in some other pair — and on the
/// docker-dev rig it is a central node, which carries the driver Akka role, replicates
/// nobody's LocalDb and does not even run the alarm historian. Deferring to it means this
/// node's events are delivered by no one, ever.
/// </para>
/// </param>
/// <returns><c>true</c> to drain; <c>false</c> only when a node holding these same rows is doing it.</returns>
public static bool ShouldDrainAlarmHistory(RedundancyRole? localRole, bool queueSharingPeerIsPrimary) =>
localRole switch
{
RedundancyRole.Primary => true,
RedundancyRole.Secondary or RedundancyRole.Detached => !queueSharingPeerIsPrimary,
_ => true, // unknown role ⇒ drain; a duplicate row beats a silent gap
};
}
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/AlarmHistorianOptions.cs
+8 -13
View File
@@ -1,5 +1,4 @@
using System.Collections.Generic;
using System.IO;
using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Historian;
@@ -7,11 +6,12 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.Historian;
/// <summary>
/// Binds the <c>AlarmHistorian</c> configuration section that gates the durable
/// store-and-forward alarm sink. When <see cref="Enabled"/> is <c>true</c>,
/// <c>AddAlarmHistorian</c> registers a <c>SqliteStoreAndForwardSink</c> (draining to the
/// <c>AddAlarmHistorian</c> registers a <c>LocalDbStoreAndForwardSink</c> (draining to the
/// gateway alarm writer supplied by the Host) in place of the
/// <c>NullAlarmHistorianSink</c> default; otherwise the Null default survives. This section
/// supplies only the <see cref="Enabled"/> gate and the SQLite store-and-forward knobs — the
/// downstream connection (endpoint/key/TLS) is sourced from the <c>ServerHistorian</c> section.
/// supplies only the <see cref="Enabled"/> gate and the store-and-forward knobs — the
/// downstream connection (endpoint/key/TLS) is sourced from the <c>ServerHistorian</c> section,
/// and the queue's storage location is the node's consolidated <c>LocalDb:Path</c> database.
/// </summary>
public sealed class AlarmHistorianOptions
{
@@ -24,9 +24,6 @@ public sealed class AlarmHistorianOptions
/// </summary>
public bool Enabled { get; init; }
/// <summary>Filesystem path to the local SQLite store-and-forward queue database.</summary>
public string DatabasePath { get; init; } = "alarm-historian.db";
/// <summary>Maximum number of queued rows the drain worker forwards in a single batch.</summary>
public int BatchSize { get; init; } = 100;
@@ -34,15 +31,15 @@ public sealed class AlarmHistorianOptions
public int DrainIntervalSeconds { get; init; } = 5;
/// <summary>Maximum queued rows before the sink evicts the oldest. Defaults to 1,000,000
/// (matches <c>SqliteStoreAndForwardSink</c>'s <c>DefaultCapacity</c>).</summary>
public long Capacity { get; init; } = SqliteStoreAndForwardSink.DefaultCapacity;
/// (matches <c>LocalDbStoreAndForwardSink</c>'s <c>DefaultCapacity</c>).</summary>
public long Capacity { get; init; } = LocalDbStoreAndForwardSink.DefaultCapacity;
/// <summary>Days to retain dead-lettered rows before purge. Defaults to 30.</summary>
public int DeadLetterRetentionDays { get; init; } = 30;
/// <summary>Maximum delivery attempts before a perpetually-retrying (poison) row is dead-lettered.
/// Defaults to 10 (matches <c>SqliteStoreAndForwardSink</c>'s <c>DefaultMaxAttempts</c>).</summary>
public int MaxAttempts { get; init; } = SqliteStoreAndForwardSink.DefaultMaxAttempts;
/// Defaults to 10 (matches <c>LocalDbStoreAndForwardSink</c>'s <c>DefaultMaxAttempts</c>).</summary>
public int MaxAttempts { get; init; } = LocalDbStoreAndForwardSink.DefaultMaxAttempts;
/// <summary>Returns operator-facing misconfiguration warnings for an <c>Enabled</c> historian
/// (empty when disabled or correctly configured). Pure — the registration logs each entry.</summary>
@@ -51,8 +48,6 @@ public sealed class AlarmHistorianOptions
{
var warnings = new List<string>();
if (!Enabled) return warnings;
if (!Path.IsPathRooted(DatabasePath))
warnings.Add($"AlarmHistorian:DatabasePath '{DatabasePath}' is relative — it resolves against the process working directory (e.g. System32 for a Windows service). Set an absolute path.");
if (DrainIntervalSeconds <= 0)
warnings.Add($"AlarmHistorian:DrainIntervalSeconds is {DrainIntervalSeconds} — must be > 0; the drain timer will throw or spin at startup.");
if (Capacity <= 0)
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/HistorianAdapterActor.cs
+1 -1
View File
@@ -15,7 +15,7 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.Historian;
/// Thin actor wrapper around <see cref="IAlarmHistorianSink"/>. Engine code (ScriptedAlarmActor,
/// Galaxy native alarm bridge, AB CIP ALMD reader) tells <see cref="AlarmHistorianEvent"/>s to this
/// actor; the actor enqueues them on the sink fire-and-forget. Production deployments register
/// <see cref="SqliteStoreAndForwardSink"/> against <c>IAlarmHistorianSink</c>; the sink owns the
/// <see cref="LocalDbStoreAndForwardSink"/> against <c>IAlarmHistorianSink</c>; the sink owns the
/// durable queue + drain-to-HistorianGateway-SendEvent loop. The actor here owns nothing operational beyond
/// the message contract — its job is to keep the engine actors on Akka's mailbox without blocking
/// them on disk I/O or gateway round-trips.
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ServerHistorianOptions.cs
+4 -2
View File
@@ -40,8 +40,10 @@ public sealed class ServerHistorianOptions
/// <summary>
/// The peppered-HMAC API key (<c>histgw_&lt;id&gt;_&lt;secret&gt;</c>) the gateway validates
/// in the <c>Authorization: Bearer</c> header. Supply via the environment variable
/// <c>ServerHistorian__ApiKey</c> — never commit it to config. Required when
/// <see cref="Enabled"/> is <c>true</c>.
/// <c>ServerHistorian__ApiKey</c> — never commit it to config. The value may be a literal
/// key or a <c>${secret:otopcua/historian/api-key}</c> token, which the pre-host secrets
/// expander resolves fail-closed (throwing if the secret is absent) before this options
/// class binds. Required when <see cref="Enabled"/> is <c>true</c>.
/// </summary>
public string ApiKey { get; init; } = "";
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs
+76 -12
View File
@@ -58,13 +58,32 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
/// <param name="Realm">The namespace realm the condition lives in — <see cref="AddressSpaceRealm.Uns"/> for
/// scripted alarms (default), <see cref="AddressSpaceRealm.Raw"/> for v3 native raw conditions.</param>
public sealed record AlarmStateUpdate(string AlarmNodeId, AlarmConditionSnapshot State, DateTime TimestampUtc, AddressSpaceRealm Realm);
/// <summary>#477 — annotate a materialised condition's source-data Quality out of band from any alarm
/// transition (the driver-connectivity path: comms lost → <see cref="OpcUaQuality.Bad"/>, restored →
/// <see cref="OpcUaQuality.Good"/>). Routed to <see cref="IOpcUaAddressSpaceSink.WriteAlarmQuality"/>,
/// which sets ONLY Quality and fires one Part 9 event on a quality-bucket change.</summary>
/// <param name="AlarmNodeId">The condition node id (RawPath for a native alarm).</param>
/// <param name="Quality">The source-data quality to annotate.</param>
/// <param name="TimestampUtc">The connectivity transition timestamp in UTC.</param>
/// <param name="Realm">The namespace realm the condition lives in (<see cref="AddressSpaceRealm.Raw"/> for native).</param>
public sealed record AlarmQualityUpdate(string AlarmNodeId, OpcUaQuality Quality, DateTime TimestampUtc, AddressSpaceRealm Realm);
/// <summary>
/// Triggers an address-space rebuild. <paramref name="DeploymentId"/> is the deployment
/// just applied by the host; the rebuild loads THAT artifact so materialisation matches the
/// applied config + the SubscribeBulk pass. It is null only for legacy/dev callers, which
/// fall back to the latest sealed deployment (lags a not-yet-sealed apply by one revision).
/// </summary>
public sealed record RebuildAddressSpace(CorrelationId Correlation, DeploymentId? DeploymentId = null);
/// <param name="Correlation">Correlation id for tracing this rebuild.</param>
/// <param name="DeploymentId">The applied deployment whose artifact to materialise, or null for the latest-sealed fallback.</param>
/// <param name="Artifact">
/// The artifact bytes already in hand, used INSTEAD of loading them from the ConfigDb. This is
/// what makes boot-from-cache actually serve its address space: on a central-SQL outage the
/// host has the cached blob but <see cref="DeploymentId"/> alone would drive a ConfigDb read
/// that cannot succeed, leaving the rebuild a no-op and clients browsing an empty server. Null
/// on the normal path, where loading from the ConfigDb by id is correct.
/// </param>
public sealed record RebuildAddressSpace(
CorrelationId Correlation, DeploymentId? DeploymentId = null, byte[]? Artifact = null);
/// <summary>Inject driver-discovered nodes (FixedTree) under an equipment at runtime (post-connect).</summary>
/// <param name="EquipmentRootNodeId">The OPC UA NodeId of the equipment root folder to inject the
@@ -93,6 +112,10 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
private int _writes;
private byte _lastServiceLevel;
private bool _publishedAtLeastOnce;
/// <summary>True once a real composition has been applied, i.e. there is a served address space worth
/// protecting. Distinguishes "the artifact went missing under a running server" (issue #485 — warn, and
/// hold what is materialised) from "nothing has been deployed here yet" (a quiet no-op at boot).</summary>
private bool _hasAppliedComposition;
private DbHealthProbeActor.DbHealthStatus? _lastDbHealth;
private RedundancyStateChanged? _lastSnapshot;
private (bool Ok, DateTime At)? _probeAboutMe;
@@ -239,6 +262,7 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
Receive<AttributeValueUpdate>(HandleAttributeUpdate);
Receive<AlarmStateUpdate>(HandleAlarmUpdate);
Receive<AlarmQualityUpdate>(HandleAlarmQualityUpdate);
Receive<RebuildAddressSpace>(HandleRebuild);
Receive<MaterialiseDiscoveredNodes>(HandleMaterialiseDiscovered);
Receive<ServiceLevelChanged>(HandleServiceLevelChanged);
@@ -296,6 +320,20 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
}
}
private void HandleAlarmQualityUpdate(AlarmQualityUpdate msg)
{
try
{
_sink.WriteAlarmQuality(msg.AlarmNodeId, msg.Quality, msg.TimestampUtc, msg.Realm);
Interlocked.Increment(ref _writes);
OtOpcUaTelemetry.OpcUaSinkWrite.Add(1, new KeyValuePair<string, object?>("kind", "alarm-quality"));
}
catch (Exception ex)
{
_log.Warning(ex, "OpcUaPublish: sink.WriteAlarmQuality threw for {Node}", msg.AlarmNodeId);
}
}
private void HandleRebuild(RebuildAddressSpace msg)
{
using var span = OtOpcUaTelemetry.StartAddressSpaceRebuildSpan();
@@ -321,13 +359,37 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
try
{
// Prefer the artifact of the deployment the host just applied — at apply time it is not
// yet Sealed, so LoadLatestArtifact would return the PREVIOUS revision and materialise a
// stale composition (variables that don't match the SubscribeBulk refs). Fall back to
// latest-sealed only for legacy callers that don't carry a DeploymentId.
var artifact = msg.DeploymentId is { } depId
? LoadArtifact(depId)
: LoadLatestArtifact();
// An in-hand artifact (boot-from-cache) wins: the host already has the cached bytes, and
// the ConfigDb read the DeploymentId path would do is exactly what is unreachable during
// the outage this exists to survive. Otherwise prefer the artifact of the deployment the
// host just applied — at apply time it is not yet Sealed, so LoadLatestArtifact would
// return the PREVIOUS revision and materialise a stale composition (variables that don't
// match the SubscribeBulk refs). Fall back to latest-sealed only for legacy callers that
// carry neither.
var artifact = msg.Artifact
?? (msg.DeploymentId is { } depId ? LoadArtifact(depId) : LoadLatestArtifact());
// Issue #485 — an artifact we could not obtain is NOT an empty configuration. Parsing zero
// bytes yields an empty composition, which the planner diffs against the live one as a
// PureRemove and the applier then faithfully tears down: a transient ConfigDb blip empties the
// served address space. Nothing legitimate produces a zero-length blob (an operator who really
// deletes everything still deploys a JSON document with empty arrays), so the only honest
// reading of "no bytes" is "no answer" — hold the last-known-good address space and let the
// next deploy (or the next boot) supply a real one. Returning here also leaves _lastApplied
// intact, so the retry diffs against what is actually materialised.
if (artifact is null or { Length: 0 })
{
if (_hasAppliedComposition)
_log.Warning(
"OpcUaPublish: no usable artifact for deployment {Id} (correlation={Correlation}) — KEEPING the currently served address space rather than tearing it down",
msg.DeploymentId, msg.Correlation);
else
_log.Debug(
"OpcUaPublish: no artifact to materialise yet (deployment={Id}, correlation={Correlation})",
msg.DeploymentId, msg.Correlation);
return;
}
var composition = _localNode is { } ln
? DeploymentArtifact.ParseComposition(artifact, ln.Value,
inconsistency => _log.Warning("OpcUaPublish {Node}: cross-cluster binding — {Message}", ln, inconsistency))
@@ -343,6 +405,7 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
var outcome = _applier.Apply(plan);
_lastApplied = composition;
_hasAppliedComposition = true;
// Sum swallowed per-node materialise failures across every pass together with Apply's own
// removal-pass tally (archreview 01/S-1). A degraded apply used to vanish into per-node
@@ -407,7 +470,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
}
/// <summary>Read a specific deployment's artifact blob from ConfigDb (the one just applied,
/// which may not be Sealed yet). Empty array on any failure — parser treats it as "no composition".</summary>
/// which may not be Sealed yet). Empty array on any failure — <see cref="HandleRebuild"/> treats that as
/// "no answer" and abandons the rebuild, leaving the served address space untouched (issue #485).</summary>
private byte[] LoadArtifact(DeploymentId deploymentId)
{
try
@@ -420,13 +484,13 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
}
catch (Exception ex)
{
_log.Warning(ex, "OpcUaPublish: failed to load artifact for deployment {Id}; rebuild becomes no-op", deploymentId);
_log.Warning(ex, "OpcUaPublish: failed to load artifact for deployment {Id}; the rebuild is abandoned", deploymentId);
return Array.Empty<byte>();
}
}
/// <summary>Read the most recent <c>Sealed</c> deployment's artifact blob from ConfigDb.
/// Empty array on any failure — the parser treats empty blob as "no composition".</summary>
/// Empty array on any failure — see <see cref="LoadArtifact"/> for how that is handled.</summary>
private byte[] LoadLatestArtifact()
{
try
@@ -440,7 +504,7 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
}
catch (Exception ex)
{
_log.Warning(ex, "OpcUaPublish: failed to load latest deployment artifact; rebuild becomes no-op");
_log.Warning(ex, "OpcUaPublish: failed to load latest deployment artifact; the rebuild is abandoned");
return Array.Empty<byte>();
}
}
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Redundancy/IRedundancyRoleView.cs
+57
View File
@@ -0,0 +1,57 @@
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Redundancy;
/// <summary>
/// A singleton snapshot of the alarm-history drain decision, so code that lives outside an actor
/// can ask "should this node be draining the alarm queue right now?".
/// </summary>
/// <remarks>
/// <para>
/// Exists for the alarm store-and-forward drain. That drain runs on a timer owned by the
/// sink, not on an actor mailbox, so it cannot receive <c>RedundancyStateChanged</c>
/// directly — but it must be role-scoped, because the queue it drains replicates to the
/// pair peer and two draining nodes would deliver every event twice.
/// </para>
/// <para>
/// Deliberately publishes the <i>decision</i> rather than the role that produced it, so
/// <see cref="PrimaryGatePolicy"/> stays the single place that decides.
/// </para>
/// <para>
/// <b>This is not the device-write gate.</b> It is fed by
/// <see cref="PrimaryGatePolicy.ShouldDrainAlarmHistory"/>, which opens on an unknown role
/// where <see cref="PrimaryGatePolicy.ShouldServiceAsPrimary"/> closes. The naming is
/// explicit about the consumer for that reason: an earlier revision published the
/// write-gate verdict here, and on a cluster with several driver nodes but no configured
/// redundancy it suspended the drain everywhere, so alarm history accumulated on every node
/// and left none.
/// </para>
/// </remarks>
public interface IRedundancyRoleView
{
/// <summary>Whether this node should be draining the alarm store-and-forward queue right now.</summary>
bool ShouldDrainAlarmHistory { get; }
/// <summary>Records a fresh decision. Called by <c>DriverHostActor</c> on every redundancy snapshot.</summary>
/// <param name="shouldDrainAlarmHistory">
/// The decision <see cref="PrimaryGatePolicy.ShouldDrainAlarmHistory"/> produced.
/// </param>
void Publish(bool shouldDrainAlarmHistory);
}
/// <summary>Thread-safe <see cref="IRedundancyRoleView"/> backed by a volatile field.</summary>
public sealed class RedundancyRoleView : IRedundancyRoleView
{
// Seeded with the decision for "role unknown" rather than a bare false. An unpublished view and a
// node whose role nothing ever reports are genuinely the same situation, and they must behave the
// same: a deployment that runs no redundancy at all never publishes here, and defaulting closed
// would silently stop its alarm history forever.
private volatile bool _shouldDrainAlarmHistory =
PrimaryGatePolicy.ShouldDrainAlarmHistory(localRole: null, queueSharingPeerIsPrimary: false);
/// <inheritdoc/>
public bool ShouldDrainAlarmHistory => _shouldDrainAlarmHistory;
/// <inheritdoc/>
public void Publish(bool shouldDrainAlarmHistory) => _shouldDrainAlarmHistory = shouldDrainAlarmHistory;
}
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmHostActor.cs
+43 -4
View File
@@ -278,11 +278,31 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
private void OnDependencyChanged(VirtualTagActor.DependencyValueChanged msg)
{
// Feed the live value into the upstream the engine subscribes from. StatusCode 0 = Good; the
// mux only forwards values it received from a driver publish, so we treat them as Good-quality.
_upstream.Push(msg.TagId, new DataValueSnapshot(msg.Value, 0u, msg.TimestampUtc, msg.TimestampUtc));
// Feed the live value into the upstream the engine subscribes from. #478 — carry the source
// driver's quality (mapped to an OPC UA StatusCode) so the engine can derive a scripted
// condition's worst-of-input quality; a Bad/Uncertain input is no longer silently treated as Good.
_upstream.Push(msg.TagId,
new DataValueSnapshot(msg.Value, StatusFromQuality(msg.Quality), msg.TimestampUtc, msg.TimestampUtc));
}
/// <summary>#478 — map the 3-state <see cref="OpcUaQuality"/> to an OPC UA StatusCode (severity bits)
/// for the engine's read cache. The inverse of <see cref="QualityFromStatus"/>.</summary>
private static uint StatusFromQuality(OpcUaQuality quality) => quality switch
{
OpcUaQuality.Bad => 0x80000000u,
OpcUaQuality.Uncertain => 0x40000000u,
_ => 0u, // Good
};
/// <summary>#478 — collapse an engine <see cref="ScriptedAlarmEvent.WorstInputStatusCode"/> (top-2
/// severity bits) back to the 3-state <see cref="OpcUaQuality"/> the Commons snapshot / node path use.</summary>
private static OpcUaQuality QualityFromStatus(uint statusCode) => (statusCode >> 30) switch
{
0 => OpcUaQuality.Good,
1 => OpcUaQuality.Uncertain,
_ => OpcUaQuality.Bad,
};
private void OnEngineEmission(EngineEmission msg)
{
var e = msg.Event;
@@ -294,6 +314,21 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
return;
}
// #478 — QualityChanged is a source-quality annotation (worst-of-input bucket moved with no Part 9
// state transition — e.g. an input went Bad, freezing the condition). Route it to the dedicated
// WriteAlarmQuality node path (sets ONLY Quality, one Part 9 event on a bucket change): NO full-state
// projection (would clobber severity/message) and NO /alerts row (it is not a state transition, so it
// must not historize). Same out-of-band quality path native comms-loss uses (#477 Layer 2).
if (e.Emission == EmissionKind.QualityChanged)
{
_publishActor.Tell(new OpcUaPublishActor.AlarmQualityUpdate(
AlarmNodeId: e.AlarmId,
Quality: QualityFromStatus(e.WorstInputStatusCode),
TimestampUtc: e.TimestampUtc,
Realm: AddressSpaceRealm.Uns));
return;
}
// Bridge to OPC UA: project the FULL Part 9 condition state (enabled/active/acked/confirmed/
// shelving/severity/message) onto the materialised condition node via the Commons snapshot.
// e.AlarmId is the materialised condition's NodeId (T14 aligned it to the ScriptedAlarmId).
@@ -539,7 +574,11 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
Enabled: e.Condition.Enabled == AlarmEnabledState.Enabled,
Shelving: MapShelving(e.Condition.Shelving.Kind),
Severity: (ushort)SeverityToInt(e.Severity),
Message: e.Message);
Message: e.Message,
// #478 — the condition's Quality is the worst quality across the script's input tags at evaluation
// time (carried on the event by the engine). A transition fired while an input is Uncertain projects
// Uncertain here so the full-snapshot write doesn't clobber quality back to Good.
Quality: QualityFromStatus(e.WorstInputStatusCode));
/// <summary>Maps the Core <see cref="ShelvingKind"/> onto the Commons <see cref="AlarmShelvingKind"/>
/// mirror (the Commons assembly can't see the Core enum).</summary>
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs
+81 -9
View File
@@ -15,11 +15,14 @@ using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
using ZB.MOM.WW.OtOpcUa.Core.VirtualTags;
using ZB.MOM.WW.OtOpcUa.OpcUaServer;
using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
using ZB.MOM.WW.OtOpcUa.Runtime.Health;
using ZB.MOM.WW.OtOpcUa.Runtime.Historian;
using ZB.MOM.WW.OtOpcUa.Runtime.OpcUa;
using ZB.MOM.WW.OtOpcUa.Runtime.Redundancy;
using ZB.MOM.WW.OtOpcUa.Runtime.VirtualTags;
using ZB.MOM.WW.LocalDb;
namespace ZB.MOM.WW.OtOpcUa.Runtime;
@@ -38,7 +41,7 @@ public static class ServiceCollectionExtensions
/// <summary>
/// Registers shared runtime services. Currently binds <see cref="IAlarmHistorianSink"/>
/// to <see cref="NullAlarmHistorianSink"/> as the default; production deployments
/// override this with <c>SqliteStoreAndForwardSink</c> wrapping the HistorianGateway alarm writer.
/// override this with <c>LocalDbStoreAndForwardSink</c> wrapping the HistorianGateway alarm writer.
/// Call this BEFORE <c>AddAkka</c>.
/// </summary>
/// <param name="services">The service collection to register with.</param>
@@ -64,12 +67,23 @@ public static class ServiceCollectionExtensions
/// <summary>
/// Config-gated durable alarm-historian sink. When the <c>AlarmHistorian</c> section has
/// <c>Enabled=true</c>, registers a <see cref="SqliteStoreAndForwardSink"/> (draining via the
/// <c>Enabled=true</c>, registers a <see cref="LocalDbStoreAndForwardSink"/> (draining via the
/// <paramref name="writerFactory"/>-supplied writer) as the <see cref="IAlarmHistorianSink"/>,
/// overriding the <see cref="NullAlarmHistorianSink"/> default. Otherwise a no-op (Null stays).
/// The writer is injected so the durable downstream (the HistorianGateway alarm writer) can be
/// supplied by the Host, which is the only project that references it.
/// </summary>
/// <remarks>
/// <para>
/// The queue lives in the node's consolidated <see cref="ILocalDb"/>, which replicates it
/// to the redundant pair peer — so undelivered alarm history survives losing the node
/// holding it. That is also why the drain is gated on <see cref="IRedundancyRoleView"/>:
/// the Secondary holds a full replica of the Primary's queue, and an ungated drain there
/// would re-deliver every event. Both services are resolved from the provider, so a
/// deployment that enables this section without registering LocalDb fails loudly at
/// resolution rather than silently buffering to nowhere.
/// </para>
/// </remarks>
/// <param name="services">The service collection to register with.</param>
/// <param name="configuration">The configuration carrying the <c>AlarmHistorian</c> section.</param>
/// <param name="writerFactory">
@@ -86,21 +100,56 @@ public static class ServiceCollectionExtensions
if (opts is not { Enabled: true }) return services; // leave the Null default from AddOtOpcUaRuntime
foreach (var warning in opts.Validate())
Serilog.Log.Logger.ForContext<SqliteStoreAndForwardSink>().Warning("Historian config: {HistorianConfigWarning}", warning);
Serilog.Log.Logger.ForContext<LocalDbStoreAndForwardSink>().Warning("Historian config: {HistorianConfigWarning}", warning);
// The view is a plain singleton so it exists even on nodes whose DriverHostActor never
// publishes to it — an unpublished view reads as "no peer, drain", which is the correct
// posture for a deployment that runs no redundancy at all.
services.TryAddSingleton<IRedundancyRoleView, RedundancyRoleView>();
// THE GATE ONLY APPLIES TO A REPLICATED QUEUE. Standing down is only ever safe because some
// other node holds the same rows and will send them instead — and the only node that does is
// this node's LocalDb replication peer. Without replication configured, these rows exist here
// and nowhere else, so deferring to anyone means they are never delivered by anyone.
//
// This is not hypothetical. The redundancy role is a CLUSTER-WIDE election
// (RedundancyStateActor keys on Akka's RoleLeader("driver")), while the queue is PAIR-LOCAL.
// On the docker-dev rig the elected driver Primary is a central node — which carries the
// driver Akka role, replicates nobody's LocalDb, and does not even run the alarm historian —
// so every site node dutifully suspended its drain in favour of a node that could not
// possibly deliver its events. Scoping the gate to "is my queue actually shared?" is what
// keeps the two scopes from disagreeing.
// BOTH halves of a pair share the queue, but only one of them dials: the initiator sets
// Replication:PeerAddress, its partner only sets SyncListenPort and waits. Testing the dial
// side alone would leave the listening half permanently ungated — one drainer per pair by
// accident rather than by role, and the wrong one whenever the roles swap.
var replicated =
!string.IsNullOrWhiteSpace(configuration["LocalDb:Replication:PeerAddress"])
|| !string.IsNullOrWhiteSpace(configuration["LocalDb:SyncListenPort"]);
if (!replicated)
{
Serilog.Log.Logger.ForContext<LocalDbStoreAndForwardSink>().Information(
"Alarm historian: LocalDb replication is not configured, so this node's queue is not "
+ "shared with any peer and the Primary drain gate does not apply — this node always "
+ "drains its own alarm queue.");
}
services.AddSingleton<IAlarmHistorianSink>(sp =>
{
// SqliteStoreAndForwardSink takes a Serilog ILogger (not Microsoft.Extensions.Logging).
// LocalDbStoreAndForwardSink takes a Serilog ILogger (not Microsoft.Extensions.Logging).
// Resolve it off the host's configured static logger so the drain worker's WARN/INFO
// lines land in the same sinks as the rest of the process.
var sink = new SqliteStoreAndForwardSink(
opts.DatabasePath,
var roleView = sp.GetRequiredService<IRedundancyRoleView>();
var sink = new LocalDbStoreAndForwardSink(
sp.GetRequiredService<ILocalDb>(),
writerFactory(opts, sp),
Serilog.Log.Logger.ForContext<SqliteStoreAndForwardSink>(),
Serilog.Log.Logger.ForContext<LocalDbStoreAndForwardSink>(),
batchSize: opts.BatchSize,
capacity: opts.Capacity,
deadLetterRetention: TimeSpan.FromDays(opts.DeadLetterRetentionDays),
maxAttempts: opts.MaxAttempts);
maxAttempts: opts.MaxAttempts,
drainGate: () => !replicated || roleView.ShouldDrainAlarmHistory);
sink.StartDrainLoop(TimeSpan.FromSeconds(opts.DrainIntervalSeconds));
return sink;
});
@@ -221,6 +270,26 @@ public static class ServiceCollectionExtensions
var serviceLevel = resolver.GetService<IServiceLevelPublisher>() ?? NullServiceLevelPublisher.Instance;
var loggerFactory = resolver.GetService<ILoggerFactory>() ?? NullLoggerFactory.Instance;
var healthPublisher = resolver.GetService<IDriverHealthPublisher>() ?? NullDriverHealthPublisher.Instance;
// Node-local deployment-artifact cache. Registered by the Host's AddOtOpcUaLocalDb on
// driver-role nodes only; deliberately left null elsewhere (admin-only graphs, test
// harnesses) rather than given a null-object, so DriverHostActor skips caching outright
// instead of pretending to cache into a sink that drops everything.
var deploymentArtifactCache = resolver.GetService<IDeploymentArtifactCache>();
// Where this actor publishes its Primary-gate verdict for the alarm store-and-forward
// drain, which runs on a timer and so cannot read RedundancyStateChanged itself.
// Registered by AddAlarmHistorian; absent when no durable sink is configured, in which
// case there is nothing downstream to inform.
var redundancyRoleView = resolver.GetService<IRedundancyRoleView>();
// Host of this node's LocalDb replication partner: the ONLY node that holds a copy of this
// node's alarm queue, and so the only node it may stand down in favour of. Null when this
// node dials nobody, which correctly means "never stand down".
var replicationPeerHost =
Uri.TryCreate(
resolver.GetService<IConfiguration>()?["LocalDb:Replication:PeerAddress"],
UriKind.Absolute,
out var peerUri)
? peerUri.Host
: null;
// Root script logger backs the ScriptedAlarm host's engine + script logging. Registered in
// Host DI inside the hasDriver block; may be absent in some role configs / test harnesses,
// in which case the DriverHostActor gracefully skips spawning the ScriptedAlarm host.
@@ -338,7 +407,10 @@ public static class ServiceCollectionExtensions
historyWriter: historyWriter,
loggerFactory: loggerFactory,
scriptRootLogger: scriptRootLogger,
invokerFactory: invokerFactory),
invokerFactory: invokerFactory,
deploymentArtifactCache: deploymentArtifactCache,
redundancyRoleView: redundancyRoleView,
replicationPeerHost: replicationPeerHost),
DriverHostActorName);
registry.Register<DriverHostActorKey>(driverHost);
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyMuxActor.cs
+1 -1
View File
@@ -100,7 +100,7 @@ public sealed class DependencyMuxActor : ReceiveActor
// space carries thousands of tags and only a fraction feed virtual-tag expressions.
return;
}
var dep = new VirtualTagActor.DependencyValueChanged(msg.FullReference, msg.Value, msg.TimestampUtc);
var dep = new VirtualTagActor.DependencyValueChanged(msg.FullReference, msg.Value, msg.TimestampUtc, msg.Quality);
foreach (var sub in subscribers)
{
sub.Tell(dep);

Some files were not shown because too many files have changed in this diff Show More