Compare commits

...

11 Commits

Author SHA1 Message Date
Joseph Doherty 4cc0215bb4 fix(adminui): the cluster-node editor was dead — 500 on load, silent no-op on save
Both defects are PRE-EXISTING (present on master, unrelated to #499) and were found
live-verifying the #499 guard, which lives on this page and was unreachable without
them fixed.

1. **HTTP 500 on load.** `FormModel.ServiceLevelBase` was `byte` to match the
   entity, and `InputNumber<T>`'s static constructor throws outright for
   `System.Byte` — "The type 'System.Byte' is not a supported numeric type". That
   escapes `BuildRenderTree` unhandled, which in Blazor takes the WHOLE page down,
   not one field. `/clusters/{id}/nodes/{nodeId}` and `.../nodes/new` have never
   rendered. Now `int` with the same `[Range(0, 255)]` and a cast on the way to the
   entity, so the stored type is unchanged. (Same failure mode as #504: an
   unhandled throw inside the render tree is a page-kill.)

2. **Save did nothing, silently.** `NodeId` was validated against
   `^[A-Za-z0-9_-]+$`, which forbids the colon — but every NodeId in this system is
   `host:port` (`ClusterRoleInfo` and `ConfigPublishCoordinator` derive it as
   `member.Address.Host:Port`, the seed writes `central-1:4053`, and
   `NodeDeploymentState.NodeId` is FK-bound to it). So every existing node failed
   validation, `OnValidSubmit` never ran, and the button was inert. Pattern now
   allows `:` and `.` (hostnames may be dotted).

3. **Added the missing `<ValidationSummary/>`.** The form had a
   `DataAnnotationsValidator` but nothing rendering its output, so a validation
   failure produced no feedback at all — which is precisely how (2) stayed
   invisible. This is the change that stops the next such defect being silent.

Live-verified on docker-dev (both central nodes rebuilt, since :9200 round-robins
the pair): page 500 → 200 on both routes, and a real edit now persists.
2026-07-26 10:42:04 -04:00
Joseph Doherty 4b9bcddbcc feat(drivers): periodic reconcile of desired vs actual subscriptions (#488)
Option A from the issue — the cheap consistency check, no interface change.

The desired ref set was re-applied on a Connected TRANSITION (`ResubscribeDesired`)
and on a deploy's `SetDesiredSubscriptions`, and nowhere else. A subscription lost
while the driver STAYED Connected — a subscribe that failed and was never retried,
a handle dropped down an error path — left the actor subscribed to nothing while
still reporting Healthy, indefinitely.

`ReconcileSubscription` now rides the existing 30 s `health-poll` tick: while
Connected, a driver holding a desired set but no live handle is re-subscribed.

Gated on `ISubscribable`. A driver that cannot subscribe at all can still be handed
a desired set, and without the guard it would be told `Subscribe` every tick and
fail every tick, forever.

This is a state-machine consistency check, NOT a probe — it can only see that this
actor holds no handle, never that a handle is present but dead server-side, because
`ISubscriptionHandle` is opaque. That is option B, and it stays deferred until a
real occurrence shows the handle outliving the subscription; it would need a
liveness member implemented across all eight drivers with different subscription
mechanics.

A redundant `Subscribe` is possible (a tick queued ahead of an already-queued one
observes the same no-handle state) and is harmless: `HandleSubscribeAsync` is a
`ReceiveAsync`, so no tick is processed mid-subscribe, and it drops any prior
subscription before establishing the new one. Suppressing it would need a
pending-subscribe flag threaded through every self-tell site — more state than the
race is worth.

`healthPollInterval` becomes an optional Props parameter, mirroring the existing
`rediscoverInterval` seam, so a test can drive the reconcile without waiting 30 s.

Three tests. The reconcile itself is proven by a stub whose FIRST subscribe throws:
nothing else would ever retry it, so a second attempt can only come from the
reconcile — disabling `ReconcileSubscription` turns it red. The two guard tests are
absence assertions and are bounded by a positive control rather than a sleep: a
counting health publisher makes ticks PROCESSED observable, so "still one subscribe"
means the reconcile ran and declined, not that the timer had not fired.

Explicitly not deploy-time drift (#486) — re-asserting an already-correct desired
set would not have helped there.
2026-07-26 10:30:19 -04:00
Joseph Doherty 2193d9f2e4 fix(config): refuse to store a Sql credential in a node config override (#499)
`ClusterNode.DriverConfigOverridesJson` is the third surface a Sql driver's config
is persisted on, and the only one nothing gated. It is a map keyed by
`DriverInstanceId` merged onto the cluster-level `DriverConfig`, so a literal
`connectionString` pasted there leaks exactly as one pasted into the driver — the
leak #498 closed for `DriverInstance.DriverConfig` and `Device.DeviceConfig`.

Gated at the SAVE, not at the deploy — a deliberate departure from both options the
issue offered:

- `DraftSnapshot` carries no `ClusterNode` rows, and adding them would widen the
  snapshot and every builder of one for a single rule about a value that never
  enters the artifact.
- More to the point, a deploy gate is the wrong instrument here. Node overrides are
  not in the artifact, so blocking a deploy would not stop the credential being
  stored — it is already in the database and replicated by then. Refusing the save
  is the only point where "refuse to store it" is literally true, which is #498's
  own framing: discarding a secret on read is not the same as refusing to store it.

The check moves into a shared `SqlCredentialGuard` so the two enforcement points
cannot drift; `DraftValidator` now calls it instead of its own private helper.

Kept deliberately narrow — the `Sql` driver's `connectionString` only, and only for
instance ids that ARE Sql drivers. The issue asked whether to generalise to
credential-shaped keys across every driver type; not doing that, for the same
reason #498 did not: a broader sweep would start refusing configs that are
legitimate today for drivers which never made an indirect-credential guarantee,
which is a regression rather than defence in depth. Widen per driver, as each gains
its own contract.

The error names the offending driver instance(s) and NEVER the value — it reaches
the AdminUI and the audit trail.

14 new cases cover both halves, including the case variants (System.Text.Json binds
`ConnectionString` to `connectionString`, so a case variant is the same key, not a
bypass), non-Sql ids being ignored, and every blank/malformed/non-object shape.
2026-07-26 10:30:04 -04:00
Joseph Doherty 8b7ea3213d fix(rig,docs): backfill ClusterNode.GrpcPort and document the upgrade step (#493)
`GrpcPort` is a nullable column added by Phase 5. Every insert in the docker-dev
seed is INSERT-if-not-exists — the right shape for a re-runnable seed, but it means
a new nullable column never reaches rows an earlier version of the file created. On
a persisted volume all six rows kept NULL, central found no dial targets, and
`TelemetryDial:Mode=Grpc` connected to nothing. (`AkkaPort` escaped this only
because it is NOT NULL with a default of 4053.)

Seed now backfills `GrpcPort IS NULL AND CreatedBy = 'docker-dev-seed'` to 4056.
NULL only — it must not overwrite a port an operator set deliberately on a
long-lived dev volume, and NULL is the unambiguous "predates the column" marker.

Deliberately NOT doing the suggested EF data migration for real deployments. The
correct value is each node's own `Telemetry:GrpcListenPort`, which is
per-deployment configuration the database cannot derive; a migration could only
invent one, and a WRONG dial target is worse than the NULL the dialer already skips
explicitly with a throttled warning. The rig can be backfilled because there the
port is fixed by docker-compose.yml and therefore actually knowable.

`docs/Telemetry.md` gains an upgrade section stating the prerequisite, the symptoms
(the throttled skip log, LISTEN with no ESTABLISHED, the pill never going live) and
the UPDATE to run before flipping any node to Grpc — plus why there is no migration.
2026-07-26 10:29:51 -04:00
Joseph Doherty 240d7aa8aa fix(test): give the three absence assertions a positive control (#501)
`AwaitAssert` returns on its FIRST success, so an assertion that is already true at
t=0 passes instantly and spends none of its duration. All three sites asserted an
absence against a collection that starts empty, so none of them ever gave the
handler the time their comments claimed.

Replaced with mailbox-ordering positive controls rather than sleeps:

- `RouteNativeAlarmAck_unknown_node_is_dropped` — follow the unmapped ack with a
  MAPPED one. The host forwards via `Tell` and the child handles `RouteAlarmAck`
  with `ReceiveAsync` (which suspends its own mailbox), so once the control reaches
  the driver the unmapped one has provably been handled.
- `RouteNativeAlarmAck_on_non_primary_is_dropped` — the role itself is the control:
  return the node to Primary and re-send. Comments (`gated` / `control`) are the
  discriminator, so a leaked ack is visible rather than merged into a count.
- `PeerProbeSupervisorTests.Single_node_snapshot_spawns_no_children` — a FOURTH site
  the issue did not list. It says the other `ChildCount.ShouldBe(0)` waits are each
  preceded by a `ShouldBe(1)`; that is true of three of the four, but not this one,
  which asserts the initial state. Now followed by a two-node snapshot: the count
  reaching 1 proves the single-node snapshot was processed, and a local-node child
  would have made it 2.

The issue warned these may turn red, which would be a real defect surfacing. They
did not — the routing is correct. Proven by breaking it deliberately instead:
disabling the Primary gate reddens the Secondary test, and routing unmapped ids to
an arbitrary mapping reddens the unknown-node test. Under the old form both breaks
passed.
2026-07-26 10:29:41 -04:00
Joseph Doherty 95295210b0 fix(test): scope the EventPump counter capture to its own pump (#503)
Root cause was cross-test measurement leakage, not a timing budget.

EventPump's `Meter` is STATIC — one instance per process — and a `MeterListener`
can only select by *meter* in `InstrumentPublished`. `StartMeterCapture` therefore
received measurements from every `EventPump` alive anywhere in the assembly. xUnit
runs test classes in parallel and several build pumps (`EventPumpStreamFaultTests`,
the `Driver-X` test in this same file), so a sibling's events landed in these
counters. That is why the flake needed a busy box: it needed two tests to overlap.
Measured directly during the gate — `Received = 11` in a run that emitted 10.

The old `Received == Dispatched + Dropped + InFlight` assertion is what turned the
leak into a failure. `InFlight` is DERIVED as `Received - Dispatched - Dropped`, so
the identity is a tautology that cannot fail on a real defect — only on a foreign
increment landing between its four separate `Interlocked.Read` calls. Hence the
reported subject, `counters.Received`. It is deleted, along with the `InFlight`
member, and replaced by direct assertions on the three MEASURED counters.

Three further defects in the same test, all found on the way:

- **The "held" dispatch loop was never held.** `OnDataChange` is `EventHandler<T>`
  — void-returning — so `async (_, _) => await gate` is async VOID: it returned to
  the dispatch loop at the first await and blocked nothing. Measured `Dispatched =
  2..3`, never 0. Drops happened only when the producer happened to outrun the
  consumer, so the arrangement the doc comment describes was fiction and the
  assertions rode on scheduling luck. Now a synchronous `ManualResetEventSlim.Wait()`
  on the loop's own thread, pinned by `Dispatched.ShouldBe(0)`.
- **Fixed `Task.Delay(150)`** replaced by a presence poll on the drop count (#500's
  rule). Dropped starts at 0, so it cannot be satisfied by the initial state.
- **Emission is staged** — one event, wait for the dispatcher to enter the handler,
  then the other nine — so the arithmetic is exact (`10 - 1 held - 2 buffered = 7`)
  rather than scheduler-dependent.

The gate release moved into `finally`, ahead of disposal: `DisposeAsync` awaits the
dispatch loop, which is now genuinely parked on that gate, so a failed assertion
would otherwise hang instead of failing.

Verified: 15/15 clean under the exact contention the issue measured (Runtime +
AdminUI + Core concurrently) — where the half-fixed version still failed on run 4.
Falsifiability: restoring the async-void handler turns the new assertions red.
2026-07-26 10:29:27 -04:00
Joseph Doherty e3155fbbf5 merge: never slice a DB-sourced string with the range operator (#504)
v2-ci / build (push) Successful in 4m11s
v2-ci / unit-tests (push) Failing after 14m22s
`@s.SourceHash[..12]` on an unvalidated nvarchar(64) threw out of
BuildRenderTree and took the whole /scripts page to an HTTP 500. Replaced with a
shared, null/short-safe `DisplayText.Abbreviate`, applied to all five unguarded
range-operator slices over DB-sourced strings in the AdminUI.

Live-verified on docker-dev: /scripts 500 -> 200 with the short hashes rendered
in full; long hashes still truncate with an ellipsis on the other pages.
2026-07-26 09:43:55 -04:00
Joseph Doherty 47d148daf9 fix(adminui): never slice a DB-sourced string with the range operator (#504)
`Scripts.razor` rendered a hash prefix as `@s.SourceHash[..12]…`. `SourceHash`
is a plain nvarchar(64) with no length floor, so any value shorter than 12
characters threw ArgumentOutOfRangeException out of BuildRenderTree — unhandled
in Blazor, which takes the WHOLE page to an HTTP 500, not just the offending
row. Live-reproduced on docker-dev, where two Script rows carry `h1` and
`h-abs-hr200`: /scripts was a hard 500.

New `Components/Shared/DisplayText.Abbreviate(string?, int)`:
- null/blank renders the Admin UI's em-dash placeholder rather than throwing;
- a value already within budget renders verbatim with NO ellipsis, so the
  display never implies text that isn't there (the old markup appended "…"
  unconditionally, outside the slice — `h1` would have shown as `h1…`);
- a maxLength < 1 is clamped, so a display bug cannot become a page crash.

Applied to every unguarded range-operator slice over a DB-sourced string found
by sweeping the AdminUI for `[..N]`:

  Scripts.razor                    SourceHash    (the reported crash)
  Certificates.razor               Thumbprint    (read off the on-disk store)
  Deployments.razor  x2            RevisionHash
  Clusters/ClusterOverview.razor   RevisionHash

The other `[..N]` hits are safe and untouched: Guid.ToString("N") slices
(always 32 chars) and the `Length > 60 ? [..60]` ternaries that self-guard.

NOTE — the issue text blamed the docker-dev seed; that was wrong and is
corrected on #504. `seed-clusters.sql` inserts only ServerCluster, ClusterNode
and LdapGroupRoleMapping. The short-hash rows came from earlier live-gate
authoring. A freshly seeded rig is fine; the trigger is any Script row that did
not go through ScriptEdit / UnsTreeService.HashSource — REST-API authored,
hand-inserted, or migrated in.

Tests: 14 new in DisplayTextTests, covering the short/null/blank/clamped cases
and a never-throws sweep over every value x budget combination. AdminUI suite
739/739.

Live-verified on docker-dev (AdminUI has no bUnit): /scripts 500 -> 200 across
both central nodes, rendering `hash=h1` and `hash=h-abs-hr200` in full;
/deployments still truncates 64-char hashes at 12 with the ellipsis;
/certificates and /clusters/MAIN unchanged.
2026-07-26 09:43:48 -04:00
Joseph Doherty 755ae1aa3f merge: re-assert non-normal scripted-alarm conditions after a (re)load (#487)
v2-ci / build (push) Successful in 4m21s
v2-ci / unit-tests (push) Failing after 3h14m40s
Closes the redeploy-reset gap on Part 9 scripted-alarm condition nodes — the
alarm analogue of the VirtualTag ReassertValue fix.

A full address-space rebuild re-materialises every condition node fresh, while
the engine's reload produces EmissionKind.None for an alarm whose state did not
change — so nothing wrote the node. Live-reproduced on docker-dev: a cleared-
but-still-UNACKNOWLEDGED alarm came back reporting acknowledged, dropping Retain
to false, so it vanished from ConditionRefresh entirely. The operator's
outstanding alarm silently disappeared from the alarm list on deploy.

ScriptedAlarmHostActor.ReassertConditionNodes now writes one AlarmStateUpdate
per alarm not in the Part 9 no-event position, sourced from the new
ScriptedAlarmEngine.GetProjections(). Node-only by construction — the projection
type carries no EmissionKind, so it cannot become an alerts row or a duplicate
historian record.

Live gate PASSED (pre-fix image vs. rebuilt, central-2): condition absent
pre-fix, present + Unacknowledged + Retain=True post-fix, stamped with its own
LastTransitionUtc; /alerts empty across two re-asserts with the panel proven
live by a real ACTIVATED transition.
2026-07-26 09:18:23 -04:00
Joseph Doherty 549c656489 fix(alarms): re-assert non-normal scripted-alarm conditions after a (re)load (#487)
A deploy that triggers a FULL address-space rebuild clears `_alarmConditions`
and re-materialises every condition node fresh — inactive, acked, confirmed,
unshelved. The engine reloads its persisted state and re-derives Active from the
predicate, so it ends up correct; but there is no transition to report, so
`LoadAsync` yields `EmissionKind.None` and `OnEngineEmission` drops it. Nothing
writes the node.

Live-reproduced on the docker-dev rig against the pre-fix image: an alarm that
had fired and cleared but was still UNACKNOWLEDGED came back from a rebuild
reporting Acknowledged with Retain=false — so it disappeared from
ConditionRefresh entirely, while the engine still held Unacknowledged. An
operator's outstanding alarm silently vanishes from the alarm list on deploy.

`ScriptedAlarmHostActor.ReassertConditionNodes` closes it. After each load it
reads the new `ScriptedAlarmEngine.GetProjections()` and sends one
`AlarmStateUpdate` per alarm NOT in the Part 9 no-event position.

Load-bearing properties:

- Node-only. It sends `AlarmStateUpdate` and nothing else — never the `alerts`
  topic, never the telemetry hub. `alerts` is the historization path, so a row
  per alarm per deploy would append a duplicate historian/AVEVA record every
  time anyone deploys. This is why it reads a projection rather than re-emitting:
  `ScriptedAlarmProjection` carries no `EmissionKind`, so there is nothing for
  the emission machinery — or a future refactor — to mistake for a transition.
  The issue's sketch said `GetState(alarmId)` + `LoadedAlarmIds`; that alone
  would have clobbered the node's severity and message, so the projection also
  carries the definition's severity, the resolved message template, and the
  last-observed worst-input quality.
- Only non-normal alarms. One in the no-event position already matches what the
  materialise built, so a never-fired alarm stays byte-identical to a deploy
  without this pass (including its Message, which the materialise seeds with the
  display name), and an all-normal deploy writes nothing at all.
- Timestamp is the condition's own `LastTransitionUtc`, not the deploy instant.
- No duplicate Part 9 events: `WriteAlarmCondition`'s delta-gate compares against
  the node's CURRENT state, so a re-assert onto a freshly materialised node is a
  genuine delta (fires once — correct, the rebuild reset what clients see) while
  one onto a surgically-preserved node is suppressed.

Tests: 6 new host tests + 6 new engine tests. Falsifiability checked by
disabling the call — 4 of the 6 host tests go red; the other 2 are absence
guards against an over-broad fix and pass either way by design.

Live gate (docker-dev, central-2, pre-fix image vs. rebuilt image):
- pre-fix: condition absent from ConditionRefresh after a rebuild;
- post-fix: present, Unacknowledged, Retain=True, stamped with its own
  LastTransitionUtc rather than the restart instant;
- /alerts stayed empty across two re-asserts, with the panel proven live by a
  real ACTIVATED transition immediately afterwards — no duplicate history.
2026-07-26 09:06:39 -04:00
Joseph Doherty 28c2866710 merge: Sql driver follow-ups #496/#497/#498 + the Runtime.Tests flake #500 (fix/sql-driver-followups)
v2-ci / build (push) Successful in 4m15s
v2-ci / unit-tests (push) Failing after 15m14s
Four commits closing the three follow-ups the Sql poll driver left behind, plus the
intermittent Runtime.Tests failure found while verifying them.

- #497 corrects 16 wrong OPC UA status-code constants across 6 drivers (the issue named
  3) and adds StatusCodeParityTests, a reflection guard checking every driver's
  hard-coded uint against the pinned SDK. That guard is what found the other 13.
- #498 fails the deploy when a Sql driver's persisted config carries a literal
  connectionString — the write-side half of a guarantee only enforced on read.
- #496 implements the design 8.1 catalog gate: authored table/column names are resolved
  against the live catalog at Initialize and replaced with the catalog's own spelling, so
  quoting becomes the backstop it was documented to be rather than the sole defence.
- #500 fixes the Runtime.Tests flake: four ordering/logic defects plus a class of tight
  presence budgets, now routed through a documented RuntimeActorTestBase.PresenceBudget.

Verified: full solution builds, all 41 unit-test projects green, Sql integration suite
21/21 against the real SQL Server on 10.100.0.35, and 30 consecutive Runtime.Tests runs
clean against a measured 13% baseline.

Still open by design: #499 (ClusterNode.DriverConfigOverridesJson is a third config
persistence surface DraftValidator cannot see) and #501 (two alarm-ack tests use
AwaitAssert for an absence assertion, so they prove nothing; the rewrite may
legitimately turn them red).
2026-07-25 22:27:26 -04:00
24 changed files with 1508 additions and 116 deletions
+22
View File
@@ -106,6 +106,28 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-b-2:4053')
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, GrpcPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy) (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, GrpcPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
VALUES ('site-b-2:4053', 'SITE-B', 'site-b-2', 4840, 8081, 4053, 4056, 'urn:OtOpcUa:site-b-2', 150, 1, 'docker-dev-seed'); VALUES ('site-b-2:4053', 'SITE-B', 'site-b-2', 4840, 8081, 4053, 4056, 'urn:OtOpcUa:site-b-2', 150, 1, 'docker-dev-seed');
------------------------------------------------------------------------------
-- Backfill: GrpcPort on rows that predate the column (#493)
--
-- Every insert above is INSERT-if-not-exists, which is the right shape for a re-runnable seed
-- but means a *new nullable* column never reaches rows created by an earlier version of this
-- file. On a persisted volume that is exactly what happened when Phase 5 added GrpcPort: the
-- six ClusterNode rows already existed, so they kept NULL, TelemetryNodeSource found no dial
-- targets, and TelemetryDial:Mode=Grpc silently connected to nothing (throttled Warning per
-- node — the code degrades gracefully, so nothing failed loudly).
--
-- Backfills NULL only. It must not overwrite a port an operator set deliberately on a
-- long-lived dev volume; a NULL is the unambiguous "this row predates the column" marker.
--
-- AkkaPort deliberately has no equivalent here: it is NOT NULL with a default of 4053, so
-- pre-existing rows already carry a usable value and there is nothing to backfill.
------------------------------------------------------------------------------
UPDATE dbo.ClusterNode
SET GrpcPort = 4056 -- matches Telemetry__GrpcListenPort on all six nodes in docker-compose.yml
WHERE GrpcPort IS NULL
AND CreatedBy = 'docker-dev-seed';
------------------------------------------------------------------------------ ------------------------------------------------------------------------------
-- Galaxy MxAccess gateway — INTENTIONALLY NOT SEEDED (retired 2026-06-15) -- Galaxy MxAccess gateway — INTENTIONALLY NOT SEEDED (retired 2026-06-15)
-- --
+39
View File
@@ -102,6 +102,45 @@ Persisted scope per plan decision #14: `Enabled`, `Acked`, `Confirmed`, `Shelvin
Every mutation the state machine produces is immediately persisted inside the engine's `_evalGate` semaphore, so the store's view is always consistent with the in-memory state. Every mutation the state machine produces is immediately persisted inside the engine's `_evalGate` semaphore, so the store's view is always consistent with the in-memory state.
### Post-(re)load condition-node re-assert (#487)
A deploy that triggers a **full address-space rebuild** clears `_alarmConditions` and re-materialises every
condition node fresh — inactive, acked, confirmed, unshelved. The engine, reloading in parallel, restores each
alarm's persisted state and re-derives `Active` from the predicate, so it ends up *correct*; but a still-active
alarm has **no transition to report**, so `LoadAsync` yields `EmissionKind.None` and `OnEngineEmission` drops it.
Nothing then writes the node. Before this fix, an active alarm with static dependencies read **normal** to every
OPC UA client after such a deploy, until its next real transition — which for a static-dependency alarm may
never come.
`ScriptedAlarmHostActor.ReassertConditionNodes` closes it. After each load completes it reads
`ScriptedAlarmEngine.GetProjections()` — a plain read returning `ScriptedAlarmProjection` (condition state +
the definition's severity + the resolved message template + last-observed worst-input quality) — and sends one
`OpcUaPublishActor.AlarmStateUpdate` per alarm that is **not** in the Part 9 no-event position.
Four properties are load-bearing:
- **Node-only.** It sends `AlarmStateUpdate` and nothing else — never the `alerts` topic, never the telemetry
hub. `alerts` is the historization path, so an alerts row per alarm per deploy would append a duplicate
historian / AVEVA record every time anyone deploys. This is why the re-assert reads a projection instead of
re-emitting: `ScriptedAlarmProjection` carries no `EmissionKind`, so there is nothing for the emission
machinery — or a future refactor — to mistake for a transition.
- **Only non-normal alarms.** An alarm in the no-event position (enabled, inactive, acked, confirmed,
unshelved) already matches what the materialise built, so it is skipped. That keeps a never-fired alarm
byte-identical to a deploy without this pass — including its Message, which the materialise seeds with the
alarm's display name — and writes nothing at all on the common all-normal deploy.
- **Ordering.** `DriverHostActor` tells `RebuildAddressSpace` to the publish actor *before* it tells
`ApplyScriptedAlarms` to the host, and the re-assert runs later still (after an awaited `LoadAsync` pipes
back). Both are local `Tell`s, which enqueue synchronously, so the re-materialise is already queued ahead of
these writes.
- **No duplicate Part 9 events.** `WriteAlarmCondition`'s delta-gate compares the incoming snapshot against
the node's *current* state: a re-assert onto a freshly materialised (normal) node is a genuine delta and
fires exactly one condition event — correct, since the rebuild reset what clients see — while a re-assert
onto a surgically-preserved node that already holds the same state is suppressed. The write is ungated by
redundancy role, like every other node write here; the Secondary keeps its address space warm for failover.
The timestamp carried is the condition's own `LastTransitionUtc`, **not** the deploy instant, so a client
reading `Time` / `ReceiveTime` after a rebuild still sees when the alarm actually went active.
## Source integration ## Source integration
`ScriptedAlarmSource` (`ScriptedAlarmSource.cs`) adapts the engine to the driver-agnostic `IAlarmSource` interface. The existing `AlarmSurfaceInvoker` + `GenericDriverNodeManager` fan-out consumes it the same way it consumes Galaxy / AB CIP / FOCAS sources — there is no scripted-alarm-specific code path in the server plumbing. From that point on, the flow into `AlarmConditionState` nodes, the `AlarmAck` session check, and the Historian sink is shared — see [AlarmTracking.md](AlarmTracking.md). `ScriptedAlarmSource` (`ScriptedAlarmSource.cs`) adapts the engine to the driver-agnostic `IAlarmSource` interface. The existing `AlarmSurfaceInvoker` + `GenericDriverNodeManager` fan-out consumes it the same way it consumes Galaxy / AB CIP / FOCAS sources — there is no scripted-alarm-specific code path in the server plumbing. From that point on, the flow into `AlarmConditionState` nodes, the `AlarmAck` session check, and the Historian sink is shared — see [AlarmTracking.md](AlarmTracking.md).
+33
View File
@@ -128,6 +128,39 @@ transports. ScadaBridge itself has since closed that gap with this identical
interceptor pattern, so Phase 5 ships authenticated from the start rather than deferring auth to a interceptor pattern, so Phase 5 ships authenticated from the start rather than deferring auth to a
later hardening pass. See the design doc's superseded note for the full history. later hardening pass. See the design doc's superseded note for the full history.
## Upgrading an existing deployment: populate `ClusterNode.GrpcPort` first (#493)
`GrpcPort` is a **nullable column added by Phase 5**, and nothing backfills it onto rows that
already existed. On an upgraded deployment every `ClusterNode` row therefore carries `NULL`,
central finds **no dial targets**, and flipping `TelemetryDial:Mode` to `Grpc` connects to
nothing. Fresh installs are unaffected. `AkkaPort` did not have this problem only because it is
`NOT NULL` with a default of 4053.
Nothing fails loudly, which is what makes this worth stating up front — the code degrades
gracefully, once per node, throttled:
```
ClusterNode <id> has no GrpcPort; it exposes no telemetry stream and is skipped in this dial refresh
(further skips of this node are silent)
```
Other symptoms: no `ESTABLISHED` sockets on the telemetry port (a `LISTEN` but nothing else in
`docker exec <node> cat /proc/net/tcp6`), and the `/alerts` / `/script-log` pill never turning
live in `Grpc` mode.
**Before flipping any node to `Grpc`**, set each driver node's `GrpcPort` to that node's own
`Telemetry:GrpcListenPort` — via the AdminUI node editor, or directly:
```sql
UPDATE dbo.ClusterNode SET GrpcPort = <that node's Telemetry:GrpcListenPort> WHERE NodeId = '<node>';
```
There is deliberately **no EF data migration** backfilling this. The correct value is that node's
own listener port, which is per-deployment configuration the database cannot derive; a migration
could only invent one, and a *wrong* dial target is worse than the `NULL` the dialer already skips
explicitly. The docker-dev seed does backfill (`GrpcPort IS NULL AND CreatedBy = 'docker-dev-seed'`
→ 4056) because there the port is fixed by `docker-compose.yml` and therefore actually knowable.
## Reconnect and reliability ## Reconnect and reliability
Central's `TelemetryDialSupervisor` (an admin-role actor) runs **one reconnecting dialer per Central's `TelemetryDialSupervisor` (an admin-role actor) runs **one reconnecting dialer per
+8 -1
View File
@@ -94,7 +94,14 @@ dual-namespace + event-delivery), Runtime 377 (+ VT reassert regression), AdminU
## Documented follow-ups (non-blocking) ## Documented follow-ups (non-blocking)
- **Scripted-alarm redeploy recovery (CONFIRMED pre-existing bug, deferred — deserves its own careful fix). Filed as issue #487.** - **Scripted-alarm redeploy recovery — FIXED (issue #487), see `docs/ScriptedAlarms.md` §"Post-(re)load
condition-node re-assert".** Landed as the proposed shape below: `ScriptedAlarmHostActor.ReassertConditionNodes`
in `OnAlarmsLoaded`, node-only, never the `alerts` topic. Two deviations from the sketch, both deliberate:
it reads a new `ScriptedAlarmEngine.GetProjections()` rather than `GetState(alarmId)` + `LoadedAlarmIds`
(a projection also carries the definition's severity, the resolved message template, and the last-observed
worst-input quality — `GetState` alone would have clobbered severity/message on the node); and it skips
alarms sitting in the Part 9 no-event position, so a deploy where nothing is in alarm writes no condition
nodes at all. The original deferral text follows for context.
The scripted-alarm Part 9 condition node has the same redeploy-reset race the VT `ReassertValue` fix closed: The scripted-alarm Part 9 condition node has the same redeploy-reset race the VT `ReassertValue` fix closed:
`RebuildAddressSpace` clears `_alarmConditions` and `MaterialiseScriptedAlarms` recreates each condition `RebuildAddressSpace` clears `_alarmConditions` and `MaterialiseScriptedAlarms` recreates each condition
fresh/normal, but `ScriptedAlarmEngine.LoadAsync` reloads from the persisted state and yields fresh/normal, but `ScriptedAlarmEngine.LoadAsync` reloads from the persisted state and yields
@@ -60,6 +60,11 @@ public static class DraftValidator
/// credential in the ConfigDb — persisted, replicated to every node's artifact cache, and readable by /// credential in the ConfigDb — persisted, replicated to every node's artifact cache, and readable by
/// anyone with config access — while the runtime silently ignored it and the driver failed to connect. /// anyone with config access — while the runtime silently ignored it and the driver failed to connect.
/// Discarding a secret on read is not the same as refusing to store it.</para> /// Discarding a secret on read is not the same as refusing to store it.</para>
/// <para><b>Two of the three surfaces are checked here.</b> The third — a node's
/// <see cref="ClusterNode.DriverConfigOverridesJson"/> — is not reachable from
/// <see cref="DraftSnapshot"/> and is gated at its save instead (#499); see
/// <see cref="SqlCredentialGuard"/> for why a deploy gate is the wrong instrument for a value that
/// never enters the artifact.</para>
/// <para>Checked on <b>both</b> config surfaces a Sql driver reads: the instance's /// <para>Checked on <b>both</b> config surfaces a Sql driver reads: the instance's
/// <see cref="DriverInstance.DriverConfig"/> and the <see cref="Device.DeviceConfig"/> of every device /// <see cref="DriverInstance.DriverConfig"/> and the <see cref="Device.DeviceConfig"/> of every device
/// beneath it, because the two are merged before the DTO sees them — a credential pasted into the /// beneath it, because the two are merged before the DTO sees them — a credential pasted into the
@@ -74,7 +79,7 @@ public static class DraftValidator
/// </remarks> /// </remarks>
private static void ValidateSqlConnectionStringNotPersisted(DraftSnapshot draft, List<ValidationError> errors) private static void ValidateSqlConnectionStringNotPersisted(DraftSnapshot draft, List<ValidationError> errors)
{ {
const string ForbiddenKey = "connectionString"; const string ForbiddenKey = SqlCredentialGuard.ForbiddenKey;
var sqlInstanceIds = draft.DriverInstances var sqlInstanceIds = draft.DriverInstances
.Where(d => string.Equals(d.DriverType, Core.Abstractions.DriverTypeNames.Sql, StringComparison.Ordinal)) .Where(d => string.Equals(d.DriverType, Core.Abstractions.DriverTypeNames.Sql, StringComparison.Ordinal))
@@ -85,7 +90,7 @@ public static class DraftValidator
foreach (var d in draft.DriverInstances) foreach (var d in draft.DriverInstances)
{ {
if (!sqlInstanceIds.Contains(d.DriverInstanceId)) continue; if (!sqlInstanceIds.Contains(d.DriverInstanceId)) continue;
if (!HasTopLevelKey(d.DriverConfig, ForbiddenKey)) continue; if (!SqlCredentialGuard.CarriesLiteralConnectionString(d.DriverConfig)) continue;
errors.Add(new("SqlConnectionStringPersisted", errors.Add(new("SqlConnectionStringPersisted",
$"Sql driver instance '{d.DriverInstanceId}' has a '{ForbiddenKey}' key in its DriverConfig. " + $"Sql driver instance '{d.DriverInstanceId}' has a '{ForbiddenKey}' key in its DriverConfig. " +
"A Sql driver must name its credentials indirectly via 'connectionStringRef', which resolves " + "A Sql driver must name its credentials indirectly via 'connectionStringRef', which resolves " +
@@ -98,7 +103,7 @@ public static class DraftValidator
foreach (var dev in draft.Devices) foreach (var dev in draft.Devices)
{ {
if (!sqlInstanceIds.Contains(dev.DriverInstanceId)) continue; if (!sqlInstanceIds.Contains(dev.DriverInstanceId)) continue;
if (!HasTopLevelKey(dev.DeviceConfig, ForbiddenKey)) continue; if (!SqlCredentialGuard.CarriesLiteralConnectionString(dev.DeviceConfig)) continue;
errors.Add(new("SqlConnectionStringPersisted", errors.Add(new("SqlConnectionStringPersisted",
$"Device '{dev.DeviceId}' on Sql driver instance '{dev.DriverInstanceId}' has a " + $"Device '{dev.DeviceId}' on Sql driver instance '{dev.DriverInstanceId}' has a " +
$"'{ForbiddenKey}' key in its DeviceConfig. DeviceConfig is merged onto DriverConfig before " + $"'{ForbiddenKey}' key in its DeviceConfig. DeviceConfig is merged onto DriverConfig before " +
@@ -107,34 +112,6 @@ public static class DraftValidator
} }
} }
/// <summary>
/// True when <paramref name="json"/> is a JSON object carrying <paramref name="key"/> at its top level,
/// matched case-insensitively. Never throws — a blank, malformed or non-object blob simply has no keys,
/// and shaping the config JSON is another rule's job.
/// </summary>
/// <param name="json">The config blob to inspect.</param>
/// <param name="key">The property name to look for.</param>
/// <returns><see langword="true"/> when the key is present at the top level.</returns>
private static bool HasTopLevelKey(string? json, string key)
{
if (string.IsNullOrWhiteSpace(json)) return false;
try
{
using var doc = System.Text.Json.JsonDocument.Parse(json);
if (doc.RootElement.ValueKind != System.Text.Json.JsonValueKind.Object) return false;
foreach (var property in doc.RootElement.EnumerateObject())
{
if (string.Equals(property.Name, key, StringComparison.OrdinalIgnoreCase)) return true;
}
return false;
}
catch (System.Text.Json.JsonException)
{
return false;
}
}
/// <summary>WP7 Calculation-driver deploy gates. For every tag bound to a <c>Calculation</c> driver: /// <summary>WP7 Calculation-driver deploy gates. For every tag bound to a <c>Calculation</c> driver:
/// <list type="number"> /// <list type="number">
/// <item><b>scriptId existence</b> — the tag's <c>TagConfig.scriptId</c> must be present and resolve to /// <item><b>scriptId existence</b> — the tag's <c>TagConfig.scriptId</c> must be present and resolve to
@@ -0,0 +1,139 @@
using System.Text.Json;
namespace ZB.MOM.WW.OtOpcUa.Configuration.Validation;
/// <summary>
/// The one place that decides whether a persisted config blob carries a literal Sql
/// <c>connectionString</c> (Gitea #498 / #499).
/// </summary>
/// <remarks>
/// <para>
/// A <c>Sql</c> driver names its credentials indirectly — <c>connectionStringRef</c> resolves from
/// the environment / secret store at Initialize — so nothing persisted should ever contain a
/// database password. The typed <c>SqlDriverConfigDto</c> already drops a <c>connectionString</c>
/// key on <b>read</b>, but discarding a secret on read is not the same as refusing to store it:
/// config blobs are schemaless JSON columns authored through raw-JSON textareas, so the key can
/// be written even though the runtime ignores it.
/// </para>
/// <para>
/// There are <b>three</b> surfaces a Sql driver's config is persisted on, and they need different
/// enforcement points:
/// <list type="bullet">
/// <item><c>DriverInstance.DriverConfig</c> and <c>Device.DeviceConfig</c> — both in the
/// deployed artifact, both visible to <c>DraftSnapshot</c>, both gated by
/// <c>DraftValidator</c>.</item>
/// <item><c>ClusterNode.DriverConfigOverridesJson</c> — the per-node override map. It is
/// <b>not</b> in <c>DraftSnapshot</c>, and deliberately does not go there: adding a
/// <c>ClusterNode</c> collection would widen the snapshot (and every builder of one) for a
/// single rule about a value that is not part of the artifact at all. More to the point, a
/// deploy gate is the wrong instrument here — the artifact never carries node overrides, so
/// blocking a deploy would not stop the credential being stored. It is already in the
/// database by then. This surface is therefore gated at the <b>save</b>, which is the only
/// point where "refuse to store it" is literally true.</item>
/// </list>
/// </para>
/// <para>
/// <b>Deliberately narrow.</b> This checks the <c>Sql</c> driver's <c>connectionString</c> key
/// only — not a general "credential-shaped key" sweep over every driver type. A broader rule
/// would start refusing configs that are legitimate today for drivers that never made this
/// guarantee, turning defence-in-depth into a regression. Widen it per driver, as each driver
/// gains an indirect-credential contract of its own.
/// </para>
/// </remarks>
public static class SqlCredentialGuard
{
/// <summary>The config key a Sql driver must never persist. Matched case-insensitively.</summary>
public const string ForbiddenKey = "connectionString";
/// <summary>
/// Whether <paramref name="configJson"/> is a JSON object carrying <see cref="ForbiddenKey"/> at
/// its top level.
/// </summary>
/// <remarks>
/// Matched <b>case-insensitively</b>: <c>System.Text.Json</c> binds <c>ConnectionString</c> to a
/// <c>connectionString</c> property by default, so a case variant is the same key, not a
/// different one. Only the top level is scanned — the DTO is flat, so a nested occurrence cannot
/// bind and is not the credential-shaped mistake this exists to catch.
/// </remarks>
/// <param name="configJson">The config blob to inspect.</param>
/// <returns><see langword="true"/> when the key is present at the top level.</returns>
public static bool CarriesLiteralConnectionString(string? configJson)
{
if (string.IsNullOrWhiteSpace(configJson)) return false;
try
{
using var doc = JsonDocument.Parse(configJson);
return doc.RootElement.ValueKind == JsonValueKind.Object
&& HasTopLevelKey(doc.RootElement, ForbiddenKey);
}
catch (JsonException)
{
// A malformed blob simply has no keys. Shaping the config JSON is another rule's job, and
// throwing here would turn a formatting mistake into a credential-guard failure.
return false;
}
}
/// <summary>
/// The <c>DriverInstanceId</c>s in a <c>ClusterNode.DriverConfigOverridesJson</c> map whose
/// override object carries a literal connection string, restricted to instances that are actually
/// Sql drivers.
/// </summary>
/// <remarks>
/// The override map is shaped <c>{ "&lt;DriverInstanceId&gt;": { …driver config keys… } }</c> and is
/// merged onto the cluster-level <c>DriverConfig</c>, so a credential pasted into one lands in
/// exactly the place <see cref="CarriesLiteralConnectionString"/> already refuses on the driver
/// itself.
/// </remarks>
/// <param name="overridesJson">The node's override map. Null / blank / malformed yields no
/// violations.</param>
/// <param name="sqlDriverInstanceIds">The ids of driver instances whose <c>DriverType</c> is
/// <c>Sql</c>. Keys outside this set are ignored — a non-Sql driver never made the indirect-credential
/// guarantee, so flagging it would be a regression, not defence in depth.</param>
/// <returns>The offending driver-instance ids, in document order; empty when there are none.</returns>
public static IReadOnlyList<string> FindNodeOverrideViolations(
string? overridesJson, IReadOnlyCollection<string> sqlDriverInstanceIds)
{
if (string.IsNullOrWhiteSpace(overridesJson) || sqlDriverInstanceIds.Count == 0)
{
return [];
}
var sqlIds = sqlDriverInstanceIds as IReadOnlySet<string>
?? sqlDriverInstanceIds.ToHashSet(StringComparer.Ordinal);
try
{
using var doc = JsonDocument.Parse(overridesJson);
if (doc.RootElement.ValueKind != JsonValueKind.Object) return [];
List<string>? offenders = null;
foreach (var entry in doc.RootElement.EnumerateObject())
{
if (!sqlIds.Contains(entry.Name)) continue;
if (entry.Value.ValueKind != JsonValueKind.Object) continue;
if (!HasTopLevelKey(entry.Value, ForbiddenKey)) continue;
(offenders ??= []).Add(entry.Name);
}
return (IReadOnlyList<string>?)offenders ?? [];
}
catch (JsonException)
{
return [];
}
}
/// <summary>Whether <paramref name="obj"/> carries <paramref name="key"/> at its top level,
/// case-insensitively.</summary>
private static bool HasTopLevelKey(JsonElement obj, string key)
{
foreach (var property in obj.EnumerateObject())
{
if (string.Equals(property.Name, key, StringComparison.OrdinalIgnoreCase)) return true;
}
return false;
}
}
@@ -334,6 +334,48 @@ public sealed class ScriptedAlarmEngine : IDisposable
public IReadOnlyCollection<AlarmConditionState> GetAllStates() public IReadOnlyCollection<AlarmConditionState> GetAllStates()
=> _alarms.Values.Select(a => a.Condition).ToArray(); => _alarms.Values.Select(a => a.Condition).ToArray();
/// <summary>
/// #487 — the currently-held condition of every loaded alarm, packaged with the definition's
/// severity, the rendered message template and the last-observed input quality, i.e. everything
/// a caller needs to <b>project</b> the alarm onto an OPC UA condition node without waiting for
/// the next transition.
/// </summary>
/// <remarks>
/// <para>
/// <b>This is a read, not an emission.</b> It deliberately returns a distinct type rather
/// than a <see cref="ScriptedAlarmEvent"/>, and it never touches <see cref="OnEvent"/>. A
/// still-active alarm produces <see cref="EmissionKind.None"/> at load (there is no
/// transition to report), so the transition stream cannot carry the current state — that is
/// exactly why the caller needs this. Routing these through the emission path instead would
/// append a duplicate historian / <c>alerts</c> row on every deploy, so the shape here is
/// intentionally one the emission machinery cannot consume.
/// </para>
/// <para>
/// <b>Synchronization.</b> Reads <c>_alarms</c> and <c>_valueCache</c> — both
/// <see cref="ConcurrentDictionary{TKey, TValue}"/> — without taking <c>_evalGate</c>, the
/// same posture as <see cref="GetState"/> / <see cref="GetAllStates"/>. Each projection is
/// individually coherent; the set as a whole is not an atomic snapshot across a concurrent
/// re-evaluation. That is sufficient for the node-projection use: an evaluation racing this
/// read publishes its own transition through the normal path.
/// </para>
/// </remarks>
/// <returns>One projection per loaded alarm; empty before the first <see cref="LoadAsync"/>.</returns>
public IReadOnlyList<ScriptedAlarmProjection> GetProjections()
{
var projections = new List<ScriptedAlarmProjection>(_alarms.Count);
foreach (var (alarmId, state) in _alarms)
{
projections.Add(new ScriptedAlarmProjection(
AlarmId: alarmId,
Severity: state.Definition.Severity,
Message: MessageTemplate.Resolve(state.Definition.MessageTemplate, TryLookup),
Condition: state.Condition,
WorstInputStatusCode: LastWorstStatus(alarmId)));
}
return projections;
}
/// <summary>Acknowledges the specified alarm on behalf of the given user.</summary> /// <summary>Acknowledges the specified alarm on behalf of the given user.</summary>
/// <param name="alarmId">The alarm identifier.</param> /// <param name="alarmId">The alarm identifier.</param>
/// <param name="user">The user performing the acknowledgment.</param> /// <param name="user">The user performing the acknowledgment.</param>
@@ -974,6 +1016,26 @@ public sealed record ScriptedAlarmEvent(
// the top-2 severity bits. Default 0u == Good keeps every existing constructor call unchanged. // the top-2 severity bits. Default 0u == Good keeps every existing constructor call unchanged.
uint WorstInputStatusCode = 0u); uint WorstInputStatusCode = 0u);
/// <summary>
/// #487 — a loaded alarm's current condition, ready to be projected onto its OPC UA node.
/// Produced by <see cref="ScriptedAlarmEngine.GetProjections"/> on demand; it is <b>not</b> an
/// emission and never travels the <see cref="ScriptedAlarmEngine.OnEvent"/> path, so it can never
/// become an <c>alerts</c> row or a historian record. Deliberately a separate type from
/// <see cref="ScriptedAlarmEvent"/> — it carries no <see cref="EmissionKind"/>, so there is nothing
/// for a future refactor to mistake for a transition.
/// </summary>
/// <param name="AlarmId">The alarm identifier (also its OPC UA condition NodeId).</param>
/// <param name="Severity">The definition's severity bucket.</param>
/// <param name="Message">The message template resolved against the current value cache.</param>
/// <param name="Condition">The Part 9 condition state the engine currently holds.</param>
/// <param name="WorstInputStatusCode">The last-observed worst OPC UA StatusCode across the alarm's inputs.</param>
public sealed record ScriptedAlarmProjection(
string AlarmId,
AlarmSeverity Severity,
string Message,
AlarmConditionState Condition,
uint WorstInputStatusCode);
/// <summary> /// <summary>
/// Upstream source abstraction — intentionally identical shape to the virtual-tag /// Upstream source abstraction — intentionally identical shape to the virtual-tag
/// engine's so Stream G can compose them behind one driver bridge. /// engine's so Stream G can compose them behind one driver bridge.
@@ -81,7 +81,7 @@ else
<tr> <tr>
<td><span class="mono small">@c.Subject</span></td> <td><span class="mono small">@c.Subject</span></td>
<td><span class="mono small">@c.Issuer</span></td> <td><span class="mono small">@c.Issuer</span></td>
<td><span class="mono small">@c.Thumbprint[..16]…</span></td> <td><span class="mono small">@DisplayText.Abbreviate(c.Thumbprint, 16)</span></td>
<td>@c.NotBefore.ToString("u")</td> <td>@c.NotBefore.ToString("u")</td>
<td>@c.NotAfter.ToString("u")</td> <td>@c.NotAfter.ToString("u")</td>
@if (store.Kind is StoreKind.Trusted or StoreKind.Rejected) @if (store.Kind is StoreKind.Trusted or StoreKind.Rejected)
@@ -58,7 +58,7 @@ else
} }
else else
{ {
<div class="kv"><span class="k">Revision</span><span class="v mono">@_lastDeployment.RevisionHash[..16]…</span></div> <div class="kv"><span class="k">Revision</span><span class="v mono">@DisplayText.Abbreviate(_lastDeployment.RevisionHash, 16)</span></div>
<div class="kv"><span class="k">Status</span><span class="v">@_lastDeployment.Status</span></div> <div class="kv"><span class="k">Status</span><span class="v">@_lastDeployment.Status</span></div>
<div class="kv"><span class="k">Created</span><span class="v">@_lastDeployment.CreatedAtUtc.ToString("u")</span></div> <div class="kv"><span class="k">Created</span><span class="v">@_lastDeployment.CreatedAtUtc.ToString("u")</span></div>
@if (_lastDeployment.SealedAtUtc is not null) @if (_lastDeployment.SealedAtUtc is not null)
@@ -9,6 +9,8 @@
@using System.ComponentModel.DataAnnotations @using System.ComponentModel.DataAnnotations
@using ZB.MOM.WW.OtOpcUa.Configuration @using ZB.MOM.WW.OtOpcUa.Configuration
@using ZB.MOM.WW.OtOpcUa.Configuration.Entities @using ZB.MOM.WW.OtOpcUa.Configuration.Entities
@using ZB.MOM.WW.OtOpcUa.Configuration.Validation
@using ZB.MOM.WW.OtOpcUa.Core.Abstractions
@inject IDbContextFactory<OtOpcUaConfigDbContext> DbFactory @inject IDbContextFactory<OtOpcUaConfigDbContext> DbFactory
@inject NavigationManager Nav @inject NavigationManager Nav
@inject AuthenticationStateProvider AuthState @inject AuthenticationStateProvider AuthState
@@ -34,6 +36,10 @@ else
{ {
<EditForm Model="_form" OnValidSubmit="SubmitAsync" FormName="nodeEdit"> <EditForm Model="_form" OnValidSubmit="SubmitAsync" FormName="nodeEdit">
<DataAnnotationsValidator /> <DataAnnotationsValidator />
@* Without this, a validation failure suppresses OnValidSubmit with NO feedback whatsoever —
the Save button simply does nothing and the operator has no way to tell why. That is how the
NodeId-regex defect above stayed invisible. *@
<ValidationSummary class="alert alert-danger py-2 px-3" />
<section class="panel rise" style="animation-delay:.02s"> <section class="panel rise" style="animation-delay:.02s">
<div class="panel-head">Identity</div> <div class="panel-head">Identity</div>
<div style="padding:1rem"> <div style="padding:1rem">
@@ -178,6 +184,33 @@ else
} }
await using var db = await DbFactory.CreateDbContextAsync(); await using var db = await DbFactory.CreateDbContextAsync();
// #499 — the third Sql-credential surface. DriverConfigOverridesJson is merged onto the
// cluster-level DriverConfig, so a literal connectionString pasted here leaks exactly as one
// pasted into the driver itself. The deploy gate cannot see it (DraftSnapshot carries no
// ClusterNode rows) and would be the wrong place anyway: node overrides never enter the
// artifact, so by the time a deploy ran the credential would already be stored. Refuse the
// save instead. The message names the driver instance but NEVER the value.
if (!string.IsNullOrWhiteSpace(_form.DriverConfigOverridesJson))
{
var sqlDriverIds = await db.DriverInstances
.Where(d => d.ClusterId == ClusterId && d.DriverType == DriverTypeNames.Sql)
.Select(d => d.DriverInstanceId)
.ToListAsync();
var offenders = SqlCredentialGuard.FindNodeOverrideViolations(
_form.DriverConfigOverridesJson, sqlDriverIds);
if (offenders.Count > 0)
{
_error =
$"Driver config overrides carry a '{SqlCredentialGuard.ForbiddenKey}' for Sql driver " +
$"instance(s) {string.Join(", ", offenders)}. A Sql driver must name its credentials " +
"indirectly via 'connectionStringRef', which resolves from the environment / secret " +
"store at Initialize; a literal connection string here would be stored in the config " +
"database, and the runtime ignores it anyway. Remove the key.";
return;
}
}
if (IsNew) if (IsNew)
{ {
if (await db.ClusterNodes.AnyAsync(n => n.NodeId == _form.NodeId)) if (await db.ClusterNodes.AnyAsync(n => n.NodeId == _form.NodeId))
@@ -194,7 +227,7 @@ else
OpcUaPort = _form.OpcUaPort, OpcUaPort = _form.OpcUaPort,
DashboardPort = _form.DashboardPort, DashboardPort = _form.DashboardPort,
ApplicationUri = _form.ApplicationUri, ApplicationUri = _form.ApplicationUri,
ServiceLevelBase = _form.ServiceLevelBase, ServiceLevelBase = (byte)_form.ServiceLevelBase,
Enabled = _form.Enabled, Enabled = _form.Enabled,
DriverConfigOverridesJson = string.IsNullOrWhiteSpace(_form.DriverConfigOverridesJson) ? null : _form.DriverConfigOverridesJson, DriverConfigOverridesJson = string.IsNullOrWhiteSpace(_form.DriverConfigOverridesJson) ? null : _form.DriverConfigOverridesJson,
CreatedAt = DateTime.UtcNow, CreatedAt = DateTime.UtcNow,
@@ -210,7 +243,7 @@ else
entity.OpcUaPort = _form.OpcUaPort; entity.OpcUaPort = _form.OpcUaPort;
entity.DashboardPort = _form.DashboardPort; entity.DashboardPort = _form.DashboardPort;
entity.ApplicationUri = _form.ApplicationUri; entity.ApplicationUri = _form.ApplicationUri;
entity.ServiceLevelBase = _form.ServiceLevelBase; entity.ServiceLevelBase = (byte)_form.ServiceLevelBase;
entity.Enabled = _form.Enabled; entity.Enabled = _form.Enabled;
entity.DriverConfigOverridesJson = string.IsNullOrWhiteSpace(_form.DriverConfigOverridesJson) ? null : _form.DriverConfigOverridesJson; entity.DriverConfigOverridesJson = string.IsNullOrWhiteSpace(_form.DriverConfigOverridesJson) ? null : _form.DriverConfigOverridesJson;
} }
@@ -254,14 +287,26 @@ else
private sealed class FormModel private sealed class FormModel
{ {
[Required, RegularExpression("^[A-Za-z0-9_-]+$")] // The ':' and '.' are REQUIRED, not permissive. Every NodeId in this system is "host:port" —
// ClusterRoleInfo and ConfigPublishCoordinator derive it as member.Address.Host:Port, the seed
// writes "central-1:4053", and NodeDeploymentState.NodeId is FK-bound to it. The previous
// pattern forbade the colon, so EVERY existing node failed validation and OnValidSubmit never
// ran: the Save button did nothing at all, silently. Hostnames may also be dotted
// (line3-opc-a.plant.local:4053).
[Required, RegularExpression("^[A-Za-z0-9_.:-]+$")]
public string NodeId { get; set; } = ""; public string NodeId { get; set; } = "";
[Required] public string Host { get; set; } = ""; [Required] public string Host { get; set; } = "";
[Range(1, 65535)] public int OpcUaPort { get; set; } = 4840; [Range(1, 65535)] public int OpcUaPort { get; set; } = 4840;
[Range(1, 65535)] public int DashboardPort { get; set; } = 8081; [Range(1, 65535)] public int DashboardPort { get; set; } = 8081;
[Required, RegularExpression("^urn:[A-Za-z0-9_:./-]+$")] [Required, RegularExpression("^urn:[A-Za-z0-9_:./-]+$")]
public string ApplicationUri { get; set; } = ""; public string ApplicationUri { get; set; } = "";
[Range(0, 255)] public byte ServiceLevelBase { get; set; } = 200; // int, NOT byte. Blazor's InputNumber<T> rejects System.Byte outright — its static
// constructor throws "The type 'System.Byte' is not a supported numeric type", which escapes
// BuildRenderTree unhandled and takes the WHOLE page to an HTTP 500. Binding this field to
// the entity's own byte type meant /clusters/{id}/nodes/{nodeId} and .../nodes/new never
// rendered at all. The Range attribute still holds it to a byte's domain, and SubmitAsync
// casts on the way to the entity, so the stored type is unchanged.
[Range(0, 255)] public int ServiceLevelBase { get; set; } = 200;
public bool Enabled { get; set; } = true; public bool Enabled { get; set; } = true;
public string? DriverConfigOverridesJson { get; set; } public string? DriverConfigOverridesJson { get; set; }
} }
@@ -55,7 +55,7 @@
{ {
<tr> <tr>
<td><code>@Short(d.DeploymentId)</code></td> <td><code>@Short(d.DeploymentId)</code></td>
<td><code>@d.RevisionHash[..12]…</code></td> <td><code>@DisplayText.Abbreviate(d.RevisionHash, 12)</code></td>
<td>@d.Status</td> <td>@d.Status</td>
<td>@d.CreatedBy</td> <td>@d.CreatedBy</td>
<td>@d.CreatedAtUtc.ToString("u")</td> <td>@d.CreatedAtUtc.ToString("u")</td>
@@ -113,7 +113,7 @@
_lastSuccess = result.Outcome == StartDeploymentOutcome.Accepted; _lastSuccess = result.Outcome == StartDeploymentOutcome.Accepted;
_lastMessage = result.Outcome switch _lastMessage = result.Outcome switch
{ {
StartDeploymentOutcome.Accepted => $"Deployment {Short(result.DeploymentId!.Value.Value)} dispatched (rev {result.RevisionHash!.Value.Value[..12]}…).", StartDeploymentOutcome.Accepted => $"Deployment {Short(result.DeploymentId!.Value.Value)} dispatched (rev {DisplayText.Abbreviate(result.RevisionHash!.Value.Value, 12)}).",
StartDeploymentOutcome.AnotherDeploymentInFlight => result.Message ?? "Another deployment is already in flight.", StartDeploymentOutcome.AnotherDeploymentInFlight => result.Message ?? "Another deployment is already in flight.",
StartDeploymentOutcome.NoChanges => "No changes detected since the last sealed deployment.", StartDeploymentOutcome.NoChanges => "No changes detected since the last sealed deployment.",
_ => result.Message ?? "Deployment rejected.", _ => result.Message ?? "Deployment rejected.",
@@ -38,7 +38,7 @@ else
<span class="mono">@s.ScriptId</span> <span class="mono">@s.ScriptId</span>
&middot; <span>@s.Name</span> &middot; <span>@s.Name</span>
&middot; <span class="chip chip-idle ms-1">@s.Language</span> &middot; <span class="chip chip-idle ms-1">@s.Language</span>
<span class="text-muted small ms-2 mono">hash=@s.SourceHash[..12]…</span> <span class="text-muted small ms-2 mono">hash=@DisplayText.Abbreviate(s.SourceHash, 12)</span>
</summary> </summary>
<div style="padding:0 1rem 1rem"> <div style="padding:0 1rem 1rem">
<div class="d-flex mb-2"> <div class="d-flex mb-2">
@@ -0,0 +1,45 @@
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Components.Shared;
/// <summary>
/// Rendering helpers for operator-facing text that comes out of the database.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why this exists (#504).</b> Several pages abbreviated a hash / thumbprint with the range
/// operator — <c>@s.SourceHash[..12]…</c> — which throws
/// <see cref="ArgumentOutOfRangeException"/> the moment the value is shorter than the slice.
/// In Blazor that escapes <c>BuildRenderTree</c> unhandled and takes the <b>whole page</b> to an
/// HTTP 500, not just the offending row. Nothing enforces a minimum length at the schema, the
/// entity, or the render layer, so any hand-written / seeded / migrated row is a live page-kill.
/// The docker-dev seed proved it: its placeholder <c>SourceHash</c> values (<c>h1</c>,
/// <c>h-abs-hr200</c>) made <c>/scripts</c> a hard 500 on every stock bring-up.
/// </para>
/// <para>
/// Use <see cref="Abbreviate"/> for <b>every</b> such prefix. It never throws, and it appends the
/// ellipsis only when it actually truncated — a 2-character hash renders as <c>h1</c>, not
/// <c>h1…</c>, so the display never implies more text than exists.
/// </para>
/// </remarks>
public static class DisplayText
{
/// <summary>Placeholder rendered for a null / blank value. Matches the em-dash convention the
/// Admin UI already uses for "nothing to show" cells.</summary>
public const string Empty = "—";
/// <summary>
/// The first <paramref name="maxLength"/> characters of <paramref name="value"/>, followed by an
/// ellipsis when (and only when) characters were dropped.
/// </summary>
/// <param name="value">The value to abbreviate. May be null, blank, or shorter than
/// <paramref name="maxLength"/> — none of which throw.</param>
/// <param name="maxLength">Maximum number of characters to keep. Values below 1 are treated as 1,
/// so a caller cannot turn a display bug into a crash.</param>
/// <returns><see cref="Empty"/> for a null/blank value; the value verbatim when it already fits;
/// otherwise the truncated prefix plus an ellipsis.</returns>
public static string Abbreviate(string? value, int maxLength)
{
if (string.IsNullOrWhiteSpace(value)) return Empty;
if (maxLength < 1) maxLength = 1;
return value.Length <= maxLength ? value : string.Concat(value.AsSpan(0, maxLength), "…");
}
}
@@ -170,6 +170,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// <summary>Interval between bounded post-connect re-discovery passes. Production default 2s; tests /// <summary>Interval between bounded post-connect re-discovery passes. Production default 2s; tests
/// inject a tiny value so the loop runs without real-time waits.</summary> /// inject a tiny value so the loop runs without real-time waits.</summary>
private readonly TimeSpan _rediscoverInterval; private readonly TimeSpan _rediscoverInterval;
private readonly TimeSpan _healthPollInterval;
/// <summary>Cap on the number of post-connect re-discovery passes — a backstop so a never-stabilising /// <summary>Cap on the number of post-connect re-discovery passes — a backstop so a never-stabilising
/// (or perpetually-empty) discovered set cannot spin the loop forever. Production default 15.</summary> /// (or perpetually-empty) discovered set cannot spin the loop forever. Production default 15.</summary>
@@ -239,6 +240,9 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// <param name="rediscoverDiscoverTimeout">Optional per-pass timeout for <see cref="ITagDiscovery.DiscoverAsync"/>; defaults to 30 seconds.</param> /// <param name="rediscoverDiscoverTimeout">Optional per-pass timeout for <see cref="ITagDiscovery.DiscoverAsync"/>; defaults to 30 seconds.</param>
/// <param name="invoker">Optional Phase 6.1 resilience invoker wrapping this driver's capability /// <param name="invoker">Optional Phase 6.1 resilience invoker wrapping this driver's capability
/// calls; defaults to <see cref="NullDriverCapabilityInvoker"/> (pass-through) when not supplied.</param> /// calls; defaults to <see cref="NullDriverCapabilityInvoker"/> (pass-through) when not supplied.</param>
/// <param name="healthPollInterval">Optional period of the health-poll heartbeat, which also drives the
/// subscription reconcile (<see cref="ReconcileSubscription"/>); defaults to
/// <see cref="HealthPollInterval"/>. Exists so a test can drive the reconcile without waiting 30 s.</param>
/// <returns>A <see cref="Akka.Actor.Props"/> instance configured to create the wrapped <see cref="DriverInstanceActor"/>.</returns> /// <returns>A <see cref="Akka.Actor.Props"/> instance configured to create the wrapped <see cref="DriverInstanceActor"/>.</returns>
public static Props Props( public static Props Props(
IDriver driver, IDriver driver,
@@ -249,7 +253,8 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
TimeSpan? rediscoverInterval = null, TimeSpan? rediscoverInterval = null,
int rediscoverMaxAttempts = DefaultRediscoverMaxAttempts, int rediscoverMaxAttempts = DefaultRediscoverMaxAttempts,
TimeSpan? rediscoverDiscoverTimeout = null, TimeSpan? rediscoverDiscoverTimeout = null,
IDriverCapabilityInvoker? invoker = null) => IDriverCapabilityInvoker? invoker = null,
TimeSpan? healthPollInterval = null) =>
Akka.Actor.Props.Create(() => new DriverInstanceActor( Akka.Actor.Props.Create(() => new DriverInstanceActor(
driver, driver,
reconnectInterval ?? DefaultReconnectInterval, reconnectInterval ?? DefaultReconnectInterval,
@@ -259,7 +264,8 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
rediscoverInterval, rediscoverInterval,
rediscoverMaxAttempts, rediscoverMaxAttempts,
rediscoverDiscoverTimeout, rediscoverDiscoverTimeout,
invoker)); invoker,
healthPollInterval));
/// <summary> /// <summary>
/// Returns true when the driver should boot in DEV-STUB mode based on host platform and /// Returns true when the driver should boot in DEV-STUB mode based on host platform and
@@ -293,6 +299,8 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// <param name="rediscoverDiscoverTimeout">Per-pass timeout for <see cref="ITagDiscovery.DiscoverAsync"/>; defaults to 30 seconds.</param> /// <param name="rediscoverDiscoverTimeout">Per-pass timeout for <see cref="ITagDiscovery.DiscoverAsync"/>; defaults to 30 seconds.</param>
/// <param name="invoker">Phase 6.1 resilience invoker wrapping this driver's capability calls; /// <param name="invoker">Phase 6.1 resilience invoker wrapping this driver's capability calls;
/// defaults to <see cref="NullDriverCapabilityInvoker"/> (pass-through) when null.</param> /// defaults to <see cref="NullDriverCapabilityInvoker"/> (pass-through) when null.</param>
/// <param name="healthPollInterval">Period of the health-poll heartbeat, which also drives the
/// subscription reconcile; defaults to <see cref="HealthPollInterval"/> when null.</param>
public DriverInstanceActor( public DriverInstanceActor(
IDriver driver, IDriver driver,
TimeSpan reconnectInterval, TimeSpan reconnectInterval,
@@ -302,7 +310,8 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
TimeSpan? rediscoverInterval = null, TimeSpan? rediscoverInterval = null,
int rediscoverMaxAttempts = DefaultRediscoverMaxAttempts, int rediscoverMaxAttempts = DefaultRediscoverMaxAttempts,
TimeSpan? rediscoverDiscoverTimeout = null, TimeSpan? rediscoverDiscoverTimeout = null,
IDriverCapabilityInvoker? invoker = null) IDriverCapabilityInvoker? invoker = null,
TimeSpan? healthPollInterval = null)
{ {
_driver = driver; _driver = driver;
_invoker = invoker ?? NullDriverCapabilityInvoker.Instance; _invoker = invoker ?? NullDriverCapabilityInvoker.Instance;
@@ -313,6 +322,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
_rediscoverInterval = rediscoverInterval ?? DefaultRediscoverInterval; _rediscoverInterval = rediscoverInterval ?? DefaultRediscoverInterval;
_rediscoverMaxAttempts = rediscoverMaxAttempts; _rediscoverMaxAttempts = rediscoverMaxAttempts;
_rediscoverDiscoverTimeout = rediscoverDiscoverTimeout ?? DefaultRediscoverDiscoverTimeout; _rediscoverDiscoverTimeout = rediscoverDiscoverTimeout ?? DefaultRediscoverDiscoverTimeout;
_healthPollInterval = healthPollInterval ?? HealthPollInterval;
OtOpcUaTelemetry.DriverInstanceLifecycle.Add(1, OtOpcUaTelemetry.DriverInstanceLifecycle.Add(1,
new KeyValuePair<string, object?>("event", startStubbed ? "spawn_stub" : "spawn"), new KeyValuePair<string, object?>("event", startStubbed ? "spawn_stub" : "spawn"),
new KeyValuePair<string, object?>("driver_type", driver.DriverType)); new KeyValuePair<string, object?>("driver_type", driver.DriverType));
@@ -335,7 +345,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
// actor starts, before any state transition fires. Also start the periodic heartbeat so // actor starts, before any state transition fires. Also start the periodic heartbeat so
// long-lived Healthy drivers keep their snapshot fresh for newly-joined SignalR clients. // long-lived Healthy drivers keep their snapshot fresh for newly-joined SignalR clients.
PublishHealthSnapshot(); PublishHealthSnapshot();
Timers.StartPeriodicTimer("health-poll", HealthPollTick.Instance, HealthPollInterval); Timers.StartPeriodicTimer("health-poll", HealthPollTick.Instance, _healthPollInterval);
} }
private void Stubbed() private void Stubbed()
@@ -489,7 +499,58 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
// to Self (HandleSubscribeAsync already logged the underlying cause). Swallow it so it doesn't dead-letter. // to Self (HandleSubscribeAsync already logged the underlying cause). Swallow it so it doesn't dead-letter.
Receive<SubscriptionFailed>(msg => Receive<SubscriptionFailed>(msg =>
_log.Debug("DriverInstance {Id}: resubscribe reported failure: {Reason}", _driverInstanceId, msg.Reason)); _log.Debug("DriverInstance {Id}: resubscribe reported failure: {Reason}", _driverInstanceId, msg.Reason));
Receive<HealthPollTick>(_ => PublishHealthSnapshot()); Receive<HealthPollTick>(_ =>
{
PublishHealthSnapshot();
ReconcileSubscription();
});
}
/// <summary>
/// Periodic desired-vs-actual subscription reconcile (#488), riding the existing <c>health-poll</c>
/// tick. While Connected, a driver that has a desired subscription set but no live handle is
/// re-subscribed.
/// </summary>
/// <remarks>
/// <para>
/// The desired set is otherwise only (re)applied on a <b>Connected transition</b>
/// (<see cref="ResubscribeDesired"/>) or on a deploy's <see cref="SetDesiredSubscriptions"/>.
/// Nothing re-asserted it for a subscription that was lost while the driver <i>stayed</i>
/// Connected — a failed subscribe whose retry never came, or a handle dropped down an error
/// path. That is the hole this closes.
/// </para>
/// <para>
/// <b>This is a state-machine consistency check, not a probe.</b> It can only see that this
/// actor holds no handle; it cannot detect a handle that is present but dead server-side,
/// because <c>ISubscriptionHandle</c> is opaque (a <c>DiagnosticId</c> string and nothing
/// else). Detecting that needs a liveness member on <c>ISubscribable</c> implemented across
/// all eight drivers, each with different subscription mechanics — deliberately deferred
/// until a real occurrence shows the handle outliving the subscription.
/// </para>
/// <para>
/// Gated on <see cref="ISubscribable"/>: a driver that cannot subscribe at all may still have
/// been handed a desired set, and re-telling <c>Subscribe</c> to it every tick would fail
/// every tick, forever.
/// </para>
/// <para>
/// A redundant <c>Subscribe</c> is possible but harmless: a tick sitting in the mailbox ahead
/// of an already-queued <c>Subscribe</c> observes the same "no handle" state and tells a
/// second one. <see cref="HandleSubscribeAsync"/> is a <c>ReceiveAsync</c> (so no tick can be
/// processed mid-subscribe) and drops any prior subscription before establishing the new one,
/// so the duplicate costs one extra round-trip and converges. Suppressing it would need a
/// pending-subscribe flag threaded through every self-tell site — more state than the race
/// is worth.
/// </para>
/// </remarks>
private void ReconcileSubscription()
{
if (_driver is not ISubscribable) return;
if (_desiredRefs.Count == 0 || _subscriptionHandle is not null) return;
_log.Info(
"DriverInstance {Id}: connected with {Count} desired subscription ref(s) but no live subscription handle — re-subscribing (periodic reconcile)",
_driverInstanceId, _desiredRefs.Count);
Self.Tell(new Subscribe(_desiredRefs, _desiredInterval));
} }
private void Reconnecting() private void Reconnecting()
@@ -288,8 +288,101 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
// so a re-Apply with a fresh union simply supersedes the old one — no explicit unregister needed. // so a re-Apply with a fresh union simply supersedes the old one — no explicit unregister needed.
_mux?.Tell(new DependencyMuxActor.RegisterInterest(msg.DepRefs, Self)); _mux?.Tell(new DependencyMuxActor.RegisterInterest(msg.DepRefs, Self));
_log.Debug("ScriptedAlarmHost: loaded; registered mux interest for {Count} dep refs", msg.DepRefs.Count); _log.Debug("ScriptedAlarmHost: loaded; registered mux interest for {Count} dep refs", msg.DepRefs.Count);
ReassertConditionNodes();
} }
/// <summary>
/// #487 — re-project every loaded alarm that is NOT in the Part 9 "no-event" position back onto
/// its condition node, right after a (re)load.
///
/// <para>
/// <b>The bug this closes.</b> A deploy that triggers a FULL address-space rebuild clears
/// <c>_alarmConditions</c> and re-materialises every condition node fresh — inactive, acked,
/// confirmed, unshelved. The engine, meanwhile, restores each alarm's persisted state and
/// re-derives Active from the predicate, so a still-active alarm ends the reload
/// <i>correctly Active in the engine</i> but produces <see cref="EmissionKind.None"/> (there
/// is no transition to report) — and <see cref="OnEngineEmission"/> drops None. Nothing then
/// writes the node, so an active alarm with static dependencies reads NORMAL to every OPC UA
/// client until its next real transition, which may never come. Same shape as the VirtualTag
/// redeploy-reset the <c>ReassertValue</c> fix closed.
/// </para>
///
/// <para>
/// <b>Node-only, by construction.</b> This sends <see cref="OpcUaPublishActor.AlarmStateUpdate"/>
/// and nothing else — it never publishes to the <c>alerts</c> topic and never touches the
/// telemetry hub. That is the whole reason it reads <see cref="ScriptedAlarmEngine.GetProjections"/>
/// (a plain read returning <see cref="ScriptedAlarmProjection"/>) rather than re-emitting: an
/// <c>alerts</c> row per alarm per deploy would append a duplicate historian / AVEVA record
/// every time anyone deploys — the alarm analogue of the VirtualTag M1 historian issue.
/// </para>
///
/// <para>
/// <b>Ordering.</b> <c>DriverHostActor</c> Tells <c>RebuildAddressSpace</c> to the publish
/// actor BEFORE it Tells <see cref="ApplyScriptedAlarms"/> here, and this runs later still —
/// after an <c>await</c>ed <see cref="ScriptedAlarmEngine.LoadAsync"/> pipes back. Both Tells
/// enqueue synchronously into the publish actor's local mailbox, so the re-materialise is
/// already queued ahead of these writes: the node exists by the time they are handled.
/// </para>
///
/// <para>
/// <b>Why only non-normal alarms.</b> An alarm sitting in the no-event position already
/// matches what the materialise just built, so writing it would be a pure no-op on state —
/// but it would still overwrite the node's Message (the materialise seeds it with the alarm's
/// display name; a projection carries the resolved template). Skipping them keeps a
/// never-fired alarm byte-identical to a deploy without this pass, and keeps the write count
/// at zero on the overwhelmingly common all-normal deploy.
/// </para>
///
/// <para>
/// <b>Duplicate events.</b> None. <c>WriteAlarmCondition</c>'s delta-gate compares the
/// incoming snapshot against the node's CURRENT state, so a re-assert onto a freshly
/// materialised (normal) node is a genuine delta and fires one Part 9 event — correct, since
/// the rebuild reset what clients see — while a re-assert onto a surgically-preserved node
/// that already holds the same state is suppressed. This write is ungated by redundancy role,
/// like every other node write here: the Secondary keeps its address space warm for failover.
/// </para>
/// </summary>
private void ReassertConditionNodes()
{
var reasserted = 0;
foreach (var projection in _engine.GetProjections())
{
if (IsInNoEventPosition(projection.Condition))
{
continue;
}
_publishActor.Tell(new OpcUaPublishActor.AlarmStateUpdate(
AlarmNodeId: projection.AlarmId,
State: ToSnapshot(projection),
// The instant the state we are re-asserting actually came about — NOT the deploy time.
// A client reading Time/ReceiveTime after a rebuild should still see when the alarm went
// active, not when the address space happened to be rebuilt.
TimestampUtc: projection.Condition.LastTransitionUtc,
Realm: AddressSpaceRealm.Uns));
reasserted++;
}
if (reasserted > 0)
{
_log.Info("ScriptedAlarmHost: re-asserted {Count} non-normal condition node(s) after (re)load", reasserted);
}
}
/// <summary>Whether <paramref name="condition"/> sits in the Part 9 "no-event" position — exactly the
/// state <see cref="AlarmConditionState.Fresh"/> produces and a freshly materialised condition node
/// already carries. Anything else (active, unacknowledged, unconfirmed, disabled, or shelved) is state
/// a rebuild would silently lose, so it is what <see cref="ReassertConditionNodes"/> re-projects.</summary>
/// <param name="condition">The engine's current condition state.</param>
/// <returns><c>true</c> when the condition carries nothing a rebuild could lose.</returns>
private static bool IsInNoEventPosition(AlarmConditionState condition)
=> condition.Enabled == AlarmEnabledState.Enabled
&& condition.Active == AlarmActiveState.Inactive
&& condition.Acked == AlarmAckedState.Acknowledged
&& condition.Confirmed == AlarmConfirmedState.Confirmed
&& condition.Shelving.Kind == ShelvingKind.Unshelved;
private void OnDependencyChanged(VirtualTagActor.DependencyValueChanged msg) private void OnDependencyChanged(VirtualTagActor.DependencyValueChanged msg)
{ {
// Feed the live value into the upstream the engine subscribes from. #478 — carry the source // Feed the live value into the upstream the engine subscribes from. #478 — carry the source
@@ -594,18 +687,38 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
/// Severity is the OPC UA 1..1000 value <see cref="SeverityToInt"/> derives from the coarse engine /// Severity is the OPC UA 1..1000 value <see cref="SeverityToInt"/> derives from the coarse engine
/// bucket, cast to the <c>ushort</c> the SDK <c>SetSeverity</c> expects. Shelving's 3-way Core kind /// bucket, cast to the <c>ushort</c> the SDK <c>SetSeverity</c> expects. Shelving's 3-way Core kind
/// maps 1:1 onto the Commons <see cref="AlarmShelvingKind"/>.</summary> /// maps 1:1 onto the Commons <see cref="AlarmShelvingKind"/>.</summary>
private static AlarmConditionSnapshot ToSnapshot(ScriptedAlarmEvent e) => new( private static AlarmConditionSnapshot ToSnapshot(ScriptedAlarmEvent e)
Active: e.Condition.Active == AlarmActiveState.Active, => ToSnapshot(e.Condition, e.Severity, e.Message, e.WorstInputStatusCode);
Acknowledged: e.Condition.Acked == AlarmAckedState.Acknowledged,
Confirmed: e.Condition.Confirmed == AlarmConfirmedState.Confirmed, /// <summary>#487 — the same projection for a <see cref="ScriptedAlarmProjection"/>: a state READ used
Enabled: e.Condition.Enabled == AlarmEnabledState.Enabled, /// to restore a condition node after a rebuild, never an emission. Shares
Shelving: MapShelving(e.Condition.Shelving.Kind), /// <see cref="ToSnapshot(AlarmConditionState, AlarmSeverity, string, uint)"/> with the transition path
Severity: (ushort)SeverityToInt(e.Severity), /// so a re-asserted node is byte-identical to what the alarm's own transition would have written.</summary>
Message: e.Message, /// <param name="p">The engine projection to map.</param>
/// <returns>The Commons snapshot the SDK sink projects onto the condition node.</returns>
private static AlarmConditionSnapshot ToSnapshot(ScriptedAlarmProjection p)
=> ToSnapshot(p.Condition, p.Severity, p.Message, p.WorstInputStatusCode);
/// <summary>The single Core → Commons condition projection, shared by the emission and re-assert
/// paths.</summary>
/// <param name="condition">The Part 9 condition state.</param>
/// <param name="severity">The engine's coarse severity bucket.</param>
/// <param name="message">The resolved condition message.</param>
/// <param name="worstInputStatusCode">The worst OPC UA StatusCode across the script's input tags.</param>
/// <returns>The Commons snapshot the SDK sink projects onto the condition node.</returns>
private static AlarmConditionSnapshot ToSnapshot(
AlarmConditionState condition, AlarmSeverity severity, string message, uint worstInputStatusCode) => new(
Active: condition.Active == AlarmActiveState.Active,
Acknowledged: condition.Acked == AlarmAckedState.Acknowledged,
Confirmed: condition.Confirmed == AlarmConfirmedState.Confirmed,
Enabled: condition.Enabled == AlarmEnabledState.Enabled,
Shelving: MapShelving(condition.Shelving.Kind),
Severity: (ushort)SeverityToInt(severity),
Message: message,
// #478 — the condition's Quality is the worst quality across the script's input tags at evaluation // #478 — the condition's Quality is the worst quality across the script's input tags at evaluation
// time (carried on the event by the engine). A transition fired while an input is Uncertain projects // time (carried on the event by the engine). A transition fired while an input is Uncertain projects
// Uncertain here so the full-snapshot write doesn't clobber quality back to Good. // Uncertain here so the full-snapshot write doesn't clobber quality back to Good.
Quality: QualityFromStatus(e.WorstInputStatusCode)); Quality: QualityFromStatus(worstInputStatusCode));
/// <summary>Maps the Core <see cref="ShelvingKind"/> onto the Commons <see cref="AlarmShelvingKind"/> /// <summary>Maps the Core <see cref="ShelvingKind"/> onto the Commons <see cref="AlarmShelvingKind"/>
/// mirror (the Commons assembly can't see the Core enum).</summary> /// mirror (the Commons assembly can't see the Core enum).</summary>
@@ -0,0 +1,132 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Configuration.Validation;
namespace ZB.MOM.WW.OtOpcUa.Configuration.Tests;
/// <summary>
/// <see cref="SqlCredentialGuard"/> — the shared "a Sql driver never persists a literal connection
/// string" check behind both the #498 deploy gate and the #499 node-override save gate.
/// </summary>
/// <remarks>
/// The node-override half is the one #499 is about: <c>ClusterNode.DriverConfigOverridesJson</c> is a
/// map keyed by <c>DriverInstanceId</c> that is merged onto the cluster-level <c>DriverConfig</c>, and
/// it is invisible to <c>DraftSnapshot</c> — so nothing gated it at all before.
/// </remarks>
[Trait("Category", "Unit")]
public sealed class SqlCredentialGuardTests
{
/// <summary>The literal an operator would paste into a raw-JSON textarea.</summary>
private const string Leaked =
"""{"provider":"SqlServer","connectionString":"Server=sql,1433;Database=Mes;User ID=sa;Password=hunter2"}""";
private const string Clean = """{"provider":"SqlServer","connectionStringRef":"Sql:Line3"}""";
// ---- CarriesLiteralConnectionString --------------------------------------------------------
/// <summary>The key is caught at the top level of a config blob.</summary>
[Fact]
public void A_literal_connectionString_is_detected()
=> SqlCredentialGuard.CarriesLiteralConnectionString(Leaked).ShouldBeTrue();
/// <summary>The supported indirect form is not a violation — otherwise the rule would block the fix
/// it tells operators to apply.</summary>
[Fact]
public void The_indirect_connectionStringRef_form_is_clean()
=> SqlCredentialGuard.CarriesLiteralConnectionString(Clean).ShouldBeFalse();
/// <summary>System.Text.Json binds <c>ConnectionString</c> to a <c>connectionString</c> property by
/// default, so a case variant is the same key — not a bypass.</summary>
[Theory]
[InlineData("""{"ConnectionString":"Server=x;Password=p"}""")]
[InlineData("""{"CONNECTIONSTRING":"Server=x;Password=p"}""")]
[InlineData("""{"connectionstring":"Server=x;Password=p"}""")]
public void Case_variants_are_the_same_key(string json)
=> SqlCredentialGuard.CarriesLiteralConnectionString(json).ShouldBeTrue();
/// <summary>Blank, malformed and non-object blobs simply have no keys. Shaping the config JSON is
/// another rule's job, and throwing here would turn a formatting mistake into a credential failure.
/// </summary>
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
[InlineData("{ not json")]
[InlineData("[1,2,3]")]
[InlineData("\"a string\"")]
public void Blank_malformed_and_non_object_blobs_are_not_violations(string? json)
=> SqlCredentialGuard.CarriesLiteralConnectionString(json).ShouldBeFalse();
/// <summary>Only the top level is scanned. The DTO is flat, so a nested occurrence cannot bind and is
/// not the credential-shaped mistake this rule exists to catch.</summary>
[Fact]
public void A_nested_occurrence_is_not_flagged()
=> SqlCredentialGuard
.CarriesLiteralConnectionString("""{"nested":{"connectionString":"Server=x"}}""")
.ShouldBeFalse();
// ---- FindNodeOverrideViolations (#499) -----------------------------------------------------
/// <summary>The #499 leak: a credential pasted into a node's per-driver override map.</summary>
[Fact]
public void A_credential_in_a_node_override_for_a_Sql_driver_is_a_violation()
{
var overrides = $$"""{"di-sql": {{Leaked}} }""";
SqlCredentialGuard.FindNodeOverrideViolations(overrides, ["di-sql"])
.ShouldBe(["di-sql"]);
}
/// <summary>Every offending instance is reported, not just the first — an operator fixing one at a
/// time would otherwise need as many save attempts as there are leaks.</summary>
[Fact]
public void Every_offending_instance_is_reported()
{
var overrides = $$"""{"di-a": {{Leaked}}, "di-clean": {{Clean}}, "di-b": {{Leaked}} }""";
SqlCredentialGuard.FindNodeOverrideViolations(overrides, ["di-a", "di-b", "di-clean"])
.ShouldBe(["di-a", "di-b"]);
}
/// <summary>Keys outside the Sql set are ignored. A non-Sql driver never made the
/// indirect-credential guarantee, so flagging it would break a config that is legitimate today —
/// a regression, not defence in depth.</summary>
[Fact]
public void An_override_for_a_non_Sql_driver_is_ignored()
{
var overrides = $$"""{"di-modbus": {{Leaked}} }""";
SqlCredentialGuard.FindNodeOverrideViolations(overrides, ["di-sql"]).ShouldBeEmpty();
}
/// <summary>The realistic override — a node-specific endpoint — must keep saving.</summary>
[Fact]
public void A_normal_override_is_clean()
{
var overrides = """{"di-sql": {"commandTimeoutSeconds": 45}}""";
SqlCredentialGuard.FindNodeOverrideViolations(overrides, ["di-sql"]).ShouldBeEmpty();
}
/// <summary>Nothing to check when the cluster has no Sql drivers, or the node has no overrides.</summary>
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
[InlineData("{ not json")]
[InlineData("[1,2,3]")]
public void Blank_and_malformed_override_maps_yield_no_violations(string? overrides)
=> SqlCredentialGuard.FindNodeOverrideViolations(overrides, ["di-sql"]).ShouldBeEmpty();
/// <summary>An empty Sql-driver set short-circuits — there is no instance the key could belong to.</summary>
[Fact]
public void No_Sql_drivers_means_no_violations()
=> SqlCredentialGuard.FindNodeOverrideViolations($$"""{"di-sql": {{Leaked}} }""", [])
.ShouldBeEmpty();
/// <summary>A non-object override entry cannot carry config keys, and must not throw.</summary>
[Fact]
public void A_non_object_override_entry_is_skipped()
=> SqlCredentialGuard.FindNodeOverrideViolations("""{"di-sql": "connectionString"}""", ["di-sql"])
.ShouldBeEmpty();
}
@@ -0,0 +1,180 @@
using Serilog;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
namespace ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms.Tests;
/// <summary>
/// #487 — <see cref="ScriptedAlarmEngine.GetProjections"/>: the read that lets a caller restore an
/// alarm's condition node after a full address-space rebuild, without waiting for the alarm's next
/// transition (which for a still-active alarm with static dependencies may never come).
/// </summary>
/// <remarks>
/// The load that motivates this is deliberately the one that emits NOTHING: an alarm whose persisted
/// state is already Active and whose predicate is still true produces <see cref="EmissionKind.None"/>,
/// so the transition stream cannot carry its state. These tests pin both halves — the projection does
/// report the state, and reading it does not fire an emission.
/// </remarks>
[Trait("Category", "Unit")]
public sealed class ScriptedAlarmProjectionTests
{
private static ScriptedAlarmDefinition Alarm(
string id = "HighTemp",
string predicate = """return (int)ctx.GetTag("Temp").Value > 100;""",
string msg = "Temp is {Temp}",
AlarmSeverity sev = AlarmSeverity.High) =>
new(AlarmId: id,
EquipmentPath: "Plant/Line1/Reactor",
AlarmName: id,
Kind: AlarmKind.AlarmCondition,
Severity: sev,
MessageTemplate: msg,
PredicateScriptSource: predicate);
/// <summary>An already-Active persisted state, as a store would hold it across a restart/redeploy.</summary>
private static AlarmConditionState PersistedActive(
string alarmId = "HighTemp",
AlarmAckedState acked = AlarmAckedState.Unacknowledged,
DateTime? lastTransitionUtc = null) =>
new(alarmId,
AlarmEnabledState.Enabled,
AlarmActiveState.Active,
acked,
AlarmConfirmedState.Confirmed,
ShelvingState.Unshelved,
lastTransitionUtc ?? DateTime.UtcNow,
LastActiveUtc: lastTransitionUtc ?? DateTime.UtcNow,
LastClearedUtc: null,
LastAckUtc: null, LastAckUser: null, LastAckComment: null,
LastConfirmUtc: null, LastConfirmUser: null, LastConfirmComment: null,
Comments: []);
private static ScriptedAlarmEngine Build(FakeUpstream up, IAlarmStateStore store)
{
var logger = new LoggerConfiguration().CreateLogger();
return new ScriptedAlarmEngine(up, store, new ScriptLoggerFactory(logger), logger);
}
/// <summary>The core of #487: a reload of a still-active alarm emits NOTHING, yet the projection
/// reports it Active — so a caller has a way to restore the node the rebuild reset.</summary>
[Fact]
public async Task A_still_active_alarm_emits_nothing_on_reload_but_projects_Active()
{
var up = new FakeUpstream();
up.Set("Temp", 150); // predicate still true — no transition to report
var store = new InMemoryAlarmStateStore();
await store.SaveAsync(PersistedActive(), TestContext.Current.CancellationToken);
using var eng = Build(up, store);
var emissions = new List<ScriptedAlarmEvent>();
eng.OnEvent += (_, e) => { lock (emissions) emissions.Add(e); }; // subscribe BEFORE the load
await eng.LoadAsync([Alarm()], TestContext.Current.CancellationToken);
// The premise. If this ever starts emitting, the bug's shape has changed and the re-assert
// in ScriptedAlarmHostActor may have become a duplicate rather than the only source of truth.
lock (emissions)
{
emissions.ShouldBeEmpty("a still-active alarm has no transition to report on reload");
}
var projection = eng.GetProjections().ShouldHaveSingleItem();
projection.AlarmId.ShouldBe("HighTemp");
projection.Condition.Active.ShouldBe(AlarmActiveState.Active);
projection.Condition.Acked.ShouldBe(AlarmAckedState.Unacknowledged);
}
/// <summary>The projection carries the definition's severity and the message template resolved against
/// the live value cache — so a re-asserted node is indistinguishable from one a real transition wrote.</summary>
[Fact]
public async Task Projection_carries_definition_severity_and_a_resolved_message()
{
var up = new FakeUpstream();
up.Set("Temp", 150);
var store = new InMemoryAlarmStateStore();
await store.SaveAsync(PersistedActive(), TestContext.Current.CancellationToken);
using var eng = Build(up, store);
await eng.LoadAsync([Alarm(sev: AlarmSeverity.Critical)], TestContext.Current.CancellationToken);
var projection = eng.GetProjections().ShouldHaveSingleItem();
projection.Severity.ShouldBe(AlarmSeverity.Critical);
projection.Message.ShouldBe("Temp is 150", "the template resolves against the current value cache");
}
/// <summary>Reading projections is a pure read — it must never raise <see cref="ScriptedAlarmEngine.OnEvent"/>,
/// or every deploy would append a duplicate historian / alerts row.</summary>
[Fact]
public async Task Reading_projections_never_fires_an_emission()
{
var up = new FakeUpstream();
up.Set("Temp", 150);
var store = new InMemoryAlarmStateStore();
await store.SaveAsync(PersistedActive(), TestContext.Current.CancellationToken);
using var eng = Build(up, store);
await eng.LoadAsync([Alarm()], TestContext.Current.CancellationToken);
var emissions = 0;
eng.OnEvent += (_, _) => Interlocked.Increment(ref emissions);
for (var i = 0; i < 5; i++) eng.GetProjections().ShouldNotBeEmpty();
Volatile.Read(ref emissions).ShouldBe(0);
}
/// <summary>An alarm that is genuinely normal projects the no-event position, so a caller can tell
/// "nothing to restore" from "restore this".</summary>
[Fact]
public async Task A_normal_alarm_projects_the_no_event_position()
{
var up = new FakeUpstream();
up.Set("Temp", 50); // predicate false
using var eng = Build(up, new InMemoryAlarmStateStore());
await eng.LoadAsync([Alarm()], TestContext.Current.CancellationToken);
var projection = eng.GetProjections().ShouldHaveSingleItem();
projection.Condition.Active.ShouldBe(AlarmActiveState.Inactive);
projection.Condition.Acked.ShouldBe(AlarmAckedState.Acknowledged);
projection.Condition.Enabled.ShouldBe(AlarmEnabledState.Enabled);
projection.Condition.Shelving.Kind.ShouldBe(ShelvingKind.Unshelved);
}
/// <summary>The worst-input quality rides the projection, so restoring a node whose script inputs are
/// Bad does not clobber its Quality back to Good (#478's invariant, on the re-assert path).</summary>
[Fact]
public async Task Projection_carries_the_last_observed_worst_input_quality()
{
var up = new FakeUpstream();
up.Set("Temp", 150);
var store = new InMemoryAlarmStateStore();
await store.SaveAsync(PersistedActive(), TestContext.Current.CancellationToken);
using var eng = Build(up, store);
await eng.LoadAsync([Alarm()], TestContext.Current.CancellationToken);
eng.GetProjections().ShouldHaveSingleItem().WorstInputStatusCode.ShouldBe(0u, "inputs start Good");
up.Push("Temp", 150, 0x80000000u); // input goes Bad — condition freezes, quality is annotated
// The push re-evaluates on a background worker; poll for the tracked bucket to move.
var deadline = DateTime.UtcNow.AddSeconds(10);
while (eng.GetProjections()[0].WorstInputStatusCode == 0u && DateTime.UtcNow < deadline)
{
await Task.Delay(20, TestContext.Current.CancellationToken);
}
(eng.GetProjections()[0].WorstInputStatusCode >> 30).ShouldBe(2u, "a Bad input projects Bad quality");
}
/// <summary>Before the first load there is nothing to project — the read is safe, not a throw.</summary>
[Fact]
public void Projections_are_empty_before_the_first_load()
{
using var eng = Build(new FakeUpstream(), new InMemoryAlarmStateStore());
eng.GetProjections().ShouldBeEmpty();
}
}
@@ -10,59 +10,109 @@ using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests.Runtime; namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests.Runtime;
/// <summary> /// <summary>
/// PR 6.2 — pins the EventPump's bounded-channel + drop-newest behavior. We /// PR 6.2 — pins the EventPump's bounded-channel + drop-newest behavior. We hold the dispatch loop
/// hold the dispatch loop with a slow handler so the channel fills, then verify /// with a blocking handler so the channel fills, then verify the producer keeps reading from the gw
/// the producer keeps reading from the gw stream and increments the /// stream and increments the <c>galaxy.events.dropped</c> counter rather than blocking.
/// <c>galaxy.events.dropped</c> counter rather than blocking.
/// </summary> /// </summary>
/// <remarks>
/// These tests observe two background loops (<c>RunAsync</c> / <c>DispatchLoopAsync</c>) through a
/// process-wide <see cref="Meter"/>, so every wait here is a <b>presence</b> assertion polled up to
/// <see cref="Budget"/> — never a fixed sleep. A fixed sleep is a bet that both loops get scheduled
/// within it, and #503 is what losing that bet looks like when the box is busy running other test
/// assemblies.
/// </remarks>
public sealed class EventPumpBoundedChannelTests public sealed class EventPumpBoundedChannelTests
{ {
/// <summary>Verifies that the event pump drops newest events when the bounded channel fills and records metrics for dropped events.</summary> /// <summary>Upper bound on how long a poll waits for a background loop to reach its terminal state.
/// Generous on purpose: it is spent only when something is genuinely broken, and it cannot turn a
/// failing assertion into a passing one.</summary>
private static readonly TimeSpan Budget = TimeSpan.FromSeconds(15);
/// <summary>
/// With the dispatch loop genuinely held and the channel therefore full, the producer keeps
/// draining the gw stream and counts every rejected write on <c>galaxy.events.dropped</c> —
/// newest-dropped, not back-pressure.
/// </summary>
/// <remarks>
/// <para>
/// The arrangement is staged so the arithmetic is exact rather than scheduler-dependent
/// (#503): emit ONE event, wait until the dispatcher has actually entered the handler (so it
/// holds that event and the channel is empty), and only then emit the remaining nine. The
/// channel accepts <c>capacity</c> of them and the rest must be dropped —
/// <c>10 1 held 2 buffered = 7</c>, every run.
/// </para>
/// </remarks>
[Fact] [Fact]
public async Task Drops_newest_when_channel_fills_and_records_metric() public async Task Drops_newest_when_channel_fills_and_records_metric()
{ {
var counters = StartMeterCapture(); const int totalEvents = 10;
const int channelCapacity = 2;
// One event is held inside the handler; `channelCapacity` more fit in the channel; the rest are dropped.
const int expectedDropped = totalEvents - 1 - channelCapacity;
// Unique per run: the counters are filtered to this exact galaxy.client tag, which is what keeps a
// parallel sibling test's pump out of them (#503). A literal name would work today and break the
// day someone reuses it.
var clientName = $"PumpTest-{Guid.NewGuid():N}";
var counters = StartMeterCapture(clientName);
// A SYNCHRONOUS gate, deliberately. OnDataChange is EventHandler<T> — void-returning — so an
// `async (_, _) => await gate` lambda is async void: it returns to the dispatch loop at the first
// await and holds nothing. That is exactly what #503 was: the loop drained freely, drops happened
// only when the producer happened to outrun the consumer, and the assertions rode on scheduling
// luck. Blocking the loop's own thread is what makes "the dispatcher is held" true.
var dispatchGate = new ManualResetEventSlim(false);
var handlerEntered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
EventPump? pump = null;
try try
{ {
var subscriber = new ManualSubscriber(); var subscriber = new ManualSubscriber();
var registry = new SubscriptionRegistry(); var registry = new SubscriptionRegistry();
registry.Register(1, [new TagBinding("Tag.A", ItemHandle: 7)]); registry.Register(1, [new TagBinding("Tag.A", ItemHandle: 7)]);
// Tiny channel + slow handler ⇒ producer hits FullMode.Wait → TryWrite false pump = new EventPump(
// for every overflow event. subscriber, registry, channelCapacity: channelCapacity, clientName: clientName);
var dispatchGate = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); pump.OnDataChange += (_, _) =>
await using var pump = new EventPump(
subscriber, registry, channelCapacity: 2, clientName: "PumpTest");
pump.OnDataChange += async (_, _) =>
{ {
// Block the dispatch loop until we've shoved enough events through to handlerEntered.TrySetResult();
// overflow the bounded channel. Consume the gate exactly once. dispatchGate.Wait();
await dispatchGate.Task.ConfigureAwait(false);
}; };
pump.Start(); pump.Start();
const int totalEvents = 10; // Stage 1 — one event, then wait for the dispatcher to take it and block. After this the
for (var i = 0; i < totalEvents; i++) // channel is provably empty and exactly one event is in flight inside the handler.
await subscriber.EmitAsync(itemHandle: 7, value: 0);
await handlerEntered.Task.WaitAsync(Budget);
// Stage 2 — the rest. `channelCapacity` are accepted; the remainder must be dropped.
for (var i = 1; i < totalEvents; i++)
{ {
await subscriber.EmitAsync(itemHandle: 7, value: i); await subscriber.EmitAsync(itemHandle: 7, value: i);
} }
// Give the producer a beat to run TryWrite for every event.
await Task.Delay(150);
// Capacity 2 + 1 in-flight in the dispatcher = 3 may have been accepted; the // PRESENCE assertion on the discriminator (#500's rule): poll until the drop count reaches its
// remainder should have hit the dropped counter. Don't pin exact counts — // terminal value rather than sleeping a fixed 150 ms and hoping the producer got scheduled.
// the scheduler can interleave; pin the invariants instead. // The budget is an upper bound before giving up — spending it costs nothing on the happy path,
counters.Received.ShouldBeGreaterThanOrEqualTo(totalEvents); // and no amount of it can make a genuinely broken pump pass. Dropped starts at 0, so this
counters.Dropped.ShouldBeGreaterThan(0, // cannot be satisfied by the initial state.
"with capacity=2 and a held dispatcher we must drop at least one of 10 events"); await WaitUntilAsync(() => counters.Dropped == expectedDropped,
(counters.Received).ShouldBe(counters.Dispatched + counters.Dropped + counters.InFlight, $"expected {expectedDropped} dropped events");
"received = dispatched + dropped + (events still queued)");
// Release the dispatcher so DisposeAsync can drain cleanly. counters.Received.ShouldBe(totalEvents,
dispatchGate.TrySetResult(); "the producer must keep reading the gw stream through the overflow, not back-pressure");
counters.Dropped.ShouldBe(expectedDropped);
counters.Dispatched.ShouldBe(0,
"the dispatch loop is still blocked in the handler, so nothing has completed dispatch — " +
"if this is non-zero the handler is not actually holding the loop and the test proves nothing");
} }
finally finally
{ {
// Release BEFORE disposing: DisposeAsync awaits the dispatch loop, and that loop is parked on
// this gate. Releasing it here (not at the end of the try) is what keeps a failed assertion a
// failure instead of a hang.
dispatchGate.Set();
if (pump is not null) await pump.DisposeAsync();
dispatchGate.Dispose();
counters.Dispose(); counters.Dispose();
} }
} }
@@ -110,23 +160,21 @@ public sealed class EventPumpBoundedChannelTests
// Poll until at least one galaxy.events.received measurement tagged // Poll until at least one galaxy.events.received measurement tagged
// galaxy.client=Driver-X lands in the listener, rather than using a // galaxy.client=Driver-X lands in the listener, rather than using a
// fixed delay that races under parallel test load on a busy box. // fixed delay that races under parallel test load on a busy box.
var deadline = DateTime.UtcNow.AddSeconds(5); // Budget, not 5 s: the same contention that produced #503 stretches how long the
bool found = false; // producer loop takes to be scheduled, and a presence poll costs nothing to over-budget.
while (DateTime.UtcNow < deadline) await WaitUntilAsync(
{ () =>
listener.RecordObservableInstruments();
bool hasMatch;
lock (captured)
{ {
hasMatch = captured.Any(c => listener.RecordObservableInstruments();
c.Instrument == "galaxy.events.received" && lock (captured)
c.Tags.Any(t => t.Key == "galaxy.client" && {
string.Equals((string?)t.Value, "Driver-X", StringComparison.Ordinal))); return captured.Any(c =>
} c.Instrument == "galaxy.events.received" &&
if (hasMatch) { found = true; break; } c.Tags.Any(t => t.Key == "galaxy.client" &&
await Task.Delay(25); string.Equals((string?)t.Value, "Driver-X", StringComparison.Ordinal)));
} }
_ = found; // assertion happens below after dispose },
"a galaxy.events.received measurement tagged galaxy.client=Driver-X");
} }
// The static Meter is shared across all EventPump instances in the test // The static Meter is shared across all EventPump instances in the test
@@ -146,7 +194,46 @@ public sealed class EventPumpBoundedChannelTests
ours.ShouldContain(c => c.Instrument == "galaxy.events.received"); ours.ShouldContain(c => c.Instrument == "galaxy.events.received");
} }
private static CounterCapture StartMeterCapture() /// <summary>Polls <paramref name="condition"/> until it holds or <see cref="Budget"/> elapses, failing
/// with <paramref name="what"/> on timeout. The presence-assertion counterpart to a fixed delay.</summary>
private static async Task WaitUntilAsync(Func<bool> condition, string what)
{
var deadline = DateTime.UtcNow + Budget;
while (DateTime.UtcNow < deadline)
{
if (condition()) return;
await Task.Delay(10);
}
condition().ShouldBeTrue($"timed out after {Budget.TotalSeconds:0}s waiting: {what}");
}
/// <summary>
/// Captures the three EventPump counters for <b>one specific pump</b>, identified by its
/// <c>galaxy.client</c> tag.
/// </summary>
/// <remarks>
/// <para>
/// The <c>galaxy.client</c> filter is the fix for #503, and it is not optional. The pump's
/// <see cref="Meter"/> is <b>static</b> — one instance for the whole process — and
/// <c>InstrumentPublished</c> can only select by <i>meter</i>, so an unfiltered listener
/// receives measurements from every <see cref="EventPump"/> alive anywhere in the assembly.
/// xUnit runs test classes in parallel and several of them build pumps
/// (<c>EventPumpStreamFaultTests</c>, the <c>Driver-X</c> test below), so a sibling's events
/// landed in these counters — that is why the flake needed a busy box: it needed the two
/// tests to overlap in time. The leak was measured directly, as <c>Received = 11</c> in a run
/// that emitted 10.
/// </para>
/// <para>
/// The old <c>Received == Dispatched + Dropped + InFlight</c> assertion is what turned the
/// leak into a failure: a foreign increment landing between its four separate
/// <see cref="Interlocked.Read(ref long)"/> calls made the two sides disagree. Hence the
/// reported subject, <c>counters.Received</c>.
/// </para>
/// </remarks>
/// <param name="clientName">The <c>galaxy.client</c> tag value to accept. Must be unique to the
/// calling test — pass a fresh GUID-suffixed name rather than a literal, so a future sibling test
/// cannot silently reintroduce the leak by reusing the same name.</param>
private static CounterCapture StartMeterCapture(string clientName)
{ {
var capture = new CounterCapture(); var capture = new CounterCapture();
var listener = new MeterListener(); var listener = new MeterListener();
@@ -154,8 +241,20 @@ public sealed class EventPumpBoundedChannelTests
{ {
if (instr.Meter.Name == EventPump.MeterName) l.EnableMeasurementEvents(instr); if (instr.Meter.Name == EventPump.MeterName) l.EnableMeasurementEvents(instr);
}; };
listener.SetMeasurementEventCallback<long>((instr, value, _, _) => listener.SetMeasurementEventCallback<long>((instr, value, tags, _) =>
{ {
var mine = false;
foreach (var tag in tags)
{
if (tag.Key == "galaxy.client" &&
string.Equals((string?)tag.Value, clientName, StringComparison.Ordinal))
{
mine = true;
break;
}
}
if (!mine) return;
switch (instr.Name) switch (instr.Name)
{ {
case "galaxy.events.received": Interlocked.Add(ref capture._received, value); break; case "galaxy.events.received": Interlocked.Add(ref capture._received, value); break;
@@ -178,8 +277,11 @@ public sealed class EventPumpBoundedChannelTests
public long Dispatched => Interlocked.Read(ref _dispatched); public long Dispatched => Interlocked.Read(ref _dispatched);
/// <summary>Gets the count of dropped events.</summary> /// <summary>Gets the count of dropped events.</summary>
public long Dropped => Interlocked.Read(ref _dropped); public long Dropped => Interlocked.Read(ref _dropped);
/// <summary>Gets the count of in-flight events.</summary> // There is deliberately no InFlight member. It could only be DERIVED as
public long InFlight => Math.Max(0, Received - Dispatched - Dropped); // Received - Dispatched - Dropped, which made the old
// `Received == Dispatched + Dropped + InFlight` assertion a tautology: it could not fail on a
// real defect, only on a torn read across the four separate Interlocked.Read calls. Assert the
// three measured counters directly instead (#503).
/// <summary>Disposes the meter listener.</summary> /// <summary>Disposes the meter listener.</summary>
public void Dispose() => Listener?.Dispose(); public void Dispose() => Listener?.Dispose();
} }
@@ -0,0 +1,95 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.AdminUI.Components.Shared;
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Tests;
/// <summary>
/// #504 — <see cref="DisplayText.Abbreviate"/>, the null/short-safe replacement for the
/// <c>@value[..N]</c> range-operator slices the Admin UI used to abbreviate hashes and thumbprints.
/// </summary>
/// <remarks>
/// The bug being pinned: a range operator over an unvalidated DB string throws
/// <see cref="System.ArgumentOutOfRangeException"/> out of <c>BuildRenderTree</c>, which is unhandled
/// in Blazor and takes the WHOLE page to an HTTP 500 — not just the offending row. Nothing enforces a
/// minimum length at the schema, entity, or render layer, so the short-input cases below are the ones
/// that matter; the happy path is almost incidental.
/// </remarks>
[Trait("Category", "Unit")]
public sealed class DisplayTextTests
{
/// <summary>A value longer than the budget is truncated and marked with an ellipsis — the normal
/// case for a 64-char SHA-256.</summary>
[Fact]
public void A_long_value_is_truncated_and_marked()
{
DisplayText.Abbreviate(new string('a', 64), 12).ShouldBe(new string('a', 12) + "…");
}
/// <summary>The regression: a value SHORTER than the budget must render, not throw. This is exactly
/// what a hand-inserted / API-authored <c>SourceHash</c> like <c>h1</c> hits.</summary>
[Theory]
[InlineData("h1", 12)]
[InlineData("h-abs-hr200", 12)] // 11 chars — one short of the old slice, the original crash
[InlineData("abc", 16)]
[InlineData("x", 64)]
public void A_value_shorter_than_the_budget_renders_verbatim(string value, int maxLength)
{
DisplayText.Abbreviate(value, maxLength).ShouldBe(value);
}
/// <summary>No ellipsis when nothing was dropped — the display must never imply text that is not
/// there. (The old markup appended "…" unconditionally, outside the slice.)</summary>
[Fact]
public void A_short_value_gets_no_ellipsis()
{
DisplayText.Abbreviate("h1", 12).ShouldNotContain("…");
}
/// <summary>A value exactly at the budget is the boundary — it fits, so it is neither truncated nor
/// marked.</summary>
[Fact]
public void A_value_exactly_at_the_budget_is_untouched()
{
DisplayText.Abbreviate("123456789012", 12).ShouldBe("123456789012");
}
/// <summary>Null / blank render as the Admin UI's em-dash placeholder rather than throwing or
/// producing a stray ellipsis.</summary>
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
public void A_null_or_blank_value_renders_the_placeholder(string? value)
{
DisplayText.Abbreviate(value, 12).ShouldBe(DisplayText.Empty);
}
/// <summary>A nonsensical budget is clamped rather than allowed to throw — a display bug must not
/// become a page crash, which is the whole point of this helper.</summary>
[Theory]
[InlineData(0)]
[InlineData(-1)]
[InlineData(int.MinValue)]
public void A_non_positive_budget_is_clamped_not_thrown(int maxLength)
{
DisplayText.Abbreviate("abcdef", maxLength).ShouldBe("a…");
}
/// <summary>Whatever the inputs, it never throws — the property that keeps a bad row from killing a
/// page. Includes the shapes that broke the old code.</summary>
[Fact]
public void It_never_throws_for_any_combination()
{
string?[] values = [null, "", " ", "h1", "h-abs-hr200", new string('f', 64), "…"];
int[] budgets = [int.MinValue, -1, 0, 1, 12, 16, 64, int.MaxValue];
foreach (var value in values)
{
foreach (var budget in budgets)
{
Should.NotThrow(() => DisplayText.Abbreviate(value, budget));
}
}
}
}
@@ -86,8 +86,19 @@ public sealed class DriverHostActorNativeAlarmAckRoutingTests : RuntimeActorTest
actor.Tell(new DriverHostActor.RouteNativeAlarmAck("Plant/gw/dev1/does-not-exist", "cmt", "alice")); actor.Tell(new DriverHostActor.RouteNativeAlarmAck("Plant/gw/dev1/does-not-exist", "cmt", "alice"));
// Give the (fire-and-forget) handler time to run; the unmapped node must produce no ack. // POSITIVE CONTROL, not a sleep (#501). `Acks` is empty at t=0, so any assertion made before the
AwaitAssert(() => recorder.Acks.ShouldBeEmpty(), duration: TimeSpan.FromMilliseconds(800)); // host has processed the message above passes without testing anything — and AwaitAssert returns on
// its FIRST success, so it spends none of its duration and is the wrong tool for an absence.
//
// Instead send a MAPPED ack afterwards. The host processes its mailbox in order and forwards to the
// child via Tell, and the child handles RouteAlarmAck with ReceiveAsync (which suspends its own
// mailbox), so by the time this one reaches the driver the unmapped one has provably already been
// handled. Whatever `Acks` holds then is the complete answer.
actor.Tell(new DriverHostActor.RouteNativeAlarmAck(AlarmRawPath, "control", "alice"));
AwaitAssert(() => recorder.Acks.ShouldContain(a => a.Comment == "control"), duration: Timeout);
recorder.Acks.Count.ShouldBe(1,
"only the control ack may reach the driver — the unmapped condition NodeId must have been dropped");
} }
/// <summary>On a SECONDARY node the ack is gated off (same primary gate as the inbound write path): the /// <summary>On a SECONDARY node the ack is gated off (same primary gate as the inbound write path): the
@@ -110,10 +121,25 @@ public sealed class DriverHostActorNativeAlarmAckRoutingTests : RuntimeActorTest
}, },
CorrelationId.NewId())); CorrelationId.NewId()));
actor.Tell(new DriverHostActor.RouteNativeAlarmAck(AlarmRawPath, "cmt", "alice")); actor.Tell(new DriverHostActor.RouteNativeAlarmAck(AlarmRawPath, "gated", "alice"));
// No ack reached the driver — the gate short-circuited before the inverse-map lookup. // POSITIVE CONTROL (#501) — same reasoning as the unmapped-node test, with the role itself as the
AwaitAssert(() => recorder.Acks.ShouldBeEmpty(), duration: TimeSpan.FromMilliseconds(800)); // control: return the node to PRIMARY and re-send the SAME mapped ack. Mailbox ordering means that
// once this one reaches the driver, the Secondary-gated one has already been processed. The
// discriminator is the comment, so a leaked ack is visible rather than merged into the count.
actor.Tell(new RedundancyStateChanged(
new[]
{
new NodeRedundancyState(TestNode, RedundancyRole.Primary,
IsClusterLeader: true, IsDriverPrimary: true, AsOfUtc: DateTime.UtcNow),
},
CorrelationId.NewId()));
actor.Tell(new DriverHostActor.RouteNativeAlarmAck(AlarmRawPath, "control", "alice"));
AwaitAssert(() => recorder.Acks.ShouldContain(a => a.Comment == "control"), duration: Timeout);
recorder.Acks.ShouldNotContain(a => a.Comment == "gated",
"the Primary gate must have dropped the ack sent while this node was Secondary");
recorder.Acks.Count.ShouldBe(1);
} }
/// <summary>Spawns the host with the recording alarm-driver factory, dispatches the deployment, and waits /// <summary>Spawns the host with the recording alarm-driver factory, dispatches the deployment, and waits
@@ -0,0 +1,122 @@
using Akka.Actor;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers;
/// <summary>
/// Gitea #488 — the periodic desired-vs-actual subscription reconcile that rides
/// <see cref="DriverInstanceActor"/>'s existing <c>health-poll</c> tick.
/// </summary>
/// <remarks>
/// <para>
/// The hole it closes: the desired ref set is re-applied on a <b>Connected transition</b> and on a
/// deploy's <c>SetDesiredSubscriptions</c>, and nowhere else. A subscription lost while the driver
/// <i>stayed</i> Connected — the shape reproduced here, a subscribe that failed and was never
/// retried — left the actor permanently subscribed to nothing while reporting Healthy.
/// </para>
/// <para>
/// All three tests drive a short <c>healthPollInterval</c> rather than the production 30 s. The
/// two absence assertions are bounded by a positive control (a counted number of ticks provably
/// processed), never by a bare sleep.
/// </para>
/// </remarks>
public sealed class DriverInstanceActorSubscriptionReconcileTests : RuntimeActorTestBase
{
private static readonly TimeSpan FastPoll = TimeSpan.FromMilliseconds(100);
/// <summary>
/// The reconcile itself: the first subscribe fails, leaving the actor Connected with a desired set
/// and no handle. Nothing else would ever retry it — no reconnect, no deploy — so a second
/// subscribe can only come from the periodic reconcile.
/// </summary>
[Fact]
public void A_lost_subscription_is_re_established_while_still_connected()
{
var driver = new SubscribableStubDriver { SubscribeFailuresBeforeSuccess = 1 };
var actor = Sys.ActorOf(DriverInstanceActor.Props(driver, healthPollInterval: FastPoll));
actor.Tell(new DriverInstanceActor.SetDesiredSubscriptions(
new[] { "tag-a" }, TimeSpan.FromMilliseconds(100)));
actor.Tell(new DriverInstanceActor.InitializeRequested("{}"));
// The connect-path subscribe runs and throws, so the handle stays null.
AwaitCondition(() => driver.SubscribeCount >= 1, PresenceBudget);
// Only the reconcile can produce this second attempt — and it must succeed this time.
AwaitCondition(() => driver.SubscribeCount >= 2, PresenceBudget);
driver.LastSubscribedRefs.ShouldBe(new[] { "tag-a" });
}
/// <summary>
/// A healthy subscription is left alone — the reconcile must not re-subscribe on every tick, which
/// would churn the backend every 30 s in production for no reason.
/// </summary>
[Fact]
public void A_healthy_subscription_is_not_re_subscribed_on_every_tick()
{
var publisher = new CountingHealthPublisher();
var driver = new SubscribableStubDriver();
var actor = Sys.ActorOf(DriverInstanceActor.Props(
driver, healthPublisher: publisher, healthPollInterval: FastPoll));
actor.Tell(new DriverInstanceActor.SetDesiredSubscriptions(
new[] { "tag-a" }, TimeSpan.FromMilliseconds(100)));
actor.Tell(new DriverInstanceActor.InitializeRequested("{}"));
AwaitCondition(() => driver.SubscribeCount >= 1, PresenceBudget);
// POSITIVE CONTROL for the absence below. Every health-poll tick publishes a snapshot, so the
// publisher count is a direct observation of ticks PROCESSED. Waiting for several of them is what
// makes "SubscribeCount is still 1" mean "the reconcile ran and declined" rather than "the timer
// had not fired yet" — the latter would pass instantly and prove nothing.
var before = publisher.Count;
AwaitCondition(() => publisher.Count >= before + 4, PresenceBudget);
driver.SubscribeCount.ShouldBe(1,
"the handle is live, so the reconcile must leave it alone");
}
/// <summary>
/// A driver that is not <see cref="ISubscribable"/> can still be handed a desired set. Reconciling
/// it would tell <c>Subscribe</c> every tick and fail every tick, forever — so the reconcile is
/// gated on the capability.
/// </summary>
[Fact]
public void A_non_subscribable_driver_is_never_reconciled()
{
var publisher = new CountingHealthPublisher();
var driver = new StubDriver(); // IDriver only — no ISubscribable
var actor = Sys.ActorOf(DriverInstanceActor.Props(
driver, healthPublisher: publisher, healthPollInterval: FastPoll));
actor.Tell(new DriverInstanceActor.SetDesiredSubscriptions(
new[] { "tag-a" }, TimeSpan.FromMilliseconds(100)));
actor.Tell(new DriverInstanceActor.InitializeRequested("{}"));
// Same positive control: assert the reconcile stays silent across several PROCESSED ticks.
// EventFilter bounds the absence to this block, and the reconcile announces itself at Info —
// the first test above is the proof that this message does appear when the reconcile fires.
EventFilter.Info(contains: "periodic reconcile").Expect(0, () =>
{
var before = publisher.Count;
AwaitCondition(() => publisher.Count >= before + 4, PresenceBudget);
});
}
/// <summary>Counts health-snapshot publishes, which is one per <c>health-poll</c> tick once the actor
/// is running — the observable that turns "time passed" into "ticks were processed".</summary>
private sealed class CountingHealthPublisher : IDriverHealthPublisher
{
private int _count;
/// <summary>Number of snapshots published so far.</summary>
public int Count => Volatile.Read(ref _count);
/// <inheritdoc />
public void Publish(string clusterId, string driverInstanceId, DriverHealth health, int errorCount5Min)
=> Interlocked.Increment(ref _count);
}
}
@@ -114,6 +114,18 @@ internal sealed class SubscribableStubDriver : StubDriver, ISubscribable
/// synchronously-completed stub task hides (the continuation otherwise runs inline).</summary> /// synchronously-completed stub task hides (the continuation otherwise runs inline).</summary>
public bool UnsubscribeYields { get; set; } public bool UnsubscribeYields { get; set; }
/// <summary>
/// How many of the first <see cref="SubscribeAsync"/> calls throw before one succeeds. Default 0
/// (always succeed) leaves existing behaviour untouched.
/// </summary>
/// <remarks>
/// This is how a test reaches the state the #488 reconcile exists for: the actor is Connected and
/// holds a desired ref set, but the subscribe that should have established the handle failed, so
/// <c>_subscriptionHandle</c> is null and nothing else will ever re-assert it — there is no
/// connect transition coming, and no deploy.
/// </remarks>
public int SubscribeFailuresBeforeSuccess { get; set; }
/// <summary>Subscribes to the specified full references.</summary> /// <summary>Subscribes to the specified full references.</summary>
/// <param name="fullReferences">The full references to subscribe to.</param> /// <param name="fullReferences">The full references to subscribe to.</param>
/// <param name="publishingInterval">The publishing interval.</param> /// <param name="publishingInterval">The publishing interval.</param>
@@ -121,8 +133,13 @@ internal sealed class SubscribableStubDriver : StubDriver, ISubscribable
public Task<ISubscriptionHandle> SubscribeAsync( public Task<ISubscriptionHandle> SubscribeAsync(
IReadOnlyList<string> fullReferences, TimeSpan publishingInterval, CancellationToken cancellationToken) IReadOnlyList<string> fullReferences, TimeSpan publishingInterval, CancellationToken cancellationToken)
{ {
Interlocked.Increment(ref SubscribeCount); var attempt = Interlocked.Increment(ref SubscribeCount);
LastSubscribedRefs = fullReferences; LastSubscribedRefs = fullReferences;
if (attempt <= SubscribeFailuresBeforeSuccess)
{
throw new InvalidOperationException($"stub subscribe failure #{attempt}");
}
return Task.FromResult<ISubscriptionHandle>(_handle); return Task.FromResult<ISubscriptionHandle>(_handle);
} }
@@ -78,7 +78,9 @@ public sealed class PeerProbeSupervisorTests : RuntimeActorTestBase
duration: PresenceBudget); duration: PresenceBudget);
} }
/// <summary>Verifies a single-node snapshot (just the local node) spawns no children.</summary> /// <summary>Verifies a single-node snapshot (just the local node) spawns no children — the local node
/// is never its own probe peer. Established against a following two-node snapshot as a positive
/// control, so the count of 1 (rather than 2) is what carries the "no local child" claim.</summary>
[Fact] [Fact]
public void Single_node_snapshot_spawns_no_children() public void Single_node_snapshot_spawns_no_children()
{ {
@@ -87,7 +89,20 @@ public sealed class PeerProbeSupervisorTests : RuntimeActorTestBase
sup.Tell(Snapshot(State(Local, RedundancyRole.Primary))); sup.Tell(Snapshot(State(Local, RedundancyRole.Primary)));
AwaitAssert(() => sup.UnderlyingActor.ChildCount.ShouldBe(0), // POSITIVE CONTROL (#501). This is an ABSENCE assertion, and ChildCount is 0 before the snapshot
// above has been processed at all — so asserting it directly passes at t=0 and proves nothing
// (AwaitAssert returns on its first success, spending none of its duration). Unlike the other
// ChildCount.ShouldBe(0) waits in this class, there is no preceding ShouldBe(1) to make it a
// transition.
//
// Follow with a snapshot that DOES spawn exactly one child. The supervisor processes its mailbox
// in order, so once the count reaches 1 the single-node snapshot has provably been handled — and
// had it spawned a child for the local node, the count would be 2.
sup.Tell(Snapshot(
State(Local, RedundancyRole.Primary),
State(Peer, RedundancyRole.Secondary)));
AwaitAssert(() => sup.UnderlyingActor.ChildCount.ShouldBe(1),
duration: PresenceBudget); duration: PresenceBudget);
} }
@@ -86,20 +86,22 @@ public sealed class ScriptedAlarmHostActorTests : RuntimeActorTestBase
Retain: false, Retain: false,
Enabled: true); Enabled: true);
private static ScriptedAlarmEngine BuildEngine(DependencyMuxTagUpstreamSource upstream) private static ScriptedAlarmEngine BuildEngine(
DependencyMuxTagUpstreamSource upstream, IAlarmStateStore? store = null)
{ {
var logger = new LoggerConfiguration().CreateLogger(); var logger = new LoggerConfiguration().CreateLogger();
return new ScriptedAlarmEngine(upstream, new InMemoryAlarmStateStore(), new ScriptLoggerFactory(logger), logger); return new ScriptedAlarmEngine(
upstream, store ?? new InMemoryAlarmStateStore(), new ScriptLoggerFactory(logger), logger);
} }
/// <summary>The local node id used by the redundancy-gating tests.</summary> /// <summary>The local node id used by the redundancy-gating tests.</summary>
private static readonly NodeId LocalNode = new("node-A"); private static readonly NodeId LocalNode = new("node-A");
private (IActorRef Host, DependencyMuxTagUpstreamSource Upstream) Spawn( private (IActorRef Host, DependencyMuxTagUpstreamSource Upstream) Spawn(
TestProbe publish, TestProbe mux, NodeId? localNode = null) TestProbe publish, TestProbe mux, NodeId? localNode = null, IAlarmStateStore? store = null)
{ {
var upstream = new DependencyMuxTagUpstreamSource(); var upstream = new DependencyMuxTagUpstreamSource();
var engine = BuildEngine(upstream); var engine = BuildEngine(upstream, store);
var host = Sys.ActorOf(ScriptedAlarmHostActor.Props(publish.Ref, mux.Ref, upstream, engine, localNode)); var host = Sys.ActorOf(ScriptedAlarmHostActor.Props(publish.Ref, mux.Ref, upstream, engine, localNode));
return (host, upstream); return (host, upstream);
} }
@@ -794,4 +796,162 @@ public sealed class ScriptedAlarmHostActorTests : RuntimeActorTestBase
evt.AlarmId.ShouldBe("alm-1"); evt.AlarmId.ShouldBe("alm-1");
evt.TransitionKind.ShouldBe("Activated"); evt.TransitionKind.ShouldBe("Activated");
} }
// ---------------------------------------------------------------------------------------------
// #487 — post-(re)load condition-node re-assert.
//
// A deploy that triggers a FULL address-space rebuild re-materialises every condition node fresh
// (inactive/acked/confirmed/unshelved). The engine reloads its persisted state and re-derives Active
// from the predicate, so it ends up CORRECT — but a still-active alarm has no transition to report,
// yielding EmissionKind.None, which OnEngineEmission drops. Without the re-assert the node reads
// NORMAL to every OPC UA client until the alarm's next real transition — which, for an alarm with
// static dependencies, may never come.
// ---------------------------------------------------------------------------------------------
/// <summary>A persisted condition state as it survives a restart / redeploy, seeded into the store the
/// host's engine loads from.</summary>
private static AlarmConditionState Persisted(
string alarmId = "alm-1",
AlarmEnabledState enabled = AlarmEnabledState.Enabled,
AlarmActiveState active = AlarmActiveState.Active,
AlarmAckedState acked = AlarmAckedState.Unacknowledged,
AlarmConfirmedState confirmed = AlarmConfirmedState.Confirmed,
ShelvingKind shelving = ShelvingKind.Unshelved,
DateTime? lastTransitionUtc = null) =>
new(alarmId,
enabled,
active,
acked,
confirmed,
shelving == ShelvingKind.Unshelved ? ShelvingState.Unshelved : new ShelvingState(shelving, null),
lastTransitionUtc ?? DateTime.UtcNow,
LastActiveUtc: active == AlarmActiveState.Active ? lastTransitionUtc ?? DateTime.UtcNow : null,
LastClearedUtc: null,
LastAckUtc: null, LastAckUser: null, LastAckComment: null,
LastConfirmUtc: null, LastConfirmUser: null, LastConfirmComment: null,
Comments: []);
/// <summary>Seed <paramref name="state"/> into a fresh store and hand it back for
/// <see cref="Spawn"/>. The engine reads it during LoadAsync, exactly as it would after a restart.</summary>
private static IAlarmStateStore StoreWith(AlarmConditionState state)
{
var store = new InMemoryAlarmStateStore();
store.SaveAsync(state, CancellationToken.None).GetAwaiter().GetResult();
return store;
}
/// <summary>#487, the regression: an alarm whose persisted state is ACTIVE emits nothing on reload
/// (no transition to report), but the host must still write its state onto the freshly materialised
/// condition node — otherwise an active alarm silently reads normal after a full-rebuild deploy.</summary>
[Fact]
public void Reload_reasserts_an_active_alarm_onto_its_condition_node()
{
var publish = CreateTestProbe();
var mux = CreateTestProbe();
var (host, _) = Spawn(publish, mux, store: StoreWith(Persisted()));
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan(severity: 800) }));
var state = publish.ExpectMsg<OpcUaPublishActor.AlarmStateUpdate>(Timeout);
state.AlarmNodeId.ShouldBe("alm-1");
state.Realm.ShouldBe(AddressSpaceRealm.Uns);
state.State.Active.ShouldBeTrue("the persisted state was Active — the rebuilt node must say so");
state.State.Acknowledged.ShouldBeFalse();
state.State.Enabled.ShouldBeTrue();
// The projection carries the definition's severity + resolved message, so a re-asserted node is
// indistinguishable from one the alarm's own transition would have written.
state.State.Severity.ShouldBe((ushort)1000); // 800 → Critical bucket → 1000
state.State.Message.ShouldBe("condition");
}
/// <summary>The re-assert is node-ONLY: it must never reach the <c>alerts</c> topic. That topic is the
/// historization path, so an alerts row per alarm per deploy would append a duplicate historian /
/// AVEVA record every time anyone deploys.</summary>
[Fact]
public void Reassert_never_publishes_to_the_alerts_topic()
{
var publish = CreateTestProbe();
var mux = CreateTestProbe();
var alerts = CreateTestProbe();
SubscribeToAlerts(alerts);
var (host, _) = Spawn(publish, mux, store: StoreWith(Persisted()));
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan() }));
// Wait for the re-assert to actually happen FIRST — otherwise the absence window below could
// elapse before the load even completed and would prove nothing.
publish.ExpectMsg<OpcUaPublishActor.AlarmStateUpdate>(Timeout);
// Absence assertion: the elapsed time IS the assertion, so this stays deliberately short.
alerts.ExpectNoMsg(TimeSpan.FromMilliseconds(500));
}
/// <summary>An alarm sitting in the Part 9 no-event position already matches what the materialise
/// built, so it is NOT re-asserted — a deploy where nothing is in alarm writes no condition nodes at
/// all, and a never-fired alarm keeps the display name the materialise seeded as its Message.</summary>
[Fact]
public void A_normal_alarm_is_not_reasserted()
{
var publish = CreateTestProbe();
var mux = CreateTestProbe();
var (host, _) = Spawn(publish, mux); // empty store ⇒ Fresh ⇒ no-event position
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan() }));
// Sequence the absence: the re-assert (if any) happens right after the mux registration, so
// waiting for that first makes the window below cover the moment the write would have landed.
mux.ExpectMsg<DependencyMuxActor.RegisterInterest>(Timeout);
publish.ExpectNoMsg(TimeSpan.FromMilliseconds(500));
}
/// <summary>Not just Active: every field a rebuild would silently lose is restored. A shelved (but
/// inactive) alarm is re-asserted too — a rebuilt node would otherwise come back unshelved, so an
/// operator's suppression would evaporate on the next deploy.</summary>
[Fact]
public void Reload_reasserts_a_shelved_but_inactive_alarm()
{
var publish = CreateTestProbe();
var mux = CreateTestProbe();
var (host, _) = Spawn(publish, mux, store: StoreWith(Persisted(
active: AlarmActiveState.Inactive,
acked: AlarmAckedState.Acknowledged,
shelving: ShelvingKind.OneShot)));
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan() }));
var state = publish.ExpectMsg<OpcUaPublishActor.AlarmStateUpdate>(Timeout);
state.State.Active.ShouldBeFalse();
state.State.Shelving.ShouldBe(AlarmShelvingKind.OneShot);
}
/// <summary>The re-asserted timestamp is when the state actually came about, NOT the deploy instant —
/// a client reading Time/ReceiveTime after a rebuild should still see when the alarm went active.</summary>
[Fact]
public void Reassert_carries_the_conditions_own_transition_timestamp()
{
var publish = CreateTestProbe();
var mux = CreateTestProbe();
var wentActiveAt = DateTime.UtcNow.AddHours(-3);
var (host, _) = Spawn(publish, mux, store: StoreWith(Persisted(lastTransitionUtc: wentActiveAt)));
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan() }));
publish.ExpectMsg<OpcUaPublishActor.AlarmStateUpdate>(Timeout)
.TimestampUtc.ShouldBe(wentActiveAt);
}
/// <summary>A disabled PLAN exposes no condition node at all (MaterialiseScriptedAlarms skips it) and
/// is not loaded into the engine — so it must not be re-asserted either, or the write would target a
/// node that does not exist.</summary>
[Fact]
public void A_disabled_plan_is_not_reasserted()
{
var publish = CreateTestProbe();
var mux = CreateTestProbe();
var (host, _) = Spawn(publish, mux, store: StoreWith(Persisted()));
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan(enabled: false) }));
publish.ExpectNoMsg(TimeSpan.FromMilliseconds(500));
}
} }