Compare commits

...

10 Commits

Author SHA1 Message Date
Joseph Doherty 0e93587b1c merge: test-correctness, config-secret and AdminUI fixes (#503 #501 #493 #499 #488 #505)
v2-ci / build (push) Successful in 4m14s
v2-ci / unit-tests (push) Failing after 14m48s
Six issues closed, one new one found, filed and fixed.

- #503 Galaxy EventPump flake — root cause was cross-test measurement leakage on a
  STATIC Meter, not a timing budget. 15/15 clean under the issue's own contention
  recipe. Three further defects in the same test fixed on the way, including a
  "held" dispatch loop that was never held (async void on EventHandler<T>) and an
  assertion built from a derived quantity, which could only fail spuriously.
- #501 Three absence assertions given positive controls (a fourth site the issue
  did not list). Proven falsifiable by breaking the production code: both breaks
  passed under the old AwaitAssert form.
- #493 docker-dev seed backfills ClusterNode.GrpcPort; upgrade step documented.
  Deliberately no EF migration — the right value is per-node config the DB cannot
  derive, and a wrong dial target is worse than a NULL that is already skipped.
- #499 Sql credentials refused in ClusterNode.DriverConfigOverridesJson, gated at
  the SAVE rather than the deploy: node overrides never enter the artifact, so a
  deploy gate would not stop the credential being stored. Live-verified.
- #488 Periodic desired-vs-actual subscription reconcile (option A), gated on
  ISubscribable so a non-subscribable driver is not Subscribe-spammed forever.
- #505 The AdminUI cluster-node editor had never worked at all — HTTP 500 on load
  (InputNumber<byte>) and Save a silent no-op (NodeId regex forbids the ':' every
  NodeId contains), with no ValidationSummary to reveal it. Pre-existing on master,
  invisible to 739 green AdminUI tests, found only by going to drive the page.

Also closed #492 (already fixed by 2964361a and drill-verified; no code change).

Verified: solution builds 0 errors; 2406 tests pass across Runtime (507), AdminUI
(739), Configuration (131), Galaxy (317), Cluster (147), Core (260),
ScriptedAlarms (82) and Sql (223). #499 and #505 live-gated on docker-dev with both
central nodes rebuilt; rig restored afterwards.
2026-07-26 10:51:43 -04:00
Joseph Doherty 4cc0215bb4 fix(adminui): the cluster-node editor was dead — 500 on load, silent no-op on save
Both defects are PRE-EXISTING (present on master, unrelated to #499) and were found
live-verifying the #499 guard, which lives on this page and was unreachable without
them fixed.

1. **HTTP 500 on load.** `FormModel.ServiceLevelBase` was `byte` to match the
   entity, and `InputNumber<T>`'s static constructor throws outright for
   `System.Byte` — "The type 'System.Byte' is not a supported numeric type". That
   escapes `BuildRenderTree` unhandled, which in Blazor takes the WHOLE page down,
   not one field. `/clusters/{id}/nodes/{nodeId}` and `.../nodes/new` have never
   rendered. Now `int` with the same `[Range(0, 255)]` and a cast on the way to the
   entity, so the stored type is unchanged. (Same failure mode as #504: an
   unhandled throw inside the render tree is a page-kill.)

2. **Save did nothing, silently.** `NodeId` was validated against
   `^[A-Za-z0-9_-]+$`, which forbids the colon — but every NodeId in this system is
   `host:port` (`ClusterRoleInfo` and `ConfigPublishCoordinator` derive it as
   `member.Address.Host:Port`, the seed writes `central-1:4053`, and
   `NodeDeploymentState.NodeId` is FK-bound to it). So every existing node failed
   validation, `OnValidSubmit` never ran, and the button was inert. Pattern now
   allows `:` and `.` (hostnames may be dotted).

3. **Added the missing `<ValidationSummary/>`.** The form had a
   `DataAnnotationsValidator` but nothing rendering its output, so a validation
   failure produced no feedback at all — which is precisely how (2) stayed
   invisible. This is the change that stops the next such defect being silent.

Live-verified on docker-dev (both central nodes rebuilt, since :9200 round-robins
the pair): page 500 → 200 on both routes, and a real edit now persists.
2026-07-26 10:42:04 -04:00
Joseph Doherty 4b9bcddbcc feat(drivers): periodic reconcile of desired vs actual subscriptions (#488)
Option A from the issue — the cheap consistency check, no interface change.

The desired ref set was re-applied on a Connected TRANSITION (`ResubscribeDesired`)
and on a deploy's `SetDesiredSubscriptions`, and nowhere else. A subscription lost
while the driver STAYED Connected — a subscribe that failed and was never retried,
a handle dropped down an error path — left the actor subscribed to nothing while
still reporting Healthy, indefinitely.

`ReconcileSubscription` now rides the existing 30 s `health-poll` tick: while
Connected, a driver holding a desired set but no live handle is re-subscribed.

Gated on `ISubscribable`. A driver that cannot subscribe at all can still be handed
a desired set, and without the guard it would be told `Subscribe` every tick and
fail every tick, forever.

This is a state-machine consistency check, NOT a probe — it can only see that this
actor holds no handle, never that a handle is present but dead server-side, because
`ISubscriptionHandle` is opaque. That is option B, and it stays deferred until a
real occurrence shows the handle outliving the subscription; it would need a
liveness member implemented across all eight drivers with different subscription
mechanics.

A redundant `Subscribe` is possible (a tick queued ahead of an already-queued one
observes the same no-handle state) and is harmless: `HandleSubscribeAsync` is a
`ReceiveAsync`, so no tick is processed mid-subscribe, and it drops any prior
subscription before establishing the new one. Suppressing it would need a
pending-subscribe flag threaded through every self-tell site — more state than the
race is worth.

`healthPollInterval` becomes an optional Props parameter, mirroring the existing
`rediscoverInterval` seam, so a test can drive the reconcile without waiting 30 s.

Three tests. The reconcile itself is proven by a stub whose FIRST subscribe throws:
nothing else would ever retry it, so a second attempt can only come from the
reconcile — disabling `ReconcileSubscription` turns it red. The two guard tests are
absence assertions and are bounded by a positive control rather than a sleep: a
counting health publisher makes ticks PROCESSED observable, so "still one subscribe"
means the reconcile ran and declined, not that the timer had not fired.

Explicitly not deploy-time drift (#486) — re-asserting an already-correct desired
set would not have helped there.
2026-07-26 10:30:19 -04:00
Joseph Doherty 2193d9f2e4 fix(config): refuse to store a Sql credential in a node config override (#499)
`ClusterNode.DriverConfigOverridesJson` is the third surface a Sql driver's config
is persisted on, and the only one nothing gated. It is a map keyed by
`DriverInstanceId` merged onto the cluster-level `DriverConfig`, so a literal
`connectionString` pasted there leaks exactly as one pasted into the driver — the
leak #498 closed for `DriverInstance.DriverConfig` and `Device.DeviceConfig`.

Gated at the SAVE, not at the deploy — a deliberate departure from both options the
issue offered:

- `DraftSnapshot` carries no `ClusterNode` rows, and adding them would widen the
  snapshot and every builder of one for a single rule about a value that never
  enters the artifact.
- More to the point, a deploy gate is the wrong instrument here. Node overrides are
  not in the artifact, so blocking a deploy would not stop the credential being
  stored — it is already in the database and replicated by then. Refusing the save
  is the only point where "refuse to store it" is literally true, which is #498's
  own framing: discarding a secret on read is not the same as refusing to store it.

The check moves into a shared `SqlCredentialGuard` so the two enforcement points
cannot drift; `DraftValidator` now calls it instead of its own private helper.

Kept deliberately narrow — the `Sql` driver's `connectionString` only, and only for
instance ids that ARE Sql drivers. The issue asked whether to generalise to
credential-shaped keys across every driver type; not doing that, for the same
reason #498 did not: a broader sweep would start refusing configs that are
legitimate today for drivers which never made an indirect-credential guarantee,
which is a regression rather than defence in depth. Widen per driver, as each gains
its own contract.

The error names the offending driver instance(s) and NEVER the value — it reaches
the AdminUI and the audit trail.

14 new cases cover both halves, including the case variants (System.Text.Json binds
`ConnectionString` to `connectionString`, so a case variant is the same key, not a
bypass), non-Sql ids being ignored, and every blank/malformed/non-object shape.
2026-07-26 10:30:04 -04:00
Joseph Doherty 8b7ea3213d fix(rig,docs): backfill ClusterNode.GrpcPort and document the upgrade step (#493)
`GrpcPort` is a nullable column added by Phase 5. Every insert in the docker-dev
seed is INSERT-if-not-exists — the right shape for a re-runnable seed, but it means
a new nullable column never reaches rows an earlier version of the file created. On
a persisted volume all six rows kept NULL, central found no dial targets, and
`TelemetryDial:Mode=Grpc` connected to nothing. (`AkkaPort` escaped this only
because it is NOT NULL with a default of 4053.)

Seed now backfills `GrpcPort IS NULL AND CreatedBy = 'docker-dev-seed'` to 4056.
NULL only — it must not overwrite a port an operator set deliberately on a
long-lived dev volume, and NULL is the unambiguous "predates the column" marker.

Deliberately NOT doing the suggested EF data migration for real deployments. The
correct value is each node's own `Telemetry:GrpcListenPort`, which is
per-deployment configuration the database cannot derive; a migration could only
invent one, and a WRONG dial target is worse than the NULL the dialer already skips
explicitly with a throttled warning. The rig can be backfilled because there the
port is fixed by docker-compose.yml and therefore actually knowable.

`docs/Telemetry.md` gains an upgrade section stating the prerequisite, the symptoms
(the throttled skip log, LISTEN with no ESTABLISHED, the pill never going live) and
the UPDATE to run before flipping any node to Grpc — plus why there is no migration.
2026-07-26 10:29:51 -04:00
Joseph Doherty 240d7aa8aa fix(test): give the three absence assertions a positive control (#501)
`AwaitAssert` returns on its FIRST success, so an assertion that is already true at
t=0 passes instantly and spends none of its duration. All three sites asserted an
absence against a collection that starts empty, so none of them ever gave the
handler the time their comments claimed.

Replaced with mailbox-ordering positive controls rather than sleeps:

- `RouteNativeAlarmAck_unknown_node_is_dropped` — follow the unmapped ack with a
  MAPPED one. The host forwards via `Tell` and the child handles `RouteAlarmAck`
  with `ReceiveAsync` (which suspends its own mailbox), so once the control reaches
  the driver the unmapped one has provably been handled.
- `RouteNativeAlarmAck_on_non_primary_is_dropped` — the role itself is the control:
  return the node to Primary and re-send. Comments (`gated` / `control`) are the
  discriminator, so a leaked ack is visible rather than merged into a count.
- `PeerProbeSupervisorTests.Single_node_snapshot_spawns_no_children` — a FOURTH site
  the issue did not list. It says the other `ChildCount.ShouldBe(0)` waits are each
  preceded by a `ShouldBe(1)`; that is true of three of the four, but not this one,
  which asserts the initial state. Now followed by a two-node snapshot: the count
  reaching 1 proves the single-node snapshot was processed, and a local-node child
  would have made it 2.

The issue warned these may turn red, which would be a real defect surfacing. They
did not — the routing is correct. Proven by breaking it deliberately instead:
disabling the Primary gate reddens the Secondary test, and routing unmapped ids to
an arbitrary mapping reddens the unknown-node test. Under the old form both breaks
passed.
2026-07-26 10:29:41 -04:00
Joseph Doherty 95295210b0 fix(test): scope the EventPump counter capture to its own pump (#503)
Root cause was cross-test measurement leakage, not a timing budget.

EventPump's `Meter` is STATIC — one instance per process — and a `MeterListener`
can only select by *meter* in `InstrumentPublished`. `StartMeterCapture` therefore
received measurements from every `EventPump` alive anywhere in the assembly. xUnit
runs test classes in parallel and several build pumps (`EventPumpStreamFaultTests`,
the `Driver-X` test in this same file), so a sibling's events landed in these
counters. That is why the flake needed a busy box: it needed two tests to overlap.
Measured directly during the gate — `Received = 11` in a run that emitted 10.

The old `Received == Dispatched + Dropped + InFlight` assertion is what turned the
leak into a failure. `InFlight` is DERIVED as `Received - Dispatched - Dropped`, so
the identity is a tautology that cannot fail on a real defect — only on a foreign
increment landing between its four separate `Interlocked.Read` calls. Hence the
reported subject, `counters.Received`. It is deleted, along with the `InFlight`
member, and replaced by direct assertions on the three MEASURED counters.

Three further defects in the same test, all found on the way:

- **The "held" dispatch loop was never held.** `OnDataChange` is `EventHandler<T>`
  — void-returning — so `async (_, _) => await gate` is async VOID: it returned to
  the dispatch loop at the first await and blocked nothing. Measured `Dispatched =
  2..3`, never 0. Drops happened only when the producer happened to outrun the
  consumer, so the arrangement the doc comment describes was fiction and the
  assertions rode on scheduling luck. Now a synchronous `ManualResetEventSlim.Wait()`
  on the loop's own thread, pinned by `Dispatched.ShouldBe(0)`.
- **Fixed `Task.Delay(150)`** replaced by a presence poll on the drop count (#500's
  rule). Dropped starts at 0, so it cannot be satisfied by the initial state.
- **Emission is staged** — one event, wait for the dispatcher to enter the handler,
  then the other nine — so the arithmetic is exact (`10 - 1 held - 2 buffered = 7`)
  rather than scheduler-dependent.

The gate release moved into `finally`, ahead of disposal: `DisposeAsync` awaits the
dispatch loop, which is now genuinely parked on that gate, so a failed assertion
would otherwise hang instead of failing.

Verified: 15/15 clean under the exact contention the issue measured (Runtime +
AdminUI + Core concurrently) — where the half-fixed version still failed on run 4.
Falsifiability: restoring the async-void handler turns the new assertions red.
2026-07-26 10:29:27 -04:00
Joseph Doherty e3155fbbf5 merge: never slice a DB-sourced string with the range operator (#504)
v2-ci / build (push) Successful in 4m11s
v2-ci / unit-tests (push) Failing after 14m22s
`@s.SourceHash[..12]` on an unvalidated nvarchar(64) threw out of
BuildRenderTree and took the whole /scripts page to an HTTP 500. Replaced with a
shared, null/short-safe `DisplayText.Abbreviate`, applied to all five unguarded
range-operator slices over DB-sourced strings in the AdminUI.

Live-verified on docker-dev: /scripts 500 -> 200 with the short hashes rendered
in full; long hashes still truncate with an ellipsis on the other pages.
2026-07-26 09:43:55 -04:00
Joseph Doherty 47d148daf9 fix(adminui): never slice a DB-sourced string with the range operator (#504)
`Scripts.razor` rendered a hash prefix as `@s.SourceHash[..12]…`. `SourceHash`
is a plain nvarchar(64) with no length floor, so any value shorter than 12
characters threw ArgumentOutOfRangeException out of BuildRenderTree — unhandled
in Blazor, which takes the WHOLE page to an HTTP 500, not just the offending
row. Live-reproduced on docker-dev, where two Script rows carry `h1` and
`h-abs-hr200`: /scripts was a hard 500.

New `Components/Shared/DisplayText.Abbreviate(string?, int)`:
- null/blank renders the Admin UI's em-dash placeholder rather than throwing;
- a value already within budget renders verbatim with NO ellipsis, so the
  display never implies text that isn't there (the old markup appended "…"
  unconditionally, outside the slice — `h1` would have shown as `h1…`);
- a maxLength < 1 is clamped, so a display bug cannot become a page crash.

Applied to every unguarded range-operator slice over a DB-sourced string found
by sweeping the AdminUI for `[..N]`:

  Scripts.razor                    SourceHash    (the reported crash)
  Certificates.razor               Thumbprint    (read off the on-disk store)
  Deployments.razor  x2            RevisionHash
  Clusters/ClusterOverview.razor   RevisionHash

The other `[..N]` hits are safe and untouched: Guid.ToString("N") slices
(always 32 chars) and the `Length > 60 ? [..60]` ternaries that self-guard.

NOTE — the issue text blamed the docker-dev seed; that was wrong and is
corrected on #504. `seed-clusters.sql` inserts only ServerCluster, ClusterNode
and LdapGroupRoleMapping. The short-hash rows came from earlier live-gate
authoring. A freshly seeded rig is fine; the trigger is any Script row that did
not go through ScriptEdit / UnsTreeService.HashSource — REST-API authored,
hand-inserted, or migrated in.

Tests: 14 new in DisplayTextTests, covering the short/null/blank/clamped cases
and a never-throws sweep over every value x budget combination. AdminUI suite
739/739.

Live-verified on docker-dev (AdminUI has no bUnit): /scripts 500 -> 200 across
both central nodes, rendering `hash=h1` and `hash=h-abs-hr200` in full;
/deployments still truncates 64-char hashes at 12 with the ellipsis;
/certificates and /clusters/MAIN unchanged.
2026-07-26 09:43:48 -04:00
Joseph Doherty 755ae1aa3f merge: re-assert non-normal scripted-alarm conditions after a (re)load (#487)
v2-ci / build (push) Successful in 4m21s
v2-ci / unit-tests (push) Failing after 3h14m40s
Closes the redeploy-reset gap on Part 9 scripted-alarm condition nodes — the
alarm analogue of the VirtualTag ReassertValue fix.

A full address-space rebuild re-materialises every condition node fresh, while
the engine's reload produces EmissionKind.None for an alarm whose state did not
change — so nothing wrote the node. Live-reproduced on docker-dev: a cleared-
but-still-UNACKNOWLEDGED alarm came back reporting acknowledged, dropping Retain
to false, so it vanished from ConditionRefresh entirely. The operator's
outstanding alarm silently disappeared from the alarm list on deploy.

ScriptedAlarmHostActor.ReassertConditionNodes now writes one AlarmStateUpdate
per alarm not in the Part 9 no-event position, sourced from the new
ScriptedAlarmEngine.GetProjections(). Node-only by construction — the projection
type carries no EmissionKind, so it cannot become an alerts row or a duplicate
historian record.

Live gate PASSED (pre-fix image vs. rebuilt, central-2): condition absent
pre-fix, present + Unacknowledged + Retain=True post-fix, stamped with its own
LastTransitionUtc; /alerts empty across two re-asserts with the panel proven
live by a real ACTIVATED transition.
2026-07-26 09:18:23 -04:00
18 changed files with 933 additions and 102 deletions
+22
View File
@@ -106,6 +106,28 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-b-2:4053')
(NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, GrpcPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
VALUES ('site-b-2:4053', 'SITE-B', 'site-b-2', 4840, 8081, 4053, 4056, 'urn:OtOpcUa:site-b-2', 150, 1, 'docker-dev-seed');
------------------------------------------------------------------------------
-- Backfill: GrpcPort on rows that predate the column (#493)
--
-- Every insert above is INSERT-if-not-exists, which is the right shape for a re-runnable seed
-- but means a *new nullable* column never reaches rows created by an earlier version of this
-- file. On a persisted volume that is exactly what happened when Phase 5 added GrpcPort: the
-- six ClusterNode rows already existed, so they kept NULL, TelemetryNodeSource found no dial
-- targets, and TelemetryDial:Mode=Grpc silently connected to nothing (throttled Warning per
-- node — the code degrades gracefully, so nothing failed loudly).
--
-- Backfills NULL only. It must not overwrite a port an operator set deliberately on a
-- long-lived dev volume; a NULL is the unambiguous "this row predates the column" marker.
--
-- AkkaPort deliberately has no equivalent here: it is NOT NULL with a default of 4053, so
-- pre-existing rows already carry a usable value and there is nothing to backfill.
------------------------------------------------------------------------------
UPDATE dbo.ClusterNode
SET GrpcPort = 4056 -- matches Telemetry__GrpcListenPort on all six nodes in docker-compose.yml
WHERE GrpcPort IS NULL
AND CreatedBy = 'docker-dev-seed';
------------------------------------------------------------------------------
-- Galaxy MxAccess gateway — INTENTIONALLY NOT SEEDED (retired 2026-06-15)
--
+33
View File
@@ -128,6 +128,39 @@ transports. ScadaBridge itself has since closed that gap with this identical
interceptor pattern, so Phase 5 ships authenticated from the start rather than deferring auth to a
later hardening pass. See the design doc's superseded note for the full history.
## Upgrading an existing deployment: populate `ClusterNode.GrpcPort` first (#493)
`GrpcPort` is a **nullable column added by Phase 5**, and nothing backfills it onto rows that
already existed. On an upgraded deployment every `ClusterNode` row therefore carries `NULL`,
central finds **no dial targets**, and flipping `TelemetryDial:Mode` to `Grpc` connects to
nothing. Fresh installs are unaffected. `AkkaPort` did not have this problem only because it is
`NOT NULL` with a default of 4053.
Nothing fails loudly, which is what makes this worth stating up front — the code degrades
gracefully, once per node, throttled:
```
ClusterNode <id> has no GrpcPort; it exposes no telemetry stream and is skipped in this dial refresh
(further skips of this node are silent)
```
Other symptoms: no `ESTABLISHED` sockets on the telemetry port (a `LISTEN` but nothing else in
`docker exec <node> cat /proc/net/tcp6`), and the `/alerts` / `/script-log` pill never turning
live in `Grpc` mode.
**Before flipping any node to `Grpc`**, set each driver node's `GrpcPort` to that node's own
`Telemetry:GrpcListenPort` — via the AdminUI node editor, or directly:
```sql
UPDATE dbo.ClusterNode SET GrpcPort = <that node's Telemetry:GrpcListenPort> WHERE NodeId = '<node>';
```
There is deliberately **no EF data migration** backfilling this. The correct value is that node's
own listener port, which is per-deployment configuration the database cannot derive; a migration
could only invent one, and a *wrong* dial target is worse than the `NULL` the dialer already skips
explicitly. The docker-dev seed does backfill (`GrpcPort IS NULL AND CreatedBy = 'docker-dev-seed'`
→ 4056) because there the port is fixed by `docker-compose.yml` and therefore actually knowable.
## Reconnect and reliability
Central's `TelemetryDialSupervisor` (an admin-role actor) runs **one reconnecting dialer per
@@ -60,6 +60,11 @@ public static class DraftValidator
/// credential in the ConfigDb — persisted, replicated to every node's artifact cache, and readable by
/// anyone with config access — while the runtime silently ignored it and the driver failed to connect.
/// Discarding a secret on read is not the same as refusing to store it.</para>
/// <para><b>Two of the three surfaces are checked here.</b> The third — a node's
/// <see cref="ClusterNode.DriverConfigOverridesJson"/> — is not reachable from
/// <see cref="DraftSnapshot"/> and is gated at its save instead (#499); see
/// <see cref="SqlCredentialGuard"/> for why a deploy gate is the wrong instrument for a value that
/// never enters the artifact.</para>
/// <para>Checked on <b>both</b> config surfaces a Sql driver reads: the instance's
/// <see cref="DriverInstance.DriverConfig"/> and the <see cref="Device.DeviceConfig"/> of every device
/// beneath it, because the two are merged before the DTO sees them — a credential pasted into the
@@ -74,7 +79,7 @@ public static class DraftValidator
/// </remarks>
private static void ValidateSqlConnectionStringNotPersisted(DraftSnapshot draft, List<ValidationError> errors)
{
const string ForbiddenKey = "connectionString";
const string ForbiddenKey = SqlCredentialGuard.ForbiddenKey;
var sqlInstanceIds = draft.DriverInstances
.Where(d => string.Equals(d.DriverType, Core.Abstractions.DriverTypeNames.Sql, StringComparison.Ordinal))
@@ -85,7 +90,7 @@ public static class DraftValidator
foreach (var d in draft.DriverInstances)
{
if (!sqlInstanceIds.Contains(d.DriverInstanceId)) continue;
if (!HasTopLevelKey(d.DriverConfig, ForbiddenKey)) continue;
if (!SqlCredentialGuard.CarriesLiteralConnectionString(d.DriverConfig)) continue;
errors.Add(new("SqlConnectionStringPersisted",
$"Sql driver instance '{d.DriverInstanceId}' has a '{ForbiddenKey}' key in its DriverConfig. " +
"A Sql driver must name its credentials indirectly via 'connectionStringRef', which resolves " +
@@ -98,7 +103,7 @@ public static class DraftValidator
foreach (var dev in draft.Devices)
{
if (!sqlInstanceIds.Contains(dev.DriverInstanceId)) continue;
if (!HasTopLevelKey(dev.DeviceConfig, ForbiddenKey)) continue;
if (!SqlCredentialGuard.CarriesLiteralConnectionString(dev.DeviceConfig)) continue;
errors.Add(new("SqlConnectionStringPersisted",
$"Device '{dev.DeviceId}' on Sql driver instance '{dev.DriverInstanceId}' has a " +
$"'{ForbiddenKey}' key in its DeviceConfig. DeviceConfig is merged onto DriverConfig before " +
@@ -107,34 +112,6 @@ public static class DraftValidator
}
}
/// <summary>
/// True when <paramref name="json"/> is a JSON object carrying <paramref name="key"/> at its top level,
/// matched case-insensitively. Never throws — a blank, malformed or non-object blob simply has no keys,
/// and shaping the config JSON is another rule's job.
/// </summary>
/// <param name="json">The config blob to inspect.</param>
/// <param name="key">The property name to look for.</param>
/// <returns><see langword="true"/> when the key is present at the top level.</returns>
private static bool HasTopLevelKey(string? json, string key)
{
if (string.IsNullOrWhiteSpace(json)) return false;
try
{
using var doc = System.Text.Json.JsonDocument.Parse(json);
if (doc.RootElement.ValueKind != System.Text.Json.JsonValueKind.Object) return false;
foreach (var property in doc.RootElement.EnumerateObject())
{
if (string.Equals(property.Name, key, StringComparison.OrdinalIgnoreCase)) return true;
}
return false;
}
catch (System.Text.Json.JsonException)
{
return false;
}
}
/// <summary>WP7 Calculation-driver deploy gates. For every tag bound to a <c>Calculation</c> driver:
/// <list type="number">
/// <item><b>scriptId existence</b> — the tag's <c>TagConfig.scriptId</c> must be present and resolve to
@@ -0,0 +1,139 @@
using System.Text.Json;
namespace ZB.MOM.WW.OtOpcUa.Configuration.Validation;
/// <summary>
/// The one place that decides whether a persisted config blob carries a literal Sql
/// <c>connectionString</c> (Gitea #498 / #499).
/// </summary>
/// <remarks>
/// <para>
/// A <c>Sql</c> driver names its credentials indirectly — <c>connectionStringRef</c> resolves from
/// the environment / secret store at Initialize — so nothing persisted should ever contain a
/// database password. The typed <c>SqlDriverConfigDto</c> already drops a <c>connectionString</c>
/// key on <b>read</b>, but discarding a secret on read is not the same as refusing to store it:
/// config blobs are schemaless JSON columns authored through raw-JSON textareas, so the key can
/// be written even though the runtime ignores it.
/// </para>
/// <para>
/// There are <b>three</b> surfaces a Sql driver's config is persisted on, and they need different
/// enforcement points:
/// <list type="bullet">
/// <item><c>DriverInstance.DriverConfig</c> and <c>Device.DeviceConfig</c> — both in the
/// deployed artifact, both visible to <c>DraftSnapshot</c>, both gated by
/// <c>DraftValidator</c>.</item>
/// <item><c>ClusterNode.DriverConfigOverridesJson</c> — the per-node override map. It is
/// <b>not</b> in <c>DraftSnapshot</c>, and deliberately does not go there: adding a
/// <c>ClusterNode</c> collection would widen the snapshot (and every builder of one) for a
/// single rule about a value that is not part of the artifact at all. More to the point, a
/// deploy gate is the wrong instrument here — the artifact never carries node overrides, so
/// blocking a deploy would not stop the credential being stored. It is already in the
/// database by then. This surface is therefore gated at the <b>save</b>, which is the only
/// point where "refuse to store it" is literally true.</item>
/// </list>
/// </para>
/// <para>
/// <b>Deliberately narrow.</b> This checks the <c>Sql</c> driver's <c>connectionString</c> key
/// only — not a general "credential-shaped key" sweep over every driver type. A broader rule
/// would start refusing configs that are legitimate today for drivers that never made this
/// guarantee, turning defence-in-depth into a regression. Widen it per driver, as each driver
/// gains an indirect-credential contract of its own.
/// </para>
/// </remarks>
public static class SqlCredentialGuard
{
/// <summary>The config key a Sql driver must never persist. Matched case-insensitively.</summary>
public const string ForbiddenKey = "connectionString";
/// <summary>
/// Whether <paramref name="configJson"/> is a JSON object carrying <see cref="ForbiddenKey"/> at
/// its top level.
/// </summary>
/// <remarks>
/// Matched <b>case-insensitively</b>: <c>System.Text.Json</c> binds <c>ConnectionString</c> to a
/// <c>connectionString</c> property by default, so a case variant is the same key, not a
/// different one. Only the top level is scanned — the DTO is flat, so a nested occurrence cannot
/// bind and is not the credential-shaped mistake this exists to catch.
/// </remarks>
/// <param name="configJson">The config blob to inspect.</param>
/// <returns><see langword="true"/> when the key is present at the top level.</returns>
public static bool CarriesLiteralConnectionString(string? configJson)
{
if (string.IsNullOrWhiteSpace(configJson)) return false;
try
{
using var doc = JsonDocument.Parse(configJson);
return doc.RootElement.ValueKind == JsonValueKind.Object
&& HasTopLevelKey(doc.RootElement, ForbiddenKey);
}
catch (JsonException)
{
// A malformed blob simply has no keys. Shaping the config JSON is another rule's job, and
// throwing here would turn a formatting mistake into a credential-guard failure.
return false;
}
}
/// <summary>
/// The <c>DriverInstanceId</c>s in a <c>ClusterNode.DriverConfigOverridesJson</c> map whose
/// override object carries a literal connection string, restricted to instances that are actually
/// Sql drivers.
/// </summary>
/// <remarks>
/// The override map is shaped <c>{ "&lt;DriverInstanceId&gt;": { …driver config keys… } }</c> and is
/// merged onto the cluster-level <c>DriverConfig</c>, so a credential pasted into one lands in
/// exactly the place <see cref="CarriesLiteralConnectionString"/> already refuses on the driver
/// itself.
/// </remarks>
/// <param name="overridesJson">The node's override map. Null / blank / malformed yields no
/// violations.</param>
/// <param name="sqlDriverInstanceIds">The ids of driver instances whose <c>DriverType</c> is
/// <c>Sql</c>. Keys outside this set are ignored — a non-Sql driver never made the indirect-credential
/// guarantee, so flagging it would be a regression, not defence in depth.</param>
/// <returns>The offending driver-instance ids, in document order; empty when there are none.</returns>
public static IReadOnlyList<string> FindNodeOverrideViolations(
string? overridesJson, IReadOnlyCollection<string> sqlDriverInstanceIds)
{
if (string.IsNullOrWhiteSpace(overridesJson) || sqlDriverInstanceIds.Count == 0)
{
return [];
}
var sqlIds = sqlDriverInstanceIds as IReadOnlySet<string>
?? sqlDriverInstanceIds.ToHashSet(StringComparer.Ordinal);
try
{
using var doc = JsonDocument.Parse(overridesJson);
if (doc.RootElement.ValueKind != JsonValueKind.Object) return [];
List<string>? offenders = null;
foreach (var entry in doc.RootElement.EnumerateObject())
{
if (!sqlIds.Contains(entry.Name)) continue;
if (entry.Value.ValueKind != JsonValueKind.Object) continue;
if (!HasTopLevelKey(entry.Value, ForbiddenKey)) continue;
(offenders ??= []).Add(entry.Name);
}
return (IReadOnlyList<string>?)offenders ?? [];
}
catch (JsonException)
{
return [];
}
}
/// <summary>Whether <paramref name="obj"/> carries <paramref name="key"/> at its top level,
/// case-insensitively.</summary>
private static bool HasTopLevelKey(JsonElement obj, string key)
{
foreach (var property in obj.EnumerateObject())
{
if (string.Equals(property.Name, key, StringComparison.OrdinalIgnoreCase)) return true;
}
return false;
}
}
@@ -81,7 +81,7 @@ else
<tr>
<td><span class="mono small">@c.Subject</span></td>
<td><span class="mono small">@c.Issuer</span></td>
<td><span class="mono small">@c.Thumbprint[..16]…</span></td>
<td><span class="mono small">@DisplayText.Abbreviate(c.Thumbprint, 16)</span></td>
<td>@c.NotBefore.ToString("u")</td>
<td>@c.NotAfter.ToString("u")</td>
@if (store.Kind is StoreKind.Trusted or StoreKind.Rejected)
@@ -58,7 +58,7 @@ else
}
else
{
<div class="kv"><span class="k">Revision</span><span class="v mono">@_lastDeployment.RevisionHash[..16]…</span></div>
<div class="kv"><span class="k">Revision</span><span class="v mono">@DisplayText.Abbreviate(_lastDeployment.RevisionHash, 16)</span></div>
<div class="kv"><span class="k">Status</span><span class="v">@_lastDeployment.Status</span></div>
<div class="kv"><span class="k">Created</span><span class="v">@_lastDeployment.CreatedAtUtc.ToString("u")</span></div>
@if (_lastDeployment.SealedAtUtc is not null)
@@ -9,6 +9,8 @@
@using System.ComponentModel.DataAnnotations
@using ZB.MOM.WW.OtOpcUa.Configuration
@using ZB.MOM.WW.OtOpcUa.Configuration.Entities
@using ZB.MOM.WW.OtOpcUa.Configuration.Validation
@using ZB.MOM.WW.OtOpcUa.Core.Abstractions
@inject IDbContextFactory<OtOpcUaConfigDbContext> DbFactory
@inject NavigationManager Nav
@inject AuthenticationStateProvider AuthState
@@ -34,6 +36,10 @@ else
{
<EditForm Model="_form" OnValidSubmit="SubmitAsync" FormName="nodeEdit">
<DataAnnotationsValidator />
@* Without this, a validation failure suppresses OnValidSubmit with NO feedback whatsoever —
the Save button simply does nothing and the operator has no way to tell why. That is how the
NodeId-regex defect above stayed invisible. *@
<ValidationSummary class="alert alert-danger py-2 px-3" />
<section class="panel rise" style="animation-delay:.02s">
<div class="panel-head">Identity</div>
<div style="padding:1rem">
@@ -178,6 +184,33 @@ else
}
await using var db = await DbFactory.CreateDbContextAsync();
// #499 — the third Sql-credential surface. DriverConfigOverridesJson is merged onto the
// cluster-level DriverConfig, so a literal connectionString pasted here leaks exactly as one
// pasted into the driver itself. The deploy gate cannot see it (DraftSnapshot carries no
// ClusterNode rows) and would be the wrong place anyway: node overrides never enter the
// artifact, so by the time a deploy ran the credential would already be stored. Refuse the
// save instead. The message names the driver instance but NEVER the value.
if (!string.IsNullOrWhiteSpace(_form.DriverConfigOverridesJson))
{
var sqlDriverIds = await db.DriverInstances
.Where(d => d.ClusterId == ClusterId && d.DriverType == DriverTypeNames.Sql)
.Select(d => d.DriverInstanceId)
.ToListAsync();
var offenders = SqlCredentialGuard.FindNodeOverrideViolations(
_form.DriverConfigOverridesJson, sqlDriverIds);
if (offenders.Count > 0)
{
_error =
$"Driver config overrides carry a '{SqlCredentialGuard.ForbiddenKey}' for Sql driver " +
$"instance(s) {string.Join(", ", offenders)}. A Sql driver must name its credentials " +
"indirectly via 'connectionStringRef', which resolves from the environment / secret " +
"store at Initialize; a literal connection string here would be stored in the config " +
"database, and the runtime ignores it anyway. Remove the key.";
return;
}
}
if (IsNew)
{
if (await db.ClusterNodes.AnyAsync(n => n.NodeId == _form.NodeId))
@@ -194,7 +227,7 @@ else
OpcUaPort = _form.OpcUaPort,
DashboardPort = _form.DashboardPort,
ApplicationUri = _form.ApplicationUri,
ServiceLevelBase = _form.ServiceLevelBase,
ServiceLevelBase = (byte)_form.ServiceLevelBase,
Enabled = _form.Enabled,
DriverConfigOverridesJson = string.IsNullOrWhiteSpace(_form.DriverConfigOverridesJson) ? null : _form.DriverConfigOverridesJson,
CreatedAt = DateTime.UtcNow,
@@ -210,7 +243,7 @@ else
entity.OpcUaPort = _form.OpcUaPort;
entity.DashboardPort = _form.DashboardPort;
entity.ApplicationUri = _form.ApplicationUri;
entity.ServiceLevelBase = _form.ServiceLevelBase;
entity.ServiceLevelBase = (byte)_form.ServiceLevelBase;
entity.Enabled = _form.Enabled;
entity.DriverConfigOverridesJson = string.IsNullOrWhiteSpace(_form.DriverConfigOverridesJson) ? null : _form.DriverConfigOverridesJson;
}
@@ -254,14 +287,26 @@ else
private sealed class FormModel
{
[Required, RegularExpression("^[A-Za-z0-9_-]+$")]
// The ':' and '.' are REQUIRED, not permissive. Every NodeId in this system is "host:port" —
// ClusterRoleInfo and ConfigPublishCoordinator derive it as member.Address.Host:Port, the seed
// writes "central-1:4053", and NodeDeploymentState.NodeId is FK-bound to it. The previous
// pattern forbade the colon, so EVERY existing node failed validation and OnValidSubmit never
// ran: the Save button did nothing at all, silently. Hostnames may also be dotted
// (line3-opc-a.plant.local:4053).
[Required, RegularExpression("^[A-Za-z0-9_.:-]+$")]
public string NodeId { get; set; } = "";
[Required] public string Host { get; set; } = "";
[Range(1, 65535)] public int OpcUaPort { get; set; } = 4840;
[Range(1, 65535)] public int DashboardPort { get; set; } = 8081;
[Required, RegularExpression("^urn:[A-Za-z0-9_:./-]+$")]
public string ApplicationUri { get; set; } = "";
[Range(0, 255)] public byte ServiceLevelBase { get; set; } = 200;
// int, NOT byte. Blazor's InputNumber<T> rejects System.Byte outright — its static
// constructor throws "The type 'System.Byte' is not a supported numeric type", which escapes
// BuildRenderTree unhandled and takes the WHOLE page to an HTTP 500. Binding this field to
// the entity's own byte type meant /clusters/{id}/nodes/{nodeId} and .../nodes/new never
// rendered at all. The Range attribute still holds it to a byte's domain, and SubmitAsync
// casts on the way to the entity, so the stored type is unchanged.
[Range(0, 255)] public int ServiceLevelBase { get; set; } = 200;
public bool Enabled { get; set; } = true;
public string? DriverConfigOverridesJson { get; set; }
}
@@ -55,7 +55,7 @@
{
<tr>
<td><code>@Short(d.DeploymentId)</code></td>
<td><code>@d.RevisionHash[..12]…</code></td>
<td><code>@DisplayText.Abbreviate(d.RevisionHash, 12)</code></td>
<td>@d.Status</td>
<td>@d.CreatedBy</td>
<td>@d.CreatedAtUtc.ToString("u")</td>
@@ -113,7 +113,7 @@
_lastSuccess = result.Outcome == StartDeploymentOutcome.Accepted;
_lastMessage = result.Outcome switch
{
StartDeploymentOutcome.Accepted => $"Deployment {Short(result.DeploymentId!.Value.Value)} dispatched (rev {result.RevisionHash!.Value.Value[..12]}…).",
StartDeploymentOutcome.Accepted => $"Deployment {Short(result.DeploymentId!.Value.Value)} dispatched (rev {DisplayText.Abbreviate(result.RevisionHash!.Value.Value, 12)}).",
StartDeploymentOutcome.AnotherDeploymentInFlight => result.Message ?? "Another deployment is already in flight.",
StartDeploymentOutcome.NoChanges => "No changes detected since the last sealed deployment.",
_ => result.Message ?? "Deployment rejected.",
@@ -38,7 +38,7 @@ else
<span class="mono">@s.ScriptId</span>
&middot; <span>@s.Name</span>
&middot; <span class="chip chip-idle ms-1">@s.Language</span>
<span class="text-muted small ms-2 mono">hash=@s.SourceHash[..12]…</span>
<span class="text-muted small ms-2 mono">hash=@DisplayText.Abbreviate(s.SourceHash, 12)</span>
</summary>
<div style="padding:0 1rem 1rem">
<div class="d-flex mb-2">
@@ -0,0 +1,45 @@
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Components.Shared;
/// <summary>
/// Rendering helpers for operator-facing text that comes out of the database.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why this exists (#504).</b> Several pages abbreviated a hash / thumbprint with the range
/// operator — <c>@s.SourceHash[..12]…</c> — which throws
/// <see cref="ArgumentOutOfRangeException"/> the moment the value is shorter than the slice.
/// In Blazor that escapes <c>BuildRenderTree</c> unhandled and takes the <b>whole page</b> to an
/// HTTP 500, not just the offending row. Nothing enforces a minimum length at the schema, the
/// entity, or the render layer, so any hand-written / seeded / migrated row is a live page-kill.
/// The docker-dev seed proved it: its placeholder <c>SourceHash</c> values (<c>h1</c>,
/// <c>h-abs-hr200</c>) made <c>/scripts</c> a hard 500 on every stock bring-up.
/// </para>
/// <para>
/// Use <see cref="Abbreviate"/> for <b>every</b> such prefix. It never throws, and it appends the
/// ellipsis only when it actually truncated — a 2-character hash renders as <c>h1</c>, not
/// <c>h1…</c>, so the display never implies more text than exists.
/// </para>
/// </remarks>
public static class DisplayText
{
/// <summary>Placeholder rendered for a null / blank value. Matches the em-dash convention the
/// Admin UI already uses for "nothing to show" cells.</summary>
public const string Empty = "—";
/// <summary>
/// The first <paramref name="maxLength"/> characters of <paramref name="value"/>, followed by an
/// ellipsis when (and only when) characters were dropped.
/// </summary>
/// <param name="value">The value to abbreviate. May be null, blank, or shorter than
/// <paramref name="maxLength"/> — none of which throw.</param>
/// <param name="maxLength">Maximum number of characters to keep. Values below 1 are treated as 1,
/// so a caller cannot turn a display bug into a crash.</param>
/// <returns><see cref="Empty"/> for a null/blank value; the value verbatim when it already fits;
/// otherwise the truncated prefix plus an ellipsis.</returns>
public static string Abbreviate(string? value, int maxLength)
{
if (string.IsNullOrWhiteSpace(value)) return Empty;
if (maxLength < 1) maxLength = 1;
return value.Length <= maxLength ? value : string.Concat(value.AsSpan(0, maxLength), "…");
}
}
@@ -170,6 +170,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// <summary>Interval between bounded post-connect re-discovery passes. Production default 2s; tests
/// inject a tiny value so the loop runs without real-time waits.</summary>
private readonly TimeSpan _rediscoverInterval;
private readonly TimeSpan _healthPollInterval;
/// <summary>Cap on the number of post-connect re-discovery passes — a backstop so a never-stabilising
/// (or perpetually-empty) discovered set cannot spin the loop forever. Production default 15.</summary>
@@ -239,6 +240,9 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// <param name="rediscoverDiscoverTimeout">Optional per-pass timeout for <see cref="ITagDiscovery.DiscoverAsync"/>; defaults to 30 seconds.</param>
/// <param name="invoker">Optional Phase 6.1 resilience invoker wrapping this driver's capability
/// calls; defaults to <see cref="NullDriverCapabilityInvoker"/> (pass-through) when not supplied.</param>
/// <param name="healthPollInterval">Optional period of the health-poll heartbeat, which also drives the
/// subscription reconcile (<see cref="ReconcileSubscription"/>); defaults to
/// <see cref="HealthPollInterval"/>. Exists so a test can drive the reconcile without waiting 30 s.</param>
/// <returns>A <see cref="Akka.Actor.Props"/> instance configured to create the wrapped <see cref="DriverInstanceActor"/>.</returns>
public static Props Props(
IDriver driver,
@@ -249,7 +253,8 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
TimeSpan? rediscoverInterval = null,
int rediscoverMaxAttempts = DefaultRediscoverMaxAttempts,
TimeSpan? rediscoverDiscoverTimeout = null,
IDriverCapabilityInvoker? invoker = null) =>
IDriverCapabilityInvoker? invoker = null,
TimeSpan? healthPollInterval = null) =>
Akka.Actor.Props.Create(() => new DriverInstanceActor(
driver,
reconnectInterval ?? DefaultReconnectInterval,
@@ -259,7 +264,8 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
rediscoverInterval,
rediscoverMaxAttempts,
rediscoverDiscoverTimeout,
invoker));
invoker,
healthPollInterval));
/// <summary>
/// Returns true when the driver should boot in DEV-STUB mode based on host platform and
@@ -293,6 +299,8 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// <param name="rediscoverDiscoverTimeout">Per-pass timeout for <see cref="ITagDiscovery.DiscoverAsync"/>; defaults to 30 seconds.</param>
/// <param name="invoker">Phase 6.1 resilience invoker wrapping this driver's capability calls;
/// defaults to <see cref="NullDriverCapabilityInvoker"/> (pass-through) when null.</param>
/// <param name="healthPollInterval">Period of the health-poll heartbeat, which also drives the
/// subscription reconcile; defaults to <see cref="HealthPollInterval"/> when null.</param>
public DriverInstanceActor(
IDriver driver,
TimeSpan reconnectInterval,
@@ -302,7 +310,8 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
TimeSpan? rediscoverInterval = null,
int rediscoverMaxAttempts = DefaultRediscoverMaxAttempts,
TimeSpan? rediscoverDiscoverTimeout = null,
IDriverCapabilityInvoker? invoker = null)
IDriverCapabilityInvoker? invoker = null,
TimeSpan? healthPollInterval = null)
{
_driver = driver;
_invoker = invoker ?? NullDriverCapabilityInvoker.Instance;
@@ -313,6 +322,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
_rediscoverInterval = rediscoverInterval ?? DefaultRediscoverInterval;
_rediscoverMaxAttempts = rediscoverMaxAttempts;
_rediscoverDiscoverTimeout = rediscoverDiscoverTimeout ?? DefaultRediscoverDiscoverTimeout;
_healthPollInterval = healthPollInterval ?? HealthPollInterval;
OtOpcUaTelemetry.DriverInstanceLifecycle.Add(1,
new KeyValuePair<string, object?>("event", startStubbed ? "spawn_stub" : "spawn"),
new KeyValuePair<string, object?>("driver_type", driver.DriverType));
@@ -335,7 +345,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
// actor starts, before any state transition fires. Also start the periodic heartbeat so
// long-lived Healthy drivers keep their snapshot fresh for newly-joined SignalR clients.
PublishHealthSnapshot();
Timers.StartPeriodicTimer("health-poll", HealthPollTick.Instance, HealthPollInterval);
Timers.StartPeriodicTimer("health-poll", HealthPollTick.Instance, _healthPollInterval);
}
private void Stubbed()
@@ -489,7 +499,58 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
// to Self (HandleSubscribeAsync already logged the underlying cause). Swallow it so it doesn't dead-letter.
Receive<SubscriptionFailed>(msg =>
_log.Debug("DriverInstance {Id}: resubscribe reported failure: {Reason}", _driverInstanceId, msg.Reason));
Receive<HealthPollTick>(_ => PublishHealthSnapshot());
Receive<HealthPollTick>(_ =>
{
PublishHealthSnapshot();
ReconcileSubscription();
});
}
/// <summary>
/// Periodic desired-vs-actual subscription reconcile (#488), riding the existing <c>health-poll</c>
/// tick. While Connected, a driver that has a desired subscription set but no live handle is
/// re-subscribed.
/// </summary>
/// <remarks>
/// <para>
/// The desired set is otherwise only (re)applied on a <b>Connected transition</b>
/// (<see cref="ResubscribeDesired"/>) or on a deploy's <see cref="SetDesiredSubscriptions"/>.
/// Nothing re-asserted it for a subscription that was lost while the driver <i>stayed</i>
/// Connected — a failed subscribe whose retry never came, or a handle dropped down an error
/// path. That is the hole this closes.
/// </para>
/// <para>
/// <b>This is a state-machine consistency check, not a probe.</b> It can only see that this
/// actor holds no handle; it cannot detect a handle that is present but dead server-side,
/// because <c>ISubscriptionHandle</c> is opaque (a <c>DiagnosticId</c> string and nothing
/// else). Detecting that needs a liveness member on <c>ISubscribable</c> implemented across
/// all eight drivers, each with different subscription mechanics — deliberately deferred
/// until a real occurrence shows the handle outliving the subscription.
/// </para>
/// <para>
/// Gated on <see cref="ISubscribable"/>: a driver that cannot subscribe at all may still have
/// been handed a desired set, and re-telling <c>Subscribe</c> to it every tick would fail
/// every tick, forever.
/// </para>
/// <para>
/// A redundant <c>Subscribe</c> is possible but harmless: a tick sitting in the mailbox ahead
/// of an already-queued <c>Subscribe</c> observes the same "no handle" state and tells a
/// second one. <see cref="HandleSubscribeAsync"/> is a <c>ReceiveAsync</c> (so no tick can be
/// processed mid-subscribe) and drops any prior subscription before establishing the new one,
/// so the duplicate costs one extra round-trip and converges. Suppressing it would need a
/// pending-subscribe flag threaded through every self-tell site — more state than the race
/// is worth.
/// </para>
/// </remarks>
private void ReconcileSubscription()
{
if (_driver is not ISubscribable) return;
if (_desiredRefs.Count == 0 || _subscriptionHandle is not null) return;
_log.Info(
"DriverInstance {Id}: connected with {Count} desired subscription ref(s) but no live subscription handle — re-subscribing (periodic reconcile)",
_driverInstanceId, _desiredRefs.Count);
Self.Tell(new Subscribe(_desiredRefs, _desiredInterval));
}
private void Reconnecting()
@@ -0,0 +1,132 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Configuration.Validation;
namespace ZB.MOM.WW.OtOpcUa.Configuration.Tests;
/// <summary>
/// <see cref="SqlCredentialGuard"/> — the shared "a Sql driver never persists a literal connection
/// string" check behind both the #498 deploy gate and the #499 node-override save gate.
/// </summary>
/// <remarks>
/// The node-override half is the one #499 is about: <c>ClusterNode.DriverConfigOverridesJson</c> is a
/// map keyed by <c>DriverInstanceId</c> that is merged onto the cluster-level <c>DriverConfig</c>, and
/// it is invisible to <c>DraftSnapshot</c> — so nothing gated it at all before.
/// </remarks>
[Trait("Category", "Unit")]
public sealed class SqlCredentialGuardTests
{
/// <summary>The literal an operator would paste into a raw-JSON textarea.</summary>
private const string Leaked =
"""{"provider":"SqlServer","connectionString":"Server=sql,1433;Database=Mes;User ID=sa;Password=hunter2"}""";
private const string Clean = """{"provider":"SqlServer","connectionStringRef":"Sql:Line3"}""";
// ---- CarriesLiteralConnectionString --------------------------------------------------------
/// <summary>The key is caught at the top level of a config blob.</summary>
[Fact]
public void A_literal_connectionString_is_detected()
=> SqlCredentialGuard.CarriesLiteralConnectionString(Leaked).ShouldBeTrue();
/// <summary>The supported indirect form is not a violation — otherwise the rule would block the fix
/// it tells operators to apply.</summary>
[Fact]
public void The_indirect_connectionStringRef_form_is_clean()
=> SqlCredentialGuard.CarriesLiteralConnectionString(Clean).ShouldBeFalse();
/// <summary>System.Text.Json binds <c>ConnectionString</c> to a <c>connectionString</c> property by
/// default, so a case variant is the same key — not a bypass.</summary>
[Theory]
[InlineData("""{"ConnectionString":"Server=x;Password=p"}""")]
[InlineData("""{"CONNECTIONSTRING":"Server=x;Password=p"}""")]
[InlineData("""{"connectionstring":"Server=x;Password=p"}""")]
public void Case_variants_are_the_same_key(string json)
=> SqlCredentialGuard.CarriesLiteralConnectionString(json).ShouldBeTrue();
/// <summary>Blank, malformed and non-object blobs simply have no keys. Shaping the config JSON is
/// another rule's job, and throwing here would turn a formatting mistake into a credential failure.
/// </summary>
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
[InlineData("{ not json")]
[InlineData("[1,2,3]")]
[InlineData("\"a string\"")]
public void Blank_malformed_and_non_object_blobs_are_not_violations(string? json)
=> SqlCredentialGuard.CarriesLiteralConnectionString(json).ShouldBeFalse();
/// <summary>Only the top level is scanned. The DTO is flat, so a nested occurrence cannot bind and is
/// not the credential-shaped mistake this rule exists to catch.</summary>
[Fact]
public void A_nested_occurrence_is_not_flagged()
=> SqlCredentialGuard
.CarriesLiteralConnectionString("""{"nested":{"connectionString":"Server=x"}}""")
.ShouldBeFalse();
// ---- FindNodeOverrideViolations (#499) -----------------------------------------------------
/// <summary>The #499 leak: a credential pasted into a node's per-driver override map.</summary>
[Fact]
public void A_credential_in_a_node_override_for_a_Sql_driver_is_a_violation()
{
var overrides = $$"""{"di-sql": {{Leaked}} }""";
SqlCredentialGuard.FindNodeOverrideViolations(overrides, ["di-sql"])
.ShouldBe(["di-sql"]);
}
/// <summary>Every offending instance is reported, not just the first — an operator fixing one at a
/// time would otherwise need as many save attempts as there are leaks.</summary>
[Fact]
public void Every_offending_instance_is_reported()
{
var overrides = $$"""{"di-a": {{Leaked}}, "di-clean": {{Clean}}, "di-b": {{Leaked}} }""";
SqlCredentialGuard.FindNodeOverrideViolations(overrides, ["di-a", "di-b", "di-clean"])
.ShouldBe(["di-a", "di-b"]);
}
/// <summary>Keys outside the Sql set are ignored. A non-Sql driver never made the
/// indirect-credential guarantee, so flagging it would break a config that is legitimate today —
/// a regression, not defence in depth.</summary>
[Fact]
public void An_override_for_a_non_Sql_driver_is_ignored()
{
var overrides = $$"""{"di-modbus": {{Leaked}} }""";
SqlCredentialGuard.FindNodeOverrideViolations(overrides, ["di-sql"]).ShouldBeEmpty();
}
/// <summary>The realistic override — a node-specific endpoint — must keep saving.</summary>
[Fact]
public void A_normal_override_is_clean()
{
var overrides = """{"di-sql": {"commandTimeoutSeconds": 45}}""";
SqlCredentialGuard.FindNodeOverrideViolations(overrides, ["di-sql"]).ShouldBeEmpty();
}
/// <summary>Nothing to check when the cluster has no Sql drivers, or the node has no overrides.</summary>
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
[InlineData("{ not json")]
[InlineData("[1,2,3]")]
public void Blank_and_malformed_override_maps_yield_no_violations(string? overrides)
=> SqlCredentialGuard.FindNodeOverrideViolations(overrides, ["di-sql"]).ShouldBeEmpty();
/// <summary>An empty Sql-driver set short-circuits — there is no instance the key could belong to.</summary>
[Fact]
public void No_Sql_drivers_means_no_violations()
=> SqlCredentialGuard.FindNodeOverrideViolations($$"""{"di-sql": {{Leaked}} }""", [])
.ShouldBeEmpty();
/// <summary>A non-object override entry cannot carry config keys, and must not throw.</summary>
[Fact]
public void A_non_object_override_entry_is_skipped()
=> SqlCredentialGuard.FindNodeOverrideViolations("""{"di-sql": "connectionString"}""", ["di-sql"])
.ShouldBeEmpty();
}
@@ -10,59 +10,109 @@ using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests.Runtime;
/// <summary>
/// PR 6.2 — pins the EventPump's bounded-channel + drop-newest behavior. We
/// hold the dispatch loop with a slow handler so the channel fills, then verify
/// the producer keeps reading from the gw stream and increments the
/// <c>galaxy.events.dropped</c> counter rather than blocking.
/// PR 6.2 — pins the EventPump's bounded-channel + drop-newest behavior. We hold the dispatch loop
/// with a blocking handler so the channel fills, then verify the producer keeps reading from the gw
/// stream and increments the <c>galaxy.events.dropped</c> counter rather than blocking.
/// </summary>
/// <remarks>
/// These tests observe two background loops (<c>RunAsync</c> / <c>DispatchLoopAsync</c>) through a
/// process-wide <see cref="Meter"/>, so every wait here is a <b>presence</b> assertion polled up to
/// <see cref="Budget"/> — never a fixed sleep. A fixed sleep is a bet that both loops get scheduled
/// within it, and #503 is what losing that bet looks like when the box is busy running other test
/// assemblies.
/// </remarks>
public sealed class EventPumpBoundedChannelTests
{
/// <summary>Verifies that the event pump drops newest events when the bounded channel fills and records metrics for dropped events.</summary>
/// <summary>Upper bound on how long a poll waits for a background loop to reach its terminal state.
/// Generous on purpose: it is spent only when something is genuinely broken, and it cannot turn a
/// failing assertion into a passing one.</summary>
private static readonly TimeSpan Budget = TimeSpan.FromSeconds(15);
/// <summary>
/// With the dispatch loop genuinely held and the channel therefore full, the producer keeps
/// draining the gw stream and counts every rejected write on <c>galaxy.events.dropped</c> —
/// newest-dropped, not back-pressure.
/// </summary>
/// <remarks>
/// <para>
/// The arrangement is staged so the arithmetic is exact rather than scheduler-dependent
/// (#503): emit ONE event, wait until the dispatcher has actually entered the handler (so it
/// holds that event and the channel is empty), and only then emit the remaining nine. The
/// channel accepts <c>capacity</c> of them and the rest must be dropped —
/// <c>10 1 held 2 buffered = 7</c>, every run.
/// </para>
/// </remarks>
[Fact]
public async Task Drops_newest_when_channel_fills_and_records_metric()
{
var counters = StartMeterCapture();
const int totalEvents = 10;
const int channelCapacity = 2;
// One event is held inside the handler; `channelCapacity` more fit in the channel; the rest are dropped.
const int expectedDropped = totalEvents - 1 - channelCapacity;
// Unique per run: the counters are filtered to this exact galaxy.client tag, which is what keeps a
// parallel sibling test's pump out of them (#503). A literal name would work today and break the
// day someone reuses it.
var clientName = $"PumpTest-{Guid.NewGuid():N}";
var counters = StartMeterCapture(clientName);
// A SYNCHRONOUS gate, deliberately. OnDataChange is EventHandler<T> — void-returning — so an
// `async (_, _) => await gate` lambda is async void: it returns to the dispatch loop at the first
// await and holds nothing. That is exactly what #503 was: the loop drained freely, drops happened
// only when the producer happened to outrun the consumer, and the assertions rode on scheduling
// luck. Blocking the loop's own thread is what makes "the dispatcher is held" true.
var dispatchGate = new ManualResetEventSlim(false);
var handlerEntered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
EventPump? pump = null;
try
{
var subscriber = new ManualSubscriber();
var registry = new SubscriptionRegistry();
registry.Register(1, [new TagBinding("Tag.A", ItemHandle: 7)]);
// Tiny channel + slow handler ⇒ producer hits FullMode.Wait → TryWrite false
// for every overflow event.
var dispatchGate = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
await using var pump = new EventPump(
subscriber, registry, channelCapacity: 2, clientName: "PumpTest");
pump.OnDataChange += async (_, _) =>
pump = new EventPump(
subscriber, registry, channelCapacity: channelCapacity, clientName: clientName);
pump.OnDataChange += (_, _) =>
{
// Block the dispatch loop until we've shoved enough events through to
// overflow the bounded channel. Consume the gate exactly once.
await dispatchGate.Task.ConfigureAwait(false);
handlerEntered.TrySetResult();
dispatchGate.Wait();
};
pump.Start();
const int totalEvents = 10;
for (var i = 0; i < totalEvents; i++)
// Stage 1 — one event, then wait for the dispatcher to take it and block. After this the
// channel is provably empty and exactly one event is in flight inside the handler.
await subscriber.EmitAsync(itemHandle: 7, value: 0);
await handlerEntered.Task.WaitAsync(Budget);
// Stage 2 — the rest. `channelCapacity` are accepted; the remainder must be dropped.
for (var i = 1; i < totalEvents; i++)
{
await subscriber.EmitAsync(itemHandle: 7, value: i);
}
// Give the producer a beat to run TryWrite for every event.
await Task.Delay(150);
// Capacity 2 + 1 in-flight in the dispatcher = 3 may have been accepted; the
// remainder should have hit the dropped counter. Don't pin exact counts —
// the scheduler can interleave; pin the invariants instead.
counters.Received.ShouldBeGreaterThanOrEqualTo(totalEvents);
counters.Dropped.ShouldBeGreaterThan(0,
"with capacity=2 and a held dispatcher we must drop at least one of 10 events");
(counters.Received).ShouldBe(counters.Dispatched + counters.Dropped + counters.InFlight,
"received = dispatched + dropped + (events still queued)");
// PRESENCE assertion on the discriminator (#500's rule): poll until the drop count reaches its
// terminal value rather than sleeping a fixed 150 ms and hoping the producer got scheduled.
// The budget is an upper bound before giving up — spending it costs nothing on the happy path,
// and no amount of it can make a genuinely broken pump pass. Dropped starts at 0, so this
// cannot be satisfied by the initial state.
await WaitUntilAsync(() => counters.Dropped == expectedDropped,
$"expected {expectedDropped} dropped events");
// Release the dispatcher so DisposeAsync can drain cleanly.
dispatchGate.TrySetResult();
counters.Received.ShouldBe(totalEvents,
"the producer must keep reading the gw stream through the overflow, not back-pressure");
counters.Dropped.ShouldBe(expectedDropped);
counters.Dispatched.ShouldBe(0,
"the dispatch loop is still blocked in the handler, so nothing has completed dispatch — " +
"if this is non-zero the handler is not actually holding the loop and the test proves nothing");
}
finally
{
// Release BEFORE disposing: DisposeAsync awaits the dispatch loop, and that loop is parked on
// this gate. Releasing it here (not at the end of the try) is what keeps a failed assertion a
// failure instead of a hang.
dispatchGate.Set();
if (pump is not null) await pump.DisposeAsync();
dispatchGate.Dispose();
counters.Dispose();
}
}
@@ -110,23 +160,21 @@ public sealed class EventPumpBoundedChannelTests
// Poll until at least one galaxy.events.received measurement tagged
// galaxy.client=Driver-X lands in the listener, rather than using a
// fixed delay that races under parallel test load on a busy box.
var deadline = DateTime.UtcNow.AddSeconds(5);
bool found = false;
while (DateTime.UtcNow < deadline)
{
listener.RecordObservableInstruments();
bool hasMatch;
lock (captured)
// Budget, not 5 s: the same contention that produced #503 stretches how long the
// producer loop takes to be scheduled, and a presence poll costs nothing to over-budget.
await WaitUntilAsync(
() =>
{
hasMatch = captured.Any(c =>
c.Instrument == "galaxy.events.received" &&
c.Tags.Any(t => t.Key == "galaxy.client" &&
string.Equals((string?)t.Value, "Driver-X", StringComparison.Ordinal)));
}
if (hasMatch) { found = true; break; }
await Task.Delay(25);
}
_ = found; // assertion happens below after dispose
listener.RecordObservableInstruments();
lock (captured)
{
return captured.Any(c =>
c.Instrument == "galaxy.events.received" &&
c.Tags.Any(t => t.Key == "galaxy.client" &&
string.Equals((string?)t.Value, "Driver-X", StringComparison.Ordinal)));
}
},
"a galaxy.events.received measurement tagged galaxy.client=Driver-X");
}
// The static Meter is shared across all EventPump instances in the test
@@ -146,7 +194,46 @@ public sealed class EventPumpBoundedChannelTests
ours.ShouldContain(c => c.Instrument == "galaxy.events.received");
}
private static CounterCapture StartMeterCapture()
/// <summary>Polls <paramref name="condition"/> until it holds or <see cref="Budget"/> elapses, failing
/// with <paramref name="what"/> on timeout. The presence-assertion counterpart to a fixed delay.</summary>
private static async Task WaitUntilAsync(Func<bool> condition, string what)
{
var deadline = DateTime.UtcNow + Budget;
while (DateTime.UtcNow < deadline)
{
if (condition()) return;
await Task.Delay(10);
}
condition().ShouldBeTrue($"timed out after {Budget.TotalSeconds:0}s waiting: {what}");
}
/// <summary>
/// Captures the three EventPump counters for <b>one specific pump</b>, identified by its
/// <c>galaxy.client</c> tag.
/// </summary>
/// <remarks>
/// <para>
/// The <c>galaxy.client</c> filter is the fix for #503, and it is not optional. The pump's
/// <see cref="Meter"/> is <b>static</b> — one instance for the whole process — and
/// <c>InstrumentPublished</c> can only select by <i>meter</i>, so an unfiltered listener
/// receives measurements from every <see cref="EventPump"/> alive anywhere in the assembly.
/// xUnit runs test classes in parallel and several of them build pumps
/// (<c>EventPumpStreamFaultTests</c>, the <c>Driver-X</c> test below), so a sibling's events
/// landed in these counters — that is why the flake needed a busy box: it needed the two
/// tests to overlap in time. The leak was measured directly, as <c>Received = 11</c> in a run
/// that emitted 10.
/// </para>
/// <para>
/// The old <c>Received == Dispatched + Dropped + InFlight</c> assertion is what turned the
/// leak into a failure: a foreign increment landing between its four separate
/// <see cref="Interlocked.Read(ref long)"/> calls made the two sides disagree. Hence the
/// reported subject, <c>counters.Received</c>.
/// </para>
/// </remarks>
/// <param name="clientName">The <c>galaxy.client</c> tag value to accept. Must be unique to the
/// calling test — pass a fresh GUID-suffixed name rather than a literal, so a future sibling test
/// cannot silently reintroduce the leak by reusing the same name.</param>
private static CounterCapture StartMeterCapture(string clientName)
{
var capture = new CounterCapture();
var listener = new MeterListener();
@@ -154,8 +241,20 @@ public sealed class EventPumpBoundedChannelTests
{
if (instr.Meter.Name == EventPump.MeterName) l.EnableMeasurementEvents(instr);
};
listener.SetMeasurementEventCallback<long>((instr, value, _, _) =>
listener.SetMeasurementEventCallback<long>((instr, value, tags, _) =>
{
var mine = false;
foreach (var tag in tags)
{
if (tag.Key == "galaxy.client" &&
string.Equals((string?)tag.Value, clientName, StringComparison.Ordinal))
{
mine = true;
break;
}
}
if (!mine) return;
switch (instr.Name)
{
case "galaxy.events.received": Interlocked.Add(ref capture._received, value); break;
@@ -178,8 +277,11 @@ public sealed class EventPumpBoundedChannelTests
public long Dispatched => Interlocked.Read(ref _dispatched);
/// <summary>Gets the count of dropped events.</summary>
public long Dropped => Interlocked.Read(ref _dropped);
/// <summary>Gets the count of in-flight events.</summary>
public long InFlight => Math.Max(0, Received - Dispatched - Dropped);
// There is deliberately no InFlight member. It could only be DERIVED as
// Received - Dispatched - Dropped, which made the old
// `Received == Dispatched + Dropped + InFlight` assertion a tautology: it could not fail on a
// real defect, only on a torn read across the four separate Interlocked.Read calls. Assert the
// three measured counters directly instead (#503).
/// <summary>Disposes the meter listener.</summary>
public void Dispose() => Listener?.Dispose();
}
@@ -0,0 +1,95 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.AdminUI.Components.Shared;
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Tests;
/// <summary>
/// #504 — <see cref="DisplayText.Abbreviate"/>, the null/short-safe replacement for the
/// <c>@value[..N]</c> range-operator slices the Admin UI used to abbreviate hashes and thumbprints.
/// </summary>
/// <remarks>
/// The bug being pinned: a range operator over an unvalidated DB string throws
/// <see cref="System.ArgumentOutOfRangeException"/> out of <c>BuildRenderTree</c>, which is unhandled
/// in Blazor and takes the WHOLE page to an HTTP 500 — not just the offending row. Nothing enforces a
/// minimum length at the schema, entity, or render layer, so the short-input cases below are the ones
/// that matter; the happy path is almost incidental.
/// </remarks>
[Trait("Category", "Unit")]
public sealed class DisplayTextTests
{
/// <summary>A value longer than the budget is truncated and marked with an ellipsis — the normal
/// case for a 64-char SHA-256.</summary>
[Fact]
public void A_long_value_is_truncated_and_marked()
{
DisplayText.Abbreviate(new string('a', 64), 12).ShouldBe(new string('a', 12) + "…");
}
/// <summary>The regression: a value SHORTER than the budget must render, not throw. This is exactly
/// what a hand-inserted / API-authored <c>SourceHash</c> like <c>h1</c> hits.</summary>
[Theory]
[InlineData("h1", 12)]
[InlineData("h-abs-hr200", 12)] // 11 chars — one short of the old slice, the original crash
[InlineData("abc", 16)]
[InlineData("x", 64)]
public void A_value_shorter_than_the_budget_renders_verbatim(string value, int maxLength)
{
DisplayText.Abbreviate(value, maxLength).ShouldBe(value);
}
/// <summary>No ellipsis when nothing was dropped — the display must never imply text that is not
/// there. (The old markup appended "…" unconditionally, outside the slice.)</summary>
[Fact]
public void A_short_value_gets_no_ellipsis()
{
DisplayText.Abbreviate("h1", 12).ShouldNotContain("…");
}
/// <summary>A value exactly at the budget is the boundary — it fits, so it is neither truncated nor
/// marked.</summary>
[Fact]
public void A_value_exactly_at_the_budget_is_untouched()
{
DisplayText.Abbreviate("123456789012", 12).ShouldBe("123456789012");
}
/// <summary>Null / blank render as the Admin UI's em-dash placeholder rather than throwing or
/// producing a stray ellipsis.</summary>
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
public void A_null_or_blank_value_renders_the_placeholder(string? value)
{
DisplayText.Abbreviate(value, 12).ShouldBe(DisplayText.Empty);
}
/// <summary>A nonsensical budget is clamped rather than allowed to throw — a display bug must not
/// become a page crash, which is the whole point of this helper.</summary>
[Theory]
[InlineData(0)]
[InlineData(-1)]
[InlineData(int.MinValue)]
public void A_non_positive_budget_is_clamped_not_thrown(int maxLength)
{
DisplayText.Abbreviate("abcdef", maxLength).ShouldBe("a…");
}
/// <summary>Whatever the inputs, it never throws — the property that keeps a bad row from killing a
/// page. Includes the shapes that broke the old code.</summary>
[Fact]
public void It_never_throws_for_any_combination()
{
string?[] values = [null, "", " ", "h1", "h-abs-hr200", new string('f', 64), "…"];
int[] budgets = [int.MinValue, -1, 0, 1, 12, 16, 64, int.MaxValue];
foreach (var value in values)
{
foreach (var budget in budgets)
{
Should.NotThrow(() => DisplayText.Abbreviate(value, budget));
}
}
}
}
@@ -86,8 +86,19 @@ public sealed class DriverHostActorNativeAlarmAckRoutingTests : RuntimeActorTest
actor.Tell(new DriverHostActor.RouteNativeAlarmAck("Plant/gw/dev1/does-not-exist", "cmt", "alice"));
// Give the (fire-and-forget) handler time to run; the unmapped node must produce no ack.
AwaitAssert(() => recorder.Acks.ShouldBeEmpty(), duration: TimeSpan.FromMilliseconds(800));
// POSITIVE CONTROL, not a sleep (#501). `Acks` is empty at t=0, so any assertion made before the
// host has processed the message above passes without testing anything — and AwaitAssert returns on
// its FIRST success, so it spends none of its duration and is the wrong tool for an absence.
//
// Instead send a MAPPED ack afterwards. The host processes its mailbox in order and forwards to the
// child via Tell, and the child handles RouteAlarmAck with ReceiveAsync (which suspends its own
// mailbox), so by the time this one reaches the driver the unmapped one has provably already been
// handled. Whatever `Acks` holds then is the complete answer.
actor.Tell(new DriverHostActor.RouteNativeAlarmAck(AlarmRawPath, "control", "alice"));
AwaitAssert(() => recorder.Acks.ShouldContain(a => a.Comment == "control"), duration: Timeout);
recorder.Acks.Count.ShouldBe(1,
"only the control ack may reach the driver — the unmapped condition NodeId must have been dropped");
}
/// <summary>On a SECONDARY node the ack is gated off (same primary gate as the inbound write path): the
@@ -110,10 +121,25 @@ public sealed class DriverHostActorNativeAlarmAckRoutingTests : RuntimeActorTest
},
CorrelationId.NewId()));
actor.Tell(new DriverHostActor.RouteNativeAlarmAck(AlarmRawPath, "cmt", "alice"));
actor.Tell(new DriverHostActor.RouteNativeAlarmAck(AlarmRawPath, "gated", "alice"));
// No ack reached the driver — the gate short-circuited before the inverse-map lookup.
AwaitAssert(() => recorder.Acks.ShouldBeEmpty(), duration: TimeSpan.FromMilliseconds(800));
// POSITIVE CONTROL (#501) — same reasoning as the unmapped-node test, with the role itself as the
// control: return the node to PRIMARY and re-send the SAME mapped ack. Mailbox ordering means that
// once this one reaches the driver, the Secondary-gated one has already been processed. The
// discriminator is the comment, so a leaked ack is visible rather than merged into the count.
actor.Tell(new RedundancyStateChanged(
new[]
{
new NodeRedundancyState(TestNode, RedundancyRole.Primary,
IsClusterLeader: true, IsDriverPrimary: true, AsOfUtc: DateTime.UtcNow),
},
CorrelationId.NewId()));
actor.Tell(new DriverHostActor.RouteNativeAlarmAck(AlarmRawPath, "control", "alice"));
AwaitAssert(() => recorder.Acks.ShouldContain(a => a.Comment == "control"), duration: Timeout);
recorder.Acks.ShouldNotContain(a => a.Comment == "gated",
"the Primary gate must have dropped the ack sent while this node was Secondary");
recorder.Acks.Count.ShouldBe(1);
}
/// <summary>Spawns the host with the recording alarm-driver factory, dispatches the deployment, and waits
@@ -0,0 +1,122 @@
using Akka.Actor;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers;
/// <summary>
/// Gitea #488 — the periodic desired-vs-actual subscription reconcile that rides
/// <see cref="DriverInstanceActor"/>'s existing <c>health-poll</c> tick.
/// </summary>
/// <remarks>
/// <para>
/// The hole it closes: the desired ref set is re-applied on a <b>Connected transition</b> and on a
/// deploy's <c>SetDesiredSubscriptions</c>, and nowhere else. A subscription lost while the driver
/// <i>stayed</i> Connected — the shape reproduced here, a subscribe that failed and was never
/// retried — left the actor permanently subscribed to nothing while reporting Healthy.
/// </para>
/// <para>
/// All three tests drive a short <c>healthPollInterval</c> rather than the production 30 s. The
/// two absence assertions are bounded by a positive control (a counted number of ticks provably
/// processed), never by a bare sleep.
/// </para>
/// </remarks>
public sealed class DriverInstanceActorSubscriptionReconcileTests : RuntimeActorTestBase
{
private static readonly TimeSpan FastPoll = TimeSpan.FromMilliseconds(100);
/// <summary>
/// The reconcile itself: the first subscribe fails, leaving the actor Connected with a desired set
/// and no handle. Nothing else would ever retry it — no reconnect, no deploy — so a second
/// subscribe can only come from the periodic reconcile.
/// </summary>
[Fact]
public void A_lost_subscription_is_re_established_while_still_connected()
{
var driver = new SubscribableStubDriver { SubscribeFailuresBeforeSuccess = 1 };
var actor = Sys.ActorOf(DriverInstanceActor.Props(driver, healthPollInterval: FastPoll));
actor.Tell(new DriverInstanceActor.SetDesiredSubscriptions(
new[] { "tag-a" }, TimeSpan.FromMilliseconds(100)));
actor.Tell(new DriverInstanceActor.InitializeRequested("{}"));
// The connect-path subscribe runs and throws, so the handle stays null.
AwaitCondition(() => driver.SubscribeCount >= 1, PresenceBudget);
// Only the reconcile can produce this second attempt — and it must succeed this time.
AwaitCondition(() => driver.SubscribeCount >= 2, PresenceBudget);
driver.LastSubscribedRefs.ShouldBe(new[] { "tag-a" });
}
/// <summary>
/// A healthy subscription is left alone — the reconcile must not re-subscribe on every tick, which
/// would churn the backend every 30 s in production for no reason.
/// </summary>
[Fact]
public void A_healthy_subscription_is_not_re_subscribed_on_every_tick()
{
var publisher = new CountingHealthPublisher();
var driver = new SubscribableStubDriver();
var actor = Sys.ActorOf(DriverInstanceActor.Props(
driver, healthPublisher: publisher, healthPollInterval: FastPoll));
actor.Tell(new DriverInstanceActor.SetDesiredSubscriptions(
new[] { "tag-a" }, TimeSpan.FromMilliseconds(100)));
actor.Tell(new DriverInstanceActor.InitializeRequested("{}"));
AwaitCondition(() => driver.SubscribeCount >= 1, PresenceBudget);
// POSITIVE CONTROL for the absence below. Every health-poll tick publishes a snapshot, so the
// publisher count is a direct observation of ticks PROCESSED. Waiting for several of them is what
// makes "SubscribeCount is still 1" mean "the reconcile ran and declined" rather than "the timer
// had not fired yet" — the latter would pass instantly and prove nothing.
var before = publisher.Count;
AwaitCondition(() => publisher.Count >= before + 4, PresenceBudget);
driver.SubscribeCount.ShouldBe(1,
"the handle is live, so the reconcile must leave it alone");
}
/// <summary>
/// A driver that is not <see cref="ISubscribable"/> can still be handed a desired set. Reconciling
/// it would tell <c>Subscribe</c> every tick and fail every tick, forever — so the reconcile is
/// gated on the capability.
/// </summary>
[Fact]
public void A_non_subscribable_driver_is_never_reconciled()
{
var publisher = new CountingHealthPublisher();
var driver = new StubDriver(); // IDriver only — no ISubscribable
var actor = Sys.ActorOf(DriverInstanceActor.Props(
driver, healthPublisher: publisher, healthPollInterval: FastPoll));
actor.Tell(new DriverInstanceActor.SetDesiredSubscriptions(
new[] { "tag-a" }, TimeSpan.FromMilliseconds(100)));
actor.Tell(new DriverInstanceActor.InitializeRequested("{}"));
// Same positive control: assert the reconcile stays silent across several PROCESSED ticks.
// EventFilter bounds the absence to this block, and the reconcile announces itself at Info —
// the first test above is the proof that this message does appear when the reconcile fires.
EventFilter.Info(contains: "periodic reconcile").Expect(0, () =>
{
var before = publisher.Count;
AwaitCondition(() => publisher.Count >= before + 4, PresenceBudget);
});
}
/// <summary>Counts health-snapshot publishes, which is one per <c>health-poll</c> tick once the actor
/// is running — the observable that turns "time passed" into "ticks were processed".</summary>
private sealed class CountingHealthPublisher : IDriverHealthPublisher
{
private int _count;
/// <summary>Number of snapshots published so far.</summary>
public int Count => Volatile.Read(ref _count);
/// <inheritdoc />
public void Publish(string clusterId, string driverInstanceId, DriverHealth health, int errorCount5Min)
=> Interlocked.Increment(ref _count);
}
}
@@ -114,6 +114,18 @@ internal sealed class SubscribableStubDriver : StubDriver, ISubscribable
/// synchronously-completed stub task hides (the continuation otherwise runs inline).</summary>
public bool UnsubscribeYields { get; set; }
/// <summary>
/// How many of the first <see cref="SubscribeAsync"/> calls throw before one succeeds. Default 0
/// (always succeed) leaves existing behaviour untouched.
/// </summary>
/// <remarks>
/// This is how a test reaches the state the #488 reconcile exists for: the actor is Connected and
/// holds a desired ref set, but the subscribe that should have established the handle failed, so
/// <c>_subscriptionHandle</c> is null and nothing else will ever re-assert it — there is no
/// connect transition coming, and no deploy.
/// </remarks>
public int SubscribeFailuresBeforeSuccess { get; set; }
/// <summary>Subscribes to the specified full references.</summary>
/// <param name="fullReferences">The full references to subscribe to.</param>
/// <param name="publishingInterval">The publishing interval.</param>
@@ -121,8 +133,13 @@ internal sealed class SubscribableStubDriver : StubDriver, ISubscribable
public Task<ISubscriptionHandle> SubscribeAsync(
IReadOnlyList<string> fullReferences, TimeSpan publishingInterval, CancellationToken cancellationToken)
{
Interlocked.Increment(ref SubscribeCount);
var attempt = Interlocked.Increment(ref SubscribeCount);
LastSubscribedRefs = fullReferences;
if (attempt <= SubscribeFailuresBeforeSuccess)
{
throw new InvalidOperationException($"stub subscribe failure #{attempt}");
}
return Task.FromResult<ISubscriptionHandle>(_handle);
}
@@ -78,7 +78,9 @@ public sealed class PeerProbeSupervisorTests : RuntimeActorTestBase
duration: PresenceBudget);
}
/// <summary>Verifies a single-node snapshot (just the local node) spawns no children.</summary>
/// <summary>Verifies a single-node snapshot (just the local node) spawns no children — the local node
/// is never its own probe peer. Established against a following two-node snapshot as a positive
/// control, so the count of 1 (rather than 2) is what carries the "no local child" claim.</summary>
[Fact]
public void Single_node_snapshot_spawns_no_children()
{
@@ -87,7 +89,20 @@ public sealed class PeerProbeSupervisorTests : RuntimeActorTestBase
sup.Tell(Snapshot(State(Local, RedundancyRole.Primary)));
AwaitAssert(() => sup.UnderlyingActor.ChildCount.ShouldBe(0),
// POSITIVE CONTROL (#501). This is an ABSENCE assertion, and ChildCount is 0 before the snapshot
// above has been processed at all — so asserting it directly passes at t=0 and proves nothing
// (AwaitAssert returns on its first success, spending none of its duration). Unlike the other
// ChildCount.ShouldBe(0) waits in this class, there is no preceding ShouldBe(1) to make it a
// transition.
//
// Follow with a snapshot that DOES spawn exactly one child. The supervisor processes its mailbox
// in order, so once the count reaches 1 the single-node snapshot has provably been handled — and
// had it spawned a child for the local node, the count would be 2.
sup.Tell(Snapshot(
State(Local, RedundancyRole.Primary),
State(Peer, RedundancyRole.Secondary)));
AwaitAssert(() => sup.UnderlyingActor.ChildCount.ShouldBe(1),
duration: PresenceBudget);
}