dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: dohertj2/ScadaBridge:15e5887b52ebb5dabfb073cb11dec1ea9eec4408
Branches Tags
dohertj2/ScadaBridge:main
...
compare: dohertj2/ScadaBridge:4da5f7cd7a4a212b910785d6b84bb8df9c8b3b3c
Branches Tags
dohertj2/ScadaBridge:main

20 Commits
15e5887b52 ... 4da5f7cd7a

Author SHA1 Message Date
Joseph Doherty 4da5f7cd7a docs(register): close row 101 — stale-snapshot race verified real (worse than guessed: spurious removal is NOT self-healing), definitions-write generation guard shipped in 0d3a90a9 2026-08-17 07:04:40 -04:00
Joseph Doherty 0d3a90a9fd fix(siteruntime): guard the DCL arbitration overlay against stale store snapshots — register row 101 
An off-thread snapshot read racing an artifact apply could land after the
apply populated the overlay; its stale absences then tombstoned and stopped
just-applied LIVE connections, which nothing re-creates until an instance
redeploy (the removal evicts the hash slot the overlay push is gated on).
Each read now captures a definitions-write generation at dispatch; a stale
result is discarded whole (additions too — a stale row would reconfigure a
live connection backward). Deterministic repro fails before the fix; a
fresh-generation guard test pins that row-90 disappearance still fires.
 2026-08-17 07:03:26 -04:00
Joseph Doherty 63d7f273e6 docs(register): Wave 6 close-out — close rows 97 (in-place DCL reconfigure, live-probed round trip), 98 (fail-closed scope gate, 5 sites not 2 — id-enumeration oracle closed), 99 (strict audit filters, 7 gates + lax-server premise corrected, live-probed), 100 (compile cache instance-scoped, Clear() deleted — zero production callers); row-88 postscript (ordinal guard); add row 101 (overlay-sweep race hypothesis) 2026-08-17 06:44:41 -04:00
Joseph Doherty 8c500d8170 test(sitecallaudit): await the upsert assertions in SingleTimestampSaturation — gating on CallCount alone races the async repo write under parallel suite load (one full-suite failure observed 2026-08-17; 3/3 green in isolation; row-95 fingerprint) 2026-08-17 06:40:33 -04:00
Joseph Doherty 16056f8eb6 fix(management): ParseDeploymentStatus rejects ordinals — Enum.TryParse accepts "2" as InProgress despite the row-88 names-only contract; same LooksNumeric guard as the audit strict parser (found by the Wave 6 rows-98/99 agent survey) 2026-08-17 06:21:12 -04:00
Joseph Doherty 8a9a8d2951 merge(agent-c): row 100 — SiteScriptCompileCache becomes an instance-scoped sealed class owned by the ScriptCompilationService DI singleton (zero production Clear() callers existed, so Clear() is deleted outright); row-91 serialized collection relaxed and delta assertions restored to exact counts; new registration test pins the load-bearing singleton lifetime 2026-08-17 06:20:17 -04:00
Joseph Doherty 152ec85e92 merge(agent-b): rows 98/99 — site-scope gate fail-closed on unresolvable targets (5 sites, not 2: the inline GetInstance/GetSite/GetDataConnection shapes were an id-enumeration oracle; all route through DenyUnresolvedTargetIfScoped, denial indistinguishable from out-of-scope); CLI audit query/export drop 7 case-sensitive AcceptOnlyFromAmong gates for help-text values + a new strict server-side ParseEnumListStrict (no silent drop, no ordinals) 2026-08-17 06:20:14 -04:00
Joseph Doherty 9dbda4a2f2 merge(agent-a): row 97 — live DCL connections reconfigure in place: ReconfigureConnectionCommand handled in all three DataConnectionActor states via a shared SwapAdapter (also unifies both failover swap sites), endpoint selection re-armed to Primary and unstable-disconnect accounting suppressed; EnsureDclConnection's change edge now sends the reconfigure instead of a dead-end Warning 2026-08-17 06:20:09 -04:00
Joseph Doherty 7c1268d85c feat(dcl+siteruntime): reconfigure a LIVE data connection in place — register row 97 
`DataConnectionManagerActor.HandleCreateConnection` is create-only (warns and
returns for a name it already holds), so `EnsureDclConnection`'s documented
"a changed config re-issues the create so the DCL adopts it" was never true on
ANY node: a corrected endpoint/credential/backup/failover-count took effect only
when the connection was next built from scratch (node restart, singleton
failover, remove-then-re-add), while the DM's hash cache recorded it as applied.
Rows 89/90 made that honest with a Warning; this closes it.

New `ReconfigureConnectionCommand` (Commons, same shape as the create):

- `DeploymentManagerActor.EnsureDclConnection` now has three outcomes off its one
  per-name hash slot — unchanged -> nothing at all, new -> create, CHANGED ->
  reconfigure — and logs "Reconfigured DCL connection {Connection}
  (protocol={Protocol}) in place" at Information in place of the old Warning. The
  winner-resolution / arbitration / tombstone logic is untouched; only what
  happens after a change is detected changed.
- `DataConnectionManagerActor` FORWARDS the command to the owning connection
  actor, falling back to a create for a name it does not hold, so the command is
  safe whatever the manager's state.
- `DataConnectionActor` handles it in ALL THREE states and never stashes it — a
  connection pointed at a wrong or dead endpoint lives in Connecting/Reconnecting,
  which is exactly when a correction arrives; stashed it would wait on the broken
  configuration forever. Handling replaces protocol/primary/backup/failover-count,
  re-arms endpoint selection to Primary (the new config's primary is the intent,
  whatever the old config had failed over to), and calls the new shared
  `SwapAdapter` helper — now used by both failover sites too, so the swap exists
  once: unhook + fire-and-forget dispose -> _factory.Create -> rewire -> bump
  _adapterGeneration LAST. `BecomeReconnecting(afterReconfigure: true)` then
  reconnects immediately (StartSingleTimer replaces any armed reconnect, so no
  racing attempts) with the unstable-disconnect accounting and failover check
  suppressed — the link was dropped on purpose, and counting it as an unstable
  cycle could trip a failover onto the NEW backup before the new primary had been
  tried once. ReSubscribeAll/ReSubscribeAllAlarms then re-issue everything from
  the untouched subscription registries. Idempotent: a reconfigure carrying the
  configuration already in force is a no-op.

Stop-and-recreate was rejected as destructive: subscriptions live inside
DataConnectionActor and InstanceActor cancels its subscribe-retry timer on first
success, so a rebuilt actor would have no subscribers and nothing would ever
re-subscribe.

Tests: new DataConnectionReconfigureTests (7; 5 fail / 2 pass against a
behavioural revert). DclConnectionArbitrationTests gains two row-97 tests and its
CDC-arrival test now asserts the second push is a reconfigure; its ping-pong
guard also asserts zero reconfigures, since a flip would now show up there.
SiteRuntime-010 renamed to ..._ReissuesReconfigureCommand. 3 fail against a
reverted EnsureDclConnection. DataConnectionLayer.Tests 327/327,
SiteRuntime.Tests 643/643, solution build 0 warnings.

Docs: Component-DataConnectionLayer.md gains an "In-place reconfiguration"
section under Endpoint Redundancy; Component-SiteRuntime.md, and the
docs/components/ mirrors, updated; the arbitration decision memo gains an
appended §9 postscript resolving its residual 1 (residuals 2 and 3 stay open).
 2026-08-17 06:19:13 -04:00
Joseph Doherty f33f25a10b fix(auth+cli): fail-closed site-scope gate (row 98); case-insensitive audit filter values with a strict server-side check (row 99) 
Register row 98 — the site-scope authorization gate was FAIL-OPEN on an
unresolvable target. The resolve-then-check shape was written as
`if (target != null) EnforceSiteScope(...)`, so a null resolve skipped the
check entirely and a site-scoped caller passed the gate for any id in the
system. Authorization now fails CLOSED: a target that cannot be PROVEN to sit
inside the caller's scope is treated as outside it.

The defect turned out to have THREE more instances than the register recorded.
Besides `EnforceSiteScopeForInstance` and `EnforceSiteScopeForIdentifier`, the
same inline shape sat in `HandleGetInstance`, `HandleGetSite` and
`HandleGetDataConnection` — read handlers that resolve their own target and so
never call the helpers. Those were the sharper half of the bug: they answered
success-with-null for a nonexistent id and Unauthorized for an out-of-scope
one, making the pair an id-enumeration oracle over the whole fleet. All five
now route through one shared `DenyUnresolvedTargetIfScoped` arm.

Two properties are load-bearing and pinned by tests:
  - System-wide callers (Administrator, or a Deployer with no site grants) are
    untouched — they return at the guards above the lookup and keep the
    handler's ordinary "not found" / null answer.
  - The denial is indistinguishable from an out-of-scope denial: same
    ManagementUnauthorized shape, and a message that never states whether the
    target exists.

Register row 99 — CLI `audit query --status delivered` failed client-side while
the server would have accepted it: `AcceptOnlyFromAmong` matches ordinally
against the capitalised literals. Same treatment row 88 gave `deploy status` —
the client-side gate is removed, the valid set moves into the option help text,
and the server becomes the single authoritative check.

That required ADDING the server check, not just relying on one: the shared
audit filter parser was LAX, silently dropping an unrecognised value and
leaving the dimension unconstrained — so removing the CLI gate alone would have
turned a loud client-side failure into a plausible, complete-looking page that
did not answer the question asked. New `ParseEnumListStrict` (case-insensitive,
rejects ordinals — `Enum.TryParse` accepts "3", and a filter is a name-based
contract) backs the ManagementService /api/audit/query + /api/audit/export
endpoints, which now return 400 BAD_REQUEST naming the valid set.

Other AcceptOnlyFromAmong sites surveyed in the audit group: `--channel`,
`--kind`, `--status` on BOTH `query` and `export` (6 uses) plus `export
--format` (rejected `--format CSV` though the server lowercases it) — all seven
fixed here. Deliberately left alone: `template folder reorder --direction`
(pure client-side literal, no server-side parse to defer to) and the global
`--format json|table` (CLI-local output selection, never sent to a server).

The CentralUI AuditLogPage drill-in parser and the CentralUI export endpoint it
feeds deliberately KEEP the lax parse — they read a query string a human may
have bookmarked, where a stale fragment must still render the page. Both
contracts now live side by side in AuditQueryParamParsers and are documented as
non-reconcilable.

Docs: Component-ManagementService.md gains a "Site-scope gate: fail-closed on
an unresolvable target" section and a strict-filter note on the audit
endpoints; Component-Security.md records the fail-closed shape under Deployer
site scoping; the CLI README marks the audit filter values case-insensitive and
states the no-silent-drop / no-ordinals contract.

Tests: 5 new ManagementActorTests fail before the row-98 fix (verified by
reverting ManagementActor.cs) and cover scoped+nonexistent -> denied,
system-wide+nonexistent -> unchanged (both Administrator and no-grant Deployer),
scoped+in-scope -> allowed, and the indistinguishability property. 21 CLI tests
fail before the row-99 fix. Four pre-existing tests that pinned the removed
client-side gate were rewritten to pin the new contract.
 2026-08-17 06:05:29 -04:00
Joseph Doherty 84b405ff5f refactor(siteruntime): scope SiteScriptCompileCache to the ScriptCompilationService singleton — close register row 100 
The compile cache was a `static` class, so its entry table and its Hits/Count
counters were one process-global blob every test class in the SiteRuntime
assembly wrote into. Row 91 last wave neutered the resulting flakes (per-test
GUID keys + one-sided counter bounds), but the shared mutable state itself
remained: any test calling Clear() evicted entries other parallel classes had
just inserted, and a Clear() from a class outside the serialized collection
would degrade the others to recompiles and re-open the row-91 flake class.

PREMISE CORRECTION: the register speculated production clears "per-instance or
per-deployment script entries" and offered ClearKeys(prefix) as the natural fit.
There are NO production callers of Clear() at all — grep of src/ finds only
GetOrAdd (ScriptCompilationService.CompileCore). Clear() existed solely so
tests could reset the global. A key-scoped API would therefore have been new
production surface built for nobody, so the instance-scoped seam is the fix and
Clear() is deleted outright rather than migrated.

- SiteScriptCompileCache: `internal static class` -> `internal sealed class`;
  Cache/EvictionLock/_hits/_accessSequence become instance fields. LRU batch
  eviction, the double-checked sweep and the ConcurrentDictionary.ToArray()
  snapshot are byte-for-byte the same policy. Clear() removed.
- ScriptCompilationService owns one cache. Public ctor (the DI path) builds its
  own; a new internal ctor takes one for tests. Production N4 sharing is
  unchanged and now rests on the DI SINGLETON registration — the one service
  instance is threaded to DeployCompileValidator and to every Instance Actor —
  so that lifetime is load-bearing and is pinned by a new registration test.
- Tests: all three Clear() callers migrated off it. Each test now owns its cache,
  which lets the assertions get STRONGER — the row-91 workarounds are retired:
  Hits/Count are exact again (no >= baseline bounds), the per-test GUID key
  token is gone, and ForeignEntrySlack (512 entries of allowance on the bound)
  drops to the racing-thread count in the concurrency test. New test
  SeparateCaches_DoNotShareEntriesOrCounters pins the scoping directly: filling
  one cache past MaxEntries leaves another's entry, hit count and identity
  intact. DeploymentWarmThenGateTests additionally asserts Count == 1 now that
  its counter is private.
- The [Collection("SiteScriptCompileCache")] serialization from row 91 is
  RELAXED (removed from all three classes) — they no longer touch any shared
  state, so there is nothing left to serialize.
- Docs: Component-SiteRuntime.md gains the ownership/singleton note and drops
  the Clear() reference; DeploymentManagerActor / DeployCompileValidator /
  ServiceCollectionExtensions comments corrected to say where the cache lives.

Build: 0 warnings. SiteRuntime.Tests 643/643, three consecutive full-parallel
runs. AuditLog.Tests correlation classes (other ScriptCompilationService
consumers) green.
 2026-08-17 06:03:58 -04:00
Joseph Doherty dcfcf822af docs(register): Wave 5 close-out — close rows 37/86/87/88 (deployment-read consolidation + Administrator system-wide fix, live-probed), 89/90 (DCL definitions-overlay arbitration, premise corrected twice, live-probed), 91/95/96 (flake family + parallelism policy); add rows 97-100 (live-connection reconfigure gap, fail-open scope gate, case-sensitive audit --status, process-global compile-cache Clear) 2026-08-17 05:41:56 -04:00
Joseph Doherty 106e467dfd merge(agent-c): rows 91/95/96 — compile-cache tests on key-scoped delta assertions; Watch/Stop/ExpectTerminated at all 23 fast-tick actor creation sites in the three remaining AuditLog.Tests classes; per-suite xunit maxParallelThreads:4 for the four MSSQL-fixture suites (matches the row-82 semaphore slots) + Test Parallelism Policy doc 2026-08-17 05:28:51 -04:00
Joseph Doherty ad0100f2f9 merge(agent-a): rows 37/86/87/88 — deployment reads consolidated on QueryDeploymentListPageAsync (QueryDeploymentSummariesAsync deleted); SiteScopeService honors the Administrator system-wide rule (claims no longer scope admins — the documented ManagementActor semantic now holds on both surfaces); site scope rides the query join (whole-table instance read gone), PageSize bounded (default 50, cap 1000); unparseable --status fails loudly naming valid values 2026-08-17 05:28:51 -04:00
Joseph Doherty 6cfb6a7970 merge(agent-b): rows 89/90 — data_connection_definitions overlay as DCL arbitration source (definitions win where present, flattened snapshot otherwise; ping-pong killed by winning-config hashing; tombstoned removals survive instance starts; premise corrected — standby DCL is empty, loss was create-time on any restart/failover) 2026-08-17 05:28:43 -04:00
Joseph Doherty d2d4db492b test(infra): close register row 96 — per-suite xunit parallelism caps for the resource-heavy suites 
maxParallelThreads was unset everywhere and only 3 of ~20 test projects carried
an xunit.runner.json at all, so xunit ran Environment.ProcessorCount collections
at once (14 on the dev box) in every suite. That is the general load multiplier
behind the row-82/91/95 flake family.

Policy: per-suite caps, NOT a global one. Pure unit suites stay uncapped for
speed; a suite gets a config only when a test CLASS costs a real external or
OS-level resource, so ProcessorCount of them at once is a resource question
rather than a CPU question. Four suites qualify, all capped at
maxParallelThreads: 4.

- ConfigurationDatabase.Tests, AuditLog.Tests, SiteCallAudit.Tests — verified as
  the three that actually take MsSqlMigrationFixture (the latter two by project
  reference). Every MSSQL-backed test class means a CREATE DATABASE plus the
  ~100-migration chain against the one shared dev SQL Server. 4 is chosen to
  MATCH the default slot count of row 82's cross-process sp_getapplock migrate
  semaphore, so the in-process and cross-process limits agree rather than
  fighting.
- Host.Tests — classes boot real ASP.NET Core hosts and real ActorSystems;
  Akka.Remote binds real sockets, Kestrel binds real ports, gRPC dials them.

Threads are capped, collections are NOT serialized. IntegrationTests
(parallelizeAssembly/parallelizeTestCollections false) and
CentralUI.PlaywrightTests (maxParallelThreads 1) already carry stricter
pre-existing settings and are left alone — the Playwright suite drives a browser
against a live rig, where shared UI state, not CPU, is the constraint.

Each capped csproj gains
<Content Include="xunit.runner.json" CopyToOutputDirectory="PreserveNewest" />;
the SDK does not copy it automatically and without it the file is silently
ignored. Verified by moving the copied file out of bin/ to take an uncapped
baseline: ConfigurationDatabase.Tests ran at 216 % CPU uncapped vs 145 % capped,
so the cap demonstrably takes effect.

Cost is close to nil — these suites are latency-bound on an external resource,
not CPU-bound (14-25 % utilisation for the two slowest). Standalone, uncapped ->
capped: AuditLog 383 tests 1m52s -> 1m51s; Host 520 tests 4m05s -> 4m08s;
ConfigurationDatabase 396 tests 10s -> 12s; SiteCallAudit 66 tests 8s -> 8s. All
green.

docs/test_infra/test_infra.md gains a Test Parallelism Policy section: the two
axes (in-process collection threads = this cap; cross-process solution-level
dotnet test = unbounded in general, bounded on the DB-migration path by the
sp_getapplock semaphore), the per-suite table with reasons, the rules (cap
threads rather than serialize; a cap is not a substitute for test isolation; the
Content copy is required), and the default for new projects (no config until the
project acquires a per-class external resource). test_infra_db.md cross-links it
from the semaphore bullet.
 2026-08-17 05:27:52 -04:00
Joseph Doherty 34a8079d15 test(auditlog): close register row 95 — stop the three remaining leaked fast-tick actors 
Row 82 (da23f2dc) fixed this leak shape in PartitionPurgeTests and
NotifyDispatcherAuditTrailTests; three classes in the same assembly still had
it. Each creates an actor with a fast periodic tick (100 ms purge override,
~0 s reconciliation override) against a shared MsSqlMigrationFixture database
and never stops it deterministically, relying on TestKit's Dispose -> Shutdown
— which is timeout-bounded and does NOT fail when it expires. A survivor keeps
mutating the fixture DB while LATER test classes run.

All three now use the same Watch(actor) / Sys.Stop(actor) /
ExpectTerminated(actor) helper da23f2dc established, applied at EVERY
actor-creation site — 11 in AuditLogPurgeActorTests, 10 in
SiteAuditReconciliationActorTests, 2 in OutageReconciliationTests.

The two real-DB cases are the ones that could corrupt another class:
AuditLogPurgeActorTests' EndToEnd_RealPartition_… keeps switching partitions
out of the shared DB, and SiteAuditReconciliationActorTests'
Tick_Duplicates_NotDoubleInserted holds a live AuditLogRepository over a
DbContext the test is about to dispose. The stub-repo cases are cheaper but not
free: each burns a DI scope and a thread every 100 ms for the rest of the
assembly run.

OutageReconciliationTests needed more than the helper. It already called
Sys.Stop, but (a) fire-and-forget, so the call returns before the in-flight tick
has finished writing to MSSQL, and (b) from the LAST line of the enclosing
`await using` block, so any earlier assertion failure skipped it entirely. Both
stops move to immediately after the last step that needs the actor alive —
before the read-back that counts rows — and the AuditLogIngestActor that
BuildCentralPipeline creates is now stopped too (test 2 was discarding its ref
outright, leaving it holding a service provider the block then disposed).

Where a test's remaining assertions read a collection the actor appends to
(client.Calls indexing, Assert.Empty on a switched-boundaries list), the stop is
placed BEFORE them so they no longer read a moving target.

AuditLog.Tests: 383/383 green (1 m 51 s), 0 skipped.
 2026-08-17 05:27:37 -04:00
Joseph Doherty 26e3a1b9a6 test(siteruntime): close register row 91 — compile-cache asserts become key-scoped deltas 
SiteScriptCompileCacheTests.GetOrAdd_SameCodeAndGlobals_ComputesOnce asserted
Assert.Equal(1, SiteScriptCompileCache.Hits) after a Clear() on what is a
process-wide static cache. The [Collection("SiteScriptCompileCache")] attribute
serializes this class against ScriptCompilationServiceTests and
DeploymentWarmThenGateTests, but NOT against the rest of the assembly: every
class that compiles a script — Instance/Script actor tests, deployment tests —
goes through the same GetOrAdd in parallel and bumps the same global Hits/Count.
Failed ~1 run in 4 under a full parallel solution run, green in isolation.

Fixed with the register's row-74 pattern (98c94588): assert the DELTA
attributable to this test's OWN keys, never an absolute global value. Two habits
carry it.

1. Every test now keys its entries on a per-test-instance guid token (the new
   Code() helper embeds it as a comment), so no other class in the assembly can
   collide with or serve them. That makes the factory-call counts and
   Assert.Same identity checks exact and fully owned — and those are the real
   claims: "computed once" and "the second lookup returned that instance".
2. The process-global counters are only ever asserted as one-sided bounds
   (>= baseline + n), since foreign traffic can only add. There is no concurrent
   Clear() to make a delta go negative: all three classes that call Clear() are
   in the serialized collection.

Four assertions changed shape, and two got stronger rather than weaker.
GetOrAdd_SameCode_DifferentGlobals_AreSeparateEntries asserted Count == 2 and
Hits == 0 — both only true while nothing else was compiling. "No cross-globals
hit" is now asserted where it is observable and ours: the second lookup HAD to
compute, the two results are not the same instance, and re-getting either serves
its own entry rather than the other globals surface's. Overflow_StaysWithinTheBound
gained an eviction probe over its own keys — a bound with foreign slack is not
proof a sweep ran; a recompute of an entry we inserted is.
Overflow_EvictsOldestBatch_… had Assert.Equal(hitsBefore + 1, Hits), the same
absolute-delta shape, now a lower bound alongside the sharp Assert.Same.

DeploymentWarmThenGateTests carried the identical latent shape one line before
its own delta assertion — Assert.Equal(0, Hits) immediately after Clear() —
and is converted to a baseline snapshot too.

SiteRuntime.Tests: 633/633 green, twice, at full parallelism (43 s, 42 s).
 2026-08-17 05:27:22 -04:00
Joseph Doherty c57e74e660 fix(siteruntime): arbitrate DCL connection config — definitions store wins (rows 89 + 90) 
Closes deferred-work-register rows 89 and 90, and corrects their premise twice.

THE RULE
The `data_connection_definitions` store wins wherever it has an opinion; an
instance's flattened `Connections` entry is a deploy-time snapshot and is
authoritative only for names the store has never owned (the deploy-only site,
where that table is legitimately empty).

PREMISE CORRECTION 1 — the peer's DCL is EMPTY, not stale
`DeploymentManagerActor` is a `site-{SiteId}`-scoped singleton and the only
sender of Create/RemoveConnectionCommand, to its OWN node's `dcl-manager`
(which exists on every node but is populated only where the singleton runs).
So the standby serves nothing. The real defect is that `_createdConnections`
is in-memory per singleton incarnation: a restart or failover rebuilds every
connection from the flattened snapshots and silently discards a newer
connection-only artifact deploy. Row 89 is therefore create-time, not
message-delivery, and NOT replication-specific — the same loss happens on an
unreplicated single-node restart.

PREMISE CORRECTION 2 — a changed config never reached a LIVE connection either
`DataConnectionManagerActor.HandleCreateConnection` is create-only (warns and
returns on a name it holds), so `EnsureDclConnection`'s documented "re-issues
the create so the DCL adopts the new configuration" has never been true on any
node. Now surfaced with a Warning saying exactly that, once per changed config.
Not fixed here: stop-and-recreate drops every tag subscription and InstanceActor
cancels its subscribe-retry timer on first success, so nothing re-subscribes.

MECHANISM
`data_connection_definitions` gains its first production reader. It is read on
the SAME off-thread task as the deployed configs (`ReadStoreSnapshotAsync`), by
the boot load and every sweep tick, into an actor-thread-confined arbitration
overlay applied BEFORE the boot batch and BEFORE the sweep's repair loop. A
CDC-delivered definition change converges within one tick on whichever node runs
the singleton — no new cross-node message. A failed definitions read yields null
and leaves the overlay untouched (distinct from an empty list, which is a real
answer); clearing it would reintroduce the ping-pong through the error path.

THE PING-PONG, KILLED BY CONSTRUCTION
§7.4 of the sweep memo rejected a store-level pass because two sources would
flip the single per-name hash slot, recreating the actor every tick. That is a
property of WHERE the hash is computed: every ensure path now resolves the
winner BEFORE hashing, so only one configuration can ever occupy the slot. The
artifact-deploy path writes its rows into the overlay in the same pass, closing
the same flip arriving through the other door.

REMOVAL (row 90)
Both removal paths share `RemoveConnectionCentrally` and record an in-memory
tombstone, so a centrally removed connection is not resurrected by the next
instance start — a pre-existing bug. Tombstones are deliberately not persisted;
a definition row reappearing clears one.

TESTS
tests/.../Actors/DclConnectionArbitrationTests.cs — 8 tests. Fail-before
(behavioral revert: overlay inert + tombstone check disabled): 5 fail, 3 pass;
the 3 passers are the two deploy-only regression guards and the ping-pong guard.

DOCS
New decision memo docs/plans/2026-08-17-dcl-connection-arbitration-decision.md
(topology, rule, ping-pong mechanism, tombstones, residuals); Component-SiteRuntime,
Component-DataConnectionLayer + their docs/components mirrors; the §7.4 residual in
the sweep memo marked closed with its premise corrected.
 2026-08-17 05:22:42 -04:00
Joseph Doherty 9f67147e7b refactor(deployments): one paged read of DeploymentRecords + close its three drifts 
Deferred-work register rows 37, 86, 87, 88 — the structural duplication and the
three behavioral divergences it had accumulated, fixed together.

Row 37 (structural). `IDeploymentManagerRepository.QueryDeploymentSummariesAsync`
is DELETED. It and `QueryDeploymentListPageAsync` were two paged reads of the same
insert-only table with different contracts — raw offset vs page number, no total vs
total, no instance join vs joined, instance-id-list scoping vs site-id scoping. The
CLI/HTTP `QueryDeployments` handler now goes through `QueryDeploymentListPageAsync`
with a `DeploymentListFilter`, exactly like the Central UI page. The
`DeploymentRecordSummary` projection went with it (`DeploymentListRow` is the single
row shape); its `MaxPageSize` const moved to `DeploymentListPage`, which also gains
`DefaultPageSize`.

Row 86 (Medium, authorization — the headline). `SiteScopeService.ResolveAsync` was
claims-only, so an Administrator carrying `SiteId` claims saw the whole fleet through
the CLI (`ManagementActor.EnforceSiteScope` has always exempted Administrator) and
only their granted sites on the Deployments page. One user, one dataset, two answers.
Administrator = system-wide is the intended semantic on both sides; the UI was the
defect. The role claim is read explicitly with an OrdinalIgnoreCase comparison rather
than via `IsInRole`, which compares claim VALUES ordinally and only resolves when the
identity carries the matching roleType — the two sides must not disagree on a
differently-cased role name. An Administrator's site claims are not collected at all,
so the invariant "system-wide => empty permitted-id set" holds for every caller.
Designer is NOT exempt: the class doc claimed "any Admin/Design user" is system-wide,
which was true of neither side, so the doc was corrected rather than the bypass
widened.

Row 87 (Low). The scoped branch used to resolve the site grant into an instance-id
list by loading EVERY instance in the system, non-transactionally, on every scoped
query; the grant now rides as `SiteIdScope` and is resolved by the query's own join.
An unparseable permitted-site id is DROPPED, never ignored — dropping can only narrow
a grant, discarding it would widen it to fleet-wide. And `PageSize <= 0` no longer
means UNLIMITED down at the repository (`--page-size 0` materialized the whole
retention window): it means "unspecified" and takes the CLI's own default of 50,
capped at 1000. Bounding without telling would be silent truncation, so the handler
now returns a `DeploymentQueryResult` envelope carrying `totalCount` plus the page
coordinates actually applied — with a CLI table projection (mirroring row 57's
`AuditConfigTableProjection`) that unwraps it and appends a "showing N of M" note.

Row 88 (Low, UX). An unrecognised `--status` fell back to "no filter", so a typo like
`--status Sucess` returned every status — a complete-looking answer to a question the
caller never asked. It is now a `ManagementError` naming the valid values. Parsing
stays case-insensitive; a numeric string is rejected too, since `Enum.TryParse`
accepts "7" and yields an undefined value (guarded with `Enum.IsDefined`). Loud
failure over silent full results, per row 46.

Tests: 7 new `SiteScopeServiceTests` pinning Administrator = system-wide (verified
failing 6/7 against the pre-fix service, the Designer pin passing both ways); the
ManagementActor deployment suite migrated onto the consolidated query with new pins
for the status error, case-insensitive parse, numeric rejection, page-size default/
cap/floor, empty-scope-on-garbage-grant, and "reads no instances at all"; a repository
test for site-scope+status composing over a bounded page; and 6 CLI tests for the new
table projection.
 2026-08-17 05:15:18 -04:00
66 changed files with 4448 additions and 653 deletions
Expand all files Collapse all files
docs/components/CentralUI.md
+3 -1
View File
@@ -51,7 +51,9 @@ Because `CookieAuthenticationStateProvider` serves a frozen principal, the circu
### Site-scoped authorization
`SiteScopeService` (scoped, registered in `AddCentralUI`) reads `SiteId` claims from the circuit principal. Deployment users whose LDAP mapping carries site-scope rules carry one integer `SiteId` claim per permitted site; Admin and Design users carry none (system-wide). Pages that enumerate sites or issue cross-site commands call `SiteScopeService.FilterSitesAsync` and `IsSiteAllowedAsync` to enforce the grant before any operation.
`SiteScopeService` (scoped, registered in `AddCentralUI`) reads the role and `SiteId` claims off the circuit principal. Deployment users whose LDAP mapping carries site-scope rules carry one integer `SiteId` claim per permitted site; a user with no `SiteId` claims is system-wide. Pages that enumerate sites or issue cross-site commands call `SiteScopeService.FilterSitesAsync` and `IsSiteAllowedAsync` to enforce the grant before any operation.
**A user holding the `Administrator` role is system-wide regardless of the `SiteId` claims they carry** (deferred-work register row 86). `ManagementActor.EnforceSiteScope` has always exempted Administrator, so before this the same user could act fleet-wide through the CLI/HTTP surface while the UI hid the same sites — one user, one dataset, two answers. The role claim is read explicitly (`JwtTokenService.RoleClaimType`, `OrdinalIgnoreCase`) rather than via `ClaimsPrincipal.IsInRole`, because `IsInRole` compares claim *values* ordinally and only resolves at all when the identity was built with the matching `roleType`; the actor compares case-insensitively, and the two sides must not disagree on a differently-cased role name. **`Designer` is NOT exempt** — the enforced rule has only ever bypassed Administrator, and this document previously claimed "Admin and Design users carry none (system-wide)", which was true of neither side. An Administrator's `SiteId` claims are not collected at all, so the invariant *system-wide ⇒ empty permitted-id set* holds for every caller and a consumer reading the id set without consulting `IsSystemWideAsync` cannot silently narrow an Administrator to a subset.
### Repository access pattern
docs/components/DataConnectionLayer.md
+36 -3
View File
@@ -161,7 +161,11 @@ A separate in-flight set `_subscribesInFlight` tracks the initial `SubscribeAsyn
### Creating a connection
The Site Runtime's `DeploymentManagerActor` sends `CreateConnectionCommand` to `DataConnectionManagerActor` whenever it creates an `InstanceActor` — the config comes from the `Connections` map inside that instance's flattened config, not from the site's `data_connection_definitions` table (which has no production reader). The manager never reads storage and has no independent view of which connections a site should have, so a start path that fails to push is indistinguishable here from a connection that genuinely does not exist: that was register row **85**, where two of the four Instance-Actor start paths omitted the push and every subscribe from the resulting instances was answered `Unknown connection: <name>` forever. The push now lives inside `CreateInstanceActor` on the Site Runtime side. Sends are hash-guarded there, so an unchanged connection produces no message at all:
The Site Runtime's `DeploymentManagerActor` sends `CreateConnectionCommand` to `DataConnectionManagerActor` whenever it creates an `InstanceActor` — the config comes from the `Connections` map inside that instance's flattened config, not from the site's `data_connection_definitions` table (which the DCL never reads — and which had no production reader at all until rows 89/90 made it the Deployment Manager's arbitration overlay; see below). The manager never reads storage and has no independent view of which connections a site should have, so a start path that fails to push is indistinguishable here from a connection that genuinely does not exist: that was register row **85**, where two of the four Instance-Actor start paths omitted the push and every subscribe from the resulting instances was answered `Unknown connection: <name>` forever. The push now lives inside `CreateInstanceActor` on the Site Runtime side. Sends are hash-guarded there, so an unchanged connection produces no message at all:
> **Arbitration (register rows 89 + 90)**: the Deployment Manager *does* read `data_connection_definitions` as of 2026-08-17 — a name with a row there is configured from that row, and only a name with no row falls to the flattened snapshot. The resolution happens **before** the hash is computed, so the two sources can never occupy the same per-name hash slot alternately. Still nothing changes here: the DCL manager remains a router over actors it is told to create. See `SiteRuntime.md` and `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md`.
>
> **`HandleCreateConnection` is create-only — which is why a CHANGE is a different command (register row 97)**: a create for a name it already holds warns and returns, so a re-issued create never reconfigured a live connection actor, on any node. The Deployment Manager now sends a `ReconfigureConnectionCommand` on the change edge; `HandleReconfigureConnection` **forwards** it to the owning connection actor, and falls back to a create for a name it does not hold (so the command is safe whatever the manager's state). A stop-and-recreate was rejected as destructive — subscriptions are held by `DataConnectionActor` and `InstanceActor` cancels its subscribe-retry timer on first success, so nothing would re-subscribe.
```csharp
// Commons/Messages/DataConnection/CreateConnectionCommand.cs
@@ -171,10 +175,39 @@ public record CreateConnectionCommand(
IDictionary<string, string> PrimaryConnectionDetails,
IDictionary<string, string>? BackupConnectionDetails = null,
int FailoverRetryCount = 3);
// Commons/Messages/DataConnection/ReconfigureConnectionCommand.cs — same shape,
// applied IN PLACE to a live connection actor.
public record ReconfigureConnectionCommand(
string ConnectionName,
string ProtocolType,
IDictionary<string, string> PrimaryConnectionDetails,
IDictionary<string, string>? BackupConnectionDetails = null,
int FailoverRetryCount = 3);
```
`DataConnectionManagerActor.HandleCreateConnection` calls `DataConnectionFactory.Create(ProtocolType, PrimaryConnectionDetails)` to produce the initial `IDataConnection` adapter and spawns a `DataConnectionActor` child.
### Reconfiguring a live connection
`DataConnectionActor` handles `ReconfigureConnectionCommand` in **all three** states — `Connecting`, `Connected` and `Reconnecting` — and never stashes it (a connection pointed at a wrong or dead endpoint lives in the two non-Connected states, which is exactly when a correction arrives; stashed, it would wait on the broken configuration forever). Handling replaces protocol/primary/backup/failover-count, resets endpoint selection to Primary, and calls `SwapAdapter` — the same helper both failover sites use:
```csharp
private void SwapAdapter(IDictionary<string, string> newConfig)
{
_adapter.Disconnected -= OnAdapterDisconnected;
_ = _adapter.DisposeAsync().AsTask(); // fire-and-forget; never awaited on the actor thread
_adapter = _factory.Create(_protocolType, newConfig);
_connectionDetails = newConfig;
_adapter.Disconnected += OnAdapterDisconnected;
_adapterGeneration++; // LAST — everything in flight on the old adapter is now stale
}
```
It then re-enters the reconnect cycle immediately (`BecomeReconnecting(afterReconfigure: true)` — zero delay, and the unstable-disconnect/failover accounting suppressed, since the link was dropped on purpose), and `ReSubscribeAll` / `ReSubscribeAllAlarms` re-issue everything from the untouched subscription registries. A reconfigure carrying the configuration already in force is a no-op.
### Subscribing and receiving values
Instance Actors send `SubscribeTagsRequest` and receive `TagValueUpdate` messages. The actor sends `TagValueUpdate` to the Instance Actor's ref for every value change notification from the adapter. On disconnect, `ConnectionQualityChanged` (with `QualityCode.Bad`) is sent to all subscribers so Instance Actors reflect staleness immediately without waiting for a per-tag callback.
@@ -275,8 +308,8 @@ Carried on `CreateConnectionCommand` from the deployment artifact.
## Dependencies & Interactions
- [Site Runtime (#3)](./SiteRuntime.md) — `DeploymentManagerActor` sends `CreateConnectionCommand` / `RemoveConnectionCommand` to create and tear down connections during instance deployment. Instance Actors send `SubscribeTagsRequest` / `UnsubscribeTagsRequest` / `WriteTagRequest`. `NativeAlarmActor` (peer to `AlarmActor` under `InstanceActor`) sends `SubscribeAlarmsRequest` / `UnsubscribeAlarmsRequest` and receives `NativeAlarmTransitionUpdate` / `NativeAlarmSourceUnavailable`.
- [Commons (#16)](./Commons.md) — owns `IDataConnection`, `IAlarmSubscribableConnection`, `IBrowsableDataConnection` interfaces; `TagValue`, `QualityCode`, `NativeAlarmTransition`, `AlarmConditionState`, `AlarmTransitionKind` types; and all message contracts (`SubscribeTagsRequest/Response`, `TagValueUpdate`, `WriteTagRequest/Response`, `SubscribeAlarmsRequest/Response`, `NativeAlarmTransitionUpdate`, `NativeAlarmSourceUnavailable`, `CreateConnectionCommand`, `BrowseNodeCommand/Result`, `DataConnectionHealthReport`).
- [Site Runtime (#3)](./SiteRuntime.md) — `DeploymentManagerActor` sends `CreateConnectionCommand` / `ReconfigureConnectionCommand` / `RemoveConnectionCommand` to create, reconfigure in place, and tear down connections during instance deployment. Instance Actors send `SubscribeTagsRequest` / `UnsubscribeTagsRequest` / `WriteTagRequest`. `NativeAlarmActor` (peer to `AlarmActor` under `InstanceActor`) sends `SubscribeAlarmsRequest` / `UnsubscribeAlarmsRequest` and receives `NativeAlarmTransitionUpdate` / `NativeAlarmSourceUnavailable`.
- [Commons (#16)](./Commons.md) — owns `IDataConnection`, `IAlarmSubscribableConnection`, `IBrowsableDataConnection` interfaces; `TagValue`, `QualityCode`, `NativeAlarmTransition`, `AlarmConditionState`, `AlarmTransitionKind` types; and all message contracts (`SubscribeTagsRequest/Response`, `TagValueUpdate`, `WriteTagRequest/Response`, `SubscribeAlarmsRequest/Response`, `NativeAlarmTransitionUpdate`, `NativeAlarmSourceUnavailable`, `CreateConnectionCommand`, `ReconfigureConnectionCommand`, `BrowseNodeCommand/Result`, `DataConnectionHealthReport`).
- [Health Monitoring (#11)](./HealthMonitoring.md) — `DataConnectionActor` calls `ISiteHealthCollector.UpdateConnectionHealth`, `UpdateTagResolution`, `UpdateTagQuality`, `UpdateConnectionEndpoint`, and `RemoveConnection` to keep the site health report current. `DataConnectionManagerActor` handles `GetAllHealthReports` by forwarding `GetHealthReport` to each child.
- [Site Event Logging (#12)](./SiteEventLogging.md) — `DataConnectionActor` logs connection lost, restored, and failover events via `ISiteEventLogger.LogEventAsync` (fire-and-forget). Absent in tests where `ISiteEventLogger` is null.
- Design spec: [Component-DataConnectionLayer.md](../requirements/Component-DataConnectionLayer.md).
docs/components/SiteRuntime.md
+5 -1
View File
@@ -91,6 +91,10 @@ Central sends a `DeployInstanceCommand` carrying a JSON `FlattenedConfiguration`
> **`CreateInstanceActor` is the single funnel for the connection push too (register row 85)**: the `EnsureDclConnections` call moved *into* `CreateInstanceActor` on 2026-08-17. It used to sit at each caller, and only two of the four callers made it — the boot-time batch and this deploy path. `StartStoredInstance` (row-67 ensure-started and the row-78 sweep) and `HandleEnableResult` did not, so a self-healed or newly-enabled instance came up with no connection actors and every tag subscribe was answered `Unknown connection: <name>` indefinitely. See the *Instance reconcile sweep* section and `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7.
> **Which config source wins (register rows 89 + 90, 2026-08-17)**: `EnsureDclConnections` no longer trusts the flattened snapshot blindly. `data_connection_definitions` is read on the same off-thread task as the deployed configs, at boot and on every sweep tick, into an actor-thread-confined **arbitration overlay**: a connection name with a row there is configured from that row, and only a name with no row uses the instance's flattened `Connections` entry (which is the correct source on a deploy-only site, where that table is legitimately empty). The winner is resolved **before** the per-name hash is computed, which is what makes the two-source ping-pong — the reason a store-level sweep was rejected in §7.4 — unrepresentable rather than merely avoided. Removals additionally leave an in-memory tombstone so a centrally deleted connection is not resurrected by the next instance start. A monotonic write-generation guard (row 101) discards a snapshot whose off-thread read raced an artifact apply — its stale absences would otherwise tombstone and stop just-applied live connections that nothing re-creates until an instance redeploy. **The register's premise was wrong in two ways** (the standby's DCL is empty rather than stale, so the real defect is create-time on the next singleton incarnation and is not replication-specific; and a re-issued create never reconfigured a *live* connection actor at all). See `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md`.
> **A changed config reaches the LIVE connection (register row 97)**: resolving the winner decides *what* to push; a third command was needed to make a push against an already-live connection mean anything, because `DataConnectionManagerActor.HandleCreateConnection` warns and returns for a name it already holds. `EnsureDclConnection` now has three outcomes off the one per-name hash slot — **unchanged → nothing**, **new → `CreateConnectionCommand`**, **changed → `ReconfigureConnectionCommand`** — and the connection actor applies the last of these in place by swapping its adapter through the failover machinery and re-issuing its subscriptions. Log line: `Reconfigured DCL connection {Connection} (protocol={Protocol}) in place` (Information), replacing the "will not take effect" Warning rows 89/90 added. See `DataConnectionLayer.md` → *Reconfiguring a live connection*.
For redeployment (instance already running), the existing actor is stopped and watched:
```csharp
@@ -219,7 +223,7 @@ Both `AlarmActor` and `NativeAlarmActor` tell the `InstanceActor` an `AlarmState
| `shared_scripts` | Shared script code from artifact deployments | No |
| `external_systems` | External system definitions (incl. `timeout_seconds`, and `max_retries`/`retry_delay_ms` — the deployed store-and-forward retry policy, `0` = use the engine default) | No |
| `database_connections` | Database connection strings | No |
| `data_connection_definitions` | OPC UA / MxGateway endpoint definitions | No |
| `data_connection_definitions` | OPC UA / MxGateway endpoint definitions — read at boot and on every sweep tick as the DCL **arbitration overlay** (rows 89/90) | No |
| `notification_lists` | Notification list definitions | No |
| `smtp_configurations` | SMTP configuration (from artifact deployment) | No |
docs/plans/2026-07-08-deferred-work-register.md
+14 -9
View File
@@ -30,7 +30,7 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
| 34 | ~~**`site_events` purge burst is unbounded in the oplog and unmeasured at scale**~~ **RESOLVED 2026-08-16 (`d9a35846`) — measured, and paced.** Measured with the REAL `EventLogPurgeService` over a real `RegisterReplicated` LocalDb file + a loopback sync pair at the rig's `MaxBatchSize: 64`: 393 B/row incl. indexes ⇒ a 1 GB event log holds **~2.73 M rows**, and expiring them in one tick took 285 s and queued **2 731 947 oplog rows = 2.7x LocalDb's `MaxOplogRows` default (1 M)** — enough to make the maintenance tick prune the oplog and flag a FULL ten-table snapshot resync, bought by a routine retention purge. The decisive nuance: the *resident* backlog is produced−minus−drained, so a healthy peer blunts it (391 k peak on loopback, drained 6.0 min) — the old safety margin was a function of peer throughput and vanished with the peer down. Fix: `MaxPurgedRowsPerTick` (default **250 000**, `0` = unlimited escape hatch), one budget SHARED across retention + cap trim; a ceiling-limited tick reschedules after `PurgeCatchUpInterval` (**5 min**, validated `> 0` and `<= PurgeInterval`) so worst-case retention converges in ~55 min instead of 11 days; purge loop is now a delay loop anchored to the END of the previous purge; rate-limited Warning + counter `scadabridge.site.event_log.purge_ceiling_hits`. No purge-exemption path added. Row-33 interplay verified empirically (500 k drained in 46 s vs the 15-min dwell). Fail-before: `PurgeBudget` neutralised → 6 of the new tests fail, all 19 pre-existing pass. Measurement + decision record: `docs/plans/2026-08-15-site-events-purge-burst-measurement.md`. Side-finding recorded as row 81 (storage cap unenforceable on a replicated node). | Found by R7 | Closed 2026-08-16. | — |
| 35 | ~~**`DeploymentRecords` index does not cover the deployment-list sort key**~~ **RESOLVED 2026-08-15 (`be29bf47`).** Migration `20260815165452_AddDeploymentRecordListSortIndex` + hand-written `docs/plans/sql/AddDeploymentRecordListSortIndex.sql`: composite `IX_DeploymentRecords_DeployedAt_Id (DeployedAt DESC, Id DESC)`; the old single-column index DROPPED as strictly redundant (`Id` is the clustered PK and was already in the old index's leaf as the row locator — same leaf width, now a declared ordered key; keeping it would cost a second index on every insert/status transition for no read benefit). Serves the three reads sharing the key (`QueryDeploymentListPageAsync`, `QueryDeploymentSummariesAsync`, `GetCurrentDeploymentStatusAsync`). `Id` in the key is also a CORRECTNESS matter: a bulk site deploy stamps many rows in one `DeployedAt` instant, so without the tie-break keyset pages can repeat or drop rows. Recorded caveat: the optimizer may already have served the ordering via a backward scan of the old index, so the "sorts per page" premise was possibly optimizer-dependent — the composite makes it deterministic and self-documenting; an actual plan capture against the rig would quantify the before/after. The ConfigurationDatabase suite exercises the descending composite as real DDL via SQLite `EnsureCreated()`, proving provider portability. | Found by R3 (`35ce1413`) | Closed 2026-08-15. | — |
| 36 | **Deployments free-text search is unindexable by construction** | Found by R3; re-verified 2026-08-17 (Low-tail sweep) | Four `LIKE '%term%'` predicates (instance name, deployment id, revision hash, user) — leading wildcards preclude index seeks, so a search is a scan of the join (`DeploymentManagerRepository.cs:100-104`; only searched column with any index is `DeploymentId`, unseekable under a leading wildcard). Acceptable at the purge-bounded row count. **2026-08-17 refresh: the row undercounted the cost by half** — `BuildListQuery` is shared with `GetDeploymentStatusCountsAsync`, so one search = TWO scans of the join (list + status tiles), mitigated by the UI's trailing-edge debounce (one pair per pause, not per keystroke). Anchoring the id/hash legs was evaluated and REJECTED as a micro-fix: `DeploymentListFilter.Search`'s doc comment promises substring matching and a 4-case `[Theory]` deliberately pins mid-string matches — changing it is a contract renegotiation, not an optimization. Fix if hot: renegotiate to prefix on id/hash legs, or full-text. *(Low)* | Search latency complaints on the Deployments page |
| 37 | **Two paged reads of `DeploymentRecords` with different contracts** | Found by R3; re-verified 2026-08-17 (Low-tail sweep: both shapes intact, `DeploymentManagerRepository.cs:110-149` and `:399-449`) | `QueryDeploymentSummariesAsync` (CLI/ManagementActor: offset, no total, no instance join, instance-id-list scoping) and R3's `QueryDeploymentListPageAsync` (UI: total + instance join + site-id scoping) duplicate query logic. Consolidating on one contract — giving the CLI a total, both scoping by site — removes the duplication. **2026-08-17: the duplication has now DRIFTED into real behavioral divergences, recorded as rows 86–88 (site-scope authorization divergence, CLI unbounded read + non-transactional scope resolution, silent `--status` parse fallback) — a consolidation should resolve those rows with it.** *(Low structurally; the drift itself is row 86, Medium)* | Rows 86–88, or next time either query's shape changes |
| 37 | ~~**Two paged reads of `DeploymentRecords` with different contracts**~~ **RESOLVED 2026-08-17 (merge `ad0100f2`, agent commit `9f67147e`).** `QueryDeploymentSummariesAsync` and `DeploymentRecordSummary` are DELETED; the CLI/HTTP `QueryDeployments` handler now rides `QueryDeploymentListPageAsync` + `DeploymentListFilter` — one row shape (`DeploymentListRow`), one scoping mechanism (site-id via the query join), total count available to both surfaces (`DeploymentListPage` now also owns `MaxPageSize` + `DefaultPageSize = 50`). Rows 86–88 were resolved with the consolidation, as this row anticipated. Row 36's 4-case substring-search Theory untouched and still passing. Remaining UI-vs-CLI asymmetry (`GetDeploymentStatusCountsAsync` status tiles are UI-only) is deliberate, not drift. **Live-probed on the rig:** CLI `deploy status --status Success --page-size 5` returns the consolidated page — rows now carry `siteId` + `instanceUniqueName`, and the response carries `totalCount` (11) the CLI never had. | Found by R3; consolidation shipped with rows 86–88 | Closed 2026-08-17. | — |
| 38 | **OtOpcUa: `RoslynVirtualTagEvaluatorTests` racing-clear test is load-fragile** (OtOpcUa repo, not this one) | Found by R4 during the full-suite bump verification | `Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed` fails under full-solution parallel test load, passes 3/3 isolated: its design assumes at most one `ClearCompiledScripts` lands inside one evaluation (so the product's one-shot retry suffices); under contention two can. Same timing-assumption family as the R5 sweep, but lives in OtOpcUa. Test-side fix: bound clear cadence relative to observed evaluation latency, or assert on retry-exhaustion instead of zero failures. | Next OtOpcUa test session, or when it fails a run that matters |
| 39 | **OtOpcUa: late-enabled replication cannot baseline pre-existing rows** (OtOpcUa repo, not this one) | Found by R4; documented in OtOpcUa's CLAUDE.md | `LocalDbSetup.OnReady` registers all four tables unconditionally on every driver node, so it can never pass `baselineExistingRows: true` (only correct when gated on replication being configured). Turning replication ON for a long-running default-OFF node converges on subsequent writes only — existing rows are never baselined. ScadaBridge closed this class with conditional registration + boot-time `DeregisterReplicated` self-heal (WP1.3/WP3.3); porting it is a behavior change beyond a pin bump (registered-set pinned by tests, deregistration must be pair-symmetric). Low current impact: site-a was enabled from fresh. | Before enabling replication on an established OtOpcUa node |
| 40 | ~~**Wall-clock sleeps used as synchronization in tests (absence-assertion class)**~~ **RESOLVED 2026-08-15 (`7a7d345b`).** All six listed sites replaced with observable edges + positive controls, strengthening proven by stash-based negative controls: `ScriptDeadlineAtEnqueueTests:128` now pins the watchdog via a genuinely-wedged positive control (armed strictly later, self-releasing) + per-script absence assertion + detach counter pinned at 1 — **disabling the entire stuck-script watchdog passed the OLD test 2/2 and fails the new one**; `:105` now proves the run was still queued when its budget expired via `QueueDepth` edges (the actual design claim — the old sleep asserted nothing about it); `DeploymentWarmThenGateTests:117` (+2 sibling `Task.Delay(500)`s) — the empty-site zero is now the manager's own observation (phantom-startup control: old passed 2/2, new fails); `SiteAlarmLiveCacheServiceTests:195,216` poll a new subscription-cancellation observable across 3 linger windows, each closing with a positive control (control: linger cancellation disabled + viewer guard dropped — `Resubscribe_Within_Linger` old passed 2/2, new fails; honest caveat: the other case's old later assertion did already catch that break, so its gain is the observable + control, not a false-pass fix); `DataConnectionActorTests:435` waits for the Connected health publication so the subscribe provably runs against Connected (the sleep guarded no assertion — determinism gain only). Bonus: the de-padded schedule exposed a LATENT race in `InstanceActorChildAttributeRaceTests` (~1-in-6 under load, A/B'd 14/14 green on the unmodified tree so pre-existing) — fixed with a bounded-deadline resolution retry; pattern recorded as row 77. | Found by the R5 sweep (`9fb52153`) | Closed 2026-08-15. | — |
@@ -78,17 +78,22 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
| 83 | ~~**Post-boot start-path tests must gate on the DM's first instance-count report — `ReconcileStartsInstanceTests` still uses a bare `Task.Delay(500)`** *(Low, test-infra)*~~ **RESOLVED 2026-08-17 (`524bdbbe`).** `ReconcileStartsInstanceTests` converted to the first-`SetInstanceCounts`-report gate; the pattern was extracted into a shared `TestSupport/DeploymentManagerStartupProbe` now used by both this suite and `InstanceSweepSelfHealTests`, so the convention has one implementation to copy instead of two to keep in sync. SiteRuntime 633/633 green. | Row-78 fix session | Closed 2026-08-17. | — |
| 84 | ~~**`LogReplicationBacklogNote` can fire twice in one purge tick** *(Low, cosmetic)*~~ **RESOLVED 2026-08-17 (`ddb3c32e`).** The note is emitted at most once per tick. Fail-before: `RunPurge_WhenBothPhasesDeleteRows_LogsTheBacklogNoteOnceForTheWholeTick` fails pre-fix with `Assert.Single(): collection contained 2 matching items`. | Row-34 session | Closed 2026-08-17. | — |
| 85 | ~~**DCL connection actors are not spawned for connection configs that arrive via LocalDb replication — row 78's gap, one layer down** *(Medium-Low, availability; proven live 2026-08-16 by the row-78 close-out probe)*~~ **RESOLVED 2026-08-17 (`8f2c2813`) — with a premise correction AND a widened blast radius.** The row's mechanism was wrong: `data_connection_definitions` has NO production reader (its getter is test-only) — connection actors are created solely from the `Connections` map in an instance's own flattened config via `EnsureDclConnections`, so there was never a store→actor path to repair. The real defect: that ensure was the CALLER'S job and only 2 of 4 instance-start paths did it (boot batch ✅, `ApplyDeployment` ✅, `StartStoredInstance` ❌ — both the row-67 ensure-started and the row-78 sweep — and `HandleEnableResult` ❌; the two misses are exactly the two paths added later). So the bug was NOT replication-specific: an ordinary unreplicated site that boots with an instance disabled and enables it later hit the same dead end, because the boot load reads only enabled configs. Fix is structural: `EnsureDclConnections` moved INTO `CreateInstanceActor` (the documented single creation funnel), before the already-exists guard so the two working paths keep exact semantics; reuses the per-name hash guard so steady state is silent; a fifth path cannot forget. Store-level sweep pass explicitly REJECTED (memo §7.4): it wouldn't have fixed this (the table is empty on deploy-only sites) and the unguarded form ping-pongs the hash slot against the deploy path, dropping live subscriptions every tick. Fail-before by behavioral revert: 4 of 7 new tests fail (sweep-started, reconcile-ensure-started, enable-after-disabled-boot, exactly-once-across-sweeps). SiteRuntime 633/633, DCL 320/320. Sibling gaps found during the fix recorded as rows 89/90. | Proven live 2026-08-16 by the row-78 close-out probe (sweep started `rig-probe-1`/`soak-notify-1` but every tag subscribe failed for 6+ min with `Unknown connection: OPC PLC Simulator — will retry`; a second pair restart healed it) | Closed 2026-08-17. **Live-proven on the rig same day** in the exact original failure sequence (pair stopped, node-a `site-localdb.db` quarantined as `.probe85-quarantine-2026-08-17`, pair restarted): reconcile "0 fetched" at 08:29:59, then at the 08:30:54 sweep tick `Created DCL connection OPC PLC Simulator` fired ALONGSIDE the two self-heal Warnings (pre-fix they appeared alone), zero `No connection actor`/`Unknown connection` lines on either node, all three `rig-probe-1` tags Good with live values ~2 min after restart — NO second restart; steady-state silence confirmed (0 Created/Updated lines across subsequent ticks); replication reseeded (`localDbReplicationConnected: true`, backlog 0), site-b/c still correctly omit the fields. | — |
| 86 | **CLI and UI disagree on deployment site-scoping for Administrators** *(Medium, authorization)* | Found by the 2026-08-17 Low-tail sweep while re-verifying row 37 | `ManagementActor.cs:2586` grants an **Administrator role bypass** of site scoping on `deploy status`, while the UI's `SiteScopeService` is role-independent (system-wide iff the user carries zero `SiteId` claims). An Administrator carrying site claims therefore sees the whole fleet via the CLI but only their sites on the Deployments page — same user, same data, two answers, and one of them is wrong. This is row 37's duplication having drifted into behavior; decide which semantic is intended (role bypass vs claims-only) and enforce it on BOTH paths, ideally as part of the row-37 consolidation. | Next ManagementService/CLI session, or the row-37 consolidation — whichever comes first |
| 87 | **CLI deployment scoping is non-transactional and unbounded** *(Low)* | Found by the 2026-08-17 Low-tail sweep | The CLI path resolves site grants to an instance-id list in a separate, non-transactional read (`ManagementActor.cs:2590-2596`), reloading EVERY instance per scoped query; and `PageSize: 0` on the wire maps to `take: null` = "no limit" (`DeploymentManagerRepository.cs:434`), materializing the entire retention window in one query. The UI path is structurally incapable of both. Bounded today by the purge window and low CLI traffic. | Row-37 consolidation, or first slow/OOM `deploy status` report |
| 88 | **Unparseable CLI `--status` silently means "no filter"** *(Low, UX)* | Found by the 2026-08-17 Low-tail sweep | `ManagementActor.ParseDeploymentStatus` returns null on an unparseable status value, so a typo (`--status Sucess`) silently returns ALL statuses instead of an error — the caller cannot distinguish "filter applied, nothing matched" from "filter ignored". Fix: fail loudly on an unrecognized value (mirror the row-46 principle: loud failure over silent empty/full results). | Next CLI touch to the deploy command group |
| 89 | **Connection-only artifact deploy does not reach the peer node's DCL** *(Medium-Low, availability)* | Found by the 2026-08-17 row-85 fix session | `data_connection_definitions` IS one of the ten replicated tables, but only the node that received the deploy runs `HandleApplyArtifactDataConnectionsToDcl` — the peer gets the row via CDC and keeps serving the OLD endpoint until a restart or an instance redeploy touches it. Not fixable by another sweep: applying a definition change to a live connection needs an arbitration decision first (does the instance's flattened-config `Connections` hash or the standalone definition hash win when they disagree?), because the row-85 fix deliberately made the flattened config the sole creation source. | Next artifact-deploy or DCL session — decide the arbitration, then apply-on-CDC becomes safe |
| 90 | **No removal path for a connection deleted on the peer** *(Low, mirror image of row 89)* | Found by the 2026-08-17 row-85 fix session | `RemoveConnectionCommand` is sent only from the artifact-deploy reconcile on the node that processed the deploy; a connection deleted on the peer leaves a live connection actor here until restart. Bounded: the orphan actor serves a connection no instance references, wasting a device session but corrupting nothing. Same arbitration prerequisite as row 89 — fix them together. | With row 89 |
| 91 | **`SiteScriptCompileCacheTests.GetOrAdd_SameCodeAndGlobals_ComputesOnce` is parallel-load fragile** *(Low, test-infra; row-82 family)* | Found by the 2026-08-17 row-85 fix session: failed once in 4 full-suite runs, green in isolation AND green on base — pre-existing, probability merely raised by the suite growing | `SiteScriptCompileCache` is a process-wide static and the test asserts an absolute `Hits == 1` after a global `Clear()`; xUnit runs other classes in parallel in the same process, so any concurrently-compiling test breaks the absolute count. Recommended fix: row-74's established delta-over-scoped-action assertion pattern (a collection fixture would serialize more than needed). | Next SiteRuntime test touch, or first flake in a run that matters |
| 86 | ~~**CLI and UI disagree on deployment site-scoping for Administrators**~~ **RESOLVED 2026-08-17 (merge `ad0100f2`, agent commit `9f67147e`) — Administrator = system-wide is the intended semantic; the UI was the defect.** The rule was already documented on BOTH sides (`ManagementActor.cs` ~:522 "Users with the Administrator role, or system-wide Deployer, are not restricted"; `SiteScopeService`'s own XML doc "any Admin/Design user — is system-wide") and enforced at 6+ ManagementActor sites — only `SiteScopeService.ResolveAsync` ignored it. It now treats an Administrator as system-wide regardless of `SiteId` claims, read via explicit `FindAll(RoleClaimType)` + `OrdinalIgnoreCase` (NOT `IsInRole`, which compares ordinally and needs a `roleType` the harness identities don't set); an Administrator's site claims aren't collected at all, preserving *system-wide ⇒ empty permitted-id set*. All 15 UI consumer call sites funnel through `ResolveAsync` (surveyed). Fail-before verified: 6/6 new Administrator pins failed with only `SiteScopeService.cs` reverted; `DesignerWithSiteClaims_IsStillScoped` pins that the bypass was NOT widened to Designer (the XML doc's "Design user" claim was inaccurate — corrected). | Found by the 2026-08-17 Low-tail sweep; fixed with the row-37 consolidation | Closed 2026-08-17. | — |
| 87 | ~~**CLI deployment scoping is non-transactional and unbounded**~~ **RESOLVED 2026-08-17 (merge `ad0100f2`, agent commit `9f67147e`), with the row-37 consolidation.** Site grants now ride as `SiteIdScope` resolved inside the query's join — the whole-table `GetAllInstancesAsync` read is gone (pinned by `..._ReadsNoInstancesAtAll`, a strictly stronger replacement for the old "bulk load exactly once" pin, consciously migrated). An unparseable permitted-site id is *dropped* (narrows the scope) rather than discarding the grant (which would widen to fleet-wide). `PageSize <= 0` no longer means unlimited: it takes the CLI's own default 50, capped at `MaxPageSize` 1000. | Found by the 2026-08-17 Low-tail sweep; fixed with the row-37 consolidation | Closed 2026-08-17. | — |
| 88 | ~~**Unparseable CLI `--status` silently means "no filter"**~~ **RESOLVED 2026-08-17 (merge `ad0100f2`, agent commit `9f67147e`).** An unrecognized non-empty status now produces a `ManagementError` naming the valid values (row-46 principle: loud failure over silent empty/full results) — enforced actor-side so HTTP API callers get it too. Case-insensitive parse retained; numeric strings rejected via `Enum.IsDefined` (`Enum.TryParse` accepts `"7"`). CLI help text documents the valid set; deliberately NOT `AcceptOnlyFromAmong`, whose case-sensitive matching is a live minor defect on `audit query --status` (row 99). **Live-probed on the rig:** `deploy status --status Sucess` → `Unknown deployment status 'Sucess'. Valid values: Pending, InProgress, Success, Failed.` **Postscript 2026-08-17 (Wave 6, `16056f8e`): the "numeric strings rejected via `Enum.IsDefined`" claim was WRONG** — `Enum.IsDefined(parsed)` is true for any valid ordinal, so `--status 2` silently resolved to `InProgress`. Fixed with the same `LooksNumeric` first-character guard the row-99 strict parser uses; live-probed: `deploy status --status 2` now fails loud naming the valid values. | Found by the 2026-08-17 Low-tail sweep | Closed 2026-08-17. | — |
| 89 | ~~**Connection-only artifact deploy does not reach the peer node's DCL**~~ **RESOLVED 2026-08-17 (merge `6cfb6a79`, agent commit `c57e74e6`) — premise corrected TWICE, then fixed.** (1) The standby's DCL is **EMPTY**, not "serving the old endpoint": `DeploymentManagerActor` (role-scoped singleton) is the sole sender of `CreateConnectionCommand` and only populates its own node's `dcl-manager`. The real defect was **create-time and not replication-specific**: `_createdConnections` is per singleton *incarnation*, and `data_connection_definitions` had NO production reader — so any restart/failover rebuilt every connection from the flattened-config snapshots and silently discarded newer connection-only artifact deploys (identical loss on an unreplicated single-node restart). (2) Arbitration decided and shipped: **the definitions store wins wherever it has an opinion; flattened config is authoritative only for names the store never owned** (deploy-only sites unchanged — memo §7.4's objection). A `_connectionDefinitions` overlay refreshes from `ReadStoreSnapshotAsync` at boot AND every sweep tick, applied before instance creation; ping-pong killed by construction (every ensure path resolves the *winner* before hashing, so one config per hash slot). Fail-before: 5/8 new `DclConnectionArbitrationTests` fail with the overlay inert. Decision memo: `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md` (records the accepted residuals: re-adds land at next instance start, not instantly — mirrors the removal's own "bad quality until redeploy" semantic, pinned by test). The discovery that a *live* connection actor is never reconfigured in place on ANY node is a separate pre-existing defect → row 97. **Live rig probe at wave close-out (post-redeploy):** the singleton node logged exactly ONE boot-time `Created DCL connection OPC PLC Simulator` with zero update churn across later sweep ticks (no ping-pong from the overlay), the standby stayed silent (topology as corrected), all three rig-probe-1 tags Good, site-a replication connected / backlog 0. | Found by the 2026-08-17 row-85 fix session | Closed 2026-08-17. | — |
| 90 | ~~**No removal path for a connection deleted on the peer**~~ **RESOLVED 2026-08-17 (merge `6cfb6a79`, agent commit `c57e74e6`), with row 89.** Both removal paths now share `RemoveConnectionCentrally` + an in-memory tombstone that `EnsureDclConnections` honors — which also closed a pre-existing bug where an artifact removal was **undone by the very next instance start**. The sweep detects a previously-definition-sourced row's disappearance via its own overlay bookkeeping. Accepted residuals, documented in the decision memo: tombstones are not durable across a restart (by decision — matches the long-standing restart-resurrection semantic), and a delete is undetectable on a node that never adopted the row (the stronger "non-empty table ⇒ complete set" inference would stop live connections on a guess; deliberately not taken). | Found by the 2026-08-17 row-85 fix session | Closed 2026-08-17. | — |
| 91 | ~~**`SiteScriptCompileCacheTests.GetOrAdd_SameCodeAndGlobals_ComputesOnce` is parallel-load fragile**~~ **RESOLVED 2026-08-17 (merge `106e467d`, agent commit `26e3a1b9`) — row-74 pattern applied (exemplar `98c94588`), and wider than the named test.** Two habits: a per-test-instance guid key token (no other class can collide with or serve this test's entries, making `factoryCalls`/`Assert.Same` exact and owned) + one-sided bounds on the process-global counters (`>= baseline + n`; safe because all `Clear()` callers share the serialized collection). Four assertions changed, two got STRONGER (`GetOrAdd_SameCode_DifferentGlobals` had a second absolute pair; `Overflow_StaysWithinTheBound` gained an eviction probe over its own keys — a slack bound is not proof a sweep ran; its `ForeignEntrySlack = 512` is a judgement call, the eviction probe is the load-bearing assertion). Sibling absolutes in the same class and in `DeploymentWarmThenGateTests` fixed too. SiteRuntime.Tests run twice at full parallelism, 633/633 both. The underlying process-global `Clear()` cross-talk hazard remains → row 100. | Found by the 2026-08-17 row-85 fix session | Closed 2026-08-17. | — |
| 92 | **The event-log cap's size model is sample-based and blind to mid-history width shifts** *(Low, accuracy limitation — documented in memo §3.1)* | Row-81 fix session (2026-08-17) | Row width is sampled 500 rows from each END of `site_events`; a stream whose row width changes sharply mid-history is mis-sized until the trim reaches that region. Self-correcting per tick (each trim exposes new tail rows to the sample), and the model's deliberate +8–13% high bias absorbs modest shifts, but a pathological width profile could over- or under-trim for several ticks. | First field report of the cap trimming visibly wide of its target, or any rework of the size model (see rows 93/94) |
| 93 | **The cap model's page-layout constants are empirical for the CURRENT `site_events` schema** *(Low, maintenance trap)* | Row-81 fix session (2026-08-17) | Fill factors (0.78/0.75), cell overheads (16/8) and especially `IndexedColumnCount = 4` are calibrated to today's table + four indexes. A schema change — new column, new index — silently invalidates them; a fifth index in particular would be silently under-counted. Any `site_events` schema migration must re-run the memo §4 calibration and update the constants together. | Any schema change to `site_events` or its indexes |
| 94 | **`SqliteCapabilityTests` is a deliberate tripwire — do not "fix" it by weakening the assertion** *(Low, standing note)* | Row-81 fix session (2026-08-17) | The test FAILS BY DESIGN if a future SQLite bump ships `dbstat`/`sqlite_dbpage` support. The correct response at that point is to REPLACE the sampled page-layout model with exact `dbstat` measurement (retiring rows 92/93 with it), not to relax the test. The test's own comment says the same. | `SqliteCapabilityTests` fails after a SQLite/provider bump |
| 95 | **The leaked 100 ms-tick test-actor pattern is systemic — three more classes carry it** *(Low, test-infra; row-82 residue)* | Row-82 fix session (2026-08-17): two classes fixed (`PartitionPurgeTests`, `NotifyDispatcherAuditTrailTests`), three more found carrying the same shape | `AuditLogPurgeActorTests` (benign today only because exactly ONE test there touches the fixture DB — adding a second DB-backed test reintroduces row 82), `OutageReconciliationTests`, and `SiteAuditReconciliationActorTests` all create 100 ms-tick actors against a shared fixture DB with no deterministic stop (only timeout-bounded `TestKit.Dispose`, which does not fail on expiry). None failed across ~15 parallel runs, so left untouched for proportionate blast radius. Fix pattern is already written: `Watch`/`Stop`/`ExpectTerminated` in the creating test. | Next touch to any of the three classes, or first flake matching the leaked-actor fingerprint (a prior test's rows mutated mid-test) |
| 96 | **xunit parallelism is uncapped almost everywhere — `MaxParallelThreads` unset, only 3 of ~20 test projects have an `xunit.runner.json`** *(Low, test-infra policy)* | Row-82 fix session (2026-08-17) | Every suite runs at processor-count collection parallelism, and a solution-level `dotnet test` multiplies that across concurrent processes — the general multiplier behind the row-82/91 flake class. The row-82 fix bounds the DB-migration axis specifically; a project-wide parallelism default (or per-suite runner configs for the resource-heavy suites) is a policy decision worth making once, deliberately, rather than per-flake. Supporting evidence from the 2026-08-17 close-out run: `AuditLogPageTests.DrillInFromParentExecution_FiltersGridToSpawnerExecution` (Playwright, untouched by the wave) hit a 30 s navigation timeout under full-suite parallel load and passed isolated in 1 s — one more member of this class. | Next test-infra session, or CI adoption of parallel suite runs |
| 95 | ~~**The leaked 100 ms-tick test-actor pattern is systemic — three more classes carry it**~~ **RESOLVED 2026-08-17 (merge `106e467d`, agent commit `34a8079d`).** All three classes live in AuditLog.Tests (premise correction: none in SiteCallAudit.Tests). `Watch`/`Stop`/`ExpectTerminated` replicated from `da23f2dc` at every creation site: 11 in `AuditLogPurgeActorTests`, 10 in `SiteAuditReconciliationActorTests`, 2 in `OutageReconciliationTests`. Second premise correction: `OutageReconciliationTests` DID call `Sys.Stop` — but fire-and-forget AND from the last line of the `await using` block, so an in-flight tick could still be writing to MSSQL during read-back and any earlier assertion failure skipped the stop entirely; stops moved before the read-backs, plus the `AuditLogIngestActor` that test 2 was discarding outright (holding a disposed provider) is now stopped too. Known lesser sibling habit left in place: EventStream probes are never unsubscribed (bounded by the per-test ActorSystem — noted, not a row). AuditLog.Tests 383/383. | Row-82 fix session (2026-08-17) | Closed 2026-08-17. | — |
| 96 | ~~**xunit parallelism is uncapped almost everywhere**~~ **RESOLVED 2026-08-17 (merge `106e467d`, agent commit `d2d4db49`) — policy made and shipped: per-suite caps for the resource-heavy suites, pure unit suites stay uncapped.** The four MSSQL-fixture users (`ConfigurationDatabase.Tests`, `AuditLog.Tests`, `SiteCallAudit.Tests`, `Host.Tests` — verified before capping) get `xunit.runner.json` with `maxParallelThreads: 4`, **deliberately matching row-82's `sp_getapplock` slot default** so the in-process and cross-process limits agree; no collections serialized. `IntegrationTests` (already fully serialized) and `CentralUI.PlaywrightTests` (already `maxParallelThreads: 1`, stricter than needed) left alone. Cap verified live (216% CPU uncapped vs 145% capped on ConfigurationDatabase.Tests); cost is nil — these suites are latency-bound on MSSQL/sockets (14–25% CPU), durations unchanged within noise. Policy documented in `docs/test_infra/test_infra.md` → *Test Parallelism Policy* (two-axis model, per-suite table, default for new projects), cross-linked from `test_infra_db.md`. Found-not-fixed: `StoreAndForward.Tests/xunit.runner.json` is a no-op (pins the default `parallelizeTestCollections: true`) — harmless, tidy on next touch. | Row-82 fix session (2026-08-17) | Closed 2026-08-17. | — |
| 97 | ~~**A live DCL connection actor is never reconfigured in place — on ANY node**~~ **RESOLVED 2026-08-17 (merge `9dbda4a2`, agent commit `7c1268d8`) — the memo's §6 sketch shipped, plus two things the sketch missed.** `EnsureDclConnection` now has three outcomes off its one hash slot: unchanged → nothing; new → create; **changed → `ReconfigureConnectionCommand`** (arbitration/tombstones byte-for-byte untouched). The manager forwards to the owning actor (unknown name falls back to create); `DataConnectionActor` handles it in **all three** lifecycle states and never stashes it, via a new shared `SwapAdapter` helper now also used by both failover sites. The sketch's two gaps, found in implementation: **endpoint selection must be re-armed to Primary** (a connection sitting on its old *backup* would otherwise dial the new backup first), and the deliberate link drop must **suppress the unstable-disconnect accounting** (else the swap itself could trip a failover). Also a premise-sharpening: "reconfigure while mid-connect" is the *primary* case, not an edge — a wrong endpoint loops in `Connecting` forever and never reaches `Connected`. Fail-before: 5/7 new `DataConnectionReconfigureTests` + 3 SiteRuntime ensure-path tests fail on the reverted behavior. **Live rig probe (post-redeploy, round trip):** connection 1's config changed (operationTimeoutMs 15000→16000→back) fired exactly TWO in-place reconfigures on the active node — `Reconfiguring DataConnectionActor for OPC PLC Simulator in place` → `Reconnected successfully on Primary endpoint` in 45 ms → `Re-subscribing 2 tags` — with all rig-probe-1 tags Good after each, the standby silent (0 reconfigure lines, topology as designed), and the old dead-end "has a new configuration" Warning never appearing. Rig config restored to original. | Found by the 2026-08-17 rows-89/90 fix session; memo `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md` (§9 postscript appended) | Closed 2026-08-17. | — |
| 98 | ~~**`EnforceSiteScopeForInstance` is fail-open for nonexistent instance ids**~~ **RESOLVED 2026-08-17 (merge `152ec85e`, agent commit `f33f25a1`) — premise corrected: the defect had FIVE instances, not two, and the worst three were worse than "harmless".** Beyond the two named helpers, the same `if (target != null) EnforceSiteScope(...)` shape sat **inline** in `HandleGetInstance` (:879), `HandleGetSite` (:1702) and `HandleGetDataConnection` (:1844) — and those returned `ManagementSuccess{JsonData=null}` for a nonexistent id vs `ManagementUnauthorized` for an out-of-scope one, i.e. a working **id-enumeration oracle across the whole fleet** for a site-scoped caller. All five now route through one shared `DenyUnresolvedTargetIfScoped`: a scoped caller's unresolvable target is DENIED, indistinguishable from out-of-scope (no existence leak); system-wide callers (Administrator / system-wide Deployer) are untouched. Also checked: `Instance.SiteId` is non-nullable `int`, so no secondary fail-open path through `EnforceSiteScope(null)`. Fail-before: 5 of the 16 new ManagementActorTests fail on the old code. | Found by the 2026-08-17 rows-37/86-88 fix session (Agent A survey) | Closed 2026-08-17. | — |
| 99 | ~~**CLI `audit query --status` client-side validation is case-sensitive**~~ **RESOLVED 2026-08-17 (merge `152ec85e`, agent commit `f33f25a1`) — premise corrected: "the server would accept it" was FALSE, and removing the client gate alone would have been a regression.** The server-side parser (`AuditQueryParamParsers.ParseEnumList`) is documented-and-tested **lax** — it silently drops unparseable values and collapses to null (filter unconstrained), i.e. exactly the row-46/88 silent-wrong-answer trap. Fix shipped both halves: **seven** `AcceptOnlyFromAmong` gates removed across `audit query`/`audit export` (`--channel`/`--kind`/`--status` on both, plus `export --format`, which rejected `CSV` the server lowercases anyway), valid sets moved to help text, and a new **strict** `ParseEnumListStrict` (throws naming the valid values; `LooksNumeric` guard rejects ordinals) wired into both ManagementService audit endpoints. The lax parser deliberately survives for the CentralUI drill-in/export query strings (bookmarked fragments must render, not fault — documented as non-reconcilable). Left alone with reasons in code: `--direction up|down` (no server parse to defer to), `--format json|table` (CLI-local), NotificationCommands' literals (unsurveyed). Four pre-existing tests were passing *vacuously* (no-credentials exit codes) and now pin the real contract; fail-before 21/26. **Found-and-fixed follow-on (orchestrator, `16056f8e`):** row 88's own `ParseDeploymentStatus` accepted ordinals (`--status 2` → InProgress) — same `LooksNumeric` guard applied + 3 tests. **Live rig probes (post-redeploy):** `audit query --status delivered` returns rows; `--status Nonsense` and `--status 3` both fail loud naming the 8 valid values; `deploy status --status 2` fails loud naming the 4 valid values. | Found by the 2026-08-17 rows-37/86-88 fix session | Closed 2026-08-17. | — |
| 100 | ~~**`SiteScriptCompileCache.Clear()` is process-global — test classes wipe each other's entries**~~ **RESOLVED 2026-08-17 (merge `8a9a8d29`, agent commit `84b405ff`) — premise corrected: there were ZERO production callers of `Clear()`** (the only production entry point is `ScriptCompilationService.CompileCore` → `GetOrAdd`; `Clear()` existed solely for tests), so neither register option was taken as written — `Clear()` was **deleted outright** and the cache became an instance-scoped `sealed class` owned by `ScriptCompilationService`. The N1 "one compile per body per process" property now rests on `AddSiteRuntime` registering that service as a **DI singleton** — a lifetime that became load-bearing, so new `SiteRuntimeServiceRegistrationTests` pins it (a Scoped/Transient downgrade would reintroduce WP3.1's recompile storms with no behavioral test failing). The 37 test sites doing `new ScriptCompilationService(logger)` each silently get their own cache — isolation for the whole assembly, not just the row-91 classes. Row-91 workarounds RETIRED, not preserved: serialized collection removed from all three classes, exact `Hits`/`Count` assertions restored, guid key token gone, `ForeignEntrySlack` 512→8; new `SeparateCaches_DoNotShareEntriesOrCounters` pins the scoping. SiteRuntime.Tests green three consecutive full-parallel runs. | Found by the 2026-08-17 row-91 fix session | Closed 2026-08-17. | — |
| 101 | ~~**`ApplyingArtifacts_RemovedDataConnection_StopsLiveDclConnection` flaked once — suspected overlay-sweep read racing the artifact-apply write**~~ **RESOLVED 2026-08-17 (`0d3a90a9`) — hypothesis VERIFIED, and the production consequence was WORSE than the row guessed.** The race is real: the snapshot is read off the actor thread but compared against the overlay only when the piped message lands, so a read starting before an artifact apply's store writes can land after `HandleApplyArtifactDataConnectionsToDcl` populated the overlay — its stale (even empty) snapshot then "proves" the just-applied names disappeared. Overlay iteration order explains the exact fingerprint (`PlcA` removed first). Two corrections to the row's guesses: (1) it is NOT "self-healing at the next tick" — the removal evicts the hash slot the overlay's push is gated on, so the next fresh sweep clears the tombstone but re-creates nothing; the stopped connection stays dead **until an instance redeploy**; (2) the mid-write "momentarily not seeing the row" framing was the lesser half — the whole-snapshot staleness (read-before-write, land-after-apply) is the dominant window. Fix: `_definitionsWriteGeneration`, bumped on the actor thread by every artifact apply and captured at every snapshot-read dispatch (boot load + sweep); a result carrying a stale generation is discarded WHOLE — additions included, since a stale row pushed at a live connection would reconfigure it backward and flip-flop with the next fresh read. Deterministic repro fails before the fix (`StaleSweepSnapshot_RacingAnArtifactApply_DoesNotRemoveJustAppliedConnections`); `FreshEmptySweepSnapshot_StillRemovesDisappearedConnections` pins that row-90's legitimate disappearance still fires. SiteRuntime.Tests 647/647; rig redeployed, zero spurious removals/discards at boot, all rig-probe-1 tags Good. Memo §10 records the mechanism. | Observed by the 2026-08-17 row-100 fix session | Closed 2026-08-17. | — |
| 68 | ~~**`deploy.sh` run from a git worktree silently rehomes every node's bind-mount data dir**~~ **RESOLVED 2026-08-15 (guard added in the same close-out).** The compose files' bind mounts are relative (`./site-a-node-a/data`), so a deploy from a worktree checkout mounts the worktree's copies — the rig keeps running and looks healthy, but its SQLite stores split from the main checkout's and everything written there is orphaned by the next deploy from main. Bit the rows-46–53 close-out live: the seeding agent's `rig-probe-1` deploy persisted into worktree-homed dirs and "vanished" after the orchestrator's redeploy (no tombstone, no product bug — LWW/tombstone forensics ruled out deletion; the row sat in `.claude/worktrees/…/docker/site-a-node-a/data/site-localdb.db` the whole time). Both `docker/deploy.sh` and `docker-env2/deploy.sh` now refuse to run from a `.claude/worktrees/` path (override: `SCADABRIDGE_ALLOW_WORKTREE_DEPLOY=1`). | Found during the rows-46–53 close-out | Closed 2026-08-15. | — |
## Resolved (verified against the code 2026-07-10)
docs/plans/2026-08-15-instance-sweep-self-heal-design.md
+15 -1
View File
@@ -229,7 +229,8 @@ The register's suggested fix was "ensure a connection actor exists for every sto
connection". Reading the code first changed the shape of the answer, because the premise is not
how the runtime works:
- **`data_connection_definitions` has no production reader.**
- **`data_connection_definitions` has no production reader** *(true when this was written; rows
89/90 gave it one on 2026-08-17 — see the residual note at the end of this section)*.
`SiteStorageService.GetAllDataConnectionDefinitionsAsync` is called from exactly one place in
the repo, and it is a test. Nothing loads that table at boot.
- A DCL connection actor is created **only as a side effect of starting an instance**, from the
@@ -311,6 +312,19 @@ sibling of row 85, it is **not** closed here, and closing it needs the arbitrati
artifact-deploy reconcile; a connection deleted on the peer via replication leaves a live actor on
this node. Same root cause, same arbitration question, same register entry.
> **CLOSED 2026-08-17 (rows 89 + 90) — and the residual above is worded on a wrong premise.**
> The arbitration decision was made and implemented: **the definitions store wins wherever it has
> an opinion; the flattened config is authoritative only for names the store has never owned.** The
> guarded-version-is-near-empty objection above does not apply to that rule, because the guard is
> not "skip names already in `_createdConnections`" but "resolve the WINNER before hashing", which
> leaves exactly one candidate per name and makes the flip unrepresentable rather than merely
> avoided. Two premise corrections came out of validating it: the peer's DCL is **empty**, not
> stale (only the singleton's node ever gets a `CreateConnectionCommand`), so the defect is
> create-time on the next singleton incarnation and is **not replication-specific**; and
> `HandleCreateConnection` is create-only, so a changed config never reached a *live* connection
> actor on any node — that one is still open and is this file's successor's §6.1. See
> `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md`.
### 7.5 Tests
`tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SelfHealCreatesDclConnectionTests.cs` —
docs/plans/2026-08-17-dcl-connection-arbitration-decision.md
+414
View File
@@ -0,0 +1,414 @@
# DCL connection arbitration — which config source owns a live connection
**Date:** 2026-08-17
**Scope:** Site Runtime (`DeploymentManagerActor`), deferred-work-register rows **89** and **90**
**Status:** implemented
**Predecessor:** `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7.4, which designed a
store-level connection sweep, **rejected** it, and recorded rows 89/90 as the residual — explicitly
blocked on "the arbitration decision above (which source wins), not another sweep pass".
---
## 1. Verified topology — and two premise corrections
Both register rows are worded in terms of "the peer node's DCL". That framing is wrong, and the
correction changes what the fix has to do, so it comes first.
### 1.1 What actually runs where
| Actor | Placement | Verified at |
|---|---|---|
| `DeploymentManagerActor` | **Cluster singleton**, role-scoped to `site-{SiteId}` (`SingletonRegistrar.Start(…, "deployment-manager", …, role: siteRole)`) | `Host/Actors/AkkaHostedService.cs` |
| `DataConnectionManagerActor` (`/user/dcl-manager`) | **Every site node**, unconditional `ActorOf` — *not* a singleton | `Host/Actors/AkkaHostedService.cs` |
| `CertStoreActor` | Every site node (documented; re-confirmed) | same file |
`CreateConnectionCommand` and `RemoveConnectionCommand` have exactly **one** sender in the whole
repo — `DeploymentManagerActor` — and it sends them to the `_dclManager` ref it was constructed
with, which is *its own node's* local `dcl-manager` (the Props are built per node, so whichever
node materialises the singleton uses that node's manager).
### 1.2 Premise correction 1 — the standby's DCL is EMPTY, not stale
Because the singleton is the only sender, **the standby node's `dcl-manager` holds no connection
actors at all**. It is not "serving the old endpoint"; it is serving nothing, and nothing subscribes
through it. So there is no per-node divergence to reconcile with a cross-node message.
The real failure mode is one layer over:
> `_createdConnections` — the per-name hash cache that records what the DM last pushed — is
> **in-memory and per singleton incarnation**. When the singleton restarts (failover, or an
> ordinary node restart), the new incarnation starts with an empty cache, loads
> `deployed_configurations`, and rebuilds every connection **from the `Connections` snapshot frozen
> into each instance's flattened config at that instance's deploy time**. Nothing has ever read
> `data_connection_definitions` — that table had **no production reader** — so a newer,
> connection-only artifact deploy is silently discarded and the site goes back to the old endpoint.
This also means **row 89 is not replication-specific**. Replication only puts the definition row on
both nodes, which is if anything helpful. The identical loss happens on an unreplicated single-node
restart. The row-89 fix is therefore a *create-time arbitration* fix, not a *message-delivery* fix.
Row 90 is the same shape: the delete is durable in the store, but the flattened configs still name
the connection, so the next instance start re-creates it.
### 1.3 Premise correction 2 — a changed config never reached a LIVE connection either
`EnsureDclConnection`'s contract, as documented in the code before this change, was that a changed
endpoint/credential *"re-issues a `CreateConnectionCommand` so the DCL adopts the new configuration
instead of keeping a stale connection"*. **That is false.**
`DataConnectionManagerActor.HandleCreateConnection` is create-only:
```csharp
if (_connectionActors.ContainsKey(command.ConnectionName))
{
_log.Warning("Connection {0} already exists", command.ConnectionName);
return;
}
```
So the system-wide artifact deploy's "make it live immediately" push has never worked on the
deploy-handling node, and the DM's hash cache recorded it as applied regardless. This is a
**separate, pre-existing defect** discovered while validating rows 89/90; see §6.
---
## 2. The arbitration rule
> **The definitions store wins wherever it has an opinion. The flattened config is authoritative
> only for names the store has never owned.**
1. **A name with a row in `data_connection_definitions`** — that row's config configures the live
DCL connection actor. The identically-named entry in an instance's flattened `Connections` map
is a deploy-time snapshot and never overrides it.
2. **A name with no definition row** — the flattened config is the source, exactly as before. This
is not a fallback; it is the *correct* answer for a deploy-only site, where the definitions
table is legitimately empty because nothing has ever sent that site a system-wide artifact
deploy while every connection that matters lives inline in the instance configs. §7.4 of the
predecessor memo identified this as the reason a blanket store-level sweep is wrong, and it
remains the reason rule 2 exists.
3. **Removal** semantics on the deploy-handling node are unchanged: the actor is stopped even if
deployed instances still reference it, and their attributes go bad quality until redeploy. What
is *added* is a tombstone (§4).
### 2.1 The overlay is an arbiter, not a creation source
`_connectionDefinitions` decides **which configuration** a connection the runtime already wants is
built from. It never creates a connection on its own: a definition row naming a connection no
deployed instance references produces no actor, because none is wanted. This is what makes it safe
to consult on every sweep tick, and it is why re-adding a centrally deleted connection brings the
actor back at the *next instance start* rather than instantly (pinned by
`ReAddedDefinition_ClearsTheTombstone_SoTheNextInstanceStartRecreatesIt`).
---
## 3. The ping-pong, and why keying the hash on the winner makes it unrepresentable
The predecessor memo rejected a store-level sweep partly because of this:
> `_createdConnections` holds one hash per name. The flattened-config hash and the
> artifact-definition hash for the same connection can legitimately differ (the supported
> version-skew case). A sweep re-hashing from the definitions table would flip the slot every tick
> against the deploy path flipping it back, **recreating the connection actor and dropping its live
> subscriptions on a loop.**
That analysis is correct, and it is a property of *where the hash is computed*, not of the sweep.
The fix is one line of ordering discipline:
> **Every ensure path resolves the winning configuration BEFORE hashing or pushing.**
`EnsureDclConnections(configJson)` no longer hands the flattened `ConnectionConfig` straight to
`EnsureDclConnection`. It first asks the overlay whether the store owns that name; if it does, the
*definition's* protocol/config/backup/failover fields are what get hashed and pushed. The
artifact-deploy path pushes those same fields. Therefore, for any given name, **only one
configuration can ever be hashed**, no matter which path calls — the two sources cannot occupy the
slot alternately because there is only ever one candidate. The flip is not guarded against; it is
unrepresentable.
The guard test `DisagreeingSources_DoNotPingPongTheConnection_AcrossManySweeps` pins exactly this
with the two sources deliberately disagreeing: one create, zero removes, across many ticks. It
passes before *and* after the fix — before, because nothing read the definitions at all — which is
the point: it is a guard against the **rejected naive design**, not a fail-before test.
### 3.1 A second flip source, closed in the same way
`HandleApplyArtifactDataConnectionsToDcl` now writes the artifact's rows into the overlay in the
same pass it pushes them. Without that there is a window between an artifact deploy and the next
sweep read in which an instance start would resolve the name to its flattened snapshot while the
artifact path had already pushed the definition — the same flip, arriving through a different door.
---
## 4. Removal and the tombstone
`_centrallyRemovedConnections` is a `HashSet<string>` of names this singleton incarnation has seen
removed centrally. It is populated by both removal paths, which share one helper
(`RemoveConnectionCentrally`) so they cannot drift:
- the artifact reconcile's `RemovedConnectionNames` (the existing path); and
- **the sweep's disappearance detection** (new — §5).
`EnsureDclConnections` skips a tombstoned name. Without the tombstone the removal does not survive
one tick: the removal evicts the hash slot, and the next instance start whose flattened config still
names the connection re-creates it. That is not hypothetical — the sweep starts instances — and it
is a pre-existing bug this change also closes
(`ArtifactReconcileRemoval_IsNotResurrectedByALaterInstanceStart` fails on the old code).
**Tombstones are deliberately in-memory only.** Persisting them was rejected: a durable "this name
is gone" marker is a second divergent authority over the same names, and the flattened configs
already resurrect a removed connection across a restart today. That pre-existing semantic is
accepted and unchanged — see the residual in §6.
A tombstone is cleared the moment a definition row for the name reappears (a definition row *is*
the store re-asserting ownership), so a centrally re-added connection is creatable again.
---
## 5. The mechanism: the definitions table as an overlay on the existing reads
No new message, no new actor, no new cross-node path. The two durable tables the DM arbitrates over
are read in **one off-thread task**, `ReadStoreSnapshotAsync`, used by both existing reads:
```
PreStart → LoadDeployedConfigs ─┐
├─ ReadStoreSnapshotAsync (off-thread, one task)
InstanceSweepTick ──────────────┘ deployed_configurations
data_connection_definitions
│
▼ piped back
StartupConfigsLoaded / InstanceSweepLoaded (actor thread)
│
ApplyConnectionDefinitionOverlay
│
┌───────────────────┴───────────────────┐
row present row disappeared
adopt + push if live tombstone + stop actor
```
Nothing blocks the actor thread on SQLite; the overlay mutation, the hash cache and the tombstone
set are all actor-thread-confined, matching the existing `StartupConfigsLoaded` /
`ApplyArtifactDataConnectionsToDcl` patterns.
**Ordering is load-bearing in both handlers.** The overlay is applied *before* the boot batch
creates any Instance Actor and *before* the sweep's repair loop, so a connection is created from the
winning config on its **first** push rather than created stale and corrected a tick later. On
failover that is the difference between a clean takeover and a needless recreate on every one.
### 5.1 The two reads fail differently, on purpose
A failed **config** read propagates and re-arms the whole load (the existing S5 path, unchanged). A
failed **definitions** read is swallowed into a `null` result, and `null` means *keep the overlay
exactly as it is* — deliberately distinct from an empty list, which is a real answer (a deploy-only
site, or a site after central deletes every data connection). Clearing the overlay on a transient
read failure would hand arbitration back to the flattened snapshots for one pass and take it away
on the next successful read: the §3 ping-pong, reintroduced through the error path.
### 5.2 Disappearance detection, and precisely how far it reaches
A name that **was** in the overlay and now has no row is a removal this incarnation can *prove*: the
store owned it a moment ago and does not now. It is tombstoned and its connection actor stopped.
A name that was **never** in the overlay is indistinguishable from one the store has never owned —
a fresh incarnation cannot tell "central deleted this" from "this site keeps its connections
inline". It is left to the flattened config. That asymmetry is deliberate; the alternative (infer
deletion from "the definitions table is non-empty, therefore it is the complete set") would stop
live connections on an inference, and is recorded as a candidate follow-up in §6.
### 5.3 Cost
One extra `SELECT` over `data_connection_definitions` per sweep tick (default 60 s) and once at
boot, on the same task as the existing deployed-configs read. Steady state is otherwise silent: the
hash guard means an unchanged connection produces no message and no log.
---
## 6. Residuals — what this does NOT close
These are precise, and each is a candidate register row.
1. **A live connection actor is still not reconfigured in place** (premise correction 1.3). The DCL
manager's `HandleCreateConnection` is create-only, so a changed configuration for a name whose
actor already exists takes effect only when the connection is next created from scratch — a node
restart, a singleton failover, or the connection being removed and re-added. This change makes
that **honest** rather than silent: `EnsureDclConnection` now logs a Warning naming exactly that,
once per changed configuration (the hash guard suppresses repeats, so a healthy site stays
silent). It is not *fixed* here because the obvious fix is destructive: stopping and re-creating
the connection actor drops every Instance Actor's tag subscription, and `InstanceActor` cancels
its per-connection subscribe-retry timer on the first success, so **nothing re-subscribes** —
trading "stale endpoint, data flowing" for "correct endpoint, no data until redeploy". The
non-destructive fix is a `ReconfigureConnectionCommand` handled *inside* `DataConnectionActor`,
reusing the adapter-swap machinery the primary→backup failover path already has
(`CountFailureAndMaybeFailover`: dispose adapter, `_factory.Create` the new one, bump
`_adapterGeneration`, re-enter the connect cycle, which re-issues everything from
`_subscriptionsByInstance`). That is a Data Connection Layer change and is out of rows 89/90's
scope.
2. **A removal is still not durable across a restart.** Tombstones are in-memory, so a connection
deleted centrally and then resurrected by an instance's flattened config on the next node restart
behaves as it always has. Closing this needs a durable marker or — better — flattened configs
that stop carrying connection bodies at all.
3. **A delete that arrives while this node has never adopted the row is undetectable** (§5.2). The
narrow case that remains open: the singleton moves to a node whose file already has the row
deleted, and some instance's flattened config still names the connection. That node re-creates
it. A stronger rule — "a non-empty definitions table is the complete set, so absence means
deleted" — would close it, at the cost of stopping live connections on an inference about what
central shipped; deliberately not taken.
4. **Dual-active (a real partition under `auto-down`)** is the only routine way a definition change
reaches a singleton that did not process the deploy itself. It is handled correctly by §5, but it
is worth naming that this is not the common path — the common path is failover/restart (§1.2).
---
## 7. Tests
`tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DclConnectionArbitrationTests.cs` — 8 tests,
asserting on the `CreateConnectionCommand` / `RemoveConnectionCommand` messages the DM sends to a
recording stand-in for the DCL manager, and on the **endpoint inside** those commands (which is what
distinguishes the two config sources).
Every test gates on `DeploymentManagerStartupProbe` for the same reason the row-78/85 suites do: a
row written before the boot load reads storage is consumed by the boot load, and an ungated test
would silently prove the boot load works instead of the path it names.
Fail-before evidence (behavioral revert: `ApplyConnectionDefinitionOverlay` made inert and the
tombstone check disabled — i.e. the exact pre-fix arbitration) — **5 fail, 3 pass**:
| Test | Pre-fix |
|---|---|
| `DefinitionRow_WinsOverAnOlderFlattenedSnapshot_WhenTheSingletonBoots` | **FAIL** — the row-89 symptom in its failover shape (create carries the old endpoint) |
| `DefinitionArrivingAfterBoot_IsAdoptedByTheNextSweepTick` | **FAIL** — the CDC-delivered change is never read |
| `DefinitionRowDisappearing_StopsTheConnection_AndItIsNotResurrected` | **FAIL** — the row-90 symptom (orphan actor stays live) |
| `ArtifactReconcileRemoval_IsNotResurrectedByALaterInstanceStart` | **FAIL** — removal undone by the next instance start |
| `ReAddedDefinition_ClearsTheTombstone_SoTheNextInstanceStartRecreatesIt` | **FAIL** |
| `DisagreeingSources_DoNotPingPongTheConnection_AcrossManySweeps` | pass (guard against the rejected naive design) |
| `NoDefinitionRow_LeavesTheFlattenedConfigAuthoritative_AtBoot` | pass (deploy-only regression guard) |
| `NoDefinitionRow_LeavesTheFlattenedConfigAuthoritative_OnTheSweepPath` | pass (deploy-only regression guard) |
The three passers are the point of passing both before and after: rule 2 must be byte-for-byte
unchanged for a deploy-only site, and the ping-pong must never appear.
---
## 8. Configuration
**None added.** The overlay refresh rides `ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds`
(default 60; `0` disables the sweep and therefore the post-boot half of this convergence — the
boot-time arbitration still applies, because it hangs off the startup load, not the sweep timer).
---
## 9. Postscript — residual 1 RESOLVED (register row 97, 2026-08-17)
*Appended; §6 above is left exactly as it was written.*
§6 residual 1 — "a live connection actor is still not reconfigured in place" — is closed, along
the sketch §6 recorded, in a Data Connection Layer change that leaves everything above untouched.
**The arbitration rule, the winner-resolution ordering and the tombstones are unchanged. Only what
happens AFTER a change is detected changed.**
### 9.1 The third command
`EnsureDclConnection` had two outcomes off its single per-name hash slot; it now has three:
| Hash slot | Command sent | Log |
|---|---|---|
| equal | **nothing** | none — what keeps a healthy site silent every sweep tick |
| absent | `CreateConnectionCommand` | `Created DCL connection {Connection} …` (Information) |
| **different** | **`ReconfigureConnectionCommand`** | `Reconfigured DCL connection {Connection} (protocol={Protocol}) in place …` (Information) |
The Warning §6 added to make the gap honest ("…is not reconfigured in place — it keeps serving its
current configuration until it is recreated") is **gone**, because the sentence is no longer true.
`DataConnectionManagerActor.HandleReconfigureConnection` **forwards** the command to the connection
actor that owns the name, and **falls back to a create** for a name it does not hold — so the
command is safe whatever the manager's state, and a sender whose view of which connections exist is
stale (a fresh incarnation, a node that has just taken over) still converges in one message.
`HandleCreateConnection` itself is untouched and stays create-only.
### 9.2 Inside the connection actor
`DataConnectionActor` handles the command in **all three** states, and **never stashes it**. That is
load-bearing rather than tidy: a connection pointed at a wrong or dead endpoint spends its life in
`Connecting` or `Reconnecting` — exactly when a corrected configuration is sent — so a stashed
reconfigure would wait for the *broken* configuration to succeed first.
The swap itself is the failover machinery, not a copy of it: the two failover sites and the
reconfigure now share one `SwapAdapter(newConfig)` helper (unhook + fire-and-forget dispose →
`_factory.Create` → rewire → bump `_adapterGeneration` **last**, so everything in flight on the
outgoing adapter is recognised as stale). Then `BecomeReconnecting(afterReconfigure: true)`:
- reconnects **immediately** rather than after `ReconnectInterval` — there is nothing to wait for,
and `StartSingleTimer` replaces any reconnect already armed, so a reconfigure landing mid-backoff
cannot leave two attempts racing;
- **suppresses the unstable-disconnect accounting and the failover check** — the link was dropped on
purpose, and counting it as an unstable cycle could trip a failover onto the *new* backup before
the new primary had been tried once;
- logs the site event at Info ("Connection reconfigured — reconnecting on the new configuration")
instead of the Warning "Connection lost".
`ReSubscribeAll` / `ReSubscribeAllAlarms` then re-issue everything from `_subscriptionsByInstance`
and `_alarmSourceSubscribers`, which the swap never touched. **Subscribers do nothing and know
nothing**: tags go bad quality across the swap and come back Good, exactly as on a failover.
Two edge cases §6 did not name:
1. **Endpoint selection is re-armed from the top.** Whatever endpoint the previous configuration had
failed over to, the new configuration's *primary* is the operator's intent — so `ActiveEndpoint`
and both failure counters reset. Without this, a connection sitting on its old backup would apply
a new configuration and immediately connect to the new *backup*.
2. **The command is idempotent.** A reconfigure carrying the configuration already in force is a
no-op inside the actor. The sender is hash-gated and does not send one, but an adapter swap drops
the link and re-subscribes every tag, so a duplicate must never be able to cost that.
A **protocol change** is supported for free: `_protocolType` became mutable and the factory keys off
it.
### 9.3 What this cost the ping-pong guard
§3's guard test asserted "one create, zero removes, across many ticks". A flip would now show up as
a stream of **reconfigures**, not creates — each one dropping the link and re-subscribing every tag.
The guard asserts on both counts now; asserting only on creates would have left it silently
guarding the wrong thing.
### 9.4 Tests
`tests/…/DataConnectionLayer.Tests/Actors/DataConnectionReconfigureTests.cs` — 7 tests. Fail-before
evidence (behavioral revert: the three `ReconfigureConnectionCommand` cases removed from the state
machine and the manager's forward replaced by the create-only path — i.e. the exact pre-fix
behaviour) — **5 fail, 2 pass**: the swap-and-re-subscribe from `Connected`, the generation bump,
the honoured-not-stashed cases in `Connecting` and in `Reconnecting`, and the manager's forward all
fail; the identical-config no-op and the unknown-name fallback-to-create pass, as guards.
`DclConnectionArbitrationTests` gained two (`ChangedConfigurationForALiveConnection_IsSentAsA
Reconfigure_NotASecondCreate`, `UnchangedConfiguration_SendsNeitherACreateNorAReconfigure`) and
`DefinitionArrivingAfterBoot_IsAdoptedByTheNextSweepTick` now asserts the second push is a
reconfigure. With `EnsureDclConnection` reverted to sending a create on the change edge, **3 fail**
across `DclConnectionArbitrationTests` + `DeploymentManagerMediumFindingsTests` (the two changed-edge
tests plus SiteRuntime-010, renamed `…_ReissuesReconfigureCommand`).
### 9.5 Residuals after this
§6 residuals **2** (a removal is not durable across a restart) and **3** (a delete is undetectable on
a node that never adopted the row) are unchanged and still open. §6 residual **4** was never a gap.
## 10. Postscript — the stale-snapshot race (register row 101, 2026-08-17)
The sweep design read the store off the actor thread (§5) and compared the result against the
overlay only when the piped message landed. That gap was a TOCTOU the original memo did not name:
a read starting before an artifact apply's store writes could land AFTER
`HandleApplyArtifactDataConnectionsToDcl` populated the overlay, and its stale snapshot would then
"prove" the just-applied names disappeared. The disappearance detection (§5) tombstoned them and
stopped the just-created live connection actors — and because the removal also evicts the hash
slot the overlay's push is gated on, the next fresh sweep clears the tombstone but re-creates
nothing: **the connection stays dead until an instance redeploy**. Surfaced as a one-off test flake
(`ApplyingArtifacts_RemovedDataConnection_StopsLiveDclConnection` received a removal for `PlcA`
where `PlcGone` was expected — overlay iteration order removing BOTH names off a stale empty
snapshot), first suspected to be test-fixture-local; verification showed the mechanism is real in
production whenever an artifact deploy lands while a sweep read is in flight.
**Fix: a definitions-write generation.** `_definitionsWriteGeneration` is bumped on the actor
thread by every artifact apply; each snapshot read captures it at dispatch and a result carrying a
stale generation is discarded WHOLE — additions included, since a stale row pushed at a live
connection would reconfigure it backward and flip-flop with the next fresh read. Discarding is
safe: the apply already updated the overlay in the same pass (§5), and the next tick reads fresh.
Applies to both snapshot consumers (boot load and sweep). Deterministic reproduction + a guard
test that the legitimate row-90 disappearance still fires on a FRESH empty snapshot:
`StaleSweepSnapshot_RacingAnArtifactApply_DoesNotRemoveJustAppliedConnections` (fails before the
fix) and `FreshEmptySweepSnapshot_StillRemovesDisappearedConnections`.
docs/plans/sql/AddDeploymentRecordListSortIndex.sql
+4 -2
View File
@@ -2,8 +2,10 @@
--
-- Replaces the single-column [IX_DeploymentRecords_DeployedAt] with the composite
-- [IX_DeploymentRecords_DeployedAt_Id] ([DeployedAt] DESC, [Id] DESC), which is exactly
-- the order DeploymentManagerRepository.QueryDeploymentListPageAsync (and
-- QueryDeploymentSummariesAsync) asks for. With both key columns descending, a forward
-- the order DeploymentManagerRepository.QueryDeploymentListPageAsync asks for. (It named
-- QueryDeploymentSummariesAsync too; register row 37 folded that second paged read into
-- QueryDeploymentListPageAsync, so both the UI and the CLI now share this one key.)
-- With both key columns descending, a forward
-- scan delivers the requested order, so a paged Skip/Take reads a Top-N off the index
-- instead of sorting the whole filtered set on every page request.
--
docs/requirements/Component-CentralUI.md
+1
View File
@@ -149,6 +149,7 @@ Central cluster only. Sites have no user interface.
- `IDeploymentManagerRepository.QueryDeploymentListPageAsync(filter, pageNumber, pageSize)` returns one page of `DeploymentListRow` — `DeploymentRecord` **inner-joined to `Instance`**, so the instance's display name and site travel with the rows that need them (the join is exact: the FK is `Restrict` and instance deletion removes the records first) — plus the **total count** of the filtered set.
- `GetDeploymentStatusCountsAsync(filter)` returns the tile counts from **one grouped aggregation**. It deliberately **ignores the filter's `Status`** — the tiles are the status *breakdown* of the otherwise-identically-filtered set, so each keeps its own total while it is the selected one.
- **Site scoping runs in the query.** The permitted-site grant is pushed in as `SiteIdScope` and resolved through the record's instance; an **empty** grant is a real filter matching nothing, never "unconstrained". `DeploymentRecord` has no `SiteId` of its own — the join is what makes this expressible.
- **This is now the ONLY paged read of `DeploymentRecords`.** The CLI/HTTP `QueryDeployments` command had a second one (`QueryDeploymentSummariesAsync`: offset instead of page number, no total, no instance join, instance-id-list scoping); it is **deleted**, and both surfaces share this query. See `Component-ManagementService.md` → *QueryDeployments* for the three behavioral drifts the duplication had accumulated (deferred-work register rows 37, 86–88).
- **Offset paging, not the Audit Log's keyset cursor — deliberately.** This page's pager is numbered and jump-to-any-page, so it needs a page count, which only a total can give it; a keyset cursor can express neither. The total is required for the tiles regardless. The deep-offset cost that pushes high-volume tables to keyset is bounded here by the terminal-record retention purge, unlike the 365-day central `AuditLog`. This mirrors the Notification Outbox, which is offset-paged for the same reason. Ordering is `DeployedAt DESC, Id DESC`; the `Id` tie-break is load-bearing, because `DeployedAt` ties on rapid redeploys and an unstable sort key makes offset paging repeat or drop rows between pages.
- The whole-table `GetAllDeploymentRecordsAsync` read was **deleted** with its last caller.
- **Filtering.** Status filter (the tiles double as the control — clicking one applies it, clicking it again clears) plus a free-text search matched DB-side against instance unique name, deployment id, revision hash and initiating user. A filter change resets to page 1. Search input is **trailing-edge debounced at 500 ms**, so a typed term is one query rather than one per keystroke.
docs/requirements/Component-DataConnectionLayer.md
+21 -2
View File
@@ -147,6 +147,20 @@ flowchart TD
See [`2026-03-22-primary-backup-data-connections-design.md`](../plans/2026-03-22-primary-backup-data-connections-design.md) for the full design.
### In-place reconfiguration (register row 97)
A `ReconfigureConnectionCommand` replaces a **live** connection's configuration without stopping the connection actor. It is the same adapter swap the failover path above performs — the two share one `SwapAdapter` helper — so the diagram applies verbatim from *dispose adapter* onwards:
1. The command lands on the actor and **is never stashed**, in *any* of the three states. That is load-bearing, not tidiness: a connection whose configured endpoint is wrong or dead spends its life looping in `Connecting` or `Reconnecting`, which is precisely when a corrected configuration is sent. Stashed, it would wait for the *old* configuration to succeed — i.e. forever.
2. Protocol, primary/backup details and failover retry count are replaced wholesale, and **endpoint selection is re-armed from the top**: whatever endpoint the previous configuration had failed over to, the new configuration's primary is the operator's intent, so `ActiveEndpoint` and both failure counters reset.
3. `SwapAdapter` unhooks and disposes the outgoing adapter (fire-and-forget, never awaited on the actor thread), builds the replacement via `IDataConnectionFactory.Create` — which is why a **protocol change** is supported too — and bumps `_adapterGeneration` **last**, so every callback and in-flight background task belonging to the outgoing adapter is recognised as stale and dropped rather than delivered as if it came from the new endpoint.
4. The actor re-enters the reconnect cycle immediately (no `ReconnectInterval` wait — there is nothing to wait for; `StartSingleTimer` replaces any reconnect already armed, so a reconfigure landing mid-backoff cannot leave two attempts racing). The unstable-disconnect accounting and the failover check are **suppressed** for this transition: the link was dropped on purpose, and counting it as an unstable cycle could trip a failover onto the new backup before the new primary had been tried once.
5. On connect, `ReSubscribeAll` / `ReSubscribeAllAlarms` re-issue everything from `_subscriptionsByInstance` and `_alarmSourceSubscribers`, which the swap never touched. **Subscribers do nothing and know nothing** — the tags go bad quality across the swap, exactly as they do on a failover, and come back Good.
**Accepted consequences**, all identical to a failover: in-flight writes against the outgoing adapter may fail (returned synchronously to the calling script, as always), and tag quality dips for the duration of the reconnect.
**Idempotent.** A reconfigure carrying the configuration already in force is a no-op. The sender is hash-guarded and does not send one, but a swap is destructive enough that a duplicate must never cost a reconnect.
## Connection Configuration Reference
All settings are parsed from the data connection's configuration JSON dictionaries (`PrimaryConfiguration` and optional `BackupConfiguration`, stored as `IDictionary<string, string>` connection details). Both endpoints use the same protocol-specific keys. Invalid numeric values fall back to defaults silently.
@@ -355,9 +369,13 @@ Each value update delivered to an Instance Actor includes:
## Connection Actor Model
> **Where connection actors come from — the Site Runtime is the only creator (register row 85)**: the DCL manager creates and removes connection actors **only** when the Deployment Manager tells it to, via `CreateConnectionCommand` / `RemoveConnectionCommand`. It never reads storage; the site's `data_connection_definitions` table has no production reader at all, and the DCL has no independent notion of "the connections this site ought to have". A connection actor's config therefore comes from the `Connections` map inside a deployed instance's **flattened config** (pushed as part of starting that instance) or from the system-wide artifact-deploy path. `HandleCreateConnection` is idempotent on the name — it warns and returns if it already holds that connection — but the Deployment Manager is the side that keeps the traffic quiet, hash-guarding on the last config pushed per name so an unchanged connection sends nothing. Row 85 was the consequence of that ownership split being incompletely honoured on the Site Runtime side: two of the four Instance-Actor start paths omitted the connection push, so instances came up whose every tag subscribe was answered `Unknown connection: <name>` here, forever. The fix is in the Site Runtime (the ensure moved into its single creation funnel) — **not** here; giving the DCL manager its own config source would create a second, divergent authority with no arbiter when the two disagree. See `Component-SiteRuntime.md` and `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7.
> **Where connection actors come from — the Site Runtime is the only creator (register row 85)**: the DCL manager creates and removes connection actors **only** when the Deployment Manager tells it to, via `CreateConnectionCommand` / `RemoveConnectionCommand`. It never reads storage, and the DCL has no independent notion of "the connections this site ought to have" — the site's `data_connection_definitions` table is read by the **Deployment Manager**, never here (see the arbitration note below; it had no reader at all until register rows 89/90). A connection actor's config therefore comes from the `Connections` map inside a deployed instance's **flattened config** (pushed as part of starting that instance) or from the system-wide artifact-deploy path. `HandleCreateConnection` is idempotent on the name — it warns and returns if it already holds that connection — but the Deployment Manager is the side that keeps the traffic quiet, hash-guarding on the last config pushed per name so an unchanged connection sends nothing. Row 85 was the consequence of that ownership split being incompletely honoured on the Site Runtime side: two of the four Instance-Actor start paths omitted the connection push, so instances came up whose every tag subscribe was answered `Unknown connection: <name>` here, forever. The fix is in the Site Runtime (the ensure moved into its single creation funnel) — **not** here; giving the DCL manager its own config source would create a second, divergent authority with no arbiter when the two disagree. See `Component-SiteRuntime.md` and `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7.
>
> Two known residual gaps of the same family, **not** closed by row 85's fix: `data_connection_definitions` is one of LocalDb's ten replicated tables, so a **connection-only** artifact deploy replicates to the peer node — but only the node that received the deploy pushes it to its DCL, so the peer keeps serving the old endpoint until an instance is redeployed or the node restarts. The removal side is the mirror image: a connection deleted on the peer leaves a live connection actor on this node.
> **The two residual gaps of that family are now closed — but not where the register said they were (rows 89 + 90, 2026-08-17)**: the peer node's DCL was never "serving the old endpoint". Only the node running the `site-{SiteId}` Deployment Manager singleton ever receives a `CreateConnectionCommand`, so the standby's manager is **empty**. The divergence surfaces when the singleton's *next* incarnation (failover, or an ordinary restart — no replication required) rebuilds every connection from the `Connections` snapshots frozen into the instances' flattened configs, because the per-name hash cache that recorded the artifact deploy's change is in-memory and dies with the incarnation. The fix lives entirely in the Site Runtime, as an **arbitration rule** — the definitions store wins wherever it has an opinion, resolved *before* the hash is computed so the two sources cannot fight over the same per-name slot — plus in-memory tombstones so a centrally removed connection is not resurrected by a flattened config. Nothing changes here. See `Component-SiteRuntime.md` → *DCL connection arbitration* and `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md`.
>
> **A live connection IS reconfigured in place, via a third command (register row 97, 2026-08-17).** `HandleCreateConnection` remains **create-only** — it warns and returns for a name it already holds — so the Deployment Manager's documented "a changed config re-issues the create so the DCL adopts it" was never true, on any node: a corrected endpoint/credential/backup/failover-count took effect only when the connection was next built from scratch (node restart, singleton failover, remove-then-re-add), while the DM's hash cache recorded it as applied. The Site Runtime now sends a **`ReconfigureConnectionCommand`** on the change edge instead. The manager **forwards** it to the connection actor that owns the name, and falls back to a create for a name it does not hold — so the command is safe whatever the manager's state and a sender never has to know whether the actor exists.
>
> **Stop-and-recreate was rejected, and the reason is worth keeping.** Subscriptions are held inside `DataConnectionActor`, and `InstanceActor` cancels its per-connection subscribe-retry timer on the first success — so a rebuilt connection actor would have *no* subscribers and nothing would ever re-subscribe. That trades "stale endpoint, data flowing" for "correct endpoint, no data until redeploy". See *In-place reconfiguration* below for what the actor does instead.
Each data connection is managed by a dedicated connection actor that uses the Akka.NET **Become/Stash** pattern to model its lifecycle as a state machine:
@@ -432,6 +450,7 @@ The DCL reports the following metrics to the Health Monitoring component via the
## Interactions
- **Site Runtime (Deployment Manager)**: Receives `CreateConnectionCommand` / `ReconfigureConnectionCommand` / `RemoveConnectionCommand`. It is the sole creator, reconfigurer and remover of connection actors; the DCL never reads storage.
- **Site Runtime (Instance Actors)**: Bidirectional — delivers value updates, receives subscription registrations and write-back commands.
- **Site Runtime (NativeAlarmActor)**: Bidirectional — receives alarm subscribe/unsubscribe requests, delivers native alarm transitions and source-unavailable notifications (read-only; no ack-back to the source).
- **Health Monitoring**: Reports connection health periodically.
docs/requirements/Component-ManagementService.md
+27 -1
View File
@@ -83,6 +83,12 @@ In addition to `/management`, the Management Service exposes a dedicated REST su
Both endpoints honour any site-scope rules attached to the caller's audit role by intersecting the caller-supplied `sourceSiteId` filter with the user's `PermittedSiteIds` (out-of-scope requests yield HTTP 403). Permission denial returns HTTP 403 with the same envelope shape used by `/management`.
**Filter parsing is strict on the enum dimensions** (`channel`, `kind`, `status`; register row 99). Values are matched **case-insensitively** against the enum member names, and an unrecognised value returns **HTTP 400** `{ error, code: "BAD_REQUEST" }` naming the full valid set — it is not silently dropped. Silently dropping a filter would leave that dimension unconstrained and hand the caller a plausible, complete-looking page that does not answer the question they asked (the same failure row 88 fixed for `deploy status`). **Ordinals are rejected**: a filter is a name-based contract, so `status=3` is an error even though `Enum.TryParse` would accept it and `3` is a real member. The `correlationId`/`executionId`/`parentExecutionId`/date dimensions stay **lax** (unparseable values are dropped).
This is the *authoritative* check for those three dimensions — the CLI deliberately carries no client-side vocabulary gate. It used to (`AcceptOnlyFromAmong`), but that check matches **case-sensitively**, so `audit query --status delivered` failed at the CLI before ever reaching a server that would have accepted it. The valid set now lives in the CLI's option help text only.
> The CentralUI `AuditLogPage` drill-in parser and the CentralUI export endpoint it feeds deliberately keep the **lax** parse. They read a query string a human may have bookmarked or hand-edited, where a stale fragment must still render the page rather than fault it. Both contracts live side by side in `AuditQueryParamParsers` (`ParseEnumList` vs `ParseEnumListStrict`); do not reconcile them to one.
## Message Groups
### Templates
@@ -147,7 +153,14 @@ The whole folder surface is reachable from the CLI as `template folder list|crea
- **DeployArtifacts**: Deploy system-wide artifacts (shared scripts, external system definitions, DB connections, data connections) to all sites or a specific site. The command's `SiteId` is honored (arch-review C2): a value routes to the single-site deploy path (`ArtifactDeploymentService.DeployToSiteAsync`), while `null` deploys fleet-wide (`DeployToAllSitesAsync`). Site scope is enforced — a site-scoped (non-Administrator) Deployer may only target a site within its `PermittedSiteIds`, and may **not** deploy fleet-wide (`SiteId is null` is rejected with `SiteScopeViolationException` → `ManagementUnauthorized`, since `EnforceSiteScope(null)` is a deliberate no-op); fleet-wide deployment requires a system-wide Deployer or Administrator.
- **DeploySite** (`MgmtDeploySite`, WP2.5): Bulk-deploy every deployable instance at one site — see the Instances section above and `Component-DeploymentManager.md`.
- **GetDeploymentStatus**: Query deployment status.
- **QueryDeployments** (`InstanceId?`, `Status?`, `Page`, `PageSize`): List deployment records. **All four arguments are honoured DB-side (WP2.5)** and the result is a `DeploymentRecordSummary` projection. The handler previously ignored `Status`, `Page` and `PageSize` entirely (the CLI had been sending them all along) and loaded the whole insert-only `DeploymentRecords` table; for a site-scoped user it additionally loaded every instance and intersected the two sets in memory. Site scope is now resolved to the set of in-scope instance ids and pushed into the query as an id filter — an **empty** scope stays a real filter (a user permitted no in-scope instances sees nothing), never a no-op. This read path also drives the opportunistic, rate-limited terminal-record retention sweep described in `Component-DeploymentManager.md`.
- **QueryDeployments** (`InstanceId?`, `Status?`, `Page`, `PageSize`): List deployment records. **All four arguments are honoured DB-side (WP2.5)** and the result is a `DeploymentQueryResult` envelope — `{ deployments, totalCount, page, pageSize }` over `DeploymentListRow`. The handler previously ignored `Status`, `Page` and `PageSize` entirely (the CLI had been sending them all along) and loaded the whole insert-only `DeploymentRecords` table; for a site-scoped user it additionally loaded every instance and intersected the two sets in memory. This read path also drives the opportunistic, rate-limited terminal-record retention sweep described in `Component-DeploymentManager.md`.
**One paged read, three drifts closed (deferred-work register rows 37, 86–88).** This handler used to own a second repository query, `QueryDeploymentSummariesAsync`, alongside the Central UI page's `QueryDeploymentListPageAsync`. That query is **deleted**; both surfaces now go through `QueryDeploymentListPageAsync` with a `DeploymentListFilter`, and the `DeploymentRecordSummary` projection went with it (`DeploymentListRow` is the single row shape). The duplication had drifted into three behavioral differences, all resolved here:
- **Site scope is a SITE-ID filter, resolved by the query's own join to `Instance`** (row 87). It used to be resolved in the handler by loading **every instance in the system** on every scoped query, purely to turn the site grant into an instance-id list — a second whole-table read, non-transactional with the one that mattered. An **empty** scope remains a real filter (a user permitted no sites sees nothing), never a no-op, and a permitted-site id that is not an integer is **dropped rather than ignored** — dropping can only narrow the grant, whereas discarding the grant would widen it to fleet-wide.
- **`PageSize` is bounded** (row 87). A non-positive value used to mean *unlimited* all the way down to the repository, so `--page-size 0` materialized the entire retention window into one response. It now means "unspecified" and takes `DeploymentListPage.DefaultPageSize` (50, the CLI's own default); every value is capped at `DeploymentListPage.MaxPageSize` (1000). `totalCount` travels with the bounded page precisely so a truncated page is distinguishable from a complete one.
- **An unrecognised `Status` is an ERROR, not "no filter"** (row 88). `--status Sucess` used to fall back to unfiltered and return every status — a plausible, complete-looking answer to a question the caller never asked. It now returns `ManagementError` naming the valid values (`Pending, InProgress, Success, Failed`). Parsing stays case-insensitive; a numeric string is rejected too (`Enum.TryParse` accepts `"7"` and yields an undefined value, so the parse is guarded by `Enum.IsDefined`). Same loud-failure principle as register row 46.
**Administrator is system-wide** on this path, as on every other site-scoped handler (`EnforceSiteScope`). The Central UI's `SiteScopeService` was claims-only and disagreed — an Administrator carrying `SiteId` claims saw the fleet here and only their sites on the Deployments page. The UI was corrected to match (register row 86); see `Component-CentralUI.md`.
### Secured Writes (MxGateway, two-person)
@@ -237,6 +250,19 @@ Every incoming message carries the authenticated user's identity and roles. The
Unauthorized commands receive an `Unauthorized` response message. Failed authorization attempts are not audit logged (consistent with existing behavior).
#### Site-scope gate: fail-closed on an unresolvable target
The site-scope helpers resolve the command's target (instance id, site id, site identifier, data-connection id) and check the resolved site against the caller's `PermittedSiteIds`. **A target that cannot be resolved is DENIED for a site-scoped caller** — `SiteScopeViolationException` → `ManagementUnauthorized`, exactly as if it were out of scope.
The gate used to be **fail-open** here (deferred-work register row 98). The resolve-then-check shape was written as `if (target != null) EnforceSiteScope(...)`, so a null resolve skipped the check entirely and a site-scoped caller fell through into the handler body — reaching `HandleGetInstance`/`HandleGetSite`/`HandleGetDataConnection`'s null answer, or a handler fault, for any id in the system. The rule is now: *a target that cannot be PROVEN to sit inside the caller's scope is treated as outside it.*
Two properties are load-bearing:
- **System-wide callers are unaffected.** `Administrator`, and a `Deployer` carrying no site grants, return at the guards above the lookup — they were never scoped, so they still reach the handler and still receive its ordinary "not found" / null answer for a missing id. Only a scoped caller is denied.
- **The denial does not leak existence.** A scoped caller gets the identical `ManagementUnauthorized` shape for "does not exist" and for "exists at a site you may not see", and the message never states which. Otherwise the gate would be an id-enumeration oracle over the whole fleet — which is what the fail-open behaviour was, since `GetInstance`/`GetSite`/`GetDataConnection` answered success-with-null for a nonexistent id and `Unauthorized` for an out-of-scope one.
The shared denial arm is `DenyUnresolvedTargetIfScoped`; `EnforceSiteScopeForInstance`, `EnforceSiteScopeForIdentifier` and the three read handlers that resolve their own target all route through it.
## Service Dependencies (DI)
The ManagementActor receives the following services and repositories via DI (injected through the actor's constructor or via a service provider):
docs/requirements/Component-Security.md
+2
View File
@@ -116,6 +116,8 @@ Set in a local or docker-dev environment via the environment variable `ScadaBrid
- Monitor and manage the Notification Outbox (retry and discard parked notifications).
- View site event logs.
- **Site scoping**: A user with site-scoped Deployer role can only perform these actions for instances at their permitted sites.
- **Administrator overrides site scoping.** A user who holds `Administrator` is system-wide **regardless of the `SiteId` claims they carry** — a scoped Deployer grant does not narrow an Administrator. This is enforced identically on both surfaces: `ManagementActor.EnforceSiteScope` (CLI / HTTP management API) and the Central UI's `SiteScopeService`. The two disagreed until deferred-work register row 86 — the UI was claims-only, so the same user saw the whole fleet through the CLI and only their sites in the UI. **No other role carries this bypass**; in particular `Designer` does not (Designer has no site-scoped permissions to bypass in the first place).
- **The scope check fails CLOSED on an unresolvable target.** Site scoping works by resolving the command's target (instance id, site id/identifier, data-connection id) and comparing its site to the permitted set. When the target does not resolve, a **site-scoped caller is DENIED** — the same `Unauthorized` result an out-of-scope target produces. The rule is *a target that cannot be PROVEN to sit inside the caller's scope is treated as outside it*; it fails open only for callers who were never scoped (Administrator, or a Deployer with no site grants), who keep their ordinary "not found" answer. The check was fail-open until deferred-work register row 98, which also made the denial **indistinguishable from an out-of-scope denial** — same response shape, and a message that never says whether the target exists — so a scoped caller cannot use the gate to enumerate ids across the fleet. See `Component-ManagementService.md` → *Site-scope gate: fail-closed on an unresolvable target*.
> **Area management (arch-review C6)**: Managing area definitions per site is gated **any-of [Designer, Deployer]**, not Administrator. Areas are authored both in the Designer tooling and inside the Central UI Deployment Topology workflow (`Topology.razor` is `RequireDeployment`), so either role qualifies; Administrator is not implicitly included in the actor gate.
docs/requirements/Component-SiteRuntime.md
+20 -2
View File
@@ -108,13 +108,31 @@ flowchart TD
>
> Guards, all shared with the row-67 handler: a **disabled** instance is never started; the sweep **defers to any deploy pipeline that owns the instance** (compile warm in flight, predecessor still terminating, deployment awaiting its persistence/init join) so it can never interleave with a real redeploy; it is **inert until boot-time Instance Actor creation has finished**, so a tick between staggered batches does not "heal" instances the boot load is about to create; it makes **one repair attempt per (instance, revision) pair**, which is what stops a periodic self-heal becoming a restart loop for an actor that dies during init or a stored config the compile gate rejects (a rejection leaves the running instance untouched — the gate returns before anything is stopped); and reads never overlap. A sweep that finds nothing is **completely silent**. A repair emits the same Warning `deployment` site-event pattern as row 67, naming the trigger (`deployment-manager sweep self-heal`) so it is distinguishable from a startup-reconcile heal, and a stale-revision restart names **both** revisions; a rejected restart adds an Error event. The synthetic redeploy replies to the singleton itself rather than to `NoSender`, because dead letters are a monitored site health metric.
> **Starting an Instance Actor also creates its DCL connections — the ensure lives in the creation funnel, not at call sites (register row 85)**: a DCL connection actor is **never** created from the `data_connection_definitions` table (nothing in production reads it). It exists only as a side effect of starting an instance, built from the `Connections` map inside that instance's **own flattened config** via `EnsureDclConnections` — plus the separate system-wide artifact-deploy path. That ensure used to be each *caller's* responsibility, and only two of the four callers made it (the boot-time staggered batch and `ApplyDeployment`). The two that did not were **`StartStoredInstance`** — the path shared by the row-67 ensure-started handler and the row-78 sweep — and **`HandleEnableResult`**. Both therefore produced a running instance whose every tag subscribe was answered `Unknown connection: <name>` by the DCL manager, indefinitely, because nothing else on the node would ever create that connection actor. Proven live 2026-08-16: a replication-seeded node's instances were correctly started by the sweep and then failed every subscribe for 6+ minutes until a second pair restart let the boot load create the connections. **It was never replication-specific** — the boot-time load reads only *enabled* configs, so a node that boots with an instance disabled never creates its connections either, and enabling it afterwards hit the identical dead end on an ordinary unreplicated site. `EnsureDclConnections` now sits at the top of **`CreateInstanceActor`**, already the single Instance Actor creation funnel (it is where the running revision is stamped for the same reason), so all four paths get it and a fifth cannot forget. It is placed *before* the already-exists guard so the two paths that already called it keep their exact previous semantics, and it stays silent in steady state because `EnsureDclConnection` is hash-guarded on the last config pushed per connection name. Deliberately **not** done: a sweep pass over `data_connection_definitions` — it would not have fixed row 85 (that table is empty on a site that has only ever received instance deploys), and re-hashing from it would fight the deploy path over the same per-name hash slot, recreating connection actors and dropping live subscriptions on a loop. See `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7.
> **Starting an Instance Actor also creates its DCL connections — the ensure lives in the creation funnel, not at call sites (register row 85)**: a DCL connection actor is **never** created from the `data_connection_definitions` table — that stayed true when rows 89/90 gave the table a reader, because the overlay it feeds is an *arbiter*, not a creation source, and at the time of row 85 nothing read the table at all. A connection actor exists only as a side effect of starting an instance, built from the `Connections` map inside that instance's **own flattened config** via `EnsureDclConnections` — plus the separate system-wide artifact-deploy path. That ensure used to be each *caller's* responsibility, and only two of the four callers made it (the boot-time staggered batch and `ApplyDeployment`). The two that did not were **`StartStoredInstance`** — the path shared by the row-67 ensure-started handler and the row-78 sweep — and **`HandleEnableResult`**. Both therefore produced a running instance whose every tag subscribe was answered `Unknown connection: <name>` by the DCL manager, indefinitely, because nothing else on the node would ever create that connection actor. Proven live 2026-08-16: a replication-seeded node's instances were correctly started by the sweep and then failed every subscribe for 6+ minutes until a second pair restart let the boot load create the connections. **It was never replication-specific** — the boot-time load reads only *enabled* configs, so a node that boots with an instance disabled never creates its connections either, and enabling it afterwards hit the identical dead end on an ordinary unreplicated site. `EnsureDclConnections` now sits at the top of **`CreateInstanceActor`**, already the single Instance Actor creation funnel (it is where the running revision is stamped for the same reason), so all four paths get it and a fifth cannot forget. It is placed *before* the already-exists guard so the two paths that already called it keep their exact previous semantics, and it stays silent in steady state because `EnsureDclConnection` is hash-guarded on the last config pushed per connection name. Deliberately **not** done at the time: a sweep pass over `data_connection_definitions` — it would not have fixed row 85 (that table is empty on a site that has only ever received instance deploys), and re-hashing from it would fight the deploy path over the same per-name hash slot, recreating connection actors and dropping live subscriptions on a loop. See `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7. That pass is now present, in a form that resolves the conflict rather than creating it — see **DCL connection arbitration** immediately below.
> **DCL connection arbitration — the definitions store wins wherever it has an opinion (register rows 89 + 90)**: `data_connection_definitions` now has a production reader. The rule is: **a connection name with a row in that table is configured from that row; a name with no row is configured from the instance's flattened `Connections` entry, exactly as before.** Rule 2 is not a fallback — it is the correct answer for a deploy-only site, where the table is legitimately empty while every connection that matters lives inline in the instance configs.
>
> **The register's premise needed two corrections, and they change what the fix is.** First, the peer node's DCL is not "serving the old endpoint" — it is **empty**. The Deployment Manager is a `site-{SiteId}`-scoped singleton and the *only* sender of `CreateConnectionCommand`/`RemoveConnectionCommand`, sending to its own node's `dcl-manager` (which exists on every node but is populated only where the singleton runs). The real defect is that `_createdConnections`, the per-name hash cache, is **in-memory and per singleton incarnation**: a restart or failover rebuilds every connection from the `Connections` snapshot frozen into each instance's flattened config at *that instance's* deploy time, and a newer connection-only artifact deploy is silently discarded. **That makes row 89 create-time, not message-delivery, and not replication-specific** — the identical loss happens on an unreplicated single-node restart. Second, `EnsureDclConnection`'s documented "a changed config re-issues the create so the DCL adopts it" was **false**: `DataConnectionManagerActor.HandleCreateConnection` is create-only (warns and returns for a name it already holds), so the artifact deploy's live push has never worked on the deploy-handling node either. That second correction was left as a residual by rows 89/90 and is closed by **row 97** — see *In-place reconfiguration* below.
>
> **Mechanism.** `data_connection_definitions` is read in the *same* off-thread task as `deployed_configurations` (`ReadStoreSnapshotAsync`), by both the boot load and every sweep tick, and applied to an actor-thread-confined **arbitration overlay** *before* the boot batch creates any Instance Actor and *before* the sweep's repair loop — so a connection is built from the winning config on its **first** push rather than created stale and corrected a tick later. The overlay is an **arbiter, not a creation source**: a definition row naming a connection no deployed instance references creates nothing. A CDC-delivered definition change therefore converges within one sweep tick on whichever node runs the singleton, with **no new cross-node message**. A definitions read that *fails* yields `null` and leaves the overlay untouched — deliberately distinct from an empty list, which is a real answer; clearing on a transient failure would reintroduce the ping-pong through the error path.
>
> **The ping-pong is killed by keying the hash on the winner.** The conflict §7.4 identified is a property of *where the hash is computed*, not of the sweep: every ensure path now resolves the winning configuration **before** hashing or pushing, so for any name only one configuration can ever occupy the hash slot regardless of which path calls. The flip is not guarded against — it is unrepresentable. `HandleApplyArtifactDataConnectionsToDcl` writes its rows into the overlay in the same pass it pushes them, closing the same flip arriving through the artifact-deploy door.
>
> **Stale-snapshot guard (row 101).** The snapshot is read off the actor thread but compared against the overlay only when its piped message lands — so a read that starts before an artifact apply's store writes can land *after* `HandleApplyArtifactDataConnectionsToDcl` populated the overlay, and its stale data would "prove" the just-applied names disappeared: tombstoned, live connection actors stopped, and — because the removal evicts the hash slot the overlay's push is gated on — **nothing re-creates them until an instance redeploy**. Each snapshot read therefore captures a monotonic definitions-write generation on the actor thread at dispatch (`_definitionsWriteGeneration`, bumped by every artifact apply), and a result carrying a stale generation is discarded **whole** — additions included, since a stale row pushed at a live connection would reconfigure it backward and flip-flop with the next fresh read. Discarding is safe: the artifact apply already updated the overlay itself, and the next tick reads fresh. Surfaced as a test flake (a raced sweep removed `PlcA` where the test expected `PlcGone`); the deterministic reproduction is `DclConnectionArbitrationTests.StaleSweepSnapshot_RacingAnArtifactApply_DoesNotRemoveJustAppliedConnections`.
>
> **Removal (row 90).** Both removal paths share one helper and now record a **tombstone** for the incarnation: the artifact reconcile's `RemovedConnectionNames`, and the sweep noticing that a name it had adopted no longer has a row (a removal it can *prove* — the store owned it a tick ago and does not now). `EnsureDclConnections` skips a tombstoned name, so a removal survives the next instance start instead of being undone by it — a pre-existing bug, since the removal evicted the hash slot but left no memory of why. Tombstones are **in-memory only** by decision: a durable marker would be a second divergent authority, and flattened configs already resurrect a removed connection across a restart. A definition row reappearing clears the tombstone.
>
> **In-place reconfiguration — a changed config now reaches the LIVE connection (register row 97)**: resolving the winner decides *what* to push; it took a third command to make a push against an already-live connection mean anything. `EnsureDclConnection` has three outcomes, all keyed off the one per-name hash slot: **unchanged → nothing at all** (no message, no log — this is what keeps a healthy site silent across every sweep tick); **new name → `CreateConnectionCommand`**; **changed → `ReconfigureConnectionCommand`**, which the DCL manager forwards to the live connection actor. The actor swaps its adapter through the very machinery the primary→backup failover uses and re-issues every subscription it holds — no redeployment, no restart. The winner-resolution and tombstone logic above are untouched; only what happens *after* a change is detected changed. The log line is `Reconfigured DCL connection {Connection} (protocol={Protocol}) in place` at Information, replacing the Warning rows 89/90 added to make the gap honest. See `Component-DataConnectionLayer.md` → *In-place reconfiguration* for what the connection actor does, and why stop-and-recreate was rejected.
>
> **Residuals still open**, recorded precisely in the decision memo: a removal is not durable across a restart; and a delete is undetectable on a node that never adopted the row. See `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md`.
> **Per-instance compilation during staggered startup — CLOSED (WP3.1)**: this was a deferred optimization (P6) affecting failover time-to-recover; each Instance Actor Roslyn-compiled its own scripts inside its own start, serialising a site's whole recovery behind compilation. Each staggered startup batch now runs a **pre-warm step**: before the batch's Instance Actors are created, the Deployment Manager compiles that batch's distinct script bodies and trigger expressions on a background task and pipes `BatchCompileWarmed` back, so every `PreStart` compile is a cache hit. Instance Actor start still calls the compilation service — it remains the correctness backstop — but in every warmed path those calls are memoised lookups. Cost: one extra message per batch.
> **The Roslyn `ScriptOptions` are process-static, and must stay that way**: the reference set backing every site-side compile is built once per process, never per compile. `ScriptOptions.WithReferences(Assembly[])` resolves each assembly through `MetadataReference.CreateFromFile`, which does **not** cache — each call mints a fresh `MetadataReference` owning an `AssemblyMetadata` → `PEReader` → `NativeHeapMemoryBlock`, an unmanaged copy of the assembly metadata that nothing disposes. Building the options per compile leaks native memory permanently: no GC reclaims it, and it is invisible to gcdump and to the managed allocation counters, so the node's working set grows without the GC heap growing. This was a live defect (a Site node at 2,885 MB working set with 150 MB of live GC heap, ~2,469 MB of it on the default process heap across ~6,700 undisposed `AssemblyMetadata` instances). Note this is orthogonal to the compile cache above — the cache dedupes *identical* script bodies, so it bounds nothing when the bodies differ. (It no longer clears wholesale on overflow: WP3.1 replaced that latency cliff with approximate-LRU batch eviction, see **Compile cache eviction** below.) Pinned by a reference-equality regression test rather than by watching memory.
> **Compile cache eviction is approximate LRU, not a wholesale clear (WP3.1)**: `SiteScriptCompileCache` is bounded at 1024 entries — deliberately smaller than the TemplateEngine verdict cache's 4096, because entries pin compiled assemblies rather than verdict strings. It used to `Clear()` **everything** on overflow. Instance scripts *and* trigger expressions share this cache, so on a site large enough to cross the bound every overflow discarded up to 1023 live compiled scripts and the next deploy or Instance-Actor start paid a full recompile storm **on actor threads**. Overflow now evicts only the oldest ⅛ (128 entries) by last-access stamp: one 1024-element scan instead of 1023 future recompiles, and hot entries survive. Recency is an `Interlocked` access sequence rather than a clock — deterministic for tests and immune to clock steps; hits update it lock-free, and only the (rare) eviction sweep takes a lock, double-checked so concurrent inserts do not stampede it. The sweep snapshots through `ConcurrentDictionary.ToArray()` and **not** LINQ over the dictionary: LINQ's `ToArray` picks the `ICollection.CopyTo` fast path, which throws when a concurrent insert lands between its Count read and its copy, and inserts are deliberately lock-free.
> **The compile cache is owned by the `ScriptCompilationService` singleton, not static (register row 100)**: `SiteScriptCompileCache` is an ordinary instance held by `ScriptCompilationService`, and "one Roslyn compile per script body per process" (N4) is exactly the statement that `AddSiteRuntime` registers that service as a **DI singleton** — the deploy gate (via `DeployCompileValidator`), every Instance Actor's `PreStart`, and `SharedScriptLibrary` all reach the one cache through the one service the Deployment Manager threads to them. The singleton lifetime is therefore load-bearing rather than incidental, and is pinned by a registration test (`SiteRuntimeServiceRegistrationTests`); downgrading it to Scoped/Transient would hand every consumer an empty cache and silently reintroduce WP3.1's recompile storms with no behavioural test failing. The class was previously `static`, which made its entry table and its `Hits`/`Count` counters one process-global blob every test class in the assembly wrote into; the `Clear()` method that existed only so tests could reset that blob is **removed** — it never had a production caller, and constructing a cache is the honest reset.
> **Compile cache eviction is approximate LRU, not a wholesale clear (WP3.1)**: `SiteScriptCompileCache` is bounded at 1024 entries — deliberately smaller than the TemplateEngine verdict cache's 4096, because entries pin compiled assemblies rather than verdict strings. It used to clear **everything** on overflow. Instance scripts *and* trigger expressions share this cache, so on a site large enough to cross the bound every overflow discarded up to 1023 live compiled scripts and the next deploy or Instance-Actor start paid a full recompile storm **on actor threads**. Overflow now evicts only the oldest ⅛ (128 entries) by last-access stamp: one 1024-element scan instead of 1023 future recompiles, and hot entries survive. Recency is an `Interlocked` access sequence rather than a clock — deterministic for tests and immune to clock steps; hits update it lock-free, and only the (rare) eviction sweep takes a lock, double-checked so concurrent inserts do not stampede it. The sweep snapshots through `ConcurrentDictionary.ToArray()` and **not** LINQ over the dictionary: LINQ's `ToArray` picks the `ICollection.CopyTo` fast path, which throws when a concurrent insert lands between its Count read and its copy, and inserts are deliberately lock-free.
> **Static options were only half the fix — the compile must also use the shared caching metadata resolver**: the static `ScriptOptions` carry only the *direct* API-surface references (5 assemblies here), and each `script.Compile()` still binds their **transitive closure** (~105 assemblies on a site node). Every transitively-referenced assembly is resolved through the options' `MetadataReferenceResolver`, and Roslyn's default one has no cross-compilation cache — it calls `MetadataReference.CreateFromFile` afresh on **every compile**, minting the same undisposed `AssemblyMetadata` → `PEReader` → `NativeHeapMemoryBlock` triple per assembly per compiled script, pinned for the process lifetime by the compile cache. That per-compile *re-resolution* — not the per-compile options construction — was the dominant term (measured live: 21 site scripts holding 6,640 metadata objects, counts bit-identical before and after the static-options fix). `ScriptCompilationService` therefore also attaches the process-wide `CachingScriptMetadataResolver.Instance` (owned by the Script Analysis component, see `Component-ScriptAnalysis.md`) via `ScriptOptions.WithMetadataResolver`, so each distinct assembly is materialized once per process. The decorator returns exactly what the undecorated resolver returned — only object identity is de-duplicated — so compile results and the trust gate are unchanged.
docs/test_infra/test_infra.md
+58
View File
@@ -46,6 +46,64 @@ Each service has a dedicated document with configuration details, verification s
- [test_infra_playwright.md](test_infra_playwright.md) — Playwright browser server (Central UI testing)
- Traefik LB — see `docker/README.md` and `docker/traefik/` (runs with the cluster, not in `infra/`)
## Test Parallelism Policy
There are **two independent parallelism axes**, and they are bounded by two different mechanisms.
Conflating them is what made the row-82/91/96 flake family hard to read: a fix on one axis leaves the
other untouched.
| Axis | What runs at once | Bounded by |
|------|-------------------|-----------|
| **In-process** — xunit collection threads | Test collections *within one test assembly*. Default is `Environment.ProcessorCount` (14 on the current dev box). | `xunit.runner.json` → `maxParallelThreads`, per suite. This section. |
| **Cross-process** — `dotnet test` on the solution | One test *process* per test project, several at a time. xunit cannot see across them. | For the MS SQL migration path only: the machine-wide `sp_getapplock` counting semaphore in `MsSqlMigrationFixture` (4 slots, `SCADABRIDGE_MSSQL_TEST_MIGRATE_SLOTS`) — see [test_infra_db.md](test_infra_db.md). Nothing bounds it in general. |
### Which suites carry a runner config, and why
The cap is **per suite, not global**: pure unit suites (Commons, ScriptAnalysis, TemplateEngine,
Transport, …) stay uncapped for speed. A suite gets a config only when a test *class* costs a real
external or OS-level resource, so that `ProcessorCount` of them at once is a resource question rather
than a CPU question.
| Suite | Setting | Why |
|-------|---------|-----|
| `ConfigurationDatabase.Tests` | `maxParallelThreads: 4` | Owns `MsSqlMigrationFixture`: every MSSQL-backed test **class** does a `CREATE DATABASE` plus the full ~100-migration chain against the one shared dev SQL Server. 4 matches the `sp_getapplock` slot count so the in-process and cross-process limits agree. |
| `AuditLog.Tests` | `maxParallelThreads: 4` | Same fixture (by project reference), plus most classes stand up an Akka TestKit `ActorSystem` with fast-ticking actors. |
| `SiteCallAudit.Tests` | `maxParallelThreads: 4` | Same fixture (by project reference) plus a TestKit `ActorSystem`. |
| `Host.Tests` | `maxParallelThreads: 4` | Classes boot real ASP.NET Core hosts and real Akka `ActorSystem`s — Akka.Remote binds real sockets, Kestrel binds real ports, gRPC channels dial them. |
| `IntegrationTests` | `parallelizeAssembly: false`, `parallelizeTestCollections: false` | Pre-existing. Fully serialized; a thread cap would be redundant. |
| `CentralUI.PlaywrightTests` | `maxParallelThreads: 1` | Pre-existing, and the strictest setting in the repo — the suite drives a browser against a **live rig**, so shared UI state, not CPU, is the constraint. |
| `StoreAndForward.Tests` | `parallelizeTestCollections: true` | Pre-existing; that is the default, so this file is a no-op pin. Uncapped by choice — SQLite files are per-test. |
### Rules
- **Cap threads, do not serialize.** `maxParallelThreads` throttles how many collections run at once;
`parallelizeTestCollections: false` runs them one at a time forever. Reach for the second only when a
suite has a genuinely global resource it cannot partition (`IntegrationTests` and the Playwright suite
are the only two, and both predate this policy).
- **A cap is not a substitute for test isolation.** Rows 82/91/95 were all *correctness* bugs — leaked
ticking actors, absolute assertions on process-global counters — that a cap only makes rarer. Fix the
test; the cap is for resource pressure.
- **The json must be copied to the output directory.** The .NET SDK does not do this automatically:
each capped `.csproj` carries
`<Content Include="xunit.runner.json" CopyToOutputDirectory="PreserveNewest" />`. Without it the file
is silently ignored and the suite runs uncapped.
- **Default for a new test project: no runner config.** Add one only when the project acquires a
per-class external resource (an MSSQL fixture, a real host/port, a browser). Record the reason in the
`.csproj` comment next to the `Content` item.
### Measured cost
The cap is close to free, because these suites are latency-bound on an external resource rather than
CPU-bound (CPU utilisation stayed at 14–25 % for the two slowest). Measured standalone on a 14-core dev
box, uncapped vs. `maxParallelThreads: 4`:
| Suite | Tests | Uncapped | Capped at 4 |
|-------|-------|----------|-------------|
| `AuditLog.Tests` | 383 | 1 m 52 s | 1 m 51 s |
| `Host.Tests` | 520 | 4 m 5 s | 4 m 8 s |
| `ConfigurationDatabase.Tests` | 396 | 10 s | 12 s |
| `SiteCallAudit.Tests` | 66 | 8 s | 8 s |
## Remote Test Infrastructure
In addition to the local Docker services, the following remote services are available for testing against real AVEVA System Platform hardware.
docs/test_infra/test_infra_db.md
+4 -1
View File
@@ -43,7 +43,10 @@ Three behaviours are worth knowing when you look at this server during a test ru
solution-level `dotnet test` runs several MSSQL-backed test *processes* at once, so the throttle has
to live on the shared server. Without it, dozens of simultaneous fresh-database migrations saturated
the container and individual DDL statements blew past EF's default 30-second command timeout,
failing whole test classes at fixture construction (deferred-work register row 82).
failing whole test classes at fixture construction (deferred-work register row 82). The
*in-process* half of the same problem is bounded separately, by a `maxParallelThreads: 4` cap in
each MSSQL-backed suite's `xunit.runner.json` — deliberately the same number, so the two limits
agree. See [test_infra.md → Test Parallelism Policy](test_infra.md#test-parallelism-policy).
- **Migration commands use a 300s timeout**, and CREATE/DROP DATABASE use 120s. Under a bounded queue
the server is busy but progressing, so the correct response to a slow statement is to wait.
- **Strays are swept.** Each test process drops, once at start and best-effort, any
src/ZB.MOM.WW.ScadaBridge.CLI/Commands/AuditCommands.cs
+26 -22
View File
@@ -41,28 +41,30 @@ public static class AuditCommands
// --channel/--kind/--status/--site are multi-valued: System.CommandLine accepts
// both repeated tokens (--channel A --channel B) and, with
// AllowMultipleArgumentsPerToken, a single token carrying several values
// (--channel A B). AcceptOnlyFromAmong validates EACH supplied value.
// (--channel A B).
//
// The valid sets are named in the help text rather than enforced with
// AcceptOnlyFromAmong: that check is case-SENSITIVE, so `--status delivered`
// failed client-side even though the server has always parsed these
// case-insensitively (register row 99 — same treatment row 88 gave
// `deploy status`). The server rejects an unrecognised value with a 400 naming
// the same list and is the authoritative check; HTTP API callers never pass
// through these options at all.
var channelOption = new Option<string[]>("--channel")
{
Description = "Filter by channel (ApiOutbound, DbOutbound, Notification, ApiInbound); repeatable",
Description = "Filter by channel (ApiOutbound, DbOutbound, Notification, ApiInbound; case-insensitive); repeatable",
AllowMultipleArgumentsPerToken = true,
};
channelOption.AcceptOnlyFromAmong("ApiOutbound", "DbOutbound", "Notification", "ApiInbound");
var kindOption = new Option<string[]>("--kind")
{
Description = "Filter by event kind (ApiCall, ApiCallCached, DbWrite, DbWriteCached, NotifySend, NotifyDeliver, InboundRequest, InboundAuthFailure, CachedSubmit, CachedResolve); repeatable",
Description = "Filter by event kind (ApiCall, ApiCallCached, DbWrite, DbWriteCached, NotifySend, NotifyDeliver, InboundRequest, InboundAuthFailure, CachedSubmit, CachedResolve; case-insensitive); repeatable",
AllowMultipleArgumentsPerToken = true,
};
kindOption.AcceptOnlyFromAmong(
"ApiCall", "ApiCallCached", "DbWrite", "DbWriteCached", "NotifySend",
"NotifyDeliver", "InboundRequest", "InboundAuthFailure", "CachedSubmit", "CachedResolve");
var statusOption = new Option<string[]>("--status")
{
Description = "Filter by status (Submitted, Forwarded, Attempted, Delivered, Failed, Parked, Discarded, Skipped); repeatable",
Description = "Filter by status (Submitted, Forwarded, Attempted, Delivered, Failed, Parked, Discarded, Skipped; case-insensitive); repeatable",
AllowMultipleArgumentsPerToken = true,
};
statusOption.AcceptOnlyFromAmong(
"Submitted", "Forwarded", "Attempted", "Delivered", "Failed", "Parked", "Discarded", "Skipped");
var siteOption = new Option<string[]>("--site")
{
Description = "Filter by source site ID; repeatable",
@@ -143,34 +145,36 @@ public static class AuditCommands
{
var sinceOption = new Option<string>("--since") { Description = "Start time: relative (1h, 24h, 7d) or ISO-8601", Required = true };
var untilOption = new Option<string>("--until") { Description = "End time: relative (1h, 24h, 7d) or ISO-8601", Required = true };
var formatExportOption = new Option<string>("--format") { Description = "Export format", Required = true };
formatExportOption.AcceptOnlyFromAmong("csv", "jsonl", "parquet");
// Same rationale as --channel/--kind/--status below and on `query` (register
// row 99): AcceptOnlyFromAmong is case-SENSITIVE, so it rejected `--format CSV`
// even though the server lowercases the value before matching. The server
// returns a 400 naming csv/jsonl/parquet and is the authoritative check.
var formatExportOption = new Option<string>("--format")
{
Description = "Export format (csv, jsonl, parquet; case-insensitive)",
Required = true,
};
var outputOption = new Option<string>("--output") { Description = "Destination file path", Required = true };
// --channel/--kind/--status/--site are multi-valued — same shape as the
// `query` subcommand: repeated tokens (--channel A --channel B) and, with
// AllowMultipleArgumentsPerToken, a single token carrying several values
// (--channel A B). AcceptOnlyFromAmong validates EACH supplied value.
// (--channel A B). Valid sets are named in the help text and enforced
// server-side, case-insensitively — see the `query` builder for the full note.
var channelOption = new Option<string[]>("--channel")
{
Description = "Filter by channel (ApiOutbound, DbOutbound, Notification, ApiInbound); repeatable",
Description = "Filter by channel (ApiOutbound, DbOutbound, Notification, ApiInbound; case-insensitive); repeatable",
AllowMultipleArgumentsPerToken = true,
};
channelOption.AcceptOnlyFromAmong("ApiOutbound", "DbOutbound", "Notification", "ApiInbound");
var kindOption = new Option<string[]>("--kind")
{
Description = "Filter by event kind (ApiCall, ApiCallCached, DbWrite, DbWriteCached, NotifySend, NotifyDeliver, InboundRequest, InboundAuthFailure, CachedSubmit, CachedResolve); repeatable",
Description = "Filter by event kind (ApiCall, ApiCallCached, DbWrite, DbWriteCached, NotifySend, NotifyDeliver, InboundRequest, InboundAuthFailure, CachedSubmit, CachedResolve; case-insensitive); repeatable",
AllowMultipleArgumentsPerToken = true,
};
kindOption.AcceptOnlyFromAmong(
"ApiCall", "ApiCallCached", "DbWrite", "DbWriteCached", "NotifySend",
"NotifyDeliver", "InboundRequest", "InboundAuthFailure", "CachedSubmit", "CachedResolve");
var statusOption = new Option<string[]>("--status")
{
Description = "Filter by status (Submitted, Forwarded, Attempted, Delivered, Failed, Parked, Discarded, Skipped); repeatable",
Description = "Filter by status (Submitted, Forwarded, Attempted, Delivered, Failed, Parked, Discarded, Skipped; case-insensitive); repeatable",
AllowMultipleArgumentsPerToken = true,
};
statusOption.AcceptOnlyFromAmong(
"Submitted", "Forwarded", "Attempted", "Delivered", "Failed", "Parked", "Discarded", "Skipped");
var siteOption = new Option<string[]>("--site")
{
Description = "Filter by source site ID; repeatable",
src/ZB.MOM.WW.ScadaBridge.CLI/Commands/DeployCommands.cs
+12 -3
View File
@@ -94,10 +94,18 @@ public static class DeployCommands
private static Command BuildStatus(Option<string> urlOption, Option<string> formatOption, Option<string> usernameOption, Option<string> passwordOption)
{
var instanceIdOption = new Option<int?>("--instance-id") { Description = "Filter by instance ID" };
var statusOption = new Option<string?>("--status") { Description = "Filter by status" };
// The valid set is named in the help text rather than enforced with
// AcceptOnlyFromAmong: that check is case-SENSITIVE, and the server has
// always accepted `--status success`. The server rejects an unrecognised
// value with the same list (register row 88) and is the authoritative
// check — HTTP API callers never pass through this option at all.
var statusOption = new Option<string?>("--status")
{
Description = "Filter by status (Pending, InProgress, Success, Failed; case-insensitive)",
};
var pageOption = new Option<int>("--page") { Description = "Page number" };
pageOption.DefaultValueFactory = _ => 1;
var pageSizeOption = new Option<int>("--page-size") { Description = "Page size" };
var pageSizeOption = new Option<int>("--page-size") { Description = "Page size (1-1000; 0 or unset uses 50)" };
pageSizeOption.DefaultValueFactory = _ => 50;
var cmd = new Command("status") { Description = "Query deployment status" };
@@ -113,7 +121,8 @@ public static class DeployCommands
var pageSize = result.GetValue(pageSizeOption);
return await CommandHelpers.ExecuteCommandAsync(
result, urlOption, formatOption, usernameOption, passwordOption,
new QueryDeploymentsCommand(instanceId, status, page, pageSize));
new QueryDeploymentsCommand(instanceId, status, page, pageSize),
tableProjector: DeploymentStatusTableProjection.ProjectRows);
});
return cmd;
}
src/ZB.MOM.WW.ScadaBridge.CLI/Commands/DeploymentStatusTableProjection.cs
+120
View File
@@ -0,0 +1,120 @@
using System.Text.Json;
using System.Text.Json.Nodes;
namespace ZB.MOM.WW.ScadaBridge.CLI.Commands;
/// <summary>
/// Table projection for <c>deploy status</c> (deferred-work register rows 37/87).
///
/// <para>
/// The handler now returns a <c>DeploymentQueryResult</c> envelope —
/// <c>{ "deployments": [ … ], "totalCount": N, "page": P, "pageSize": S }</c> — where
/// it used to return a bare row array. Handed to the generic renderer unprojected,
/// an object takes the Property/Value branch and the whole <c>deployments</c> array
/// lands in a single unreadable cell. This unwraps the envelope to the row array the
/// renderer expects, exactly as <see cref="AuditConfigTableProjection"/> does for
/// <c>audit-config query</c>.
/// </para>
///
/// <para>
/// The envelope's counts would be lost by a plain unwrap, and losing THOSE is the one
/// thing this projection must not do: the page size is bounded server-side now, so a
/// full page is no longer proof that there are no more rows. When the page is a strict
/// subset a single trailing row carries a <c>note</c> cell saying how many of how many
/// are shown and how to get the rest. When the page IS the whole result set the column
/// is not emitted at all.
/// </para>
///
/// <para>
/// JSON output is never touched — this runs only on the table path.
/// </para>
/// </summary>
internal static class DeploymentStatusTableProjection
{
/// <summary>
/// Projects a <c>DeploymentQueryResult</c> envelope (or a bare row array) to the
/// row array the table renderer consumes. Returns the input unchanged when it is
/// not JSON or not one of those shapes, so the renderer's own fallbacks apply.
/// </summary>
/// <param name="json">The raw success JSON body from the management API.</param>
/// <returns>JSON suitable for table rendering.</returns>
internal static string ProjectRows(string json)
{
JsonDocument doc;
try
{
doc = JsonDocument.Parse(json);
}
catch (JsonException)
{
// Not JSON (e.g. a proxy error page) — let the renderer print it verbatim.
return json;
}
using (doc)
{
var root = doc.RootElement;
// A bare array is accepted so the command still renders usefully against
// an older server that returns the pre-envelope shape.
if (root.ValueKind == JsonValueKind.Array)
return BuildRows(root, totalCount: null).ToJsonString();
if (root.ValueKind == JsonValueKind.Object
&& TryGetPropertyCI(root, "deployments", out var rows)
&& rows.ValueKind == JsonValueKind.Array)
{
int? total = TryGetPropertyCI(root, "totalCount", out var tc)
&& tc.ValueKind == JsonValueKind.Number
&& tc.TryGetInt32(out var n)
? n
: null;
return BuildRows(rows, total).ToJsonString();
}
// Some other object (e.g. an error envelope) — leave it to the renderer.
return json;
}
}
private static JsonArray BuildRows(JsonElement rows, int? totalCount)
{
var arr = new JsonArray();
var shown = 0;
foreach (var item in rows.EnumerateArray())
{
arr.Add(item.ValueKind == JsonValueKind.Object
? JsonNode.Parse(item.GetRawText())!
: JsonValue.Create(item.ToString())!);
shown++;
}
// Only when the page is a strict subset — otherwise the count is just the row
// count the operator can already see, and the extra column would be noise.
if (totalCount is int total && total > shown)
{
arr.Add(new JsonObject
{
["note"] = JsonValue.Create(
$"showing {shown} of {total} — use --page / --page-size for the rest"),
});
}
return arr;
}
private static bool TryGetPropertyCI(JsonElement obj, string name, out JsonElement value)
{
foreach (var prop in obj.EnumerateObject())
{
if (string.Equals(prop.Name, name, StringComparison.OrdinalIgnoreCase))
{
value = prop.Value;
return true;
}
}
value = default;
return false;
}
}
src/ZB.MOM.WW.ScadaBridge.CLI/README.md
+32 -9
View File
@@ -1011,9 +1011,23 @@ scadabridge --url <url> deploy status [--instance-id <int>] [--status <string>]
| Option | Required | Default | Description |
|--------|----------|---------|-------------|
| `--instance-id` | no | — | Filter by instance ID |
| `--status` | no | — | Filter by deployment status string |
| `--status` | no | — | Filter by deployment status: `Pending`, `InProgress`, `Success`, `Failed` (case-insensitive) |
| `--page` | no | `1` | Page number |
| `--page-size` | no | `50` | Results per page |
| `--page-size` | no | `50` | Results per page; capped at `1000`. `0` means "unspecified" and takes the default — it does **not** mean unlimited |
Output is an envelope: `deployments` (the page of rows, newest first), `totalCount`
(rows matching the filter across all pages), plus the `page` and `pageSize` the server
actually applied. `--format table` renders the rows and appends a `showing N of M`
note when the page is a strict subset.
An unrecognised `--status` is an **error** naming the valid values, not a silent
fallback to "all statuses" — a typo like `--status Sucess` used to return every
deployment record, which looks exactly like a complete answer to the question you
meant to ask.
Site scoping: a site-scoped Deployer sees only deployments for instances at their
permitted sites. A user holding `Administrator` is system-wide regardless of any site
grant, matching the Central UI.
---
@@ -1674,9 +1688,9 @@ scadabridge --url <url> audit query [options]
|--------|----------|---------|-------------|
| `--since` | no | — | Start time: relative (`1h`, `24h`, `7d`) or ISO-8601 |
| `--until` | no | — | End time: relative (`1h`, `24h`, `7d`) or ISO-8601 |
| `--channel` | no | — | Filter by channel (`ApiOutbound`, `DbOutbound`, `Notification`, `ApiInbound`); repeatable — multiple values are OR-combined |
| `--kind` | no | — | Filter by event kind (`ApiCall`, `ApiCallCached`, `DbWrite`, `DbWriteCached`, `NotifySend`, `NotifyDeliver`, `InboundRequest`, `InboundAuthFailure`, `CachedSubmit`, `CachedResolve`); repeatable — multiple values are OR-combined |
| `--status` | no | — | Filter by status (`Submitted`, `Forwarded`, `Attempted`, `Delivered`, `Failed`, `Parked`, `Discarded`, `Skipped`); repeatable — multiple values are OR-combined |
| `--channel` | no | — | Filter by channel (`ApiOutbound`, `DbOutbound`, `Notification`, `ApiInbound`; case-insensitive); repeatable — multiple values are OR-combined |
| `--kind` | no | — | Filter by event kind (`ApiCall`, `ApiCallCached`, `DbWrite`, `DbWriteCached`, `NotifySend`, `NotifyDeliver`, `InboundRequest`, `InboundAuthFailure`, `CachedSubmit`, `CachedResolve`; case-insensitive); repeatable — multiple values are OR-combined |
| `--status` | no | — | Filter by status (`Submitted`, `Forwarded`, `Attempted`, `Delivered`, `Failed`, `Parked`, `Discarded`, `Skipped`; case-insensitive); repeatable — multiple values are OR-combined |
| `--site` | no | — | Filter by source site ID; repeatable — multiple values are OR-combined |
| `--target` | no | — | Filter by target (external system, DB connection, notification list) |
| `--actor` | no | — | Filter by actor |
@@ -1693,6 +1707,15 @@ either as repeated flags (`--channel ApiOutbound --channel DbOutbound`) or
space-separated after one flag (`--channel ApiOutbound DbOutbound`). Values
within one filter are OR-combined; the different filters are AND-combined.
**Filter values are case-insensitive** — `--status delivered`, `--status Delivered`
and `--status DELIVERED` are the same query. The CLI does not validate the
vocabulary itself; the server does, and an unrecognised value is an **error naming
the valid set**, not a silently-dropped filter. That matters: dropping the filter
would return a plausible, complete-looking page that does not answer the question
asked (the same failure mode as `deploy status`). Ordinals are rejected — a filter
is a name-based contract, so `--status 3` is an error even though `3` is a real
enum value.
With `--format table`, events render as an aligned text table with columns
`OccurredAtUtc`, `Channel`, `Kind`, `Status`, `Target`, `Actor`, `DurationMs`,
`HttpStatus`; long `Target`/`Actor` values are truncated with an ellipsis. With
@@ -1712,11 +1735,11 @@ scadabridge --url <url> audit export --since <time> --until <time> --format <fmt
|--------|----------|---------|-------------|
| `--since` | yes | — | Start time: relative (`1h`, `24h`, `7d`) or ISO-8601 |
| `--until` | yes | — | End time: relative (`1h`, `24h`, `7d`) or ISO-8601 |
| `--format` | yes | — | Export format: `csv`, `jsonl`, or `parquet` |
| `--format` | yes | — | Export format: `csv`, `jsonl`, or `parquet` (case-insensitive) |
| `--output` | yes | — | Destination file path |
| `--channel` | no | — | Filter by channel |
| `--kind` | no | — | Filter by event kind |
| `--status` | no | — | Filter by status |
| `--channel` | no | — | Filter by channel (case-insensitive; same valid set as `audit query`) |
| `--kind` | no | — | Filter by event kind (case-insensitive; same valid set as `audit query`) |
| `--status` | no | — | Filter by status (case-insensitive; same valid set as `audit query`) |
| `--site` | no | — | Filter by source site ID |
| `--target` | no | — | Filter by target |
| `--actor` | no | — | Filter by actor |
src/ZB.MOM.WW.ScadaBridge.CentralUI/Auth/SiteScopeService.cs
+40 -10
View File
@@ -6,14 +6,26 @@ namespace ZB.MOM.WW.ScadaBridge.CentralUI.Auth;
/// <summary>
/// Resolves the set of sites the current user is permitted to operate on, from
/// the <c>SiteId</c> claims attached at login.
/// the roles and <c>SiteId</c> claims attached at login.
/// <para>
/// The design (Component-CentralUI, CLAUDE.md "Security &amp; Auth") makes the
/// Deployment role site-scoped: a Deployment user mapped through an LDAP group
/// with site-scope rules carries one <see cref="JwtTokenService.SiteIdClaimType"/>
/// claim per permitted site (the claim value is the integer <c>Site.Id</c>).
/// A Deployment user with no <c>SiteId</c> claim — and any Admin/Design user — is
/// system-wide.
/// A user with no <c>SiteId</c> claim is system-wide, and so is any user holding
/// the <see cref="Roles.Administrator"/> role — regardless of the <c>SiteId</c>
/// claims they carry.
/// </para>
/// <para>
/// <b>The Administrator bypass is not a convenience, it is the agreed semantic on
/// the other side of the same rule.</b> <c>ManagementActor</c> has always exempted
/// Administrator from site scoping (<c>EnforceSiteScope</c> and every handler that
/// calls it), so a user holding Administrator ALONGSIDE a scoped Deployer grant
/// could act fleet-wide through the CLI/HTTP surface while this service hid the
/// same sites in the UI — the UI was the side that disagreed with the documented
/// rule (deferred-work register row 86). Note that Designer is NOT exempt: the
/// enforced rule has only ever bypassed Administrator, and this doc used to claim
/// "any Admin/Design user" is system-wide, which was never true of either side.
/// </para>
/// <para>
/// Deployment and Monitoring pages must filter every site/instance list through
@@ -35,8 +47,9 @@ public sealed class SiteScopeService
}
/// <summary>
/// True when the user is not restricted to a site subset (no <c>SiteId</c>
/// claims). System-wide users see and act on every site.
/// True when the user is not restricted to a site subset — either because they
/// hold <see cref="Roles.Administrator"/> or because they carry no
/// <c>SiteId</c> claims. System-wide users see and act on every site.
/// </summary>
/// <returns>A task that resolves to <c>true</c> if the user has no site-scope restriction.</returns>
public async Task<bool> IsSystemWideAsync()
@@ -83,19 +96,36 @@ public sealed class SiteScopeService
return cached;
var state = await _authStateProvider.GetAuthenticationStateAsync();
var siteClaims = state.User.FindAll(JwtTokenService.SiteIdClaimType);
// Administrator => system-wide, whatever SiteId claims are present. Read
// the role CLAIM rather than IsInRole: the principal's role claim type is
// only bound to JwtTokenService.RoleClaimType when the identity was built
// by SessionClaimBuilder, and the comparison is OrdinalIgnoreCase to match
// ManagementActor's — IsInRole compares claim VALUES ordinally, which would
// make the two sides disagree on a differently-cased role name.
var isAdministrator = state.User
.FindAll(JwtTokenService.RoleClaimType)
.Any(c => string.Equals(c.Value, Roles.Administrator, StringComparison.OrdinalIgnoreCase));
var ids = new HashSet<int>();
foreach (var claim in siteClaims)
if (!isAdministrator)
{
if (int.TryParse(claim.Value, out var id))
ids.Add(id);
// An Administrator's SiteId claims are not merely overridden, they are
// not collected at all: the invariant "system-wide => empty set" then
// holds for every caller, so a consumer that reads the id set without
// first consulting IsSystemWide cannot silently narrow an Administrator
// to a subset of sites.
foreach (var claim in state.User.FindAll(JwtTokenService.SiteIdClaimType))
{
if (int.TryParse(claim.Value, out var id))
ids.Add(id);
}
}
// No SiteId claims => system-wide. Absence of scope rules means an
// unrestricted deployer (this service is the sole
// site-scoping mechanism — there is no separate handler to mirror).
var result = (IsSystemWide: ids.Count == 0, Sites: (IReadOnlySet<int>)ids);
var result = (IsSystemWide: isAdministrator || ids.Count == 0, Sites: (IReadOnlySet<int>)ids);
_cached = result;
return result;
}
src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IDeploymentManagerRepository.cs
+16 -28
View File
@@ -16,8 +16,21 @@ public interface IDeploymentManagerRepository
/// <returns>The deployment record, or null if not found.</returns>
Task<DeploymentRecord?> GetDeploymentRecordByIdAsync(int id, CancellationToken cancellationToken = default);
/// <summary>
/// Database-side paged, filtered and instance-joined deployment listing —
/// the read path behind the Central UI deployment-status page.
/// Database-side paged, filtered and instance-joined deployment listing — THE
/// paged read of <c>DeploymentRecords</c>, shared by the Central UI
/// deployment-status page and the CLI/HTTP <c>QueryDeployments</c> command.
///
/// <para>
/// It was two reads until deferred-work register row 37. The other one
/// (<c>QueryDeploymentSummariesAsync</c>) differed in three ways that were all
/// defects rather than requirements: it took a raw <c>skip</c> instead of a page
/// number, returned no total (so a truncated page was indistinguishable from a
/// complete one), skipped the <c>Instance</c> join, and scoped by a caller-resolved
/// INSTANCE-ID LIST — which its one caller built by loading every instance in the
/// system on every scoped query. Site scoping is expressed here as
/// <see cref="Types.Deployment.DeploymentListFilter.SiteIdScope"/> and resolved by
/// the join, so no caller needs that read.
/// </para>
///
/// <para>
/// Replaces a <c>GetAllDeploymentRecords</c> whole-table read. That method's only
@@ -50,7 +63,7 @@ public interface IDeploymentManagerRepository
/// </summary>
/// <param name="filter">Status / search / site-scope / instance constraints; all applied DB-side.</param>
/// <param name="pageNumber">1-based page number; values below 1 are floored at 1.</param>
/// <param name="pageSize">Rows per page; clamped to <c>[1, DeploymentRecordSummary.MaxPageSize]</c>.</param>
/// <param name="pageSize">Rows per page; clamped to <c>[1, DeploymentListPage.MaxPageSize]</c>. There is no "unlimited" value.</param>
/// <param name="cancellationToken">A cancellation token that can be used to cancel the operation.</param>
/// <returns>The requested page of rows plus the total count of the filtered set.</returns>
Task<Types.Deployment.DeploymentListPage> QueryDeploymentListPageAsync(
@@ -80,31 +93,6 @@ public interface IDeploymentManagerRepository
Types.Deployment.DeploymentListFilter filter,
CancellationToken cancellationToken = default);
/// <summary>
/// Database-side paged and filtered deployment query returning row summaries.
/// Backs the <c>QueryDeployments</c> management command, which previously
/// loaded the ENTIRE deployment-record table (insert-only, one row per deploy
/// attempt for the life of the system) and ignored the page arguments the CLI
/// was already sending.
/// </summary>
/// <param name="instanceId">Restrict to one instance's history, or <see langword="null"/> for all.</param>
/// <param name="status">Restrict to one deployment status, or <see langword="null"/> for all.</param>
/// <param name="instanceIdScope">
/// When non-null, restrict to these instance ids. Used to apply a site-scoped
/// user's permitted-site filter IN THE DATABASE rather than by loading every
/// record and every instance and filtering in memory.
/// </param>
/// <param name="skip">Rows to skip; negative values are floored at 0.</param>
/// <param name="take">Page size; non-positive or null returns every remaining row.</param>
/// <param name="cancellationToken">A cancellation token that can be used to cancel the operation.</param>
/// <returns>One page of deployment summaries, newest first.</returns>
Task<IReadOnlyList<Types.Deployment.DeploymentRecordSummary>> QueryDeploymentSummariesAsync(
int? instanceId,
DeploymentStatus? status,
IReadOnlyCollection<int>? instanceIdScope,
int skip,
int? take,
CancellationToken cancellationToken = default);
/// <summary>
/// Deletes TERMINAL deployment records (<see cref="DeploymentStatus.Success"/> /
/// <see cref="DeploymentStatus.Failed"/>) that completed before
/// <paramref name="cutoffUtc"/>, in one bounded batch.
src/ZB.MOM.WW.ScadaBridge.Commons/Messages/DataConnection/ReconfigureConnectionCommand.cs
+38
View File
@@ -0,0 +1,38 @@
namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
/// <summary>
/// Command to apply a new configuration to an <b>already-live</b> data connection
/// <b>in place</b> — sent from DeploymentManagerActor to the DCL Manager Actor whenever the
/// arbitration overlay resolves a NEW winning configuration for a connection name the DCL
/// already holds (deferred register row 97).
///
/// <para>
/// <b>Why this is a separate command from <see cref="CreateConnectionCommand"/>.</b>
/// <c>DataConnectionManagerActor.HandleCreateConnection</c> is create-only — it warns and
/// returns for a name it already holds — so a re-issued create never reached the live
/// connection actor, and a changed endpoint/credential took effect only when the connection was
/// next built from scratch (node restart, singleton failover, or a remove-then-add). Stopping
/// and re-creating the actor is NOT the fix: the subscriptions live inside
/// <c>DataConnectionActor</c> and <c>InstanceActor</c> cancels its subscribe-retry timer on the
/// first success, so nothing would re-subscribe — it would trade "stale endpoint, data flowing"
/// for "correct endpoint, no data until redeploy". Instead the connection actor swaps its
/// adapter through the same machinery the primary→backup failover path uses and re-issues every
/// subscription it already holds.
/// </para>
///
/// <para>
/// <b>Safe whatever the manager's state.</b> A reconfigure naming a connection the manager does
/// not hold falls back to a create, so a sender never has to know whether the actor exists.
/// </para>
/// </summary>
/// <param name="ConnectionName">Name of the connection to reconfigure (unchanged by this command).</param>
/// <param name="ProtocolType">Protocol type of the NEW configuration; may differ from the current one.</param>
/// <param name="PrimaryConnectionDetails">Flattened connection details for the new primary endpoint.</param>
/// <param name="BackupConnectionDetails">Flattened connection details for the new backup endpoint, or null for none.</param>
/// <param name="FailoverRetryCount">Consecutive failures before switching endpoints under the new configuration.</param>
public record ReconfigureConnectionCommand(
string ConnectionName,
string ProtocolType,
IDictionary<string, string> PrimaryConnectionDetails,
IDictionary<string, string>? BackupConnectionDetails = null,
int FailoverRetryCount = 3);
src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/DeploymentCommands.cs
+32
View File
@@ -23,4 +23,36 @@ public record MgmtDeployArtifactsCommand(int? SiteId = null);
/// <param name="SiteId">Database id of the target site. Required — this command is never fleet-wide.</param>
public record MgmtDeploySiteCommand(int SiteId);
public record QueryDeploymentsCommand(int? InstanceId = null, string? Status = null, int Page = 1, int PageSize = 50);
/// <summary>
/// Result wrapper for <see cref="QueryDeploymentsCommand"/>.
///
/// <para>
/// The handler used to return a bare row array. It cannot any more: the page size
/// is now BOUNDED (a non-positive <c>PageSize</c> no longer means "unlimited" —
/// deferred-work register row 87), and a bounded page without a total is a silent
/// truncation. <see cref="TotalCount"/> is what lets a caller tell "that is all of
/// them" from "there is more, ask for page 2". Mirrors
/// <c>AuditLogQueryResult</c> / <c>SecuredWriteListResult</c>, the other paged
/// management query results.
/// </para>
///
/// <para>
/// <see cref="Page"/> and <see cref="PageSize"/> echo the values the server
/// actually applied, which are not necessarily the ones requested: both are
/// floored/clamped/defaulted server-side, so the echo is the only way a caller
/// learns the effective slice. Additive contract; new fields go on the end with a
/// default.
/// </para>
/// </summary>
/// <param name="Deployments">The requested page of deployment rows, newest first.</param>
/// <param name="TotalCount">Total rows matching the filter across all pages.</param>
/// <param name="Page">The 1-based page number actually served.</param>
/// <param name="PageSize">The page size actually applied.</param>
public record DeploymentQueryResult(
IReadOnlyList<Types.Deployment.DeploymentListRow> Deployments,
int TotalCount,
int Page,
int PageSize);
public record GetDeploymentDiffCommand(int InstanceId);
src/ZB.MOM.WW.ScadaBridge.Commons/Types/Audit/AuditFilterParseException.cs
+21
View File
@@ -0,0 +1,21 @@
namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
/// <summary>
/// Thrown by <see cref="AuditQueryParamParsers.ParseEnumListStrict{TEnum}"/> when an
/// Audit Log filter value is not a member name of its enum.
/// </summary>
/// <remarks>
/// The message is composed to be shown VERBATIM to the caller — it names the offending
/// value and the full valid set, and carries no server-internal detail — so the
/// ManagementService audit endpoints surface it directly as the body of a
/// <c>400 BAD_REQUEST</c> rather than replacing it with a generic fault string.
/// </remarks>
public class AuditFilterParseException : Exception
{
/// <summary>
/// Initializes a new instance with a caller-safe message naming the bad value
/// and the valid set.
/// </summary>
/// <param name="message">The caller-safe error message.</param>
public AuditFilterParseException(string message) : base(message) { }
}
src/ZB.MOM.WW.ScadaBridge.Commons/Types/Audit/AuditQueryParamParsers.cs
+82 -4
View File
@@ -21,11 +21,24 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
/// </para>
///
/// <para>
/// <b>Lax-parse contract.</b> Every value of a repeated parameter is parsed
/// independently; an unparseable or blank element is silently dropped (NO 400)
/// rather than failing the whole set. An empty result collapses to <c>null</c> so
/// the corresponding filter dimension stays unconstrained.
/// <b>Two parse contracts, chosen per surface.</b>
/// </para>
/// <list type="bullet">
/// <item><b>Lax</b> (<see cref="ParseEnumList{TEnum}"/>): every value of a repeated
/// parameter is parsed independently; an unparseable or blank element is silently
/// dropped (NO 400) rather than failing the whole set. An empty result collapses to
/// <c>null</c> so the corresponding filter dimension stays unconstrained. This is the
/// right contract for the CentralUI <c>AuditLogPage</c> drill-in parser and the
/// CentralUI export endpoint it feeds: those read a query string a human may have
/// bookmarked or hand-edited, and a stale fragment must still render the page rather
/// than fault it.</item>
/// <item><b>Strict</b> (<see cref="ParseEnumListStrict{TEnum}"/>): an unrecognised
/// value is an ERROR naming the valid set. This is the right contract for the
/// ManagementService <c>/api/audit/query</c> + <c>/api/audit/export</c> endpoints,
/// which answer a CLI/API caller who asked a specific question: silently dropping
/// their filter returns a plausible, complete-looking result that does NOT answer it
/// (register rows 46/88/99 — the same loud-failure principle).</item>
/// </list>
/// </summary>
public static class AuditQueryParamParsers
{
@@ -57,6 +70,71 @@ public static class AuditQueryParamParsers
return parsed.Count > 0 ? parsed : null;
}
/// <summary>
/// Parses each raw value as <typeparamref name="TEnum"/> (case-insensitive),
/// REJECTING an unrecognised value with an
/// <see cref="AuditFilterParseException"/> that names the valid set. Blank
/// elements are still dropped silently (a repeated query param can legitimately
/// carry an empty slot); a set that yields no value collapses to <c>null</c> so
/// the filter dimension stays unconstrained.
/// </summary>
/// <remarks>
/// Register row 99. Two traps this closes that a bare
/// <see cref="Enum.TryParse{TEnum}(string, bool, out TEnum)"/> does not:
/// <list type="number">
/// <item><b>Numeric strings.</b> <c>TryParse</c> happily converts <c>"7"</c> —
/// and <c>"3"</c>, which lands on a REAL member — into an enum value. A filter
/// is a caller-facing name-based contract, never an ordinal one, so any value
/// starting with a digit or sign is rejected outright. Enum member names can
/// never start with those characters, so the check costs no legitimate input.</item>
/// <item><b>Undefined composites.</b> <see cref="Enum.IsDefined{TEnum}(TEnum)"/>
/// catches anything that parsed to a value with no declared member.</item>
/// </list>
/// </remarks>
/// <typeparam name="TEnum">The enum type to parse each raw value into.</typeparam>
/// <param name="rawValues">Raw query-parameter string values to parse; may be null.</param>
/// <param name="parameterName">Query-parameter name, used in the error message (e.g. <c>"status"</c>).</param>
/// <returns>A non-empty list of parsed values, or <c>null</c> if no values were supplied.</returns>
/// <exception cref="AuditFilterParseException">A supplied value is not a member name of <typeparamref name="TEnum"/>.</exception>
public static IReadOnlyList<TEnum>? ParseEnumListStrict<TEnum>(
IEnumerable<string?>? rawValues, string parameterName)
where TEnum : struct, Enum
{
if (rawValues is null)
{
return null;
}
var parsed = new List<TEnum>();
foreach (var raw in rawValues)
{
if (string.IsNullOrWhiteSpace(raw))
{
continue;
}
var trimmed = raw.Trim();
if (LooksNumeric(trimmed)
|| !Enum.TryParse<TEnum>(trimmed, ignoreCase: true, out var value)
|| !Enum.IsDefined(value))
{
throw new AuditFilterParseException(
$"Unknown {parameterName} '{trimmed}'. Valid values: {string.Join(", ", Enum.GetNames<TEnum>())}.");
}
parsed.Add(value);
}
return parsed.Count > 0 ? parsed : null;
}
/// <summary>
/// True when the value opens with a digit or a sign — i.e. an ordinal, not an
/// enum member name. No enum member name can start with these characters.
/// </summary>
private static bool LooksNumeric(string value)
=> value.Length > 0 && (char.IsAsciiDigit(value[0]) || value[0] == '+' || value[0] == '-');
/// <summary>
/// Trims each raw value and drops blank entries. Returns <c>null</c> when
/// <paramref name="rawValues"/> is <c>null</c>, empty, or every value was
src/ZB.MOM.WW.ScadaBridge.Commons/Types/Deployment/DeploymentListPage.cs
+18
View File
@@ -19,6 +19,24 @@ public sealed record DeploymentListPage(
IReadOnlyList<DeploymentListRow> Rows,
int TotalCount)
{
/// <summary>
/// Hard ceiling applied to a requested page size by every caller of the paged
/// deployment read. Lives on the page contract because the page is the single
/// shape that read now has: the older <c>DeploymentRecordSummary</c> projection
/// (which owned this constant, and whose query accepted an unlimited
/// <c>take</c>) was folded into this one by deferred-work register row 37.
/// </summary>
public const int MaxPageSize = 1000;
/// <summary>
/// Page size used when a caller supplies none (or a non-positive one). Matches
/// the CLI's own <c>--page-size</c> default, so <c>deploy status</c> with no
/// paging flags behaves identically whether the value travels on the wire or
/// is defaulted here (register row 87 — a non-positive size used to mean
/// "unlimited" and materialized the whole retention window).
/// </summary>
public const int DefaultPageSize = 50;
/// <summary>An empty page — no rows, no matches.</summary>
public static DeploymentListPage Empty { get; } = new(Array.Empty<DeploymentListRow>(), 0);
src/ZB.MOM.WW.ScadaBridge.Commons/Types/Deployment/DeploymentListRow.cs
+6 -3
View File
@@ -3,8 +3,11 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Deployment;
/// <summary>
/// One rendered row of the Central UI deployment-status list: a
/// <c>DeploymentRecord</c> projection already joined to its <c>Instance</c>.
/// One row of a deployment listing: a <c>DeploymentRecord</c> projection already
/// joined to its <c>Instance</c>. THE row projection — the Central UI
/// deployment-status page and the CLI/HTTP <c>QueryDeployments</c> command share
/// it, having previously had a projection each (deferred-work register row 37;
/// the CLI-side <c>DeploymentRecordSummary</c> was the one that went).
///
/// <para>
/// The join is what removes the page's SECOND full-table read. Rendering a row
@@ -14,7 +17,7 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Deployment;
/// </para>
///
/// <para>
/// Like <see cref="DeploymentRecordSummary"/>, the <c>RowVersion</c>
/// The <c>RowVersion</c>
/// optimistic-concurrency token is deliberately absent — this is a READ projection
/// and must never be mistaken for something that can be written back. Mutating
/// paths keep loading the tracked entity.
src/ZB.MOM.WW.ScadaBridge.Commons/Types/Deployment/DeploymentRecordSummary.cs
-54
View File
@@ -1,54 +0,0 @@
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Deployment;
/// <summary>
/// Row-shaped projection of a <c>DeploymentRecord</c> for list surfaces (CLI
/// <c>deploy list</c> / <c>instance</c> history, the Central UI deployment-status
/// page).
///
/// <para>
/// Exists so a deployment listing can be paged and filtered in the DATABASE.
/// Deployment records are insert-only — one row per deploy attempt, retained for
/// the configured retention window — so the previous "load the whole table, then
/// filter in memory" read path degraded with the age of the system rather than
/// with the size of the requested page.
/// </para>
///
/// <para>
/// The <c>RowVersion</c> optimistic-concurrency token is deliberately absent: this
/// is a READ projection, and a summary must never be mistaken for something that
/// can be written back. Mutating paths keep loading the tracked entity.
/// </para>
///
/// <para>
/// Message-contract evolution rule: additive-only. New fields go on the end with a
/// default.
/// </para>
/// </summary>
/// <param name="Id">Deployment record row id.</param>
/// <param name="DeploymentId">The logical deployment id (GUID, "N" format).</param>
/// <param name="InstanceId">Instance the deployment targeted.</param>
/// <param name="Status">Terminal or in-flight deployment status.</param>
/// <param name="RevisionHash">Revision hash of the deployed configuration.</param>
/// <param name="DeployedBy">User who initiated the deployment.</param>
/// <param name="DeployedAt">When the deployment was initiated.</param>
/// <param name="CompletedAt">When the deployment reached a terminal status, if it has.</param>
/// <param name="ErrorMessage">Failure detail when the deployment did not succeed.</param>
public record DeploymentRecordSummary(
int Id,
string DeploymentId,
int InstanceId,
DeploymentStatus Status,
string? RevisionHash,
string DeployedBy,
DateTimeOffset DeployedAt,
DateTimeOffset? CompletedAt,
string? ErrorMessage)
{
/// <summary>
/// Hard ceiling applied to a requested page size, mirroring the in-memory
/// <c>Page</c> clamp the management actor already applies elsewhere.
/// </summary>
public const int MaxPageSize = 1000;
}
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/DeploymentConfiguration.cs
+3 -2
View File
@@ -41,8 +41,9 @@ public class DeploymentRecordConfiguration : IEntityTypeConfiguration<Deployment
builder.HasIndex(d => d.InstanceId);
// Deployment-list sort key. Every list read orders DeployedAt DESC, Id DESC
// (DeploymentManagerRepository.QueryDeploymentListPageAsync and
// QueryDeploymentSummariesAsync) — Id is the deterministic tie-break, because
// (DeploymentManagerRepository.QueryDeploymentListPageAsync, which register
// row 37 made the single paged read behind BOTH the Central UI page and the
// CLI/HTTP QueryDeployments command) — Id is the deterministic tie-break, because
// a bulk site deploy stamps many rows in the same DeployedAt instant and a
// non-deterministic order would let a row repeat or vanish across pages.
// Declaring both key columns descending lets a forward scan feed the paged
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/DeploymentManagerRepository.cs
+1 -53
View File
@@ -115,7 +115,7 @@ public class DeploymentManagerRepository : IDeploymentManagerRepository
CancellationToken cancellationToken = default)
{
var page = Math.Max(1, pageNumber);
var size = Math.Clamp(pageSize, 1, DeploymentRecordSummary.MaxPageSize);
var size = Math.Clamp(pageSize, 1, DeploymentListPage.MaxPageSize);
var query = BuildListQuery(filter, applyStatus: true);
@@ -396,58 +396,6 @@ public class DeploymentManagerRepository : IDeploymentManagerRepository
return expired.Count;
}
/// <inheritdoc />
public async Task<IReadOnlyList<DeploymentRecordSummary>> QueryDeploymentSummariesAsync(
int? instanceId,
DeploymentStatus? status,
IReadOnlyCollection<int>? instanceIdScope,
int skip,
int? take,
CancellationToken cancellationToken = default)
{
var query = _dbContext.DeploymentRecords.AsNoTracking();
if (instanceId.HasValue)
query = query.Where(d => d.InstanceId == instanceId.Value);
if (status.HasValue)
query = query.Where(d => d.Status == status.Value);
if (instanceIdScope != null)
{
// Site scoping applied DB-side. An empty scope is a real, meaningful
// filter — a user permitted no in-scope instances sees nothing — so it
// must not be short-circuited into "no filter".
var scope = instanceIdScope as int[] ?? instanceIdScope.ToArray();
query = query.Where(d => scope.Contains(d.InstanceId));
}
// ThenByDescending(Id) for the same reason GetCurrentDeploymentStatusAsync
// does it: DeployedAt ties on rapid redeploys, and an unstable sort key
// makes Skip/Take paging non-deterministic (rows repeat or vanish between
// pages).
var paged = query
.OrderByDescending(d => d.DeployedAt)
.ThenByDescending(d => d.Id)
.Skip(Math.Max(0, skip));
if (take is > 0)
paged = paged.Take(Math.Min(take.Value, DeploymentRecordSummary.MaxPageSize));
return await paged
.Select(d => new DeploymentRecordSummary(
d.Id,
d.DeploymentId,
d.InstanceId,
d.Status,
d.RevisionHash,
d.DeployedBy,
d.DeployedAt,
d.CompletedAt,
d.ErrorMessage))
.ToListAsync(cancellationToken);
}
/// <inheritdoc />
public async Task<int> PurgeTerminalDeploymentRecordsAsync(
DateTimeOffset cutoffUtc,
src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs
+196 -78
View File
@@ -37,7 +37,12 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
private readonly DataConnectionOptions _options;
private readonly ISiteHealthCollector _healthCollector;
private readonly IDataConnectionFactory _factory;
private readonly string _protocolType;
/// <summary>
/// Protocol of the configuration currently in force. NOT readonly: an in-place
/// reconfiguration (<see cref="ReconfigureConnectionCommand"/>) may change the protocol, and
/// the factory keys off it when the replacement adapter is built.
/// </summary>
private string _protocolType;
private readonly ISiteEventLogger? _siteEventLogger;
/// <summary>Stash for holding messages while the connection is not in the Connected state.</summary>
@@ -202,9 +207,13 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
private readonly HashSet<string> _alarmSourcesWithoutSeparator = new(StringComparer.Ordinal);
private IDictionary<string, string> _connectionDetails;
private readonly IDictionary<string, string> _primaryConfig;
private readonly IDictionary<string, string>? _backupConfig;
private readonly int _failoverRetryCount;
// The three fields below describe the configuration currently in force. None are readonly:
// an in-place reconfiguration (ReconfigureConnectionCommand, deferred register row 97)
// replaces them wholesale and then swaps the adapter through the same machinery the
// primary→backup failover uses.
private IDictionary<string, string> _primaryConfig;
private IDictionary<string, string>? _backupConfig;
private int _failoverRetryCount;
private ActiveEndpoint _activeEndpoint = ActiveEndpoint.Primary;
private int _consecutiveFailures;
private int _consecutiveUnstableDisconnects;
@@ -339,6 +348,13 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
case UnsubscribeAlarmsRequest:
Stash.Stash();
break;
case ReconfigureConnectionCommand rc:
// NEVER stashed. A reconfigure that arrived while the initial connect was still
// in flight would otherwise sit in the stash until the OLD configuration
// succeeded — and if it never succeeds (the very case an endpoint correction is
// sent to fix) it would sit there forever.
HandleReconfigure(rc);
break;
case SubscribeCompleted sc:
// A subscribe started while Connected can complete after a transition;
// apply it so its state survives into the next ReSubscribeAll.
@@ -454,6 +470,9 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
case AdapterDisconnected:
HandleDisconnect();
break;
case ReconfigureConnectionCommand rc:
HandleReconfigure(rc);
break;
case RetryTagResolution:
HandleRetryTagResolution();
break;
@@ -477,67 +496,65 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
// ── Reconnecting State ──
private void BecomeReconnecting()
/// <param name="afterReconfigure">
/// True when this transition was caused by an in-place reconfiguration
/// (<see cref="HandleReconfigure"/>) rather than by losing the link. It suppresses the
/// unstable-disconnect accounting and the failover check — the adapter has just been
/// swapped ON PURPOSE against a brand-new configuration, so counting it as an unstable
/// cycle would let a reconfigure trip a failover onto the new backup before the new primary
/// has been tried even once — and it reconnects immediately instead of waiting out
/// <see cref="DataConnectionOptions.ReconnectInterval"/>, since there is nothing to wait for.
/// </param>
private void BecomeReconnecting(bool afterReconfigure = false)
{
_log.Warning("[{0}] Entering Reconnecting state", _connectionName);
_log.Warning("[{0}] Entering Reconnecting state{1}", _connectionName,
afterReconfigure ? " (reconfigured in place)" : string.Empty);
_healthCollector.UpdateConnectionHealth(_connectionName, ConnectionHealth.Disconnected);
_healthCollector.UpdateConnectionEndpoint(_connectionName, "Disconnected");
// Register row 54 — drops this connection out of the connected gauge while the
// total holds, which is exactly the connected < total alert condition.
ScadaBridgeTelemetry.RecordDataConnectionState(_connectionName, ConnectionHealth.Disconnected);
// Track unstable connections toward failover.
// If we were connected for less than the stability threshold, this counts
// as an unstable cycle (e.g., connect succeeded but heartbeat went stale).
var connectionDuration = DateTimeOffset.UtcNow - _lastConnectedAt;
if (_lastConnectedAt != default && connectionDuration < _options.StableConnectionThreshold)
if (!afterReconfigure)
{
_consecutiveUnstableDisconnects++;
_log.Warning("[{0}] Unstable connection (lasted {1:F0}s) — consecutive unstable disconnects: {2}/{3}",
_connectionName, connectionDuration.TotalSeconds, _consecutiveUnstableDisconnects,
_backupConfig != null ? _failoverRetryCount : 0);
}
else
{
_consecutiveUnstableDisconnects = 0;
}
// Failover if we keep connecting and going stale repeatedly
if (_backupConfig != null && _consecutiveUnstableDisconnects >= _failoverRetryCount)
{
var previousEndpoint = _activeEndpoint;
_activeEndpoint = _activeEndpoint == ActiveEndpoint.Primary
? ActiveEndpoint.Backup
: ActiveEndpoint.Primary;
_consecutiveUnstableDisconnects = 0;
_consecutiveFailures = 0;
var newConfig = _activeEndpoint == ActiveEndpoint.Primary
? _primaryConfig
: _backupConfig;
// Dispose old adapter
_adapter.Disconnected -= OnAdapterDisconnected;
_ = _adapter.DisposeAsync().AsTask();
// Create new adapter for the target endpoint
_adapter = _factory.Create(_protocolType, newConfig);
_connectionDetails = newConfig;
_adapter.Disconnected += OnAdapterDisconnected;
// New adapter — bump the generation so callbacks
// from the disposed adapter are recognised as stale and dropped.
_adapterGeneration++;
_log.Warning("[{0}] Failing over from {1} to {2} (unstable connection pattern)",
_connectionName, previousEndpoint, _activeEndpoint);
if (_siteEventLogger != null)
// Track unstable connections toward failover.
// If we were connected for less than the stability threshold, this counts
// as an unstable cycle (e.g., connect succeeded but heartbeat went stale).
var connectionDuration = DateTimeOffset.UtcNow - _lastConnectedAt;
if (_lastConnectedAt != default && connectionDuration < _options.StableConnectionThreshold)
{
_ = _siteEventLogger.LogEventAsync(
"connection", "Warning", null, _connectionName,
$"Failover from {previousEndpoint} to {_activeEndpoint} (unstable connection)",
$"Connection lasted {connectionDuration.TotalSeconds:F0}s, threshold {_options.StableConnectionThreshold.TotalSeconds:F0}s");
_consecutiveUnstableDisconnects++;
_log.Warning("[{0}] Unstable connection (lasted {1:F0}s) — consecutive unstable disconnects: {2}/{3}",
_connectionName, connectionDuration.TotalSeconds, _consecutiveUnstableDisconnects,
_backupConfig != null ? _failoverRetryCount : 0);
}
else
{
_consecutiveUnstableDisconnects = 0;
}
// Failover if we keep connecting and going stale repeatedly
if (_backupConfig != null && _consecutiveUnstableDisconnects >= _failoverRetryCount)
{
var previousEndpoint = _activeEndpoint;
_activeEndpoint = _activeEndpoint == ActiveEndpoint.Primary
? ActiveEndpoint.Backup
: ActiveEndpoint.Primary;
_consecutiveUnstableDisconnects = 0;
_consecutiveFailures = 0;
SwapAdapter(_activeEndpoint == ActiveEndpoint.Primary ? _primaryConfig : _backupConfig);
_log.Warning("[{0}] Failing over from {1} to {2} (unstable connection pattern)",
_connectionName, previousEndpoint, _activeEndpoint);
if (_siteEventLogger != null)
{
_ = _siteEventLogger.LogEventAsync(
"connection", "Warning", null, _connectionName,
$"Failover from {previousEndpoint} to {_activeEndpoint} (unstable connection)",
$"Connection lasted {connectionDuration.TotalSeconds:F0}s, threshold {_options.StableConnectionThreshold.TotalSeconds:F0}s");
}
}
}
@@ -545,8 +562,11 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
if (_siteEventLogger != null)
{
_ = _siteEventLogger.LogEventAsync(
"connection", "Warning", null, _connectionName,
$"Connection lost — entering reconnect cycle", null);
"connection", afterReconfigure ? "Info" : "Warning", null, _connectionName,
afterReconfigure
? "Connection reconfigured — reconnecting on the new configuration"
: "Connection lost — entering reconnect cycle",
null);
}
Become(Reconnecting);
@@ -558,8 +578,118 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
// (mark mirrored alarms uncertain; the reconnect snapshot reconciles them).
PushAlarmSourceUnavailable();
// Schedule reconnect attempt
Timers.StartSingleTimer("reconnect", new AttemptConnect(), _options.ReconnectInterval);
// Schedule reconnect attempt. StartSingleTimer REPLACES any reconnect already armed,
// so a reconfigure landing mid-backoff cannot leave two AttemptConnect messages racing.
Timers.StartSingleTimer("reconnect", new AttemptConnect(),
afterReconfigure ? TimeSpan.Zero : _options.ReconnectInterval);
}
/// <summary>
/// Replaces the live adapter with one built for <paramref name="newConfig"/> — the single
/// swap shared by both failover sites and by in-place reconfiguration
/// (<see cref="HandleReconfigure"/>).
///
/// <para>
/// The old adapter is unhooked first and disposed fire-and-forget (never awaited on the
/// actor thread), and <see cref="_adapterGeneration"/> is bumped LAST so every callback and
/// in-flight background task started against the outgoing adapter is recognised as stale and
/// dropped rather than delivered as if it came from the new endpoint. Subscriptions are
/// untouched: they live in <see cref="_subscriptionsByInstance"/> and are re-issued by
/// <see cref="ReSubscribeAll"/> once the replacement adapter connects.
/// </para>
/// </summary>
private void SwapAdapter(IDictionary<string, string> newConfig)
{
_adapter.Disconnected -= OnAdapterDisconnected;
_ = _adapter.DisposeAsync().AsTask();
_adapter = _factory.Create(_protocolType, newConfig);
_connectionDetails = newConfig;
_adapter.Disconnected += OnAdapterDisconnected;
_adapterGeneration++;
}
/// <summary>
/// Applies a new configuration to this LIVE connection <b>in place</b> (deferred register
/// row 97) — the non-destructive alternative to stopping and re-creating the connection
/// actor, which would strand every subscriber (the subscription registry lives here, and
/// <c>InstanceActor</c> cancels its subscribe-retry timer on the first success, so nothing
/// would ever re-subscribe).
///
/// <para>
/// It reuses the primary→backup failover machinery verbatim: <see cref="SwapAdapter"/>
/// disposes the current adapter, builds one from the new configuration and bumps the
/// generation, then the reconnect cycle re-issues every subscription from
/// <see cref="_subscriptionsByInstance"/> and every alarm feed from
/// <see cref="_alarmSourceSubscribers"/>. In-flight writes against the outgoing adapter may
/// fail — the same accepted consequence a failover has, and the failure is returned
/// synchronously to the calling script exactly as it would be there.
/// </para>
///
/// <para>
/// <b>Endpoint selection is re-armed from the top.</b> Whatever endpoint the PREVIOUS
/// configuration had failed over to, the new configuration's primary is the operator's
/// intent, so the active endpoint and both failure counters reset.
/// </para>
///
/// <para>
/// <b>Idempotent.</b> A reconfigure carrying the configuration already in force is a no-op.
/// The sender is hash-gated and does not send one, but a swap is destructive enough that a
/// duplicate must never cost a reconnect.
/// </para>
/// </summary>
private void HandleReconfigure(ReconfigureConnectionCommand command)
{
if (string.Equals(_protocolType, command.ProtocolType, StringComparison.Ordinal)
&& _failoverRetryCount == command.FailoverRetryCount
&& DetailsEqual(_primaryConfig, command.PrimaryConnectionDetails)
&& DetailsEqual(_backupConfig, command.BackupConnectionDetails))
{
_log.Debug("[{0}] Reconfigure carries the configuration already in force — ignoring",
_connectionName);
return;
}
_log.Warning(
"[{0}] Reconfiguring in place (protocol {1} → {2}) — swapping the adapter and "
+ "re-subscribing {3} tag(s) for {4} instance(s) and {5} alarm source(s)",
_connectionName, _protocolType, command.ProtocolType,
_instancesByTag.Count, _subscriptionsByInstance.Count, _alarmSourceSubscribers.Count);
_protocolType = command.ProtocolType;
_primaryConfig = command.PrimaryConnectionDetails;
_backupConfig = command.BackupConnectionDetails;
_failoverRetryCount = command.FailoverRetryCount;
_activeEndpoint = ActiveEndpoint.Primary;
_consecutiveFailures = 0;
_consecutiveUnstableDisconnects = 0;
SwapAdapter(_primaryConfig);
BecomeReconnecting(afterReconfigure: true);
}
/// <summary>
/// Order-insensitive value comparison of two flattened connection-detail maps, used only by
/// <see cref="HandleReconfigure"/>'s no-op guard.
/// </summary>
private static bool DetailsEqual(
IDictionary<string, string>? left, IDictionary<string, string>? right)
{
if (ReferenceEquals(left, right)) return true;
if (left == null || right == null) return false;
if (left.Count != right.Count) return false;
foreach (var (key, value) in left)
{
if (!right.TryGetValue(key, out var other)
|| !string.Equals(value, other, StringComparison.Ordinal))
return false;
}
return true;
}
private void Reconnecting(object message)
@@ -586,6 +716,11 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
// Allow alarm unsubscribe during reconnect (cleanup on instance stop).
HandleUnsubscribeAlarms(areq);
break;
case ReconfigureConnectionCommand rc:
// NEVER stashed — this is the state a connection to a wrong or dead endpoint
// spends its life in, i.e. exactly when a corrected configuration arrives.
HandleReconfigure(rc);
break;
case TagValueReceived:
// Ignore — stale callback from previous connection
break;
@@ -720,24 +855,7 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
: ActiveEndpoint.Primary;
_consecutiveFailures = 0;
var newConfig = _activeEndpoint == ActiveEndpoint.Primary
? _primaryConfig
: _backupConfig;
// Dispose old adapter (fire-and-forget — don't await in actor context)
_adapter.Disconnected -= OnAdapterDisconnected;
_ = _adapter.DisposeAsync().AsTask();
// Create new adapter for the target endpoint
_adapter = _factory.Create(_protocolType, newConfig);
_connectionDetails = newConfig;
// Wire disconnect handler on new adapter
_adapter.Disconnected += OnAdapterDisconnected;
// New adapter — bump the generation so callbacks
// from the disposed adapter are recognised as stale and dropped.
_adapterGeneration++;
SwapAdapter(_activeEndpoint == ActiveEndpoint.Primary ? _primaryConfig : _backupConfig);
_log.Warning("[{0}] Failing over from {1} to {2}",
_connectionName, previousEndpoint, _activeEndpoint);
src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionManagerActor.cs
+35
View File
@@ -58,6 +58,7 @@ public class DataConnectionManagerActor : ReceiveActor
_opcUaGlobalOptions = opcUaGlobalOptions ?? new OpcUaGlobalOptions();
Receive<CreateConnectionCommand>(HandleCreateConnection);
Receive<ReconfigureConnectionCommand>(HandleReconfigureConnection);
Receive<SubscribeTagsRequest>(HandleRoute);
Receive<UnsubscribeTagsRequest>(HandleRoute);
Receive<SubscribeAlarmsRequest>(HandleRouteAlarms);
@@ -102,6 +103,40 @@ public class DataConnectionManagerActor : ReceiveActor
command.ConnectionName, command.ProtocolType);
}
/// <summary>
/// Applies a new configuration to a LIVE connection actor (deferred register row 97) by
/// forwarding the command to it — the actor swaps its adapter in place and re-issues every
/// subscription it holds, rather than being stopped and rebuilt (which would silently strand
/// every subscriber; see <see cref="ReconfigureConnectionCommand"/>).
///
/// <para>
/// A name this manager does not hold falls back to <see cref="HandleCreateConnection"/>, so
/// the command is safe regardless of manager state: a sender whose view of which connections
/// exist is stale (a fresh singleton incarnation, a node that has just taken over) still
/// converges on the intended configuration in one message.
/// </para>
/// </summary>
private void HandleReconfigureConnection(ReconfigureConnectionCommand command)
{
if (_connectionActors.TryGetValue(command.ConnectionName, out var actor))
{
_log.Info("Reconfiguring DataConnectionActor for {0} in place (protocol={1})",
command.ConnectionName, command.ProtocolType);
actor.Forward(command);
return;
}
_log.Info(
"Reconfigure for {0} arrived with no live connection actor — creating it instead",
command.ConnectionName);
HandleCreateConnection(new CreateConnectionCommand(
command.ConnectionName,
command.ProtocolType,
command.PrimaryConnectionDetails,
command.BackupConnectionDetails,
command.FailoverRetryCount));
}
private void HandleRoute(SubscribeTagsRequest request)
{
if (_connectionActors.TryGetValue(request.ConnectionName, out var actor))
src/ZB.MOM.WW.ScadaBridge.ManagementService/AuditEndpoints.cs
+60 -8
View File
@@ -133,7 +133,16 @@ public static class AuditEndpoints
return Forbidden("OperationalAudit");
}
var filter = ParseFilter(context.Request.Query);
AuditLogQueryFilter filter;
try
{
filter = ParseFilter(context.Request.Query);
}
catch (AuditFilterParseException ex)
{
return BadFilter(ex);
}
var restricted = ApplySiteScope(filter, auth.User!);
if (restricted is null)
{
@@ -216,7 +225,16 @@ public static class AuditEndpoints
statusCode: 400);
}
var filter = ParseFilter(context.Request.Query);
AuditLogQueryFilter filter;
try
{
filter = ParseFilter(context.Request.Query);
}
catch (AuditFilterParseException ex)
{
return BadFilter(ex);
}
var restricted = ApplySiteScope(filter, auth.User!);
if (restricted is null)
{
@@ -532,6 +550,21 @@ public static class AuditEndpoints
private static IResult Forbidden(string permission) => Results.Json(
new { error = $"Permission '{permission}' required.", code = "UNAUTHORIZED" }, statusCode: 403);
/// <summary>
/// Maps an <see cref="AuditFilterParseException"/> to the endpoint's existing
/// <c>400 BAD_REQUEST</c> shape — the same envelope the unsupported-export-format
/// branch already returns, so the CLI needs no new error handling.
/// </summary>
/// <remarks>
/// The exception message is surfaced VERBATIM. It is composed by the parser to be
/// caller-safe (offending value + the full valid set) and naming the valid set is
/// the entire point of the check (register row 99).
/// </remarks>
/// <param name="ex">The parse failure to surface.</param>
/// <returns>A 400 result carrying the caller-safe message.</returns>
private static IResult BadFilter(AuditFilterParseException ex) => Results.Json(
new { error = ex.Message, code = "BAD_REQUEST" }, statusCode: 400);
/// <summary>
/// Applies the caller's <see cref="AuthenticatedUser.PermittedSiteIds"/> to the
/// audit-log filter. System-wide callers (empty PermittedSiteIds —
@@ -587,23 +620,42 @@ public static class AuditEndpoints
/// <c>channel</c>/<c>kind</c>/<c>status</c>/<c>sourceSiteId</c> dimensions are
/// multi-value: a repeated query param (<c>channel=A&amp;channel=B</c>) yields
/// a multi-element filter list, while a single param yields a one-element
/// list. Unknown enum names / un-parseable Guids / dates are silently dropped
/// (no 400) — the same lax contract the CentralUI export endpoint uses; an
/// unparseable value within a repeated set is dropped, not the whole set.
/// list.
/// </summary>
/// <remarks>
/// <para>
/// <b>The enum filters parse STRICTLY (register row 99); the Guid/date filters
/// stay lax.</b> An unrecognised <c>channel</c>/<c>kind</c>/<c>status</c> throws
/// <see cref="AuditFilterParseException"/>, which the two handlers turn into a
/// <c>400 BAD_REQUEST</c> naming the valid set. Silently dropping a filter here
/// used to hand the CLI a plausible, complete-looking page that did not answer
/// the question asked — the same failure row 88 fixed for <c>deploy status</c>.
/// The CLI reaches this endpoint with case-insensitive values now that its own
/// case-SENSITIVE <c>AcceptOnlyFromAmong</c> guard is gone, so this is the
/// authoritative check for those three dimensions.
/// </para>
/// <para>
/// The CentralUI <c>AuditLogPage</c> drill-in parser and the CentralUI export
/// endpoint it feeds deliberately keep the LAX
/// <see cref="AuditQueryParamParsers.ParseEnumList{TEnum}"/> — they read a query
/// string a human may have bookmarked or hand-edited, where a stale fragment
/// must still render the page. Do not "reconcile" the two.
/// </para>
/// <para>
/// This endpoint reads the source-site filter from the <c>sourceSiteId</c>
/// query key, whereas the CentralUI export endpoint reads it as <c>site</c>.
/// The divergence is deliberate — each endpoint matches its own CLI / UI URL
/// builder — so do NOT "fix" the two to a single key name.
/// </para>
/// </remarks>
/// <param name="query">The HTTP query string collection to parse filter parameters from.</param>
/// <returns>An <see cref="AuditLogQueryFilter"/> built from the supplied query parameters.</returns>
/// <exception cref="AuditFilterParseException">A channel/kind/status value is not a recognised member name.</exception>
public static AuditLogQueryFilter ParseFilter(IQueryCollection query)
{
var channels = AuditQueryParamParsers.ParseEnumList<AuditChannel>(query["channel"]);
var kinds = AuditQueryParamParsers.ParseEnumList<AuditKind>(query["kind"]);
var statuses = AuditQueryParamParsers.ParseEnumList<AuditStatus>(query["status"]);
var channels = AuditQueryParamParsers.ParseEnumListStrict<AuditChannel>(query["channel"], "channel");
var kinds = AuditQueryParamParsers.ParseEnumListStrict<AuditKind>(query["kind"], "kind");
var statuses = AuditQueryParamParsers.ParseEnumListStrict<AuditStatus>(query["status"], "status");
var sourceSiteIds = AuditQueryParamParsers.ParseStringList(query["sourceSiteId"]);
Guid? correlationId = null;
src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs
+198 -43
View File
@@ -25,6 +25,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Transport;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.Commons.Types.InboundApi;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Transport;
@@ -535,8 +536,49 @@ public class ManagementActor : ReceiveActor
}
}
/// <summary>
/// The fail-CLOSED arm of the scope gate: a target that could NOT be resolved is
/// denied for a site-scoped caller, and ignored for a system-wide one.
/// </summary>
/// <remarks>
/// Register row 98. The scope gate used to be fail-OPEN on an unresolvable id —
/// the resolve-then-check shape was written as <c>if (x != null) EnforceSiteScope(…)</c>,
/// so a null resolve skipped the check entirely and a site-scoped caller passed the
/// gate. Authorization now fails CLOSED: a target that cannot be PROVEN to sit
/// inside the caller's scope is treated as outside it.
///
/// <para>
/// <b>System-wide callers are untouched.</b> Administrator, or a Deployer with no
/// site grants (the rule documented on <see cref="EnforceSiteScope"/>), return at
/// the guards below — they were never scoped, so they still reach the handler and
/// still get its ordinary "not found"/null answer for a missing id.
/// </para>
///
/// <para>
/// <b>The denial is deliberately indistinguishable from an out-of-scope one.</b>
/// It raises the same <see cref="SiteScopeViolationException"/> (→
/// <c>ManagementUnauthorized</c>), and <paramref name="targetDescription"/> must
/// never state whether the target exists — otherwise the gate becomes an
/// enumeration oracle letting a scoped caller probe ids across the whole fleet.
/// </para>
/// </remarks>
/// <param name="user">The authenticated caller.</param>
/// <param name="targetDescription">Caller-safe description of the target, e.g. <c>"instance 42"</c>.</param>
/// <exception cref="SiteScopeViolationException">The caller is site-scoped.</exception>
private static void DenyUnresolvedTargetIfScoped(AuthenticatedUser user, string targetDescription)
{
if (user.PermittedSiteIds.Length == 0) return;
if (user.Roles.Contains(Roles.Administrator, StringComparer.OrdinalIgnoreCase)) return;
throw new SiteScopeViolationException(
$"Access denied: {targetDescription} is not within your permitted sites " +
$"[{string.Join(", ", user.PermittedSiteIds)}].");
}
/// <summary>
/// Resolves the site ID for an instance and enforces site-scope.
/// An instance that cannot be resolved is DENIED for a site-scoped caller
/// (see <see cref="DenyUnresolvedTargetIfScoped"/>).
/// </summary>
private static async Task EnforceSiteScopeForInstance(IServiceProvider sp, AuthenticatedUser user, int instanceId)
{
@@ -545,13 +587,20 @@ public class ManagementActor : ReceiveActor
var repo = sp.GetRequiredService<ITemplateEngineRepository>();
var instance = await repo.GetInstanceByIdAsync(instanceId);
if (instance != null)
EnforceSiteScope(user, instance.SiteId);
if (instance is null)
{
DenyUnresolvedTargetIfScoped(user, $"instance {instanceId}");
return;
}
EnforceSiteScope(user, instance.SiteId);
}
/// <summary>
/// Resolves a site by its string identifier and enforces site-scope.
/// Used by remote-query handlers that key off the site identifier rather than its ID.
/// A site identifier that cannot be resolved is DENIED for a site-scoped caller
/// (see <see cref="DenyUnresolvedTargetIfScoped"/>).
/// </summary>
private static async Task EnforceSiteScopeForIdentifier(IServiceProvider sp, AuthenticatedUser user, string siteIdentifier)
{
@@ -560,8 +609,13 @@ public class ManagementActor : ReceiveActor
var repo = sp.GetRequiredService<ISiteRepository>();
var site = await repo.GetSiteByIdentifierAsync(siteIdentifier);
if (site != null)
EnforceSiteScope(user, site.Id);
if (site is null)
{
DenyUnresolvedTargetIfScoped(user, $"site '{siteIdentifier}'");
return;
}
EnforceSiteScope(user, site.Id);
}
/// <summary>
@@ -835,12 +889,28 @@ public class ManagementActor : ReceiveActor
return Page(instances, cmd.Skip, cmd.Take);
}
/// <summary>
/// Reads one instance, scope-checked against its own site.
/// </summary>
/// <remarks>
/// This handler resolves the instance ITSELF rather than calling
/// <see cref="EnforceSiteScopeForInstance"/> — the helper would repeat the same
/// repository read — so it carries its own copy of the fail-CLOSED arm
/// (register row 98). A scoped caller must not be able to tell a nonexistent
/// instance (which used to answer success/null) from one at a site they may not
/// see, or the read becomes an id-enumeration oracle over the whole fleet.
/// </remarks>
private static async Task<object?> HandleGetInstance(IServiceProvider sp, GetInstanceCommand cmd, AuthenticatedUser user)
{
var repo = sp.GetRequiredService<ITemplateEngineRepository>();
var instance = await repo.GetInstanceByIdAsync(cmd.InstanceId);
if (instance != null)
EnforceSiteScope(user, instance.SiteId);
if (instance is null)
{
DenyUnresolvedTargetIfScoped(user, $"instance {cmd.InstanceId}");
return null;
}
EnforceSiteScope(user, instance.SiteId);
return instance;
}
@@ -1658,12 +1728,22 @@ public class ManagementActor : ReceiveActor
return sites;
}
/// <summary>
/// Reads one site, scope-checked against itself. Fail-CLOSED on an unresolvable
/// id for a scoped caller — see <see cref="DenyUnresolvedTargetIfScoped"/>
/// (register row 98).
/// </summary>
private static async Task<object?> HandleGetSite(IServiceProvider sp, GetSiteCommand cmd, AuthenticatedUser user)
{
var repo = sp.GetRequiredService<ISiteRepository>();
var site = await repo.GetSiteByIdAsync(cmd.SiteId);
if (site != null)
EnforceSiteScope(user, site.Id);
if (site is null)
{
DenyUnresolvedTargetIfScoped(user, $"site {cmd.SiteId}");
return null;
}
EnforceSiteScope(user, site.Id);
return site;
}
@@ -1800,13 +1880,23 @@ public class ManagementActor : ReceiveActor
return (await repo.GetAllDataConnectionsAsync()).Select(DataConnectionPublicShape).ToList();
}
/// <summary>
/// Reads one data connection, scope-checked against its owning site. Fail-CLOSED
/// on an unresolvable id for a scoped caller — see
/// <see cref="DenyUnresolvedTargetIfScoped"/> (register row 98).
/// </summary>
private static async Task<object?> HandleGetDataConnection(IServiceProvider sp, GetDataConnectionCommand cmd, AuthenticatedUser user)
{
var repo = sp.GetRequiredService<ISiteRepository>();
var conn = await repo.GetDataConnectionByIdAsync(cmd.DataConnectionId);
if (conn != null)
EnforceSiteScope(user, conn.SiteId);
return conn is null ? null : DataConnectionPublicShape(conn);
if (conn is null)
{
DenyUnresolvedTargetIfScoped(user, $"data connection {cmd.DataConnectionId}");
return null;
}
EnforceSiteScope(user, conn.SiteId);
return DataConnectionPublicShape(conn);
}
private static async Task<object?> HandleCreateDataConnection(IServiceProvider sp, CreateDataConnectionCommand cmd, string user)
@@ -2558,53 +2648,118 @@ public class ManagementActor : ReceiveActor
if (sp.GetService<DeploymentService>() is { } deploymentService)
await deploymentService.TryPurgeTerminalDeploymentRecordsAsync();
// Page and Status are honoured DB-side. Both arrived on the command
// contract (and the CLI has been sending --page / --page-size / --status
// all along) but the handler previously ignored all three and returned the
// whole table.
// Page, PageSize and Status are all honoured DB-side, through the SAME
// paged read the Central UI deployment page uses (register row 37 — there
// used to be a second, subtly different repository query behind this
// handler alone).
var status = ParseDeploymentStatus(cmd.Status);
var skip = Math.Max(0, (Math.Max(1, cmd.Page) - 1) * Math.Max(1, cmd.PageSize));
var take = cmd.PageSize > 0 ? cmd.PageSize : (int?)null;
var page = Math.Max(1, cmd.Page);
var pageSize = ResolveDeploymentPageSize(cmd.PageSize);
// Instance-scoped query: enforce against the target instance's site
// before reading its deployment history.
if (cmd.InstanceId.HasValue)
{
await EnforceSiteScopeForInstance(sp, user, cmd.InstanceId.Value);
return await repo.QueryDeploymentSummariesAsync(
cmd.InstanceId.Value, status, instanceIdScope: null, skip, take);
return await QueryDeploymentPage(
repo, new DeploymentListFilter(status, InstanceId: cmd.InstanceId.Value), page, pageSize);
}
// Unfiltered query: a site-scoped Deployment user must only see records
// for instances at sites within their scope. DeploymentRecord has no
// SiteId, so the scope is resolved to the set of in-scope instance ids and
// pushed into the query as an id filter — rather than loading every record
// AND every instance and intersecting them in memory, which made an
// unfiltered listing cost two full-table loads regardless of page size.
if (user.PermittedSiteIds.Length == 0 || user.Roles.Contains(Roles.Administrator, StringComparer.OrdinalIgnoreCase))
return await repo.QueryDeploymentSummariesAsync(null, status, instanceIdScope: null, skip, take);
var permittedIds = new HashSet<string>(user.PermittedSiteIds);
var templateRepo = sp.GetRequiredService<ITemplateEngineRepository>();
var allInstances = await templateRepo.GetAllInstancesAsync();
var scopedInstanceIds = allInstances
.Where(i => permittedIds.Contains(i.SiteId.ToString()))
.Select(i => i.Id)
.ToArray();
return await repo.QueryDeploymentSummariesAsync(null, status, scopedInstanceIds, skip, take);
// SiteId of its own, so the scope rides as a SITE-ID filter resolved by the
// query's own join to Instance. It used to be resolved here instead, by
// loading EVERY instance in the system on every scoped query to turn the
// site grant into an instance-id list (register row 87).
var siteScope = ResolveSiteIdScope(user);
return await QueryDeploymentPage(
repo, new DeploymentListFilter(status, SiteIdScope: siteScope), page, pageSize);
}
/// <summary>
/// Parses the wire <c>Status</c> filter into a <see cref="DeploymentStatus"/>.
/// An unparseable value is treated as "no filter" rather than an error, which
/// preserves the previous handler's behaviour of ignoring the field entirely.
/// Runs the shared paged deployment read and wraps it in the command's result
/// envelope, echoing the page coordinates that were actually applied.
/// </summary>
private static DeploymentStatus? ParseDeploymentStatus(string? status) =>
!string.IsNullOrWhiteSpace(status) && Enum.TryParse<DeploymentStatus>(status, ignoreCase: true, out var parsed)
? parsed
: null;
private static async Task<DeploymentQueryResult> QueryDeploymentPage(
IDeploymentManagerRepository repo, DeploymentListFilter filter, int page, int pageSize)
{
var result = await repo.QueryDeploymentListPageAsync(filter, page, pageSize);
return new DeploymentQueryResult(result.Rows, result.TotalCount, page, pageSize);
}
/// <summary>
/// Resolves the caller's site-scope grant into the query's
/// <see cref="DeploymentListFilter.SiteIdScope"/>: <see langword="null"/> for a
/// system-wide caller, otherwise the permitted site ids.
/// </summary>
/// <remarks>
/// Administrator is system-wide here, matching <see cref="EnforceSiteScope"/>
/// and the rest of this actor. A permitted-site id that is not an integer is
/// DROPPED rather than ignored: dropping can only narrow the grant, whereas
/// treating the whole grant as unparseable would widen it to fleet-wide. A
/// grant of nothing but garbage therefore yields an EMPTY scope, which the
/// filter treats as a real "sees nothing" constraint.
/// </remarks>
private static IReadOnlyCollection<int>? ResolveSiteIdScope(AuthenticatedUser user)
{
if (user.PermittedSiteIds.Length == 0) return null;
if (user.Roles.Contains(Roles.Administrator, StringComparer.OrdinalIgnoreCase)) return null;
var ids = new List<int>(user.PermittedSiteIds.Length);
foreach (var raw in user.PermittedSiteIds)
{
if (int.TryParse(raw, out var id))
ids.Add(id);
}
return ids;
}
/// <summary>
/// Bounds the requested deployment page size: a non-positive value means
/// "unspecified" and takes the default, and every value is capped at
/// <see cref="DeploymentListPage.MaxPageSize"/>.
/// </summary>
/// <remarks>
/// Register row 87. A non-positive size used to mean UNLIMITED all the way down
/// to the repository, so <c>--page-size 0</c> materialized the entire
/// terminal-record retention window into one response. The result envelope
/// carries the total count precisely so bounding the page does not hide rows.
/// </remarks>
private static int ResolveDeploymentPageSize(int requested) =>
requested <= 0
? DeploymentListPage.DefaultPageSize
: Math.Min(requested, DeploymentListPage.MaxPageSize);
/// <summary>
/// Parses the wire <c>Status</c> filter into a <see cref="DeploymentStatus"/>.
/// Absent/blank means "no filter"; an unrecognised value is an ERROR naming the
/// valid ones.
/// </summary>
/// <remarks>
/// Register row 88. An unparseable value used to fall back to "no filter", so
/// <c>--status Sucess</c> silently returned every status — the caller saw a
/// plausible, complete-looking result that did not answer the question they
/// asked. Same loud-failure principle as row 46.
/// </remarks>
/// <exception cref="ManagementCommandException">The value is not a deployment status.</exception>
private static DeploymentStatus? ParseDeploymentStatus(string? status)
{
if (string.IsNullOrWhiteSpace(status)) return null;
var trimmed = status.Trim();
// Reject ordinals before TryParse — "2" parses to InProgress, which the
// documented names-only contract does not admit. No enum member name can
// start with a digit or a sign (same guard as AuditQueryParamParsers).
var looksNumeric = char.IsAsciiDigit(trimmed[0]) || trimmed[0] == '+' || trimmed[0] == '-';
if (!looksNumeric
&& Enum.TryParse<DeploymentStatus>(trimmed, ignoreCase: true, out var parsed)
&& Enum.IsDefined(parsed))
{
return parsed;
}
throw new ManagementCommandException(
$"Unknown deployment status '{status}'. Valid values: {string.Join(", ", Enum.GetNames<DeploymentStatus>())}.");
}
// ========================================================================
// Audit Log handler
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs
+422 -44
View File
@@ -82,6 +82,17 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// </summary>
private readonly Func<Task<List<DeployedInstance>>> _configLoader;
/// <summary>
/// Reads the durable <c>data_connection_definitions</c> rows (deferred register rows 89 + 90).
/// Defaults to <see cref="SiteStorageService.GetAllDataConnectionDefinitionsAsync"/>;
/// overridable via the ctor so a test can drive a read failure.
/// <para>
/// This table had NO production reader before rows 89/90 — it was written by the artifact
/// deploy and never read back, which is precisely why a definition change survived only as
/// long as the singleton incarnation that applied it.
/// </para>
/// </summary>
private readonly Func<Task<List<StoredDataConnectionDefinition>>> _connectionDefinitionLoader;
/// <summary>
/// Set once the startup deployed-config load has succeeded. Guards against a
/// stale <see cref="RetryStartupLoad"/> re-running the load after success.
/// </summary>
@@ -246,6 +257,12 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// towards <see cref="ScriptExecutionScheduler.ComputeTargetThreads"/> on every instance-
/// count change; null uses the process-wide shared scheduler.
/// </param>
/// <param name="connectionDefinitionLoader">
/// Rows 89 + 90: optional override for reading the durable
/// <c>data_connection_definitions</c> rows that arbitrate the live DCL configuration;
/// defaults to reading from <paramref name="storage"/>. Primarily for tests that need to
/// drive a read failure.
/// </param>
public DeploymentManagerActor(
SiteStorageService storage,
ScriptCompilationService compilationService,
@@ -261,7 +278,8 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
TimeSpan? startupLoadRetryInterval = null,
Func<Task<List<DeployedInstance>>>? configLoader = null,
ExternalSystemDefinitionCache? externalSystemCache = null,
ScriptExecutionScheduler? scriptScheduler = null)
ScriptExecutionScheduler? scriptScheduler = null,
Func<Task<List<StoredDataConnectionDefinition>>>? connectionDefinitionLoader = null)
{
_scriptScheduler = scriptScheduler;
_storage = storage;
@@ -280,6 +298,8 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
? TimeSpan.FromSeconds(options.InstanceSweepIntervalSeconds)
: TimeSpan.Zero;
_configLoader = configLoader ?? (() => _storage.GetAllDeployedConfigsAsync());
_connectionDefinitionLoader = connectionDefinitionLoader
?? (() => _storage.GetAllDataConnectionDefinitionsAsync());
_logger = logger;
// Reuse a single logger factory for all Instance Actors.
// Prefer an explicitly injected factory, fall back to one resolved from
@@ -478,14 +498,54 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// </summary>
private void LoadDeployedConfigs()
{
_configLoader().ContinueWith(t =>
// Row 101: capture on the actor thread BEFORE the off-thread read starts, so a landing
// handler can tell whether an artifact apply mutated the overlay while it was in flight.
var generationAtRead = _definitionsWriteGeneration;
ReadStoreSnapshotAsync().ContinueWith(t =>
{
if (t.IsCompletedSuccessfully)
return new StartupConfigsLoaded(t.Result, null);
return new StartupConfigsLoaded(
t.Result.Configs, null, t.Result.Definitions, generationAtRead);
return new StartupConfigsLoaded([], t.Exception?.GetBaseException().Message);
}).PipeTo(Self);
}
/// <summary>
/// Reads the two durable tables the Deployment Manager arbitrates over — the deployed
/// instance configs and the <c>data_connection_definitions</c> rows — in ONE off-thread
/// task, so the boot load and every sweep tick see a single consistent snapshot and the
/// actor thread is never blocked on SQLite.
///
/// <para>
/// <b>The two reads fail differently on purpose.</b> A failed CONFIG read propagates: it is
/// the existing S5 path and re-arms the whole load at a fixed interval. A failed DEFINITIONS
/// read is swallowed into a <see langword="null"/> result, because the DCL arbitration overlay
/// must be left EXACTLY as it was rather than cleared — an overlay cleared by a transient
/// read failure would hand arbitration back to the flattened snapshots for one pass and then
/// take it away again on the next successful read, which is the recreate-on-every-flip
/// ping-pong the whole design exists to make impossible.
/// </para>
/// </summary>
private async Task<(List<DeployedInstance> Configs, IReadOnlyList<StoredDataConnectionDefinition>? Definitions)>
ReadStoreSnapshotAsync()
{
var configs = await _configLoader().ConfigureAwait(false);
IReadOnlyList<StoredDataConnectionDefinition>? definitions = null;
try
{
definitions = await _connectionDefinitionLoader().ConfigureAwait(false);
}
catch (Exception ex)
{
_logger.LogWarning(
ex, "Could not read the data connection definitions — the DCL arbitration "
+ "overlay keeps its previous contents until the next read succeeds");
}
return (configs, definitions);
}
/// <inheritdoc />
protected override void PostStop()
{
@@ -544,6 +604,15 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
// Mark the startup load complete so a stale retry timer is ignored.
_startupLoadCompleted = true;
// Rows 89 + 90: seed the DCL arbitration overlay BEFORE any Instance Actor is created,
// so the boot batch's EnsureDclConnections already resolves to the definition-owned
// config. This is the failover half of the row-89 fix: a singleton incarnation starting
// on the peer node rebuilds the whole DCL from flattened snapshots, and without the
// overlay in place first it would rebuild them STALE and only correct itself a sweep
// tick later — a needless create-then-diverge on every failover.
ApplyConnectionDefinitionOverlayIfFresh(
msg.ConnectionDefinitions, msg.DefinitionsGenerationAtRead, "boot load");
var enabledConfigs = msg.Configs.Where(c => c.IsEnabled).ToList();
_deployedInstanceNames.Clear();
foreach (var c in msg.Configs)
@@ -613,7 +682,8 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
{
// Verdict discarded: startup does not gate on compile failures (a failing
// script is logged by the Instance Actor and leaves the rest running).
// This call exists only to populate SiteScriptCompileCache.
// This call exists only to populate the compile cache owned by the
// shared ScriptCompilationService (SiteScriptCompileCache).
validator.Validate(configJson);
}
catch
@@ -734,8 +804,8 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
}
/// <summary>
/// Compiles the deployment's scripts off the actor thread purely to populate the
/// process-wide <see cref="SiteScriptCompileCache"/>, then pipes
/// Compiles the deployment's scripts off the actor thread purely to populate the compile
/// cache owned by the shared <c>ScriptCompilationService</c> singleton, then pipes
/// <see cref="DeployCompileWarmed"/> back so the authoritative gate can run on the actor
/// thread against a warm cache. The warm is best-effort: its verdict is discarded and any
/// exception swallowed, because <see cref="RunDeployGateAndProceed"/> re-derives the
@@ -846,9 +916,10 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
// already been warmed off-thread (StartDeployCompileWarm), so this call is
// pure cache hits and no longer holds the singleton — while the per-instance
// warm guard keeps this instance's own command ordering intact. Redeploys and
// multi-instance deploys of unchanged scripts hit the process-wide compile
// cache (SiteScriptCompileCache) too, and the Instance Actor's PreStart reuses
// the gate's compile (N4).
// multi-instance deploys of unchanged scripts hit the compile cache
// (SiteScriptCompileCache, owned by the ScriptCompilationService singleton this
// actor shares with every Instance Actor) too, and the Instance Actor's PreStart
// reuses the gate's compile (N4).
var errors = _deployCompileValidator.Validate(command.FlattenedConfigurationJson);
if (errors.Count > 0)
{
@@ -1563,8 +1634,10 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
_instanceSweepInFlight = true;
_configLoader().ContinueWith(t => t.IsCompletedSuccessfully
? new InstanceSweepLoaded(t.Result, null)
// Row 101: capture on the actor thread BEFORE the off-thread read starts.
var generationAtRead = _definitionsWriteGeneration;
ReadStoreSnapshotAsync().ContinueWith(t => t.IsCompletedSuccessfully
? new InstanceSweepLoaded(t.Result.Configs, null, t.Result.Definitions, generationAtRead)
: new InstanceSweepLoaded([], t.Exception?.GetBaseException().Message))
.PipeTo(Self);
}
@@ -1589,6 +1662,15 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
return;
}
// Rows 89 + 90: refresh the DCL arbitration overlay FIRST, so any instance this same
// sweep pass goes on to start resolves its connections against the current definitions
// rather than against the snapshot frozen into its flattened config. This is the
// convergence path for a definition row that arrived by LocalDb CDC replication: no new
// cross-node message exists (or is needed) — whichever node runs the singleton reads the
// row out of its own file on the next tick.
ApplyConnectionDefinitionOverlayIfFresh(
msg.ConnectionDefinitions, msg.DefinitionsGenerationAtRead, "instance sweep");
foreach (var config in msg.Configs)
{
var instanceName = config.InstanceUniqueName;
@@ -2083,17 +2165,211 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// <summary>
/// Tracks the configuration last sent to the DCL for each connection name, keyed by
/// a hash of the connection's protocol/endpoints/credentials/failover count.
/// A name whose hash is unchanged is skipped; a name whose config
/// changed re-issues a <c>CreateConnectionCommand</c> so the DCL adopts the new
/// configuration instead of keeping a stale connection after a redeployment.
/// A name whose hash is unchanged is skipped; a name that is new gets a
/// <c>CreateConnectionCommand</c>, and a name whose config CHANGED gets a
/// <c>ReconfigureConnectionCommand</c> so the live connection actor adopts the new
/// configuration in place instead of serving a stale one until it is next rebuilt.
/// </summary>
private readonly Dictionary<string, string> _createdConnections = new();
/// <summary>
/// Sets up DCL connections from the flattened config. Idempotent on unchanged
/// configuration, but re-issues the create command when a connection's endpoint,
/// credentials, backup endpoint, or failover retry count has changed since it was
/// last sent.
/// <b>The DCL connection arbitration overlay</b> (deferred register rows 89 + 90) — the
/// <c>data_connection_definitions</c> rows this singleton incarnation currently believes the
/// site store owns, keyed by connection name.
///
/// <para>
/// <b>The rule: the definitions store wins wherever it has an opinion.</b> A connection name
/// with a row here is configured from that row; the identically-named entry inside an
/// instance's flattened config is a deploy-time SNAPSHOT and never overrides it. A name with
/// no row here falls through to the flattened config exactly as before — that is not a
/// fallback, it is the correct answer for a deploy-only site, where this table is legitimately
/// EMPTY because it is written only by the system-wide artifact deploy while the connections
/// that matter live inline in the instance configs.
/// </para>
///
/// <para>
/// <b>This overlay is an arbiter, not a creation source.</b> A definition row for a name no
/// deployed instance references creates nothing: there is no connection actor wanted for it.
/// The overlay only decides WHICH configuration a connection the runtime already wants is
/// created from — which is what makes it safe to consult on every sweep tick.
/// </para>
///
/// <para>
/// Refreshed by <see cref="ApplyConnectionDefinitionOverlay"/> from the boot load and from
/// every sweep tick, and kept consistent in-flight by
/// <see cref="HandleApplyArtifactDataConnectionsToDcl"/>. Actor-thread-confined.
/// </para>
/// </summary>
private readonly Dictionary<string, StoredDataConnectionDefinition> _connectionDefinitions =
new(StringComparer.Ordinal);
/// <summary>
/// Connection names this singleton incarnation has seen REMOVED centrally (deferred register
/// row 90) — either by an artifact reconcile it processed itself, or by a definition row that
/// disappeared from the store between two sweep reads.
///
/// <para>
/// Without it the removal does not stick for as long as one tick: the removal evicts the name
/// from <see cref="_createdConnections"/>, and the very next instance start whose flattened
/// config still names that connection re-creates it — the flattened snapshot resurrecting a
/// connection central has deleted. That is not hypothetical; the sweep starts instances.
/// </para>
///
/// <para>
/// <b>Deliberately in-memory only.</b> Persisting tombstones was rejected: a durable
/// "this connection is gone" marker is a second, divergent authority over the same names, and
/// the flattened configs already resurrect a removed connection across a restart today — that
/// pre-existing semantic is accepted and unchanged, and is recorded as the residual in
/// <c>docs/plans/2026-08-17-dcl-connection-arbitration-decision.md</c>. Cleared the moment a
/// definition row for the name reappears, so a centrally re-added connection comes straight
/// back.
/// </para>
/// </summary>
private readonly HashSet<string> _centrallyRemovedConnections = new(StringComparer.Ordinal);
/// <summary>
/// Monotonic count of actor-thread definition mutations from artifact applies (deferred
/// register row 101). A store snapshot is read OFF the actor thread and compared against
/// the overlay only when its piped message lands — so a read that started before an
/// artifact apply's store writes can land AFTER <see
/// cref="HandleApplyArtifactDataConnectionsToDcl"/> populated the overlay, and its stale
/// data would then "prove" the just-applied names disappeared, tombstoning and stopping
/// live connections that nothing re-creates until an instance redeploy. Each snapshot read
/// captures this value on the actor thread at dispatch; a result carrying a stale
/// generation skips the overlay apply entirely (additions too — a stale row pushed via
/// <see cref="EnsureDclConnection"/> would reconfigure a live connection BACKWARD and set
/// up a flip-flop with the next fresh read). The next tick reads fresh and converges.
/// </summary>
private long _definitionsWriteGeneration;
/// <summary>
/// Row 101 guard in front of <see cref="ApplyConnectionDefinitionOverlay"/>: applies the
/// snapshot only when no artifact apply mutated the definitions while the read was in
/// flight. A stale snapshot is discarded WHOLE — its absences would spuriously tombstone
/// and stop just-applied live connections (which nothing re-creates until an instance
/// redeploy), and its rows would push superseded configs onto live connections. Discarding
/// is safe because the artifact apply already updated the overlay itself and the next
/// tick's read starts after the write, so convergence is one tick away.
/// </summary>
private void ApplyConnectionDefinitionOverlayIfFresh(
IReadOnlyList<StoredDataConnectionDefinition>? definitions,
long generationAtRead,
string source)
{
if (generationAtRead != _definitionsWriteGeneration)
{
_logger.LogDebug(
"Discarding the {Source} data-connection definitions snapshot — an artifact "
+ "apply changed the definitions while the read was in flight (generation "
+ "{ReadGeneration} vs {CurrentGeneration}); the next sweep tick reads fresh",
source, generationAtRead, _definitionsWriteGeneration);
return;
}
ApplyConnectionDefinitionOverlay(definitions);
}
/// <summary>
/// Refreshes <see cref="_connectionDefinitions"/> from a store read and reconciles the live
/// DCL against it (deferred register rows 89 + 90).
///
/// <para>
/// <paramref name="definitions"/> is <see langword="null"/> when the read FAILED, which is
/// deliberately distinct from an empty list: a failed read leaves the overlay untouched,
/// while an empty list is a real answer (the deploy-only site shape, and also what a site
/// looks like after central deletes every data connection).
/// </para>
///
/// <para>
/// <b>Disappearance detection, and exactly how far it reaches.</b> A name that WAS in this
/// overlay and no longer has a row is a removal this incarnation can prove: the store owned
/// it a moment ago and does not now. It is tombstoned and its live connection actor stopped.
/// A name that was never in the overlay is indistinguishable from one the store has never
/// owned — a fresh incarnation cannot tell "central deleted this" from "this site keeps its
/// connections inline" — so it is left to the flattened config. That asymmetry is the
/// documented residual of rows 89/90, not an oversight.
/// </para>
///
/// <para>
/// Reached only through <see cref="ApplyConnectionDefinitionOverlayIfFresh"/> (row 101) —
/// never with a snapshot proven stale.
/// </para>
/// </summary>
private void ApplyConnectionDefinitionOverlay(
IReadOnlyList<StoredDataConnectionDefinition>? definitions)
{
if (definitions == null)
return;
var disappeared = _connectionDefinitions.Keys
.Where(name => !definitions.Any(d => string.Equals(d.Name, name, StringComparison.Ordinal)))
.ToList();
foreach (var definition in definitions)
{
// A definition row is the store re-asserting ownership: any tombstone for the name
// is stale, so a centrally re-added connection is creatable again immediately.
_centrallyRemovedConnections.Remove(definition.Name);
_connectionDefinitions[definition.Name] = definition;
// Push only to connections the runtime already holds. A definition nothing
// references stays inert — see the overlay's "arbiter, not a creation source" note.
if (_createdConnections.ContainsKey(definition.Name))
{
EnsureDclConnection(
definition.Name,
definition.Protocol,
definition.ConfigurationJson,
definition.BackupConfigurationJson,
definition.FailoverRetryCount);
}
}
foreach (var name in disappeared)
RemoveConnectionCentrally(name, "its definition row is no longer in the site store");
}
/// <summary>
/// Applies a central removal of a data connection: tombstones the name for this singleton
/// incarnation, drops the overlay row and the hash-cache slot, and stops the live DCL
/// connection actor. Shared by the artifact-reconcile removal and the sweep's
/// disappearance detection so both produce identical state.
///
/// <para>
/// A connection still referenced by a deployed instance's config no longer exists centrally
/// either — its attributes go bad quality, the standard disconnected signal, until the
/// instance is redeployed against current central config. That is the pre-existing removal
/// semantic and it is unchanged here; what IS new is that the removal now survives the next
/// instance start instead of being undone by it.
/// </para>
/// </summary>
private void RemoveConnectionCentrally(string name, string because)
{
_connectionDefinitions.Remove(name);
_centrallyRemovedConnections.Add(name);
_createdConnections.Remove(name);
_dclManager?.Tell(new DataConnectionLayer.Actors.RemoveConnectionCommand(name));
_logger.LogWarning(
"Data connection '{Name}' removed centrally ({Because}); its DCL connection actor "
+ "has been stopped and will not be recreated from an instance's flattened config",
name, because);
}
/// <summary>
/// Sets up DCL connections from the flattened config, <b>arbitrated against
/// <see cref="_connectionDefinitions"/></b> (deferred register rows 89 + 90). Idempotent on
/// unchanged configuration.
///
/// <para>
/// <b>Why the arbitration has to happen HERE, before the hash is computed.</b>
/// <see cref="_createdConnections"/> holds one hash slot per connection name. The flattened
/// snapshot and the definition row for the same name can legitimately differ — that is the
/// supported version-skew case, an artifact deploy newer than the instance deploys that
/// reference it. Two callers hashing from two different sources would flip that one slot back
/// and forth, and every flip is a fresh <c>CreateConnectionCommand</c>. Resolving the WINNER
/// first means only one configuration can ever be hashed for a name, whichever path calls, so
/// the flip is not merely avoided by convention — it is unrepresentable.
/// </para>
/// </summary>
private void EnsureDclConnections(string configJson)
{
@@ -2106,6 +2382,29 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
foreach (var (name, connConfig) in config.Connections)
{
// Row 90: never resurrect a name the artifact reconcile removed.
if (_centrallyRemovedConnections.Contains(name))
{
_logger.LogDebug(
"Skipping DCL connection '{Connection}' named by a flattened config — it "
+ "was removed centrally in this incarnation", name);
continue;
}
if (_connectionDefinitions.TryGetValue(name, out var owned))
{
// Rule 1: the store has an opinion, so the store wins.
EnsureDclConnection(
name,
owned.Protocol,
owned.ConfigurationJson,
owned.BackupConfigurationJson,
owned.FailoverRetryCount);
continue;
}
// Rule 2: no definition row has ever owned this name — the flattened config is
// the source, exactly as before. This is the whole of a deploy-only site.
EnsureDclConnection(
name,
connConfig.Protocol,
@@ -2124,13 +2423,29 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// Hash-guarded DCL connection push shared by the inline
/// per-instance path (<see cref="EnsureDclConnections(string)"/>) and the
/// system-wide artifact-deploy path (<see cref="HandleDeployArtifacts"/>).
/// Unchanged config is a no-op; a changed endpoint/credentials/backup/
/// failover-count re-issues a <c>CreateConnectionCommand</c> so a system-
/// wide artifact-deploy makes its data-connection change live immediately
/// (the artifact-deploy path previously only persisted to SQLite — the
/// DCL didn't see the change until next instance redeploy or node
/// restart, contradicting the "site is self-contained after artifact
/// deployment" intent).
///
/// <para>
/// <b>Three outcomes, one hash slot.</b> An unchanged configuration is a no-op (no message,
/// no log — this is what keeps a healthy site silent across every sweep tick). A name the
/// hash cache has never held gets a <c>CreateConnectionCommand</c>. A name whose
/// endpoint/credentials/backup/failover-count CHANGED gets a
/// <c>ReconfigureConnectionCommand</c>, which the DCL applies to the live connection actor
/// <b>in place</b> — it swaps its adapter through the same machinery the primary→backup
/// failover uses and re-issues every subscription it holds.
/// </para>
///
/// <para>
/// <b>Why the changed case is not a re-issued create</b> (deferred register row 97).
/// <c>DataConnectionManagerActor.HandleCreateConnection</c> is CREATE-ONLY — it warns and
/// returns for a name it already holds — so the re-issued create this path used to send
/// never reached the live actor at all, while the hash cache recorded it as applied. A
/// changed endpoint therefore took effect only when the connection was next built from
/// scratch (node restart, singleton failover, or a remove-then-add). Stopping and
/// re-creating the actor is not the fix either: subscriptions live inside the connection
/// actor and <c>InstanceActor</c> cancels its subscribe-retry timer on first success, so
/// nothing would re-subscribe. See
/// <c>docs/plans/2026-08-17-dcl-connection-arbitration-decision.md</c> §9.
/// </para>
/// </summary>
private void EnsureDclConnection(
string name,
@@ -2151,14 +2466,29 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
? null
: FlattenConnectionConfig(protocol, backupConfigurationJson);
var changed = _createdConnections.ContainsKey(name);
_createdConnections[name] = configHash;
if (changed)
{
_dclManager.Tell(new Commons.Messages.DataConnection.ReconfigureConnectionCommand(
name, protocol, primaryDetails, backupDetails, failoverRetryCount));
// Emitted once per changed configuration (the hash guard above suppresses repeats),
// so a healthy site stays silent.
_logger.LogInformation(
"Reconfigured DCL connection {Connection} (protocol={Protocol}) in place — the "
+ "live connection actor swaps its adapter onto the new configuration and "
+ "re-issues its subscriptions; no redeployment or restart is needed",
name, protocol);
return;
}
_dclManager.Tell(new Commons.Messages.DataConnection.CreateConnectionCommand(
name, protocol, primaryDetails, backupDetails, failoverRetryCount));
var changed = _createdConnections.ContainsKey(name);
_createdConnections[name] = configHash;
_logger.LogInformation(
"{Action} DCL connection {Connection} (protocol={Protocol})",
changed ? "Updated" : "Created", name, protocol);
"Created DCL connection {Connection} (protocol={Protocol})", name, protocol);
}
/// <summary>
@@ -2791,8 +3121,27 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// </summary>
private void HandleApplyArtifactDataConnectionsToDcl(ApplyArtifactDataConnectionsToDcl msg)
{
// Row 101: any snapshot read dispatched before this point carries data that predates
// these mutations — mark it stale so its overlay half is discarded when it lands.
_definitionsWriteGeneration++;
foreach (var dc in msg.DataConnections)
{
// Rows 89 + 90: the rows have just been committed to data_connection_definitions, so
// update the arbitration overlay in the same pass rather than waiting for the next
// sweep read. Skipping this would leave a window in which an instance start resolves
// the name to its flattened snapshot while the artifact path has already pushed the
// definition — the exact two-sources-one-hash-slot flip the overlay exists to prevent.
_centrallyRemovedConnections.Remove(dc.Name);
_connectionDefinitions[dc.Name] = new StoredDataConnectionDefinition
{
Name = dc.Name,
Protocol = dc.Protocol,
ConfigurationJson = dc.PrimaryConfigurationJson,
BackupConfigurationJson = dc.BackupConfigurationJson,
FailoverRetryCount = dc.FailoverRetryCount
};
EnsureDclConnection(
dc.Name,
dc.Protocol,
@@ -2801,20 +3150,12 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
dc.FailoverRetryCount);
}
// Reconciled removals: the definition row is already gone from SQLite; evict
// the hash cache (so a later re-add with the same name recreates cleanly) and
// stop the live DCL connection actor. A connection still referenced by a
// deployed instance's config no longer exists centrally either — its attributes
// go bad quality, the standard disconnected signal, until the instance is
// redeployed against current central config.
// Reconciled removals: the definition row is already gone from SQLite; drop the overlay
// row, tombstone the name for this incarnation (row 90 — otherwise the next instance
// start recreates it from that instance's flattened config), evict the hash cache so a
// later re-add recreates cleanly, and stop the live DCL connection actor.
foreach (var name in msg.RemovedConnectionNames)
{
_createdConnections.Remove(name);
_dclManager?.Tell(new DataConnectionLayer.Actors.RemoveConnectionCommand(name));
_logger.LogWarning(
"Artifact reconcile removed data connection '{Name}' (deleted centrally); " +
"its DCL connection actor has been stopped", name);
}
RemoveConnectionCentrally(name, "deleted centrally, reconciled by an artifact deploy");
}
/// <summary>
@@ -2861,7 +3202,30 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// </summary>
internal sealed record SiteNodeJoined(Address Address);
internal record StartupConfigsLoaded(List<DeployedInstance> Configs, string? Error);
/// <summary>
/// Carries the boot-time store read back onto the actor thread.
///
/// <para>
/// <paramref name="ConnectionDefinitions"/> (deferred register rows 89 + 90) is the
/// <c>data_connection_definitions</c> snapshot read in the SAME off-thread task, so the DCL
/// arbitration overlay is populated BEFORE the boot batch creates any Instance Actor — a
/// connection whose definition row is newer than the flattened snapshot inside the instance
/// config is therefore created from the definition on the very first push, not created stale
/// and corrected a sweep later. <see langword="null"/> means the definitions read FAILED and
/// the previous overlay must be kept; an EMPTY list means the table is genuinely empty, which
/// is the deploy-only site shape (see <see cref="ApplyConnectionDefinitionOverlay"/>).
/// </para>
/// </summary>
/// <param name="DefinitionsGenerationAtRead">
/// Row 101: <see cref="_definitionsWriteGeneration"/> captured on the actor thread when the
/// read was dispatched. A mismatch on landing means an artifact apply mutated the overlay
/// while this read was in flight — the snapshot's definitions half is stale and is skipped.
/// </param>
internal record StartupConfigsLoaded(
List<DeployedInstance> Configs,
string? Error,
IReadOnlyList<StoredDataConnectionDefinition>? ConnectionDefinitions = null,
long DefinitionsGenerationAtRead = 0);
/// <summary>
/// Internal singleton signal that re-attempts the startup deployed-config load
@@ -2946,8 +3310,22 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// <summary>
/// Carries one sweep's off-thread read of the durable deployed-config rows back onto the
/// actor thread, where every comparison and repair happens.
///
/// <para>
/// <paramref name="ConnectionDefinitions"/> (rows 89 + 90) rides the same read so the DCL
/// arbitration overlay is refreshed on every tick — that is what makes a definition row
/// delivered by LocalDb CDC converge without any new cross-node message. Same null-vs-empty
/// contract as <see cref="StartupConfigsLoaded"/>: null = read failed, keep the overlay.
/// </para>
/// </summary>
internal record InstanceSweepLoaded(List<DeployedInstance> Configs, string? Error);
/// <param name="DefinitionsGenerationAtRead">
/// Row 101: same stale-snapshot guard as <see cref="StartupConfigsLoaded"/> — see there.
/// </param>
internal record InstanceSweepLoaded(
List<DeployedInstance> Configs,
string? Error,
IReadOnlyList<StoredDataConnectionDefinition>? ConnectionDefinitions = null,
long DefinitionsGenerationAtRead = 0);
internal record DeployPersistenceResult(
string DeploymentId, string InstanceName, bool Success, string? Error,
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs
+6 -5
View File
@@ -13,12 +13,13 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment;
/// do not compile on the site (design spec: site-side compile failure rejects the
/// deployment; no partial state is applied).
///
/// Because it holds no actor state and only calls the (stateless)
/// <see cref="ScriptCompilationService"/>, it is pure and thread-safe; the Deployment
/// Manager nevertheless invokes it <em>synchronously on the singleton's actor thread</em>
/// as a pure prefix step, preserving mailbox-FIFO deploy ordering (see
/// Because it holds no actor state and only calls <see cref="ScriptCompilationService"/>
/// (whose only state is a thread-safe compile cache), it is pure and thread-safe; the
/// Deployment Manager nevertheless invokes it <em>synchronously on the singleton's actor
/// thread</em> as a pure prefix step, preserving mailbox-FIFO deploy ordering (see
/// <see cref="Actors.DeploymentManagerActor.HandleDeploy"/>). Compiles of unchanged
/// script bodies are deduped by the process-wide compile cache.
/// script bodies are deduped by that cache, which is shared across the whole site because
/// the compilation service is a DI singleton.
/// </summary>
public sealed class DeployCompileValidator
{
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptCompilationService.cs
+33 -7
View File
@@ -14,22 +14,48 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
/// execution-path compile of the script against <see cref="ScriptGlobals"/> /
/// <see cref="TriggerExpressionGlobals"/>.
///
/// Compile results — success AND failure — are memoised process-wide in
/// Compile results — success AND failure — are memoised in this service's
/// <see cref="SiteScriptCompileCache"/> (keyed on code hash + globals type), so an
/// unchanged script body compiles once per process lifetime and the deploy gate,
/// the Instance Actor's PreStart recompile, and SharedScriptLibrary all reuse one
/// compiled <c>Script&lt;object?&gt;</c>. Error text is name-free by construction, so a
/// cached failure renders correctly for every caller.
///
/// <para><b>The cache is owned per service instance, not static (register row 100).</b>
/// "Once per process" is a property of this service being registered as a DI
/// <b>singleton</b> (<see cref="ServiceCollectionExtensions.AddSiteRuntime"/>, pinned by
/// <c>SiteRuntimeServiceRegistrationTests</c>) — the whole site reaches the cache through
/// that one instance, which the Deployment Manager threads into its
/// <see cref="Deployment.DeployCompileValidator"/> and into every Instance Actor. Making the
/// cache an instance field rather than static state means a test (or a second host in one
/// process) that constructs its own service gets its own cache instead of mutating everyone
/// else's; that shared-static cross-talk is what row 100 closed.</para>
/// </summary>
public class ScriptCompilationService
{
private readonly ILogger<ScriptCompilationService> _logger;
private readonly SiteScriptCompileCache _compileCache;
/// <summary>Initializes a new instance of the ScriptCompilationService class.</summary>
/// <param name="logger">Logger instance.</param>
public ScriptCompilationService(ILogger<ScriptCompilationService> logger)
: this(logger, new SiteScriptCompileCache())
{
}
/// <summary>
/// Test seam: constructs the service over a caller-supplied compile cache, so a test can
/// observe <see cref="SiteScriptCompileCache.Hits"/> / <see cref="SiteScriptCompileCache.Count"/>
/// for exactly the compiles it caused. Production always uses the public constructor.
/// </summary>
/// <param name="logger">Logger instance.</param>
/// <param name="compileCache">The compile cache this service memoises into.</param>
internal ScriptCompilationService(
ILogger<ScriptCompilationService> logger,
SiteScriptCompileCache compileCache)
{
_logger = logger;
_compileCache = compileCache;
}
/// <summary>
@@ -136,16 +162,16 @@ public class ScriptCompilationService
/// <summary>
/// Shared compilation funnel for every site-side compile (the deploy gate, the
/// Instance Actor's PreStart recompile, and SharedScriptLibrary). Results — success
/// AND failure — are memoised process-wide in <see cref="SiteScriptCompileCache"/>,
/// AND failure — are memoised in this service's <see cref="SiteScriptCompileCache"/>,
/// keyed on code hash + globals type: an unchanged script body compiles once per
/// process lifetime and the compiled <c>Script&lt;object?&gt;</c> is shared across all
/// call sites (N4). Cached error text is name-free by construction (Roslyn diagnostics /
/// the compile-exception message never embed the caller's script name), so a cached
/// failure renders correctly for every caller.
/// process lifetime (this service is a DI singleton) and the compiled
/// <c>Script&lt;object?&gt;</c> is shared across all call sites (N4). Cached error text is
/// name-free by construction (Roslyn diagnostics / the compile-exception message never
/// embed the caller's script name), so a cached failure renders correctly for every caller.
/// </summary>
private ScriptCompilationResult CompileCore(string name, string code, Type globalsType)
{
var result = SiteScriptCompileCache.GetOrAdd(code, globalsType,
var result = _compileCache.GetOrAdd(code, globalsType,
() => CompileUncached(name, code, globalsType));
if (!result.IsSuccess)
_logger.LogDebug(
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/SiteScriptCompileCache.cs
+34 -29
View File
@@ -5,13 +5,25 @@ using System.Text;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
/// <summary>
/// Process-wide cache of site-side script <see cref="ScriptCompilationResult"/>s, keyed on
/// Cache of site-side script <see cref="ScriptCompilationResult"/>s, keyed on
/// the globals type plus a SHA-256 hash of the script code. Unlike the TemplateEngine's
/// <c>ScriptCompileVerdictCache</c> (which caches central-validation <em>verdicts</em>), this
/// caches the full compiled result — the immutable Roslyn <c>Script&lt;object?&gt;</c> itself —
/// so the synchronous deploy gate and the <c>InstanceActor.PreStart</c> recompile share one
/// Roslyn compile per script body per process lifetime (N4).
///
/// <para><b>Owned by <see cref="ScriptCompilationService"/>, not static (register row 100).</b>
/// This used to be a <c>static</c> class, which made its <c>Hits</c>/<c>Count</c> counters and
/// its entry table one process-global blob that every test class in the assembly shared. The
/// process-wide N4 sharing property is unchanged in production — it now comes from
/// <c>ScriptCompilationService</c> being a DI <b>singleton</b> (<c>ServiceCollectionExtensions</c>,
/// pinned by <c>SiteRuntimeServiceRegistrationTests</c>), so exactly one cache exists per host and
/// the deploy gate, the Instance Actor's PreStart recompile and SharedScriptLibrary all reach it
/// through that one service. What changed is that a test constructing its own
/// <c>ScriptCompilationService</c> now gets its own cache instead of writing into everyone else's.
/// The old <c>Clear()</c> escape hatch is deliberately GONE: it had no production caller — it
/// existed only so tests could reset the global — and a fresh instance is the honest reset.</para>
///
/// <para>Soundness rests on three invariants:</para>
/// <list type="number">
/// <item>A compile result is a pure function of (code, globals type, compile-time-static
@@ -30,7 +42,7 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
/// </para>
///
/// <para><b>WP3.1 — approximate LRU replaced the overflow cliff.</b> The cache used to
/// <c>Clear()</c> wholesale on overflow. Instance scripts AND trigger expressions share this
/// clear itself wholesale on overflow. Instance scripts AND trigger expressions share this
/// cache, so on a site large enough to cross 1024 entries every overflow discarded up to 1023
/// live compiled scripts and the next deploy or Instance-Actor start paid a full recompile
/// storm — on actor threads. Now an insert at the bound evicts only the oldest
@@ -42,7 +54,7 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
/// only lock is the (rare) eviction sweep, double-checked so concurrent inserts do not
/// stampede it.</para>
/// </summary>
internal static class SiteScriptCompileCache
internal sealed class SiteScriptCompileCache
{
/// <summary>Upper bound on cached entries; crossing it evicts the oldest batch.</summary>
internal const int MaxEntries = 1024;
@@ -54,18 +66,18 @@ internal static class SiteScriptCompileCache
/// </summary>
private const int EvictionBatchDivisor = 8;
private static readonly ConcurrentDictionary<string, CacheEntry> Cache = new();
private static readonly object EvictionLock = new();
private static long _hits;
private readonly ConcurrentDictionary<string, CacheEntry> _cache = new();
private readonly object _evictionLock = new();
private long _hits;
/// <summary>Monotonic access sequence; the recency stamp written onto entries.</summary>
private static long _accessSequence;
private long _accessSequence;
/// <summary>Number of cache hits observed since the last <see cref="Clear"/>.</summary>
public static long Hits => Interlocked.Read(ref _hits);
/// <summary>Number of cache hits observed by this cache instance.</summary>
public long Hits => Interlocked.Read(ref _hits);
/// <summary>Current number of cached compile results.</summary>
public static int Count => Cache.Count;
public int Count => _cache.Count;
/// <summary>
/// Returns the cached compile result for <paramref name="code"/> against
@@ -77,11 +89,11 @@ internal static class SiteScriptCompileCache
/// <param name="globalsType">The Roslyn globals surface the script compiles against; part of the key so identical source under different globals stays distinct.</param>
/// <param name="factory">Computes the compile result on a cache miss.</param>
/// <returns>The cached or newly computed compile result.</returns>
public static ScriptCompilationResult GetOrAdd(string code, Type globalsType, Func<ScriptCompilationResult> factory)
public ScriptCompilationResult GetOrAdd(string code, Type globalsType, Func<ScriptCompilationResult> factory)
{
var key = globalsType.FullName + ":" + Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(code)));
if (Cache.TryGetValue(key, out var entry))
if (_cache.TryGetValue(key, out var entry))
{
Interlocked.Increment(ref _hits);
Touch(entry);
@@ -90,35 +102,28 @@ internal static class SiteScriptCompileCache
var result = factory();
if (Cache.Count >= MaxEntries)
if (_cache.Count >= MaxEntries)
EvictOldestBatch();
Cache[key] = new CacheEntry(result) { LastAccess = NextStamp() };
_cache[key] = new CacheEntry(result) { LastAccess = NextStamp() };
return result;
}
/// <summary>Clears all cached compile results and resets the hit counter.</summary>
public static void Clear()
{
Cache.Clear();
Interlocked.Exchange(ref _hits, 0);
}
/// <summary>Refreshes an entry's recency stamp. Lock-free — a lost race only costs accuracy, never correctness.</summary>
private static void Touch(CacheEntry entry) => Volatile.Write(ref entry.LastAccess, NextStamp());
private void Touch(CacheEntry entry) => Volatile.Write(ref entry.LastAccess, NextStamp());
private static long NextStamp() => Interlocked.Increment(ref _accessSequence);
private long NextStamp() => Interlocked.Increment(ref _accessSequence);
/// <summary>
/// Evicts the oldest <see cref="MaxEntries"/> / <see cref="EvictionBatchDivisor"/> entries
/// by recency stamp. Double-checked under <see cref="EvictionLock"/> so several concurrent
/// by recency stamp. Double-checked under <see cref="_evictionLock"/> so several concurrent
/// inserts crossing the bound together perform ONE sweep rather than one each.
/// </summary>
private static void EvictOldestBatch()
private void EvictOldestBatch()
{
lock (EvictionLock)
lock (_evictionLock)
{
if (Cache.Count < MaxEntries) return; // another thread already swept
if (_cache.Count < MaxEntries) return; // another thread already swept
var batch = Math.Max(1, MaxEntries / EvictionBatchDivisor);
@@ -128,14 +133,14 @@ internal static class SiteScriptCompileCache
// copies, and throws ArgumentException when a concurrent insert lands between
// the two. The eviction lock only excludes other EVICTORS — inserts run
// lock-free by design — so the snapshot must be the thread-safe one.
var victims = Cache.ToArray()
var victims = _cache.ToArray()
.OrderBy(kvp => Volatile.Read(ref kvp.Value.LastAccess))
.Take(batch)
.Select(kvp => kvp.Key)
.ToArray();
foreach (var victim in victims)
Cache.TryRemove(victim, out _);
_cache.TryRemove(victim, out _);
}
}
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs
+8 -1
View File
@@ -32,7 +32,14 @@ public static class ServiceCollectionExtensions
services.AddHostedService<SiteStorageInitializer>();
// Script compilation service
// Script compilation service.
//
// SINGLETON IS LOAD-BEARING (register row 100): this service owns the
// SiteScriptCompileCache, so "one Roslyn compile per script body per process"
// (N4) is exactly the statement that there is one of these per host. Downgrading
// it to Scoped/Transient would silently give every consumer its own cache and
// reintroduce the recompile storms WP3.1 removed — no test would fail on
// behaviour, only the registration pin in SiteRuntimeServiceRegistrationTests.
services.AddSingleton<ScriptCompilationService>();
// Shared script library
tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/AuditLogPurgeActorTests.cs
+71 -11
View File
@@ -138,6 +138,38 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
NullLogger<AuditLogPurgeActor>.Instance)));
}
/// <summary>
/// Stops a purge actor and blocks until it is actually dead.
/// </summary>
/// <remarks>
/// <para>
/// Deferred-work register row 95, the same leak shape row 82 fixed in
/// <c>PartitionPurgeTests</c> / <c>NotifyDispatcherAuditTrailTests</c>. Every actor this
/// class creates ticks every <c>100 ms</c>
/// (<see cref="AuditLogPurgeOptions.IntervalOverride"/>) and was only ever stopped
/// implicitly by <see cref="TestKit"/>'s <c>Dispose</c> → <c>Shutdown</c> — which is
/// bounded by a timeout and does NOT fail when that timeout expires. So on a loaded
/// machine an actor kept ticking while LATER test classes ran: the stub-repo tests burn a
/// DI scope and a thread every 100 ms for the rest of the assembly run, and
/// <c>EndToEnd_RealPartition_…</c>'s actor keeps switching partitions out of the shared
/// <see cref="MsSqlMigrationFixture"/> database that other classes in this assembly seed
/// against.
/// </para>
/// <para>
/// Stopping the actor inside the test that created it removes the dependency on
/// ActorSystem-shutdown timing entirely.
/// </para>
/// </remarks>
private void StopPurgeActor(IActorRef actor)
{
Watch(actor);
Sys.Stop(actor);
// A stop is processed after the in-flight tick completes, so this also guarantees no
// partition switch is still running against the fixture DB.
ExpectTerminated(actor, TimeSpan.FromSeconds(30));
Unwatch(actor);
}
private static AuditLogPurgeOptions FastTickOptions(TimeSpan? interval = null) => new()
{
IntervalHours = 24,
@@ -163,7 +195,7 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
public void Tick_Fires_OnDailyInterval()
{
var repo = new RecordingRepo();
CreateActor(repo, FastTickOptions());
var actor = CreateActor(repo, FastTickOptions());
// The first scheduled tick fires after the configured interval. We
// assert the visible side effect (the enumerator was called) rather
@@ -173,6 +205,8 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
$"expected >= 1 enumerator call, got {repo.ThresholdQueries.Count}"),
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
StopPurgeActor(actor);
}
// ---------------------------------------------------------------------
@@ -192,7 +226,7 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
RowsPerBoundary = _ => 42L,
};
CreateActor(repo, FastTickOptions());
var actor = CreateActor(repo, FastTickOptions());
AwaitAssert(
() =>
@@ -202,6 +236,8 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
},
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
StopPurgeActor(actor);
}
// ---------------------------------------------------------------------
@@ -217,7 +253,7 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
// and asserting the actor switched nothing despite the tick firing.
var repo = new RecordingRepo { Boundaries = new List<DateTime>() };
CreateActor(repo, FastTickOptions());
var actor = CreateActor(repo, FastTickOptions());
// Wait for at least one tick (visible via the enumerator call) then
// assert no switch happened.
@@ -226,6 +262,10 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
// Stop before the negative assertion: a still-ticking actor is exactly what would
// make it read a moving target (register row 95).
StopPurgeActor(actor);
Assert.Empty(repo.SwitchedBoundaries);
}
@@ -244,9 +284,10 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
};
var probe = SubscribePurged();
CreateActor(repo, FastTickOptions());
var actor = CreateActor(repo, FastTickOptions());
var msg = probe.ExpectMsg<AuditLogPurgedEvent>(TimeSpan.FromSeconds(5));
StopPurgeActor(actor);
Assert.Equal(boundary, msg.MonthBoundary);
Assert.Equal(1234L, msg.RowsDeleted);
Assert.True(msg.DurationMs >= 0,
@@ -269,7 +310,7 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
BoundaryException = new InvalidOperationException("simulated switch failure for poison boundary"),
};
CreateActor(repo, FastTickOptions());
var actor = CreateActor(repo, FastTickOptions());
AwaitAssert(
() =>
@@ -281,6 +322,8 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
},
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
StopPurgeActor(actor);
}
// ---------------------------------------------------------------------
@@ -349,7 +392,7 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
};
var probe = SubscribePurged();
Sys.ActorOf(Props.Create(() => new AuditLogPurgeActor(
var actor = Sys.ActorOf(Props.Create(() => new AuditLogPurgeActor(
sp,
Options.Create(purgeOptions),
Options.Create(auditOptions),
@@ -363,6 +406,11 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
isMessage: m => m.MonthBoundary == janBoundary,
max: TimeSpan.FromSeconds(30));
// This is the one actor in the class that mutates the SHARED fixture database, so it
// must not outlive the test and start switching partitions out from under the next
// class's seed rows (register row 95).
StopPurgeActor(actor);
Assert.True(matched.RowsDeleted >= 1,
$"Expected RowsDeleted >= 1 for the Jan-2026 partition; got {matched.RowsDeleted}.");
@@ -394,7 +442,7 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
// or some other accidental default. We use a non-default retention
// (30 days) so the assertion isn't satisfied by the 365 default.
var repo = new RecordingRepo();
CreateActor(
var actor = CreateActor(
repo,
FastTickOptions(),
auditOptions: new AuditLogOptions { RetentionDays = 30 });
@@ -404,6 +452,8 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
StopPurgeActor(actor);
var threshold = repo.ThresholdQueries[0];
var expected = DateTime.UtcNow - TimeSpan.FromDays(30);
// 1-minute slack covers test-thread scheduling jitter between the
@@ -437,7 +487,7 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
var purgeOptionsWrapped = Options.Create(purgeOptions);
var sp = BuildScopedProvider(repo);
Sys.ActorOf(Props.Create(() => new AuditLogPurgeActor(
var actor = Sys.ActorOf(Props.Create(() => new AuditLogPurgeActor(
sp,
purgeOptionsWrapped,
auditOptions,
@@ -448,6 +498,8 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
StopPurgeActor(actor);
var purge = repo.ChannelPurges.First(p => p.Channel == "ApiOutbound");
Assert.Equal(1234, purge.BatchSize);
@@ -483,7 +535,7 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
var purgeOptions = Options.Create(FastTickOptions());
var sp = BuildScopedProvider(repo);
Sys.ActorOf(Props.Create(() => new AuditLogPurgeActor(
var actor = Sys.ActorOf(Props.Create(() => new AuditLogPurgeActor(
sp,
purgeOptions,
auditOptions,
@@ -496,6 +548,10 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
// Stop before the negative assertion: a still-ticking actor is exactly what would
// make it read a moving target (register row 95).
StopPurgeActor(actor);
Assert.Empty(repo.ChannelPurges);
}
@@ -527,7 +583,7 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
});
var sp = BuildScopedProvider(repo);
Sys.ActorOf(Props.Create(() => new AuditLogPurgeActor(
var actor = Sys.ActorOf(Props.Create(() => new AuditLogPurgeActor(
sp,
Options.Create(purgeOptions),
auditOptions,
@@ -542,6 +598,8 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
},
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
StopPurgeActor(actor);
}
// ---------------------------------------------------------------------
@@ -572,7 +630,7 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
var probe = CreateTestProbe();
Sys.EventStream.Subscribe(probe.Ref, typeof(AuditLogPurgeFailedEvent));
Sys.ActorOf(Props.Create(() => new AuditLogPurgeActor(
var actor = Sys.ActorOf(Props.Create(() => new AuditLogPurgeActor(
sp,
Options.Create(FastTickOptions()),
Options.Create(new AuditLogOptions()),
@@ -588,5 +646,7 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
$"expected PurgeFailures >= 1, got {snapshot.PurgeFailures}"),
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
StopPurgeActor(actor);
}
}
tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/SiteAuditReconciliationActorTests.cs
+68 -11
View File
@@ -252,6 +252,37 @@ public class SiteAuditReconciliationActorTests : TestKit, IClassFixture<MsSqlMig
NullLogger<SiteAuditReconciliationActor>.Instance)));
}
/// <summary>
/// Stops a reconciliation actor and blocks until it is actually dead.
/// </summary>
/// <remarks>
/// <para>
/// Deferred-work register row 95, the same leak shape row 82 fixed in
/// <c>PartitionPurgeTests</c> / <c>NotifyDispatcherAuditTrailTests</c>. Every actor this
/// class creates ticks on a ~0 s repeat interval (<see cref="FastTickOptions"/>) and was
/// only ever stopped implicitly by <see cref="TestKit"/>'s <c>Dispose</c> →
/// <c>Shutdown</c>, which is bounded by a timeout and does NOT fail when it expires. A
/// survivor keeps pulling and INSERTING audit rows: in
/// <c>Tick_Duplicates_NotDoubleInserted</c> it holds a real
/// <see cref="AuditLogRepository"/> over the shared <see cref="MsSqlMigrationFixture"/>
/// database (and over a <c>DbContext</c> the test is about to dispose), so it writes into
/// the DB LATER test classes read from.
/// </para>
/// <para>
/// Stopping the actor inside the test that created it removes the dependency on
/// ActorSystem-shutdown timing entirely.
/// </para>
/// </remarks>
private void StopReconciliationActor(IActorRef actor)
{
Watch(actor);
Sys.Stop(actor);
// A stop is processed after the in-flight tick completes, so this also guarantees no
// pull or insert is still running against the fixture DB.
ExpectTerminated(actor, TimeSpan.FromSeconds(30));
Unwatch(actor);
}
/// <summary>
/// Subscribes to the EventStream and collects every
/// <see cref="SiteAuditTelemetryStalledChanged"/> publication into a list
@@ -278,7 +309,7 @@ public class SiteAuditReconciliationActorTests : TestKit, IClassFixture<MsSqlMig
var repo = new RecordingRepo();
var opts = FastTickOptions();
CreateActor(sites, client, repo, opts);
var actor = CreateActor(sites, client, repo, opts);
// The first scheduled tick fires after `ReconciliationIntervalSeconds`,
// which is 0 for the test — Akka's scheduler still respects the
@@ -289,6 +320,8 @@ public class SiteAuditReconciliationActorTests : TestKit, IClassFixture<MsSqlMig
() => Assert.True(client.Calls.Count >= 1, $"expected >= 1 pull call, got {client.Calls.Count}"),
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
StopReconciliationActor(actor);
}
// ---------------------------------------------------------------------
@@ -304,7 +337,7 @@ public class SiteAuditReconciliationActorTests : TestKit, IClassFixture<MsSqlMig
var client = new ScriptedPullClient();
var repo = new RecordingRepo();
CreateActor(sites, client, repo, FastTickOptions());
var actor = CreateActor(sites, client, repo, FastTickOptions());
AwaitAssert(() =>
{
@@ -313,6 +346,8 @@ public class SiteAuditReconciliationActorTests : TestKit, IClassFixture<MsSqlMig
},
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
StopReconciliationActor(actor);
}
// ---------------------------------------------------------------------
@@ -329,11 +364,14 @@ public class SiteAuditReconciliationActorTests : TestKit, IClassFixture<MsSqlMig
new PullAuditEventsResponse(new[] { e1, e2 }, MoreAvailable: false));
var repo = new RecordingRepo();
CreateActor(sites, client, repo, FastTickOptions());
var actor = CreateActor(sites, client, repo, FastTickOptions());
AwaitAssert(() => Assert.Equal(2, repo.InsertCallCount),
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
StopReconciliationActor(actor);
Assert.Contains(repo.Inserted, e => e.EventId == e1.EventId);
Assert.Contains(repo.Inserted, e => e.EventId == e2.EventId);
}
@@ -371,13 +409,18 @@ public class SiteAuditReconciliationActorTests : TestKit, IClassFixture<MsSqlMig
await using var context = CreateContext();
var repo = new AuditLogRepository(context);
CreateActor(sites, client, repo, FastTickOptions());
var actor = CreateActor(sites, client, repo, FastTickOptions());
// Wait for the actor to ingest both rows.
await Task.Delay(TimeSpan.FromSeconds(1));
AwaitAssert(() => Assert.True(client.Calls.Count >= 1),
duration: TimeSpan.FromSeconds(3));
// Stop before reading: this actor holds a real repository over the SHARED fixture
// database (and over the DbContext this test is about to dispose), so it must not
// outlive the test (register row 95).
StopReconciliationActor(actor);
// Even though the pull returned 2 events, only 1 fresh row should
// exist in MSSQL alongside the pre-existing one — InsertIfNotExistsAsync
// is first-write-wins on EventId.
@@ -413,7 +456,7 @@ public class SiteAuditReconciliationActorTests : TestKit, IClassFixture<MsSqlMig
new PullAuditEventsResponse(new[] { e1, e2, e3 }, MoreAvailable: false));
var repo = new RecordingRepo();
CreateActor(sites, client, repo, FastTickOptions());
var actor = CreateActor(sites, client, repo, FastTickOptions());
// Wait until we have at least two pulls — the second one must use t3
// as its `since` argument because that was the max OccurredAtUtc in
@@ -423,6 +466,10 @@ public class SiteAuditReconciliationActorTests : TestKit, IClassFixture<MsSqlMig
duration: TimeSpan.FromSeconds(5),
interval: TimeSpan.FromMilliseconds(50));
// Stop before indexing into client.Calls: the list must not grow mid-assertion
// (register row 95).
StopReconciliationActor(actor);
Assert.Equal(DateTime.MinValue, client.Calls[0].SinceUtc);
Assert.Equal(t3, client.Calls[1].SinceUtc);
@@ -451,13 +498,15 @@ public class SiteAuditReconciliationActorTests : TestKit, IClassFixture<MsSqlMig
new PullAuditEventsResponse(new[] { low, high, mid }, MoreAvailable: true));
var repo = new RecordingRepo();
CreateActor(sites, client, repo, FastTickOptions());
var actor = CreateActor(sites, client, repo, FastTickOptions());
AwaitAssert(() => Assert.True(client.Calls.Count >= 2,
$"need at least 2 pulls, got {client.Calls.Count}"),
duration: TimeSpan.FromSeconds(5),
interval: TimeSpan.FromMilliseconds(50));
StopReconciliationActor(actor);
Assert.Equal(t, client.Calls[1].SinceUtc);
Assert.Equal(high.EventId.ToString(), client.Calls[1].AfterId);
}
@@ -475,13 +524,15 @@ public class SiteAuditReconciliationActorTests : TestKit, IClassFixture<MsSqlMig
new PullAuditEventsResponse(new[] { evt }, MoreAvailable: false));
var repo = new AlwaysThrowingRepo();
CreateActor(sites, client, repo, FastTickOptions());
var actor = CreateActor(sites, client, repo, FastTickOptions());
AwaitAssert(() => Assert.True(client.Calls.Count >= 2,
$"need at least 2 pulls, got {client.Calls.Count}"),
duration: TimeSpan.FromSeconds(5),
interval: TimeSpan.FromMilliseconds(50));
StopReconciliationActor(actor);
Assert.Equal(DateTime.MinValue, client.Calls[1].SinceUtc);
Assert.Null(client.Calls[1].AfterId);
}
@@ -543,7 +594,7 @@ public class SiteAuditReconciliationActorTests : TestKit, IClassFixture<MsSqlMig
new PullAuditEventsResponse(new[] { bEvent }, MoreAvailable: false));
var repo = new RecordingRepo();
CreateActor(sites, client, repo, FastTickOptions());
var actor = CreateActor(sites, client, repo, FastTickOptions());
AwaitAssert(() =>
{
@@ -552,6 +603,8 @@ public class SiteAuditReconciliationActorTests : TestKit, IClassFixture<MsSqlMig
},
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
StopReconciliationActor(actor);
}
// ---------------------------------------------------------------------
@@ -574,12 +627,13 @@ public class SiteAuditReconciliationActorTests : TestKit, IClassFixture<MsSqlMig
var repo = new RecordingRepo();
var (probe, _) = SubscribeStalled();
CreateActor(sites, client, repo, FastTickOptions(stalledAfter: 2));
var actor = CreateActor(sites, client, repo, FastTickOptions(stalledAfter: 2));
// Expect Stalled=true after the second non-draining tick. The probe
// waits with its own timeout (a few seconds gives the 0 s repeat
// interval ample slack).
var msg = probe.ExpectMsg<SiteAuditTelemetryStalledChanged>(TimeSpan.FromSeconds(5));
StopReconciliationActor(actor);
Assert.Equal("siteA", msg.SiteId);
Assert.True(msg.Stalled);
}
@@ -606,7 +660,7 @@ public class SiteAuditReconciliationActorTests : TestKit, IClassFixture<MsSqlMig
var repo = new RecordingRepo();
var (probe, _) = SubscribeStalled();
CreateActor(sites, client, repo, FastTickOptions(stalledAfter: 2));
var actor = CreateActor(sites, client, repo, FastTickOptions(stalledAfter: 2));
// First publication is the stalled=true transition; second is the
// back-to-draining flip. The actor publishes ONLY on transitions so we
@@ -615,6 +669,7 @@ public class SiteAuditReconciliationActorTests : TestKit, IClassFixture<MsSqlMig
Assert.True(first.Stalled);
var second = probe.ExpectMsg<SiteAuditTelemetryStalledChanged>(TimeSpan.FromSeconds(5));
StopReconciliationActor(actor);
Assert.False(second.Stalled);
Assert.Equal("siteA", second.SiteId);
}
@@ -643,7 +698,7 @@ public class SiteAuditReconciliationActorTests : TestKit, IClassFixture<MsSqlMig
var repo = new PoisonInsertRepo(
poison.EventId, new InvalidOperationException("poison row will never insert"));
CreateActor(sites, client, repo, FastTickOptions());
var actor = CreateActor(sites, client, repo, FastTickOptions());
// After the 5th failed attempt the actor abandons the row and writes ONE
// synthetic ReconciliationAbandoned row (fresh EventId) whose details carry
@@ -669,5 +724,7 @@ public class SiteAuditReconciliationActorTests : TestKit, IClassFixture<MsSqlMig
() => Assert.Contains(client.Calls, c => c.SinceUtc == poisonTime),
duration: TimeSpan.FromSeconds(5),
interval: TimeSpan.FromMilliseconds(100));
StopReconciliationActor(actor);
}
}
tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/OutageReconciliationTests.cs
+45 -7
View File
@@ -165,6 +165,37 @@ public class OutageReconciliationTests : TestKit, IClassFixture<MsSqlMigrationFi
return (sp, ingest);
}
/// <summary>
/// Stops an actor and blocks until it is actually dead.
/// </summary>
/// <remarks>
/// <para>
/// Deferred-work register row 95, the same leak shape row 82 fixed in
/// <c>PartitionPurgeTests</c> / <c>NotifyDispatcherAuditTrailTests</c>. Both tests here
/// already called <c>Sys.Stop</c>, but that is fire-and-forget: it returns before the
/// actor has processed the stop, so an in-flight 100 ms tick could still be pulling and
/// INSERTING into the shared <see cref="MsSqlMigrationFixture"/> database — and into the
/// <see cref="IServiceProvider"/> the enclosing <c>await using</c> is about to dispose.
/// Worse, the call sat at the very END of the block, so any earlier assertion failure
/// skipped it entirely and left the actor ticking for the rest of the assembly run,
/// stopped only by <see cref="TestKit"/>'s <c>Dispose</c> → <c>Shutdown</c>, which is
/// timeout-bounded and does not fail when it expires.
/// </para>
/// <para>
/// Watch/Stop/ExpectTerminated makes the stop synchronous, and each call site now sits
/// immediately after the last step that needs the actor alive rather than at the end.
/// </para>
/// </remarks>
private void StopActor(IActorRef actor)
{
Watch(actor);
Sys.Stop(actor);
// A stop is processed after the in-flight tick completes, so this also guarantees no
// pull or insert is still running against the fixture DB.
ExpectTerminated(actor, TimeSpan.FromSeconds(30));
Unwatch(actor);
}
private static SiteAuditReconciliationOptions FastTickOptions(int batchSize = 256) => new()
{
ReconciliationIntervalSeconds = 300,
@@ -265,6 +296,12 @@ public class OutageReconciliationTests : TestKit, IClassFixture<MsSqlMigrationFi
duration: TimeSpan.FromSeconds(10),
interval: TimeSpan.FromMilliseconds(100));
// Both actors have done everything the test needs; stop them HERE, before the
// read-back, so nothing is still writing to the shared fixture DB while step 5
// counts rows and so a step-5 failure cannot leak them (register row 95).
StopActor(reconciliationActor);
StopActor(ingest);
// Step 5: assert no duplicates by EventId — central must have
// exactly the 200 rows we wrote at the site (one row per EventId).
await using var verify = CreateContext();
@@ -277,10 +314,6 @@ public class OutageReconciliationTests : TestKit, IClassFixture<MsSqlMigrationFi
// And every EventId we wrote at the site is present centrally.
Assert.True(written.All(w => centralIds.Contains(w.EventId)),
"every site-written EventId should be present centrally.");
// Tear the actor down before disposing the harness; the actor's
// PostStop cancels its scheduled timer.
Sys.Stop(reconciliationActor);
}
}
@@ -303,7 +336,7 @@ public class OutageReconciliationTests : TestKit, IClassFixture<MsSqlMigrationFi
await sqliteWriter.WriteAsync(NewEvent(siteId, baseOccurred.AddMilliseconds(i)));
}
var (sp, _) = BuildCentralPipeline();
var (sp, ingest) = BuildCentralPipeline();
await using (sp as IAsyncDisposable ?? throw new InvalidOperationException())
{
var pullClient = new QueueBackedPullClient(sqliteWriter);
@@ -338,6 +371,13 @@ public class OutageReconciliationTests : TestKit, IClassFixture<MsSqlMigrationFi
Assert.True(pullClient.CallCount > callsAfterDrain,
$"expected additional pull calls after drain to validate idempotency, got {pullClient.CallCount} after {callsAfterDrain}");
// Both actors have done everything the test needs; stop them HERE, before the
// read-back, so nothing is still writing to the shared fixture DB while the
// duplicate check counts rows and so a failure below cannot leak them
// (register row 95).
StopActor(reconciliationActor);
StopActor(ingest);
// Central count must still be exactly totalEvents — no duplicates
// even though the cursor + read-Reconciled-too semantics could
// theoretically re-fetch on the second cycle.
@@ -347,8 +387,6 @@ public class OutageReconciliationTests : TestKit, IClassFixture<MsSqlMigrationFi
.ToListAsync();
Assert.Equal(totalEvents, rows.Count);
Assert.Equal(totalEvents, rows.Select(r => r.EventId).Distinct().Count());
Sys.Stop(reconciliationActor);
}
}
}
tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests.csproj
+16
View File
@@ -46,6 +46,22 @@
<Using Include="Xunit" />
</ItemGroup>
<ItemGroup>
<!--
Deferred-work register row 96 — per-suite in-process parallelism cap.
Resource-heavy suite: MSSQL-backed classes take an MsSqlMigrationFixture
per test CLASS (CREATE DATABASE + the full migration chain on the ONE
shared dev SQL Server) and most classes stand up an Akka TestKit
ActorSystem with fast-ticking actors. Uncapped, xunit runs
Environment.ProcessorCount collections at once. The cap is threads only —
collections stay parallel — and 4 matches the default slot count of the
cross-process sp_getapplock migrate semaphore (row 82), so the in-process
and cross-process limits agree. Policy: docs/test_infra/test_infra.md.
The json must be copied to the output directory to take effect.
-->
<Content Include="xunit.runner.json" CopyToOutputDirectory="PreserveNewest" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.AuditLog/ZB.MOM.WW.ScadaBridge.AuditLog.csproj" />
<!--
tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/xunit.runner.json
+4
View File
@@ -0,0 +1,4 @@
{
"$schema": "https://xunit.net/schema/current/xunit.runner.schema.json",
"maxParallelThreads": 4
}
tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/CommandTreeTests.cs
+81 -2
View File
@@ -231,8 +231,12 @@ public class CommandTreeTests
[Theory]
[InlineData("sideways")]
// AcceptOnlyFromAmong is case-SENSITIVE (same contract as the audit
// --channel / --kind / --status options), so the literals are lowercase only.
// AcceptOnlyFromAmong is case-SENSITIVE, so the literals are lowercase only.
// The audit --channel/--kind/--status options USED to share this contract; they
// dropped AcceptOnlyFromAmong in register row 99 precisely because the
// case-sensitivity rejected values the server accepts. --direction keeps it: it is
// a pure client-side literal that never reaches a server-side parse, so there is no
// authoritative check behind it to defer to.
[InlineData("Up")]
public void TemplateFolderReorder_RejectsAnythingElse(string direction)
{
@@ -255,6 +259,81 @@ public class CommandTreeTests
Assert.NotEmpty(parse.Errors);
}
// ========================================================================
// Register row 99: the audit filter options must NOT be case-sensitively
// gated client-side.
//
// `audit query --status delivered` used to fail at the CLI parse layer while
// the server would have accepted it — AcceptOnlyFromAmong matches ordinally
// against the capitalised literals. The valid set now lives in the help text
// and the server's strict, case-insensitive check is authoritative (the same
// treatment row 88 gave `deploy status`). These tests fail before that change.
// ========================================================================
public static TheoryData<string, string, string> AuditFilterValues() => new()
{
// subcommand, option, value — lowercase, uppercase and mixed case of a real member.
{ "query", "--status", "delivered" },
{ "query", "--status", "DELIVERED" },
{ "query", "--channel", "apioutbound" },
{ "query", "--kind", "apicall" },
{ "export", "--status", "parked" },
{ "export", "--channel", "notification" },
{ "export", "--kind", "dbwrite" },
};
[Theory]
[MemberData(nameof(AuditFilterValues))]
public void AuditFilterOptions_AcceptValuesInAnyCase(string subcommand, string option, string value)
{
var audit = AuditCommands.Build(Url, Format, Username, Password);
var cmd = audit.Subcommands.Single(c => c.Name == subcommand);
// `export` has three required options; supply them so the only thing under
// test is the filter value's own acceptance.
var args = subcommand == "export"
? new[] { "--since", "1h", "--until", "now", "--format", "csv", "--output", "out.csv", option, value }
: new[] { option, value };
var parse = cmd.Parse(args);
Assert.Empty(parse.Errors);
}
[Theory]
[InlineData("csv")]
[InlineData("CSV")]
[InlineData("jsonl")]
public void AuditExport_FormatOption_AcceptsValuesInAnyCase(string format)
{
// The server lowercases --format before matching and returns a 400 naming
// csv/jsonl/parquet for anything else, so the client-side literal gate was
// pure friction (register row 99).
var audit = AuditCommands.Build(Url, Format, Username, Password);
var export = audit.Subcommands.Single(c => c.Name == "export");
var parse = export.Parse(new[]
{
"--since", "1h", "--until", "now", "--format", format, "--output", "out.csv",
});
Assert.Empty(parse.Errors);
}
[Fact]
public void AuditQuery_UnrecognisedFilterValue_IsNotRejectedClientSide()
{
// Deliberate: the CLI no longer second-guesses the filter vocabulary. A bad
// value must reach the server, which answers 400 naming the valid set —
// one authoritative check instead of two that can drift apart.
var audit = AuditCommands.Build(Url, Format, Username, Password);
var query = audit.Subcommands.Single(c => c.Name == "query");
var parse = query.Parse(new[] { "--status", "NotAStatus" });
Assert.Empty(parse.Errors);
}
[Fact]
public void SiteArea_HasMoveVerb()
{
tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/Commands/AuditExportCommandTests.cs
+30 -9
View File
@@ -40,15 +40,34 @@ public class AuditExportCommandTests
}
[Fact]
public void Export_InvalidFormat_Rejected()
public void Export_InvalidFormat_IsNotRejectedAtTheParseLayer()
{
// Register row 99: the client-side AcceptOnlyFromAmong gate is gone. It
// matched case-SENSITIVELY, so it rejected `--format CSV` even though the
// server lowercases the value before matching. The server returns a 400
// naming csv/jsonl/parquet and is the authoritative check.
var root = AuditCommandTestHarness.BuildRoot();
var parse = root.Parse(new[]
{
"audit", "export", "--since", "1h", "--until", "0h",
"--format", "xml", "--output", "/tmp/out.xml",
});
Assert.NotEmpty(parse.Errors);
Assert.Empty(parse.Errors);
}
[Theory]
[InlineData("CSV")]
[InlineData("Csv")]
[InlineData("JSONL")]
public void Export_FormatInAnyCase_IsAccepted(string format)
{
var root = AuditCommandTestHarness.BuildRoot();
var parse = root.Parse(new[]
{
"audit", "export", "--since", "1h", "--until", "0h",
"--format", format, "--output", "/tmp/out.csv",
});
Assert.Empty(parse.Errors);
}
// ---- Query string -----------------------------------------------------
@@ -148,16 +167,18 @@ public class AuditExportCommandTests
}
[Fact]
public void Export_MultiValueChannel_WithOneInvalidName_FailsFast()
public void Export_MultiValueChannel_WithOneUnrecognisedName_StillParses()
{
// AcceptOnlyFromAmong validates EACH value of the multi-value option.
// Register row 99: the vocabulary gate moved server-side; the multi-value
// shape itself is unchanged, so both values reach the server to adjudicate.
var root = AuditCommandTestHarness.BuildRoot();
var (exit, _, err) = AuditCommandTestHarness.Invoke(
root, "audit", "export", "--since", "1h", "--until", "0h",
var parse = root.Parse(new[]
{
"audit", "export", "--since", "1h", "--until", "0h",
"--format", "csv", "--output", "/tmp/out.csv",
"--channel", "ApiOutbound", "OutboundApi");
Assert.NotEqual(0, exit);
Assert.NotEqual("", err);
"--channel", "ApiOutbound", "OutboundApi",
});
Assert.Empty(parse.Errors);
}
// ---- Streaming export to file -----------------------------------------
tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/Commands/AuditQueryCommandTests.cs
+35 -27
View File
@@ -424,43 +424,51 @@ public class AuditQueryCommandTests
Assert.Empty(parse.Errors);
}
[Fact]
public void Query_MultiValueChannel_WithOneInvalidName_FailsFast()
// ========================================================================
// Register row 99: filter-vocabulary validation moved OFF the CLI.
//
// These four tests used to assert that AcceptOnlyFromAmong rejected an
// unrecognised --channel/--kind/--status at the parse layer. That gate matched
// case-SENSITIVELY, so it also rejected `--status delivered`, which the server
// has always accepted — the CLI was stricter than the system it drives. The
// gate is gone; the parse layer now accepts any string and the server answers
// 400 naming the valid set (AuditEndpointsTests pins that side).
//
// What is asserted here is the CLIENT half of that contract: the value is
// carried through to the request rather than rejected locally.
// ========================================================================
[Theory]
[InlineData("--channel", "OutboundApi")] // old (non-existent) name; the real enum is "ApiOutbound"
[InlineData("--kind", "CachedCall")]
[InlineData("--status", "Bogus")]
public void Query_UnrecognisedFilterValue_IsNotRejectedAtTheParseLayer(string option, string value)
{
// AcceptOnlyFromAmong validates EACH value of the multi-value option.
var root = AuditCommandTestHarness.BuildRoot();
var (exit, _, err) = AuditCommandTestHarness.Invoke(
root, "audit", "query", "--channel", "ApiOutbound", "OutboundApi");
Assert.NotEqual(0, exit);
Assert.NotEqual("", err);
var parse = root.Parse(new[] { "audit", "query", option, value });
Assert.Empty(parse.Errors);
}
[Fact]
public void Query_ChannelWithInvalidName_FailsFast_NonZeroExit()
public void Query_MultiValueChannel_WithOneUnrecognisedName_StillParses()
{
// "OutboundApi" is the old (non-existent) name; the real enum is "ApiOutbound".
// The multi-value shape is unchanged; only the vocabulary gate is gone, so
// BOTH values reach the server for it to adjudicate.
var root = AuditCommandTestHarness.BuildRoot();
var (exit, _, err) = AuditCommandTestHarness.Invoke(root, "audit", "query", "--channel", "OutboundApi");
Assert.NotEqual(0, exit);
Assert.NotEqual("", err);
Assert.Contains("OutboundApi", err);
var parse = root.Parse(new[] { "audit", "query", "--channel", "ApiOutbound", "OutboundApi" });
Assert.Empty(parse.Errors);
}
[Fact]
public void Query_KindWithInvalidName_FailsFast_NonZeroExit()
[Theory]
[InlineData("--channel", "apioutbound")]
[InlineData("--kind", "apicall")]
[InlineData("--status", "delivered")]
[InlineData("--status", "DELIVERED")]
public void Query_FilterValueInAnyCase_IsAccepted(string option, string value)
{
// The defect row 99 names: these all failed client-side before the fix.
var root = AuditCommandTestHarness.BuildRoot();
var (exit, _, err) = AuditCommandTestHarness.Invoke(root, "audit", "query", "--kind", "CachedCall");
Assert.NotEqual(0, exit);
Assert.NotEqual("", err);
}
[Fact]
public void Query_StatusWithInvalidName_FailsFast_NonZeroExit()
{
var root = AuditCommandTestHarness.BuildRoot();
var (exit, _, err) = AuditCommandTestHarness.Invoke(root, "audit", "query", "--status", "Bogus");
Assert.NotEqual(0, exit);
Assert.NotEqual("", err);
var parse = root.Parse(new[] { "audit", "query", option, value });
Assert.Empty(parse.Errors);
}
}
tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/DeploymentStatusTableProjectionTests.cs
+89
View File
@@ -0,0 +1,89 @@
using System.Text.Json;
using ZB.MOM.WW.ScadaBridge.CLI.Commands;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
namespace ZB.MOM.WW.ScadaBridge.CLI.Tests;
/// <summary>
/// Tests for the <c>deploy status</c> table projection (deferred-work register rows
/// 37/87). The handler now returns a <see cref="DeploymentQueryResult"/> OBJECT where
/// it used to return a bare row array; without a projector the generic renderer takes
/// its Property/Value branch and prints the whole <c>deployments</c> array as raw JSON
/// inside one cell. The projection unwraps the envelope and keeps the total visible as
/// a trailing note — which matters more here than for the audit query, because the
/// page size is bounded server-side and a full page is no longer proof there is
/// nothing more.
/// </summary>
public class DeploymentStatusTableProjectionTests
{
/// <summary>
/// The real server shape, serialised from the REAL <see cref="DeploymentQueryResult"/>
/// / <see cref="DeploymentListRow"/> types rather than hand-written JSON, so renaming
/// one of their properties fails this test instead of silently blanking a column.
/// </summary>
private static string RealPayload(int rowCount, int totalCount) =>
JsonSerializer.Serialize(
new DeploymentQueryResult(
Enumerable.Range(1, rowCount).Select(i => new DeploymentListRow(
i, $"dep-{i:D3}", i, $"Line3.Pump{i}", 2, DeploymentStatus.Success,
$"rev-{i}", "alice",
new DateTimeOffset(2026, 8, 15, 12, 0, 0, TimeSpan.Zero), null, null)).ToList(),
totalCount,
Page: 1,
PageSize: rowCount),
new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.CamelCase });
[Fact]
public void Projection_UnwrapsTheEnvelopeToARowArray()
{
var projected = DeploymentStatusTableProjection.ProjectRows(RealPayload(2, 2));
using var doc = JsonDocument.Parse(projected);
Assert.Equal(JsonValueKind.Array, doc.RootElement.ValueKind);
Assert.Equal(2, doc.RootElement.GetArrayLength());
Assert.Equal("dep-001", doc.RootElement[0].GetProperty("deploymentId").GetString());
Assert.Equal("Line3.Pump1", doc.RootElement[0].GetProperty("instanceUniqueName").GetString());
}
[Fact]
public void Projection_WhenPageIsAStrictSubset_AppendsATotalNote()
{
var projected = DeploymentStatusTableProjection.ProjectRows(RealPayload(2, 57));
using var doc = JsonDocument.Parse(projected);
Assert.Equal(3, doc.RootElement.GetArrayLength());
var note = doc.RootElement[2].GetProperty("note").GetString();
Assert.Contains("showing 2 of 57", note);
Assert.Contains("--page-size", note);
}
[Fact]
public void Projection_WhenPageIsTheWholeResult_EmitsNoNoteColumn()
{
var projected = DeploymentStatusTableProjection.ProjectRows(RealPayload(2, 2));
using var doc = JsonDocument.Parse(projected);
Assert.All(doc.RootElement.EnumerateArray(),
e => Assert.False(e.TryGetProperty("note", out _)));
}
[Fact]
public void Projection_AcceptsABareRowArray_FromAPreEnvelopeServer()
{
var projected = DeploymentStatusTableProjection.ProjectRows("""[{"deploymentId":"dep-001"}]""");
using var doc = JsonDocument.Parse(projected);
Assert.Equal("dep-001", Assert.Single(doc.RootElement.EnumerateArray())
.GetProperty("deploymentId").GetString());
}
[Theory]
[InlineData("not json at all")]
[InlineData("""{"error":"boom"}""")]
public void Projection_LeavesUnrecognisedBodiesUntouched(string body)
{
Assert.Equal(body, DeploymentStatusTableProjection.ProjectRows(body));
}
}
tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Auth/SiteScopeServiceTests.cs
+71
View File
@@ -90,4 +90,75 @@ public class SiteScopeServiceTests
Assert.True(await svc.IsSiteAllowedAsync(1));
Assert.True(await svc.IsSiteAllowedAsync(999));
}
// ── Administrator is system-wide regardless of SiteId claims (register 86) ──
//
// ManagementActor has always exempted Administrator from site scoping
// (EnforceSiteScope and every handler that calls it). This service was
// claims-only, so an Administrator who ALSO carried a scoped Deployer grant
// could act fleet-wide through the CLI/HTTP surface while the UI hid the same
// sites. The UI was the side that disagreed with the documented rule.
[Fact]
public async Task AdministratorWithSiteClaims_IsSystemWide()
{
var svc = ForUser(Role("Administrator"), Role("Deployer"), SiteClaim(1), SiteClaim(3));
Assert.True(await svc.IsSystemWideAsync());
}
[Fact]
public async Task AdministratorWithSiteClaims_FilterSites_ReturnsAllSites()
{
var svc = ForUser(Role("Administrator"), Role("Deployer"), SiteClaim(1));
var filtered = await svc.FilterSitesAsync(Sites(1, 2, 3));
Assert.Equal(new[] { 1, 2, 3 }, filtered.Select(s => s.Id));
}
[Fact]
public async Task AdministratorWithSiteClaims_IsSiteAllowed_ForSitesOutsideTheClaims()
{
var svc = ForUser(Role("Administrator"), SiteClaim(1));
Assert.True(await svc.IsSiteAllowedAsync(2));
}
[Fact]
public async Task AdministratorWithSiteClaims_ReportsNoPermittedIds()
{
// The invariant "system-wide => empty id set" must hold for every caller, so
// a consumer that reads the ids without consulting IsSystemWide cannot
// silently narrow an Administrator to a subset of sites.
var svc = ForUser(Role("Administrator"), SiteClaim(1), SiteClaim(3));
Assert.Empty(await svc.PermittedSiteIdsAsync());
}
[Theory]
[InlineData("administrator")]
[InlineData("ADMINISTRATOR")]
public async Task AdministratorRoleMatch_IsCaseInsensitive(string role)
{
// ManagementActor compares roles with OrdinalIgnoreCase. ClaimsPrincipal
// .IsInRole compares claim VALUES ordinally, so matching the actor requires
// reading the role claims explicitly — otherwise the two sides would
// disagree on a differently-cased role name.
var svc = ForUser(Role(role), SiteClaim(1));
Assert.True(await svc.IsSystemWideAsync());
}
[Fact]
public async Task DesignerWithSiteClaims_IsStillScoped()
{
// The class doc used to claim "any Admin/Design user" is system-wide. Only
// Administrator is — on either side of the rule — and the doc was corrected
// rather than the bypass widened.
var svc = ForUser(Role("Designer"), SiteClaim(1));
Assert.False(await svc.IsSystemWideAsync());
Assert.False(await svc.IsSiteAllowedAsync(2));
}
}
tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/AuditQueryParamParsersTests.cs
+97 -5
View File
@@ -4,11 +4,17 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
namespace ZB.MOM.WW.ScadaBridge.Commons.Tests.Types;
/// <summary>
/// Audit Log #23 (M8): tests for the shared lax multi-value query-param parsers
/// used by the ManagementService + CentralUI audit endpoints and the
/// <c>AuditLogPage</c> drill-in parser. The contract under test: parse each
/// repeated value independently, silently drop unparseable/blank elements, and
/// collapse an empty result to <c>null</c>.
/// Audit Log #23 (M8): tests for the shared multi-value query-param parsers used by
/// the ManagementService + CentralUI audit endpoints and the <c>AuditLogPage</c>
/// drill-in parser. Two contracts are under test:
/// <list type="bullet">
/// <item><b>Lax</b> (<c>ParseEnumList</c>/<c>ParseStringList</c>, used by the
/// CentralUI surfaces): parse each repeated value independently, silently drop
/// unparseable/blank elements, collapse an empty result to <c>null</c>.</item>
/// <item><b>Strict</b> (<c>ParseEnumListStrict</c>, register row 99, used by the
/// ManagementService endpoints the CLI calls): an unrecognised value is an error
/// naming the valid set; blanks are still dropped.</item>
/// </list>
/// </summary>
public class AuditQueryParamParsersTests
{
@@ -53,6 +59,92 @@ public class AuditQueryParamParsersTests
Assert.Null(AuditQueryParamParsers.ParseEnumList<AuditStatus>(new[] { "Bogus", "" }));
}
// ========================================================================
// Strict variant (register row 99) — the contract the ManagementService
// /api/audit/query + /api/audit/export endpoints use, so the CLI can drop its
// case-SENSITIVE AcceptOnlyFromAmong guard and still fail loudly on garbage.
// ========================================================================
[Fact]
public void ParseEnumListStrict_NullInput_ReturnsNull()
{
Assert.Null(AuditQueryParamParsers.ParseEnumListStrict<AuditStatus>(null, "status"));
}
[Fact]
public void ParseEnumListStrict_EmptyInput_ReturnsNull()
{
Assert.Null(AuditQueryParamParsers.ParseEnumListStrict<AuditStatus>(Array.Empty<string?>(), "status"));
}
[Theory]
[InlineData("delivered")]
[InlineData("DELIVERED")]
[InlineData("Delivered")]
[InlineData(" delivered ")]
public void ParseEnumListStrict_IsCaseInsensitiveAndTrims(string raw)
{
var result = AuditQueryParamParsers.ParseEnumListStrict<AuditStatus>(new[] { raw }, "status");
Assert.Equal(new[] { AuditStatus.Delivered }, result);
}
[Fact]
public void ParseEnumListStrict_AllValuesValid_ParsesEverything()
{
var result = AuditQueryParamParsers.ParseEnumListStrict<AuditChannel>(
new[] { "ApiOutbound", "dboutbound" }, "channel");
Assert.Equal(new[] { AuditChannel.ApiOutbound, AuditChannel.DbOutbound }, result);
}
[Fact]
public void ParseEnumListStrict_UnrecognisedValue_ThrowsNamingTheValidSet()
{
var ex = Assert.Throws<AuditFilterParseException>(
() => AuditQueryParamParsers.ParseEnumListStrict<AuditStatus>(
new[] { "Delivered", "NotAStatus" }, "status"));
// The offending value AND the whole valid vocabulary must be in the message —
// naming the valid set is the entire point of the check.
Assert.Contains("NotAStatus", ex.Message);
Assert.Contains("status", ex.Message);
foreach (var name in Enum.GetNames<AuditStatus>())
{
Assert.Contains(name, ex.Message);
}
}
[Theory]
// "7" is out of range; "3" lands on a REAL member. Enum.TryParse accepts BOTH —
// a filter is a name-based contract, never an ordinal one, so both are rejected.
[InlineData("7")]
[InlineData("3")]
[InlineData("0")]
[InlineData("-1")]
[InlineData("+2")]
public void ParseEnumListStrict_NumericString_IsRejected(string raw)
{
var ex = Assert.Throws<AuditFilterParseException>(
() => AuditQueryParamParsers.ParseEnumListStrict<AuditStatus>(new[] { raw }, "status"));
Assert.Contains(raw, ex.Message);
}
[Fact]
public void ParseEnumListStrict_BlankElements_AreStillDroppedSilently()
{
// A repeated query param can legitimately carry an empty slot; only a
// NON-blank unrecognised value is an error.
var result = AuditQueryParamParsers.ParseEnumListStrict<AuditStatus>(
new[] { "Delivered", "", " ", null }, "status");
Assert.Equal(new[] { AuditStatus.Delivered }, result);
}
[Fact]
public void ParseEnumListStrict_AllBlank_ReturnsNull()
{
Assert.Null(AuditQueryParamParsers.ParseEnumListStrict<AuditStatus>(
new[] { "", " ", null }, "status"));
}
[Fact]
public void ParseStringList_NullInput_ReturnsNull()
{
tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/DeploymentListQueryRepositoryTests.cs
+32 -1
View File
@@ -164,7 +164,7 @@ public class DeploymentListQueryRepositoryTests : IDisposable
new DeploymentListFilter(), pageNumber: 1, pageSize: int.MaxValue);
Assert.Single(page.Rows);
Assert.True(DeploymentRecordSummary.MaxPageSize > 0);
Assert.True(DeploymentListPage.MaxPageSize > 0);
}
[Fact]
@@ -300,6 +300,37 @@ public class DeploymentListQueryRepositoryTests : IDisposable
Assert.Empty(page.Rows);
}
[Fact]
public async Task QueryPage_SiteScopeAndStatus_Compose_ForTheCliQueryPath()
{
// Register row 37: this is now the ONLY paged read of DeploymentRecords —
// the CLI/HTTP QueryDeployments command reaches it through the same filter
// the UI page uses, where it used to have a second repository method with
// instance-id-list scoping and no total. Its combination (site scope AND a
// status filter, over more rows than one page holds) is pinned here.
var s1 = await SeedSiteAsync("S1");
var s2 = await SeedSiteAsync("S2");
var a = await SeedInstanceAsync("Inst-A", s1);
var b = await SeedInstanceAsync("Inst-B", s2);
for (var i = 0; i < 4; i++)
{
await SeedRecordAsync($"dep-a{i}", a, DeploymentStatus.Failed, _base.AddMinutes(i));
await SeedRecordAsync($"dep-b{i}", b, DeploymentStatus.Failed, _base.AddMinutes(i));
}
await SeedRecordAsync("dep-a-ok", a, DeploymentStatus.Success, _base.AddHours(1));
var page = await _repository.QueryDeploymentListPageAsync(
new DeploymentListFilter(Status: DeploymentStatus.Failed, SiteIdScope: new[] { s1 }),
pageNumber: 1, pageSize: 2);
// Four in-scope failures exist; the page is bounded to two and the total
// reports the rest, so a bounded page is never mistaken for a complete one.
Assert.Equal(4, page.TotalCount);
Assert.Equal(2, page.Rows.Count);
Assert.All(page.Rows, r => Assert.Equal(s1, r.SiteId));
Assert.All(page.Rows, r => Assert.Equal(DeploymentStatus.Failed, r.Status));
}
// ── Status counts ─────────────────────────────────────────────────────────
[Fact]
tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests.csproj
+17
View File
@@ -42,6 +42,23 @@
<Using Include="Xunit" />
</ItemGroup>
<ItemGroup>
<!--
Deferred-work register row 96 — per-suite in-process parallelism cap.
This is the suite that owns MsSqlMigrationFixture: every MSSQL-backed test
CLASS gets its own database, i.e. a CREATE DATABASE plus the full
~100-migration chain against the ONE shared dev SQL Server. Uncapped,
xunit runs Environment.ProcessorCount collections at once, so a 14-core
dev box asked the server for 14 concurrent fresh-database migrations from
this process alone. The cap is threads only — collections stay parallel —
and 4 matches the default slot count of the cross-process sp_getapplock
migrate semaphore (row 82), so the in-process and cross-process limits
agree. Policy: docs/test_infra/test_infra.md.
The json must be copied to the output directory to take effect.
-->
<Content Include="xunit.runner.json" CopyToOutputDirectory="PreserveNewest" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.csproj" />
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.Commons/ZB.MOM.WW.ScadaBridge.Commons.csproj" />
tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/xunit.runner.json
+4
View File
@@ -0,0 +1,4 @@
{
"$schema": "https://xunit.net/schema/current/xunit.runner.schema.json",
"maxParallelThreads": 4
}
tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Actors/DataConnectionReconfigureTests.cs
+351
View File
@@ -0,0 +1,351 @@
using Akka.Actor;
using Akka.TestKit.Xunit2;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Actors;
using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests.Actors;
/// <summary>
/// Deferred register row <b>97</b> — <b>in-place reconfiguration of a LIVE data connection</b>.
///
/// <para>
/// <b>The gap these pin.</b> <c>DataConnectionManagerActor.HandleCreateConnection</c> is
/// create-only: it warns and returns for a name it already holds. So the Deployment Manager's
/// documented "a changed configuration re-issues the create so the DCL adopts it" was never true
/// on ANY node — a corrected endpoint or credential took effect only when the connection was next
/// built from scratch (node restart, singleton failover, or a remove-then-add), while the DM's
/// hash cache recorded it as applied. Discovered while validating rows 89/90 and recorded as
/// residual 1 of <c>docs/plans/2026-08-17-dcl-connection-arbitration-decision.md</c>.
/// </para>
///
/// <para>
/// <b>Why the fix is a swap and not a restart.</b> Stopping and re-creating the connection actor
/// is destructive: the subscription registry lives inside <c>DataConnectionActor</c>, and
/// <c>InstanceActor</c> cancels its per-connection subscribe-retry timer on the first success —
/// so nothing would ever re-subscribe. That trades "stale endpoint, data flowing" for "correct
/// endpoint, no data until redeploy". <see cref="ReconfigureConnectionCommand"/> instead reuses
/// the primary→backup failover machinery: dispose the adapter, build one from the new config,
/// bump the adapter generation, and let the reconnect cycle re-issue every subscription from the
/// registry that was never touched.
/// </para>
/// </summary>
[Collection("DataConnectionManagerActor")]
public class DataConnectionReconfigureTests : TestKit
{
private const string OldEndpoint = "opc.tcp://old-plc:4840";
private const string NewEndpoint = "opc.tcp://new-plc:4840";
private readonly DataConnectionOptions _options;
private readonly ISiteHealthCollector _healthCollector;
private readonly IDataConnectionFactory _factory;
public DataConnectionReconfigureTests()
: base(@"akka.loglevel = DEBUG")
{
_healthCollector = Substitute.For<ISiteHealthCollector>();
_factory = Substitute.For<IDataConnectionFactory>();
_options = new DataConnectionOptions
{
ReconnectInterval = TimeSpan.FromMilliseconds(100),
TagResolutionRetryInterval = TimeSpan.FromMilliseconds(200),
WriteTimeout = TimeSpan.FromSeconds(5),
SeedReadMaxAttempts = 1,
SeedReadRetryDelay = TimeSpan.FromMilliseconds(10)
};
}
private static Dictionary<string, string> Config(string endpoint) =>
new() { ["Endpoint"] = endpoint };
private IActorRef CreateConnectionActor(
string name, IDataConnection adapter, IDictionary<string, string> primaryConfig) =>
Sys.ActorOf(Props.Create(() =>
new DataConnectionActor(
name, adapter, _options, _healthCollector, _factory, "OpcUa",
primaryConfig, null, 3)), name);
/// <summary>An adapter that connects, subscribes and reads without incident.</summary>
private static IDataConnection HealthyAdapter(string subscriptionId)
{
var adapter = Substitute.For<IDataConnection>();
adapter.ConnectAsync(Arg.Any<IDictionary<string, string>>(), Arg.Any<CancellationToken>())
.Returns(Task.CompletedTask);
adapter.Status.Returns(ConnectionHealth.Connected);
adapter.SubscribeAsync(Arg.Any<string>(), Arg.Any<SubscriptionCallback>(), Arg.Any<CancellationToken>())
.Returns(subscriptionId);
adapter.ReadAsync(Arg.Any<string>(), Arg.Any<CancellationToken>())
.Returns(new ReadResult(false, null, null));
return adapter;
}
private static void RaiseDisconnected(IDataConnection adapter) =>
adapter.Disconnected += Raise.Event<Action>();
private bool HasEnteredConnectedState(string connectionName) =>
_healthCollector.ReceivedCalls().Any(c =>
c.GetMethodInfo().Name == nameof(ISiteHealthCollector.UpdateConnectionHealth) &&
c.GetArguments() is [string name, ConnectionHealth.Connected, ..] &&
name == connectionName);
private bool CreatedAdapterFor(string endpoint) =>
_factory.ReceivedCalls().Any(c =>
c.GetMethodInfo().Name == nameof(IDataConnectionFactory.Create) &&
c.GetArguments()[1] is IDictionary<string, string> d &&
d.TryGetValue("Endpoint", out var e) && e == endpoint);
private static bool Subscribed(IDataConnection adapter) =>
adapter.ReceivedCalls().Any(c => c.GetMethodInfo().Name == "SubscribeAsync");
private static bool Connected(IDataConnection adapter) =>
adapter.ReceivedCalls().Any(c => c.GetMethodInfo().Name == "ConnectAsync");
// ── The connection actor ──
/// <summary>
/// The headline behaviour: a live, CONNECTED connection told to reconfigure builds a new
/// adapter from the new configuration and re-issues the subscription it already holds against
/// it — without the subscriber doing anything, and without the connection actor being stopped.
///
/// <para>
/// <b>Fails before the fix:</b> there was no reconfigure command at all, and the create the
/// Deployment Manager re-issued was dropped by the manager's create-only guard, so the old
/// adapter kept serving <c>old-plc</c> indefinitely.
/// </para>
/// </summary>
[Fact]
public void Reconfigure_WhileConnected_SwapsTheAdapter_AndReSubscribesExistingTags()
{
const string name = "reconf-connected";
var oldAdapter = HealthyAdapter("sub-old");
var newAdapter = HealthyAdapter("sub-new");
_factory.Create("OpcUa", Arg.Is<IDictionary<string, string>>(d => d["Endpoint"] == NewEndpoint))
.Returns(newAdapter);
var actor = CreateConnectionActor(name, oldAdapter, Config(OldEndpoint));
AwaitCondition(() => HasEnteredConnectedState(name), TimeSpan.FromSeconds(5),
TimeSpan.FromMilliseconds(10));
actor.Tell(new SubscribeTagsRequest(
"c1", "inst1", name, ["sensor/temp"], DateTimeOffset.UtcNow));
ExpectMsg<SubscribeTagsResponse>(TimeSpan.FromSeconds(5));
AwaitCondition(() => Subscribed(oldAdapter), TimeSpan.FromSeconds(5));
actor.Tell(new ReconfigureConnectionCommand(
name, "OpcUa", Config(NewEndpoint), null, 3));
AwaitCondition(() => CreatedAdapterFor(NewEndpoint), TimeSpan.FromSeconds(5),
TimeSpan.FromMilliseconds(10));
AwaitCondition(() => Connected(newAdapter), TimeSpan.FromSeconds(5));
// The whole point: the subscription survives the swap and is re-issued for free.
AwaitCondition(() => Subscribed(newAdapter), TimeSpan.FromSeconds(5),
TimeSpan.FromMilliseconds(10));
}
/// <summary>
/// The adapter generation must be bumped by the swap, exactly as it is on failover: the
/// outgoing adapter's subscription callbacks captured Self and keep firing, and a value from
/// the endpoint the operator has just corrected AWAY from must never be delivered as if it
/// came from the new one.
/// </summary>
[Fact]
public void Reconfigure_DropsValuesStillArrivingFromTheOldAdapter()
{
const string name = "reconf-generation";
SubscriptionCallback? oldCallback = null;
var oldAdapter = HealthyAdapter("sub-old");
oldAdapter.SubscribeAsync("sensor/temp", Arg.Any<SubscriptionCallback>(), Arg.Any<CancellationToken>())
.Returns(ci =>
{
oldCallback = (SubscriptionCallback)ci[1];
return Task.FromResult("sub-old");
});
var newAdapter = HealthyAdapter("sub-new");
_factory.Create("OpcUa", Arg.Is<IDictionary<string, string>>(d => d["Endpoint"] == NewEndpoint))
.Returns(newAdapter);
var actor = CreateConnectionActor(name, oldAdapter, Config(OldEndpoint));
AwaitCondition(() => HasEnteredConnectedState(name), TimeSpan.FromSeconds(5),
TimeSpan.FromMilliseconds(10));
actor.Tell(new SubscribeTagsRequest(
"c1", TestActor.Path.Name, name, ["sensor/temp"], DateTimeOffset.UtcNow));
ExpectMsg<SubscribeTagsResponse>(TimeSpan.FromSeconds(5));
AwaitCondition(() => oldCallback != null, TimeSpan.FromSeconds(5));
actor.Tell(new ReconfigureConnectionCommand(
name, "OpcUa", Config(NewEndpoint), null, 3));
// The swap flips every subscribed tag to bad quality, same as a disconnect.
ExpectMsg<ConnectionQualityChanged>(TimeSpan.FromSeconds(5));
AwaitCondition(() => Subscribed(newAdapter), TimeSpan.FromSeconds(5));
ExpectNoMsg(TimeSpan.FromMilliseconds(300));
oldCallback!("sensor/temp", new TagValue(999, QualityCode.Good, DateTimeOffset.UtcNow));
ExpectNoMsg(TimeSpan.FromSeconds(1));
}
/// <summary>
/// <b>The state that matters most, and the one a stash would break.</b> A connection whose
/// configured endpoint is wrong or dead never reaches Connected at all — it loops in the
/// Connecting state retrying — which is EXACTLY when a corrected configuration is sent. If
/// the reconfigure were stashed with the subscribe/write requests it would wait for the old
/// (broken) configuration to succeed first, i.e. forever.
/// </summary>
[Fact]
public void Reconfigure_WhileStillConnecting_IsHonoured_NotStashed()
{
const string name = "reconf-connecting";
var deadAdapter = Substitute.For<IDataConnection>();
deadAdapter.ConnectAsync(Arg.Any<IDictionary<string, string>>(), Arg.Any<CancellationToken>())
.Returns(Task.FromException(new Exception("Connection refused")));
var newAdapter = HealthyAdapter("sub-new");
_factory.Create("OpcUa", Arg.Is<IDictionary<string, string>>(d => d["Endpoint"] == NewEndpoint))
.Returns(newAdapter);
var actor = CreateConnectionActor(name, deadAdapter, Config(OldEndpoint));
AwaitCondition(() => Connected(deadAdapter), TimeSpan.FromSeconds(5));
Assert.False(HasEnteredConnectedState(name));
actor.Tell(new ReconfigureConnectionCommand(
name, "OpcUa", Config(NewEndpoint), null, 3));
AwaitCondition(() => CreatedAdapterFor(NewEndpoint), TimeSpan.FromSeconds(5),
TimeSpan.FromMilliseconds(10));
AwaitCondition(() => HasEnteredConnectedState(name), TimeSpan.FromSeconds(5),
TimeSpan.FromMilliseconds(10));
}
/// <summary>
/// The same, one state over: a connection that HAD been up and is now looping in the
/// Reconnecting state. Reconnecting stashes subscribes and writes too, so the reconfigure
/// must be handled there as well — and the subscriptions it re-issues are the ones the
/// registry has been holding across the outage.
/// </summary>
[Fact]
public void Reconfigure_WhileReconnecting_IsHonoured_AndReSubscribes()
{
const string name = "reconf-reconnecting";
var connects = 0;
var oldAdapter = HealthyAdapter("sub-old");
oldAdapter.ConnectAsync(Arg.Any<IDictionary<string, string>>(), Arg.Any<CancellationToken>())
.Returns(_ => Interlocked.Increment(ref connects) == 1
? Task.CompletedTask
: Task.FromException(new Exception("still down")));
var newAdapter = HealthyAdapter("sub-new");
_factory.Create("OpcUa", Arg.Is<IDictionary<string, string>>(d => d["Endpoint"] == NewEndpoint))
.Returns(newAdapter);
var actor = CreateConnectionActor(name, oldAdapter, Config(OldEndpoint));
AwaitCondition(() => HasEnteredConnectedState(name), TimeSpan.FromSeconds(5),
TimeSpan.FromMilliseconds(10));
actor.Tell(new SubscribeTagsRequest(
"c1", "inst1", name, ["sensor/temp"], DateTimeOffset.UtcNow));
ExpectMsg<SubscribeTagsResponse>(TimeSpan.FromSeconds(5));
// Drop the link; with no backup configured it retries the same dead endpoint forever.
RaiseDisconnected(oldAdapter);
AwaitCondition(() => connects >= 2, TimeSpan.FromSeconds(5));
actor.Tell(new ReconfigureConnectionCommand(
name, "OpcUa", Config(NewEndpoint), null, 3));
AwaitCondition(() => CreatedAdapterFor(NewEndpoint), TimeSpan.FromSeconds(5),
TimeSpan.FromMilliseconds(10));
AwaitCondition(() => Subscribed(newAdapter), TimeSpan.FromSeconds(5),
TimeSpan.FromMilliseconds(10));
}
/// <summary>
/// A reconfigure carrying the configuration already in force must cost nothing. The sender is
/// hash-gated and does not send one, but an adapter swap drops the link and re-subscribes
/// every tag, so a redundant command must never be able to do that.
/// </summary>
[Fact]
public void Reconfigure_WithTheConfigurationAlreadyInForce_IsANoOp()
{
const string name = "reconf-identical";
var adapter = HealthyAdapter("sub-old");
var actor = CreateConnectionActor(name, adapter, Config(OldEndpoint));
AwaitCondition(() => HasEnteredConnectedState(name), TimeSpan.FromSeconds(5),
TimeSpan.FromMilliseconds(10));
actor.Tell(new ReconfigureConnectionCommand(
name, "OpcUa", Config(OldEndpoint), null, 3));
// No adapter is ever built: the factory is only consulted by a swap.
ExpectNoMsg(TimeSpan.FromMilliseconds(500));
Assert.False(CreatedAdapterFor(OldEndpoint));
Assert.DoesNotContain(_factory.ReceivedCalls(),
c => c.GetMethodInfo().Name == nameof(IDataConnectionFactory.Create));
}
// ── The manager's routing ──
/// <summary>
/// The manager forwards a reconfigure to the connection actor that owns the name — which is
/// the whole difference from <see cref="CreateConnectionCommand"/>, whose handler warns and
/// returns for a name it already holds.
/// </summary>
[Fact]
public void Manager_Reconfigure_ForALiveConnection_ReachesTheConnectionActor()
{
var oldAdapter = HealthyAdapter("sub-old");
var newAdapter = HealthyAdapter("sub-new");
_factory.Create("OpcUa", Arg.Is<IDictionary<string, string>>(d => d["Endpoint"] == OldEndpoint))
.Returns(oldAdapter);
_factory.Create("OpcUa", Arg.Is<IDictionary<string, string>>(d => d["Endpoint"] == NewEndpoint))
.Returns(newAdapter);
var manager = Sys.ActorOf(Props.Create(() =>
new DataConnectionManagerActor(_factory, _options, _healthCollector)));
manager.Tell(new CreateConnectionCommand("conn1", "OpcUa", Config(OldEndpoint), null, 3));
AwaitCondition(() => Connected(oldAdapter), TimeSpan.FromSeconds(10));
manager.Tell(new ReconfigureConnectionCommand("conn1", "OpcUa", Config(NewEndpoint), null, 3));
AwaitCondition(() => Connected(newAdapter), TimeSpan.FromSeconds(10),
TimeSpan.FromMilliseconds(10));
}
/// <summary>
/// A reconfigure for a name the manager does not hold falls back to a create, so the command
/// is safe whatever the manager's state — a sender whose view of which connections exist is
/// stale still converges on the intended configuration in one message rather than leaving
/// the connection missing until something else happens to create it.
/// </summary>
[Fact]
public void Manager_Reconfigure_ForAnUnknownConnection_FallsBackToCreate()
{
var adapter = HealthyAdapter("sub-new");
_factory.Create("OpcUa", Arg.Is<IDictionary<string, string>>(d => d["Endpoint"] == NewEndpoint))
.Returns(adapter);
var manager = Sys.ActorOf(Props.Create(() =>
new DataConnectionManagerActor(_factory, _options, _healthCollector)));
manager.Tell(new ReconfigureConnectionCommand("never-created", "OpcUa", Config(NewEndpoint), null, 3));
AwaitCondition(() => Connected(adapter), TimeSpan.FromSeconds(10),
TimeSpan.FromMilliseconds(10));
// And it is a real, routable connection afterwards — not an orphan.
manager.Tell(new SubscribeTagsRequest(
"c1", "inst1", "never-created", ["sensor/temp"], DateTimeOffset.UtcNow));
var response = ExpectMsg<SubscribeTagsResponse>(TimeSpan.FromSeconds(10));
Assert.True(response.Success);
}
}
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ZB.MOM.WW.ScadaBridge.Host.Tests.csproj
+13
View File
@@ -28,6 +28,19 @@
<Using Include="Xunit" />
</ItemGroup>
<ItemGroup>
<!--
Deferred-work register row 96 — per-suite in-process parallelism cap.
Resource-heavy suite: classes boot real ASP.NET Core hosts and real Akka
ActorSystems — Akka.Remote binds real sockets, Kestrel binds real ports,
gRPC channels dial them. Uncapped that is Environment.ProcessorCount full
hosts standing up at once. The cap is threads only; collections stay
parallel. Policy: docs/test_infra/test_infra.md.
The json must be copied to the output directory to take effect.
-->
<Content Include="xunit.runner.json" CopyToOutputDirectory="PreserveNewest" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.Host/ZB.MOM.WW.ScadaBridge.Host.csproj" />
</ItemGroup>
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/xunit.runner.json
+4
View File
@@ -0,0 +1,4 @@
{
"$schema": "https://xunit.net/schema/current/xunit.runner.schema.json",
"maxParallelThreads": 4
}
tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/AuditEndpointsTests.cs
+63 -6
View File
@@ -416,23 +416,80 @@ public class AuditEndpointsTests
Assert.Null(filter.SourceSiteIds);
}
// ========================================================================
// Register row 99: the enum filter dimensions parse STRICTLY here.
//
// These used to assert the LAX contract — an unrecognised enum name was
// dropped and the query ran unfiltered on that dimension, so `--status
// Nonsense` returned a plausible, complete-looking page that did not answer
// the question asked. That is exactly the failure row 88 fixed for `deploy
// status`, and it became reachable from the CLI once the (case-SENSITIVE,
// and therefore over-strict) client-side AcceptOnlyFromAmong guard was
// removed. The Guid/date dimensions stay lax — see ParseFilter's remarks.
// ========================================================================
[Fact]
public void ParseFilter_UnparseableValuesInRepeatedSet_AreDroppedSilently()
public void ParseFilter_UnrecognisedEnumValue_Throws()
{
// Lax-parse contract: an unrecognised enum name is dropped, the rest of
// the repeated set survives — no 400, no whole-set drop.
var query = new Microsoft.AspNetCore.Http.QueryCollection(
new Dictionary<string, Microsoft.Extensions.Primitives.StringValues>
{
["channel"] = new[] { "ApiOutbound", "Bogus", "Notification" },
});
var ex = Assert.Throws<AuditFilterParseException>(() => AuditEndpoints.ParseFilter(query));
Assert.Contains("Bogus", ex.Message);
Assert.Contains("channel", ex.Message);
}
[Fact]
public void ParseFilter_UnrecognisedStatus_ThrowsNamingTheValidSet()
{
var query = new Microsoft.AspNetCore.Http.QueryCollection(
new Dictionary<string, Microsoft.Extensions.Primitives.StringValues>
{
["status"] = new[] { "Nonsense" },
});
var ex = Assert.Throws<AuditFilterParseException>(() => AuditEndpoints.ParseFilter(query));
foreach (var name in Enum.GetNames<AuditStatus>())
{
Assert.Contains(name, ex.Message);
}
}
[Fact]
public void ParseFilter_LowercaseEnumValues_AreAccepted()
{
// The CLI sends whatever the operator typed now that its own case-sensitive
// guard is gone; `--status delivered` must resolve here.
var query = new Microsoft.AspNetCore.Http.QueryCollection(
new Dictionary<string, Microsoft.Extensions.Primitives.StringValues>
{
["channel"] = new[] { "apioutbound" },
["kind"] = new[] { "apicall" },
["status"] = new[] { "delivered", "PARKED" },
});
var filter = AuditEndpoints.ParseFilter(query);
Assert.Equal(new[] { AuditChannel.ApiOutbound, AuditChannel.Notification }, filter.Channels);
// Every value unparseable → the dimension stays unconstrained (null).
Assert.Null(filter.Statuses);
Assert.Equal(new[] { AuditChannel.ApiOutbound }, filter.Channels);
Assert.Equal(new[] { AuditKind.ApiCall }, filter.Kinds);
Assert.Equal(new[] { AuditStatus.Delivered, AuditStatus.Parked }, filter.Statuses);
}
[Fact]
public void ParseFilter_NumericEnumValue_IsRejected()
{
// Enum.TryParse would accept "3" as an ordinal; a filter is a name-based
// contract, so it must not.
var query = new Microsoft.AspNetCore.Http.QueryCollection(
new Dictionary<string, Microsoft.Extensions.Primitives.StringValues>
{
["status"] = new[] { "3" },
});
Assert.Throws<AuditFilterParseException>(() => AuditEndpoints.ParseFilter(query));
}
[Fact]
tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/ManagementActorTests.cs
+403 -64
View File
@@ -1,3 +1,4 @@
using System.Text.Json;
using Akka.Actor;
using Akka.TestKit.Xunit2;
using Microsoft.Extensions.DependencyInjection;
@@ -984,6 +985,193 @@ public class ManagementActorTests : TestKit, IDisposable
ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
}
// ========================================================================
// Register row 98: the scope gate fails CLOSED on an unresolvable target.
//
// EnforceSiteScopeForInstance / EnforceSiteScopeForIdentifier used to skip
// EnforceSiteScope entirely when the id did not resolve, so a site-scoped
// caller PASSED the gate and fell through into the handler body. A gate that
// cannot prove the target is inside the caller's scope must treat it as
// outside. System-wide callers return at the guards above the lookup and are
// untouched — they still get the handler's ordinary "not found".
// ========================================================================
[Fact]
public void GetInstance_NonexistentIdForSiteScopedUser_ReturnsUnauthorized()
{
// _templateRepo is unstubbed for id 999 → GetInstanceByIdAsync returns null.
var actor = CreateActor();
var envelope = ScopedEnvelope(new GetInstanceCommand(999), new[] { "1" }, "Deployer");
actor.Tell(envelope);
var response = ExpectMsg<ManagementUnauthorized>(TimeSpan.FromSeconds(5));
Assert.Equal(envelope.CorrelationId, response.CorrelationId);
// The denial must not double as an existence oracle: the message says the
// instance is not in scope, never whether it exists.
Assert.DoesNotContain("not found", response.Message, StringComparison.OrdinalIgnoreCase);
Assert.DoesNotContain("exist", response.Message, StringComparison.OrdinalIgnoreCase);
}
[Fact]
public void GetInstance_NonexistentIdForSiteScopedUser_IsIndistinguishableFromOutOfScope()
{
// The security property: a scoped caller cannot tell "instance 7 lives at a
// site you may not see" apart from "instance 999 does not exist" — both are
// the same ManagementUnauthorized shape.
_templateRepo.GetInstanceByIdAsync(7, Arg.Any<CancellationToken>())
.Returns(new Instance("Pump7") { Id = 7, SiteId = 2 });
var actor = CreateActor();
actor.Tell(ScopedEnvelope(new GetInstanceCommand(7), new[] { "1" }, "Deployer"));
var outOfScope = ExpectMsg<ManagementUnauthorized>(TimeSpan.FromSeconds(5));
actor.Tell(ScopedEnvelope(new GetInstanceCommand(999), new[] { "1" }, "Deployer"));
var nonexistent = ExpectMsg<ManagementUnauthorized>(TimeSpan.FromSeconds(5));
Assert.Equal(outOfScope.GetType(), nonexistent.GetType());
}
[Fact]
public void GetInstance_NonexistentIdForAdminUser_IsUnchanged()
{
// An Administrator is system-wide, so the gate returns before the lookup:
// the handler runs and answers for itself. NOT a ManagementUnauthorized.
var actor = CreateActor();
var envelope = ScopedEnvelope(new GetInstanceCommand(999), new[] { "1" }, "Administrator");
actor.Tell(envelope);
var response = ExpectMsg<object>(TimeSpan.FromSeconds(5));
Assert.IsNotType<ManagementUnauthorized>(response);
}
[Fact]
public void GetInstance_NonexistentIdForSystemWideDeployer_IsUnchanged()
{
// The other system-wide arm: a Deployer with NO site grants. Same rule,
// same untouched behaviour.
var actor = CreateActor();
var envelope = ScopedEnvelope(new GetInstanceCommand(999), Array.Empty<string>(), "Deployer");
actor.Tell(envelope);
var response = ExpectMsg<object>(TimeSpan.FromSeconds(5));
Assert.IsNotType<ManagementUnauthorized>(response);
}
[Fact]
public void GetInstance_InScopeIdForSiteScopedUser_IsStillAllowed()
{
// The fail-closed change must not deny a legitimate in-scope target.
_templateRepo.GetInstanceByIdAsync(7, Arg.Any<CancellationToken>())
.Returns(new Instance("Pump7") { Id = 7, SiteId = 1 });
var actor = CreateActor();
var envelope = ScopedEnvelope(new GetInstanceCommand(7), new[] { "1" }, "Deployer");
actor.Tell(envelope);
ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
}
[Fact]
public void GetSite_NonexistentIdForSiteScopedUser_ReturnsUnauthorized()
{
// HandleGetSite resolves the site itself and carried its own copy of the
// fail-OPEN shape (`if (site != null) EnforceSiteScope(...)`), so a scoped
// caller used to get success/null for any id — an enumeration oracle.
var siteRepo = Substitute.For<ISiteRepository>();
siteRepo.GetSiteByIdAsync(999, Arg.Any<CancellationToken>())
.Returns((Commons.Entities.Sites.Site?)null);
_services.AddScoped(_ => siteRepo);
var actor = CreateActor();
var envelope = ScopedEnvelope(new GetSiteCommand(999), new[] { "1" }, "Deployer");
actor.Tell(envelope);
var response = ExpectMsg<ManagementUnauthorized>(TimeSpan.FromSeconds(5));
Assert.Equal(envelope.CorrelationId, response.CorrelationId);
}
[Fact]
public void GetSite_NonexistentIdForAdminUser_IsUnchanged()
{
var siteRepo = Substitute.For<ISiteRepository>();
siteRepo.GetSiteByIdAsync(999, Arg.Any<CancellationToken>())
.Returns((Commons.Entities.Sites.Site?)null);
_services.AddScoped(_ => siteRepo);
var actor = CreateActor();
var envelope = ScopedEnvelope(new GetSiteCommand(999), new[] { "1" }, "Administrator");
actor.Tell(envelope);
var response = ExpectMsg<object>(TimeSpan.FromSeconds(5));
Assert.IsNotType<ManagementUnauthorized>(response);
}
[Fact]
public void GetDataConnection_NonexistentIdForSiteScopedUser_ReturnsUnauthorized()
{
// Third copy of the same inline fail-OPEN shape.
var siteRepo = Substitute.For<ISiteRepository>();
siteRepo.GetDataConnectionByIdAsync(999, Arg.Any<CancellationToken>())
.Returns((Commons.Entities.Sites.DataConnection?)null);
_services.AddScoped(_ => siteRepo);
var actor = CreateActor();
var envelope = ScopedEnvelope(new GetDataConnectionCommand(999), new[] { "1" }, "Deployer");
actor.Tell(envelope);
var response = ExpectMsg<ManagementUnauthorized>(TimeSpan.FromSeconds(5));
Assert.Equal(envelope.CorrelationId, response.CorrelationId);
}
[Fact]
public void QueryEventLogs_NonexistentSiteIdentifierForSiteScopedUser_ReturnsUnauthorized()
{
// EnforceSiteScopeForIdentifier shares the shape: an identifier that
// resolves to no Site row is a denial for a scoped caller.
var siteRepo = Substitute.For<ISiteRepository>();
siteRepo.GetSiteByIdentifierAsync("GHOST", Arg.Any<CancellationToken>())
.Returns((Commons.Entities.Sites.Site?)null);
_services.AddScoped(_ => siteRepo);
var actor = CreateActor();
var envelope = ScopedEnvelope(
new QueryEventLogsCommand("GHOST"),
new[] { "1" },
"Deployer");
actor.Tell(envelope);
var response = ExpectMsg<ManagementUnauthorized>(TimeSpan.FromSeconds(5));
Assert.Equal(envelope.CorrelationId, response.CorrelationId);
}
[Fact]
public void QueryEventLogs_InScopeSiteIdentifierForSiteScopedUser_IsStillAllowed()
{
AddSiteRepoWithSite(1, "SITE1");
var actor = CreateActor();
var envelope = ScopedEnvelope(
new QueryEventLogsCommand("SITE1"),
new[] { "1" },
"Deployer");
actor.Tell(envelope);
// The scope gate passes; whatever the (unwired) CommunicationService does
// next is not this test's business — only that it is NOT a scope denial.
var response = ExpectMsg<object>(TimeSpan.FromSeconds(5));
Assert.IsNotType<ManagementUnauthorized>(response);
}
[Fact]
public void GetSite_OutOfScopeForSiteScopedUser_ReturnsUnauthorized()
{
@@ -1222,51 +1410,63 @@ public class ManagementActorTests : TestKit, IDisposable
new("deploy-" + instanceId, "operator") { Id = instanceId, InstanceId = instanceId };
/// <summary>
/// Row-summary equivalent of <see cref="DeploymentRecordFor"/>. QueryDeployments
/// now returns DB-side projections instead of whole tracked entities.
/// Row-projection equivalent of <see cref="DeploymentRecordFor"/>, on the site
/// given. QueryDeployments returns DB-side projections joined to their instance,
/// not whole tracked entities — and the join is what carries the
/// <c>SiteId</c> the scope filter is evaluated against.
/// </summary>
private static DeploymentRecordSummary DeploymentSummaryFor(int instanceId) =>
new(instanceId, "deploy-" + instanceId, instanceId, DeploymentStatus.Success,
null, "operator", DateTimeOffset.UtcNow, DateTimeOffset.UtcNow, null);
private static DeploymentListRow DeploymentRowFor(int instanceId, int siteId) =>
new(instanceId, "deploy-" + instanceId, instanceId, "Pump" + instanceId, siteId,
DeploymentStatus.Success, null, "operator", DateTimeOffset.UtcNow, DateTimeOffset.UtcNow, null);
/// <summary>
/// Stubs <c>QueryDeploymentSummariesAsync</c> so it applies the same
/// instance / status / scope / paging predicates the database would.
/// Stubs the single paged deployment read so it applies the same
/// instance / status / site-scope / paging predicates the database would.
///
/// <para>
/// The filtering moved OUT of the handler and INTO the query, so a stub that
/// ignored the arguments would make these tests pass no matter what the handler
/// passed down. Honouring them here is what keeps the scope-enforcement
/// assertions meaningful.
/// The filtering lives entirely in the query now (register row 37 folded the
/// handler's own second repository method away), so a stub that ignored the
/// filter would make these tests pass no matter what the handler passed down.
/// Honouring it here is what keeps the scope-enforcement assertions meaningful.
/// </para>
/// </summary>
private static void StubDeploymentSummaryQuery(
/// <param name="repo">The substituted repository to stub.</param>
/// <param name="rows">The seed rows, as (instance id, site id) pairs.</param>
private static void StubDeploymentPageQuery(
IDeploymentManagerRepository repo,
IEnumerable<int> instanceIds)
params (int InstanceId, int SiteId)[] rows)
{
var rows = instanceIds.Select(DeploymentSummaryFor).ToList();
repo.QueryDeploymentSummariesAsync(
Arg.Any<int?>(), Arg.Any<DeploymentStatus?>(), Arg.Any<IReadOnlyCollection<int>?>(),
Arg.Any<int>(), Arg.Any<int?>(), Arg.Any<CancellationToken>())
var all = rows.Select(r => DeploymentRowFor(r.InstanceId, r.SiteId)).ToList();
repo.QueryDeploymentListPageAsync(
Arg.Any<DeploymentListFilter>(), Arg.Any<int>(), Arg.Any<int>(), Arg.Any<CancellationToken>())
.Returns(ci =>
{
var instanceId = ci.ArgAt<int?>(0);
var scope = ci.ArgAt<IReadOnlyCollection<int>?>(2);
var skip = ci.ArgAt<int>(3);
var take = ci.ArgAt<int?>(4);
var filter = ci.ArgAt<DeploymentListFilter>(0);
var page = Math.Max(1, ci.ArgAt<int>(1));
var size = Math.Max(1, ci.ArgAt<int>(2));
IEnumerable<DeploymentRecordSummary> q = rows;
if (instanceId.HasValue)
q = q.Where(r => r.InstanceId == instanceId.Value);
if (scope != null)
q = q.Where(r => scope.Contains(r.InstanceId));
q = q.Skip(Math.Max(0, skip));
if (take is > 0)
q = q.Take(take.Value);
return (IReadOnlyList<DeploymentRecordSummary>)q.ToList();
IEnumerable<DeploymentListRow> q = all;
if (filter.InstanceId.HasValue)
q = q.Where(r => r.InstanceId == filter.InstanceId.Value);
if (filter.Status.HasValue)
q = q.Where(r => r.Status == filter.Status.Value);
if (filter.SiteIdScope != null)
q = q.Where(r => filter.SiteIdScope.Contains(r.SiteId));
var matched = q.ToList();
var slice = matched.Skip((page - 1) * size).Take(size).ToList();
return new DeploymentListPage(slice, matched.Count);
});
}
/// <summary>Captures the filter the handler pushed into the paged read.</summary>
private static DeploymentListFilter CapturedDeploymentFilter(IDeploymentManagerRepository repo)
{
var call = repo.ReceivedCalls()
.Single(c => c.GetMethodInfo().Name == nameof(IDeploymentManagerRepository.QueryDeploymentListPageAsync));
return (DeploymentListFilter)call.GetArguments()[0]!;
}
/// <summary>
/// Stubs <c>GetTemplateSummariesAsync</c> over <paramref name="count"/> synthetic
/// templates, applying the requested Skip/Take exactly as the database would.
@@ -1305,7 +1505,7 @@ public class ManagementActorTests : TestKit, IDisposable
public void QueryDeployments_UnfilteredWithDeploymentRole_ReturnsAllRecords()
{
var deployRepo = Substitute.For<IDeploymentManagerRepository>();
StubDeploymentSummaryQuery(deployRepo, [1, 2]);
StubDeploymentPageQuery(deployRepo, (1, 1), (2, 2));
_services.AddScoped(_ => deployRepo);
var actor = CreateActor();
@@ -1322,7 +1522,7 @@ public class ManagementActorTests : TestKit, IDisposable
public void QueryDeployments_FilteredByInstanceId_ReturnsInstanceRecords()
{
var deployRepo = Substitute.For<IDeploymentManagerRepository>();
StubDeploymentSummaryQuery(deployRepo, [5]);
StubDeploymentPageQuery(deployRepo, (5, 1));
_services.AddScoped(_ => deployRepo);
var actor = CreateActor();
@@ -1351,8 +1551,7 @@ public class ManagementActorTests : TestKit, IDisposable
var response = ExpectMsg<ManagementUnauthorized>(TimeSpan.FromSeconds(5));
Assert.Equal(envelope.CorrelationId, response.CorrelationId);
// The out-of-scope instance's deployment history must not be queried.
deployRepo.DidNotReceiveWithAnyArgs().QueryDeploymentSummariesAsync(
default, default, default, default, default);
deployRepo.DidNotReceiveWithAnyArgs().QueryDeploymentListPageAsync(default!, default, default);
}
[Fact]
@@ -1361,7 +1560,7 @@ public class ManagementActorTests : TestKit, IDisposable
_templateRepo.GetInstanceByIdAsync(5, Arg.Any<CancellationToken>())
.Returns(new Instance("Pump5") { Id = 5, SiteId = 1 });
var deployRepo = Substitute.For<IDeploymentManagerRepository>();
StubDeploymentSummaryQuery(deployRepo, [5]);
StubDeploymentPageQuery(deployRepo, (5, 1));
_services.AddScoped(_ => deployRepo);
var actor = CreateActor();
@@ -1376,17 +1575,8 @@ public class ManagementActorTests : TestKit, IDisposable
[Fact]
public void QueryDeployments_UnfilteredForSiteScopedUser_DropsOutOfScopeRecords()
{
// ManagementService-023: the unfiltered branch now bulk-loads instances
// once via GetAllInstancesAsync (instead of N+1 GetInstanceByIdAsync per
// distinct InstanceId). Mock the bulk path accordingly.
_templateRepo.GetAllInstancesAsync(Arg.Any<CancellationToken>())
.Returns(new List<Instance>
{
new("Pump1") { Id = 1, SiteId = 1 },
new("Pump2") { Id = 2, SiteId = 2 },
});
var deployRepo = Substitute.For<IDeploymentManagerRepository>();
StubDeploymentSummaryQuery(deployRepo, [1, 2]);
StubDeploymentPageQuery(deployRepo, (1, 1), (2, 2));
_services.AddScoped(_ => deployRepo);
var actor = CreateActor();
@@ -1397,28 +1587,19 @@ public class ManagementActorTests : TestKit, IDisposable
var response = ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
Assert.Contains("deploy-1", response.JsonData);
Assert.DoesNotContain("deploy-2", response.JsonData);
// The per-instance lookup must NOT have been used for the unfiltered
// branch — that was the N+1 the bulk load replaced.
_templateRepo.DidNotReceiveWithAnyArgs().GetInstanceByIdAsync(default);
Assert.Equal(new[] { 1 }, CapturedDeploymentFilter(deployRepo).SiteIdScope);
}
[Fact]
public void QueryDeployments_UnfilteredForSiteScopedUser_UsesBulkInstanceLoad_NotPerRecordLookup()
public void QueryDeployments_UnfilteredForSiteScopedUser_ReadsNoInstancesAtAll()
{
// ManagementService-023 regression pin: the unfiltered branch must issue
// GetAllInstancesAsync ONCE and never call GetInstanceByIdAsync, regardless
// of how many DeploymentRecords reference distinct InstanceIds. Before the
// fix, three distinct instance ids would have produced three per-instance
// lookups (textbook N+1).
_templateRepo.GetAllInstancesAsync(Arg.Any<CancellationToken>())
.Returns(new List<Instance>
{
new("Pump1") { Id = 1, SiteId = 1 },
new("Pump2") { Id = 2, SiteId = 2 },
new("Pump3") { Id = 3, SiteId = 1 },
});
// Register row 87. The scoped branch used to resolve the site grant into an
// instance-id LIST, which meant loading EVERY instance in the system on
// every scoped query (a bulk load that had itself replaced an N+1 of
// per-record lookups). The grant now rides as a site-id filter resolved by
// the query's own join, so neither instance read may happen.
var deployRepo = Substitute.For<IDeploymentManagerRepository>();
StubDeploymentSummaryQuery(deployRepo, [1, 2, 3]);
StubDeploymentPageQuery(deployRepo, (1, 1), (2, 2), (3, 1));
_services.AddScoped(_ => deployRepo);
var actor = CreateActor();
@@ -1427,17 +1608,41 @@ public class ManagementActorTests : TestKit, IDisposable
actor.Tell(envelope);
ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
_templateRepo.Received(1).GetAllInstancesAsync(Arg.Any<CancellationToken>());
_templateRepo.DidNotReceiveWithAnyArgs().GetAllInstancesAsync();
_templateRepo.DidNotReceiveWithAnyArgs().GetInstanceByIdAsync(default);
}
[Fact]
public void QueryDeployments_SiteScopedUserWithUnparseableGrant_SeesNothing()
{
// A permitted-site id that is not an integer is dropped, never ignored:
// ignoring it would silently widen the grant to fleet-wide. A grant of
// nothing but garbage therefore yields an EMPTY scope — a real "sees
// nothing" filter, not "no filter".
var deployRepo = Substitute.For<IDeploymentManagerRepository>();
StubDeploymentPageQuery(deployRepo, (1, 1), (2, 2));
_services.AddScoped(_ => deployRepo);
var actor = CreateActor();
var envelope = ScopedEnvelope(new QueryDeploymentsCommand(), new[] { "not-an-id" }, "Deployer");
actor.Tell(envelope);
var response = ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
Assert.DoesNotContain("deploy-1", response.JsonData);
Assert.DoesNotContain("deploy-2", response.JsonData);
Assert.Empty(CapturedDeploymentFilter(deployRepo).SiteIdScope!);
}
[Fact]
public void QueryDeployments_UnfilteredForAdminUser_ReturnsAllRecords()
{
// Admin role bypasses site scoping even with PermittedSiteIds set.
// (The user also holds Deployment so it passes the role gate.)
// The UI's SiteScopeService now agrees — see SiteScopeServiceTests
// (register row 86).
var deployRepo = Substitute.For<IDeploymentManagerRepository>();
StubDeploymentSummaryQuery(deployRepo, [1, 2]);
StubDeploymentPageQuery(deployRepo, (1, 1), (2, 2));
_services.AddScoped(_ => deployRepo);
var actor = CreateActor();
@@ -1448,6 +1653,140 @@ public class ManagementActorTests : TestKit, IDisposable
var response = ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
Assert.Contains("deploy-1", response.JsonData);
Assert.Contains("deploy-2", response.JsonData);
Assert.Null(CapturedDeploymentFilter(deployRepo).SiteIdScope);
}
// ── Loud status parsing + bounded page size (register rows 87 / 88) ────────
[Fact]
public void QueryDeployments_UnknownStatus_FailsLoudlyNamingTheValidValues()
{
// Register row 88: `--status Sucess` used to fall back to "no filter" and
// return EVERY status — a plausible, complete-looking answer to a question
// the caller did not ask.
var deployRepo = Substitute.For<IDeploymentManagerRepository>();
StubDeploymentPageQuery(deployRepo, (1, 1));
_services.AddScoped(_ => deployRepo);
var actor = CreateActor();
actor.Tell(Envelope(new QueryDeploymentsCommand(Status: "Sucess"), "Deployer"));
var response = ExpectMsg<ManagementError>(TimeSpan.FromSeconds(5));
Assert.Contains("Sucess", response.Error);
Assert.Contains("Success", response.Error);
Assert.Contains("Failed", response.Error);
deployRepo.DidNotReceiveWithAnyArgs().QueryDeploymentListPageAsync(default!, default, default);
}
[Theory]
[InlineData("2")]
[InlineData("+1")]
[InlineData("-1")]
public void QueryDeployments_OrdinalStatus_IsRejectedLikeAnyUnknownValue(string status)
{
// Register row 88 postscript (Wave 6): Enum.TryParse accepts ordinals, so
// `--status 2` silently resolved to InProgress despite the documented
// names-only contract. Same guard as AuditQueryParamParsers.LooksNumeric.
var deployRepo = Substitute.For<IDeploymentManagerRepository>();
StubDeploymentPageQuery(deployRepo, (1, 1));
_services.AddScoped(_ => deployRepo);
var actor = CreateActor();
actor.Tell(Envelope(new QueryDeploymentsCommand(Status: status), "Deployer"));
var response = ExpectMsg<ManagementError>(TimeSpan.FromSeconds(5));
Assert.Contains("Success", response.Error);
deployRepo.DidNotReceiveWithAnyArgs().QueryDeploymentListPageAsync(default!, default, default);
}
[Theory]
[InlineData("success")]
[InlineData("SUCCESS")]
public void QueryDeployments_KnownStatus_IsCaseInsensitive(string status)
{
var deployRepo = Substitute.For<IDeploymentManagerRepository>();
StubDeploymentPageQuery(deployRepo, (1, 1));
_services.AddScoped(_ => deployRepo);
var actor = CreateActor();
actor.Tell(Envelope(new QueryDeploymentsCommand(Status: status), "Deployer"));
ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
Assert.Equal(DeploymentStatus.Success, CapturedDeploymentFilter(deployRepo).Status);
}
[Fact]
public void QueryDeployments_NumericStatus_IsRejected()
{
// Enum.TryParse happily accepts "7" and yields an undefined enum value, so
// the parse needs the IsDefined guard as well as the TryParse.
var deployRepo = Substitute.For<IDeploymentManagerRepository>();
StubDeploymentPageQuery(deployRepo, (1, 1));
_services.AddScoped(_ => deployRepo);
var actor = CreateActor();
actor.Tell(Envelope(new QueryDeploymentsCommand(Status: "7"), "Deployer"));
ExpectMsg<ManagementError>(TimeSpan.FromSeconds(5));
}
[Theory]
[InlineData(0)]
[InlineData(-1)]
public void QueryDeployments_NonPositivePageSize_TakesTheDefault_NotUnlimited(int requested)
{
// Register row 87: a non-positive page size used to mean "unlimited" all the
// way down to the repository, materializing the whole retention window.
var deployRepo = Substitute.For<IDeploymentManagerRepository>();
StubDeploymentPageQuery(deployRepo, Enumerable.Range(1, 200).Select(i => (i, 1)).ToArray());
_services.AddScoped(_ => deployRepo);
var actor = CreateActor();
actor.Tell(Envelope(new QueryDeploymentsCommand(PageSize: requested), "Deployer"));
var response = ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
var call = deployRepo.ReceivedCalls()
.Single(c => c.GetMethodInfo().Name == nameof(IDeploymentManagerRepository.QueryDeploymentListPageAsync));
Assert.Equal(DeploymentListPage.DefaultPageSize, (int)call.GetArguments()[2]!);
// The total travels with the bounded page, so a caller can tell a truncated
// page from a complete one.
var result = JsonSerializer.Deserialize<JsonElement>(response.JsonData);
Assert.Equal(200, result.GetProperty("totalCount").GetInt32());
Assert.Equal(DeploymentListPage.DefaultPageSize, result.GetProperty("deployments").GetArrayLength());
Assert.Equal(DeploymentListPage.DefaultPageSize, result.GetProperty("pageSize").GetInt32());
}
[Fact]
public void QueryDeployments_OversizedPageSize_IsCappedAtTheMaximum()
{
var deployRepo = Substitute.For<IDeploymentManagerRepository>();
StubDeploymentPageQuery(deployRepo, (1, 1));
_services.AddScoped(_ => deployRepo);
var actor = CreateActor();
actor.Tell(Envelope(new QueryDeploymentsCommand(PageSize: int.MaxValue), "Deployer"));
ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
var call = deployRepo.ReceivedCalls()
.Single(c => c.GetMethodInfo().Name == nameof(IDeploymentManagerRepository.QueryDeploymentListPageAsync));
Assert.Equal(DeploymentListPage.MaxPageSize, (int)call.GetArguments()[2]!);
}
[Fact]
public void QueryDeployments_PageIsFlooredAtOne()
{
var deployRepo = Substitute.For<IDeploymentManagerRepository>();
StubDeploymentPageQuery(deployRepo, (1, 1));
_services.AddScoped(_ => deployRepo);
var actor = CreateActor();
actor.Tell(Envelope(new QueryDeploymentsCommand(Page: 0), "Deployer"));
var response = ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
Assert.Contains("deploy-1", response.JsonData);
var result = JsonSerializer.Deserialize<JsonElement>(response.JsonData);
Assert.Equal(1, result.GetProperty("page").GetInt32());
}
// ========================================================================
tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests/SiteCallAuditReconciliationTests.cs
+11 -2
View File
@@ -492,8 +492,17 @@ public class SiteCallAuditReconciliationTests : TestKit
interval: TimeSpan.FromMilliseconds(50));
// The rows it saw were still upserted (idempotent mirror refresh).
Assert.True(repo.Upserted.ContainsKey(r1.TrackedOperationId));
Assert.True(repo.Upserted.ContainsKey(r2.TrackedOperationId));
// Awaited, not asserted flat: the upsert follows the pull asynchronously,
// so gating on CallCount alone races the repo write under parallel load
// (observed once in a full-suite run, 2026-08-17).
AwaitAssert(
() =>
{
Assert.True(repo.Upserted.ContainsKey(r1.TrackedOperationId));
Assert.True(repo.Upserted.ContainsKey(r2.TrackedOperationId));
},
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
// Critical SiteCallAudit-009 invariant: the within-tick drain BROKE on the
// no-progress pin rather than looping to the 50-page ceiling. With a 2s
tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests.csproj
+15
View File
@@ -36,6 +36,21 @@
<Using Include="Xunit" />
</ItemGroup>
<ItemGroup>
<!--
Deferred-work register row 96 — per-suite in-process parallelism cap.
This suite's actor tests take an MsSqlMigrationFixture per test CLASS: a
CREATE DATABASE plus the full migration chain on the ONE shared dev SQL
Server, alongside an Akka TestKit ActorSystem. Uncapped, xunit runs
Environment.ProcessorCount collections at once. The cap is threads only —
collections stay parallel — and 4 matches the default slot count of the
cross-process sp_getapplock migrate semaphore (row 82), so the in-process
and cross-process limits agree. Policy: docs/test_infra/test_infra.md.
The json must be copied to the output directory to take effect.
-->
<Content Include="xunit.runner.json" CopyToOutputDirectory="PreserveNewest" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.SiteCallAudit/ZB.MOM.WW.ScadaBridge.SiteCallAudit.csproj" />
<!--
tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests/xunit.runner.json
+4
View File
@@ -0,0 +1,4 @@
{
"$schema": "https://xunit.net/schema/current/xunit.runner.schema.json",
"maxParallelThreads": 4
}
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DclConnectionArbitrationTests.cs
+651
View File
@@ -0,0 +1,651 @@
using System.Collections.Concurrent;
using System.Text.Json;
using Akka.Actor;
using Akka.TestKit.Xunit2;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.TestSupport;
using ZB.MOM.WW.ScadaBridge.TestSupport;
using RemoveConnectionCommand = ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Actors.RemoveConnectionCommand;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors;
/// <summary>
/// Regression tests for deferred register rows <b>89</b>, <b>90</b> and <b>97</b> — <b>which
/// configuration source owns a live DCL connection, and how a change reaches it</b>.
///
/// <para>
/// <b>The arbitration rule under test.</b> The <c>data_connection_definitions</c> store wins
/// wherever it has an opinion; an instance's flattened <c>Connections</c> entry is a deploy-time
/// SNAPSHOT and is authoritative only for names the store has never owned (the deploy-only site,
/// where that table is legitimately empty). Every ensure path resolves the winner BEFORE hashing,
/// so the single per-name hash slot in <c>_createdConnections</c> can only ever hold one source's
/// configuration — which is what makes the two-source recreate-on-every-flip ping-pong
/// unrepresentable rather than merely avoided.
/// </para>
///
/// <para>
/// <b>Verified topology (this is what the register rows really describe).</b> The Deployment
/// Manager is a cluster singleton scoped to <c>site-{SiteId}</c> and is the ONLY sender of
/// <see cref="CreateConnectionCommand"/> / <see cref="RemoveConnectionCommand"/>; the
/// <c>dcl-manager</c> actor exists on every site node but only the singleton's node ever
/// populates one. So the standby's DCL is not "serving the old endpoint" — it is empty, and the
/// divergence surfaces when the singleton's NEXT incarnation rebuilds every connection from the
/// flattened snapshots and never consults the newer definition rows. Design record:
/// <c>docs/plans/2026-08-17-dcl-connection-arbitration-decision.md</c>.
/// </para>
///
/// <para>
/// <b>Row 97 — the third command.</b> Resolving the winner decides WHAT to push; it took a
/// <see cref="ReconfigureConnectionCommand"/> to make a push against an ALREADY-LIVE connection
/// mean anything at all, because the DCL manager's create handler warns and returns for a name it
/// already holds. So a name the hash cache has never held gets a create, a changed one gets a
/// reconfigure, and an unchanged one gets nothing.
/// </para>
/// </summary>
public class DclConnectionArbitrationTests : TestKit, IDisposable
{
private const string ConnectionName = "OPC PLC Simulator";
/// <summary>The endpoint frozen into an instance's flattened config at its deploy time.</summary>
private const string FlattenedEndpoint = "opc.tcp://old-plc:4840";
/// <summary>The endpoint a later, connection-only artifact deploy wrote to the store.</summary>
private const string DefinitionEndpoint = "opc.tcp://new-plc:4840";
private readonly SiteStorageService _storage;
private readonly ScriptCompilationService _compilationService;
private readonly SharedScriptLibrary _sharedScriptLibrary;
private readonly TestLocalDb _localDb;
private readonly ConcurrentQueue<CreateConnectionCommand> _creates = new();
private readonly ConcurrentQueue<ReconfigureConnectionCommand> _reconfigures = new();
private readonly ConcurrentQueue<RemoveConnectionCommand> _removes = new();
public DclConnectionArbitrationTests()
{
_localDb = TestLocalDb.CreateTemp("dcl-arbitration-test");
_storage = new SiteStorageService(_localDb.Db, NullLogger<SiteStorageService>.Instance);
_storage.InitializeAsync().GetAwaiter().GetResult();
_compilationService = new ScriptCompilationService(
NullLogger<ScriptCompilationService>.Instance);
_sharedScriptLibrary = new SharedScriptLibrary(
_compilationService, NullLogger<SharedScriptLibrary>.Instance);
}
void IDisposable.Dispose()
{
// Shut the actors down BEFORE the ILocalDb they write through is disposed.
Shutdown();
var path = _localDb.Path;
_localDb.Dispose();
TestLocalDb.DeleteFiles(path);
}
/// <summary>Shortest sweep the (whole-second) option allows.</summary>
private const int SweepSeconds = 1;
/// <summary>Long enough for at least three sweep ticks to have come and gone.</summary>
private static readonly TimeSpan SeveralSweeps = TimeSpan.FromSeconds(4);
/// <summary>
/// Starts a Deployment Manager wired to a recording stand-in for the DCL manager, and blocks
/// until its boot-time store read has PROVABLY completed (see
/// <see cref="DeploymentManagerStartupProbe"/>). Rows written before that point are consumed
/// by the boot load, which is a different path from the post-boot sweep — the gate is what
/// keeps each test proving the path it names.
/// </summary>
private IActorRef CreateBootedDeploymentManager(int sweepSeconds = SweepSeconds)
{
var dclSpy = ActorOf(Props.Create(() =>
new DclConnectionSpy(_creates, _reconfigures, _removes)));
var health = new DeploymentManagerStartupProbe();
var dm = ActorOf(Props.Create(() => new DeploymentManagerActor(
_storage,
_compilationService,
_sharedScriptLibrary,
null,
new SiteRuntimeOptions { InstanceSweepIntervalSeconds = sweepSeconds },
NullLogger<DeploymentManagerActor>.Instance,
dclManager: dclSpy,
healthCollector: health,
serviceProvider: null,
loggerFactory: null,
configFetcher: null)));
AwaitCondition(() => health.ReportCount > 0, TimeSpan.FromSeconds(20),
TimeSpan.FromMilliseconds(25));
return dm;
}
/// <summary>
/// A flattened config carrying one data connection. The <c>Connections</c> map is the
/// deploy-time snapshot the arbitration rule demotes whenever a definition row exists.
/// </summary>
private static string MakeConfigJson(string instanceName, string endpoint = FlattenedEndpoint) =>
JsonSerializer.Serialize(new FlattenedConfiguration
{
InstanceUniqueName = instanceName,
Attributes =
[
new ResolvedAttribute
{
CanonicalName = "TestAttr", Value = "42", DataType = "Int32"
}
],
Connections = new Dictionary<string, ConnectionConfig>
{
[ConnectionName] = new()
{
Protocol = "OpcUa",
ConfigurationJson = EndpointJson(endpoint),
FailoverRetryCount = 3
}
}
});
private static string EndpointJson(string endpoint) => $"{{\"endpointUrl\":\"{endpoint}\"}}";
/// <summary>Writes the <c>data_connection_definitions</c> row an artifact deploy would.</summary>
private Task StoreDefinitionAsync(string endpoint) =>
_storage.StoreDataConnectionDefinitionAsync(
ConnectionName, "OpcUa", EndpointJson(endpoint), null, 3);
private IReadOnlyList<CreateConnectionCommand> CreatesFor(string connectionName) =>
_creates.Where(c => c.ConnectionName == connectionName).ToArray();
private IReadOnlyList<ReconfigureConnectionCommand> ReconfiguresFor(string connectionName) =>
_reconfigures.Where(c => c.ConnectionName == connectionName).ToArray();
/// <summary>
/// The endpoint the DCL was actually told to connect to — the whole point of the arbitration.
/// </summary>
private static string EndpointOf(CreateConnectionCommand command) =>
command.PrimaryConnectionDetails["endpoint"];
/// <inheritdoc cref="EndpointOf(CreateConnectionCommand)"/>
private static string EndpointOf(ReconfigureConnectionCommand command) =>
command.PrimaryConnectionDetails["endpoint"];
private async Task AwaitCreateCountAsync(int count, string because) =>
await AwaitAssertAsync(() =>
{
Assert.True(CreatesFor(ConnectionName).Count >= count, because);
return Task.CompletedTask;
}, TimeSpan.FromSeconds(20));
private async Task AwaitReconfigureCountAsync(int count, string because) =>
await AwaitAssertAsync(() =>
{
Assert.True(ReconfiguresFor(ConnectionName).Count >= count, because);
return Task.CompletedTask;
}, TimeSpan.FromSeconds(20));
// ── Rule 1: a definition row outranks an older flattened snapshot ──
/// <summary>
/// <b>THE row-89 reproduction, in its real (failover / restart) shape.</b> The store holds a
/// definition row pointing at the new endpoint — put there by a connection-only artifact
/// deploy and, on a replicated pair, delivered to this node by LocalDb CDC — while the
/// instance's own flattened config still carries the endpoint frozen in at ITS deploy time.
/// A singleton incarnation booting here rebuilds the entire DCL from those flattened
/// snapshots, so this is exactly what the peer node does the moment it takes over.
///
/// <para>
/// <b>Fails before the fix:</b> the create carries <c>old-plc</c>, because nothing in the
/// runtime ever read <c>data_connection_definitions</c> — the table had no production reader
/// at all, so the artifact deploy's change survived only inside the singleton incarnation
/// that applied it.
/// </para>
/// </summary>
[Fact]
public async Task DefinitionRow_WinsOverAnOlderFlattenedSnapshot_WhenTheSingletonBoots()
{
const string instanceName = "ArbitratedPump";
await StoreDefinitionAsync(DefinitionEndpoint);
await _storage.StoreDeployedConfigAsync(
instanceName, MakeConfigJson(instanceName), "dep-a", "rev-a", true);
CreateBootedDeploymentManager(sweepSeconds: 0);
await AwaitCreateCountAsync(1, "the boot load never created the connection");
Assert.Equal(DefinitionEndpoint, EndpointOf(CreatesFor(ConnectionName).Single()));
}
/// <summary>
/// <b>The ping-pong guard.</b> With the two sources disagreeing, the naive form of this fix —
/// a sweep pass that re-hashes the connection from the definitions table while the deploy
/// paths re-hash it from the flattened configs — flips the single per-name hash slot on every
/// tick, and each flip is another <see cref="CreateConnectionCommand"/>, i.e. the live
/// connection actor recreated and every subscription on it dropped, forever.
///
/// <para>
/// This passes both before and after the fix (before, because nothing read the definitions at
/// all) and is the reason the ensure paths resolve the WINNER before hashing rather than
/// guarding the flip after the fact.
/// </para>
/// </summary>
[Fact]
public async Task DisagreeingSources_DoNotPingPongTheConnection_AcrossManySweeps()
{
const string instanceName = "NoPingPongPump";
await StoreDefinitionAsync(DefinitionEndpoint);
await _storage.StoreDeployedConfigAsync(
instanceName, MakeConfigJson(instanceName), "dep-p", "rev-p", true);
CreateBootedDeploymentManager();
await AwaitCreateCountAsync(1, "the boot load never created the connection");
await Task.Delay(SeveralSweeps);
Assert.Single(CreatesFor(ConnectionName));
// Row 97 moved the cost of a flip: a ping-pong now shows up as a stream of RECONFIGURES,
// each of which drops the link and re-subscribes every tag. Assert on both, or the guard
// silently stops guarding the thing that would actually happen.
Assert.Empty(ReconfiguresFor(ConnectionName));
Assert.Empty(_removes);
}
// ── Rule 2: no definition row means the flattened config is the source, unchanged ──
/// <summary>
/// The deploy-only site: <c>data_connection_definitions</c> is EMPTY because nothing has ever
/// sent this site a system-wide artifact deploy, and every connection that matters lives
/// inline in the instance configs. Rule 2 says the flattened config is the source there — not
/// as a fallback, but as the correct answer — so this must behave exactly as it did before.
///
/// <para>Passes before and after: a deploy-only regression guard.</para>
/// </summary>
[Fact]
public async Task NoDefinitionRow_LeavesTheFlattenedConfigAuthoritative_AtBoot()
{
const string instanceName = "DeployOnlyPump";
await _storage.StoreDeployedConfigAsync(
instanceName, MakeConfigJson(instanceName), "dep-d", "rev-d", true);
CreateBootedDeploymentManager(sweepSeconds: 0);
await AwaitCreateCountAsync(1, "the boot load never created the connection");
Assert.Equal(FlattenedEndpoint, EndpointOf(CreatesFor(ConnectionName).Single()));
}
/// <summary>
/// The same rule on the POST-boot sweep path (register row 78's start, row 85's connection):
/// a config that appears in the store on its own starts an instance whose connection, with no
/// definition row anywhere, still comes from that instance's flattened config.
///
/// <para>Passes before and after: a deploy-only regression guard for the sweep path.</para>
/// </summary>
[Fact]
public async Task NoDefinitionRow_LeavesTheFlattenedConfigAuthoritative_OnTheSweepPath()
{
const string instanceName = "DeployOnlySweptPump";
CreateBootedDeploymentManager();
await _storage.StoreDeployedConfigAsync(
instanceName, MakeConfigJson(instanceName), "dep-ds", "rev-ds", true);
await AwaitCreateCountAsync(1, "the sweep never created the connection");
Assert.Equal(FlattenedEndpoint, EndpointOf(CreatesFor(ConnectionName).Single()));
}
// ── Row 89's live half: a definition arriving after boot converges on the next tick ──
/// <summary>
/// <b>A definition row appearing in the store while the node runs</b> — what LocalDb's CDC
/// applier does on the peer of a replicated pair when a connection-only artifact deploy lands
/// on the other node. There is no cross-node message for this and none is needed: the sweep
/// re-reads the table on the next tick, adopts the row into the arbitration overlay, and
/// pushes the new configuration for the connection the runtime already holds.
///
/// <para>
/// <b>Fails before the fix:</b> nothing is ever issued for the change at all, and no sweep
/// tick ever looks at the definitions table.
/// </para>
///
/// <para>
/// <b>Register row 97 changed the SHAPE of the second push.</b> The connection already exists
/// at this point, so the change goes out as a
/// <see cref="ReconfigureConnectionCommand"/> — a second create would be dropped by the DCL
/// manager's create-only guard and change nothing.
/// </para>
/// </summary>
[Fact]
public async Task DefinitionArrivingAfterBoot_IsAdoptedByTheNextSweepTick()
{
const string instanceName = "CdcDeliveredPump";
CreateBootedDeploymentManager();
// The instance starts first, from its own (older) flattened snapshot.
await _storage.StoreDeployedConfigAsync(
instanceName, MakeConfigJson(instanceName), "dep-c", "rev-c", true);
await AwaitCreateCountAsync(1, "the sweep never created the connection");
Assert.Equal(FlattenedEndpoint, EndpointOf(CreatesFor(ConnectionName).First()));
// Now the definition row lands — a bare storage write, the CDC applier's only side effect.
await StoreDefinitionAsync(DefinitionEndpoint);
await AwaitReconfigureCountAsync(1,
"a definition row that arrived by replication was never picked up: the live DCL kept "
+ "the endpoint frozen into an instance's flattened config (register row 89)");
Assert.Equal(DefinitionEndpoint, EndpointOf(ReconfiguresFor(ConnectionName).Last()));
// And it settles: the overlay now owns the name, so further ticks are silent.
await Task.Delay(SeveralSweeps);
Assert.Single(CreatesFor(ConnectionName));
Assert.Single(ReconfiguresFor(ConnectionName));
}
// ── Row 97: a changed configuration reaches the LIVE connection ──
/// <summary>
/// <b>THE row-97 reproduction on the sender side.</b> A connection the DCL already holds gets
/// a new endpoint. What must go out is a <see cref="ReconfigureConnectionCommand"/>, which the
/// live connection actor applies in place by swapping its adapter and re-issuing its
/// subscriptions.
///
/// <para>
/// <b>Fails before the fix:</b> the Deployment Manager re-issued a
/// <see cref="CreateConnectionCommand"/> and logged a Warning saying it would not take effect
/// — which was accurate, because <c>DataConnectionManagerActor.HandleCreateConnection</c>
/// warns and returns for a name it already holds. The hash cache recorded the change as
/// applied regardless, so a corrected endpoint sat unapplied until a node restart, a singleton
/// failover, or a remove-then-add.
/// </para>
/// </summary>
[Fact]
public async Task ChangedConfigurationForALiveConnection_IsSentAsAReconfigure_NotASecondCreate()
{
const string instanceName = "ReconfiguredPump";
CreateBootedDeploymentManager();
await _storage.StoreDeployedConfigAsync(
instanceName, MakeConfigJson(instanceName), "dep-rc1", "rev-rc1", true);
await AwaitCreateCountAsync(1, "the sweep never created the connection");
// The connection-only artifact deploy that corrects the endpoint.
await StoreDefinitionAsync(DefinitionEndpoint);
await AwaitReconfigureCountAsync(1,
"a changed configuration for an ALREADY-LIVE connection was not sent as a "
+ "ReconfigureConnectionCommand, so the live connection actor keeps serving the old "
+ "endpoint until it is rebuilt from scratch (register row 97)");
Assert.Equal(DefinitionEndpoint, EndpointOf(ReconfiguresFor(ConnectionName).Single()));
// A create would have been dropped by the manager's create-only guard — so the change must
// NOT have been shipped as one.
Assert.Single(CreatesFor(ConnectionName));
}
/// <summary>
/// The hash guard is unchanged by row 97: an UNCHANGED configuration still sends nothing at
/// all. This is what keeps a healthy site silent across every sweep tick, and it is the reason
/// a reconfigure — which drops the link and re-subscribes every tag — is safe to send on the
/// change edge.
/// </summary>
[Fact]
public async Task UnchangedConfiguration_SendsNeitherACreateNorAReconfigure()
{
const string instanceName = "SteadyPump";
await StoreDefinitionAsync(DefinitionEndpoint);
await _storage.StoreDeployedConfigAsync(
instanceName, MakeConfigJson(instanceName), "dep-st", "rev-st", true);
CreateBootedDeploymentManager();
await AwaitCreateCountAsync(1, "the boot load never created the connection");
// Re-write the SAME definition row several sweeps running, and start a second instance
// naming the same connection: neither is a configuration change.
await StoreDefinitionAsync(DefinitionEndpoint);
await _storage.StoreDeployedConfigAsync(
"SteadyPumpTwo", MakeConfigJson("SteadyPumpTwo"), "dep-st2", "rev-st2", true);
await Task.Delay(SeveralSweeps);
Assert.Single(CreatesFor(ConnectionName));
Assert.Empty(ReconfiguresFor(ConnectionName));
}
// ── Row 90: removal, and not being resurrected by a flattened snapshot ──
/// <summary>
/// <b>THE row-90 reproduction.</b> A definition row this incarnation had adopted disappears
/// from the store — a central delete, reconciled by an artifact deploy on the other node and
/// replicated here as a row deletion. The sweep can PROVE that removal (the store owned the
/// name a tick ago and does not now), so it stops the live connection actor; and the name is
/// tombstoned, so the next instance start whose flattened config still lists that connection
/// does not quietly bring it back.
///
/// <para>
/// <b>Fails before the fix</b> at the removal assert: nothing ever noticed, and the orphan
/// connection actor stayed live until the node restarted.
/// </para>
/// </summary>
[Fact]
public async Task DefinitionRowDisappearing_StopsTheConnection_AndItIsNotResurrected()
{
const string firstInstance = "RemovedConnPump";
const string secondInstance = "LaterPump";
await StoreDefinitionAsync(DefinitionEndpoint);
await _storage.StoreDeployedConfigAsync(
firstInstance, MakeConfigJson(firstInstance), "dep-r1", "rev-r1", true);
CreateBootedDeploymentManager();
await AwaitCreateCountAsync(1, "the boot load never created the connection");
// Central deleted the connection; the reconcile (or CDC) removes the row.
await _storage.DeleteDataConnectionDefinitionsExceptAsync([]);
await AwaitAssertAsync(() =>
{
Assert.True(
_removes.Any(r => r.ConnectionName == ConnectionName),
"the definition row vanished and the live DCL connection actor was left running "
+ "(register row 90)");
return Task.CompletedTask;
}, TimeSpan.FromSeconds(20));
var createsAfterRemoval = CreatesFor(ConnectionName).Count;
// A second instance whose flattened config still names the removed connection is started
// by the sweep. It must NOT resurrect a connection central has deleted.
await _storage.StoreDeployedConfigAsync(
secondInstance, MakeConfigJson(secondInstance), "dep-r2", "rev-r2", true);
await Task.Delay(SeveralSweeps);
Assert.Equal(createsAfterRemoval, CreatesFor(ConnectionName).Count);
}
/// <summary>
/// The same tombstone, reached through the path that actually removes connections today: an
/// artifact deploy whose complete data-connection set no longer contains the name, which the
/// site reconciles by deleting the row and stopping the actor.
///
/// <para>
/// <b>Fails before the fix</b> at the final assert: the removal evicted the hash slot but
/// left no memory of WHY, so the very next instance start re-created the connection from that
/// instance's flattened config — a centrally-deleted connection resurrected within one sweep
/// tick.
/// </para>
/// </summary>
[Fact]
public async Task ArtifactReconcileRemoval_IsNotResurrectedByALaterInstanceStart()
{
const string instanceName = "PostReconcilePump";
var dm = CreateBootedDeploymentManager();
// Artifact deploy 1 — the connection exists centrally.
dm.Tell(new DeployArtifactsCommand(
"artifact-1", null, null, null, null,
[new DataConnectionArtifact(ConnectionName, "OpcUa", EndpointJson(DefinitionEndpoint), null)],
null, DateTimeOffset.UtcNow));
Assert.True(ExpectMsg<ArtifactDeploymentResponse>(TimeSpan.FromSeconds(20)).Success);
// Artifact deploy 2 — central no longer has it, so the site reconciles it away.
dm.Tell(new DeployArtifactsCommand(
"artifact-2", null, null, null, null, [], null, DateTimeOffset.UtcNow));
Assert.True(ExpectMsg<ArtifactDeploymentResponse>(TimeSpan.FromSeconds(20)).Success);
await AwaitAssertAsync(() =>
{
Assert.True(
_removes.Any(r => r.ConnectionName == ConnectionName),
"the artifact reconcile did not stop the removed connection");
return Task.CompletedTask;
}, TimeSpan.FromSeconds(20));
var createsAfterRemoval = CreatesFor(ConnectionName).Count;
// An instance start naming the removed connection must not undo the removal.
await _storage.StoreDeployedConfigAsync(
instanceName, MakeConfigJson(instanceName), "dep-pr", "rev-pr", true);
await Task.Delay(SeveralSweeps);
Assert.Equal(createsAfterRemoval, CreatesFor(ConnectionName).Count);
}
/// <summary>
/// The tombstone must not be permanent: a definition row IS the store re-asserting ownership
/// of the name, so a connection central deletes and then re-adds is creatable again.
///
/// <para>
/// <b>Note what the re-add does and does not do.</b> The overlay is an arbiter, not a creation
/// source — clearing the tombstone does not by itself resurrect a connection actor, exactly as
/// a definition row for a name no instance references creates nothing. The connection comes
/// back with the next instance start, and it comes back at the RE-ADDED endpoint, which is
/// what proves both halves: the tombstone was cleared, and rule 1 still governs the winner.
/// This mirrors the removal's own long-standing semantic — a removed connection's attributes
/// stay bad quality until the referencing instance is redeployed.
/// </para>
/// </summary>
[Fact]
public async Task ReAddedDefinition_ClearsTheTombstone_SoTheNextInstanceStartRecreatesIt()
{
const string firstInstance = "ReAddedPump";
const string laterInstance = "ReAddedLaterPump";
await StoreDefinitionAsync(DefinitionEndpoint);
await _storage.StoreDeployedConfigAsync(
firstInstance, MakeConfigJson(firstInstance), "dep-ra", "rev-ra", true);
CreateBootedDeploymentManager();
await AwaitCreateCountAsync(1, "the boot load never created the connection");
await _storage.DeleteDataConnectionDefinitionsExceptAsync([]);
await AwaitAssertAsync(() =>
{
Assert.True(_removes.Any(r => r.ConnectionName == ConnectionName),
"the removal never happened");
return Task.CompletedTask;
}, TimeSpan.FromSeconds(20));
// Central re-adds it, at a third endpoint.
const string reAddedEndpoint = "opc.tcp://re-added-plc:4840";
await StoreDefinitionAsync(reAddedEndpoint);
// A later instance start is what actually wants a connection actor again.
await _storage.StoreDeployedConfigAsync(
laterInstance, MakeConfigJson(laterInstance), "dep-ral", "rev-ral", true);
await AwaitCreateCountAsync(2,
"the tombstone outlived the re-add, so a connection central had restored stayed dead");
Assert.Equal(reAddedEndpoint, EndpointOf(CreatesFor(ConnectionName).Last()));
}
/// <summary>
/// Deferred register row 101 — the stale-snapshot race, made deterministic. A sweep's store
/// read runs OFF the actor thread; one that starts before an artifact apply's store writes
/// can land AFTER <c>HandleApplyArtifactDataConnectionsToDcl</c> populated the overlay. Its
/// snapshot then lacks the just-applied rows, and the disappearance detection would tombstone
/// and STOP the just-created live connections — which nothing re-creates until an instance
/// redeploy, because the removal also evicts the hash slot the overlay's push is gated on.
///
/// <para>
/// The crafted <c>InstanceSweepLoaded</c> below IS that race: an empty snapshot stamped with
/// the pre-apply write generation, delivered after the apply. <b>Fails before the fix</b>:
/// without the generation guard the actor removes the connection it created a moment ago.
/// </para>
/// </summary>
[Fact]
public async Task StaleSweepSnapshot_RacingAnArtifactApply_DoesNotRemoveJustAppliedConnections()
{
// Sweep effectively disabled so the crafted message is the only sweep result in play.
var dm = CreateBootedDeploymentManager(sweepSeconds: 3600);
dm.Tell(new DeployArtifactsCommand(
"artifact-race-1", null, null, null, null,
[new DataConnectionArtifact(ConnectionName, "OpcUa", EndpointJson(DefinitionEndpoint), null)],
null, DateTimeOffset.UtcNow));
Assert.True(ExpectMsg<ArtifactDeploymentResponse>(TimeSpan.FromSeconds(20)).Success);
await AwaitCreateCountAsync(1, "the artifact apply never created the connection");
// Generation 0 = captured before the apply bumped it; empty list = the pre-write table.
dm.Tell(new DeploymentManagerActor.InstanceSweepLoaded(
[], null, ConnectionDefinitions: [], DefinitionsGenerationAtRead: 0));
await Task.Delay(TimeSpan.FromSeconds(2));
Assert.DoesNotContain(_removes, r => r.ConnectionName == ConnectionName);
}
/// <summary>
/// The row-101 guard must not blunt row 90's legitimate disappearance detection: the same
/// empty snapshot stamped with the CURRENT write generation is a real answer (the store
/// genuinely holds no rows any more) and still stops the connection.
/// </summary>
[Fact]
public async Task FreshEmptySweepSnapshot_StillRemovesDisappearedConnections()
{
var dm = CreateBootedDeploymentManager(sweepSeconds: 3600);
dm.Tell(new DeployArtifactsCommand(
"artifact-race-2", null, null, null, null,
[new DataConnectionArtifact(ConnectionName, "OpcUa", EndpointJson(DefinitionEndpoint), null)],
null, DateTimeOffset.UtcNow));
Assert.True(ExpectMsg<ArtifactDeploymentResponse>(TimeSpan.FromSeconds(20)).Success);
await AwaitCreateCountAsync(1, "the artifact apply never created the connection");
// Exactly one artifact apply has run, so the current write generation is 1.
dm.Tell(new DeploymentManagerActor.InstanceSweepLoaded(
[], null, ConnectionDefinitions: [], DefinitionsGenerationAtRead: 1));
await AwaitAssertAsync(() =>
{
Assert.Contains(_removes, r => r.ConnectionName == ConnectionName);
return Task.CompletedTask;
}, TimeSpan.FromSeconds(10));
}
/// <summary>
/// Stand-in for the DCL manager actor: records the two commands the Deployment Manager is the
/// sole sender of, and swallows everything else (Instance Actors send their tag subscribes
/// here too, and those are not what these tests assert on).
/// </summary>
private sealed class DclConnectionSpy : ReceiveActor
{
public DclConnectionSpy(
ConcurrentQueue<CreateConnectionCommand> creates,
ConcurrentQueue<ReconfigureConnectionCommand> reconfigures,
ConcurrentQueue<RemoveConnectionCommand> removes)
{
Receive<CreateConnectionCommand>(creates.Enqueue);
Receive<ReconfigureConnectionCommand>(reconfigures.Enqueue);
Receive<RemoveConnectionCommand>(removes.Enqueue);
ReceiveAny(_ => { });
}
}
}
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerMediumFindingsTests.cs
+16 -8
View File
@@ -166,12 +166,19 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable
/// <summary>
/// SiteRuntime-010: when a redeployment changes a connection's configuration
/// (here the failover retry count and endpoint), the Deployment Manager must
/// re-issue a <see cref="ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection.CreateConnectionCommand"/>
/// so the DCL adopts the new configuration rather than keeping the stale one.
/// (here the failover retry count and endpoint), the Deployment Manager must push the new
/// configuration to the DCL rather than leaving it serving the stale one.
///
/// <para>
/// <b>Register row 97 changed WHICH command that is.</b> It used to be a re-issued
/// <c>CreateConnectionCommand</c> — which the create-only manager silently dropped for a name
/// it already held, so this test proved a message was sent and nothing more. It is now a
/// <see cref="ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection.ReconfigureConnectionCommand"/>,
/// which the live connection actor actually applies in place.
/// </para>
/// </summary>
[Fact]
public async Task EnsureDclConnections_ConnectionConfigChanged_ReissuesCreateCommand()
public async Task EnsureDclConnections_ConnectionConfigChanged_ReissuesReconfigureCommand()
{
var storage = NewStorage(_localDb.Db);
await storage.InitializeAsync();
@@ -198,11 +205,12 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable
MakeConfigJsonWithConnection("ConnPump", "opc.tcp://host-b:4840", 7),
"admin", DateTimeOffset.UtcNow));
// The DCL must receive a fresh create command reflecting the new config.
var secondCreate = dcl.ExpectMsg<ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection.CreateConnectionCommand>(
// The DCL must receive a RECONFIGURE reflecting the new config — not a second create,
// which the create-only manager would drop for a name it already holds.
var reconfigure = dcl.ExpectMsg<ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection.ReconfigureConnectionCommand>(
TimeSpan.FromSeconds(10));
Assert.Equal("Conn1", secondCreate.ConnectionName);
Assert.Equal(7, secondCreate.FailoverRetryCount);
Assert.Equal("Conn1", reconfigure.ConnectionName);
Assert.Equal(7, reconfigure.FailoverRetryCount);
}
/// <summary>
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentWarmThenGateTests.cs
+20 -8
View File
@@ -29,13 +29,17 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors;
/// answer its deployer instead of leaving it to Ask-timeout, and commands for DIFFERENT
/// instances must not block each other.</para>
///
/// <para>Shares the <c>SiteScriptCompileCache</c> collection because the batch pre-warm test
/// asserts on that process-wide cache's hit counter.</para>
/// <para>The batch pre-warm test asserts on the compile cache's hit counter. As of register
/// row 100 that cache is owned by the <c>ScriptCompilationService</c> instance this fixture
/// builds — the one it threads into the Deployment Manager and, through it, into every Instance
/// Actor — so the counter is private to the test. The old
/// <c>[Collection("SiteScriptCompileCache")]</c> serialisation against the two
/// <c>Scripts/</c> test classes is therefore gone: nothing here is shared any more.</para>
/// </summary>
[Collection("SiteScriptCompileCache")]
public class DeploymentWarmThenGateTests : TestKit, IDisposable
{
private readonly SiteStorageService _storage;
private readonly SiteScriptCompileCache _compileCache;
private readonly ScriptCompilationService _compilationService;
private readonly SharedScriptLibrary _sharedScriptLibrary;
private readonly TestLocalDb _localDb;
@@ -45,8 +49,9 @@ public class DeploymentWarmThenGateTests : TestKit, IDisposable
_localDb = TestLocalDb.CreateTemp("dm-warm-gate-test");
_storage = new SiteStorageService(_localDb.Db, NullLogger<SiteStorageService>.Instance);
_storage.InitializeAsync().GetAwaiter().GetResult();
_compileCache = new SiteScriptCompileCache();
_compilationService = new ScriptCompilationService(
NullLogger<ScriptCompilationService>.Instance);
NullLogger<ScriptCompilationService>.Instance, _compileCache);
_sharedScriptLibrary = new SharedScriptLibrary(
_compilationService, NullLogger<SharedScriptLibrary>.Instance);
}
@@ -247,16 +252,23 @@ public class DeploymentWarmThenGateTests : TestKit, IDisposable
await _storage.StoreDeployedConfigAsync(
"BatchTwo", ConfigJson("BatchTwo", sharedCode), "d2", "h2", true);
SiteScriptCompileCache.Clear();
Assert.Equal(0, SiteScriptCompileCache.Hits);
// No Clear() and no baseline subtraction: this cache is the fixture's own and nothing
// has compiled into it yet (register rows 91/100 — the counter used to be a
// process-global that other test classes bumped in parallel, forcing a delta-over-
// baseline reading). Only the Deployment Manager below writes to it.
Assert.Equal(0, _compileCache.Hits);
CreateDeploymentManager();
AwaitAssert(() =>
{
// One compile, then repeated hits: the second pre-warm plus both PreStarts.
Assert.True(SiteScriptCompileCache.Hits >= 3,
$"expected the pre-warmed body to be served from cache, saw {SiteScriptCompileCache.Hits} hits");
Assert.True(_compileCache.Hits >= 3,
$"expected the pre-warmed body to be served from cache, saw {_compileCache.Hits} hits");
}, TimeSpan.FromSeconds(20));
// …and exactly one entry for the one shared body: the hits above were the SAME compiled
// script being reused, not several bodies each compiled once.
Assert.Equal(1, _compileCache.Count);
}
}
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptCompilationServiceTests.cs
+14 -12
View File
@@ -15,15 +15,20 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Scripts;
/// <c>System.Diagnostics.Process</c> is blocked under System.Diagnostics —
/// <c>Stopwatch</c> stays allowed. The real execution-path compile against
/// <c>ScriptGlobals</c> / <c>TriggerExpressionGlobals</c> is unchanged.
///
/// <para><b>Register row 100:</b> the compile cache is owned by the service instance, not
/// static, so this class's service — and therefore its cache — is private to each test. The
/// <c>[Collection("SiteScriptCompileCache")]</c> serialisation and the <c>Clear()</c> calls that
/// used to open the cache-sensitive tests are gone; <c>Hits</c> is now exact.</para>
/// </summary>
[Collection("SiteScriptCompileCache")]
public class ScriptCompilationServiceTests
{
private readonly SiteScriptCompileCache _cache = new();
private readonly ScriptCompilationService _service;
public ScriptCompilationServiceTests()
{
_service = new ScriptCompilationService(NullLogger<ScriptCompilationService>.Instance);
_service = new ScriptCompilationService(NullLogger<ScriptCompilationService>.Instance, _cache);
}
[Fact]
@@ -38,28 +43,27 @@ public class ScriptCompilationServiceTests
[Fact]
public void Compile_SameCodeTwice_SharesOneRoslynCompile()
{
SiteScriptCompileCache.Clear();
var r1 = _service.Compile("deploy-gate-copy", "return 1 + 1;");
var r2 = _service.Compile("prestart-copy", "return 1 + 1;");
Assert.True(r1.IsSuccess);
Assert.Same(r1.CompiledScript, r2.CompiledScript); // one compile, shared Script<T> (N4) — the definitive proof
// Hits is a process-global counter; other test classes in this assembly compile scripts
// concurrently (they are not in this serialized collection), so the exact post-Clear count
// is not deterministic under a full-assembly parallel run. The shared-Script<T> assertion
// above is the real proof of cache reuse; here we only require the second lookup registered
// a hit (>= 1) rather than pinning an exact global count.
Assert.True(SiteScriptCompileCache.Hits >= 1);
// The cache belongs to this service instance (register row 100), so the hit count is
// exactly the one the second Compile caused — it used to be a >= 1 lower bound on a
// process-global counter other test classes were bumping in parallel.
Assert.Equal(1, _cache.Hits);
Assert.Equal(1, _cache.Count);
}
[Fact]
public void Compile_ScriptAndTriggerExpression_DoNotCrossContaminate()
{
SiteScriptCompileCache.Clear();
var script = _service.Compile("s", "1 > 0");
var trigger = _service.CompileTriggerExpression("t", "1 > 0");
Assert.NotSame(script.CompiledScript, trigger.CompiledScript); // different globals surfaces
Assert.Equal(2, _cache.Count); // two entries, no cross-globals hit
Assert.Equal(0, _cache.Hits);
}
[Fact]
@@ -231,7 +235,6 @@ public class ScriptCompilationServiceTests
[Fact]
public void Compile_DistinctScripts_ShareOneMetadataReferenceSet_SoNativeMemoryDoesNotGrow()
{
SiteScriptCompileCache.Clear();
var first = _service.Compile("first", "return 1 + 1;");
var second = _service.Compile("second", "return 2 + 2;");
@@ -255,7 +258,6 @@ public class ScriptCompilationServiceTests
[Fact]
public void Compile_UsesProcessWideCachingMetadataResolver()
{
SiteScriptCompileCache.Clear();
var result = _service.Compile("resolver-pin", "return 41 + 1;");
Assert.True(result.IsSuccess);
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/SiteScriptCompileCacheTests.cs
+124 -54
View File
@@ -5,51 +5,131 @@ using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Scripts;
/// <summary>
/// Tests for the process-wide <see cref="SiteScriptCompileCache"/>. The cache is
/// static/process-wide state, so this class shares the "SiteScriptCompileCache" xunit
/// collection with <c>ScriptCompilationServiceTests</c> (no cross-class parallelism) and
/// calls <see cref="SiteScriptCompileCache.Clear"/> at the top of each test.
/// Tests for <see cref="SiteScriptCompileCache"/>.
/// </summary>
[Collection("SiteScriptCompileCache")]
/// <remarks>
/// <para>
/// <b>Deferred-work register row 100 — the cache is no longer static.</b> Every test here builds
/// its OWN <see cref="SiteScriptCompileCache"/>, so <see cref="SiteScriptCompileCache.Hits"/> and
/// <see cref="SiteScriptCompileCache.Count"/> count this test's compiles and nothing else. That
/// retires three workarounds this file used to carry:
/// </para>
/// <list type="bullet">
/// <item>the <c>[Collection("SiteScriptCompileCache")]</c> serialisation with
/// <c>ScriptCompilationServiceTests</c> and <c>DeploymentWarmThenGateTests</c> — those three
/// classes shared a mutable global and had to be kept off each other; they no longer touch any
/// shared state and run fully in parallel again;</item>
/// <item>the per-test GUID key token (register row 91) — no other class can reach this cache at
/// all, so keys need not be globally unique;</item>
/// <item>the one-sided counter bounds and the <c>ForeignEntrySlack = 512</c> allowance on
/// <c>Count</c> — foreign inserts are impossible now, so every assertion below is exact.</item>
/// </list>
/// <para>
/// The old escape hatch, <c>Clear()</c>, is gone from the type: it had no production caller and
/// existed only to reset the global between tests. Constructing a cache is the reset.
/// </para>
/// </remarks>
public class SiteScriptCompileCacheTests
{
/// <summary>The cache under test. Private to this test instance — xunit builds one per test.</summary>
private readonly SiteScriptCompileCache _cache = new();
private static ScriptCompilationResult Ok() =>
ScriptCompilationResult.Succeeded(CSharpScript.Create<object?>("1", ScriptOptions.Default, typeof(ScriptGlobals)));
[Fact]
public void GetOrAdd_SameCodeAndGlobals_ComputesOnce()
{
SiteScriptCompileCache.Clear();
var factoryCalls = 0;
ScriptCompilationResult Factory() { factoryCalls++; return Ok(); }
var r1 = SiteScriptCompileCache.GetOrAdd("return 1;", typeof(ScriptGlobals), Factory);
var r2 = SiteScriptCompileCache.GetOrAdd("return 1;", typeof(ScriptGlobals), Factory);
var r1 = _cache.GetOrAdd("return 1;", typeof(ScriptGlobals), Factory);
var r2 = _cache.GetOrAdd("return 1;", typeof(ScriptGlobals), Factory);
Assert.Equal(1, factoryCalls);
Assert.Same(r1, r2);
Assert.Equal(1, SiteScriptCompileCache.Hits);
Assert.Equal(1, _cache.Hits);
Assert.Equal(1, _cache.Count);
}
[Fact]
public void GetOrAdd_SameCode_DifferentGlobals_AreSeparateEntries()
{
SiteScriptCompileCache.Clear();
SiteScriptCompileCache.GetOrAdd("1 > 0", typeof(ScriptGlobals), Ok);
SiteScriptCompileCache.GetOrAdd("1 > 0", typeof(TriggerExpressionGlobals), Ok);
var scriptCompiles = 0;
var triggerCompiles = 0;
Assert.Equal(2, SiteScriptCompileCache.Count);
Assert.Equal(0, SiteScriptCompileCache.Hits); // no cross-globals hit
var script = _cache.GetOrAdd("1 > 0", typeof(ScriptGlobals),
() => { scriptCompiles++; return Ok(); });
var trigger = _cache.GetOrAdd("1 > 0", typeof(TriggerExpressionGlobals),
() => { triggerCompiles++; return Ok(); });
Assert.Equal(1, scriptCompiles);
Assert.Equal(1, triggerCompiles);
Assert.NotSame(script, trigger);
// Two distinct entries and no cross-globals hit — exact, because this cache is ours.
Assert.Equal(2, _cache.Count);
Assert.Equal(0, _cache.Hits);
// Each entry stays independently addressable afterwards.
Assert.Same(script, _cache.GetOrAdd("1 > 0", typeof(ScriptGlobals),
() => throw new InvalidOperationException("script entry was not cached")));
Assert.Same(trigger, _cache.GetOrAdd("1 > 0", typeof(TriggerExpressionGlobals),
() => throw new InvalidOperationException("trigger entry was not cached")));
}
/// <summary>
/// Register row 100: two caches are genuinely independent stores. This is the property the
/// whole change exists for — one test class's cache activity can no longer evict, serve, or
/// be counted by another's. It is the direct replacement for the old
/// <c>Clear()</c>-under-a-serialized-collection arrangement.
/// </summary>
[Fact]
public void SeparateCaches_DoNotShareEntriesOrCounters()
{
var other = new SiteScriptCompileCache();
var mine = _cache.GetOrAdd("return 1;", typeof(ScriptGlobals), Ok);
_cache.GetOrAdd("return 1;", typeof(ScriptGlobals), Ok); // a hit on OUR cache
// The other cache has neither the entry nor the hit.
Assert.Equal(0, other.Count);
Assert.Equal(0, other.Hits);
// Same key, other store: it must compute its own value, and get a distinct instance.
var theirCompiles = 0;
var theirs = other.GetOrAdd("return 1;", typeof(ScriptGlobals),
() => { theirCompiles++; return Ok(); });
Assert.Equal(1, theirCompiles);
Assert.NotSame(mine, theirs);
// …and filling the other cache past its bound — the closest thing left to the old
// wholesale Clear() — leaves ours completely untouched.
for (var i = 0; i <= SiteScriptCompileCache.MaxEntries; i++)
other.GetOrAdd($"return {i};", typeof(ScriptGlobals), Ok);
Assert.Equal(1, _cache.Count);
Assert.Equal(1, _cache.Hits);
Assert.Same(mine, _cache.GetOrAdd("return 1;", typeof(ScriptGlobals),
() => throw new InvalidOperationException("our entry was evicted by another cache")));
}
[Fact]
public void Overflow_StaysWithinTheBound()
{
SiteScriptCompileCache.Clear();
for (var i = 0; i <= SiteScriptCompileCache.MaxEntries; i++)
SiteScriptCompileCache.GetOrAdd($"return {i};", typeof(ScriptGlobals), Ok);
_cache.GetOrAdd($"return {i};", typeof(ScriptGlobals), Ok);
Assert.True(SiteScriptCompileCache.Count <= SiteScriptCompileCache.MaxEntries);
// MaxEntries + 1 distinct inserts, single-threaded: the sweep ran and the ceiling holds
// exactly (no foreign inserts can land here any more — register row 100).
Assert.True(_cache.Count <= SiteScriptCompileCache.MaxEntries,
$"cache grew past its bound: {_cache.Count} vs max {SiteScriptCompileCache.MaxEntries}");
var recomputed = 0;
for (var i = 0; i < 16; i++)
_cache.GetOrAdd($"return {i};", typeof(ScriptGlobals), () => { recomputed++; return Ok(); });
Assert.True(recomputed > 0, "no oldest-batch entry was evicted — the overflow sweep did not run");
}
// ── WP3.1: approximate LRU replaced the wholesale-Clear overflow cliff ──────────
@@ -62,46 +142,37 @@ public class SiteScriptCompileCacheTests
[Fact]
public void Overflow_EvictsOldestBatch_AndKeepsRecentlyTouchedEntries()
{
// Note: the cache is process-wide static and other test classes compile into it
// concurrently, so the assertions below are phrased as properties of the eviction
// policy (what survives, what does not, nothing is wiped) rather than as exact
// counts, which no test can own here.
SiteScriptCompileCache.Clear();
// Fill to the bound. Entries 1..N are inserted oldest-first.
for (var i = 0; i < SiteScriptCompileCache.MaxEntries; i++)
SiteScriptCompileCache.GetOrAdd($"return {i};", typeof(ScriptGlobals), Ok);
_cache.GetOrAdd($"return {i};", typeof(ScriptGlobals), Ok);
// Touch entry 0 so it carries the NEWEST access stamp despite being inserted first.
var hot = SiteScriptCompileCache.GetOrAdd("return 0;", typeof(ScriptGlobals), Ok);
var hitsAfterTouch = SiteScriptCompileCache.Hits;
var hot = _cache.GetOrAdd("return 0;", typeof(ScriptGlobals), Ok);
var hitsAfterTouch = _cache.Hits;
Assert.Equal(1, hitsAfterTouch); // exactly one hit so far: the touch
// Push well past the bound so at least one eviction sweep definitely runs.
for (var i = 0; i < 200; i++)
SiteScriptCompileCache.GetOrAdd($"return overflow{i};", typeof(ScriptGlobals), Ok);
_cache.GetOrAdd($"return overflow{i};", typeof(ScriptGlobals), Ok);
// Nothing was cleared wholesale: the cache is still most of the way full and the hit
// counter survived (Clear() would have reset it to 0).
Assert.True(SiteScriptCompileCache.Count > SiteScriptCompileCache.MaxEntries / 2,
$"cache collapsed to {SiteScriptCompileCache.Count} entries — this looks like a wholesale clear");
Assert.True(SiteScriptCompileCache.Hits >= hitsAfterTouch);
// counter survived (a wholesale clear used to reset it to 0).
Assert.True(_cache.Count > SiteScriptCompileCache.MaxEntries / 2,
$"cache collapsed to {_cache.Count} entries — this looks like a wholesale clear");
Assert.Equal(hitsAfterTouch, _cache.Hits); // 200 distinct inserts, none of them hits
// The touched entry survived the sweep — a hit, not a recompile. This is the LRU
// property: recency, not insertion order, decides what stays.
var hitsBefore = SiteScriptCompileCache.Hits;
var again = SiteScriptCompileCache.GetOrAdd("return 0;", typeof(ScriptGlobals),
var again = _cache.GetOrAdd("return 0;", typeof(ScriptGlobals),
() => throw new InvalidOperationException("hot entry was evicted — LRU is not keeping recently-used entries"));
Assert.Same(hot, again);
Assert.Equal(hitsBefore + 1, SiteScriptCompileCache.Hits);
Assert.Equal(hitsAfterTouch + 1, _cache.Hits);
// …and untouched old entries genuinely were evicted, so the sweep really ran.
var recomputed = 0;
for (var i = 1; i <= 16; i++)
{
var code = $"return {i};";
SiteScriptCompileCache.GetOrAdd(code, typeof(ScriptGlobals),
() => { recomputed++; return Ok(); });
}
_cache.GetOrAdd($"return {i};", typeof(ScriptGlobals), () => { recomputed++; return Ok(); });
Assert.True(recomputed > 0, "no old entry was evicted — the overflow sweep did not run");
}
@@ -113,26 +184,25 @@ public class SiteScriptCompileCacheTests
[Fact]
public async Task ConcurrentGetOrAddStorm_KeepsTheBound()
{
SiteScriptCompileCache.Clear();
const int perTask = 400;
var tasks = Enumerable.Range(0, 8).Select(t => Task.Run(() =>
const int tasks = 8;
await Task.WhenAll(Enumerable.Range(0, tasks).Select(t => Task.Run(() =>
{
for (var i = 0; i < perTask; i++)
SiteScriptCompileCache.GetOrAdd($"return {t}_{i};", typeof(ScriptGlobals), Ok);
}));
await Task.WhenAll(tasks);
_cache.GetOrAdd($"return {t}_{i};", typeof(ScriptGlobals), Ok);
})));
// A small transient overshoot is by design: threads that lose the double-checked
// eviction race still insert their own entry afterwards, so the ceiling is "bounded",
// not "never exceeded by one". The property under test is that 3200 distinct inserts
// do not accumulate — before WP3.1 this was a wholesale Clear(), and a per-entry
// eviction bug would show up here as unbounded growth, not as a handful of extra rows.
// (Other test classes compile into this process-wide cache concurrently, which is the
// other reason the bound is asserted with slack rather than exactly.)
Assert.True(SiteScriptCompileCache.Count <= SiteScriptCompileCache.MaxEntries + 512,
$"cache grew past its bound: {SiteScriptCompileCache.Count} vs max {SiteScriptCompileCache.MaxEntries} " +
$"after {8 * perTask} distinct inserts");
// not "never exceeded by one". The slack is bounded by the number of racing threads
// (register row 100 — it used to be 512, to absorb inserts from other test classes
// sharing the process-global cache; nothing foreign can reach this instance).
// The property under test is that 3200 distinct inserts do not accumulate — before
// WP3.1 this was a wholesale clear, and a per-entry eviction bug would show up here as
// unbounded growth, not as a handful of extra rows.
Assert.True(_cache.Count <= SiteScriptCompileCache.MaxEntries + tasks,
$"cache grew past its bound: {_cache.Count} vs max {SiteScriptCompileCache.MaxEntries} " +
$"(+{tasks} racing-insert slack) after {tasks * perTask} distinct inserts");
}
}
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/SiteRuntimeServiceRegistrationTests.cs
+36
View File
@@ -0,0 +1,36 @@
using Microsoft.Extensions.DependencyInjection;
using ZB.MOM.WW.ScadaBridge.SiteRuntime;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests;
/// <summary>
/// Pins the DI lifetimes that carry a behavioural contract rather than a convention.
/// </summary>
public class SiteRuntimeServiceRegistrationTests
{
/// <summary>
/// Register row 100. <c>ScriptCompilationService</c> owns the
/// <c>SiteScriptCompileCache</c>, so "one Roslyn compile per script body per process"
/// (N4 / WP3.1's warm-then-gate deploy path) is precisely the statement that DI hands
/// out ONE of these per host. The cache stopped being <c>static</c> when row 100 removed
/// the process-global state tests were clobbering; this registration is what replaced
/// the static as the guarantee, so it is pinned here.
///
/// <para>Downgrading this to Scoped or Transient would give every consumer — the deploy
/// gate, each Instance Actor's PreStart, SharedScriptLibrary — its own empty cache and
/// silently reintroduce the recompile storms WP3.1 removed, with no functional test
/// failing. This assertion is the only thing that would notice.</para>
/// </summary>
[Fact]
public void AddSiteRuntime_RegistersScriptCompilationServiceAsSingleton_SoOneCompileCacheExistsPerProcess()
{
var services = new ServiceCollection();
services.AddSiteRuntime();
var descriptor = Assert.Single(
services, d => d.ServiceType == typeof(ScriptCompilationService));
Assert.Equal(ServiceLifetime.Singleton, descriptor.Lifetime);
}
}