dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: dohertj2/ScadaBridge:d18f344e66c4e13eaedae82d986ee2d9b95ee6b1
Branches Tags
dohertj2/ScadaBridge:main
...
compare: dohertj2/ScadaBridge:9b67c7cb4b4041934ab4a07d1abc40223880df05
Branches Tags
dohertj2/ScadaBridge:main

19 Commits
d18f344e66 ... 9b67c7cb4b

Author SHA1 Message Date
Joseph Doherty 9b67c7cb4b docs(plans)+docs(register): 2026-08-19 Medium-findings remediation plan (all 17 tasks executed); reopen+re-close register rows 95/99 with the P8-6/P6-3 defect proofs; add row 103 (deferred P6-6 lesser instances); CLAUDE.md — bulk deploy skips Disabled, LocalDb 0.3.2 2026-08-19 15:07:24 -04:00
Joseph Doherty 8569a493f6 deps(localdb): consume 0.3.2 — cap-prune session fault + sync keepalives + newest-wins stream guard (arch-review 2026-08-19 P4-1 + P4-3) 2026-08-19 15:07:24 -04:00
Joseph Doherty d9d72f1bc5 test(communication)+test(sitecallaudit): cap parallelism, delta/collection-serialize gauge asserts, replace sleep-then-assert, stop leaked 100ms tickers (arch-review 2026-08-19 P8-5 + P8-6, register rows 40/82/95/96/102 patterns) 2026-08-19 15:03:49 -04:00
Joseph Doherty 9a3f006721 fix(management)+fix(cli): neutralize CSV formula injection + align export columns; deadline + Ctrl-C on audit export streaming (arch-review 2026-08-19 P6-7 + P6-8) 
P6-7 (CWE-1236): AuditEndpoints CSV export quoted cells on comma/quote/CR/LF
but never neutralized a leading =, +, -, @, TAB, or CR — inbound-API-caller
text landing in Target/ErrorMessage/RequestSummary/ResponseSummary/Extra
could execute as a formula/DDE trigger when opened in Excel. AppendField now
prefixes a literal single quote on free-text columns only (never the
GUID/enum/numeric/date/bool columns the exporter formats itself), applied
before RFC 4180 quoting. The CSV was also missing ExecutionId/
ParentExecutionId/SourceNode that the JSONL export already carried; both are
now 24 columns in the same AuditExportRow order. A second, independently
maintained CSV writer (CentralUI's AuditLogExportService, feeding the Audit
Log page's bulk-export button) had the identical gap and got the identical
fix + column alignment.

P6-8: `audit export` was the one CLI HTTP path with no deadline at all — a
residual gap from the ManagementHttpClient timeout-honesty fix, which made
HttpClient.Timeout infinite and the per-call token the single deadline
everywhere except this streamed GET. RunExportAsync now takes a headers-
received deadline (CancelAfter, reusing ManagementHttpClient.
DefaultConnectTimeout) around the initial GetAsync, an inactivity deadline
around the streaming copy (rearmed on every successful read, so large exports
are bounded by silence, not total size), and an external CancellationToken
wired to Console.CancelKeyPress in AuditCommands so Ctrl-C cancels promptly.
A stalled/faulted stream now produces a friendly error noting the partial
output file instead of hanging or dumping a stack trace.
 2026-08-19 15:01:45 -04:00
Joseph Doherty 50a77a8170 fix(transport): carry ExternalSystem TimeoutSeconds through bundles + diff (arch-review 2026-08-19 P7-1, row-27 shape) 
TimeoutSeconds is the per-system HTTP timeout ExternalSystemClient reads at
call time and is deployed to sites (proto field 6), but Transport bundles
silently dropped it: the DTO never carried it, export never wrote it, the
importer's Add path reset it to 0, the Overwrite path left the target's
stale value untouched, and ArtifactDiff never compared it (so a bundle
whose only change was a tuned timeout previewed as Identical).

Adds TimeoutSeconds as an additive-optional trailing field on
ExternalSystemDto (schemaVersion stays 1.1, bundleFormatVersion unchanged)
and threads it through EntitySerializer (export + import), BundleImporter
(Add and Overwrite), and ArtifactDiff.CompareExternalSystem. Old-bundle
semantics pinned: an absent field (null) falls back to the entity default
on Add and leaves the target's existing value untouched on Overwrite/diff.

Beefed up the RoundTripEquivalenceTests external-system fixture with a
non-default TimeoutSeconds so the reflection round-trip guard actually
exercises the field instead of comparing default against default.

Swept the rest of the ExternalSystemDto shape against
ExternalSystemDefinition field-by-field; no other operative field is
missing.
 2026-08-19 14:58:17 -04:00
Joseph Doherty 013257ec7c fix(centralui): per-IP spray ceiling on LoginThrottle (arch-review 2026-08-19 P6-9) 
LoginThrottle keyed failures only as {username}|{ip}, so a password spray
from one IP against many distinct usernames never accumulated
MaxLoginFailuresPerWindow failures on any single pair and was never
throttled, despite the class doc claiming spray protection.

Add a secondary per-IP counter (SecurityOptions.PerIpMaxFailures, default
25, validated strictly greater than MaxLoginFailuresPerWindow) that shares
the existing fixed window/lockout machinery but tracks failures by IP
alone. IsLockedOut now refuses a bind when either the per-pair or the
per-IP counter is locked. RecordSuccess clears only the per-pair counter
(never the per-IP one), so a spray that stumbles onto one valid account
cannot use that success to reset its IP budget and keep spraying; window
expiry is the only relief for the per-IP counter. Per-pair behavior for
normal (non-spray) flows is unchanged.

Updates the class doc and Component-Security.md to describe the two-tier
model instead of the single-tier spray claim.
 2026-08-19 14:54:57 -04:00
Joseph Doherty 561582747e fix(centralui): stop debug-stream sessions abandoned mid-start — dispose-vs-connect race (arch-review 2026-08-19 P6-6) 
A debug-stream session created after the consumer had already gone away was
leaked permanently. Both consumers awaited StartStreamAsync (up to 30s) and only
then published the session; a circuit teardown / client disconnect inside that
window ran the release path first, found nothing registered and stopped nothing.
The session stayed registered in the singleton DebugStreamService, whose
keepalive renews registered sessions forever (the 5-minute orphan reaper only
collects UNregistered ones), pinning the site gRPC subscription, the
DebugStreamBridgeActor and the dead consumer's object graph until process
restart.

Both call sites now publish the session and re-check liveness under a single
monitor, so exactly one side tears it down and neither can miss it:

- DebugView.Connect/Dispose/Disconnect share a _sessionLock; Dispose sets
  _disposed and takes the session in one step, Connect publishes only if not
  disposed and otherwise stops the session it just created. Dispose can run off
  the renderer's thread, so the Blazor sync context is not relied on. Disconnect
  and the onTerminated callback go through the same take-once helper, which also
  makes a repeated Dispose stop-once instead of stop-again.
- DebugStreamHub uses the per-connection Items monitor: OnDisconnectedAsync marks
  the connection gone before releasing, SubscribeInstance publishes only if
  neither that flag nor ConnectionAborted is set, and UnsubscribeInstance takes
  the id atomically.

DebugStreamService.StartStreamAsync/StopStream become virtual purely as a test
seam (no behaviour change, keepalive/orphan contract untouched) so the race can
be driven deterministically by a gated stub.

Tests: DebugViewConnectDisposeRaceTests (3) + DebugStreamHubSubscribeRaceTests
(3). Verified fail-unfixed: with the two source fixes stashed, both
dispose-mid-start tests report zero StopStream calls and the DebugView
double-dispose test reports two.
 2026-08-19 14:53:01 -04:00
Joseph Doherty 5cbc143b01 fix(deploymentmanager): bulk deploy skips Disabled + reconcile circuit-breaker + probe captures pre-flatten versions (arch-review 2026-08-19 P7-2 + P7-3 + P7-4) 2026-08-19 14:52:33 -04:00
Joseph Doherty b68f553f1e fix(host): h2 keepalive on site gRPC listener (arch-review 2026-08-19 P4-3) 2026-08-19 14:50:45 -04:00
Joseph Doherty e63ffd90ee fix(siteruntime): stamp the ensure-started heal with the store-write generation + order the redeploy override reset ahead of the successor (arch-review 2026-08-19 P1-1 + P1-2) 
P1-1 — the row-67 ensure-started self-heal re-read the durable row off-thread but
captured no _storeWriteGeneration and never ran IsStoreSnapshotStale, unlike the boot
load and the reconcile sweep. A Disable or Delete committing inside that window left the
result carrying an enabled config that passed every guard (neither command sets any of
the three pipeline markers ShouldAttemptEnsureStart re-checks), so the heal started an
instance the operator had just stopped — PERMANENTLY, since the sweep iterates store rows
only and has no runtime→store stop leg. The generation is now captured at dispatch,
threaded on EnsureInstanceStartedResult and checked on landing, and the comment claiming
the stored row already covered this is corrected. The heal reads through _configLoader,
the same seam the boot load and sweep use (identical production call), so a test can hold
the read open across a committed disable/delete.

P1-2 — the "static overrides reset on redeployment" invariant was racy in both
directions. ApplyDeployment created the successor Instance Actor first and only then
dispatched the clear, queued behind a full config-JSON write, so the successor's PreStart
override load routinely won and applied the PREDECESSOR's overrides — keys absent from
the new config included — to in-memory state for the actor's lifetime while the disk
ended cleared; a later restart then silently flipped the values. The clear is now
dispatched BEFORE CreateInstanceActor and handed to the actor as a gate its override load
chains behind; the persistence task joins the same task, so a failed clear still fails the
deploy exactly as before. Leg 2: the predecessor's PostStop flush was fire-and-forget and
could commit up to one coalescing window AFTER the clear (the Deployment Manager gates the
clear on Terminated), durably resurrecting overrides — PostStop now waits for it, bounded
at 5s with a Warning on overrun.

Tests: new RedeployStaticOverrideResetTests (5) pins both legs plus the gate mechanism and
first-deploy behaviour; three new cases in ReconcileStartsInstanceTests pin the discarded
disable/delete heals and that an unraced heal still starts. InstanceActorTests'
StaticOverride_ResetOnRedeployment comment claimed to simulate an ordering the production
path did not have — corrected.
 2026-08-19 14:49:24 -04:00
Joseph Doherty 64e6ba70e8 fix(management): site-scope ListDataConnections + hoist scope checks above resolve/status oracles (arch-review 2026-08-19 P6-4 + P6-5) 
P6-4: HandleListDataConnections took no user at all and returned EVERY
site's connections to any authenticated caller, nullifying the row-98
fail-closed gate on HandleGetDataConnection — Get denied exactly what List
handed over. DataConnectionPublicShape already scrubs secrets, so the leak
was site topology plus a connection-id existence oracle, but it still
defeated scoping. List now takes the AuthenticatedUser: an explicit SiteId
goes through EnforceSiteScope (the ListAreas shape), and an unfiltered list
from a site-scoped non-Administrator caller is narrowed to their permitted
sites (the ListSites shape). GetRequiredRoles is untouched — scoping is not
a role gate, so the frozen 138-command authz matrix is unaffected.

The Central UI Design > Connections page bypasses the actor entirely
(ISiteRepository.GetAllDataConnectionsAsync direct) and IS reachable by a
scoped user: it is gated RequireDesign, and SiteScopeService explicitly does
NOT exempt Designer (register row 86). It now filters sites through
SiteScope.FilterSitesAsync and keeps only their connections, mirroring
EventLogs/AlarmSummary.

P6-5: four handlers resolved their target — and on approve/reject reported
its status — BEFORE the scope check, so an out-of-scope caller could tell a
real id from a fake one, and read a secured write's lifecycle status, from
the error shape alone. The scope check is hoisted above the not-found and
status checks in HandleSubmitSecuredWrite, HandleApproveSecuredWrite,
HandleRejectSecuredWrite and HandleDebugSnapshot, with the unresolvable case
routed through the existing DenyUnresolvedTargetIfScoped so the denial is
identical whether or not the target exists. System-wide callers keep their
ordinary not-found/status answers, and the P6-1 CAS semantics on
approve/reject are unchanged (only the checks above the CAS moved).

Tests: 8 new ManagementService tests + 2 CentralUI bUnit tests, all verified
to FAIL against the unfixed handlers (src reverted, tests re-run) and pass
after — including the unscoped/Administrator control cases proving behaviour
for system-wide callers is unchanged. Full ManagementService.Tests 545 green,
CentralUI.Tests 1073 green.
 2026-08-19 14:49:24 -04:00
Joseph Doherty d0a748e632 fix(communication)+fix(siteruntime): carry DetachedScriptThreads/ScriptRunShedCount/SiteStreamAlarmDropCount across the health wire; restore drained counters on failed send (arch-review 2026-08-19 P8-1) 2026-08-19 14:49:24 -04:00
Joseph Doherty eca0ab768d fix(auditlog)+fix(configdb): budget the ingest fallback loop; index + varchar-type the SiteCalls list query (arch-review 2026-08-19 P5-1 + P5-2) 
P5-1: AuditLogIngestActor.OnCachedTelemetryAsync's per-entry fallback loop
(used when the cached-telemetry batch fast path fails) ran with no
budget/cancellation, unlike OnIngestAsync's fallback. A slow/wedged DB could
freeze the central ingest singleton's mailbox for minutes. The loop now runs
under its own fresh IngestFallbackBudget CancellationTokenSource (mirroring
OnIngestAsync), passes the token into BeginTransactionAsync/
InsertIfNotExistsAsync/UpsertAsync, and stops early on budget exhaustion,
logging how many entries were left for the periodic reconciliation pull to
pick up.

P5-2: SiteCallAuditRepository.QueryAsync ordered by
(CreatedAtUtc DESC, TrackedOperationId DESC) with no index leading
unconditionally on CreatedAtUtc, forcing a full scan+sort on an unfiltered
page. Added IX_SiteCalls_Created_TrackedOperationId (migration
AddSiteCallsListSortIndex + matching hand-written production-parity script
under docs/plans/sql/). Also switched the query from FromSqlInterpolated
(which binds string filters as nvarchar(4000)) to FromSqlRaw with explicitly
SqlDbType.VarChar-typed parameters, matching the SiteCalls columns'
IsUnicode(false) declaration and the write path's existing Param(...)
pattern, so seeks — including the keyset predicate — are no longer blocked
by an implicit conversion.

Tests: new AuditLogIngestActorCombinedTelemetryTests case proves the
fallback budget bounds the wait (hanging repo stub, ~5s not indefinite),
that remaining entries are skipped, and that a non-default/cancellable
token reaches the repository call; verified fail-unfixed by reverting the
actor change (test times out at its 20s bound). Existing
SiteCallAuditRepositoryTests QueryAsync/keyset/stuck-cutoff cases (real
MSSQL) confirm the raw-SQL rewrite still returns correct pages.
 2026-08-19 14:49:24 -04:00
Joseph Doherty 6c11362d9f fix(auditlog)+fix(siteeventlog): complete audit write tasks only after commit; run event-log retention on standby (arch-review 2026-08-19 P4-2 + P4-4) 
P4-2: SqliteAuditWriter.FlushBatch completed each pending row's Task
(TrySetResult) inside the per-row insert loop, before transaction.Commit()
ran. A later failure in the same batch (another row's insert throwing, or
the commit itself failing) rolled the whole transaction back, but the
earlier rows' Tasks had already completed successfully -- the caller and
FallbackAuditWriter (whose ring-buffer fallback exists precisely for this
outage) both saw false success while the rows were actually gone from the
database; the outer catch's TrySetException became a silent no-op against
an already-completed TaskCompletionSource. Every row's Completion is now
deferred and only resolved (TrySetResult for the whole batch, or
TrySetException for the whole batch) after transaction.Commit() succeeds
or the transaction rolls back, mirroring SiteEventLogger.FlushBatch.

P4-4: EventLogPurgeService gated the daily site_events purge on the
active-node check, early-exiting on the standby, on the stale premise that
"the standby receives no writes". DataConnectionActor runs on both site
nodes and writes site_events regardless of active/standby status (register
rows 89/90/97), so an unreplicated standby (the rig's site-b/site-c
posture) grew its site_events file forever with no retention and no cap.
The gate is removed; every node now purges its own file every tick. Safe
on a replicated pair because the retention/cap deletes are CDC-captured
and converge under LWW regardless of which node issues them; required on
an unreplicated pair because each node's file is independent. The
active-node check is retained only to annotate the purge tick's log line.

Tests: new SqliteAuditWriterCommitFailureTests forces a genuine
non-swallowed mid-batch INSERT failure via a real SQLite trigger + custom
scalar function on the writer's own connection (reflection seam), proving
(a) a row whose own insert succeeded still faults when a later row in the
same batch fails, and (b) FallbackAuditWriter rings every faulted row and
increments the failure counter for each -- both assertions fail against
the pre-fix code (proof-of-failure verified by temporarily reverting the
fix). EventLogPurgeServiceTests' standby case is inverted to assert the
purge now runs (not skips) on a standby node.
 2026-08-19 14:49:24 -04:00
Joseph Doherty 26ab6f8f09 fix(dcl): generation-fence ConnectResult + roll back failed OPC UA subscribe batches (arch-review 2026-08-19 P2-3 + P2-4) 
P2-3. ConnectResult was the last piped completion with no adapter generation on
it, so a connect launched against generation N and answered after the adapter had
been swapped (failover, or the row-97 in-place reconfigure) was applied to its
REPLACEMENT. A stale success drove ReSubscribeAll + BecomeConnected against an
adapter that had never connected, and the replacement's own result then landed in
the Connected state where nothing handles it. With a bad new endpoint the batch
subscribes failed at connection level with a log line only, no AdapterDisconnected
ever fired, and the idempotent-reconfigure guard refused a re-send of the same
configuration — only a restart recovered the connection. A stale FAILURE was as
wrong the other way: it counted toward failover and re-armed the reconnect timer
underneath the attempt genuinely in flight.

HandleAttemptConnect now stamps the generation and both handlers open with the
same guard the subscribe leg got in c4d3c8b8, dropping a mismatched result whole.
The adapter instance is deliberately NOT carried: unlike a subscribe completion a
connect result holds no adapter-scoped handles to release, and SwapAdapter has
already disposed the outgoing adapter. Dropping cannot wedge the machine — every
generation bump (CountFailureAndMaybeFailover, BecomeReconnecting's failover
branch, HandleReconfigure through it) arms an AttemptConnect for the new
generation, and the state stays Connecting/Reconnecting until that answers.

P2-4. CreateSubscriptionsCoreAsync registers each monitored item as it places it
(_callbacks, _itemShard, shard budget, the SDK subscription's item list) and only
then issues ApplyChanges, so a throw there propagated with every registration
still in place and nothing holding a handle to release them. The actor reads the
throw as a whole-chunk fault and re-issues the SAME batch on its backoff timer, so
each probe grew the bookkeeping again. Placement + apply moved to
PlaceAndApplyAsync behind a try/catch that unwinds via the existing
ReleaseFailedItem (per-item try, so one failure neither stops the rest nor
replaces the original fault); no ApplyChanges is re-issued, and items a partial
apply did reach the server with are reclaimed when the shard and session are torn
down on reconnect.

That reconnect is the other half: IsConnectionLevelFailure knew nothing about
ServiceResultException, which is how the SDK reports a dead session — so a dead
session classified as a per-tag resolution failure for the whole chunk. It now
also matches a narrow set of session/channel status codes (BadSessionIdInvalid,
BadSessionClosed, BadSessionNotActivated, BadSecureChannelClosed,
BadSecureChannelIdInvalid, BadServerNotConnected, BadConnectionClosed,
BadCommunicationError, BadNotConnected). Per-node codes such as BadNodeIdUnknown
stay per-tag — that is exactly the device-still-booting case the resolution timer
exists for.

Tests: DataConnectionStaleConnectResultTests (stale success neither connects nor
re-subscribes against the new adapter; stale failure does not disturb the current
attempt; the current generation's result is still processed after a stale drop) —
all 3 fail with the guards removed. DataConnectionOpcUaFaultClassificationTests
(bad-session drives reconnect — fails unfixed; bad-node-id stays per-tag).
RealOpcUaClientBatchRollbackTests (no residual registrations after an ApplyChanges
throw; repeated failed batches do not grow the bookkeeping) — both fail with the
rollback removed. Full DCL suite 342/342 green.
 2026-08-19 14:49:24 -04:00
Joseph Doherty e2d149089c infra(docker-env2)+infra(docker): init:true + secrets posture parity for env2; regen-proto EXIT trap (arch-review 2026-08-19 P8-2 + P8-3 + P8-4) 2026-08-19 14:49:24 -04:00
Joseph Doherty 412406e7c5 fix(communication): cancel-only CTS replacement, deferred channel disposal, single-endpoint debug stream (arch-review 2026-08-19 P3-1 + P3-2 + P3-3) 2026-08-19 14:49:24 -04:00
Joseph Doherty fd4cb30273 fix(siteruntime): arm stuck-script watchdog + health/summary on alarm runs (arch-review 2026-08-19 P1-3) 
Alarm on-trigger runs (ScriptRunLauncher.RunAlarmScriptAsync) bypassed the
stuck-script watchdog, the script-error health counter, and
ScriptRunSummaryRecorder entirely, even though they run on the same bounded
ScriptExecutionScheduler pool as instance scripts. A wedged alarm body cost a
pool thread forever with no site event and no detach/replace, and a timed-out
or failed on-trigger run was invisible to health monitoring and the run
summary.

Bring the alarm path to parity with the instance-script path (RunScriptAsync):
capture the worker slot/run stamp in the first synchronous segment and arm
ArmStuckScriptWatchdog with the same detach-and-replace semantics and gauge;
resolve ScriptRunSummaryRecorder/ISiteEventLogger from the same DI provider and
thread RecordStarted/RecordCompleted/RecordFailed/RecordTimedOut; increment
ISiteHealthCollector.IncrementScriptError on both the timeout and general
failure catch blocks. LaunchAlarmScript/RunAlarmScriptAsync now take
healthCollector and serviceProvider parameters, passed from AlarmActor's
existing _healthCollector/_serviceProvider fields. No change to the
enqueue-anchored deadline, the shed cap, or "a stop does NOT cancel an
in-flight run" semantics.

Adds AlarmScriptWatchdogTests (previously zero coverage of LaunchAlarmScript):
wedged-worker detach + replacement, timeout-at-dequeue health counter
increment, and successful-completion summary recording. All three fail
against the pre-fix code (verified via stash) and pass after.
 2026-08-19 14:49:24 -04:00
Joseph Doherty 91fcb5b951 fix(commons): reject comma-joined enum tokens in strict audit query parsers (arch-review 2026-08-19 P6-3, register row 99 defect) 2026-08-19 14:49:24 -04:00
107 changed files with 9148 additions and 716 deletions
Expand all files Collapse all files
CLAUDE.md
+2 -1
View File
@@ -92,6 +92,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
- **Batching is by BYTE BUDGET as of LocalDb 0.2.0** — `LocalDb:Replication:MaxBatchBytes` (default **2 MB**, sized under the 4 MB gRPC cap) bounds a delta/snapshot message by summed serialized size via a per-message split in `SyncSession.PumpLoopAsync`, with `MaxBatchSize` demoted to a secondary row cap; a single row over budget is sent alone rather than stalling the stream. **The rig's old `MaxBatchSize = 16` pin (a hand-computed byte-budget proxy: ~70 KB worst-case `config_json` x the 500 default is ~35 MB) is retired, but `MaxBatchSize` is NOT fully redundant with `MaxBatchBytes`** — it also bounds the separate DB READ page in `OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer`, which materializes the whole page into memory *before* the byte-budget split runs, so an unset (500-default) `MaxBatchSize` still lets a reconnect drain transiently allocate ~35 MB per read even though every wire message stays under `MaxBatchBytes` (arch-review adversarial finding F2). Both site-a nodes on `docker/` therefore pin an explicit `"MaxBatchSize": 64` to bound that transient allocation, while `MaxBatchBytes` stays unset (its 2 MB default) to bound the wire message; site-b/site-c stay unreplicated so the key doesn't apply there.
- **CDC registration is conditional, and both directions self-heal at boot** (arch-review WP1.3 + WP3.3, `Host/SiteLocalDbSetup.cs`). Capture triggers are installed only when this node has replication configured — `PeerAddress` **or** `ApiKey`, an OR because only the dialling half sets `PeerAddress` while the passive half carries the key alone. An unreplicated node calls **`DeregisterReplicated`** on all ten tables at boot, dropping triggers an earlier build left behind and pruning their oplog/row-version rows (idempotent; logs once at Information when something was actually cleaned). A replicating node registers with **`baselineExistingRows: true`**, which seeds `__localdb_row_version` for pre-existing rows at the LWW floor (HLC `0`, this node's id) and flags a snapshot resync — so turning replication ON for a site that has been running without it now converges on the rows already in the file instead of only on writes made after the restart. Deregistration must be symmetric (the handshake compares registered-table digests fail-closed), so replication is a both-nodes-together change in either direction. LocalDb 0.2.0 also makes backlog depth O(1) and drops the unused `__localdb_oplog_hlc` index; the on-disk bookkeeping schema goes to v2, upgraded in place on open, with no wire change (0.1.x peers still sync).
- **LocalDb 0.3.0 (2026-08-15) makes that snapshot flag PER-TABLE** — the residual 0.2.1 explicitly deferred. On-disk bookkeeping schema goes to **v3** (a new `__localdb_snapshot_state` table, upgraded in place on open); still wire-compatible, and `lib_schema_version` deliberately stays `1` because the handshake compares it fail-closed for *equality*. The extension is negotiated by **capability** (`Handshake.supports_partial_snapshot`, `SnapshotBegin.tables` where **empty == all tables**), so a peer that does not advertise it is served a full snapshot — a scoped one would make it advance `last_applied_remote_seq` past deltas it never received. For the same reason **a partial snapshot moves no watermark on either side**; the overlapping deltas are still pumped and absorbed by LWW. Snapshot debt now clears on the receiver's new `SnapshotAck` rather than on enqueue, so a crash mid-snapshot re-streams the same scope. **`SiteLocalDbSetup.cs` needed no restructuring** — it already calls `RegisterReplicated` once per table in a loop, which is exactly what per-table flagging keys off. The consequence to know: **adding an eleventh table to `ReplicatedTables` on an already-replicating site now snapshots that one table**, where through 0.2.x the same edit re-streamed all eleven in full in both directions. First-boot behaviour is unchanged (all ten seed at once ⇒ the flagged set is every registered table ⇒ an ordinary full snapshot), and so is upgrading the rig in place (the ten tables are already ledgered, nothing seeds, nothing is flagged). A future "register the whole list in one call" API would silently give the coarse behaviour back.
- **LocalDb 0.3.2 (2026-08-19, ScadaBridge arch-review P4-1 + P4-3; wire-compatible, no schema change):** (1) the sync pump now **faults the live session when a cap-prune invalidates its position** (horizon check on the oplog's min seq before each read batch + on the batch head after it; throws the new public `LocalDbOplogGapException` through the 0.3.1 fault-teardown) — previously a slow-but-alive peer silently skipped the pruned gap and the pair diverged until the next reconnect; (2) **transport keepalives both sides** — client `SocketsHttpHandler` pings via new validated `LocalDb:Replication:KeepAlivePingDelay`/`KeepAlivePingTimeout` (defaults 60s/20s, `Always` policy), and ScadaBridge's site gRPC listener sets Kestrel h2 keepalive (`Host/SiteKestrelSetup.cs` — server-wide, also governs the metrics port); (3) the server's single-stream guard is now **newest-wins with an awaited handover** — a fresh authenticated `Sync` cancels a dead/wedged incumbent and *waits* for it to finish (`TakeoverTimeout` 10s) before starting, so a half-open link can no longer wedge the pair behind `AlreadyExists` while the zombie side reports `LocalDbReplicationConnected=true`.
- **LocalDb 0.2.1** (two adversarial-review findings against 0.2.0, wire-compatible, no schema change) fixes: (1) `DeregisterReplicated` was deleting the HLC clock's only durable crash anchor along with the table rows — an ungraceful exit (crash, SIGKILL, power loss) on a fully-deregistered file left nothing to recover from, so the reopened clock could restart from wall-clock time and re-issue stamps below HLCs peers already hold (silently discarded by LWW); the current clock is now flushed inside the deregistration transaction, before the prune. (2) The sync inbox was an **unbounded** channel, so inbound memory was a function of what the peer sends — a mutual full snapshot (both sides of a pair baseline unconditionally when replication is enabled late) buffered whole, in memory, on both nodes at once; it is now **bounded at 64 messages** with wait-mode backpressure (unrelated to, and not to be confused with, the rig's separate `MaxBatchSize: 64` DB-read-page pin above), which required moving the snapshot to run alongside the receive loop (not inside the pre-loop handshake) so the inbox is drained for the whole session and can't deadlock itself.
- All timestamps are UTC throughout the system.
- Inter-cluster communication uses **three** transports, not two — **all cross-cluster command/control and data now rides gRPC** after the ClusterClient→gRPC migration's Phase 4 (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`) deleted Akka `ClusterClient`/`ClusterClientReceptionist`: (1) **gRPC command/control** — site→central over the central-hosted `CentralControlService` (`GrpcCentralTransport`, sticky central-a→central-b channel pair; deployments/notifications/health/heartbeat/audit-ingest/reconcile), and central→site over the site-hosted `SiteCommandService` (`GrpcSiteTransport`, per-site NodeA→NodeB channel pair; the 28 lifecycle/OPC-UA/query/parked/route/failover commands); (2) **gRPC** server-streaming for real-time data (attribute values, alarm states, `SiteStreamService`); and (3) **plain token-gated HTTP** for the deployment config itself — notify-and-fetch, the site pulls the config from `DeploymentConfigEndpoints` (`ManagementService/DeploymentConfigEndpoints.cs`) with an `X-Deployment-Token` header, `AllowAnonymous` with the per-deployment token as the entire security boundary. The gRPC boundary is per-site PSK-authenticated (`ControlPlaneAuthInterceptor`, unchanged). There is **no receptionist registration** — discovery is by dialling configured endpoints; central builds one `SitePairChannelProvider` per site (addresses from `Site.GrpcNodeAAddress`/`GrpcNodeBAddress`, refreshed from the DB every 60s and on admin changes), sites dial `ScadaBridge:Communication:CentralGrpcEndpoints` (both central nodes, h2c on `CentralGrpcPort` 8083, **NOT** via Traefik). **Discovery is asymmetric by design:** central discovers site gRPC addresses from the *database* (refreshable at runtime), sites discover central from *appsettings* (`CentralGrpcEndpoints`, static — restart required; `StartupValidator` requires a Site node to list at least one). `Akka.Cluster.Tools` stays for ClusterSingleton; only the ClusterClient part is gone. **Central never buffers for an unreachable site** — the send fails with the caller's Ask/deadline timing out; a `ConnectionStateChanged` mechanism built for this was deleted as dead code.
@@ -124,7 +125,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
- Flattened configs include a revision hash for staleness detection.
- Deployment identity: unique deployment ID + revision hash for idempotency.
- Per-instance operation lock covers all mutating commands (deploy, disable, enable, delete).
- **Bulk site deployment (`DeploymentService.DeploySiteAsync`, arch-review WP2.5)** redeploys every out-of-date instance at a site through one flatten session (a template chain shared by N instances is walked once) in three phases — serial Prepare, bounded-parallel Send (`SiteDeploymentMaxParallelism`, default 4), serial Finalize — still **not all-or-nothing**: any one instance's failure doesn't block the rest. The `PendingDeployment` staging row (5-minute TTL) is written per-instance immediately before that instance's round-trip in the Send phase, not upfront in Prepare, so a large batch's tail instances don't have their TTL clock already half-spent by the time they're reached.
- **Bulk site deployment (`DeploymentService.DeploySiteAsync`, arch-review WP2.5)** redeploys every out-of-date instance at a site through one flatten session (a template chain shared by N instances is walked once) in three phases — serial Prepare, bounded-parallel Send (`SiteDeploymentMaxParallelism`, default 4), serial Finalize — still **not all-or-nothing**: any one instance's failure doesn't block the rest. **Disabled instances are SKIPPED by bulk deploy as of 2026-08-19 (arch-review P7-2)** — they are reported in the summary's additive `SkippedInstances` list, never flattened/sent/re-enabled (single-instance deploy remains the explicit operator path that deploys+enables a Disabled instance); Prepare's per-instance reconcile also runs under a short `SiteDeploymentReconcileTimeout` (5s) with a once-per-batch unreachable-site circuit-breaker, so a down site no longer costs 30s × N serial before the first send (P7-3). The `PendingDeployment` staging row (5-minute TTL) is written per-instance immediately before that instance's round-trip in the Send phase, not upfront in Prepare, so a large batch's tail instances don't have their TTL clock already half-spent by the time they're reached.
- Site-side apply is all-or-nothing per instance.
- System-wide artifact version skew across sites is supported.
- Last-write-wins for concurrent template editing (no optimistic concurrency on templates).
Directory.Packages.props
+3 -3
View File
@@ -109,9 +109,9 @@
<PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.6.2" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" Version="0.6.2" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.Grpc" Version="0.6.2" />
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.3.1" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.3.1" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.3.1" />
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.3.2" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.3.2" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.3.2" />
</ItemGroup>
<!--
docker-env2/central-node-a/appsettings.Central.json
+12 -5
View File
@@ -20,20 +20,27 @@
"MinNrOfMembers": 1
},
"Database": {
"ConfigurationDb": "Server=scadabridge-mssql,1433;Database=ScadaBridgeConfig2;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true",
"MachineDataDb": "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData2;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
"_comment": "ConfigurationDb/MachineDataDb (which carry the dev SQL password) are supplied as ScadaBridge__Database__* whole-key env overrides in docker-env2/docker-compose.yml (dev-only-insecure, mirroring docker/docker-compose.yml and the ApiKeyPepper convention). The same dev password already ships committed for the sibling scadabridge-mssql container (infra/docker-compose.yml MSSQL_SA_PASSWORD) — this is consolidation, not new exposure. Env overrides layer over JSON before the ${secret:} expander runs, so the dev cluster boots with no KEK/secret store. Real/prod config uses ${secret:} tokens in src/.../appsettings.Central.json (T4)."
},
"Security": {
"Ldap": {
"Server": "10.100.0.35",
"Server": "host.docker.internal",
"Port": 3893,
"FallbackServers": [
"host.docker.internal:3894"
],
"_comment_FallbackServers": "Redundant local GLAuth pair (scadaproj/infra/glauth-redundant): zb-ldap-primary :3893 + zb-ldap-backup :3894 on the Docker host — mirrors docker/central-node-a/appsettings.Central.json. Replaces the shared 10.100.0.35 GLAuth, whose serviceaccount password was rotated (SEC-36) away from the rig's dev bind creds; env2 hit this directly (every login failing) before this fix.",
"Transport": "None",
"AllowInsecure": true,
"SearchBase": "dc=zb,dc=local",
"ServiceAccountDn": "cn=serviceaccount,dc=zb,dc=local",
"ServiceAccountPassword": "serviceaccount123"
"_comment_ServiceAccountPassword": "Supplied as ScadaBridge__Security__Ldap__ServiceAccountPassword env override in docker-env2/docker-compose.yml (dev-only-insecure)."
},
"JwtSigningKey": "scadabridge-env2-dev-jwt-signing-key-must-be-at-least-32-characters-long",
"Auth": {
"DisableLogin": false,
"User": "multi-role"
},
"_comment_JwtSigningKey": "Supplied as ScadaBridge__Security__JwtSigningKey env override in docker-env2/docker-compose.yml (dev-only-insecure).",
"JwtExpiryMinutes": 15,
"IdleTimeoutMinutes": 30,
"RequireHttpsCookie": false,
docker-env2/central-node-b/appsettings.Central.json
+12 -5
View File
@@ -20,20 +20,27 @@
"MinNrOfMembers": 1
},
"Database": {
"ConfigurationDb": "Server=scadabridge-mssql,1433;Database=ScadaBridgeConfig2;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true",
"MachineDataDb": "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData2;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
"_comment": "ConfigurationDb/MachineDataDb (which carry the dev SQL password) are supplied as ScadaBridge__Database__* whole-key env overrides in docker-env2/docker-compose.yml (dev-only-insecure, mirroring docker/docker-compose.yml and the ApiKeyPepper convention). The same dev password already ships committed for the sibling scadabridge-mssql container (infra/docker-compose.yml MSSQL_SA_PASSWORD) — this is consolidation, not new exposure. Env overrides layer over JSON before the ${secret:} expander runs, so the dev cluster boots with no KEK/secret store. Real/prod config uses ${secret:} tokens in src/.../appsettings.Central.json (T4)."
},
"Security": {
"Ldap": {
"Server": "10.100.0.35",
"Server": "host.docker.internal",
"Port": 3893,
"FallbackServers": [
"host.docker.internal:3894"
],
"_comment_FallbackServers": "Redundant local GLAuth pair (scadaproj/infra/glauth-redundant): zb-ldap-primary :3893 + zb-ldap-backup :3894 on the Docker host — mirrors docker/central-node-b/appsettings.Central.json. Replaces the shared 10.100.0.35 GLAuth, whose serviceaccount password was rotated (SEC-36) away from the rig's dev bind creds; env2 hit this directly (every login failing) before this fix.",
"Transport": "None",
"AllowInsecure": true,
"SearchBase": "dc=zb,dc=local",
"ServiceAccountDn": "cn=serviceaccount,dc=zb,dc=local",
"ServiceAccountPassword": "serviceaccount123"
"_comment_ServiceAccountPassword": "Supplied as ScadaBridge__Security__Ldap__ServiceAccountPassword env override in docker-env2/docker-compose.yml (dev-only-insecure)."
},
"JwtSigningKey": "scadabridge-env2-dev-jwt-signing-key-must-be-at-least-32-characters-long",
"Auth": {
"DisableLogin": false,
"User": "multi-role"
},
"_comment_JwtSigningKey": "Supplied as ScadaBridge__Security__JwtSigningKey env override in docker-env2/docker-compose.yml (dev-only-insecure).",
"JwtExpiryMinutes": 15,
"IdleTimeoutMinutes": 30,
"RequireHttpsCookie": false,
docker-env2/docker-compose.yml
+45
View File
@@ -11,6 +11,10 @@ x-secrets-store-env: &secrets-store-env
services:
central-a:
image: scadabridge:latest
# An init process (tini) as PID 1, so a crashed dotnet process actually dies
# instead of wedging the container at 100% CPU with the restart policy never
# firing — mirrors docker/docker-compose.yml's init: true (ScadaBridge#34).
init: true
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
@@ -27,6 +31,25 @@ services:
# pepper per the "different per environment" guidance; real deployments inject a
# true secret out-of-band, never from source control. Both Central nodes share it.
ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-env2-cluster-0001"
# DEV-ONLY local-cluster secrets — NOT real secrets. Relocated out of the mounted
# appsettings.Central.json (mirrors docker/docker-compose.yml's G-4 T5 secrets
# cleanup) so that app-config file carries no plaintext credentials. Env overrides
# layer over JSON BEFORE the ${secret:} pre-host expander runs. The SQL password is
# the same value already committed for the sibling scadabridge-mssql container
# (infra/docker-compose.yml MSSQL_SA_PASSWORD) and for docker/'s central nodes —
# consolidation, not new exposure; env2's Config2/MachineData2 databases live on
# that SAME shared mssql container. Real deployments use ${secret:} tokens +
# a seeded store.
ScadaBridge__Database__ConfigurationDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeConfig2;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData2;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
# LDAP bind credential for the redundant local GLAuth pair (scadaproj/infra/glauth-redundant,
# host.docker.internal:3893/:3894 — see central-node-a/appsettings.Central.json). Same
# value docker/'s central nodes use for the SAME local pair; the shared 10.100.0.35
# GLAuth's serviceaccount password was rotated (SEC-36) and is no longer valid.
ScadaBridge__Security__Ldap__ServiceAccountPassword: "serviceaccount123"
# DEV-ONLY local-cluster secret — NOT a real secret. Distinct from docker/'s JWT key
# so sessions cannot cross environments (see docker-env2/README.md).
ScadaBridge__Security__JwtSigningKey: "scadabridge-env2-dev-jwt-signing-key-must-be-at-least-32-characters-long"
# DEV-ONLY gRPC control-plane preshared key for site-x — NOT a real secret.
# Must match ScadaBridge:Communication:GrpcPsk in site-x-node-*/appsettings.Site.json.
# Production seeds SB-GRPC-PSK-<siteId> into the secret store instead.
@@ -47,6 +70,7 @@ services:
central-b:
image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
@@ -63,6 +87,25 @@ services:
# pepper per the "different per environment" guidance; real deployments inject a
# true secret out-of-band, never from source control. Both Central nodes share it.
ScadaBridge__InboundApi__ApiKeyPepper: "dev-only-insecure-pepper-env2-cluster-0001"
# DEV-ONLY local-cluster secrets — NOT real secrets. Relocated out of the mounted
# appsettings.Central.json (mirrors docker/docker-compose.yml's G-4 T5 secrets
# cleanup) so that app-config file carries no plaintext credentials. Env overrides
# layer over JSON BEFORE the ${secret:} pre-host expander runs. The SQL password is
# the same value already committed for the sibling scadabridge-mssql container
# (infra/docker-compose.yml MSSQL_SA_PASSWORD) and for docker/'s central nodes —
# consolidation, not new exposure; env2's Config2/MachineData2 databases live on
# that SAME shared mssql container. Real deployments use ${secret:} tokens +
# a seeded store.
ScadaBridge__Database__ConfigurationDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeConfig2;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
ScadaBridge__Database__MachineDataDb: "Server=scadabridge-mssql,1433;Database=ScadaBridgeMachineData2;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
# LDAP bind credential for the redundant local GLAuth pair (scadaproj/infra/glauth-redundant,
# host.docker.internal:3893/:3894 — see central-node-b/appsettings.Central.json). Same
# value docker/'s central nodes use for the SAME local pair; the shared 10.100.0.35
# GLAuth's serviceaccount password was rotated (SEC-36) and is no longer valid.
ScadaBridge__Security__Ldap__ServiceAccountPassword: "serviceaccount123"
# DEV-ONLY local-cluster secret — NOT a real secret. Distinct from docker/'s JWT key
# so sessions cannot cross environments (see docker-env2/README.md).
ScadaBridge__Security__JwtSigningKey: "scadabridge-env2-dev-jwt-signing-key-must-be-at-least-32-characters-long"
# DEV-ONLY gRPC control-plane preshared key for site-x — NOT a real secret.
# Must match ScadaBridge:Communication:GrpcPsk in site-x-node-*/appsettings.Site.json.
# Production seeds SB-GRPC-PSK-<siteId> into the secret store instead.
@@ -83,6 +126,7 @@ services:
site-x-a:
image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
@@ -111,6 +155,7 @@ services:
site-x-b:
image: scadabridge:latest
init: true # PID-1 crash wedge, ScadaBridge#34 — see central-a
# CoordinatedShutdown needs cluster-leave (15s budget) + cluster-exiting +
# actor-system-terminate + Serilog flush; the 10s SIGTERM default SIGKILLed
# mid-drain, turning every redeploy into the crash path (review 01 [Medium]).
docker/regen-proto.sh
+20 -8
View File
@@ -50,12 +50,25 @@ if [[ ! -f "$CSPROJ" ]]; then
exit 1
fi
# Backup so we can always restore the comment state on failure. Leaving the
# csproj with an active Protobuf item is the one outcome that breaks Docker, so
# every exit path restores this copy.
# Backup so we can always restore the comment state, however this script ends.
# Leaving the csproj with an active Protobuf item is the one outcome that breaks
# Docker, so every exit path restores this copy — success, a failed `dotnet
# build` (ERR), or Ctrl-C (INT/TERM) mid-build. An EXIT trap covers all of
# those (bash runs it on any shell termination, signals included), and the
# guard on $BACKUP's existence makes it idempotent: the normal success path
# below restores explicitly for prompt feedback, so by the time the trap fires
# on a clean exit $BACKUP is already gone and it's a no-op instead of a second
# restore/rm.
BACKUP="$(mktemp)"
cp "$CSPROJ" "$BACKUP"
trap 'cp "$BACKUP" "$CSPROJ"; rm -f "$BACKUP"; echo "Restored csproj from backup."' ERR
restore_csproj() {
if [[ -f "$BACKUP" ]]; then
cp "$BACKUP" "$CSPROJ"
rm -f "$BACKUP"
echo "Restored csproj from backup."
fi
}
trap restore_csproj EXIT
# 1. Inject an ItemGroup holding just the selected protos, immediately before
# the closing </Project>. The documented commented-out block is left alone.
@@ -117,10 +130,9 @@ if [[ "$TARGET" == "sitecommand" || "$TARGET" == "all" ]]; then
fi
# 5. Restore the backed-up csproj — i.e. drop the injected ItemGroup — so Docker
# builds keep working.
cp "$BACKUP" "$CSPROJ"
rm -f "$BACKUP"
trap - ERR
# builds keep working. (Also covered by the EXIT trap above; calling it here
# too just gives the success path its "Restored..." feedback before "Done".)
restore_csproj
echo ""
echo "Done. Review and commit:"
docs/plans/2026-07-08-deferred-work-register.md
+3 -2
View File
@@ -87,14 +87,15 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
| 92 | **The event-log cap's size model is sample-based and blind to mid-history width shifts** *(Low, accuracy limitation — documented in memo §3.1)* | Row-81 fix session (2026-08-17) | Row width is sampled 500 rows from each END of `site_events`; a stream whose row width changes sharply mid-history is mis-sized until the trim reaches that region. Self-correcting per tick (each trim exposes new tail rows to the sample), and the model's deliberate +8–13% high bias absorbs modest shifts, but a pathological width profile could over- or under-trim for several ticks. **Reviewed 2026-08-18: accepted by design** — inherent to any sampled model, not closable by code short of the row-94 `dbstat` replacement; the row-93 schema pin does not (and cannot) cover it. | First field report of the cap trimming visibly wide of its target, or the row-94 `dbstat` replacement (which retires this row with the model) |
| 93 | ~~**The cap model's page-layout constants are empirical for the CURRENT `site_events` schema**~~ **RESOLVED 2026-08-18 — the silent trap is now loud.** `SiteEventsSchemaCalibrationTests` (SiteEventLogging.Tests) pins the exact shape the calibration assumed: the eight-column list the sampler's `octet_length` sum covers, the index set as exactly the four secondary indexes `IndexedColumnCount = 4` charges plus the PK autoindex (absorbed into the measured fill factors), and each index's single covered column. Any `site_events` schema migration — new column, fifth index, composite widening — now fails the suite with a message directing to the memo §4 recalibration and the constants + sampling-SQL update, instead of silently mis-sizing the cap. Suite green 110/110. | Row-81 fix session (2026-08-17); pinned by test 2026-08-18 | Closed 2026-08-18. | — |
| 94 | **`SqliteCapabilityTests` is a deliberate tripwire — do not "fix" it by weakening the assertion** *(Low, standing note)* | Row-81 fix session (2026-08-17) | The test FAILS BY DESIGN if a future SQLite bump ships `dbstat`/`sqlite_dbpage` support. The correct response at that point is to REPLACE the sampled page-layout model with exact `dbstat` measurement (retiring row 92 — and the row-93 schema pin `SiteEventsSchemaCalibrationTests`, added 2026-08-18, goes with the model too), not to relax the test. The test's own comment says the same. | `SqliteCapabilityTests` fails after a SQLite/provider bump |
| 95 | ~~**The leaked 100 ms-tick test-actor pattern is systemic — three more classes carry it**~~ **RESOLVED 2026-08-17 (merge `106e467d`, agent commit `34a8079d`).** All three classes live in AuditLog.Tests (premise correction: none in SiteCallAudit.Tests). `Watch`/`Stop`/`ExpectTerminated` replicated from `da23f2dc` at every creation site: 11 in `AuditLogPurgeActorTests`, 10 in `SiteAuditReconciliationActorTests`, 2 in `OutageReconciliationTests`. Second premise correction: `OutageReconciliationTests` DID call `Sys.Stop` — but fire-and-forget AND from the last line of the `await using` block, so an in-flight tick could still be writing to MSSQL during read-back and any earlier assertion failure skipped the stop entirely; stops moved before the read-backs, plus the `AuditLogIngestActor` that test 2 was discarding outright (holding a disposed provider) is now stopped too. Known lesser sibling habit left in place: EventStream probes are never unsubscribed (bounded by the per-test ActorSystem — noted, not a row). AuditLog.Tests 383/383. | Row-82 fix session (2026-08-17) | Closed 2026-08-17. | — |
| 95 | ~~**The leaked 100 ms-tick test-actor pattern is systemic — three more classes carry it**~~ **RESOLVED 2026-08-17 (merge `106e467d`, agent commit `34a8079d`).** All three classes live in AuditLog.Tests (premise correction: none in SiteCallAudit.Tests). `Watch`/`Stop`/`ExpectTerminated` replicated from `da23f2dc` at every creation site: 11 in `AuditLogPurgeActorTests`, 10 in `SiteAuditReconciliationActorTests`, 2 in `OutageReconciliationTests`. Second premise correction: `OutageReconciliationTests` DID call `Sys.Stop` — but fire-and-forget AND from the last line of the `await using` block, so an in-flight tick could still be writing to MSSQL during read-back and any earlier assertion failure skipped the stop entirely; stops moved before the read-backs, plus the `AuditLogIngestActor` that test 2 was discarding outright (holding a disposed provider) is now stopped too. Known lesser sibling habit left in place: EventStream probes are never unsubscribed (bounded by the per-test ActorSystem — noted, not a row). AuditLog.Tests 383/383. **Postscript 2026-08-19 (arch-review P8-6): the premise correction was itself WRONG — SiteCallAudit.Tests DOES leak 100 ms-tick actors.** `SiteCallAuditPurgeTests` (4 creation sites incl. a discarded `Sys.ActorOf`) and `SiteCallAuditReconciliationTests` (11 sites, two in one test) had ZERO `Watch`/`Stop`/`ExpectTerminated`. Reopened and re-closed 2026-08-19 (agent commit `102e7a2b`): row-82 treatment via a per-file `StopTicker` helper (`try/finally`, blocked test doubles released first) at every site; the same sweep also fixed `PendingDeploymentPurgeActorTests.cs:90` in Communication.Tests (same class), capped Communication.Tests with the row-96 `xunit.runner.json` (`maxParallelThreads: 4` — it was the last uncapped TestKit+Kestrel suite), collection-serialized the two `aggregators.active` gauge-moving classes (row-102 shape), and reworked 11 sleep-then-assert-absence sites in `SiteAlarmAggregatorActorTests` per row-40 (`AssertHoldsFor` polling helper; settle-sleeps before *actions* left alone). Both suites 3× green (717/717, 66/66), zero flakes observed. | Row-82 fix session (2026-08-17) | Closed 2026-08-17; reopened + re-closed 2026-08-19 (arch-review P8-6). | — |
| 96 | ~~**xunit parallelism is uncapped almost everywhere**~~ **RESOLVED 2026-08-17 (merge `106e467d`, agent commit `d2d4db49`) — policy made and shipped: per-suite caps for the resource-heavy suites, pure unit suites stay uncapped.** The four MSSQL-fixture users (`ConfigurationDatabase.Tests`, `AuditLog.Tests`, `SiteCallAudit.Tests`, `Host.Tests` — verified before capping) get `xunit.runner.json` with `maxParallelThreads: 4`, **deliberately matching row-82's `sp_getapplock` slot default** so the in-process and cross-process limits agree; no collections serialized. `IntegrationTests` (already fully serialized) and `CentralUI.PlaywrightTests` (already `maxParallelThreads: 1`, stricter than needed) left alone. Cap verified live (216% CPU uncapped vs 145% capped on ConfigurationDatabase.Tests); cost is nil — these suites are latency-bound on MSSQL/sockets (14–25% CPU), durations unchanged within noise. Policy documented in `docs/test_infra/test_infra.md` → *Test Parallelism Policy* (two-axis model, per-suite table, default for new projects), cross-linked from `test_infra_db.md`. Found-not-fixed: `StoreAndForward.Tests/xunit.runner.json` is a no-op (pins the default `parallelizeTestCollections: true`) — harmless, tidy on next touch. | Row-82 fix session (2026-08-17) | Closed 2026-08-17. | — |
| 97 | ~~**A live DCL connection actor is never reconfigured in place — on ANY node**~~ **RESOLVED 2026-08-17 (merge `9dbda4a2`, agent commit `7c1268d8`) — the memo's §6 sketch shipped, plus two things the sketch missed.** `EnsureDclConnection` now has three outcomes off its one hash slot: unchanged → nothing; new → create; **changed → `ReconfigureConnectionCommand`** (arbitration/tombstones byte-for-byte untouched). The manager forwards to the owning actor (unknown name falls back to create); `DataConnectionActor` handles it in **all three** lifecycle states and never stashes it, via a new shared `SwapAdapter` helper now also used by both failover sites. The sketch's two gaps, found in implementation: **endpoint selection must be re-armed to Primary** (a connection sitting on its old *backup* would otherwise dial the new backup first), and the deliberate link drop must **suppress the unstable-disconnect accounting** (else the swap itself could trip a failover). Also a premise-sharpening: "reconfigure while mid-connect" is the *primary* case, not an edge — a wrong endpoint loops in `Connecting` forever and never reaches `Connected`. Fail-before: 5/7 new `DataConnectionReconfigureTests` + 3 SiteRuntime ensure-path tests fail on the reverted behavior. **Live rig probe (post-redeploy, round trip):** connection 1's config changed (operationTimeoutMs 15000→16000→back) fired exactly TWO in-place reconfigures on the active node — `Reconfiguring DataConnectionActor for OPC PLC Simulator in place` → `Reconnected successfully on Primary endpoint` in 45 ms → `Re-subscribing 2 tags` — with all rig-probe-1 tags Good after each, the standby silent (0 reconfigure lines, topology as designed), and the old dead-end "has a new configuration" Warning never appearing. Rig config restored to original. | Found by the 2026-08-17 rows-89/90 fix session; memo `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md` (§9 postscript appended) | Closed 2026-08-17. | — |
| 98 | ~~**`EnforceSiteScopeForInstance` is fail-open for nonexistent instance ids**~~ **RESOLVED 2026-08-17 (merge `152ec85e`, agent commit `f33f25a1`) — premise corrected: the defect had FIVE instances, not two, and the worst three were worse than "harmless".** Beyond the two named helpers, the same `if (target != null) EnforceSiteScope(...)` shape sat **inline** in `HandleGetInstance` (:879), `HandleGetSite` (:1702) and `HandleGetDataConnection` (:1844) — and those returned `ManagementSuccess{JsonData=null}` for a nonexistent id vs `ManagementUnauthorized` for an out-of-scope one, i.e. a working **id-enumeration oracle across the whole fleet** for a site-scoped caller. All five now route through one shared `DenyUnresolvedTargetIfScoped`: a scoped caller's unresolvable target is DENIED, indistinguishable from out-of-scope (no existence leak); system-wide callers (Administrator / system-wide Deployer) are untouched. Also checked: `Instance.SiteId` is non-nullable `int`, so no secondary fail-open path through `EnforceSiteScope(null)`. Fail-before: 5 of the 16 new ManagementActorTests fail on the old code. | Found by the 2026-08-17 rows-37/86-88 fix session (Agent A survey) | Closed 2026-08-17. | — |
| 99 | ~~**CLI `audit query --status` client-side validation is case-sensitive**~~ **RESOLVED 2026-08-17 (merge `152ec85e`, agent commit `f33f25a1`) — premise corrected: "the server would accept it" was FALSE, and removing the client gate alone would have been a regression.** The server-side parser (`AuditQueryParamParsers.ParseEnumList`) is documented-and-tested **lax** — it silently drops unparseable values and collapses to null (filter unconstrained), i.e. exactly the row-46/88 silent-wrong-answer trap. Fix shipped both halves: **seven** `AcceptOnlyFromAmong` gates removed across `audit query`/`audit export` (`--channel`/`--kind`/`--status` on both, plus `export --format`, which rejected `CSV` the server lowercases anyway), valid sets moved to help text, and a new **strict** `ParseEnumListStrict` (throws naming the valid values; `LooksNumeric` guard rejects ordinals) wired into both ManagementService audit endpoints. The lax parser deliberately survives for the CentralUI drill-in/export query strings (bookmarked fragments must render, not fault — documented as non-reconcilable). Left alone with reasons in code: `--direction up|down` (no server parse to defer to), `--format json|table` (CLI-local), NotificationCommands' literals (unsurveyed). Four pre-existing tests were passing *vacuously* (no-credentials exit codes) and now pin the real contract; fail-before 21/26. **Found-and-fixed follow-on (orchestrator, `16056f8e`):** row 88's own `ParseDeploymentStatus` accepted ordinals (`--status 2` → InProgress) — same `LooksNumeric` guard applied + 3 tests. **Live rig probes (post-redeploy):** `audit query --status delivered` returns rows; `--status Nonsense` and `--status 3` both fail loud naming the 8 valid values; `deploy status --status 2` fails loud naming the 4 valid values. | Found by the 2026-08-17 rows-37/86-88 fix session | Closed 2026-08-17. | — |
| 99 | ~~**CLI `audit query --status` client-side validation is case-sensitive**~~ **RESOLVED 2026-08-17 (merge `152ec85e`, agent commit `f33f25a1`) — premise corrected: "the server would accept it" was FALSE, and removing the client gate alone would have been a regression.** The server-side parser (`AuditQueryParamParsers.ParseEnumList`) is documented-and-tested **lax** — it silently drops unparseable values and collapses to null (filter unconstrained), i.e. exactly the row-46/88 silent-wrong-answer trap. Fix shipped both halves: **seven** `AcceptOnlyFromAmong` gates removed across `audit query`/`audit export` (`--channel`/`--kind`/`--status` on both, plus `export --format`, which rejected `CSV` the server lowercases anyway), valid sets moved to help text, and a new **strict** `ParseEnumListStrict` (throws naming the valid values; `LooksNumeric` guard rejects ordinals) wired into both ManagementService audit endpoints. The lax parser deliberately survives for the CentralUI drill-in/export query strings (bookmarked fragments must render, not fault — documented as non-reconcilable). Left alone with reasons in code: `--direction up|down` (no server parse to defer to), `--format json|table` (CLI-local), NotificationCommands' literals (unsurveyed). Four pre-existing tests were passing *vacuously* (no-credentials exit codes) and now pin the real contract; fail-before 21/26. **Found-and-fixed follow-on (orchestrator, `16056f8e`):** row 88's own `ParseDeploymentStatus` accepted ordinals (`--status 2` → InProgress) — same `LooksNumeric` guard applied + 3 tests. **Live rig probes (post-redeploy):** `audit query --status delivered` returns rows; `--status Nonsense` and `--status 3` both fail loud naming the 8 valid values; `deploy status --status 2` fails loud naming the 4 valid values. **Postscript 2026-08-19 (arch-review P6-3): the strict parser this row shipped was itself fail-open — `Enum.TryParse` splits and ORs comma-joined names inside ONE token for any enum (not just `[Flags]`), and dense 0-based `AuditStatus` lands composites on defined members:** `--status Forwarded,Attempted` = 1|2 = 3 = **Delivered** (silently wrong data), `Submitted,Parked` = 0|5 drops Submitted; the server passes raw query values whole (ASP.NET Core does not split commas), so the trap was live on `/api/audit/query` + export. Reopened and re-closed 2026-08-19 (agent commit `db174633`): any token containing `,` is rejected before `TryParse` with the same valid-values error; proof-of-failure verified on the two name+name composites. Nuance pinned by probe: the `"Failed,7"` name+ordinal variant does NOT reproduce — `TryParse` rejects mixed name/numeric comma lists outright — the test is kept as a regression guard. Repeated-param lists of valid names unaffected. | Found by the 2026-08-17 rows-37/86-88 fix session | Closed 2026-08-17; reopened + re-closed 2026-08-19 (arch-review P6-3). | — |
| 100 | ~~**`SiteScriptCompileCache.Clear()` is process-global — test classes wipe each other's entries**~~ **RESOLVED 2026-08-17 (merge `8a9a8d29`, agent commit `84b405ff`) — premise corrected: there were ZERO production callers of `Clear()`** (the only production entry point is `ScriptCompilationService.CompileCore` → `GetOrAdd`; `Clear()` existed solely for tests), so neither register option was taken as written — `Clear()` was **deleted outright** and the cache became an instance-scoped `sealed class` owned by `ScriptCompilationService`. The N1 "one compile per body per process" property now rests on `AddSiteRuntime` registering that service as a **DI singleton** — a lifetime that became load-bearing, so new `SiteRuntimeServiceRegistrationTests` pins it (a Scoped/Transient downgrade would reintroduce WP3.1's recompile storms with no behavioral test failing). The 37 test sites doing `new ScriptCompilationService(logger)` each silently get their own cache — isolation for the whole assembly, not just the row-91 classes. Row-91 workarounds RETIRED, not preserved: serialized collection removed from all three classes, exact `Hits`/`Count` assertions restored, guid key token gone, `ForeignEntrySlack` 512→8; new `SeparateCaches_DoNotShareEntriesOrCounters` pins the scoping. SiteRuntime.Tests green three consecutive full-parallel runs. | Found by the 2026-08-17 row-91 fix session | Closed 2026-08-17. | — |
| 101 | ~~**`ApplyingArtifacts_RemovedDataConnection_StopsLiveDclConnection` flaked once — suspected overlay-sweep read racing the artifact-apply write**~~ **RESOLVED 2026-08-17 (`0d3a90a9`) — hypothesis VERIFIED, and the production consequence was WORSE than the row guessed.** The race is real: the snapshot is read off the actor thread but compared against the overlay only when the piped message lands, so a read starting before an artifact apply's store writes can land after `HandleApplyArtifactDataConnectionsToDcl` populated the overlay — its stale (even empty) snapshot then "proves" the just-applied names disappeared. Overlay iteration order explains the exact fingerprint (`PlcA` removed first). Two corrections to the row's guesses: (1) it is NOT "self-healing at the next tick" — the removal evicts the hash slot the overlay's push is gated on, so the next fresh sweep clears the tombstone but re-creates nothing; the stopped connection stays dead **until an instance redeploy**; (2) the mid-write "momentarily not seeing the row" framing was the lesser half — the whole-snapshot staleness (read-before-write, land-after-apply) is the dominant window. Fix: `_definitionsWriteGeneration`, bumped on the actor thread by every artifact apply and captured at every snapshot-read dispatch (boot load + sweep); a result carrying a stale generation is discarded WHOLE — additions included, since a stale row pushed at a live connection would reconfigure it backward and flip-flop with the next fresh read. Deterministic repro fails before the fix (`StaleSweepSnapshot_RacingAnArtifactApply_DoesNotRemoveJustAppliedConnections`); `FreshEmptySweepSnapshot_StillRemovesDisappearedConnections` pins that row-90's legitimate disappearance still fires. SiteRuntime.Tests 647/647; rig redeployed, zero spurious removals/discards at boot, all rig-probe-1 tags Good. Memo §10 records the mechanism. **Postscript 2026-08-18 (merge `505f3569`, agent commit `f02e1166`): extended to the WHOLE snapshot.** The instance-config half had the same TOCTOU, and all three hazards were PROVEN real pre-fix: a stale sweep restarts an instance **backward** (`TryRestartStaleInstance` feeds the snapshot's old `ConfigJson` to `HandleDeploy` — a literal old-config redeploy; pipeline ownership only covers in-flight deploys and the old revision was never claimed), a read-pre-delete/disable **resurrects** a just-stopped instance (delete even clears the sweep claim), and the boot path skews `_deployedInstanceNames`. Fix: one `_storeWriteGeneration` over both tables, bumped at BOTH ends of every actor-thread config mutation (deploy persist, enable, disable, delete, rollback, artifact apply) — both ends because `ApplyDeployment`/`HandleDisable`/`HandleDelete` advance actor state BEFORE the off-thread write, so a completion-only bump (the row-101 original argument, which holds only for the artifact apply) leaves a window. Stale sweep discarded whole; stale boot load re-dispatched, capped at 5 consecutive (then applied with a Warning). CDC/`SiteReconciliationActor` writes deliberately never bump — discovering them is the sweep's purpose. 3 new fail-before tests + 2 fresh-generation guards pinning that row-72/78 legitimate repairs survive; SiteRuntime.Tests 652/652 x3; full suite green (one unrelated Communication.Tests gauge flake → row 102); rig probe: disable/enable/redeploy churn over multiple sweep ticks produced zero self-heal events on either site-a node, tags Good. Memo §11. | Observed by the 2026-08-17 row-100 fix session | Closed 2026-08-17; whole-snapshot extension 2026-08-18. | — |
| 102 | ~~**`SiteStreamSubscribersActiveGauge_GoesToOneOnConnect_AndBackToZeroOnCancel` is delta-hardened but still flakes — the baseline can't survive other contributors CLOSING**~~ **RESOLVED 2026-08-18 via the preferred option (row-96 pattern).** The survey found the gauge moves ONLY inside the real `SiteStreamGrpcServer.SubscribeInstance` handler, and exactly two test classes drive it: `SiteStreamGrpcServerTests` (the gauge test's own class) and `CleanupVerificationTests` — `GrpcOptionsWiringTests` instantiates the server but never opens a stream, and the client/aggregator suites subscribe against fakes, so none of those can perturb the counter. Both drivers now share the `[Collection("SiteStreamSubscriberGauge")]` xunit collection (comments on both classes + at the baseline poll explain why the relative read alone was insufficient), so no other contributor can close a stream between the baseline capture and the assertions. Verified 4/4 consecutive full-suite runs green (717/717), durations unchanged (22–25 s vs the prior 24 s) — the serialization is free. Original evidence: the test polls `baseline + 1`, but the process-global gauge's baseline is captured ONCE; capture while parallel classes' streams are up → they close → the gauge settles at this test's own 1 and `baseline+1` is unreachable forever — a fixed-baseline delta only survives contributors that hold constant or increase. | Observed 2026-08-18 (one full-suite failure, `Expected: 4, Actual: 1`; 3/3 green in isolation); fixed same day | Closed 2026-08-18. | — |
| 103 | **Two lesser TTL-bounded circuit-teardown-vs-async-setup races remain in CentralUI** (arch-review 2026-08-19 P6-6's optional tail, deliberately deferred by the fix). The permanent debug-stream leak was closed at `DebugView.razor` + `DebugStreamHub` (dispose-vs-connect take-once pattern, commit `11e623c2`), but two same-family instances with a *different shape* were left: (1) `TransportImport.razor.cs TryLoadAsync` (~:244) — a circuit torn down mid-load strands 1 of the 8 `MaxConcurrentImportSessions` slots until the 30-min `BundleSession` TTL reaps it (partially defeats row 43); (2) `Topology.razor` (~:201-214) — timer created before a disposal re-check. Both are TTL/lifetime-bounded (no permanent leak), which is why they missed the Medium bar. Fix is the same post-await `_disposed` re-check + take-once teardown, but routed through `BundleSession.ReleaseSession`/the wizard-step lifecycle rather than a session-stop call. | Arch-review 2026-08-19 P6-6 remediation (T10 explicitly scoped them out) | Open. | Next CentralUI hardening pass, or opportunistically when either file is next touched. |
| 68 | ~~**`deploy.sh` run from a git worktree silently rehomes every node's bind-mount data dir**~~ **RESOLVED 2026-08-15 (guard added in the same close-out).** The compose files' bind mounts are relative (`./site-a-node-a/data`), so a deploy from a worktree checkout mounts the worktree's copies — the rig keeps running and looks healthy, but its SQLite stores split from the main checkout's and everything written there is orphaned by the next deploy from main. Bit the rows-46–53 close-out live: the seeding agent's `rig-probe-1` deploy persisted into worktree-homed dirs and "vanished" after the orchestrator's redeploy (no tombstone, no product bug — LWW/tombstone forensics ruled out deletion; the row sat in `.claude/worktrees/…/docker/site-a-node-a/data/site-localdb.db` the whole time). Both `docker/deploy.sh` and `docker-env2/deploy.sh` now refuse to run from a `.claude/worktrees/` path (override: `SCADABRIDGE_ALLOW_WORKTREE_DEPLOY=1`). | Found during the rows-46–53 close-out | Closed 2026-08-15. | — |
## Resolved (verified against the code 2026-07-10)
docs/plans/2026-08-19-medium-findings-remediation-plan.md
+374
View File
@@ -0,0 +1,374 @@
# Medium-Findings Remediation Plan — 2026-08-19 Arch Review
Scope: all **Medium** and **Medium-High** findings from the 2026-08-19 updated architectural
review (report artifact `43c7fa20-9626-4d59-a9c9-061524794660`). The four Highs (P2-1, P2-2,
P6-1, P6-2) were fixed in the preceding wave (`c4d3c8b8..d18f344e`, deployed to the rig).
Baseline: `main @ d18f344e`.
Two borderline findings are rounded **in** because they are cheap and adjacent to in-scope
work: P3-3 (Low-Med, Debug View dual-address requirement) and P4-4 (Med-Low, standby
event-log retention). All plain-Low findings stay out of scope (P1-4, P2-5, P2-6, P3-4,
P3-5, P4-5..P4-7, P5-3..P5-6, P6-10..P6-12, P7-5, P7-6, P8-7..P8-10).
Register consequences (handled at integration, NOT by task agents, to avoid file
contention): row 99 reopened+closed by T8 (P6-3 shows the closure was defective);
row 95 reopened+closed by T17 (P8-6 shows its premise correction was wrong).
Execution model (same as the High wave): file-disjoint tasks, one dedicated manual git
worktree + model-matched subagent each, pathspec commits, cherry-pick onto main,
combined build + affected suites, `git diff` review, single merge session. Two waves.
Decision made in this plan (owner may override later): **P7-2 takes the conservative
option** — bulk `DeploySiteAsync` skips `Disabled` instances and surfaces them in the
result as skipped, matching the reconcile path's deliberate preservation of Disabled.
Rationale: the alternative (silently force-enabling deliberately-stopped equipment on a
bulk push) is the hazard the finding names.
---
## Wave 1
### T1 (Opus) — P1-1 + P1-2: DeploymentManager races (heal resurrection; redeploy override reset)
**Files:** `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs`,
`InstanceActor.cs`, the `EnsureInstanceStartedResult` message, `InstanceActorTests.cs`.
- **P1-1 (Med):** `HandleEnsureInstanceStarted` (:1571) re-reads the store off-thread but
captures no `_storeWriteGeneration`; the result handler never calls
`IsStoreSnapshotStale` (unlike boot :521 and sweep :1726). A Disable/Delete processed
between read and result resurrects the instance permanently (sweep has no runtime→store
stop leg; disabled rows `continue`, deleted rows are absent). Verifier note: trigger is
the ONE-SHOT startup reconcile (`SiteReconciliationActor.cs:267`), so the window is
boot-adjacent; the stale comment at :1567-1570 must go. **Fix:** thread the generation
captured at read through `EnsureInstanceStartedResult`; discard on mismatch (boot/sweep
pattern). Test: heal result arriving after a Disable/Delete is dropped.
- **P1-2 (Med, arguably Med-High):** ApplyDeployment creates the new InstanceActor (:1202)
BEFORE the Task.Run that ends in `ClearStaticOverridesAsync` (:1222-1234); the new
actor's PreStart override load typically beats the clear (verifier: likely interleaving)
→ pre-redeploy overrides applied for the actor's lifetime, violating "reset on
redeployment". Leg 2: old actor's PostStop fire-and-forget `FlushDirtyStaticOverrides`
(InstanceActor.cs:363-364) can durably commit AFTER the clear. **Fix:** on redeploy,
clear overrides in the persistence task BEFORE `CreateInstanceActor` (or gate the
override load behind the persistence result), and order the old actor's final flush
before the clear (e.g. await/flush before actor creation — Terminated is already awaited
:1004-1066, so flushing synchronously in PostStop's stop path or draining before clear
are both acceptable; pick the minimal ordering that closes both legs).
`InstanceActorTests.cs:355-356` pins the WRONG contract — fix the test to pin the right
one. Tests: redeploy never applies pre-redeploy overrides; late flush cannot resurrect.
### T2 (Sonnet) — P1-3: Alarm on-trigger runs bypass the stuck-script watchdog
**File:** `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptRunLauncher.cs`
(`RunAlarmScriptAsync` :454-537 / `LaunchAlarmScript` :149-186).
Script path arms watchdog + slot capture (:244-253); alarm path does none on the same
bounded pool — a wedged alarm body silently costs a pool thread forever (bounded 4/alarm
by the in-flight cap), with no site event, no detach, no healthCollector error increment,
no `ScriptRunSummaryRecorder`. **Fix:** mirror the script path exactly — slot/stamp/
watchdog + health/summary threading (timeout catch today is LogWarning-only :518-524).
Tests: first tests to drive `LaunchAlarmScript` — wedged alarm body triggers watchdog
detach + site event + `DetachedScriptThreads`; timeout increments health error.
### T3 (Opus) — P2-3 + P2-4: DCL ConnectResult generation; OPC UA batch rollback
**Files:** `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs`,
`Adapters/RealOpcUaClient.cs`.
- **P2-3 (Med):** `ConnectResult` piped untagged (:777-787). Reconfigure mid-connect: old
adapter's `ConnectResult(true)` drives ReSubscribeAll+BecomeConnected against the NEW
never-connected adapter; new adapter's result dropped in Connected; idempotent-
reconfigure guard (:644-652) then blocks recovery until restart/redeploy. **Fix:** tag
`ConnectResult` with the adapter generation (stamped at connect launch, exactly the
P2-1 pattern now on `SubscribeCompleted` — reuse it); drop mismatches. Tests mirror
`DataConnectionStaleCompletionTests` for the connect leg.
- **P2-4 (Med):** `CreateSubscriptionsCoreAsync` (RealOpcUaClient.cs:663): mid-batch
`ApplyChangesAsync` throw leaves all `created` items registered (_callbacks/_itemShard/
ItemCount/SDK items :628-661) with no rollback, and `IsConnectionLevelFailure`
(DataConnectionActor.cs:1464-1471) misses `ServiceResultException` → chunk treated
per-tag unresolved, each backoff probe RE-ADDS items → unbounded shard growth. **Fix:**
unwind `created` via `ReleaseFailedItem` on throw; classify SDK bad-session codes
connection-level. Tests: fake-SDK mid-batch throw leaves no residue; bad-session code
drives the connection-level path.
### T4 (Opus) — P3-1 + P3-2 + P3-3: gRPC client lifecycle/disposal races
**Files:** `src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs`,
`Grpc/SitePairChannelProvider.cs`, `DebugStreamService.cs`.
- **P3-1 (Med):** `RegisterSubscription` (:147-153) disposes the prior handler's CTS while
its `ConsumeStreamAsync` may still read the token (ODE at :219/:294/:359/:364);
`Unsubscribe` (:455-462) unguarded Cancel; `ReleaseResources` (:609-619) unguarded
iteration whose ODE aborts the factory's DisposeAsync loop leaving later clients
undisposed; an ODE from `prior.Cancel()` escapes the NEW SubscribeAsync with the CURRENT
generation → phantom fault. **Fix:** cancel-only on replacement (dispose only from the
owning path), ODE-guard every Cancel, keep factory disposal loop fault-isolated.
`SiteStreamGrpcClientTests.cs:680` pins current behavior — re-pin.
- **P3-2 (Med):** `UpdateSite`/`RemoveSite` (:123-131/:155-159) Dispose channels under
`pair.Gate` while `ExecuteAsync` invokes outside the lock (:203-215) → 60s DB refresh
with an edited address kills in-flight RPCs (multi-minute DeployArtifactsCommand) with
ODE/Cancelled (not Unavailable → no cross-node retry). **Fix:** deferred disposal — park
replaced channels until quiescent or time-delayed; mirror the stream factory's redesign
rationale (SiteStreamGrpcClientFactory.cs:12-22).
- **P3-3 (Low-Med, rounded in):** `DebugStreamService` (:111-114) throws unless BOTH gRPC
node addresses are set, though single-endpoint sites are supported elsewhere. **Fix:**
apply the N9 degeneration (`SiteAlarmLiveCacheService.cs:324-330`): reuse the sole
endpoint.
### T5 (Opus) — P4-1 + P4-3: LocalDb sync divergence + liveness (cross-repo)
**Files (scadaproj):** `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/src/.../Internal/SyncSession.cs`,
`OplogStore.cs`, `SyncBackgroundService.cs`, tests. **Files (here):**
`src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (site Kestrel h2 keepalive),
`Directory.Packages.props` (0.3.1 → 0.3.2 when published).
- **P4-1 (Med):** `PumpLoopAsync` never checks `needs_snapshot` after a cap-prune though
`OplogStore.ReadBatchAboveAsync` documents the obligation (OplogStore.cs:27-29) —
maintenance prunes above `_sentThruSeq` on a slow-but-alive peer → pump silently skips
the gap, pair diverges for the session's lifetime (heals only on reconnect). **Fix:**
fault the live session when `EnforceCapsAsync` prunes into the in-flight range (the
0.3.1 fault-teardown control-lane flush already exists — reuse it). Test: prune during
an active session faults it; reconnect serves the snapshot.
- **P4-3 (Med):** no keepalive anywhere on the sync stream: bare `GrpcChannel.ForAddress`
client-side, no Kestrel `Http2.KeepAlivePingDelay` on site 8083 → hard crash/partition
of the dialing node leaves the passive node's handler hung on MoveNext holding
`_active=1`, rejecting the rebooted initiator's reconnects `AlreadyExists`, possibly
forever; zombie side reports `LocalDbReplicationConnected=true`. **Fix:** client
`SocketsHttpHandler` KeepAlivePing (delay/timeout, always-with-active-requests) in
`SyncBackgroundService`; Kestrel h2 keepalive on the site gRPC listener in Host
`Program.cs`; AND newest-wins eviction (or a bounded read deadline) on the server's
single-stream guard so a fresh authenticated Sync can displace a dead one.
- **Version/publish:** bump LocalDb to 0.3.2, commit in scadaproj (do NOT push);
`build/push.sh` needs the user-held `GITEA_NUGET_KEY` — if absent, leave the
ScadaBridge `Directory.Packages.props` bump as a flagged follow-up and land only the
Host keepalive here. The ScadaBridge-side Kestrel change is independent and lands
regardless.
### T6 (Sonnet) — P4-2 + P4-4: audit-writer durability; standby event-log retention
- **P4-2 (Med, arguably Med-High):** `AuditLog/Site/SqliteAuditWriter.cs` — per-row
`TrySetResult` (:487/:499) BEFORE `transaction.Commit()` (:503); commit failure
(SQLITE_FULL/IOERR) no-ops `TrySetException` → `FallbackAuditWriter` sees success, no
ring fallback, whole batch silently lost in exactly the outage the ring exists for;
`SiteAuditWriteFailures` stays 0. Bonus (verifier): a mid-batch insert failure on row k
loses rows 0..k-1 the same way. **Fix:** complete TCSes only after commit (the pattern
`SiteEventLogger.FlushBatch` already uses :319/:357-361); on commit/insert failure fail
EVERY row's TCS so the ring fallback engages. Tests: failed commit → all TCSes faulted +
ring receives batch + failure counter increments.
- **P4-4 (Med-Low, rounded in):** `SiteEventLogging/EventLogPurgeService.cs:215-219`
early-exits on the standby on the stale premise "standby receives no writes" — DCL
actors run on both nodes since rows 89/90/97 and write site events. On an unreplicated
site (site-b/c posture) the standby's `site_events` grows forever. **Fix:** let the
standby run retention/cap against its own file (no-op-safe on a replicated pair —
deletes are CDC-captured and LWW-converge; say so in a comment). Test: purge runs when
not primary.
### T7 (Sonnet) — P5-1 + P5-2: central ingest fallback budget; SiteCalls query shape
- **P5-1 (Med):** `AuditLog/Central/AuditLogIngestActor.cs:372-405` — per-entry fallback
loop passes NO token/timeout (contrast `OnIngestAsync`'s 5s `IngestFallbackBudget`
:268); each entry sits inside `SqlServerRetryingExecutionStrategy` 5×30s → minutes
inside `ReceiveAsync`, singleton mailbox frozen, site re-sends amplify. **Fix:** mirror
`OnIngestAsync` — short linked CTS + `IngestSqlCommandTimeout` per entry, pass the token
to `InsertIfNotExistsAsync`/`UpsertAsync`. Test: fallback loop honors budget.
- **P5-2 (Med):** `ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs:301-331`
orders `(CreatedAtUtc DESC, TrackedOperationId DESC)` with no index leading on
`CreatedAtUtc` (full scan+sort per unfiltered page, row-35 class) and binds
nvarchar(4000) params against `IsUnicode(false)` columns (may block seeks, incl. the
keyset predicate :316; write path types VarChar deliberately :230). **Fix:** add the
`(CreatedAtUtc DESC, TrackedOperationId DESC)` index via EF migration (BUILD FIRST —
`--no-build` scaffolds empty) + typed varchar params on the read path. Update the
handwritten production SQL script if one exists for this table (migration-parity rule).
### T8 (Sonnet) — P6-3: ParseEnumListStrict comma smuggling (register row 99 defective)
**File:** `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Audit/AuditQueryParamParsers.cs:117-119`.
`Enum.TryParse` ORs comma-joined names for any enum; `LooksNumeric` checks first char
only; dense 0-based `AuditStatus` makes composites land on defined members —
`"Forwarded,Attempted"` = 1|2 = 3 = `Delivered` (silent wrong data); comma also smuggles
ordinals past the row-88/99 guard (`"Failed,7"` → `Skipped`). Server path confirmed:
`AuditEndpoints.cs:656-658` passes the raw query value whole. **Fix:** reject any value
containing ',' before TryParse (per-token error, same UX as existing invalid-name error).
Tests: comma-joined names rejected; `"Failed,7"` rejected; single names/known-good paths
unchanged. Do NOT edit the register (integration does).
## Wave 2
### T9 (Opus) — P6-4 + P6-5: ManagementActor scoping gaps (single-writer lane)
**File:** `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs` (sole owner
this wave).
- **P6-4 (Med):** `HandleListDataConnections` (:1875-1881) takes no user, null role gate →
lists every site's connections (scrubbed but topology-revealing existence oracle) while
`HandleGetDataConnection` fail-closes (:1888-1900) — Get denies what List hands over.
**Fix:** filter by `PermittedSiteIds` exactly as `HandleListSites` (:1719-1729) /
`HandleListSecuredWrites` (:1665-1695) do. Check the Design UI page's separate
`SiteRepository.GetAllDataConnectionsAsync` bypass (DataConnections.razor:309) — if it
serves scoped users unfiltered, route it through scope too (same one-line filter).
- **P6-5 (Med):** resolve-before-scope oracles outside row 98's five shapes: approve
(:1465-1474) and reject (:1627-1635) check not-found + status (leaking "is Executed,
not Pending") BEFORE `EnforceSiteScopeForIdentifier`; same shape `HandleDebugSnapshot`
(:3604-3608) + `HandleSubmitSecuredWrite` (:1417-1423). **Fix:** hoist scope checks
above resolution/status checks at all four sites (row-98 `DenyUnresolvedTargetIfScoped`
semantics). Tests: out-of-scope caller gets the scope denial, not the status oracle,
at each site.
### T10 (Opus) — P6-6: Debug-stream session abandoned mid-start leaks permanently
**Files:** `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/DebugView.razor(.cs)`
(:493), `DebugStreamHub.cs` (:195).
`Connect()` awaits `StartStreamAsync` (≤30s) then assigns `_session` with no `_disposed`
re-check; if Dispose already ran, StopStream never fires and the singleton
`DebugStreamService` keepalive renews the registered session forever (:157/:213-219; the
5-min orphan net only reaps UNregistered sessions) — holds the site gRPC debug sub +
bridge actor + dead circuit graph until restart. Identical shape in `DebugStreamHub.cs:195`.
**Fix:** post-await `_disposed` re-check that stops the just-created session (both sites);
make Dispose vs late-assign race-safe (interlocked/lock — Blazor Sync context helps but
Dispose can run on a different flow). Tests: dispose-during-start stops the session and
leaves nothing registered. The lesser TTL-bounded instances (TransportImport slot,
Topology timer) are OPTIONAL — take them only if trivial with the same pattern.
### T11 (Sonnet) — P6-7 + P6-8: audit export injection + CLI deadline
- **P6-7 (Med):** `ManagementService/AuditEndpoints.cs:810-840` CSV quotes only on
`, " \r \n`; leading `= + - @ \t` pass through → formula/DDE execution on operator open
(CWE-1236); exported Target/ErrorMessage/Request/ResponseSummary/Extra carry inbound-
API-caller text verbatim. **Fix:** prefix-escape cells starting with those chars
(`'` prefix, the standard neutralization). Secondary: CSV omits ExecutionId/
ParentExecutionId/SourceNode vs JSONL and the `AuditExportRow.cs:11-13` "24-field"
claim — add the columns and align the doc comment.
- **P6-8 (Med):** `CLI/Commands/AuditExportHelpers.cs:129` —
`SendGetStreamAsync(..., CancellationToken.None)` after `HttpClient.Timeout = Infinite`
(residue of the e0e4b246 timeout-honesty fix): a hung server hangs the CLI forever, and
body reads (:141/:152)/`CopyToAsync` (:171) are tokenless (mid-stream fault → raw stack
+ truncated file). **Fix:** headers-received deadline + overall/inactivity deadline +
Ctrl-C token wiring through all reads; friendly error on mid-stream fault. Tests:
deadline fires; cancellation propagates.
### T12 (Sonnet) — P6-9: LoginThrottle never throttles single-IP password spray
**File:** `src/ZB.MOM.WW.ScadaBridge.CentralUI/Security/LoginThrottle.cs:189-190`.
Key is `{username}|{ip}` → independent 5-failure budget per username per IP; a spray
(many usernames, one IP) is never throttled though the class doc claims spray protection.
**Fix:** add a secondary per-IP counter with a higher ceiling (e.g. 20-30 failures/window
→ per-IP lockout), layered on the existing per-pair key; keep the doc honest. Tests:
spray locks the IP; normal per-user behavior unchanged; distinct IPs unaffected.
### T13 (Sonnet) — P7-1: ExternalSystem TimeoutSeconds silently dropped by bundles
**Files:** `src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntityDtos.cs:248`,
`EntitySerializer.cs:127-141`, `BundleImporter.cs:3087-3128`, `ArtifactDiff.cs:166-179`,
`RoundTripEquivalenceTests.cs:445-452`.
Row-27 SHAPE recurrence: DTO lacks `TimeoutSeconds` (operative at runtime and deployed as
proto field 6); export never writes it, Add resets to 0, Overwrite keeps old, diff never
compares → a tuned-timeout-only bundle previews Identical and the target env silently runs
the gateway default. The round-trip guard is vacuous because the fixture never sets the
field. **Fix:** add the DTO field + both serializer directions + both import branches +
`AddIfDifferent` in the diff + a NON-DEFAULT fixture value so the reflection round-trip
guard actually covers it. Schema stays additive (schemaVersion unchanged — additive field).
### T14 (Opus) — P7-2 + P7-3 + P7-4: bulk-deploy semantics, reconcile deadline, probe TOCTOU
**Files:** `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentService.cs`,
`StaleInstanceProbe.cs`, `FlattenSession.cs`.
- **P7-2 (Med, decided above):** `DeploySiteAsync` deploys ALL instances (:804) and every
success runs `forceEnabledState:true` (:610-612) — bulk push re-enables deliberately-
Disabled equipment while the reconcile path deliberately preserves Disabled (:1638-1649,
pinned). **Fix:** skip Disabled instances in bulk deploy; surface them in the result as
a skipped list with reason; pin with tests (zero Disabled cases exist in
DeploySiteAsyncTests today). Update `Component-DeploymentManager.md` (one sentence).
- **P7-3 (Med):** phase-1 `TryReconcileWithSiteAsync` per instance is bounded only by the
30s QueryTimeout → unreachable site × N instances ≈ hours serial before a single send;
CLI's 5-min BulkDeployTimeout cancels and re-arms the identical cost. **Fix:** in bulk
mode, short linked per-reconcile deadline AND a once-per-batch site reachability probe —
first reconcile timeout marks the site unreachable for the rest of Prepare (instances
proceed on DB state, which is what happens on timeout today anyway).
- **P7-4 (Med):** `StaleInstanceProbe.cs:106-109` re-reads per-template versions AFTER
flatten (comment claims before) → a non-structural template edit during flatten pairs
pre-edit hash with post-edit version; memo permanently serves not-stale. FlattenSession
captures correctly (FlattenSession.cs:114) but doesn't expose it. **Fix:** expose the
session's captured `MemberVersions`; probe stores those instead of re-reading. Test:
template edit mid-flatten still reports stale.
### T15 (Sonnet) — P8-1: three SiteHealthReport fields never cross the wire
**Files:** `src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/CentralControlDtoMapper.cs:452`,
the vendored centralcontrol proto (regen via `bash docker/regen-proto.sh centralcontrol`),
`SiteRuntime SiteHealthCollector.cs:243/:317-327`, `HealthReportSender.cs:180`.
`DetachedScriptThreads`, `ScriptRunShedCount`, `SiteStreamAlarmDropCount` are populated
site-side but `ToDto` omits all three (proto ends at field 29) — a watchdog-detaching/
shedding/alarm-dropping site looks healthy centrally. Aggravator: `AddIntervalCounters`
takes only the 5 original counters and `HealthReportSender` never passes the shed count →
a failed send permanently loses the interval's shed count (drained by
`Interlocked.Exchange`). **Fix:** proto fields **30-32** (additive, never reuse numbers),
mapper both directions, extend `AddIntervalCounters` so failed sends restore all drained
counters. Follow the vendored-proto workflow exactly (regen script restores the csproj; an
active `<Protobuf>` item must never be committed). Tests: mapper round-trips the three
fields; failed send restores shed count.
### T16 (Sonnet) — P8-2 + P8-3 + P8-4: env2 topology drift + regen-proto trap
- **P8-2 (Med):** `docker-env2/docker-compose.yml` — add `init: true` to all 4 services
(docker/ has it ×8 with the issue-#34 rationale; env2's PID-1 boot failure wedges at
100% CPU instead of restart-looping).
- **P8-3 (Med):** env2 centrals still carry the SEC-36-rotated LDAP bind cred in plaintext
(`central-node-a/appsettings.Central.json:28,34` + node-b), plus inline SQL passwords +
JWT key. **Fix:** port the docker/ posture — compose env-var overrides for secrets, and
the same LDAP fallback/DisableLogin arrangement docker/ uses. Do NOT invent new secret
values; reference the same env vars docker/ consumes.
- **P8-4 (Med):** `docker/regen-proto.sh:58` — `trap ... ERR` doesn't fire on Ctrl-C
during the long build → active `<Protobuf>` left in the csproj (breaks Docker build).
**Fix:** idempotent restore trap on `EXIT` (covers ERR/INT/TERM/success; guard so a
successful run doesn't double-restore).
### T17 (Sonnet) — P8-5 + P8-6: test-health (register rows 95/96/40/102 classes)
- **P8-5 (Med):** Communication.Tests: (a) add `xunit.runner.json` `maxParallelThreads: 4`
(row-96 policy; ~14 TestKit classes + real Kestrel, last uncapped resource-heavy suite);
(b) `SiteAlarmLiveCacheTelemetryTests.cs:55-93` asserts fixed baselines on the
process-global `aggregators.active` gauge with no `[Collection]` while two sibling
suites move the instruments (row-102 shape) — apply the row-102 treatment (serialize via
collection or delta-based assertions); (c) rework the 8 sleep-then-assert-absence sites
in `SiteAlarmAggregatorActorTests` (:385-386, :624-625 et al) per row-40 (await the
observable effect / use TestKit expectations, never wall-clock sleeps), and fix the
discarded 100ms-tick actor in `PendingDeploymentPurgeActorTests.cs:90` (row-95 class:
Watch + Stop + ExpectTerminated).
- **P8-6 (Med):** SiteCallAudit.Tests DOES leak 100ms-tick actors (row 95's premise
correction was wrong): `SiteCallAuditPurgeTests` actors at :127/:152/:174 + discarded
`Sys.ActorOf` :209-212, `SiteCallAuditReconciliationTests` :58/:248/:372/:592 — zero
Watch/Stop/ExpectTerminated in either file. **Fix:** row-82 treatment on every leaked
actor. Do NOT edit the register (integration reopens/closes row 95 with a citation).
- Test-only task: verify by running both suites (Communication.Tests 3× for flake
confidence).
---
## Shared constraints (all tasks)
- Fix + tests only; no drive-by refactors. Match surrounding style/comment density.
- Targeted verification per task: build touched project(s) + test project(s); run affected
suites filtered. Full solution sweep once, at integration.
- Each agent works ONLY inside its own worktree (absolute paths, `git -C`), commits with a
pathspec limited to its own files, reports commit SHA + test results. Agents do NOT
touch `docs/plans/2026-07-08-deferred-work-register.md` or `CLAUDE.md` (integration
does).
- Component-doc updates only where the doc states the now-contradicted behavior; one
sentence each.
- Wire/proto changes additive only; never reuse field numbers.
## Integration (per wave, then final)
1. Cherry-pick wave commits onto `main` (verify file-disjointness first; T4/T15/T17 share
the Communication project — files must stay disjoint).
2. `dotnet build ZB.MOM.WW.ScadaBridge.slnx`; run the touched suites; full
`dotnet test` sweep after wave 2.
3. Register edits: reopen+close rows 95 and 99 citing P8-6/P6-3; add rows for anything an
agent flags as deliberately deferred.
4. `git diff` review; commit plan doc + register/docs touch-ups; remove worktrees.
5. No push, no rig redeploy without the user.
docs/plans/sql/AddSiteCallsListSortIndex.sql
+50
View File
@@ -0,0 +1,50 @@
-- arch-review 2026-08-19 P5-2 (register row-35 class): the SiteCalls list page's
-- sort key gets its own index.
--
-- SiteCallAuditRepository.QueryAsync orders (CreatedAtUtc DESC, TrackedOperationId
-- DESC), and its keyset predicate is exactly that pair. None of the existing
-- SiteCalls indexes lead unconditionally on CreatedAtUtc:
-- - IX_SiteCalls_Source_Created requires an equality on SourceSite first.
-- - IX_SiteCalls_NonTerminal / IX_SiteCalls_Terminal are each filtered to one
-- side of TerminalAtUtc.
-- So an unfiltered (or lightly filtered) page fell back to a full clustered-index
-- scan + sort. IX_SiteCalls_Created_TrackedOperationId (CreatedAtUtc DESC,
-- TrackedOperationId DESC) makes a forward scan of the index deliver rows in the
-- requested order directly, so a keyset page reads a Top-N off the index instead
-- of sorting the whole table on every page request.
--
-- SiteCalls is small enough today that a plain (offline) index build is fine. On
-- an Enterprise edition instance where the table has grown large and the Central
-- UI must stay responsive, add WITH (ONLINE = ON) to the CREATE INDEX below.
--
-- Run with QUOTED_IDENTIFIER ON. Several tables in this database carry filtered
-- indexes and persisted computed columns, and SQL Server refuses DDL/DML against
-- them with error 1934 when the session has QUOTED_IDENTIFIER OFF -- which is
-- exactly what `sqlcmd` gives you by default (pass -I, or keep this header). The
-- SETs sit in their own batch so they are in force when the statements after the
-- GO parse.
SET QUOTED_IDENTIFIER ON;
SET ANSI_NULLS ON;
GO
BEGIN TRANSACTION;
IF NOT EXISTS (
SELECT * FROM [__EFMigrationsHistory]
WHERE [MigrationId] = N'20260819183326_AddSiteCallsListSortIndex'
)
BEGIN
EXEC(N'CREATE INDEX [IX_SiteCalls_Created_TrackedOperationId]
ON [SiteCalls] ([CreatedAtUtc] DESC, [TrackedOperationId] DESC)');
END;
IF NOT EXISTS (
SELECT * FROM [__EFMigrationsHistory]
WHERE [MigrationId] = N'20260819183326_AddSiteCallsListSortIndex'
)
BEGIN
INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion])
VALUES (N'20260819183326_AddSiteCallsListSortIndex', N'10.0.7');
END;
COMMIT;
GO
docs/requirements/Component-DeploymentManager.md
+20
View File
@@ -169,6 +169,26 @@ one is parallel:
3. **Finalize (serial).** Commit terminal statuses, apply post-success side
effects, write audit rows, release each operation lock.
**Disabled instances are skipped.** A bulk site deploy never touches an instance
in `Disabled` — no flatten, no send, no state change — because deploy is a legal
transition out of `Disabled` and forces `Enabled` on success, so an unfiltered
site-wide push would silently restart every piece of equipment an operator had
deliberately stopped; they are reported in the summary's `SkippedInstances` with
the reason "Disabled — bulk deploy preserves disabled state; deploy individually
to re-enable", and the single-instance deploy remains the explicit re-enable path.
**Reconciliation is bounded per batch.** Phase 1 is serial, so an unreachable site
would otherwise cost the batch one full communication-layer `QueryTimeout` (30 s)
per instance before a single deploy was sent — hours for a large site, past the
CLI's 5-minute bulk deadline, which cancels and re-arms the identical cost on the
next attempt. Each reconcile in a bulk deploy therefore runs under the short
`SiteDeploymentReconcileTimeout` (default **5 s**), and the FIRST one to hit that
deadline opens a once-per-batch circuit-breaker that skips the remaining
reconciles (one summary log line, not N). Skipping costs nothing: a timed-out
reconcile already falls through to a normal deploy on central's recorded state,
with the site's own stale-revision rejection as the safety net. The
single-instance path is unbounded by this option and keeps the full `QueryTimeout`.
**Not all-or-nothing.** An instance that fails for any reason (wrong state, failed
validation, lock already held by another operation, site round-trip timed out) is
reported as a failed row while the rest proceed, and is individually retryable via
docs/requirements/Component-ManagementService.md
+1 -1
View File
@@ -141,7 +141,7 @@ The whole folder surface is reachable from the CLI as `template folder list|crea
### Data Connections
- **ListDataConnections** / **GetDataConnection**: Query data connection definitions.
- **ListDataConnections** / **GetDataConnection**: Query data connection definitions. Both are site-scoped for a site-scoped non-Administrator caller — `List` filters an explicit `SiteId` through `EnforceSiteScope` and narrows an unfiltered list to the caller's permitted sites, rather than returning the fleet and nullifying `Get`'s fail-closed gate (arch-review 2026-08-19 P6-4).
- **CreateDataConnection** / **UpdateDataConnection** / **DeleteDataConnection**: Manage data connection definitions.
- **AssignDataConnectionToSite** / **UnassignDataConnectionFromSite**: Manage site assignments.
docs/requirements/Component-Security.md
+1
View File
@@ -209,6 +209,7 @@ To blunt online password-spray and brute-force against the directory, every LDAP
- **Scope — every LDAP-bind surface**: the Central UI interactive login (`POST /auth/login`) and CLI token issue (`POST /auth/token`), plus the HTTP-Basic management/CLI surfaces fronted by `ManagementAuthenticator` — `POST /management`, the audit REST endpoints, **and the debug-stream hub (`DebugStreamHub.OnConnectedAsync` delegates its whole credential path to `ManagementAuthenticator.AuthenticateAsync`, closing the round-2 N1 gap where the hub bound LDAP directly and bypassed the throttle)**. No bind happens outside these paths. The dev/test `DisableLogin` bypass sits **above** the throttle: a login-disabled deployment never binds, so it is never throttled.
- **Key and window**: counters are keyed per `{username}|{IP}` pair (lower-cased username; the IP half is the **forwarded-header-resolved client IP** when `ScadaBridge:Security:ForwardedHeaders` names a trusted proxy, otherwise the raw connection peer). A fixed window of `LoginFailureWindowMinutes` (default 5) opens on the first failure; reaching `MaxLoginFailuresPerWindow` failures (default 5) within it locks that key out for `LoginLockoutMinutes` (default 5). A successful bind clears the key; an expired window resets the count. Setting `MaxLoginFailuresPerWindow` to `0` disables throttling entirely.
- **Per-IP spray ceiling (arch-review 2026-08-19 P6-9).** The per-pair key alone is blind to a *spray* — one attacker IP trying one password against many distinct usernames never reaches `MaxLoginFailuresPerWindow` on any single pair — so a secondary counter keyed on the IP alone, same window/lockout but a higher ceiling (`PerIpMaxFailures`, default 25, validated strictly greater than `MaxLoginFailuresPerWindow`), locks out *new* attempts from that IP once it is reached; a successful bind does **not** reset the per-IP counter (only window/lockout expiry does), so a spray that stumbles onto one valid account cannot ride that success to keep spraying.
- **Proxy topologies and the username-lockout DoS.** The key is deliberately `{username}|{IP}`, not `{username}` alone: keying on username alone would let any network-adjacent actor lock any operator out of a SCADA control surface with five wrong passwords, repeatable indefinitely. That isolation only exists when the throttle sees real client IPs — behind a proxy **without** ForwardedHeaders trust, all clients collapse onto the proxy's IP and the DoS returns. Mitigations: (a) the shipped Traefik topologies enable trusted-proxy ForwardedHeaders (so per-IP isolation is real in the documented deployment); (b) the lockout window is short by default (`LoginLockoutMinutes` = 5) and never touches the directory account itself; (c) residual risk — an attacker spraying from their *own* IP locking out a victim username *at that IP only* — is the throttle working as designed. A success-path bypass ("a correct password unlocks") was considered and rejected: it would let an attacker who has the password bypass the lockout entirely, and it converts the throttle into a password oracle during the lockout window.
- **Behaviour when locked**: the surface refuses the bind without contacting LDAP. The Basic-Auth surfaces (`ManagementAuthenticator`) and `POST /auth/token` return HTTP `429` with `code = "AUTH_THROTTLED"`; `POST /auth/login` redirects to `/login` with a "Too many failed attempts. Try again later." message.
- **Best-effort, per-node**: the throttle is in-memory with no external state. The two central nodes maintain independent counters (a spray hitting both simply burns its budget on each). Memory is bounded — writes prune expired keys and cap the tracked-key count. This is a rate-limit guard, not an account-lockout policy; it never disables the directory account itself.
docs/requirements/Component-SiteEventLogging.md
+6 -3
View File
@@ -77,8 +77,11 @@ Each event entry contains:
## Storage
- Events are stored in the **consolidated site LocalDb** SQLite file (`site_events` table;
see CLAUDE.md → Consolidated site database). Only the **active node** generates and stores
events (a standby runs no scripts, subsystems, or deployments to log).
see CLAUDE.md → Consolidated site database). The active node generates most events (a
standby runs no scripts, subsystems, or deployments to log), but DataConnectionActor
runs on **both** site nodes and writes its own `site_events` rows regardless of
active/standby status (register rows 89/90/97) — see *Retention* below for the
purge consequence.
- **`site_events` IS replicated** (`SiteLocalDbSetup.ReplicatedTables`) on a site pair with
replication configured — this reverses the component's pre-LocalDb-Phase-1 behavior, and the
reason is **failover history continuity, not either-node queries**: the query handler
@@ -92,7 +95,7 @@ Each event entry contains:
On a site pair running **without** replication configured (by deliberate choice, e.g. the
rig's site-b/site-c), the log stays node-local and a failover does start it fresh — the
documented trade of not configuring a peer, unchanged by this policy.
- **Retention**: 30 days. A **daily background job** runs on the active node and deletes all events older than 30 days. Hard delete — no archival. Deletes are sliced into bounded 1000-row batches per DELETE statement rather than one unbounded statement, so a large expired backlog does not hold the write lock (and every concurrent recorder flush) for the duration of the purge.
- **Retention**: 30 days. A **daily background job** runs on **every** site node (arch-review 2026-08-19 P4-4 — previously active-node-only, but the standby writes its own `site_events` rows too, so an unreplicated standby's file grew unbounded under the old active-only gate) and deletes that node's own events older than 30 days. Hard delete — no archival. Deletes are sliced into bounded 1000-row batches per DELETE statement rather than one unbounded statement, so a large expired backlog does not hold the write lock (and every concurrent recorder flush) for the duration of the purge.
- **Storage cap**: A configurable maximum size (default: 1 GB) is enforced. If the cap is reached
before the 30-day retention window, the oldest events are purged first. This prevents disk
exhaustion from alarm storms, script failure loops, or connection flapping.
src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogIngestActor.cs
+34 -5
View File
@@ -325,7 +325,12 @@ public class AuditLogIngestActor : ReceiveActor
/// <see cref="Microsoft.EntityFrameworkCore.RelationalDatabaseFacadeExtensions.BeginTransactionAsync"/>
/// scope and a try/catch around it). Audit-write failure NEVER aborts the
/// user-facing action — the site keeps the row Pending and retries on the
/// next drain.
/// next drain. The per-entry loop (used when <see cref="TryIngestCachedBatchAsync"/>
/// falls back) runs under its own fresh <see cref="IngestFallbackBudget"/>
/// token, mirroring <see cref="OnIngestAsync"/>'s fallback — a slow or wedged
/// database can only cost this handler that one short budget, not an
/// unbounded per-entry loop, so the singleton's mailbox is never frozen for
/// the life of a large packet.
/// </remarks>
private async Task OnCachedTelemetryAsync(IngestCachedTelemetryCommand cmd)
{
@@ -369,14 +374,38 @@ public class AuditLogIngestActor : ReceiveActor
return;
}
// The per-entry fallback gets its OWN fresh, short budget — mirroring
// OnIngestAsync's fallback (see IngestFallbackBudget above). Without
// this, an unbounded loop of per-entry transactions (each wrapped in
// SqlServerRetryingExecutionStrategy's 5x30s retry ladder) can run for
// minutes inside this ReceiveAsync handler, freezing the singleton's
// mailbox: every queued IngestAuditEventsCommand/IngestCachedTelemetryCommand
// stalls behind it, sites time out and re-send, and the pile-up
// compounds. Budget exhaustion stops the loop early; entries left
// over are simply never added to `accepted`, so the site keeps them
// Pending and the existing lost-telemetry path (periodic
// reconciliation pull) picks up the remainder.
using var fallbackBudget = new CancellationTokenSource(IngestFallbackBudget);
var entryIndex = 0;
foreach (var entry in cmd.Entries)
{
entryIndex++;
if (fallbackBudget.IsCancellationRequested)
{
var remaining = cmd.Entries.Count - entryIndex + 1;
_logger.LogWarning(
"Combined telemetry per-entry fallback exhausted its {FallbackSeconds}s budget; leaving {Remaining} entr(ies) unaccepted for this drain — the site keeps them Pending and periodic reconciliation will pick up the remainder.",
IngestFallbackBudget.TotalSeconds,
remaining);
break;
}
try
{
await strategy.ExecuteAsync(async () =>
{
await using var tx = await dbContext.Database
.BeginTransactionAsync()
.BeginTransactionAsync(fallbackBudget.Token)
.ConfigureAwait(false);
// Stamp IngestedAtUtc on both rows from a single
@@ -396,12 +425,12 @@ public class AuditLogIngestActor : ReceiveActor
var auditStamped = AuditRowProjection.WithIngestedAtUtc(filteredAudit, ingestedAt);
var siteCallStamped = entry.SiteCall with { IngestedAtUtc = ingestedAt };
await auditRepo.InsertIfNotExistsAsync(auditStamped)
await auditRepo.InsertIfNotExistsAsync(auditStamped, fallbackBudget.Token)
.ConfigureAwait(false);
await siteCallRepo.UpsertAsync(siteCallStamped)
await siteCallRepo.UpsertAsync(siteCallStamped, fallbackBudget.Token)
.ConfigureAwait(false);
await tx.CommitAsync().ConfigureAwait(false);
await tx.CommitAsync(fallbackBudget.Token).ConfigureAwait(false);
}).ConfigureAwait(false);
accepted.Add(entry.Audit.EventId);
src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/SqliteAuditWriter.cs
+26 -3
View File
@@ -484,7 +484,16 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
{
eventCmd.ExecuteNonQuery();
fwdCmd.ExecuteNonQuery();
pending.Completion.TrySetResult();
// Do NOT complete pending.Completion here (arch-review 2026-08-19
// P4-2). A row's INSERTs succeeding is not durable until the
// transaction actually commits below — completing the caller's Task
// this early means a later commit failure (SQLITE_FULL, IOERR, or a
// later row in this same batch throwing) reports false success: the
// row is rolled back out of the database but the caller — and
// FallbackAuditWriter, whose ring-buffer fallback exists precisely
// for this outage — already saw it as delivered. Every row in this
// batch is completed with TrySetResult() together, AFTER
// transaction.Commit() succeeds, mirroring SiteEventLogger.FlushBatch.
}
catch (SqliteException ex) when (ex.SqliteErrorCode == SqliteErrorConstraint)
{
@@ -492,21 +501,35 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
// audit_event PRIMARY KEY throws before the sidecar insert
// runs, so neither table gains a second row. Treat as
// success: the lifecycle event is durably recorded under
// the first writer's payload.
// the first writer's payload. Still deferred to after Commit()
// below — the surrounding transaction (this row's canonical/sidecar
// insert never ran, but earlier rows in the batch did) must still
// commit for that to be true.
_logger.LogDebug(ex,
"Duplicate EventId {EventId} swallowed by SqliteAuditWriter",
evt.EventId);
pending.Completion.TrySetResult();
}
}
transaction.Commit();
_flushCountForTests++;
// Only NOW — after the whole batch is durably committed — tell every
// caller (and FallbackAuditWriter) that their row landed.
foreach (var pending in batch)
{
pending.Completion.TrySetResult();
}
}
catch (Exception ex)
{
transaction.Rollback();
_logger.LogError(ex, "SqliteAuditWriter batch insert failed; faulting {Count} pending events", batch.Count);
// Fault EVERY row in the batch, including any whose own INSERTs succeeded
// above — none of that is true anymore, since the whole transaction (this
// row's insert included) just rolled back. No TrySetResult has run yet for
// any row in this batch (see above), so every TrySetException here is a
// real transition, not a no-op racing an already-completed Task.
foreach (var pending in batch)
{
pending.Completion.TrySetException(ex);
src/ZB.MOM.WW.ScadaBridge.CLI/Commands/AuditCommands.cs
+17 -1
View File
@@ -218,16 +218,32 @@ public static class AuditCommands
Actor = result.GetValue(actorOption),
};
// Ctrl-C is the one way to abort an in-flight export other than the
// headers/inactivity deadlines below — mirrors the CancellationTokenSource
// owned by DebugCommands' streaming debug-view command (arch-review
// 2026-08-19 P6-8: this was the one CLI HTTP path with NO deadline at all).
using var cts = new CancellationTokenSource();
ConsoleCancelEventHandler onCancel = (_, e) =>
{
e.Cancel = true;
cts.Cancel();
};
Console.CancelKeyPress += onCancel;
try
{
using var client = new ManagementHttpClient(connection.Url!, connection.Username!, connection.Password!);
return await AuditExportHelpers.RunExportAsync(client, args, Console.Out, DateTimeOffset.UtcNow);
return await AuditExportHelpers.RunExportAsync(client, args, Console.Out, DateTimeOffset.UtcNow, cts.Token);
}
catch (FormatException ex)
{
OutputFormatter.WriteError(ex.Message, "INVALID_ARGUMENT");
return 1;
}
finally
{
Console.CancelKeyPress -= onCancel;
}
});
return cmd;
}
src/ZB.MOM.WW.ScadaBridge.CLI/Commands/AuditExportHelpers.cs
+110 -7
View File
@@ -106,10 +106,34 @@ public static class AuditExportHelpers
return "?" + string.Join("&", parts);
}
/// <summary>
/// Deadline for receiving the export response's headers (i.e. the server has
/// started responding), measured from the start of the request. This is the
/// one bound <see cref="ManagementHttpClient.SendGetStreamAsync"/> does NOT
/// carry on its own — <see cref="ManagementHttpClient"/>'s <c>HttpClient.Timeout</c>
/// is deliberately <see cref="Timeout.InfiniteTimeSpan"/> (see its doc comment)
/// and <see cref="SocketsHttpHandler.ConnectTimeout"/> bounds only the TCP/TLS
/// connect phase, not a connected-but-silent server. Reuses
/// <see cref="ManagementHttpClient.DefaultConnectTimeout"/> as the standard CLI
/// per-call default (arch-review 2026-08-19 P6-8).
/// </summary>
public static readonly TimeSpan DefaultHeadersTimeout = ManagementHttpClient.DefaultConnectTimeout;
/// <summary>
/// Inactivity deadline for the streaming body copy: reset on every chunk
/// successfully read, so a large export keeps streaming indefinitely as long
/// as bytes keep arriving, but a stream that goes silent mid-transfer (a
/// hung/crashed server holding the connection open) is bounded rather than
/// hanging the CLI forever — the residual gap the timeout-honesty fix
/// (<c>ManagementHttpClient.DefaultConnectTimeout</c> doc comment) left in this
/// one path (arch-review 2026-08-19 P6-8).
/// </summary>
public static readonly TimeSpan DefaultInactivityTimeout = ManagementHttpClient.DefaultConnectTimeout;
/// <summary>
/// Executes the export: GETs <c>/api/audit/export</c> and copies the response body
/// stream directly to <see cref="AuditExportArgs.Output"/>. The body is never fully
/// buffered — <see cref="Stream.CopyToAsync(Stream)"/> streams in fixed-size chunks.
/// buffered — the copy streams in fixed-size chunks via <see cref="Stream.ReadAsync(Memory{byte}, CancellationToken)"/>.
/// A <c>501 Not Implemented</c> (parquet not yet supported server-side) prints the
/// server message and returns a non-zero exit code.
/// </summary>
@@ -117,16 +141,43 @@ public static class AuditExportHelpers
/// <param name="args">The export arguments containing filters and output file path.</param>
/// <param name="output">Text writer for command output messages.</param>
/// <param name="now">The current time for resolving relative time specifications.</param>
/// <param name="cancellationToken">
/// External cancellation (e.g. the CLI's Ctrl-C handler) — cancelling this
/// stops the export promptly at either the headers wait or the next stream
/// read, with a "cancelled" (not timed-out) message.
/// </param>
/// <param name="headersTimeout">
/// Override for <see cref="DefaultHeadersTimeout"/> — test seam.
/// </param>
/// <param name="inactivityTimeout">
/// Override for <see cref="DefaultInactivityTimeout"/> — test seam.
/// </param>
/// <returns>0 on success, 1 on general error, or 2 on authorization failure.</returns>
public static async Task<int> RunExportAsync(
ManagementHttpClient client, AuditExportArgs args, TextWriter output, DateTimeOffset now)
ManagementHttpClient client, AuditExportArgs args, TextWriter output, DateTimeOffset now,
CancellationToken cancellationToken = default,
TimeSpan? headersTimeout = null,
TimeSpan? inactivityTimeout = null)
{
var qs = BuildQueryString(args, now);
using var headersCts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
headersCts.CancelAfter(headersTimeout ?? DefaultHeadersTimeout);
HttpResponseMessage response;
try
{
response = await client.SendGetStreamAsync("api/audit/export" + qs, CancellationToken.None);
response = await client.SendGetStreamAsync("api/audit/export" + qs, headersCts.Token);
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
OutputFormatter.WriteError("Export cancelled.", "CANCELLED");
return 1;
}
catch (OperationCanceledException)
{
OutputFormatter.WriteError("Request timed out waiting for the server to respond.", "TIMEOUT");
return 1;
}
catch (HttpRequestException ex)
{
@@ -138,7 +189,7 @@ public static class AuditExportHelpers
{
if (response.StatusCode == HttpStatusCode.NotImplemented)
{
var message = await response.Content.ReadAsStringAsync();
var message = await response.Content.ReadAsStringAsync(cancellationToken);
OutputFormatter.WriteError(
string.IsNullOrWhiteSpace(message)
? "Export format not implemented by the server."
@@ -149,7 +200,7 @@ public static class AuditExportHelpers
if (!response.IsSuccessStatusCode)
{
var message = await response.Content.ReadAsStringAsync();
var message = await response.Content.ReadAsStringAsync(cancellationToken);
// Honour the documented "authorization failure → exit 2"
// contract on the REST audit surface as well. HTTP 403 is the
// primary signal; the server may also surface UNAUTHORIZED /
@@ -164,17 +215,69 @@ public static class AuditExportHelpers
return isAuthFailure ? 2 : 1;
}
await using var source = await response.Content.ReadAsStreamAsync();
await using var source = await response.Content.ReadAsStreamAsync(cancellationToken);
await using var destination = new FileStream(
args.Output, FileMode.Create, FileAccess.Write, FileShare.None,
bufferSize: 81920, useAsync: true);
await source.CopyToAsync(destination);
try
{
await CopyWithInactivityTimeoutAsync(
source, destination, inactivityTimeout ?? DefaultInactivityTimeout, cancellationToken);
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
OutputFormatter.WriteError($"Export cancelled; partial file written to {args.Output}.", "CANCELLED");
return 1;
}
catch (OperationCanceledException)
{
OutputFormatter.WriteError(
"Export stalled — no data received from the server for "
+ $"{(inactivityTimeout ?? DefaultInactivityTimeout).TotalSeconds:0}s; "
+ $"partial file written to {args.Output}.",
"TIMEOUT");
return 1;
}
catch (Exception ex) when (ex is IOException or HttpRequestException)
{
OutputFormatter.WriteError(
$"Export failed while streaming the response: {ex.Message}; partial file written to {args.Output}.",
"STREAM_FAILED");
return 1;
}
}
output.WriteLine($"Exported audit log to {args.Output}");
return 0;
}
/// <summary>
/// Copies <paramref name="source"/> to <paramref name="destination"/> in fixed-size
/// chunks, rearming an inactivity deadline before every read so the copy is bounded
/// only by silence, not by total size — a multi-hundred-MB export keeps streaming as
/// long as bytes keep arriving. <paramref name="cancellationToken"/> (external, e.g.
/// Ctrl-C) and the inactivity deadline share one linked token; callers distinguish the
/// two causes via <c>cancellationToken.IsCancellationRequested</c> after catching
/// <see cref="OperationCanceledException"/>.
/// </summary>
private static async Task CopyWithInactivityTimeoutAsync(
Stream source, Stream destination, TimeSpan inactivityTimeout, CancellationToken cancellationToken)
{
using var readCts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
var buffer = new byte[81920];
while (true)
{
readCts.CancelAfter(inactivityTimeout);
var bytesRead = await source.ReadAsync(buffer.AsMemory(), readCts.Token);
if (bytesRead == 0)
{
break;
}
await destination.WriteAsync(buffer.AsMemory(0, bytesRead), cancellationToken);
}
}
/// <summary>
/// Best-effort parse of the server's JSON error envelope (<c>{ "error": ..., "code": ... }</c>)
/// to extract the <c>code</c> field. Returns null if the body is empty, not valid JSON, or
src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Deployment/DebugView.razor
+60 -8
View File
@@ -343,6 +343,16 @@
}
private DebugStreamSession? _session;
/// <summary>
/// Guards the <see cref="_session"/> slot together with <see cref="_disposed"/>
/// (arch-review 2026-08-19 P6-6). <c>Connect</c> awaits <c>StartStreamAsync</c> for up
/// to 30s; the circuit can be torn down inside that window, and <c>Dispose</c> may run
/// on a non-UI thread, so publishing the session and observing disposal must be one
/// atomic step. Whoever loses the race is the one that stops the stream.
/// </summary>
private readonly object _sessionLock = new();
private ToastNotification _toast = default!;
private string? _initError;
@@ -468,7 +478,7 @@
onTerminated: () =>
{
_connected = false;
_session = null;
_ = TakeSession();
// CentralUI-009: skip the toast/render if already disposed.
if (_disposed) return;
_ = SafeInvokeAsync(() =>
@@ -490,7 +500,25 @@
return;
}
_session = session;
// P6-6: the component may have been disposed while StartStreamAsync was in
// flight. Dispose() would have found _session still null and stopped nothing,
// so this session would stay registered in the singleton DebugStreamService —
// whose keepalive renews registered sessions forever, pinning the site
// subscription, the bridge actor and this dead circuit until process restart.
// Publish and re-check disposal atomically so exactly one side tears it down.
bool abandoned;
lock (_sessionLock)
{
abandoned = _disposed;
if (!abandoned) _session = session;
}
if (abandoned)
{
DebugStreamService.StopStream(session.SessionId);
_connecting = false;
return;
}
// Populate initial state from snapshot
_attributeValues.Clear();
@@ -521,12 +549,26 @@
_connecting = false;
}
/// <summary>
/// Atomically clears the <see cref="_session"/> slot and returns what was there, so
/// only the caller that actually took the session stops it (P6-6).
/// </summary>
private DebugStreamSession? TakeSession()
{
lock (_sessionLock)
{
var session = _session;
_session = null;
return session;
}
}
private async Task Disconnect()
{
if (_session != null)
var session = TakeSession();
if (session != null)
{
DebugStreamService.StopStream(_session.SessionId);
_session = null;
DebugStreamService.StopStream(session.SessionId);
}
// Clear persisted selection — user explicitly disconnected
@@ -725,10 +767,20 @@
{
// CentralUI-009: mark disposed first so any in-flight stream callback
// sees the flag and no-ops, then stop the stream synchronously.
_disposed = true;
if (_session != null)
// P6-6: the flag is set and the session taken under the same lock Connect
// publishes through, so a Connect still awaiting StartStreamAsync either
// hands the session over here or stops it itself — never neither.
DebugStreamSession? session;
lock (_sessionLock)
{
DebugStreamService.StopStream(_session.SessionId);
_disposed = true;
session = _session;
_session = null;
}
if (session != null)
{
DebugStreamService.StopStream(session.SessionId);
}
}
}
src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Design/DataConnections.razor
+11 -2
View File
@@ -8,6 +8,7 @@
@attribute [Authorize(Policy = AuthorizationPolicies.RequireDesign)]
@implements IDisposable
@inject ISiteRepository SiteRepository
@inject ZB.MOM.WW.ScadaBridge.CentralUI.Auth.SiteScopeService SiteScope
@inject IConnectionHealthQueryService ConnectionHealthQuery
@inject NavigationManager NavigationManager
@inject IDialogService Dialog
@@ -305,8 +306,16 @@
_errorMessage = null;
try
{
var sites = await SiteRepository.GetAllSitesAsync();
_connections = (await SiteRepository.GetAllDataConnectionsAsync()).ToList();
// Site scoping (arch-review 2026-08-19 P6-4): this page reads the repository
// directly, bypassing ManagementActor's scope gate, and Designer is NOT
// exempt from site scope (see SiteScopeService) — so a scoped Designer used
// to see every site's connections here. Filter the sites, then keep only the
// connections belonging to them, mirroring EventLogs/AlarmSummary.
var sites = await SiteScope.FilterSitesAsync(await SiteRepository.GetAllSitesAsync());
var permittedSiteIds = sites.Select(s => s.Id).ToHashSet();
_connections = (await SiteRepository.GetAllDataConnectionsAsync())
.Where(c => permittedSiteIds.Contains(c.SiteId))
.ToList();
var connBySite = _connections.GroupBy(c => c.SiteId).ToDictionary(g => g.Key, g => g.ToList());
_treeRoots = sites.Select(site => new DcTreeNode(
src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/AuditLogExportService.cs
+48 -19
View File
@@ -20,8 +20,9 @@ namespace ZB.MOM.WW.ScadaBridge.CentralUI.Services;
/// Output is capped at a caller-supplied <c>maxRows</c> ceiling; when the cap
/// is hit the service appends a <c># Capped at … rows. Use the CLI for larger
/// exports.</c> footer line so an operator can tell a truncated download from
/// a complete one. The header row contains the 21 columns of
/// <see cref="AuditEvent"/> in declaration order.
/// a complete one. The header row contains the same 24 columns, in the same
/// order, as the ManagementService CLI's CSV export (arch-review 2026-08-19
/// P6-7 — the two CSV writers were found to have drifted and are kept aligned).
/// </para>
/// </summary>
public interface IAuditLogExportService
@@ -92,7 +93,7 @@ public sealed class AuditLogExportService : IAuditLogExportService
leaveOpen: true);
writer.NewLine = "\r\n"; // RFC 4180
// Header — 21 columns in AuditEvent declaration order.
// Header — 24 columns in AuditEventView declaration order.
await writer.WriteLineAsync(Header);
int written = 0;
@@ -160,13 +161,14 @@ public sealed class AuditLogExportService : IAuditLogExportService
}
// ─────────────────────────────────────────────────────────────────────
// CSV helpers
// CSV helpers — 24 columns, kept aligned with the ManagementService CLI's
// CSV export (arch-review 2026-08-19 P6-7)
// ─────────────────────────────────────────────────────────────────────
/// <summary>The 21 column names in <see cref="AuditEvent"/> declaration order.</summary>
/// <summary>The 24 column names, in <see cref="AuditEventView"/> declaration order.</summary>
internal const string Header =
"EventId,OccurredAtUtc,IngestedAtUtc,Channel,Kind,CorrelationId," +
"SourceSiteId,SourceInstanceId,SourceScript,Actor,Target,Status," +
"EventId,OccurredAtUtc,IngestedAtUtc,Channel,Kind,CorrelationId,ExecutionId,ParentExecutionId," +
"SourceSiteId,SourceNode,SourceInstanceId,SourceScript,Actor,Target,Status," +
"HttpStatus,DurationMs,ErrorMessage,ErrorDetail,RequestSummary," +
"ResponseSummary,PayloadTruncated,Extra,ForwardState";
@@ -174,7 +176,12 @@ public sealed class AuditLogExportService : IAuditLogExportService
/// Serialises one <see cref="AuditEventView"/> as a CSV row (no trailing newline).
/// Each nullable column renders as the empty string when null; non-null
/// scalars use invariant culture so an export taken on one locale parses
/// cleanly on another.
/// cleanly on another. Free-text columns are run through
/// <see cref="NeedsFormulaNeutralization"/> before RFC 4180 quoting so a
/// leading <c>=</c>/<c>+</c>/<c>-</c>/<c>@</c>/TAB/CR can't be interpreted as
/// a formula or DDE trigger by a spreadsheet application (CWE-1236;
/// arch-review 2026-08-19 P6-7). GUID/enum/numeric/date/bool columns this
/// formatter generates itself are never neutralized.
/// </summary>
/// <param name="evt">The audit event view to format as a CSV row.</param>
/// <returns>An RFC 4180 CSV row string (no trailing newline) for the supplied event.</returns>
@@ -187,20 +194,23 @@ public sealed class AuditLogExportService : IAuditLogExportService
AppendField(sb, evt.Channel.ToString());
AppendField(sb, evt.Kind.ToString());
AppendField(sb, evt.CorrelationId?.ToString());
AppendField(sb, evt.SourceSiteId);
AppendField(sb, evt.SourceInstanceId);
AppendField(sb, evt.SourceScript);
AppendField(sb, evt.Actor);
AppendField(sb, evt.Target);
AppendField(sb, evt.ExecutionId?.ToString());
AppendField(sb, evt.ParentExecutionId?.ToString());
AppendField(sb, evt.SourceSiteId, freeText: true);
AppendField(sb, evt.SourceNode, freeText: true);
AppendField(sb, evt.SourceInstanceId, freeText: true);
AppendField(sb, evt.SourceScript, freeText: true);
AppendField(sb, evt.Actor, freeText: true);
AppendField(sb, evt.Target, freeText: true);
AppendField(sb, evt.Status.ToString());
AppendField(sb, evt.HttpStatus?.ToString(CultureInfo.InvariantCulture));
AppendField(sb, evt.DurationMs?.ToString(CultureInfo.InvariantCulture));
AppendField(sb, evt.ErrorMessage);
AppendField(sb, evt.ErrorDetail);
AppendField(sb, evt.RequestSummary);
AppendField(sb, evt.ResponseSummary);
AppendField(sb, evt.ErrorMessage, freeText: true);
AppendField(sb, evt.ErrorDetail, freeText: true);
AppendField(sb, evt.RequestSummary, freeText: true);
AppendField(sb, evt.ResponseSummary, freeText: true);
AppendField(sb, evt.PayloadTruncated.ToString());
AppendField(sb, evt.Extra);
AppendField(sb, evt.Extra, freeText: true);
AppendField(sb, evt.ForwardState?.ToString());
return sb.ToString();
}
@@ -211,7 +221,7 @@ public sealed class AuditLogExportService : IAuditLogExportService
private static string FormatDate(DateTime value) =>
value.ToString("O", CultureInfo.InvariantCulture);
private static void AppendField(StringBuilder sb, string? value, bool first = false)
private static void AppendField(StringBuilder sb, string? value, bool first = false, bool freeText = false)
{
if (!first) sb.Append(',');
if (string.IsNullOrEmpty(value))
@@ -219,6 +229,14 @@ public sealed class AuditLogExportService : IAuditLogExportService
return;
}
// OWASP CSV-injection neutralization (CWE-1236): prefix a literal single
// quote before RFC 4180 quoting so a free-text cell starting with a
// formula/DDE trigger character renders as inert text in Excel.
if (freeText && NeedsFormulaNeutralization(value))
{
value = "'" + value;
}
// RFC 4180: quote on comma / quote / CR / LF; double-up embedded quotes.
bool needsQuoting = value.IndexOfAny(s_quoteTriggers) >= 0;
if (!needsQuoting)
@@ -242,5 +260,16 @@ public sealed class AuditLogExportService : IAuditLogExportService
sb.Append('"');
}
/// <summary>
/// True when <paramref name="value"/> starts with a character a spreadsheet
/// application would treat as a formula/DDE-launch trigger — the OWASP CSV
/// Injection standard set: <c>=</c>, <c>+</c>, <c>-</c>, <c>@</c>, TAB, CR.
/// </summary>
private static bool NeedsFormulaNeutralization(string value)
{
var c = value[0];
return c == '=' || c == '+' || c == '-' || c == '@' || c == '\t' || c == '\r';
}
private static readonly char[] s_quoteTriggers = { ',', '"', '\r', '\n' };
}
src/ZB.MOM.WW.ScadaBridge.Commons/Types/Audit/AuditQueryParamParsers.cs
+15 -2
View File
@@ -79,7 +79,8 @@ public static class AuditQueryParamParsers
/// the filter dimension stays unconstrained.
/// </summary>
/// <remarks>
/// Register row 99. Two traps this closes that a bare
/// Register row 99 (closure defect fixed by arch-review 2026-08-19 P6-3). Three
/// traps this closes that a bare
/// <see cref="Enum.TryParse{TEnum}(string, bool, out TEnum)"/> does not:
/// <list type="number">
/// <item><b>Numeric strings.</b> <c>TryParse</c> happily converts <c>"7"</c> —
@@ -87,6 +88,17 @@ public static class AuditQueryParamParsers
/// is a caller-facing name-based contract, never an ordinal one, so any value
/// starting with a digit or sign is rejected outright. Enum member names can
/// never start with those characters, so the check costs no legitimate input.</item>
/// <item><b>Comma-joined tokens.</b> <c>TryParse</c> splits ANY comma-separated
/// string on <c>,</c> and bitwise-ORs the pieces together — regardless of
/// whether <typeparamref name="TEnum"/> is a <c>[Flags]</c> enum. For a dense
/// 0-based enum like <c>AuditStatus</c> that composite almost always collides
/// with a real, but WRONG, member (<c>"Forwarded,Attempted"</c> = 1|2 = 3 =
/// <c>Delivered</c>), and it smuggles an ordinal past the numeric guard above
/// (<c>"Failed,7"</c> = 7 = <c>Skipped</c>). A single token containing <c>,</c>
/// is therefore rejected outright, before <c>TryParse</c> ever sees it — a
/// value that legitimately arrives as multiple tokens (a repeated query
/// parameter, or an already-split list) is unaffected, since each of those
/// tokens is comma-free on its own.</item>
/// <item><b>Undefined composites.</b> <see cref="Enum.IsDefined{TEnum}(TEnum)"/>
/// catches anything that parsed to a value with no declared member.</item>
/// </list>
@@ -95,7 +107,7 @@ public static class AuditQueryParamParsers
/// <param name="rawValues">Raw query-parameter string values to parse; may be null.</param>
/// <param name="parameterName">Query-parameter name, used in the error message (e.g. <c>"status"</c>).</param>
/// <returns>A non-empty list of parsed values, or <c>null</c> if no values were supplied.</returns>
/// <exception cref="AuditFilterParseException">A supplied value is not a member name of <typeparamref name="TEnum"/>.</exception>
/// <exception cref="AuditFilterParseException">A supplied value is not a member name of <typeparamref name="TEnum"/>, or a single token contains a comma.</exception>
public static IReadOnlyList<TEnum>? ParseEnumListStrict<TEnum>(
IEnumerable<string?>? rawValues, string parameterName)
where TEnum : struct, Enum
@@ -115,6 +127,7 @@ public static class AuditQueryParamParsers
var trimmed = raw.Trim();
if (LooksNumeric(trimmed)
|| trimmed.Contains(',')
|| !Enum.TryParse<TEnum>(trimmed, ignoreCase: true, out var value)
|| !Enum.IsDefined(value))
{
src/ZB.MOM.WW.ScadaBridge.Communication/CentralControlGrpc/CentralControl.cs
+166 -38
View File
@@ -72,7 +72,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
"ZXkYASABKAkSQQoFdmFsdWUYAiABKAsyMi5zY2FkYWJyaWRnZS5jZW50cmFs",
"Y29udHJvbC52MS5UYWdRdWFsaXR5Q291bnRzRHRvOgI4ASJQChFOb2RlU3Rh",
"dHVzTGlzdER0bxI7CgVub2RlcxgBIAMoCzIsLnNjYWRhYnJpZGdlLmNlbnRy",
"YWxjb250cm9sLnYxLk5vZGVTdGF0dXNEdG8iyQ4KE1NpdGVIZWFsdGhSZXBv",
"YWxjb250cm9sLnYxLk5vZGVTdGF0dXNEdG8irw8KE1NpdGVIZWFsdGhSZXBv",
"cnREdG8SDwoHc2l0ZV9pZBgBIAEoCRIXCg9zZXF1ZW5jZV9udW1iZXIYAiAB",
"KAMSNAoQcmVwb3J0X3RpbWVzdGFtcBgDIAEoCzIaLmdvb2dsZS5wcm90b2J1",
"Zi5UaW1lc3RhbXAScAoYZGF0YV9jb25uZWN0aW9uX3N0YXR1c2VzGAQgAygL",
@@ -107,42 +107,45 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
"ASgLMhouZ29vZ2xlLnByb3RvYnVmLkJvb2xWYWx1ZRI7ChZsb2NhbF9kYl9v",
"cGxvZ19iYWNrbG9nGBwgASgLMhsuZ29vZ2xlLnByb3RvYnVmLkludDY0VmFs",
"dWUSPQoZc3RhcnR1cF9yZWNvbmNpbGVyX2FjdGl2ZRgdIAEoCzIaLmdvb2ds",
"ZS5wcm90b2J1Zi5Cb29sVmFsdWUacgobRGF0YUNvbm5lY3Rpb25TdGF0dXNl",
"c0VudHJ5EgsKA2tleRgBIAEoCRJCCgV2YWx1ZRgCIAEoDjIzLnNjYWRhYnJp",
"ZGdlLmNlbnRyYWxjb250cm9sLnYxLkNvbm5lY3Rpb25IZWFsdGhFbnVtOgI4",
"ARpxChhUYWdSZXNvbHV0aW9uQ291bnRzRW50cnkSCwoDa2V5GAEgASgJEkQK",
"BXZhbHVlGAIgASgLMjUuc2NhZGFicmlkZ2UuY2VudHJhbGNvbnRyb2wudjEu",
"VGFnUmVzb2x1dGlvblN0YXR1c0R0bzoCOAEaQgogU3RvcmVBbmRGb3J3YXJk",
"QnVmZmVyRGVwdGhzRW50cnkSCwoDa2V5GAEgASgJEg0KBXZhbHVlGAIgASgF",
"OgI4ASJjChZTaXRlSGVhbHRoUmVwb3J0QWNrRHRvEg8KB3NpdGVfaWQYASAB",
"KAkSFwoPc2VxdWVuY2VfbnVtYmVyGAIgASgDEhAKCGFjY2VwdGVkGAMgASgI",
"Eg0KBWVycm9yGAQgASgJIosBCgxIZWFydGJlYXREdG8SDwoHc2l0ZV9pZBgB",
"IAEoCRIVCg1ub2RlX2hvc3RuYW1lGAIgASgJEhEKCWlzX2FjdGl2ZRgDIAEo",
"CBItCgl0aW1lc3RhbXAYBCABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0",
"YW1wEhEKCXN5bnRoZXRpYxgFIAEoCCq9AQoUQ29ubmVjdGlvbkhlYWx0aEVu",
"dW0SIQodQ09OTkVDVElPTl9IRUFMVEhfVU5TUEVDSUZJRUQQABIfChtDT05O",
"RUNUSU9OX0hFQUxUSF9DT05ORUNURUQQARIiCh5DT05ORUNUSU9OX0hFQUxU",
"SF9ESVNDT05ORUNURUQQAhIgChxDT05ORUNUSU9OX0hFQUxUSF9DT05ORUNU",
"SU5HEAMSGwoXQ09OTkVDVElPTl9IRUFMVEhfRVJST1IQBDKhBgoVQ2VudHJh",
"bENvbnRyb2xTZXJ2aWNlEoMBChJTdWJtaXROb3RpZmljYXRpb24SNC5zY2Fk",
"YWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5Ob3RpZmljYXRpb25TdWJtaXRE",
"dG8aNy5zY2FkYWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5Ob3RpZmljYXRp",
"b25TdWJtaXRBY2tEdG8SkgEKF1F1ZXJ5Tm90aWZpY2F0aW9uU3RhdHVzEjku",
"c2NhZGFicmlkZ2UuY2VudHJhbGNvbnRyb2wudjEuTm90aWZpY2F0aW9uU3Rh",
"dHVzUXVlcnlEdG8aPC5zY2FkYWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5O",
"b3RpZmljYXRpb25TdGF0dXNSZXNwb25zZUR0bxJHChFJbmdlc3RBdWRpdEV2",
"ZW50cxIbLnNpdGVzdHJlYW0uQXVkaXRFdmVudEJhdGNoGhUuc2l0ZXN0cmVh",
"bS5Jbmdlc3RBY2sSUAoVSW5nZXN0Q2FjaGVkVGVsZW1ldHJ5EiAuc2l0ZXN0",
"cmVhbS5DYWNoZWRUZWxlbWV0cnlCYXRjaBoVLnNpdGVzdHJlYW0uSW5nZXN0",
"QWNrEoABCg1SZWNvbmNpbGVTaXRlEjYuc2NhZGFicmlkZ2UuY2VudHJhbGNv",
"bnRyb2wudjEuUmVjb25jaWxlU2l0ZVJlcXVlc3REdG8aNy5zY2FkYWJyaWRn",
"ZS5jZW50cmFsY29udHJvbC52MS5SZWNvbmNpbGVTaXRlUmVzcG9uc2VEdG8S",
"fQoQUmVwb3J0U2l0ZUhlYWx0aBIyLnNjYWRhYnJpZGdlLmNlbnRyYWxjb250",
"cm9sLnYxLlNpdGVIZWFsdGhSZXBvcnREdG8aNS5zY2FkYWJyaWRnZS5jZW50",
"cmFsY29udHJvbC52MS5TaXRlSGVhbHRoUmVwb3J0QWNrRHRvElAKCUhlYXJ0",
"YmVhdBIrLnNjYWRhYnJpZGdlLmNlbnRyYWxjb250cm9sLnYxLkhlYXJ0YmVh",
"dER0bxoWLmdvb2dsZS5wcm90b2J1Zi5FbXB0eUIrqgIoWkIuTU9NLldXLlNj",
"YWRhQnJpZGdlLkNvbW11bmljYXRpb24uR3JwY2IGcHJvdG8z"));
"ZS5wcm90b2J1Zi5Cb29sVmFsdWUSHwoXZGV0YWNoZWRfc2NyaXB0X3RocmVh",
"ZHMYHiABKAUSHQoVc2NyaXB0X3J1bl9zaGVkX2NvdW50GB8gASgFEiQKHHNp",
"dGVfc3RyZWFtX2FsYXJtX2Ryb3BfY291bnQYICABKAMacgobRGF0YUNvbm5l",
"Y3Rpb25TdGF0dXNlc0VudHJ5EgsKA2tleRgBIAEoCRJCCgV2YWx1ZRgCIAEo",
"DjIzLnNjYWRhYnJpZGdlLmNlbnRyYWxjb250cm9sLnYxLkNvbm5lY3Rpb25I",
"ZWFsdGhFbnVtOgI4ARpxChhUYWdSZXNvbHV0aW9uQ291bnRzRW50cnkSCwoD",
"a2V5GAEgASgJEkQKBXZhbHVlGAIgASgLMjUuc2NhZGFicmlkZ2UuY2VudHJh",
"bGNvbnRyb2wudjEuVGFnUmVzb2x1dGlvblN0YXR1c0R0bzoCOAEaQgogU3Rv",
"cmVBbmRGb3J3YXJkQnVmZmVyRGVwdGhzRW50cnkSCwoDa2V5GAEgASgJEg0K",
"BXZhbHVlGAIgASgFOgI4ASJjChZTaXRlSGVhbHRoUmVwb3J0QWNrRHRvEg8K",
"B3NpdGVfaWQYASABKAkSFwoPc2VxdWVuY2VfbnVtYmVyGAIgASgDEhAKCGFj",
"Y2VwdGVkGAMgASgIEg0KBWVycm9yGAQgASgJIosBCgxIZWFydGJlYXREdG8S",
"DwoHc2l0ZV9pZBgBIAEoCRIVCg1ub2RlX2hvc3RuYW1lGAIgASgJEhEKCWlz",
"X2FjdGl2ZRgDIAEoCBItCgl0aW1lc3RhbXAYBCABKAsyGi5nb29nbGUucHJv",
"dG9idWYuVGltZXN0YW1wEhEKCXN5bnRoZXRpYxgFIAEoCCq9AQoUQ29ubmVj",
"dGlvbkhlYWx0aEVudW0SIQodQ09OTkVDVElPTl9IRUFMVEhfVU5TUEVDSUZJ",
"RUQQABIfChtDT05ORUNUSU9OX0hFQUxUSF9DT05ORUNURUQQARIiCh5DT05O",
"RUNUSU9OX0hFQUxUSF9ESVNDT05ORUNURUQQAhIgChxDT05ORUNUSU9OX0hF",
"QUxUSF9DT05ORUNUSU5HEAMSGwoXQ09OTkVDVElPTl9IRUFMVEhfRVJST1IQ",
"BDKhBgoVQ2VudHJhbENvbnRyb2xTZXJ2aWNlEoMBChJTdWJtaXROb3RpZmlj",
"YXRpb24SNC5zY2FkYWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5Ob3RpZmlj",
"YXRpb25TdWJtaXREdG8aNy5zY2FkYWJyaWRnZS5jZW50cmFsY29udHJvbC52",
"MS5Ob3RpZmljYXRpb25TdWJtaXRBY2tEdG8SkgEKF1F1ZXJ5Tm90aWZpY2F0",
"aW9uU3RhdHVzEjkuc2NhZGFicmlkZ2UuY2VudHJhbGNvbnRyb2wudjEuTm90",
"aWZpY2F0aW9uU3RhdHVzUXVlcnlEdG8aPC5zY2FkYWJyaWRnZS5jZW50cmFs",
"Y29udHJvbC52MS5Ob3RpZmljYXRpb25TdGF0dXNSZXNwb25zZUR0bxJHChFJ",
"bmdlc3RBdWRpdEV2ZW50cxIbLnNpdGVzdHJlYW0uQXVkaXRFdmVudEJhdGNo",
"GhUuc2l0ZXN0cmVhbS5Jbmdlc3RBY2sSUAoVSW5nZXN0Q2FjaGVkVGVsZW1l",
"dHJ5EiAuc2l0ZXN0cmVhbS5DYWNoZWRUZWxlbWV0cnlCYXRjaBoVLnNpdGVz",
"dHJlYW0uSW5nZXN0QWNrEoABCg1SZWNvbmNpbGVTaXRlEjYuc2NhZGFicmlk",
"Z2UuY2VudHJhbGNvbnRyb2wudjEuUmVjb25jaWxlU2l0ZVJlcXVlc3REdG8a",
"Ny5zY2FkYWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5SZWNvbmNpbGVTaXRl",
"UmVzcG9uc2VEdG8SfQoQUmVwb3J0U2l0ZUhlYWx0aBIyLnNjYWRhYnJpZGdl",
"LmNlbnRyYWxjb250cm9sLnYxLlNpdGVIZWFsdGhSZXBvcnREdG8aNS5zY2Fk",
"YWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5TaXRlSGVhbHRoUmVwb3J0QWNr",
"RHRvElAKCUhlYXJ0YmVhdBIrLnNjYWRhYnJpZGdlLmNlbnRyYWxjb250cm9s",
"LnYxLkhlYXJ0YmVhdER0bxoWLmdvb2dsZS5wcm90b2J1Zi5FbXB0eUIrqgIo",
"WkIuTU9NLldXLlNjYWRhQnJpZGdlLkNvbW11bmljYXRpb24uR3JwY2IGcHJv",
"dG8z"));
descriptor = pbr::FileDescriptor.FromGeneratedCode(descriptorData,
new pbr::FileDescriptor[] { global::Google.Protobuf.WellKnownTypes.EmptyReflection.Descriptor, global::Google.Protobuf.WellKnownTypes.TimestampReflection.Descriptor, global::Google.Protobuf.WellKnownTypes.WrappersReflection.Descriptor, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor, },
new pbr::GeneratedClrTypeInfo(new[] {typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ConnectionHealthEnum), }, null, new pbr::GeneratedClrTypeInfo[] {
@@ -160,7 +163,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ConnectionEndpointMapDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ConnectionEndpointMapDto.Parser, new[]{ "Entries" }, null, null, null, new pbr::GeneratedClrTypeInfo[] { null, }),
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TagQualityMapDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TagQualityMapDto.Parser, new[]{ "Entries" }, null, null, null, new pbr::GeneratedClrTypeInfo[] { null, }),
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NodeStatusListDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NodeStatusListDto.Parser, new[]{ "Nodes" }, null, null, null, null),
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto.Parser, new[]{ "SiteId", "SequenceNumber", "ReportTimestamp", "DataConnectionStatuses", "TagResolutionCounts", "ScriptErrorCount", "AlarmEvaluationErrorCount", "StoreAndForwardBufferDepths", "DeadLetterCount", "DeployedInstanceCount", "EnabledInstanceCount", "DisabledInstanceCount", "NodeRole", "NodeHostname", "DataConnectionEndpoints", "DataConnectionTagQuality", "ParkedMessageCount", "ClusterNodes", "SiteAuditWriteFailures", "AuditRedactionFailure", "SiteAuditBacklog", "SiteEventLogWriteFailures", "OldestParkedMessageAgeSeconds", "ScriptQueueDepth", "ScriptBusyThreads", "ScriptOldestBusyAgeSeconds", "LocalDbReplicationConnected", "LocalDbOplogBacklog", "StartupReconcilerActive" }, null, null, null, new pbr::GeneratedClrTypeInfo[] { null, null, null, }),
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto.Parser, new[]{ "SiteId", "SequenceNumber", "ReportTimestamp", "DataConnectionStatuses", "TagResolutionCounts", "ScriptErrorCount", "AlarmEvaluationErrorCount", "StoreAndForwardBufferDepths", "DeadLetterCount", "DeployedInstanceCount", "EnabledInstanceCount", "DisabledInstanceCount", "NodeRole", "NodeHostname", "DataConnectionEndpoints", "DataConnectionTagQuality", "ParkedMessageCount", "ClusterNodes", "SiteAuditWriteFailures", "AuditRedactionFailure", "SiteAuditBacklog", "SiteEventLogWriteFailures", "OldestParkedMessageAgeSeconds", "ScriptQueueDepth", "ScriptBusyThreads", "ScriptOldestBusyAgeSeconds", "LocalDbReplicationConnected", "LocalDbOplogBacklog", "StartupReconcilerActive", "DetachedScriptThreads", "ScriptRunShedCount", "SiteStreamAlarmDropCount" }, null, null, null, new pbr::GeneratedClrTypeInfo[] { null, null, null, }),
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto.Parser, new[]{ "SiteId", "SequenceNumber", "Accepted", "Error" }, null, null, null, null),
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto.Parser, new[]{ "SiteId", "NodeHostname", "IsActive", "Timestamp", "Synthetic" }, null, null, null, null)
}));
@@ -4268,6 +4271,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
LocalDbReplicationConnected = other.LocalDbReplicationConnected;
LocalDbOplogBacklog = other.LocalDbOplogBacklog;
StartupReconcilerActive = other.StartupReconcilerActive;
detachedScriptThreads_ = other.detachedScriptThreads_;
scriptRunShedCount_ = other.scriptRunShedCount_;
siteStreamAlarmDropCount_ = other.siteStreamAlarmDropCount_;
_unknownFields = pb::UnknownFieldSet.Clone(other._unknownFields);
}
@@ -4665,6 +4671,56 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
}
/// <summary>Field number for the "detached_script_threads" field.</summary>
public const int DetachedScriptThreadsFieldNumber = 30;
private int detachedScriptThreads_;
/// <summary>
/// WP3.1: script-execution worker threads detached and replaced by the stuck-script
/// watchdog and not yet exited. Point-in-time gauge (like script_busy_threads above),
/// not interval-drained. Zero is the healthy state.
/// </summary>
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
public int DetachedScriptThreads {
get { return detachedScriptThreads_; }
set {
detachedScriptThreads_ = value;
}
}
/// <summary>Field number for the "script_run_shed_count" field.</summary>
public const int ScriptRunShedCountFieldNumber = 31;
private int scriptRunShedCount_;
/// <summary>
/// WP3.1: per-interval count of script/alarm on-trigger runs shed by
/// MaxConcurrentRunsPerScript. Interval-drained like script_error_count.
/// </summary>
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
public int ScriptRunShedCount {
get { return scriptRunShedCount_; }
set {
scriptRunShedCount_ = value;
}
}
/// <summary>Field number for the "site_stream_alarm_drop_count" field.</summary>
public const int SiteStreamAlarmDropCountFieldNumber = 32;
private long siteStreamAlarmDropCount_;
/// <summary>
/// WP2.6d: cumulative count of alarm state changes dropped from SiteStreamManager's
/// dedicated alarm publish queue because it was at capacity. Point-in-time
/// (not reset on collect), like site_event_log_write_failures.
/// </summary>
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
public long SiteStreamAlarmDropCount {
get { return siteStreamAlarmDropCount_; }
set {
siteStreamAlarmDropCount_ = value;
}
}
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
public override bool Equals(object other) {
@@ -4709,6 +4765,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
if (LocalDbReplicationConnected != other.LocalDbReplicationConnected) return false;
if (LocalDbOplogBacklog != other.LocalDbOplogBacklog) return false;
if (StartupReconcilerActive != other.StartupReconcilerActive) return false;
if (DetachedScriptThreads != other.DetachedScriptThreads) return false;
if (ScriptRunShedCount != other.ScriptRunShedCount) return false;
if (SiteStreamAlarmDropCount != other.SiteStreamAlarmDropCount) return false;
return Equals(_unknownFields, other._unknownFields);
}
@@ -4745,6 +4804,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
if (localDbReplicationConnected_ != null) hash ^= LocalDbReplicationConnected.GetHashCode();
if (localDbOplogBacklog_ != null) hash ^= LocalDbOplogBacklog.GetHashCode();
if (startupReconcilerActive_ != null) hash ^= StartupReconcilerActive.GetHashCode();
if (DetachedScriptThreads != 0) hash ^= DetachedScriptThreads.GetHashCode();
if (ScriptRunShedCount != 0) hash ^= ScriptRunShedCount.GetHashCode();
if (SiteStreamAlarmDropCount != 0L) hash ^= SiteStreamAlarmDropCount.GetHashCode();
if (_unknownFields != null) {
hash ^= _unknownFields.GetHashCode();
}
@@ -4865,6 +4927,18 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
if (startupReconcilerActive_ != null) {
_single_startupReconcilerActive_codec.WriteTagAndValue(output, StartupReconcilerActive);
}
if (DetachedScriptThreads != 0) {
output.WriteRawTag(240, 1);
output.WriteInt32(DetachedScriptThreads);
}
if (ScriptRunShedCount != 0) {
output.WriteRawTag(248, 1);
output.WriteInt32(ScriptRunShedCount);
}
if (SiteStreamAlarmDropCount != 0L) {
output.WriteRawTag(128, 2);
output.WriteInt64(SiteStreamAlarmDropCount);
}
if (_unknownFields != null) {
_unknownFields.WriteTo(output);
}
@@ -4977,6 +5051,18 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
if (startupReconcilerActive_ != null) {
_single_startupReconcilerActive_codec.WriteTagAndValue(ref output, StartupReconcilerActive);
}
if (DetachedScriptThreads != 0) {
output.WriteRawTag(240, 1);
output.WriteInt32(DetachedScriptThreads);
}
if (ScriptRunShedCount != 0) {
output.WriteRawTag(248, 1);
output.WriteInt32(ScriptRunShedCount);
}
if (SiteStreamAlarmDropCount != 0L) {
output.WriteRawTag(128, 2);
output.WriteInt64(SiteStreamAlarmDropCount);
}
if (_unknownFields != null) {
_unknownFields.WriteTo(ref output);
}
@@ -5068,6 +5154,15 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
if (startupReconcilerActive_ != null) {
size += _single_startupReconcilerActive_codec.CalculateSizeWithTag(StartupReconcilerActive);
}
if (DetachedScriptThreads != 0) {
size += 2 + pb::CodedOutputStream.ComputeInt32Size(DetachedScriptThreads);
}
if (ScriptRunShedCount != 0) {
size += 2 + pb::CodedOutputStream.ComputeInt32Size(ScriptRunShedCount);
}
if (SiteStreamAlarmDropCount != 0L) {
size += 2 + pb::CodedOutputStream.ComputeInt64Size(SiteStreamAlarmDropCount);
}
if (_unknownFields != null) {
size += _unknownFields.CalculateSize();
}
@@ -5186,6 +5281,15 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
StartupReconcilerActive = other.StartupReconcilerActive;
}
}
if (other.DetachedScriptThreads != 0) {
DetachedScriptThreads = other.DetachedScriptThreads;
}
if (other.ScriptRunShedCount != 0) {
ScriptRunShedCount = other.ScriptRunShedCount;
}
if (other.SiteStreamAlarmDropCount != 0L) {
SiteStreamAlarmDropCount = other.SiteStreamAlarmDropCount;
}
_unknownFields = pb::UnknownFieldSet.MergeFrom(_unknownFields, other._unknownFields);
}
@@ -5351,6 +5455,18 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
}
break;
}
case 240: {
DetachedScriptThreads = input.ReadInt32();
break;
}
case 248: {
ScriptRunShedCount = input.ReadInt32();
break;
}
case 256: {
SiteStreamAlarmDropCount = input.ReadInt64();
break;
}
}
}
#endif
@@ -5516,6 +5632,18 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
}
break;
}
case 240: {
DetachedScriptThreads = input.ReadInt32();
break;
}
case 248: {
ScriptRunShedCount = input.ReadInt32();
break;
}
case 256: {
SiteStreamAlarmDropCount = input.ReadInt64();
break;
}
}
}
}
src/ZB.MOM.WW.ScadaBridge.Communication/DebugStreamService.cs
+24 -6
View File
@@ -81,7 +81,12 @@ public class DebugStreamService : IDisposable
/// <param name="onTerminated">Callback invoked when the stream terminates.</param>
/// <param name="ct">A cancellation token that can be used to cancel the operation.</param>
/// <returns>A debug stream session with the initial snapshot.</returns>
public async Task<DebugStreamSession> StartStreamAsync(
/// <remarks>
/// Virtual purely as a test seam: the consumers' dispose-vs-start race (P6-6) can only
/// be driven deterministically by a stub that holds the start open. Behaviour and the
/// keepalive/orphan contract are unchanged.
/// </remarks>
public virtual async Task<DebugStreamSession> StartStreamAsync(
int instanceId,
Action<object> onEvent,
Action onTerminated,
@@ -108,10 +113,22 @@ public class DebugStreamService : IDisposable
instanceUniqueName = instance.UniqueName;
siteIdentifier = site.SiteIdentifier;
grpcNodeAAddress = site.GrpcNodeAAddress
?? throw new InvalidOperationException($"Site {siteIdentifier} has no GrpcNodeAAddress configured.");
grpcNodeBAddress = site.GrpcNodeBAddress
?? throw new InvalidOperationException($"Site {siteIdentifier} has no GrpcNodeBAddress configured.");
// Single-endpoint site (arch-review 2026-08-19 P3-3, applying the same N9
// degeneration SiteAlarmLiveCacheService.ResolveSiteAsync already uses): reuse the
// sole configured endpoint for both slots, so the bridge actor's NodeA↔NodeB
// failover flip degenerates to a reconnect against the same node. Requiring BOTH
// addresses made Debug View the only surface that refused a one-node site.
var nodeA = site.GrpcNodeAAddress;
var nodeB = site.GrpcNodeBAddress;
if (string.IsNullOrWhiteSpace(nodeA) && string.IsNullOrWhiteSpace(nodeB))
{
throw new InvalidOperationException(
$"Site {siteIdentifier} has no gRPC node addresses configured.");
}
grpcNodeAAddress = !string.IsNullOrWhiteSpace(nodeA) ? nodeA! : nodeB!;
grpcNodeBAddress = !string.IsNullOrWhiteSpace(nodeB) ? nodeB! : nodeA!;
}
var sessionId = Guid.NewGuid().ToString("N");
@@ -197,7 +214,8 @@ public class DebugStreamService : IDisposable
/// Stops an active debug stream session.
/// </summary>
/// <param name="sessionId">The session ID of the debug stream to stop.</param>
public void StopStream(string sessionId)
/// <remarks>Virtual purely as a test seam — see <see cref="StartStreamAsync"/>.</remarks>
public virtual void StopStream(string sessionId)
{
if (_sessions.TryRemove(sessionId, out var bridgeActor))
{
src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/CentralControlDtoMapper.cs
+6
View File
@@ -476,6 +476,9 @@ public static class CentralControlDtoMapper
LocalDbReplicationConnected = msg.LocalDbReplicationConnected,
LocalDbOplogBacklog = msg.LocalDbOplogBacklog,
StartupReconcilerActive = msg.StartupReconcilerActive,
DetachedScriptThreads = msg.DetachedScriptThreads,
ScriptRunShedCount = msg.ScriptRunShedCount,
SiteStreamAlarmDropCount = msg.SiteStreamAlarmDropCount,
};
foreach (var (name, health) in msg.DataConnectionStatuses)
@@ -651,6 +654,9 @@ public static class CentralControlDtoMapper
LocalDbReplicationConnected = dto.LocalDbReplicationConnected,
LocalDbOplogBacklog = dto.LocalDbOplogBacklog,
StartupReconcilerActive = dto.StartupReconcilerActive,
DetachedScriptThreads = dto.DetachedScriptThreads,
ScriptRunShedCount = dto.ScriptRunShedCount,
SiteStreamAlarmDropCount = dto.SiteStreamAlarmDropCount,
};
}
src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SitePairChannelProvider.cs
+96 -7
View File
@@ -53,6 +53,14 @@ public sealed class SitePairChannelProvider : IDisposable
private static readonly TimeSpan MaxBackoff = TimeSpan.FromSeconds(60);
private static readonly TimeSpan FailbackSweepInterval = TimeSpan.FromSeconds(1);
/// <summary>
/// How long a replaced/removed channel is PARKED before it is disposed (arch-review
/// 2026-08-19 P3-2). Sized to exceed the longest command deadline
/// (<c>CommunicationOptions.DeploymentTimeout</c> and friends, minutes at most), so no call
/// that was already in flight when the channel was replaced can still be using it.
/// </summary>
internal static readonly TimeSpan ChannelDisposalGrace = TimeSpan.FromMinutes(5);
private readonly ISitePskProvider _pskProvider;
private readonly CommunicationOptions _options;
private readonly ILogger<SitePairChannelProvider> _logger;
@@ -60,6 +68,14 @@ public sealed class SitePairChannelProvider : IDisposable
private readonly Func<GrpcChannel, CancellationToken, Task<bool>> _reachabilityProbe;
private readonly ConcurrentDictionary<string, SitePair> _sites = new(StringComparer.Ordinal);
/// <summary>
/// Channels replaced by <see cref="UpdateSite"/> or dropped by <see cref="RemoveSite"/>,
/// each with the instant it becomes safe to dispose. FIFO order is disposal order because
/// the grace window is constant.
/// </summary>
private readonly ConcurrentQueue<(GrpcChannel Channel, DateTime DisposeAfterUtc)> _graveyard = new();
private readonly CancellationTokenSource _shutdown = new();
private Task? _failbackLoop;
private readonly object _loopGate = new();
@@ -98,8 +114,9 @@ public sealed class SitePairChannelProvider : IDisposable
/// <summary>
/// (Re)builds a site's channel pair from its gRPC node endpoints. A no-op when neither endpoint
/// changed. Disposes and rebuilds a channel whose endpoint changed, and resets stickiness to the
/// preferred node when the pair is (re)created. Called on the DB refresh tick.
/// changed. Parks (see <see cref="ChannelDisposalGrace"/>) and rebuilds a channel whose endpoint
/// changed, and resets stickiness to the preferred node when the pair is (re)created. Called on
/// the DB refresh tick.
/// </summary>
/// <param name="siteId">Site identifier.</param>
/// <param name="nodeAEndpoint">NodeA gRPC base address, or null when unconfigured.</param>
@@ -107,6 +124,7 @@ public sealed class SitePairChannelProvider : IDisposable
public void UpdateSite(string siteId, string? nodeAEndpoint, string? nodeBEndpoint)
{
ArgumentException.ThrowIfNullOrWhiteSpace(siteId);
SweepGraveyard();
var pair = _sites.GetOrAdd(siteId, id => new SitePair(id));
lock (pair.Gate)
@@ -120,13 +138,13 @@ public sealed class SitePairChannelProvider : IDisposable
if (changedA)
{
pair.ChannelA?.Dispose();
Park(pair.ChannelA);
pair.ChannelA = string.IsNullOrWhiteSpace(nodeAEndpoint) ? null : BuildChannel(nodeAEndpoint, siteId);
pair.NodeAEndpoint = nodeAEndpoint;
}
if (changedB)
{
pair.ChannelB?.Dispose();
Park(pair.ChannelB);
pair.ChannelB = string.IsNullOrWhiteSpace(nodeBEndpoint) ? null : BuildChannel(nodeBEndpoint, siteId);
pair.NodeBEndpoint = nodeBEndpoint;
}
@@ -140,7 +158,10 @@ public sealed class SitePairChannelProvider : IDisposable
}
}
/// <summary>Disposes a removed site's channels and invalidates its cached preshared key.</summary>
/// <summary>
/// Parks a removed site's channels for deferred disposal (see <see cref="ChannelDisposalGrace"/>)
/// and invalidates its cached preshared key.
/// </summary>
/// <param name="siteId">Site identifier to remove.</param>
public void RemoveSite(string siteId)
{
@@ -149,12 +170,14 @@ public sealed class SitePairChannelProvider : IDisposable
return;
}
SweepGraveyard();
if (_sites.TryRemove(siteId, out var pair))
{
lock (pair.Gate)
{
pair.ChannelA?.Dispose();
pair.ChannelB?.Dispose();
Park(pair.ChannelA);
Park(pair.ChannelB);
pair.ChannelA = null;
pair.ChannelB = null;
}
@@ -284,6 +307,7 @@ public sealed class SitePairChannelProvider : IDisposable
{
while (await timer.WaitForNextTickAsync(ct).ConfigureAwait(false))
{
SweepGraveyard();
var now = DateTime.UtcNow;
foreach (var kvp in _sites)
{
@@ -317,6 +341,67 @@ public sealed class SitePairChannelProvider : IDisposable
}
}
/// <summary>
/// Parks a channel that is no longer reachable through <see cref="_sites"/> instead of
/// disposing it inline (arch-review 2026-08-19 P3-2).
/// <para>
/// <b>Why deferred.</b> <see cref="ExecuteAsync{T}"/> reads the channel under
/// <c>pair.Gate</c> but invokes the RPC OUTSIDE it, so an inline
/// <c>Dispose</c> from the 60 s DB address refresh (or an admin edit) aborted whatever was
/// in flight — a multi-minute <c>DeployArtifactsCommand</c> included — with
/// <see cref="ObjectDisposedException"/> or <see cref="StatusCode.Cancelled"/>. Neither is
/// <see cref="StatusCode.Unavailable"/>, so the cross-node retry deliberately never engaged
/// and the command simply failed.
/// </para>
/// <para>
/// <b>Why a timed park rather than reference counting.</b> A refcount would have to be taken
/// and released around every <c>ExecuteAsync</c> call — bookkeeping on the hot path, plus a
/// quiescence handshake — to buy nothing a fixed window does not already give: the window
/// only has to outlast the longest command deadline. It also matches the precedent set by
/// <see cref="SiteStreamGrpcClientFactory"/>, which likewise keeps a superseded endpoint's
/// channel alive rather than disposing it under a concurrent session. Cost is a handful of
/// idle HTTP/2 channels for at most <see cref="ChannelDisposalGrace"/>.
/// </para>
/// </summary>
/// <param name="channel">The channel to park; null is ignored.</param>
private void Park(GrpcChannel? channel)
{
if (channel is not null)
{
_graveyard.Enqueue((channel, DateTime.UtcNow + ChannelDisposalGrace));
}
}
/// <summary>
/// Disposes every parked channel whose grace window has elapsed. Called from the DB refresh
/// path (<see cref="UpdateSite"/>/<see cref="RemoveSite"/>) and from the failback loop, so a
/// park always drains even on a provider that never runs a call again.
/// </summary>
/// <param name="force">True to dispose every parked channel regardless of its window (shutdown, and tests).</param>
internal void SweepGraveyard(bool force = false)
{
var now = DateTime.UtcNow;
while (_graveyard.TryPeek(out var parked) && (force || now >= parked.DisposeAfterUtc))
{
if (!_graveyard.TryDequeue(out parked))
{
break;
}
try
{
parked.Channel.Dispose();
}
catch (Exception ex)
{
_logger.LogDebug(ex, "Disposing a parked gRPC channel faulted");
}
}
}
/// <summary>Test/diagnostic accessor: how many replaced channels are awaiting deferred disposal.</summary>
internal int ParkedChannelCount => _graveyard.Count;
private void FlipTo(SitePair pair, bool toIsA)
{
lock (pair.Gate)
@@ -396,6 +481,10 @@ public sealed class SitePairChannelProvider : IDisposable
}
}
_sites.Clear();
// Process shutdown — nothing can still be in flight, so drain the graveyard regardless
// of the remaining grace on each entry.
SweepGraveyard(force: true);
}
/// <summary>Per-site mutable channel state, guarded by <see cref="Gate"/>.</summary>
src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs
+82 -27
View File
@@ -138,7 +138,18 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable
/// Registers a subscription's CancellationTokenSource for a correlation ID.
/// If an entry already exists for that correlation ID (a reconnect race where two
/// <see cref="SubscribeAsync"/> calls briefly share an ID), the prior CTS is
/// cancelled and disposed so it cannot leak. Internal for testability.
/// <b>cancelled only</b> — never disposed from here. Internal for testability.
/// <para>
/// <b>Cancel-only is the contract</b> (arch-review 2026-08-19 P3-1, the client half of the
/// row-47 class): the prior handler's <see cref="ConsumeStreamAsync"/> is still running and
/// still reads <c>cts.Token</c> (opening the call, <c>ReadAllAsync</c>, the
/// <c>IsCancellationRequested</c> checks), so disposing it from the REPLACING path throws
/// <see cref="ObjectDisposedException"/> inside the prior stream — and, worse, out of the
/// <c>prior.Cancel()</c> call itself into the NEW <c>SubscribeAsync</c>, where it reads as a
/// fault of the current generation and burns a reconnect. Disposal belongs to the OWNING
/// path: <see cref="SubscribeAsync"/>/<see cref="SubscribeSiteAsync"/> dispose the CTS they
/// created once their stream loop has fully unwound.
/// </para>
/// </summary>
/// <param name="correlationId">Unique identifier for the subscription.</param>
/// <param name="cts">CancellationTokenSource for managing the subscription lifecycle.</param>
@@ -146,12 +157,30 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable
{
if (_subscriptions.TryGetValue(correlationId, out var prior) && !ReferenceEquals(prior, cts))
{
prior.Cancel();
prior.Dispose();
CancelQuietly(prior);
}
_subscriptions[correlationId] = cts;
}
/// <summary>
/// Cancels a subscription token source, swallowing <see cref="ObjectDisposedException"/>.
/// A CTS whose owning stream loop has already unwound and disposed it is simply a
/// subscription that no longer needs cancelling; the caller must never see that race as a
/// failure (arch-review 2026-08-19 P3-1).
/// </summary>
/// <param name="cts">The token source to cancel.</param>
private static void CancelQuietly(CancellationTokenSource cts)
{
try
{
cts.Cancel();
}
catch (ObjectDisposedException)
{
// Already disposed by its owning stream loop — nothing left to cancel.
}
}
/// <summary>
/// Removes the subscription entry for a correlation ID only if the stored CTS is
/// exactly the one supplied. A racing replacement stream may already own the slot,
@@ -213,13 +242,23 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable
onEvent(domainEvent);
}
await ConsumeStreamAsync(
correlationId,
cts,
() => _client.SubscribeInstance(request, cancellationToken: cts.Token),
frame => ForEachEvent(frame, Deliver),
onError,
onCompleted);
// This call OWNS `cts` — it created it, so it disposes it once the stream loop has
// fully unwound (P3-1). No other path may dispose it while this one is still reading
// cts.Token; a replacing subscription cancels only.
try
{
await ConsumeStreamAsync(
correlationId,
cts,
() => _client.SubscribeInstance(request, cancellationToken: cts.Token),
frame => ForEachEvent(frame, Deliver),
onError,
onCompleted);
}
finally
{
cts.Dispose();
}
}
/// <summary>
@@ -288,14 +327,22 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable
onAlarmEvent(alarm);
}
await ConsumeStreamAsync(
correlationId,
cts,
() => _client.SubscribeSite(request, cancellationToken: cts.Token),
frame => ForEachEvent(frame, Deliver),
onError,
onCompleted,
onConnected);
// Owns `cts` — see SubscribeAsync.
try
{
await ConsumeStreamAsync(
correlationId,
cts,
() => _client.SubscribeSite(request, cancellationToken: cts.Token),
frame => ForEachEvent(frame, Deliver),
onError,
onCompleted,
onConnected);
}
finally
{
cts.Dispose();
}
}
/// <summary>
@@ -449,15 +496,17 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable
}
/// <summary>
/// Cancels an active subscription by correlation ID.
/// Cancels an active subscription by correlation ID. Cancel-only: the subscription's own
/// <see cref="SubscribeAsync"/>/<see cref="SubscribeSiteAsync"/> call disposes the token
/// source once its stream loop unwinds (P3-1). Cancelling an entry whose owner already
/// disposed it — or unsubscribing after the client was disposed — is a silent no-op.
/// </summary>
/// <param name="correlationId">Unique identifier of the subscription to cancel.</param>
public virtual void Unsubscribe(string correlationId)
{
if (_subscriptions.TryRemove(correlationId, out var cts))
{
cts.Cancel();
cts.Dispose();
CancelQuietly(cts);
}
}
@@ -601,17 +650,23 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable
};
/// <summary>
/// Releases all subscription CancellationTokenSources and the underlying
/// gRPC channel. All teardown here is synchronous (CTS disposal and
/// <see cref="GrpcChannel.Dispose"/>), so a synchronous <see cref="Dispose"/>
/// can release everything without sync-over-async blocking.
/// Cancels every live subscription and releases the underlying gRPC channel. All teardown
/// here is synchronous (CTS cancellation and <see cref="GrpcChannel.Dispose"/>), so a
/// synchronous <see cref="Dispose"/> can release everything without sync-over-async blocking.
/// <para>
/// Cancel-only, per-entry guarded (P3-1). The token sources are owned by their
/// <see cref="SubscribeAsync"/>/<see cref="SubscribeSiteAsync"/> calls, which dispose them
/// as their loops unwind in response to this cancellation; disposing them here raced those
/// loops, and one <see cref="ObjectDisposedException"/> aborted the whole foreach — which in
/// turn aborted <see cref="SiteStreamGrpcClientFactory.DisposeAsync"/>'s client loop and left
/// every later client (and its channel) undisposed.
/// </para>
/// </summary>
private void ReleaseResources()
{
foreach (var cts in _subscriptions.Values)
{
cts.Cancel();
cts.Dispose();
CancelQuietly(cts);
}
_subscriptions.Clear();
src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClientFactory.cs
+39 -6
View File
@@ -120,7 +120,7 @@ public class SiteStreamGrpcClientFactory : IAsyncDisposable, IDisposable
foreach (var key in _clients.Keys.Where(k => k.Site == siteIdentifier).ToList())
{
if (_clients.TryRemove(key, out var client))
await client.DisposeAsync();
await DisposeClientAsync(client, key);
}
// Drop the cached preshared key too, so a site removed and re-added under the same
@@ -129,16 +129,39 @@ public class SiteStreamGrpcClientFactory : IAsyncDisposable, IDisposable
}
/// <summary>
/// Asynchronously disposes all cached clients and clears the cache.
/// Asynchronously disposes all cached clients and clears the cache. Each client's disposal
/// is fault-isolated: one throwing client must not abort the loop and strand every later
/// client's channel undisposed (arch-review 2026-08-19 P3-1).
/// </summary>
/// <returns>A value task that completes when all clients have been disposed.</returns>
public async ValueTask DisposeAsync()
{
foreach (var client in _clients.Values)
foreach (var entry in _clients)
{
await DisposeClientAsync(entry.Value, entry.Key);
}
_clients.Clear();
}
/// <summary>
/// Disposes one client, logging and swallowing any failure so the caller's disposal loop
/// continues to the remaining clients.
/// </summary>
/// <param name="client">The client to dispose.</param>
/// <param name="key">Cache key of the client, for diagnostics.</param>
/// <returns>A task that completes when the client has been disposed (or its failure logged).</returns>
private async Task DisposeClientAsync(SiteStreamGrpcClient client, (string Site, string Endpoint) key)
{
try
{
await client.DisposeAsync();
}
_clients.Clear();
catch (Exception ex)
{
_loggerFactory.CreateLogger<SiteStreamGrpcClientFactory>().LogWarning(
ex, "Disposing the site stream gRPC client for site {SiteId} endpoint {Endpoint} failed",
key.Site, key.Endpoint);
}
}
/// <summary>
@@ -147,12 +170,22 @@ public class SiteStreamGrpcClientFactory : IAsyncDisposable, IDisposable
/// a stall/deadlock risk during host shutdown). Each
/// <see cref="SiteStreamGrpcClient"/> releases all of its resources
/// synchronously, so we dispose them directly with no async path.
/// Fault-isolated per client, exactly as <see cref="DisposeAsync"/> is.
/// </summary>
public void Dispose()
{
foreach (var client in _clients.Values)
foreach (var entry in _clients)
{
client.Dispose();
try
{
entry.Value.Dispose();
}
catch (Exception ex)
{
_loggerFactory.CreateLogger<SiteStreamGrpcClientFactory>().LogWarning(
ex, "Disposing the site stream gRPC client for site {SiteId} endpoint {Endpoint} failed",
entry.Key.Site, entry.Key.Endpoint);
}
}
_clients.Clear();
}
src/ZB.MOM.WW.ScadaBridge.Communication/Protos/central_control.proto
+11
View File
@@ -234,6 +234,17 @@ message SiteHealthReportDto {
// self-heal path, which is a misconfiguration a healthy-looking site would
// otherwise hide entirely.
google.protobuf.BoolValue startup_reconciler_active = 29;
// WP3.1: script-execution worker threads detached and replaced by the stuck-script
// watchdog and not yet exited. Point-in-time gauge (like script_busy_threads above),
// not interval-drained. Zero is the healthy state.
int32 detached_script_threads = 30;
// WP3.1: per-interval count of script/alarm on-trigger runs shed by
// MaxConcurrentRunsPerScript. Interval-drained like script_error_count.
int32 script_run_shed_count = 31;
// WP2.6d: cumulative count of alarm state changes dropped from SiteStreamManager's
// dedicated alarm publish queue because it was at capacity. Point-in-time
// (not reset on collect), like site_event_log_write_failures.
int64 site_stream_alarm_drop_count = 32;
}
// Central -> Site: health report ack, so a lost report is observable.
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/SiteCallEntityTypeConfiguration.cs
+12
View File
@@ -106,5 +106,17 @@ public class SiteCallEntityTypeConfiguration : IEntityTypeConfiguration<SiteCall
builder.HasIndex(s => s.TerminalAtUtc)
.HasDatabaseName("IX_SiteCalls_Terminal")
.HasFilter("[TerminalAtUtc] IS NOT NULL");
// Created_TrackedOperationId backs QueryAsync's list-page ORDER BY
// (CreatedAtUtc DESC, TrackedOperationId DESC) and its matching keyset
// predicate — the unfiltered/lightly-filtered Central UI Site Calls page.
// None of the indexes above lead on CreatedAtUtc unconditionally
// (Source_Created requires an equality on SourceSite first;
// NonTerminal/Terminal are filtered to one side of TerminalAtUtc), so an
// unfiltered page previously fell back to a full clustered-index scan +
// sort (arch-review 2026-08-19 P5-2, register row-35 class).
builder.HasIndex(s => new { s.CreatedAtUtc, s.TrackedOperationId })
.IsDescending(true, true)
.HasDatabaseName("IX_SiteCalls_Created_TrackedOperationId");
}
}
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260819183326_AddSiteCallsListSortIndex.Designer.cs
Generated
+2127
View File
File diff suppressed because it is too large Load Diff
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/20260819183326_AddSiteCallsListSortIndex.cs
+47
View File
@@ -0,0 +1,47 @@
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations
{
/// <summary>
/// Adds <c>IX_SiteCalls_Created_TrackedOperationId (CreatedAtUtc DESC,
/// TrackedOperationId DESC)</c> — arch-review 2026-08-19 P5-2, register
/// row-35 class.
/// </summary>
/// <remarks>
/// <see cref="ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Repositories.SiteCallAuditRepository.QueryAsync"/>
/// orders <c>(CreatedAtUtc DESC, TrackedOperationId DESC)</c> and its keyset
/// predicate is exactly that pair. None of the existing SiteCalls indexes
/// lead unconditionally on <c>CreatedAtUtc</c> — <c>IX_SiteCalls_Source_Created</c>
/// requires an equality on <c>SourceSite</c> first, and
/// <c>IX_SiteCalls_NonTerminal</c>/<c>IX_SiteCalls_Terminal</c> are each
/// filtered to one side of <c>TerminalAtUtc</c> — so an unfiltered (or
/// lightly filtered) list page fell back to a full clustered-index scan +
/// sort. With both key columns descending, a forward scan of this index
/// already delivers rows in the requested order, so a keyset page reads a
/// Top-N off the index instead of sorting the whole table on every page
/// request. See <c>docs/plans/sql/AddSiteCallsListSortIndex.sql</c> for the
/// handwritten production-parity script.
/// </remarks>
public partial class AddSiteCallsListSortIndex : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.CreateIndex(
name: "IX_SiteCalls_Created_TrackedOperationId",
table: "SiteCalls",
columns: new[] { "CreatedAtUtc", "TrackedOperationId" },
descending: new bool[0]);
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropIndex(
name: "IX_SiteCalls_Created_TrackedOperationId",
table: "SiteCalls");
}
}
}
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Migrations/ScadaBridgeDbContextModelSnapshot.cs
+4
View File
@@ -171,6 +171,10 @@ namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations
.HasDatabaseName("IX_SiteCalls_Terminal")
.HasFilter("[TerminalAtUtc] IS NOT NULL");
b.HasIndex("CreatedAtUtc", "TrackedOperationId")
.IsDescending()
.HasDatabaseName("IX_SiteCalls_Created_TrackedOperationId");
b.HasIndex("SourceSite", "CreatedAtUtc")
.IsDescending(false, true)
.HasDatabaseName("IX_SiteCalls_Source_Created");
src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/SiteCallAuditRepository.cs
+47 -24
View File
@@ -274,12 +274,19 @@ WHERE TrackedOperationId = @Id
throw new ArgumentNullException(nameof(paging));
}
// FormattableString interpolation parameterises every value (no concatenation)
// so this is injection-safe. EF Core resolves the parameter values, the
// composed sql is shaped to SQL Server's grammar and projected into the
// SiteCall entity via FromSqlInterpolated. The CASE expressions wrap each
// optional predicate so a null filter field degrades to a no-op (matches
// every row) instead of branching at C# level into N variants.
// Raw SQL with NAMED parameters (not FromSqlInterpolated) — arch-review
// 2026-08-19 P5-2. FromSqlInterpolated binds every interpolated string
// value as an nvarchar(4000) parameter by default; every SiteCalls string
// column is IsUnicode(false) (varchar), so those parameters forced an
// implicit CONVERT on the column side of each comparison and blocked
// index seeks — including the keyset predicate below, which is exactly
// what IX_SiteCalls_Created_TrackedOperationId exists to serve. Each
// string filter now gets an explicitly-typed SqlDbType.VarChar parameter,
// mirroring the write path's Param(...) helper above. A named ADO.NET
// parameter can appear more than once in the SQL text (unlike a
// positional FromSqlInterpolated hole) while still being added to the
// command's Parameters collection exactly once, so the "IS NULL OR
// col = @p" shape needs no duplicate parameter objects.
var afterCreated = paging.AfterCreatedAtUtc;
var afterIdString = paging.AfterId?.Value.ToString("D");
var hasCursor = afterCreated is not null && afterIdString is not null;
@@ -294,37 +301,53 @@ WHERE TrackedOperationId = @Id
// never under-filled with a non-null next cursor. Mirrors how
// NotificationOutboxRepository.QueryAsync applies NotificationOutboxFilter.StuckCutoff.
//
// SELECT-list maintenance: EF Core's FromSqlInterpolated requires every
// SELECT-list maintenance: EF Core's FromSqlRaw requires every
// entity-tracked column to appear in the result set. Adding a new column
// to the SiteCall entity means extending the list below too — otherwise
// every read trips "The required column 'X' was not present" at runtime.
FormattableString sql = $@"
SELECT TOP ({paging.PageSize})
const string sql = @"
SELECT TOP (@PageSize)
TrackedOperationId, Channel, Target, SourceSite, SourceNode, Status, RetryCount,
LastError, HttpStatus, CreatedAtUtc, UpdatedAtUtc, TerminalAtUtc, IngestedAtUtc
FROM dbo.SiteCalls
WHERE ({filter.Channel} IS NULL OR Channel = {filter.Channel})
AND ({filter.SourceSite} IS NULL OR SourceSite = {filter.SourceSite})
AND ({filter.SourceNode} IS NULL OR SourceNode = {filter.SourceNode})
AND ({filter.Status} IS NULL OR Status = {filter.Status})
AND ({filter.Target} IS NULL OR Target = {filter.Target})
AND ({fromUtc} IS NULL OR CreatedAtUtc >= {fromUtc})
AND ({toUtc} IS NULL OR CreatedAtUtc <= {toUtc})
AND ({stuckCutoff} IS NULL OR (TerminalAtUtc IS NULL AND CreatedAtUtc < {stuckCutoff}))
AND ({(hasCursor ? 1 : 0)} = 0
OR CreatedAtUtc < {afterCreated}
OR (CreatedAtUtc = {afterCreated} AND TrackedOperationId < {afterIdString}))
WHERE (@Channel IS NULL OR Channel = @Channel)
AND (@SourceSite IS NULL OR SourceSite = @SourceSite)
AND (@SourceNode IS NULL OR SourceNode = @SourceNode)
AND (@Status IS NULL OR Status = @Status)
AND (@Target IS NULL OR Target = @Target)
AND (@FromUtc IS NULL OR CreatedAtUtc >= @FromUtc)
AND (@ToUtc IS NULL OR CreatedAtUtc <= @ToUtc)
AND (@StuckCutoff IS NULL OR (TerminalAtUtc IS NULL AND CreatedAtUtc < @StuckCutoff))
AND (@HasCursor = 0
OR CreatedAtUtc < @AfterCreated
OR (CreatedAtUtc = @AfterCreated AND TrackedOperationId < @AfterId))
ORDER BY CreatedAtUtc DESC, TrackedOperationId DESC
-- Every filter above is the (@p IS NULL OR col = @p) optional-parameter shape, so a
-- single cached plan would be parameter-sniffed for whichever filters happened to be
-- non-null on first compile. RECOMPILE lets the optimizer prune the dead (@p IS NULL)
-- predicates per invocation and pick IX_SiteCalls_Status_Updated / IX_SiteCalls_NonTerminal
-- for the filters actually supplied (arch-review 04, P5). Per-invocation compile cost is
-- negligible at UI-page cadence.
-- predicates per invocation and pick IX_SiteCalls_Status_Updated / IX_SiteCalls_NonTerminal /
-- IX_SiteCalls_Created_TrackedOperationId for the filters actually supplied (arch-review 04,
-- P5). Per-invocation compile cost is negligible at UI-page cadence.
OPTION (RECOMPILE);";
var parameters = new object[]
{
Param("@PageSize", SqlDbType.Int, paging.PageSize),
Param("@Channel", SqlDbType.VarChar, filter.Channel),
Param("@SourceSite", SqlDbType.VarChar, filter.SourceSite),
Param("@SourceNode", SqlDbType.VarChar, filter.SourceNode),
Param("@Status", SqlDbType.VarChar, filter.Status),
Param("@Target", SqlDbType.VarChar, filter.Target),
Param("@FromUtc", SqlDbType.DateTime2, fromUtc),
Param("@ToUtc", SqlDbType.DateTime2, toUtc),
Param("@StuckCutoff", SqlDbType.DateTime2, stuckCutoff),
Param("@HasCursor", SqlDbType.Int, hasCursor ? 1 : 0),
Param("@AfterCreated", SqlDbType.DateTime2, afterCreated),
Param("@AfterId", SqlDbType.VarChar, afterIdString),
};
var rows = await _context.Set<SiteCall>()
.FromSqlInterpolated(sql)
.FromSqlRaw(sql, parameters)
.AsNoTracking()
.ToListAsync(ct);
src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs
+85 -6
View File
@@ -778,16 +778,60 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
{
_log.Debug("[{0}] Attempting connection...", _connectionName);
var self = Self;
// Stamp the generation this attempt belongs to, exactly as every other piped
// completion does. A failover or an in-place reconfigure landing while ConnectAsync
// is still in flight swaps the adapter and bumps the counter, and the OUTGOING
// adapter's answer must never be applied to its replacement.
var generation = _adapterGeneration;
_adapter.ConnectAsync(_connectionDetails).ContinueWith(t =>
{
if (t.IsCompletedSuccessfully)
return new ConnectResult(true, null);
return new ConnectResult(false, t.Exception?.GetBaseException().Message);
return new ConnectResult(true, null, generation);
return new ConnectResult(false, t.Exception?.GetBaseException().Message, generation);
}).PipeTo(self);
}
/// <summary>
/// True when this connect result was produced by an adapter that has since been swapped
/// out, in which case it is dropped WHOLE (arch-review 2026-08-19 P2-3).
///
/// <para>
/// A success from the dead adapter used to drive <see cref="ReSubscribeAll"/> and
/// <see cref="BecomeConnected"/> against the NEW, never-connected adapter — and the new
/// adapter's own result then arrived in the Connected state, where nothing handles it. If the
/// new endpoint was bad the batch subscribes failed at connection level with a log line only,
/// no <c>AdapterDisconnected</c> ever fired, and the idempotent-reconfigure guard rejected
/// re-sending the same configuration: only a restart recovered the connection. A failure from
/// the dead adapter is just as wrong — it would count toward failover and re-arm the reconnect
/// timer underneath the attempt that is genuinely in flight.
/// </para>
///
/// <para>
/// Dropping cannot wedge the state machine: every site that bumps the generation
/// (<see cref="CountFailureAndMaybeFailover"/>, the failover branch of
/// <see cref="BecomeReconnecting"/>, and <see cref="HandleReconfigure"/> through it) arms an
/// <see cref="AttemptConnect"/> for the new generation, and the state stays Connecting /
/// Reconnecting until that attempt's own result arrives.
/// </para>
/// </summary>
private bool IsStaleConnectResult(ConnectResult result)
{
if (result.AdapterGeneration == _adapterGeneration)
return false;
_log.Debug(
"[{0}] Dropping {1} connect result from adapter generation {2} (current {3}) — "
+ "the adapter it ran against has been replaced",
_connectionName, result.Success ? "successful" : "failed",
result.AdapterGeneration, _adapterGeneration);
return true;
}
private void HandleConnectResult(ConnectResult result)
{
if (IsStaleConnectResult(result))
return;
if (result.Success)
{
_log.Info("[{0}] Connection established", _connectionName);
@@ -809,6 +853,9 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
private void HandleReconnectResult(ConnectResult result)
{
if (IsStaleConnectResult(result))
return;
if (result.Success)
{
_log.Info("[{0}] Reconnected successfully on {1} endpoint", _connectionName, _activeEndpoint);
@@ -1538,11 +1585,38 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
{
var baseEx = ex is AggregateException agg ? agg.GetBaseException() : ex;
return baseEx is InvalidOperationException
or System.Net.Sockets.SocketException
or TimeoutException
or System.IO.IOException;
or System.Net.Sockets.SocketException
or TimeoutException
or System.IO.IOException
|| (baseEx is Opc.Ua.ServiceResultException sre && IsConnectionLevelStatus(sre.StatusCode));
}
/// <summary>
/// OPC UA status codes that mean the SESSION or the SECURE CHANNEL is gone, not that one
/// node is unusable (arch-review 2026-08-19 P2-4). The SDK reports a dead session by
/// throwing <see cref="Opc.Ua.ServiceResultException"/> out of the batch subscribe, which
/// classified as a per-tag fault: every tag in the chunk was marked unresolved and each
/// backoff probe re-issued the whole batch against the same dead session, growing the
/// adapter's monitored-item bookkeeping on every round instead of reconnecting once.
///
/// <para>
/// Deliberately narrow. A per-node code — <c>BadNodeIdUnknown</c>, <c>BadNodeIdInvalid</c>,
/// <c>BadUserAccessDenied</c> — must stay a tag-resolution failure, because that is exactly
/// the "device still booting / tag path wrong" case the retry timer exists for; treating it
/// as connection-level would put the connection into a reconnect loop over one bad tag path.
/// </para>
/// </summary>
private static bool IsConnectionLevelStatus(uint statusCode) => statusCode is
Opc.Ua.StatusCodes.BadSessionIdInvalid
or Opc.Ua.StatusCodes.BadSessionClosed
or Opc.Ua.StatusCodes.BadSessionNotActivated
or Opc.Ua.StatusCodes.BadSecureChannelClosed
or Opc.Ua.StatusCodes.BadSecureChannelIdInvalid
or Opc.Ua.StatusCodes.BadServerNotConnected
or Opc.Ua.StatusCodes.BadConnectionClosed
or Opc.Ua.StatusCodes.BadCommunicationError
or Opc.Ua.StatusCodes.BadNotConnected;
private void HandleUnsubscribe(UnsubscribeTagsRequest request)
{
_log.Debug("[{0}] Unsubscribing all tags for instance {1}",
@@ -2881,7 +2955,12 @@ public class DataConnectionActor : UntypedActor, IWithStash, IWithTimers
// ── Internal messages ──
internal record AttemptConnect;
internal record ConnectResult(bool Success, string? Error);
/// <summary>
/// Outcome of one <see cref="AttemptConnect"/>. Carries the adapter generation it ran
/// against so a result produced by an adapter that has since been swapped out is dropped
/// rather than applied to its replacement (see <see cref="IsStaleConnectResult"/>).
/// </summary>
internal record ConnectResult(bool Success, string? Error, int AdapterGeneration);
internal record AdapterDisconnected;
internal record TagValueReceived(string TagPath, TagValue Value, int AdapterGeneration);
internal record RetryTagResolution;
src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/RealOpcUaClient.cs
+111 -40
View File
@@ -621,48 +621,21 @@ public class RealOpcUaClient : IOpcUaClient
var created = new List<(int Index, string NodeId, string Handle, MonitoredItem Item)>(resolved.Count);
var touched = new List<DataShard>();
foreach (var (index, nodeId, startNode) in resolved)
try
{
var shard = await PlaceOnDataShardAsync(cancellationToken);
var handle = Guid.NewGuid().ToString();
var monitoredItem = new MonitoredItem(shard.Subscription.DefaultItem)
{
DisplayName = nodeId,
StartNodeId = startNode,
AttributeId = Attributes.Value,
SamplingInterval = _options.SamplingIntervalMs,
QueueSize = (uint)_options.QueueSize,
DiscardOldest = _options.DiscardOldest,
Filter = BuildDataChangeFilter(_options.Deadband)
};
_callbacks[handle] = onValueChanged;
monitoredItem.Notification += (_, e) =>
{
if (e.NotificationValue is MonitoredItemNotification notification)
{
var value = notification.Value?.Value;
var timestamp = notification.Value?.SourceTimestamp ?? DateTime.UtcNow;
var statusCode = notification.Value?.StatusCode.Code ?? 0;
if (_callbacks.TryGetValue(handle, out var cb))
{
cb(nodeId, value, timestamp, statusCode);
}
}
};
shard.Subscription.AddItem(monitoredItem);
shard.ItemCount++;
_itemShard[handle] = shard.Subscription;
created.Add((index, nodeId, handle, monitoredItem));
if (!touched.Contains(shard))
touched.Add(shard);
await PlaceAndApplyAsync(resolved, onValueChanged, created, touched, cancellationToken);
}
catch
{
// Anything thrown from placement or ApplyChanges (a dead session is the common
// one) leaves every item created so far registered in _callbacks / _itemShard and
// counted against its shard's budget, with nothing to ever release them: the
// actor sees the throw as a whole-chunk fault and re-issues the SAME batch on its
// backoff timer, so each probe grew the bookkeeping again (arch-review 2026-08-19
// P2-4). Unwind before propagating.
UnwindCreatedItems(created);
throw;
}
// ONE ApplyChanges per touched shard — the whole point of the batch seam.
foreach (var shard in touched)
await shard.Subscription.ApplyChangesAsync(cancellationToken);
foreach (var (index, nodeId, handle, item) in created)
{
@@ -687,6 +660,104 @@ public class RealOpcUaClient : IOpcUaClient
}, cancellationToken);
}
/// <summary>
/// Places every resolved node on a data shard and issues ONE ApplyChanges per touched shard.
/// Callers must hold <see cref="_applyGate"/>. <paramref name="created"/> and
/// <paramref name="touched"/> are filled as the work proceeds so a throw can be unwound by
/// <see cref="UnwindCreatedItems"/>.
/// </summary>
private async Task PlaceAndApplyAsync(
List<(int Index, string NodeId, NodeId Resolved)> resolved,
Action<string, object?, DateTime, uint> onValueChanged,
List<(int Index, string NodeId, string Handle, MonitoredItem Item)> created,
List<DataShard> touched,
CancellationToken cancellationToken)
{
foreach (var (index, nodeId, startNode) in resolved)
{
var shard = await PlaceOnDataShardAsync(cancellationToken);
var handle = Guid.NewGuid().ToString();
var monitoredItem = new MonitoredItem(shard.Subscription.DefaultItem)
{
DisplayName = nodeId,
StartNodeId = startNode,
AttributeId = Attributes.Value,
SamplingInterval = _options.SamplingIntervalMs,
QueueSize = (uint)_options.QueueSize,
DiscardOldest = _options.DiscardOldest,
Filter = BuildDataChangeFilter(_options.Deadband)
};
_callbacks[handle] = onValueChanged;
monitoredItem.Notification += (_, e) =>
{
if (e.NotificationValue is MonitoredItemNotification notification)
{
var value = notification.Value?.Value;
var timestamp = notification.Value?.SourceTimestamp ?? DateTime.UtcNow;
var statusCode = notification.Value?.StatusCode.Code ?? 0;
if (_callbacks.TryGetValue(handle, out var cb))
{
cb(nodeId, value, timestamp, statusCode);
}
}
};
shard.Subscription.AddItem(monitoredItem);
shard.ItemCount++;
_itemShard[handle] = shard.Subscription;
created.Add((index, nodeId, handle, monitoredItem));
if (!touched.Contains(shard))
touched.Add(shard);
}
// ONE ApplyChanges per touched shard — the whole point of the batch seam.
foreach (var shard in touched)
await shard.Subscription.ApplyChangesAsync(cancellationToken);
}
/// <summary>
/// Rolls back the monitored items a FAILED batch had already placed, so a throw leaves no
/// residue behind: no <c>_callbacks</c> entry (a leak that also keeps a dead callback
/// reachable from the SDK's publish thread), no <c>_itemShard</c> entry, no phantom shard
/// occupancy, and no item left on the shard's local item list. Callers must hold
/// <see cref="_applyGate"/>.
///
/// <para>
/// Best-effort and exception-safe by design: it runs on the failure path, so one item that
/// cannot be released must not stop the rest from being released and must not replace the
/// original fault. No ApplyChanges is re-issued — the session that just threw is generally
/// dead, and where a partial apply DID reach the server the items it created are reclaimed
/// when the shard and session are torn down on the reconnect that the connection-level
/// classification of that fault now drives.
/// </para>
/// </summary>
private void UnwindCreatedItems(
List<(int Index, string NodeId, string Handle, MonitoredItem Item)> created)
{
if (created.Count == 0)
return;
foreach (var (_, nodeId, handle, item) in created)
{
try
{
ReleaseFailedItem(handle, item);
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"OPC UA batch subscribe rollback could not release the monitored item for {NodeId}",
nodeId);
}
}
_logger.LogWarning(
"OPC UA batch subscribe failed — rolled back {Count} partially created monitored item(s)",
created.Count);
}
/// <summary>
/// Returns the shard this item belongs on per
/// <see cref="MonitoredItemShardPlanner"/> — first shard with free capacity, else a
src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentManagerOptions.cs
+19
View File
@@ -41,6 +41,25 @@ public class DeploymentManagerOptions
/// </summary>
public TimeSpan SiteDeploymentTimeoutPerInstance { get; set; } = TimeSpan.FromSeconds(120);
/// <summary>
/// Deadline applied to each query-before-redeploy reconciliation round-trip
/// inside a bulk <c>DeploySiteAsync</c> (the single-instance path keeps the
/// communication layer's own <c>QueryTimeout</c>).
///
/// <para>
/// Phase 1 is strictly serial, so an unreachable site otherwise costs
/// <c>QueryTimeout</c> (30 s) PER INSTANCE before a single deploy is sent — for
/// a 500-instance site roughly four hours, far past the CLI's 5-minute bulk
/// deploy timeout, which then cancels and re-arms the identical cost on the next
/// attempt. Deliberately short (seconds): a reconcile that does not answer
/// promptly falls through to a normal deploy anyway, which is exactly what its
/// own timeout produces. Works together with the once-per-batch reachability
/// circuit-breaker, which skips the remaining reconciles after the first
/// timeout.
/// </para>
/// </summary>
public TimeSpan SiteDeploymentReconcileTimeout { get; set; } = TimeSpan.FromSeconds(5);
/// <summary>
/// Retention window for TERMINAL deployment records (<c>Success</c>,
/// <c>Failed</c>). Deployments are insert-only — one row per attempt, forever —
src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentManagerOptionsValidator.cs
+5
View File
@@ -38,6 +38,11 @@ public sealed class DeploymentManagerOptionsValidator : OptionsValidatorBase<Dep
$"(was {options.SiteDeploymentTimeoutPerInstance}); it bounds each instance's site round-trip " +
$"inside a bulk site deployment.");
builder.RequireThat(options.SiteDeploymentReconcileTimeout > TimeSpan.Zero,
$"ScadaBridge:DeploymentManager:SiteDeploymentReconcileTimeout must be a positive duration " +
$"(was {options.SiteDeploymentReconcileTimeout}); it bounds each query-before-redeploy " +
$"reconciliation inside a bulk site deployment's serial prepare phase.");
builder.RequireThat(options.TerminalDeploymentRecordRetention >= TimeSpan.Zero,
$"ScadaBridge:DeploymentManager:TerminalDeploymentRecordRetention must not be negative " +
$"(was {options.TerminalDeploymentRecordRetention}); use TimeSpan.Zero to disable purging of " +
src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentService.cs
+198 -17
View File
@@ -57,6 +57,13 @@ public class DeploymentService
/// </summary>
private const string TimeoutFailurePrefix = "Communication failure:";
/// <summary>
/// Reason reported on every <see cref="SkippedInstanceResult"/> a bulk site
/// deploy produces for a <see cref="InstanceState.Disabled"/> instance.
/// </summary>
internal const string DisabledSkipReason =
"Disabled — bulk deploy preserves disabled state; deploy individually to re-enable";
/// <summary>Rows deleted per terminal-record purge batch.</summary>
private const int TerminalPurgeBatchSize = 500;
@@ -185,8 +192,13 @@ public class DeploymentService
// out of the same phase helpers is what keeps deployment identity,
// idempotency, lock coverage and optimistic concurrency from drifting apart
// between them.
// reconcileGate is null on this path on purpose: the batch-wide reconcile
// budget exists to stop a serial phase-1 loop multiplying one unreachable
// site's query timeout by the instance count. A single instance pays it
// once, and this is the explicit operator path — it keeps the full
// communication-layer QueryTimeout.
var prepared = await PrepareDeploymentAsync(
instanceId, user, session: null, cancellationToken);
instanceId, user, session: null, reconcileGate: null, cancellationToken);
if (prepared.EarlyResult is { } early)
return early;
@@ -230,6 +242,11 @@ public class DeploymentService
/// one. Sharing it across a batch is what collapses N identical template-chain
/// walks into one.
/// </param>
/// <param name="reconcileGate">
/// Batch-wide reconcile budget + reachability circuit-breaker for a bulk
/// deploy, or <see langword="null"/> on the single-instance path (which keeps
/// the communication layer's own query timeout).
/// </param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>
/// A prepared deployment ready to send, or one carrying
@@ -240,6 +257,7 @@ public class DeploymentService
int instanceId,
string user,
FlattenSession? session,
BulkReconcileGate? reconcileGate,
CancellationToken cancellationToken)
{
// Load instance
@@ -258,7 +276,8 @@ public class DeploymentService
try
{
return await PrepareUnderLockAsync(instance, user, session, lockHandle, cancellationToken);
return await PrepareUnderLockAsync(
instance, user, session, reconcileGate, lockHandle, cancellationToken);
}
catch
{
@@ -279,6 +298,7 @@ public class DeploymentService
Instance instance,
string user,
FlattenSession? session,
BulkReconcileGate? reconcileGate,
IDisposable lockHandle,
CancellationToken cancellationToken)
{
@@ -329,8 +349,11 @@ public class DeploymentService
// duplicate deployment is not produced (design: "Deployment Identity &
// Idempotency"). A clean prior Success or a fresh first-time deploy
// skips this extra round-trip.
var reconciled = await TryReconcileWithSiteAsync(
instance, revisionHash, configJson, user, cancellationToken);
var reconciled = reconcileGate is null
? await TryReconcileWithSiteAsync(
instance, revisionHash, configJson, user, cancellationToken)
: await ReconcileUnderBatchGateAsync(
instance, revisionHash, configJson, user, reconcileGate, cancellationToken);
if (reconciled != null)
{
lockHandle.Dispose();
@@ -745,6 +768,36 @@ public class DeploymentService
/// </summary>
private readonly record struct SendOutcome(DeploymentStatusResponse? Response, Exception? Error);
/// <summary>
/// A bulk deploy's shared query-before-redeploy budget: the short per-reconcile
/// deadline every instance in the batch runs under, plus the once-per-batch
/// reachability circuit-breaker that skips the rest after the first timeout.
/// See <see cref="ReconcileUnderBatchGateAsync"/> for why both bounds exist.
///
/// <para>
/// Deliberately not thread-safe: it is only ever touched from the serial
/// prepare phase.
/// </para>
/// </summary>
/// <param name="reconcileTimeout">Deadline applied to each reconcile round-trip.</param>
private sealed class BulkReconcileGate(TimeSpan reconcileTimeout)
{
/// <summary>Deadline applied to each reconcile round-trip in this batch.</summary>
public TimeSpan ReconcileTimeout { get; } = reconcileTimeout;
/// <summary>True once a reconcile in this batch hit its deadline.</summary>
public bool SiteUnreachable { get; private set; }
/// <summary>Number of reconciles the circuit-breaker skipped, for the batch's one summary log line.</summary>
public int SkippedCount { get; private set; }
/// <summary>Opens the circuit for the remainder of the batch.</summary>
public void MarkUnreachable() => SiteUnreachable = true;
/// <summary>Counts one reconcile skipped because the circuit is open.</summary>
public void CountSkipped() => SkippedCount++;
}
/// <summary>
/// Deploy every deployable instance at a site in one operation.
///
@@ -783,9 +836,18 @@ public class DeploymentService
/// <see cref="DeployInstanceAsync"/> — its own deployment id and revision hash,
/// its own operation lock held across the whole operation, its own
/// optimistically-concurrent status record, and the same query-before-redeploy
/// idempotency check. Instances that cannot be deployed (wrong state, failed
/// validation, lock already held) are reported as failed rows; the batch is
/// NOT all-or-nothing, mirroring the artifact deployment path.
/// idempotency check — with two batch-level differences. Instances that cannot
/// be deployed (wrong state, failed validation, lock already held) are reported
/// as failed rows; the batch is NOT all-or-nothing, mirroring the artifact
/// deployment path.
/// </para>
///
/// <para>
/// <b>Disabled instances are skipped entirely</b> and reported in
/// <see cref="SiteDeploymentSummary.SkippedInstances"/> — see the filter below
/// for why. <b>Reconciliation is bounded per batch</b> by
/// <see cref="ReconcileUnderBatchGateAsync"/>, so an unreachable site costs the
/// batch one short timeout rather than one 30-second query per instance.
/// </para>
/// </summary>
/// <param name="siteId">The database ID of the site whose instances are deployed.</param>
@@ -803,12 +865,30 @@ public class DeploymentService
var instances = await _siteRepository.GetInstancesBySiteIdAsync(siteId, cancellationToken);
if (instances.Count == 0)
return Result<SiteDeploymentSummary>.Success(new SiteDeploymentSummary(site.SiteIdentifier, [], 0, 0));
return Result<SiteDeploymentSummary>.Success(new SiteDeploymentSummary(site.SiteIdentifier, [], 0, 0, []));
// Deliberately-Disabled instances are NOT part of a bulk deploy. Deploy is a
// legal transition out of Disabled and the single-instance path uses it to
// deploy-and-enable, so including them here would have one site-wide push
// silently START every piece of equipment an operator had deliberately
// stopped. They are reported as skipped rows instead — no flatten, no send,
// no state change — and remain individually deployable, which is the
// explicit "yes, re-enable this one" action.
var deployable = new List<Instance>(instances.Count);
var skipped = new List<SkippedInstanceResult>();
foreach (var instance in instances)
{
if (instance.State == InstanceState.Disabled)
skipped.Add(new SkippedInstanceResult(instance.Id, instance.UniqueName, DisabledSkipReason));
else
deployable.Add(instance);
}
// ---- Phase 1: prepare, serially, on ONE shared flatten session. ----
var session = _flatteningPipeline.CreateSession();
var prepared = new List<PreparedDeployment>(instances.Count);
var results = new List<InstanceDeploymentResult>(instances.Count);
var reconcileGate = new BulkReconcileGate(_options.SiteDeploymentReconcileTimeout);
var prepared = new List<PreparedDeployment>(deployable.Count);
var results = new List<InstanceDeploymentResult>(deployable.Count);
// Index of the first prepared deployment phase 3 has NOT yet finalised.
// Everything from here to the end of `prepared` still owns an operation
@@ -817,14 +897,15 @@ public class DeploymentService
var finalizedUpTo = 0;
try
{
foreach (var instance in instances)
foreach (var instance in deployable)
{
cancellationToken.ThrowIfCancellationRequested();
PreparedDeployment step;
try
{
step = await PrepareDeploymentAsync(instance.Id, user, session, cancellationToken);
step = await PrepareDeploymentAsync(
instance.Id, user, session, reconcileGate, cancellationToken);
}
catch (Exception ex)
{
@@ -891,13 +972,26 @@ public class DeploymentService
site.SiteIdentifier,
results,
results.Count(r => r.Success),
results.Count(r => !r.Success));
results.Count(r => !r.Success),
skipped);
// ONE summary line for the whole batch, never one per skipped reconcile:
// an unreachable site would otherwise log N warnings for a single fact.
if (reconcileGate.SkippedCount > 0)
{
_logger.LogWarning(
"Site {SiteId} did not answer the query-before-redeploy reconciliation within {Timeout}; " +
"skipped it for the remaining {SkippedCount} instance(s) of this bulk deploy — they proceed " +
"on central's recorded state, with the site's own stale-revision rejection as the safety net",
site.SiteIdentifier, reconcileGate.ReconcileTimeout, reconcileGate.SkippedCount);
}
_logger.LogInformation(
"Bulk deployment of site {SiteId} complete: {SuccessCount} succeeded, {FailureCount} failed " +
"Bulk deployment of site {SiteId} complete: {SuccessCount} succeeded, {FailureCount} failed, " +
"{SkippedCount} skipped as Disabled " +
"({ChainLoads} template-chain load(s) across {InstanceCount} instance(s))",
site.SiteIdentifier, summary.SuccessCount, summary.FailureCount,
session.ChainLoads, instances.Count);
site.SiteIdentifier, summary.SuccessCount, summary.FailureCount, skipped.Count,
session.ChainLoads, deployable.Count);
return Result<SiteDeploymentSummary>.Success(summary);
}
@@ -1671,6 +1765,73 @@ public class DeploymentService
return null;
}
/// <summary>
/// Runs <see cref="TryReconcileWithSiteAsync"/> for one instance of a BULK
/// deploy, under the batch's short per-reconcile deadline and its
/// once-per-batch reachability circuit-breaker.
///
/// <para>
/// Phase 1 is strictly serial, so an unreachable site makes every instance pay
/// the full communication-layer <c>QueryTimeout</c> (30 s) in turn before a
/// single deploy is sent — 500 instances is roughly four hours, so the CLI's
/// 5-minute bulk-deploy deadline cancels the operation and the next attempt
/// re-arms the identical cost. A bulk redeploy could therefore never converge
/// while the site was down. Two bounds fix that: a short
/// <see cref="DeploymentManagerOptions.SiteDeploymentReconcileTimeout"/> per
/// reconcile, and — after the FIRST timeout — skipping the remaining reconciles
/// for the rest of the batch.
/// </para>
///
/// <para>
/// Skipping loses nothing: a reconcile that times out already returns
/// <see langword="null"/> and the instance proceeds on DB state with the site's
/// own stale-revision rejection as the safety net — which is exactly what every
/// subsequent reconcile against a down site would have produced, one 30-second
/// wait at a time.
/// </para>
///
/// <para>
/// The gate needs no synchronisation: it is touched only from the serial
/// prepare phase.
/// </para>
/// </summary>
/// <param name="instance">Instance being prepared.</param>
/// <param name="targetRevisionHash">Revision hash this deploy would apply.</param>
/// <param name="configJson">Serialized flattened config for the reconciled snapshot.</param>
/// <param name="currentUser">User attributed on the reconciliation audit row.</param>
/// <param name="gate">The batch's reconcile budget and reachability state.</param>
/// <param name="cancellationToken">Cancellation token for the batch.</param>
/// <returns>The reconciled prior record, or <see langword="null"/> to proceed with a normal deploy.</returns>
private async Task<DeploymentRecord?> ReconcileUnderBatchGateAsync(
Instance instance,
string targetRevisionHash,
string configJson,
string currentUser,
BulkReconcileGate gate,
CancellationToken cancellationToken)
{
if (gate.SiteUnreachable)
{
gate.CountSkipped();
return null;
}
using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
cts.CancelAfter(gate.ReconcileTimeout);
var reconciled = await TryReconcileWithSiteAsync(
instance, targetRevisionHash, configJson, currentUser, cts.Token);
// TryReconcileWithSiteAsync swallows its own query faults (it must — a
// failed query never aborts a deploy), so the deadline is detected on the
// token rather than on an exception. An outer cancellation is the caller
// aborting the whole batch, not evidence about the site.
if (reconciled == null && cts.IsCancellationRequested && !cancellationToken.IsCancellationRequested)
gate.MarkUnreachable();
return reconciled;
}
/// <summary>
/// The site is queried before a re-deploy only when a
/// prior record is stuck <see cref="DeploymentStatus.InProgress"/>, or is
@@ -1901,6 +2062,18 @@ public record InstanceDeploymentResult(
bool Success,
string? ErrorMessage);
/// <summary>
/// One instance a bulk site deployment did not attempt at all — no flatten, no
/// site round-trip, and no state change of any kind.
/// </summary>
/// <param name="InstanceId">Database id of the instance.</param>
/// <param name="UniqueName">The instance's unique name.</param>
/// <param name="Reason">Why it was skipped, phrased for the operator.</param>
public record SkippedInstanceResult(
int InstanceId,
string UniqueName,
string Reason);
/// <summary>
/// Result matrix of a bulk site deployment. Mirrors the shape of
/// <c>ArtifactDeploymentSummary</c>: successes are not rolled back when other
@@ -1911,8 +2084,16 @@ public record InstanceDeploymentResult(
/// <param name="InstanceResults">Per-instance outcomes.</param>
/// <param name="SuccessCount">Number of instances the site confirmed.</param>
/// <param name="FailureCount">Number of instances that failed for any reason.</param>
/// <param name="SkippedInstances">
/// Instances the batch deliberately did not attempt — currently the
/// <see cref="InstanceState.Disabled"/> ones, which a bulk push must not silently
/// re-enable. They are neither successes nor failures, so they are counted in
/// neither total. Additive and nullable so an older client deserialising this
/// summary is unaffected.
/// </param>
public record SiteDeploymentSummary(
string SiteIdentifier,
IReadOnlyList<InstanceDeploymentResult> InstanceResults,
int SuccessCount,
int FailureCount);
int FailureCount,
IReadOnlyList<SkippedInstanceResult>? SkippedInstances = null);
src/ZB.MOM.WW.ScadaBridge.DeploymentManager/FlattenSession.cs
+17
View File
@@ -86,6 +86,23 @@ public sealed class FlattenSession
/// </summary>
public IReadOnlyCollection<int> VisitedTemplateIds => _templates.Keys;
/// <summary>
/// Every visited template id paired with the watermark version this session
/// READ it at — the pre-flatten watermark the staleness fast path must memoise
/// against.
///
/// <para>
/// Re-reading the versions after the flatten instead is a correctness bug, not
/// an optimisation detail: an edit that lands mid-flatten pairs the PRE-edit
/// content (and therefore the pre-edit hash) with the POST-edit version, and
/// the memo then answers "not stale" until something else moves the watermark.
/// Recording the version each template was read at can only ever cause an extra
/// flatten, never a stale answer.
/// </para>
/// </summary>
public IReadOnlyDictionary<int, long> VisitedTemplateVersions =>
_templates.ToDictionary(entry => entry.Key, entry => entry.Value.Version);
/// <summary>
/// Returns the inheritance chain for <paramref name="templateId"/> (the
/// template itself, then each ancestor), loading it through
src/ZB.MOM.WW.ScadaBridge.DeploymentManager/StaleInstanceProbe.cs
+12 -11
View File
@@ -62,7 +62,7 @@ public sealed class StaleInstanceProbe : IStaleInstanceProbe
/// </summary>
/// <param name="Structure">Graph structure version at flatten time.</param>
/// <param name="InstanceVersion">The instance's own version at flatten time.</param>
/// <param name="TemplateVersions">Version of every template the flatten walked, by template id.</param>
/// <param name="TemplateVersions">Version every template the flatten walked was READ at, by template id.</param>
/// <param name="RevisionHash">The hash that reading produced.</param>
private sealed record ProbeMemo(
long Structure,
@@ -85,10 +85,15 @@ public sealed class StaleInstanceProbe : IStaleInstanceProbe
// script-compilation stage (this probe is called per-instance across a whole
// bundle import in BundleImporter.ComputeStaleInstanceIdsAsync).
//
// Watermark readings are taken BEFORE the flatten. A mutation that lands
// during the flatten therefore bumps a version past the recorded one and
// invalidates the memo on the next read, rather than being captured as if
// it had already been included.
// Watermark readings are taken BEFORE the flatten they describe. A mutation
// that lands during the flatten therefore bumps a version past the recorded
// one and invalidates the memo on the next read, rather than being captured
// as if it had already been included. The structure and instance versions
// are read here; the per-template versions come from the session, which
// records each template's version at the moment it read that template (see
// FlattenSession.VisitedTemplateVersions) — re-reading them after the
// flatten would pair a pre-edit hash with a post-edit version and make the
// memo serve "not stale" indefinitely.
var structure = _watermark.StructureVersion;
var instanceVersion = _watermark.GetInstanceVersion(instanceId);
@@ -103,12 +108,8 @@ public sealed class StaleInstanceProbe : IStaleInstanceProbe
if (Memos.Count >= MaxCachedProbes)
Memos.Clear();
var visited = session.VisitedTemplateIds;
var versions = new Dictionary<int, long>(visited.Count);
foreach (var templateId in visited)
versions[templateId] = _watermark.GetTemplateVersion(templateId);
Memos[instanceId] = new ProbeMemo(structure, instanceVersion, versions, result.Value.RevisionHash);
Memos[instanceId] = new ProbeMemo(
structure, instanceVersion, session.VisitedTemplateVersions, result.Value.RevisionHash);
return result.Value.RevisionHash;
}
src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/CentralHealthReportLoop.cs
+2 -1
View File
@@ -132,7 +132,8 @@ public class CentralHealthReportLoop : BackgroundService
alarmErrors: report.AlarmEvaluationErrorCount,
deadLetters: report.DeadLetterCount,
siteAuditWriteFailures: report.SiteAuditWriteFailures,
auditRedactionFailures: report.AuditRedactionFailure);
auditRedactionFailures: report.AuditRedactionFailure,
scriptRunShedCount: report.ScriptRunShedCount);
throw;
}
src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/HealthReportSender.cs
+2 -1
View File
@@ -182,7 +182,8 @@ public class HealthReportSender : BackgroundService
alarmErrors: report.AlarmEvaluationErrorCount,
deadLetters: report.DeadLetterCount,
siteAuditWriteFailures: report.SiteAuditWriteFailures,
auditRedactionFailures: report.AuditRedactionFailure);
auditRedactionFailures: report.AuditRedactionFailure,
scriptRunShedCount: report.ScriptRunShedCount);
throw;
}
src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ISiteHealthCollector.cs
+8 -1
View File
@@ -286,12 +286,19 @@ public interface ISiteHealthCollector
/// <param name="deadLetters">Dead letter count to add back.</param>
/// <param name="siteAuditWriteFailures">Site audit write failure count to add back.</param>
/// <param name="auditRedactionFailures">Audit redaction failure count to add back.</param>
/// <param name="scriptRunShedCount">
/// Interval-drained script/alarm run-shed count to add back (arch-review 2026-08-19
/// P8-1). Additive trailing parameter defaulting to 0 so existing callers keep
/// compiling; <c>DetachedScriptThreads</c>/<c>SiteStreamAlarmDropCount</c> are
/// point-in-time gauges, not drained on collect, so they need no restore path.
/// </param>
void AddIntervalCounters(
int scriptErrors,
int alarmErrors,
int deadLetters,
int siteAuditWriteFailures,
int auditRedactionFailures)
int auditRedactionFailures,
int scriptRunShedCount = 0)
{
// Default no-op so test fakes do not need to be updated. The real
// SiteHealthCollector overrides this with the Interlocked.Add restore.
src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthCollector.cs
+9 -3
View File
@@ -245,19 +245,25 @@ public class SiteHealthCollector : ISiteHealthCollector
int alarmErrors,
int deadLetters,
int siteAuditWriteFailures,
int auditRedactionFailures)
int auditRedactionFailures,
int scriptRunShedCount = 0)
{
// Each counter is restored atomically via
// Interlocked.Add so an increment that arrived during the failed Send
// (and therefore accumulated against the zero left by CollectReport's
// Exchange) is correctly summed with the values being put back. No
// ordering between the five Adds is required — they target independent
// fields.
// ordering between the six Adds is required — they target independent
// fields. scriptRunShedCount joins this restore path (arch-review
// 2026-08-19 P8-1) because it is drained by CollectReport the same way
// as the other five; DetachedScriptThreads and SiteStreamAlarmDropCount
// are point-in-time gauges (CompareExchange/Read, never zeroed on
// collect) so a failed Send loses nothing for them to restore.
if (scriptErrors != 0) Interlocked.Add(ref _scriptErrorCount, scriptErrors);
if (alarmErrors != 0) Interlocked.Add(ref _alarmErrorCount, alarmErrors);
if (deadLetters != 0) Interlocked.Add(ref _deadLetterCount, deadLetters);
if (siteAuditWriteFailures != 0) Interlocked.Add(ref _siteAuditWriteFailures, siteAuditWriteFailures);
if (auditRedactionFailures != 0) Interlocked.Add(ref _auditRedactionFailures, auditRedactionFailures);
if (scriptRunShedCount != 0) Interlocked.Add(ref _scriptRunShedCount, scriptRunShedCount);
}
/// <inheritdoc />
src/ZB.MOM.WW.ScadaBridge.Host/Program.cs
+3 -14
View File
@@ -757,21 +757,10 @@ try
// metrics listener never collides with the Akka remoting port on site nodes.
var metricsPort = configuration.GetValue<int>("ScadaBridge:Node:MetricsPort", 8084);
// Configure Kestrel for HTTP/2 only on the gRPC port
// Configure Kestrel for HTTP/2 only on the gRPC port (plus the h2 keepalive that
// makes a half-open inbound stream detectable) — see SiteKestrelSetup.
builder.WebHost.ConfigureKestrel(options =>
{
options.ListenAnyIP(grpcPort, listenOptions =>
{
listenOptions.Protocols = Microsoft.AspNetCore.Server.Kestrel.Core.HttpProtocols.Http2;
});
// Dedicated HTTP/1.1 (and HTTP/2) listener for the Prometheus /metrics
// scrape endpoint, reachable by an HTTP/1.1 scraper.
options.ListenAnyIP(metricsPort, listenOptions =>
{
listenOptions.Protocols = Microsoft.AspNetCore.Server.Kestrel.Core.HttpProtocols.Http1AndHttp2;
});
});
SiteKestrelSetup.Configure(options, grpcPort, metricsPort));
// gRPC server registration. Two interceptors, two disjoint service prefixes, two
// separate keys — neither one's absence weakens the other:
src/ZB.MOM.WW.ScadaBridge.Host/SiteKestrelSetup.cs
+66
View File
@@ -0,0 +1,66 @@
using Microsoft.AspNetCore.Server.Kestrel.Core;
namespace ZB.MOM.WW.ScadaBridge.Host;
/// <summary>
/// Kestrel configuration for a Site node: the h2c gRPC listener, the HTTP/1.1 metrics listener, and
/// the HTTP/2 keepalive that governs both.
/// </summary>
/// <remarks>
/// Extracted from <c>Program.cs</c>'s Site branch so the keepalive settings are assertable — the
/// existing Host wiring tests re-declare their own <c>ConfigureKestrel</c> lambda and therefore
/// prove nothing about the one that ships.
/// </remarks>
internal static class SiteKestrelSetup
{
/// <summary>
/// How long an HTTP/2 connection may sit idle before Kestrel sends a keepalive PING.
/// </summary>
/// <remarks>
/// <para>
/// Kestrel's default is <see cref="TimeSpan.MaxValue"/> — no server-initiated pings at all — and
/// every long-lived stream this port carries is legitimately silent for long stretches: the
/// LocalDb replication sync stream on a converged pair, and a <c>SiteStreamService</c>
/// subscription for an idle site. A hard crash or network partition of the peer therefore leaves
/// this node holding a half-open stream that will never produce another byte and never fail
/// either. For LocalDb sync that is the expensive one: the handler sits inside its single-stream
/// guard, so the rebooted peer's reconnects are refused while this node still reports itself
/// replication-connected (ScadaBridge arch-review 2026-08-19, P4-3).
/// </para>
/// <para>
/// 60s costs one frame per idle minute per connection and bounds detection at roughly
/// <see cref="Http2KeepAlivePingDelay"/> + <see cref="Http2KeepAlivePingTimeout"/>. Pings are
/// ordinary HTTP/2 frames, so this is unrelated to the listener being h2c rather than TLS.
/// </para>
/// </remarks>
internal static readonly TimeSpan Http2KeepAlivePingDelay = TimeSpan.FromSeconds(60);
/// <summary>How long Kestrel waits for a keepalive PING ack before closing the connection.</summary>
internal static readonly TimeSpan Http2KeepAlivePingTimeout = TimeSpan.FromSeconds(20);
/// <summary>
/// Binds the site node's two listeners and applies the server-wide HTTP/2 keepalive.
/// </summary>
/// <remarks>
/// The keepalive lives on <c>Limits</c>, which is per-server rather than per-listener, so it
/// governs the gRPC port and the metrics port alike. That is the intent: the metrics listener
/// also speaks HTTP/2, and a scrape connection has nothing to lose from a ping.
/// </remarks>
public static void Configure(KestrelServerOptions options, int grpcPort, int metricsPort)
{
options.Limits.Http2.KeepAlivePingDelay = Http2KeepAlivePingDelay;
options.Limits.Http2.KeepAlivePingTimeout = Http2KeepAlivePingTimeout;
options.ListenAnyIP(grpcPort, listenOptions =>
{
listenOptions.Protocols = HttpProtocols.Http2;
});
// Dedicated HTTP/1.1 (and HTTP/2) listener for the Prometheus /metrics
// scrape endpoint, reachable by an HTTP/1.1 scraper.
options.ListenAnyIP(metricsPort, listenOptions =>
{
listenOptions.Protocols = HttpProtocols.Http1AndHttp2;
});
}
}
src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs
+11 -8
View File
@@ -210,20 +210,23 @@ public static class SiteServiceRegistration
return new AkkaClusterNodeProvider(akkaService, siteRole);
});
// The EventLogPurgeService runs on every
// site host node but consults this optional gate each tick and early-exits on
// the standby. Register it to delegate to IClusterNodeProvider.SelfIsPrimary
// (the canonical "this node is the oldest Up member (singleton host)" check) so purge
// runs ONLY on the active node — no duplicated cluster logic. Non-clustered test hosts that
// never call SiteServiceRegistration leave it unregistered, so the purge defaults
// to always-run (the pre-fix behaviour, preserved).
// The EventLogPurgeService runs on every site host node and, as of arch-review
// 2026-08-19 P4-4, purges its own site_events file on EVERY node — the old
// early-exit-on-standby gate this delegate drove is gone (DataConnectionActor
// writes site_events on both site nodes, so a standby that never purged would grow
// unbounded on an unreplicated site). Register it to delegate to
// IClusterNodeProvider.SelfIsPrimary (the canonical "this node is the oldest Up
// member (singleton host)" check) purely so the purge tick's log line can say
// whether this node is active — no duplicated cluster logic, no gating. Non-clustered
// test hosts that never call SiteServiceRegistration leave it unregistered, so the
// purge just logs every tick as active.
services.AddSingleton<SiteEventLogActiveNodeCheck>(sp =>
{
var nodeProvider = sp.GetRequiredService<IClusterNodeProvider>();
return () => nodeProvider.SelfIsPrimary;
});
// Wording-only companion to the gate above: when a purge on a replication-enabled node
// Wording-only companion to the delegate above: when a purge on a replication-enabled node
// actually deletes rows, it appends an operator note explaining the LocalDbOplogBacklog
// spike that follows. site_events is a replicated table and a retention/cap DELETE is
// CDC-captured like any other write (LocalDb Phase 2 — no purge-exemption path), so the
src/ZB.MOM.WW.ScadaBridge.ManagementService/AuditEndpoints.cs
+51 -14
View File
@@ -760,17 +760,30 @@ public static class AuditEndpoints
}
// ─────────────────────────────────────────────────────────────────────
// CSV helpers — 21 columns in AuditEvent declaration order
// CSV helpers — 24 columns, same order/naming as the JSONL export
// (AuditExportRow property declaration order; arch-review 2026-08-19 P6-7)
// ─────────────────────────────────────────────────────────────────────
public const string CsvHeader =
"EventId,OccurredAtUtc,IngestedAtUtc,Channel,Kind,CorrelationId," +
"SourceSiteId,SourceInstanceId,SourceScript,Actor,Target,Status," +
"EventId,OccurredAtUtc,IngestedAtUtc,Channel,Kind,CorrelationId,ExecutionId,ParentExecutionId," +
"SourceSiteId,SourceNode,SourceInstanceId,SourceScript,Actor,Target,Status," +
"HttpStatus,DurationMs,ErrorMessage,ErrorDetail,RequestSummary," +
"ResponseSummary,PayloadTruncated,Extra,ForwardState";
/// <summary>
/// Formats a single <see cref="AuditEvent"/> as an RFC 4180 CSV row matching <see cref="CsvHeader"/>.
/// Free-text columns (anything that can carry inbound-API-caller-controlled or
/// site-script-controlled text — <see cref="AuditExportRow.SourceSiteId"/>,
/// <see cref="AuditExportRow.SourceNode"/>, <see cref="AuditExportRow.SourceInstanceId"/>,
/// <see cref="AuditExportRow.SourceScript"/>, <see cref="AuditExportRow.Actor"/>,
/// <see cref="AuditExportRow.Target"/>, <see cref="AuditExportRow.ErrorMessage"/>,
/// <see cref="AuditExportRow.ErrorDetail"/>, <see cref="AuditExportRow.RequestSummary"/>,
/// <see cref="AuditExportRow.ResponseSummary"/>, <see cref="AuditExportRow.Extra"/>) are run
/// through <see cref="NeedsFormulaNeutralization"/> before RFC 4180 quoting so a leading
/// <c>=</c>/<c>+</c>/<c>-</c>/<c>@</c>/TAB/CR can't be interpreted as a formula or DDE
/// trigger by a spreadsheet application (CWE-1236). GUID/enum/numeric/date/bool columns the
/// exporter formats itself are never neutralized — none of them can start with a trigger
/// character, and neutralizing a signed numeric column would corrupt it.
/// </summary>
/// <param name="evt">The audit event to format.</param>
/// <returns>An RFC 4180 CSV row string (no trailing newline) for the supplied event.</returns>
@@ -783,20 +796,23 @@ public static class AuditEndpoints
AppendField(sb, evt.Channel.ToString());
AppendField(sb, evt.Kind.ToString());
AppendField(sb, evt.CorrelationId?.ToString());
AppendField(sb, evt.SourceSiteId);
AppendField(sb, evt.SourceInstanceId);
AppendField(sb, evt.SourceScript);
AppendField(sb, evt.Actor);
AppendField(sb, evt.Target);
AppendField(sb, evt.ExecutionId?.ToString());
AppendField(sb, evt.ParentExecutionId?.ToString());
AppendField(sb, evt.SourceSiteId, freeText: true);
AppendField(sb, evt.SourceNode, freeText: true);
AppendField(sb, evt.SourceInstanceId, freeText: true);
AppendField(sb, evt.SourceScript, freeText: true);
AppendField(sb, evt.Actor, freeText: true);
AppendField(sb, evt.Target, freeText: true);
AppendField(sb, evt.Status.ToString());
AppendField(sb, evt.HttpStatus?.ToString(CultureInfo.InvariantCulture));
AppendField(sb, evt.DurationMs?.ToString(CultureInfo.InvariantCulture));
AppendField(sb, evt.ErrorMessage);
AppendField(sb, evt.ErrorDetail);
AppendField(sb, evt.RequestSummary);
AppendField(sb, evt.ResponseSummary);
AppendField(sb, evt.ErrorMessage, freeText: true);
AppendField(sb, evt.ErrorDetail, freeText: true);
AppendField(sb, evt.RequestSummary, freeText: true);
AppendField(sb, evt.ResponseSummary, freeText: true);
AppendField(sb, evt.PayloadTruncated.ToString());
AppendField(sb, evt.Extra);
AppendField(sb, evt.Extra, freeText: true);
AppendField(sb, evt.ForwardState?.ToString());
return sb.ToString();
}
@@ -807,7 +823,7 @@ public static class AuditEndpoints
private static string FormatDate(DateTime value) =>
value.ToString("O", CultureInfo.InvariantCulture);
private static void AppendField(StringBuilder sb, string? value, bool first = false)
private static void AppendField(StringBuilder sb, string? value, bool first = false, bool freeText = false)
{
if (!first) sb.Append(',');
if (string.IsNullOrEmpty(value))
@@ -815,6 +831,16 @@ public static class AuditEndpoints
return;
}
// OWASP CSV-injection neutralization (CWE-1236): a free-text cell whose
// first character would be interpreted by a spreadsheet application as a
// formula/DDE trigger is prefixed with a literal single quote. Applied
// BEFORE RFC 4180 quoting, and only to free-text columns — never to the
// GUID/enum/numeric/date/bool columns this formatter generates itself.
if (freeText && NeedsFormulaNeutralization(value))
{
value = "'" + value;
}
// RFC 4180: quote on comma / quote / CR / LF; double-up embedded quotes.
if (value.IndexOfAny(s_quoteTriggers) < 0)
{
@@ -837,5 +863,16 @@ public static class AuditEndpoints
sb.Append('"');
}
/// <summary>
/// True when <paramref name="value"/> starts with a character a spreadsheet
/// application would treat as a formula/DDE-launch trigger — the OWASP CSV
/// Injection standard set: <c>=</c>, <c>+</c>, <c>-</c>, <c>@</c>, TAB, CR.
/// </summary>
private static bool NeedsFormulaNeutralization(string value)
{
var c = value[0];
return c == '=' || c == '+' || c == '-' || c == '@' || c == '\t' || c == '\r';
}
private static readonly char[] s_quoteTriggers = { ',', '"', '\r', '\n' };
}
src/ZB.MOM.WW.ScadaBridge.ManagementService/DebugStreamHub.cs
+59 -3
View File
@@ -19,6 +19,7 @@ public class DebugStreamHub : Hub
private const string SessionIdKey = "DebugStreamSessionId";
private const string RolesKey = "DebugStreamRoles";
private const string PermittedSiteIdsKey = "DebugStreamPermittedSiteIds";
private const string DisconnectedKey = "DebugStreamDisconnected";
/// <summary>
/// Pure site-scope authorization check for a debug-stream subscription.
@@ -192,7 +193,35 @@ public class DebugStreamHub : Hub
CancellationToken.None, TaskContinuationOptions.OnlyOnFaulted, TaskScheduler.Default);
});
Context.Items[SessionIdKey] = session.SessionId;
// arch-review 2026-08-19 P6-6: the client may have disconnected while
// StartStreamAsync was in flight (up to 30s). OnDisconnectedAsync already ran
// and found no session id to release, so publishing it here would leave the
// session registered in the singleton DebugStreamService — whose keepalive
// renews registered sessions forever, pinning the site subscription and its
// bridge actor until process restart. Publish and re-check liveness under the
// per-connection Items monitor (the same one UnsubscribeInstance takes) so
// exactly one side tears the session down.
var abandoned = false;
lock (Context.Items)
{
if (Context.Items.ContainsKey(DisconnectedKey) || Context.ConnectionAborted.IsCancellationRequested)
{
abandoned = true;
}
else
{
Context.Items[SessionIdKey] = session.SessionId;
}
}
if (abandoned)
{
_debugStreamService.StopStream(session.SessionId);
_logger.LogInformation(
"DebugStreamHub: {ConnectionId} disconnected while subscribing to instance {InstanceId} — session released",
connectionId, instanceId);
return;
}
// Send the initial snapshot
await Clients.Caller.SendAsync("OnSnapshot", session.InitialSnapshot);
@@ -214,18 +243,45 @@ public class DebugStreamHub : Hub
/// <returns>A task that represents the asynchronous operation.</returns>
public Task UnsubscribeInstance()
{
if (Context.Items.TryGetValue(SessionIdKey, out var sessionIdObj) && sessionIdObj is string sessionId)
var sessionId = TakeSessionId();
if (sessionId is not null)
{
_debugStreamService.StopStream(sessionId);
Context.Items.Remove(SessionIdKey);
_logger.LogInformation("DebugStreamHub: {ConnectionId} unsubscribed", Context.ConnectionId);
}
return Task.CompletedTask;
}
/// <summary>
/// Atomically removes and returns the session id registered for this connection, so a
/// release path and the late publish in <see cref="SubscribeInstance"/> cannot both
/// claim (or both miss) the same session — see P6-6 there.
/// </summary>
/// <returns>The session id that was registered, or <see langword="null"/> if none was.</returns>
private string? TakeSessionId()
{
lock (Context.Items)
{
if (Context.Items.TryGetValue(SessionIdKey, out var sessionIdObj) && sessionIdObj is string sessionId)
{
Context.Items.Remove(SessionIdKey);
return sessionId;
}
}
return null;
}
/// <inheritdoc />
public override async Task OnDisconnectedAsync(Exception? exception)
{
// P6-6: mark the connection gone BEFORE releasing, so a SubscribeInstance still
// awaiting StartStreamAsync sees the flag and stops its own session rather than
// registering it against a connection nothing will ever unsubscribe.
lock (Context.Items)
{
Context.Items[DisconnectedKey] = true;
}
await UnsubscribeInstance();
await base.OnDisconnectedAsync(exception);
}
src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs
+80 -21
View File
@@ -394,7 +394,7 @@ public class ManagementActor : ReceiveActor
MoveAreaCommand cmd => await HandleMoveArea(sp, cmd, user.Username),
// Data Connections
ListDataConnectionsCommand cmd => await HandleListDataConnections(sp, cmd),
ListDataConnectionsCommand cmd => await HandleListDataConnections(sp, cmd, user),
GetDataConnectionCommand cmd => await HandleGetDataConnection(sp, cmd, user),
CreateDataConnectionCommand cmd => await HandleCreateDataConnection(sp, cmd, user.Username),
UpdateDataConnectionCommand cmd => await HandleUpdateDataConnection(sp, cmd, user.Username),
@@ -1413,13 +1413,22 @@ public class ManagementActor : ReceiveActor
private static async Task<object?> HandleSubmitSecuredWrite(
IServiceProvider sp, SubmitSecuredWriteCommand cmd, AuthenticatedUser user)
{
var siteRepo = sp.GetRequiredService<ISiteRepository>();
var site = await siteRepo.GetSiteByIdentifierAsync(cmd.SiteId)
?? throw new ManagementCommandException($"Site '{cmd.SiteId}' not found.");
// Site scope (arch-review R2 N3): an LDAP-mapping-scoped Operator may only
// submit device writes to their permitted sites — same rule C2 applies to
// DeployArtifacts, on a higher-consequence path. Checked before the row exists.
//
// The unresolvable case is fail-CLOSED and answered BEFORE the "not found"
// (arch-review 2026-08-19 P6-5): resolving first and reporting not-found made
// this handler a site-identifier existence oracle for an out-of-scope caller,
// who now gets the same scope denial whether or not the site exists.
var siteRepo = sp.GetRequiredService<ISiteRepository>();
var site = await siteRepo.GetSiteByIdentifierAsync(cmd.SiteId);
if (site is null)
{
DenyUnresolvedTargetIfScoped(user, $"site '{cmd.SiteId}'");
throw new ManagementCommandException($"Site '{cmd.SiteId}' not found.");
}
EnforceSiteScope(user, site.Id);
var connections = await siteRepo.GetDataConnectionsBySiteIdAsync(site.Id);
@@ -1461,18 +1470,27 @@ public class ManagementActor : ReceiveActor
IServiceProvider sp, ApproveSecuredWriteCommand cmd, AuthenticatedUser user)
{
var repo = sp.GetRequiredService<ISecuredWriteRepository>();
var row = await repo.GetAsync(cmd.Id)
?? throw new ManagementCommandException($"Secured write {cmd.Id} not found.");
if (!string.Equals(row.Status, "Pending", StringComparison.Ordinal))
throw new ManagementCommandException(
$"Secured write {cmd.Id} is '{row.Status}', not Pending; it cannot be approved.");
var row = await repo.GetAsync(cmd.Id);
if (row is null)
{
DenyUnresolvedTargetIfScoped(user, $"secured write {cmd.Id}");
throw new ManagementCommandException($"Secured write {cmd.Id} not found.");
}
// Site scope (arch-review R2 N3): the approving Verifier must be permitted on
// the row's target site — checked BEFORE the TTL/self-approval/CAS chain so an
// out-of-scope verifier can neither consume the Pending transition nor relay.
//
// It is also checked BEFORE the status test (arch-review 2026-08-19 P6-5): the
// "is 'Executed', not Pending" message told an out-of-scope caller both that the
// id existed and what had happened to it. An out-of-scope caller now gets the
// same scope denial for every id, existing or not.
await EnforceSiteScopeForIdentifier(sp, user, row.SiteId);
if (!string.Equals(row.Status, "Pending", StringComparison.Ordinal))
throw new ManagementCommandException(
$"Secured write {cmd.Id} is '{row.Status}', not Pending; it cannot be approved.");
// Server-side TTL: a Pending write older than the configured age is transitioned
// to Expired and can never be approved — the stale setpoint is never relayed
// (arch-review S2). Checked BEFORE the self-approval/CAS so an overdue row never
@@ -1631,17 +1649,24 @@ public class ManagementActor : ReceiveActor
IServiceProvider sp, RejectSecuredWriteCommand cmd, AuthenticatedUser user)
{
var repo = sp.GetRequiredService<ISecuredWriteRepository>();
var entity = await repo.GetAsync(cmd.Id)
?? throw new ManagementCommandException($"Secured write {cmd.Id} not found.");
var entity = await repo.GetAsync(cmd.Id);
if (entity is null)
{
DenyUnresolvedTargetIfScoped(user, $"secured write {cmd.Id}");
throw new ManagementCommandException($"Secured write {cmd.Id} not found.");
}
// Site scope (arch-review R2 N3): the rejecting Verifier must be permitted on
// the row's target site — checked BEFORE the TTL/self-approval chain, and (as on
// approve) BEFORE the status test, so the not-found/status errors stop being an
// existence-and-status oracle for an out-of-scope caller (arch-review 2026-08-19
// P6-5). The CAS below is unchanged.
await EnforceSiteScopeForIdentifier(sp, user, entity.SiteId);
if (!string.Equals(entity.Status, "Pending", StringComparison.Ordinal))
throw new ManagementCommandException(
$"Secured write {cmd.Id} is '{entity.Status}', not Pending; it cannot be rejected.");
// Site scope (arch-review R2 N3): the rejecting Verifier must be permitted on
// the row's target site — checked BEFORE the TTL/self-approval chain.
await EnforceSiteScopeForIdentifier(sp, user, entity.SiteId);
// Server-side TTL: an overdue Pending write expires here too, so a reject on a
// stale row deterministically reports the expiry rather than recording a
// human decision (arch-review S2).
@@ -1902,12 +1927,38 @@ public class ManagementActor : ReceiveActor
};
}
private static async Task<object?> HandleListDataConnections(IServiceProvider sp, ListDataConnectionsCommand cmd)
/// <summary>
/// Lists data connections, constrained to the caller's permitted sites.
/// </summary>
/// <remarks>
/// Arch-review 2026-08-19 P6-4. The list used to take no user at all and return
/// EVERY site's connections, which nullified the row-98 fail-closed gate on
/// <see cref="HandleGetDataConnection"/> — Get denied exactly what List handed
/// over, and the secret-elided <see cref="DataConnectionPublicShape"/> still
/// leaked site topology plus an id-existence oracle. Scoping now mirrors
/// <see cref="HandleListSites"/>: an explicit site filter is scope-checked like
/// any other site-keyed command, and an unfiltered list from a site-scoped
/// non-admin caller is narrowed to their permitted sites. System-wide callers
/// (Administrator, or a Deployer with no site grants) are unaffected.
/// </remarks>
private static async Task<object?> HandleListDataConnections(
IServiceProvider sp, ListDataConnectionsCommand cmd, AuthenticatedUser user)
{
var repo = sp.GetRequiredService<ISiteRepository>();
if (cmd.SiteId.HasValue)
{
EnforceSiteScope(user, cmd.SiteId.Value);
return (await repo.GetDataConnectionsBySiteIdAsync(cmd.SiteId.Value)).Select(DataConnectionPublicShape).ToList();
return (await repo.GetAllDataConnectionsAsync()).Select(DataConnectionPublicShape).ToList();
}
IEnumerable<DataConnection> connections = await repo.GetAllDataConnectionsAsync();
if (user.PermittedSiteIds.Length > 0 && !user.Roles.Contains(Roles.Administrator, StringComparer.OrdinalIgnoreCase))
{
var permittedIds = new HashSet<string>(user.PermittedSiteIds);
connections = connections.Where(c => permittedIds.Contains(c.SiteId.ToString()));
}
return connections.Select(DataConnectionPublicShape).ToList();
}
/// <summary>
@@ -3631,9 +3682,17 @@ public class ManagementActor : ReceiveActor
private static async Task<object?> HandleDebugSnapshot(IServiceProvider sp, DebugSnapshotCommand cmd, AuthenticatedUser user)
{
// Fail-CLOSED before the not-found (arch-review 2026-08-19 P6-5): reporting
// "Instance {id} not found" ahead of the scope check made this handler an
// instance-id existence oracle for an out-of-scope caller. The scope denial is
// now the same answer whether or not the instance exists.
var instanceRepo = sp.GetRequiredService<ITemplateEngineRepository>();
var instance = await instanceRepo.GetInstanceByIdAsync(cmd.InstanceId)
?? throw new ManagementCommandException($"Instance {cmd.InstanceId} not found.");
var instance = await instanceRepo.GetInstanceByIdAsync(cmd.InstanceId);
if (instance is null)
{
DenyUnresolvedTargetIfScoped(user, $"instance {cmd.InstanceId}");
throw new ManagementCommandException($"Instance {cmd.InstanceId} not found.");
}
EnforceSiteScope(user, instance.SiteId);
src/ZB.MOM.WW.ScadaBridge.Security/LoginThrottle.cs
+127 -53
View File
@@ -4,7 +4,8 @@ using Microsoft.Extensions.Options;
namespace ZB.MOM.WW.ScadaBridge.Security;
/// <summary>
/// Fixed-window LDAP-bind failure lockout, keyed per <c>{username}|{ip}</c>.
/// Fixed-window LDAP-bind failure lockout, layering a per-<c>{username}|{ip}</c> budget with a
/// secondary, coarser per-IP budget.
/// </summary>
/// <remarks>
/// <para>
@@ -17,27 +18,51 @@ namespace ZB.MOM.WW.ScadaBridge.Security;
/// determined spray hitting both nodes still burns its budget twice as fast on each).
/// </para>
/// <para>
/// <b>Two tiers, same clock.</b> The primary counter is keyed per <c>{username}|{ip}</c> pair
/// and stops a brute-force attempt against one account. On its own that counter is blind to a
/// <b>spray</b> — one attacker IP trying one password against many distinct usernames never
/// accumulates <see cref="SecurityOptions.MaxLoginFailuresPerWindow"/> failures on any single
/// pair. A secondary counter keyed on the IP alone closes that gap: it uses the same fixed
/// window (<see cref="SecurityOptions.LoginFailureWindowMinutes"/>) and lockout duration
/// (<see cref="SecurityOptions.LoginLockoutMinutes"/>) but a higher ceiling
/// (<see cref="SecurityOptions.PerIpMaxFailures"/>), so a shared NAT gateway absorbing a
/// handful of ordinary typos across a few users stays under it while a spray does not.
/// <see cref="IsLockedOut"/> refuses a bind attempt when EITHER counter for the pair's IP is
/// currently locked out.
/// </para>
/// <para>
/// <b>Success-path semantics are asymmetric by design.</b> <see cref="RecordSuccess"/> clears
/// only the per-pair counter, never the per-IP counter — a spray that happens to guess (or
/// already knows) one valid account on the sprayed IP must not be able to reset the IP's
/// failure budget and keep spraying indefinitely. The per-IP counter's only relief is window
/// expiry (or lockout expiry once armed); this is the cheapest safe choice and intentionally
/// does not distinguish "this IP's failures are all from one confused legitimate user" from
/// "this IP just found one valid account mid-spray" — the throttle cannot tell those apart.
/// </para>
/// <para>
/// The counter uses a <b>fixed window</b>: the first failure opens a window of
/// <see cref="SecurityOptions.LoginFailureWindowMinutes"/>; reaching
/// <see cref="SecurityOptions.MaxLoginFailuresPerWindow"/> failures inside that window locks
/// the key out for <see cref="SecurityOptions.LoginLockoutMinutes"/>. A successful bind or an
/// expired window resets the counter. Setting <see cref="SecurityOptions.MaxLoginFailuresPerWindow"/>
/// to 0 disables throttling entirely.
/// <see cref="SecurityOptions.LoginFailureWindowMinutes"/>; reaching the tier's ceiling
/// (<see cref="SecurityOptions.MaxLoginFailuresPerWindow"/> per pair,
/// <see cref="SecurityOptions.PerIpMaxFailures"/> per IP) inside that window locks the key out
/// for <see cref="SecurityOptions.LoginLockoutMinutes"/>. An expired window resets the count.
/// Setting <see cref="SecurityOptions.MaxLoginFailuresPerWindow"/> to 0 disables both tiers
/// entirely.
/// </para>
/// <para>
/// Registered as a singleton (<see cref="ServiceCollectionExtensions.AddSecurity"/>) so the
/// counters are shared across all requests on a node. Thread-safe via
/// <see cref="ConcurrentDictionary{TKey,TValue}"/> and atomic entry swaps. Memory is bounded:
/// each write opportunistically prunes expired entries, and if the map still exceeds
/// <see cref="MaxTrackedKeys"/> the oldest-windowed entries are dropped — so a spray across a
/// large IP/username space cannot grow the map without limit.
/// each write opportunistically prunes expired entries in both maps, and if either map still
/// exceeds <see cref="MaxTrackedKeys"/> the oldest-windowed entries are dropped — so a spray
/// across a large IP/username space cannot grow either map without limit.
/// </para>
/// </remarks>
public sealed class LoginThrottle
{
/// <summary>
/// Hard cap on the number of tracked <c>{username}|{ip}</c> keys. Above this the
/// oldest-windowed entries are dropped on write to bound memory under a spray attack.
/// Hard cap on the number of tracked keys in EITHER the per-pair (<c>{username}|{ip}</c>)
/// or per-IP map. Above this the oldest-windowed entries are dropped on write to bound
/// memory under a spray attack.
/// </summary>
public const int MaxTrackedKeys = 10_000;
@@ -48,16 +73,23 @@ public sealed class LoginThrottle
/// window/lockout expiry directly and never depends on pruning.</summary>
internal const int PruneEveryNFailures = 64;
/// <summary>Diagnostics/test accessor: number of currently tracked keys.</summary>
/// <summary>Diagnostics/test accessor: number of currently tracked per-pair keys.</summary>
internal int TrackedKeyCount => _entries.Count;
/// <summary>Diagnostics/test accessor: number of currently tracked per-IP keys.</summary>
internal int TrackedIpCount => _ipEntries.Count;
private int _failureWrites;
private readonly TimeProvider _clock;
private readonly IOptions<SecurityOptions> _options;
/// <summary>Primary tier: one entry per <c>{username}|{ip}</c> pair.</summary>
private readonly ConcurrentDictionary<string, Entry> _entries = new(StringComparer.Ordinal);
/// <summary>Secondary tier: one entry per IP, catching a spray across many usernames.</summary>
private readonly ConcurrentDictionary<string, Entry> _ipEntries = new(StringComparer.Ordinal);
/// <summary>
/// One tracked key's state: the failure <paramref name="Count"/>, the
/// <paramref name="WindowStart"/> that anchors the fixed failure window, and — once the
@@ -76,26 +108,36 @@ public sealed class LoginThrottle
/// <summary>
/// Returns true when the given <paramref name="username"/> + <paramref name="ip"/> pair is
/// currently locked out and a bind must be refused without contacting LDAP. Never mutates
/// state (an expired lockout simply reads as unlocked; the entry is cleared on the next write).
/// currently locked out, OR <paramref name="ip"/> itself is locked out under the secondary
/// spray ceiling, and a bind must be refused without contacting LDAP. Never mutates state
/// (an expired lockout simply reads as unlocked; the entry is cleared on the next write).
/// </summary>
/// <param name="username">The username being authenticated.</param>
/// <param name="ip">The client IP address of the bind attempt.</param>
/// <returns><c>true</c> when the key is currently locked out; otherwise <c>false</c>.</returns>
/// <returns><c>true</c> when the pair or its IP is currently locked out; otherwise <c>false</c>.</returns>
public bool IsLockedOut(string username, string ip)
{
var opts = _options.Value;
if (opts.MaxLoginFailuresPerWindow <= 0) return false; // throttling disabled
var now = _clock.GetUtcNow();
if (_ipEntries.TryGetValue(IpKey(ip), out var ipEntry)
&& ipEntry.LockedUntil is { } ipUntil && ipUntil > now)
{
return true;
}
if (!_entries.TryGetValue(Key(username, ip), out var entry)) return false;
return entry.LockedUntil is { } until && until > _clock.GetUtcNow();
return entry.LockedUntil is { } until && until > now;
}
/// <summary>
/// Records a failed bind for the given <paramref name="username"/> + <paramref name="ip"/>.
/// Opens or advances the fixed window and, on reaching the configured threshold, arms the
/// lockout. No-op when throttling is disabled.
/// Opens or advances the fixed window for BOTH the per-pair and the per-IP counter and, on
/// either reaching its configured threshold, arms that counter's lockout. No-op when
/// throttling is disabled.
/// </summary>
/// <param name="username">The username that failed authentication.</param>
/// <param name="ip">The client IP address of the failed bind attempt.</param>
@@ -107,12 +149,53 @@ public sealed class LoginThrottle
var now = _clock.GetUtcNow();
var window = TimeSpan.FromMinutes(opts.LoginFailureWindowMinutes);
var lockout = TimeSpan.FromMinutes(opts.LoginLockoutMinutes);
var key = Key(username, ip);
_entries.AddOrUpdate(
RecordFailureIn(_entries, Key(username, ip), opts.MaxLoginFailuresPerWindow, window, lockout, now);
RecordFailureIn(_ipEntries, IpKey(ip), opts.PerIpMaxFailures, window, lockout, now);
// Amortized prune (R2 N8): every Nth failure, or immediately when either map is over
// the hard cap — the cap is the memory-safety invariant and must never wait for the
// cadence.
if (Interlocked.Increment(ref _failureWrites) % PruneEveryNFailures == 0
|| _entries.Count > MaxTrackedKeys || _ipEntries.Count > MaxTrackedKeys)
{
Prune(now);
}
}
/// <summary>
/// Records a successful bind for the given <paramref name="username"/> +
/// <paramref name="ip"/>, clearing any accumulated failure count and lockout for that PAIR
/// only. Deliberately does <b>not</b> touch the IP's secondary counter — see the class
/// remarks on success-path asymmetry: resetting the per-IP counter on any successful bind
/// would let a spray that stumbles onto (or already knows) one valid account on the
/// sprayed IP keep spraying indefinitely. The per-IP counter's only relief is window/lockout
/// expiry.
/// </summary>
/// <param name="username">The username that authenticated successfully.</param>
/// <param name="ip">The client IP address of the successful bind attempt.</param>
public void RecordSuccess(string username, string ip)
{
_entries.TryRemove(Key(username, ip), out _);
}
/// <summary>
/// Records one failure against <paramref name="key"/> in <paramref name="map"/>, opening or
/// advancing its fixed window and arming its lockout once <paramref name="maxFailures"/> is
/// reached. Shared by the per-pair and per-IP tiers — only the map, key, and ceiling differ.
/// </summary>
private static void RecordFailureIn(
ConcurrentDictionary<string, Entry> map,
string key,
int maxFailures,
TimeSpan window,
TimeSpan lockout,
DateTimeOffset now)
{
map.AddOrUpdate(
key,
// First observed failure for this key: open a fresh window.
_ => Arm(new Entry(1, now, null), opts, now, lockout),
_ => Arm(new Entry(1, now, null), maxFailures, now, lockout),
(_, existing) =>
{
// A live lockout stands until it expires — further failures don't extend it.
@@ -124,68 +207,59 @@ public sealed class LoginThrottle
// If the current window has lapsed (or a prior lockout has expired), restart it.
if (existing.LockedUntil is not null || now - existing.WindowStart >= window)
{
return Arm(new Entry(1, now, null), opts, now, lockout);
return Arm(new Entry(1, now, null), maxFailures, now, lockout);
}
return Arm(existing with { Count = existing.Count + 1 }, opts, now, lockout);
return Arm(existing with { Count = existing.Count + 1 }, maxFailures, now, lockout);
});
// Amortized prune (R2 N8): every Nth failure, or immediately when over the hard
// cap — the cap is the memory-safety invariant and must never wait for the cadence.
if (Interlocked.Increment(ref _failureWrites) % PruneEveryNFailures == 0
|| _entries.Count > MaxTrackedKeys)
{
Prune(now);
}
}
/// <summary>
/// Records a successful bind for the given <paramref name="username"/> +
/// <paramref name="ip"/>, clearing any accumulated failure count and lockout for that key.
/// </summary>
/// <param name="username">The username that authenticated successfully.</param>
/// <param name="ip">The client IP address of the successful bind attempt.</param>
public void RecordSuccess(string username, string ip)
{
_entries.TryRemove(Key(username, ip), out _);
}
/// <summary>Arms the lockout on <paramref name="entry"/> if it has reached the threshold.</summary>
private static Entry Arm(Entry entry, SecurityOptions opts, DateTimeOffset now, TimeSpan lockout) =>
entry.Count >= opts.MaxLoginFailuresPerWindow
/// <summary>Arms the lockout on <paramref name="entry"/> if it has reached <paramref name="maxFailures"/>.</summary>
private static Entry Arm(Entry entry, int maxFailures, DateTimeOffset now, TimeSpan lockout) =>
entry.Count >= maxFailures
? entry with { LockedUntil = now + lockout }
: entry;
/// <summary>
/// Opportunistic memory management on write: drop entries whose window has fully expired
/// (no active lockout), and if the map still exceeds <see cref="MaxTrackedKeys"/> evict the
/// oldest-windowed entries so a spray across many keys cannot grow the map unbounded.
/// (no active lockout) in BOTH maps, and if either map still exceeds
/// <see cref="MaxTrackedKeys"/> evict its oldest-windowed entries so a spray across many
/// keys cannot grow either map unbounded.
/// </summary>
private void Prune(DateTimeOffset now)
{
var opts = _options.Value;
var window = TimeSpan.FromMinutes(opts.LoginFailureWindowMinutes);
foreach (var kvp in _entries)
PruneMap(_entries, window, now);
PruneMap(_ipEntries, window, now);
}
private static void PruneMap(ConcurrentDictionary<string, Entry> map, TimeSpan window, DateTimeOffset now)
{
foreach (var kvp in map)
{
var e = kvp.Value;
var lockActive = e.LockedUntil is { } until && until > now;
if (!lockActive && now - e.WindowStart >= window)
{
_entries.TryRemove(kvp);
map.TryRemove(kvp);
}
}
if (_entries.Count <= MaxTrackedKeys) return;
if (map.Count <= MaxTrackedKeys) return;
// Still over the cap after pruning expired entries — evict the oldest-windowed keys.
var excess = _entries.Count - MaxTrackedKeys;
foreach (var kvp in _entries.OrderBy(k => k.Value.WindowStart).Take(excess))
var excess = map.Count - MaxTrackedKeys;
foreach (var kvp in map.OrderBy(k => k.Value.WindowStart).Take(excess))
{
_entries.TryRemove(kvp);
map.TryRemove(kvp);
}
}
private static string Key(string username, string ip) =>
$"{(username ?? string.Empty).ToLowerInvariant()}|{ip ?? string.Empty}";
/// <summary>Key for the secondary, IP-only tier — deliberately no username component.</summary>
private static string IpKey(string ip) => ip ?? string.Empty;
}
src/ZB.MOM.WW.ScadaBridge.Security/SecurityOptions.cs
+26
View File
@@ -124,6 +124,18 @@ public class SecurityOptions
/// <see cref="MaxLoginFailuresPerWindow"/> failures within the window. Default: <b>5</b>.
/// </summary>
public int LoginLockoutMinutes { get; set; } = 5;
/// <summary>
/// Secondary, IP-only failure ceiling layered on top of the per-<c>{username}|{ip}</c>
/// budget (<see cref="MaxLoginFailuresPerWindow"/>) — closes the password-spray gap where
/// one attacker IP tries one password against many distinct usernames, never accumulating
/// <see cref="MaxLoginFailuresPerWindow"/> failures on any single pair. Reuses the same
/// <see cref="LoginFailureWindowMinutes"/> window and <see cref="LoginLockoutMinutes"/>
/// lockout duration, just counted per IP instead of per pair. Set higher than the per-pair
/// ceiling (validated by <see cref="SecurityOptionsValidator"/>) so a handful of ordinary
/// typos behind a shared NAT gateway don't trip it. Default: <b>25</b>.
/// </summary>
public int PerIpMaxFailures { get; set; } = 25;
}
/// <summary>
@@ -186,6 +198,20 @@ public sealed class SecurityOptionsValidator : Microsoft.Extensions.Options.IVal
$"({options.LoginLockoutMinutes}) must be non-negative.");
}
// SECURITY: the per-IP spray ceiling must sit strictly above the per-pair ceiling.
// If it didn't, a single unlucky (or malicious) username/IP pair could trip the
// IP-wide lockout before its own per-pair budget is exhausted, locking out every
// OTHER username sharing that IP (e.g. a NAT gateway) on account of one pair's
// failures — defeating the "don't punish shared IPs for a couple of typos" intent.
if (options.PerIpMaxFailures <= options.MaxLoginFailuresPerWindow)
{
return Microsoft.Extensions.Options.ValidateOptionsResult.Fail(
$"{nameof(SecurityOptions.PerIpMaxFailures)} " +
$"({options.PerIpMaxFailures}) must be strictly greater than " +
$"{nameof(SecurityOptions.MaxLoginFailuresPerWindow)} " +
$"({options.MaxLoginFailuresPerWindow}).");
}
return Microsoft.Extensions.Options.ValidateOptionsResult.Success;
}
}
src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogPurgeService.cs
+29 -27
View File
@@ -8,17 +8,16 @@ using ZB.MOM.WW.ScadaBridge.Commons.Observability;
namespace ZB.MOM.WW.ScadaBridge.SiteEventLogging;
/// <summary>
/// Predicate the <see cref="EventLogPurgeService"/>
/// consults at the top of every purge tick to decide whether THIS node should
/// run the daily purge. The design states "a daily background job runs on the
/// active node and deletes all events older than 30 days"; the standby's local
/// SQLite receives no writes, so purging there is harmless but unnecessary —
/// and silently doing it anyway diverges from the design.
/// Predicate the <see cref="EventLogPurgeService"/> consults at the top of every purge
/// tick. As of arch-review 2026-08-19 P4-4 it no longer GATES the purge — DataConnectionActor
/// writes <c>site_events</c> on both site nodes (register rows 89/90/97), so a standby that
/// skipped its own purge would grow its file unboundedly on an unreplicated site. Every node
/// now purges its own file every tick; this predicate is consulted only to annotate the tick's
/// log line with whether this node is the active one.
///
/// Registration is the Host's responsibility (it knows the cluster topology);
/// when no implementation is registered the purge service defaults to "always
/// active" so non-clustered hosts and unit tests are unaffected — backward
/// compatible with the prior "run on every host" behaviour.
/// when no implementation is registered the purge service treats every tick as active for
/// logging purposes — backward compatible with hosts that never register this delegate.
/// </summary>
/// <returns><c>true</c> if this node is the active site member and should run the purge; <c>false</c> to skip.</returns>
public delegate bool SiteEventLogActiveNodeCheck();
@@ -116,11 +115,11 @@ public class EventLogPurgeService : BackgroundService
/// <param name="options">Site event log options (retention days, storage cap, purge interval).</param>
/// <param name="logger">Logger instance.</param>
/// <param name="isActiveNode">
/// Optional active-node check. When <c>null</c>, the
/// service runs the purge on every tick (preserves the pre-fix behaviour
/// for non-clustered hosts and existing tests). When supplied — e.g. by
/// the Host on a site node — each tick early-exits on the standby so the
/// daily purge runs only on the active node, matching the design.
/// Optional active-node check, consulted only to annotate the purge tick's log line
/// (arch-review 2026-08-19 P4-4 removed the early-exit this used to drive — every node
/// purges its own <c>site_events</c> file every tick, since DataConnectionActor writes
/// on both site nodes and a standby that skipped its own purge would grow unbounded on
/// an unreplicated site). When <c>null</c> every tick logs as active.
/// </param>
/// <param name="isReplicationConfigured">
/// Optional LocalDb-replication check. When <c>null</c> — non-clustered hosts,
@@ -193,13 +192,20 @@ public class EventLogPurgeService : BackgroundService
{
try
{
// Gate every tick on the active-node check.
// The standby's local SQLite receives no writes, so purging there
// is harmless but unnecessary; the design (Component-SiteEventLogging
// → Storage) explicitly states the purge runs on the active node.
// Defensive try/catch: a transient cluster-state read failure must
// not stop the purge loop — fall back to running the purge (the
// pre-fix behaviour was "always run", which is harmless on standby).
// arch-review 2026-08-19 P4-4: purge no longer gates on the active-node check.
// DataConnectionActor writes site_events on BOTH site nodes (register rows
// 89/90/97), so the old "standby receives no writes" premise this gate was
// built on is stale — a standby that never purges its own file grows
// site_events forever (no retention, no cap) on an UNREPLICATED site (the rig's
// site-b/site-c posture). On a REPLICATED pair this was masked because the
// retention/cap DELETEs are CDC-captured and converge under LWW regardless of
// which node issues them (same reasoning as the existing purge-on-replicated
// design — see the replication-backlog note below and CLAUDE.md's LocalDb
// bullet); it is REQUIRED on an unreplicated pair because each node's file is
// independent. Both nodes now run retention + cap trim against their own file
// every tick. isActive is retained only to annotate the log line below (and
// defensively defaults to true if the check itself throws) — it no longer
// skips any work.
bool isActive;
try
{
@@ -208,15 +214,11 @@ public class EventLogPurgeService : BackgroundService
catch (Exception checkEx)
{
_logger.LogDebug(checkEx,
"Active-node check threw during purge tick; running purge to be safe");
"Active-node check threw during purge tick; treating as active for logging purposes");
isActive = true;
}
if (!isActive)
{
_logger.LogDebug("Skipping event log purge tick — this node is not the active site member");
return false;
}
_logger.LogDebug("Running event log purge tick (active={IsActive})", isActive);
var budget = new PurgeBudget(_options.MaxPurgedRowsPerTick);
PurgeByRetention(budget);
src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/ServiceCollectionExtensions.cs
+8 -7
View File
@@ -23,13 +23,14 @@ public static class ServiceCollectionExtensions
services.AddSingleton<ISiteEventLogger>(sp => sp.GetRequiredService<SiteEventLogger>());
services.AddSingleton<IEventLogQueryService, EventLogQueryService>();
// The purge service still registers on every host
// node, but it consults an optional SiteEventLogActiveNodeCheck on each
// tick and early-exits on the standby. The Host registers the real
// active-node check on site nodes; tests and non-clustered hosts leave
// it unregistered, and the purge defaults to "always run" (the
// pre-fix behaviour). Building the service via a factory so the
// optional delegate flows from DI rather than the constructor default.
// The purge service registers on every host node and purges its own
// site_events file on every tick, on every node (arch-review 2026-08-19 P4-4 —
// the standby writes site_events too, so it can no longer skip its own purge).
// It still consults an optional SiteEventLogActiveNodeCheck, but only to annotate
// the tick's log line, not to gate it. The Host registers the real active-node
// check on site nodes; tests and non-clustered hosts leave it unregistered, and
// the log line just reports every tick as active. Building the service via a
// factory so the optional delegate flows from DI rather than the constructor default.
//
// SiteEventLogReplicationCheck flows the same way and is likewise Host-supplied:
// it only selects the wording of the purge log line (whether to add the
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs
+5
View File
@@ -825,6 +825,11 @@ public class AlarmActor : ReceiveActor
// Context.Parent.Tell from the execution actor.
Self,
_logger,
// arch-review 2026-08-19 P1-3: same health collector + service provider as
// the instance-script path, so the alarm on-trigger run gets the same
// stuck-script watchdog, script-error counting, and run-summary threading.
_healthCollector,
_serviceProvider,
runId,
// Per-script timeout from the on-trigger script (null = global).
_onTriggerExecutionTimeoutSeconds,
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs
+67 -13
View File
@@ -1197,9 +1197,27 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
{
var instanceName = command.InstanceUniqueName;
// Static overrides and mirrored native alarm state are reset on redeployment per
// design decision — the new config may bind different sources, and the source snapshot
// re-seeds the mirror on (re)subscribe. Dispatched HERE, BEFORE the successor exists,
// and handed to it as a gate (arch-review 2026-08-19 P1-2). It used to run inside the
// persistence task below, i.e. after the actor was already created AND queued behind a
// full config-JSON write, so the successor's PreStart override load routinely won the
// race and applied the PREDECESSOR's overrides — including keys absent from the new
// config — to in-memory state for the actor's whole lifetime, while the disk ended up
// cleared. That combination is worse than either half alone: the running instance
// silently disagreed with its own store, and a later restart flipped the values.
var staticStateResetGate = Task.Run(async () =>
{
await _storage.ClearStaticOverridesAsync(instanceName);
await _storage.ClearNativeAlarmsForInstanceAsync(instanceName);
});
// Create the Instance Actor immediately. CreateInstanceActor first ensures the DCL
// connections this config's data-sourced attributes subscribe through (register row 85).
CreateInstanceActor(instanceName, command.FlattenedConfigurationJson, command.RevisionHash);
CreateInstanceActor(
instanceName, command.FlattenedConfigurationJson, command.RevisionHash,
staticStateResetGate);
if (!isRedeploy)
_deployedInstanceNames.Add(instanceName);
UpdateInstanceCounts();
@@ -1228,11 +1246,9 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
command.RevisionHash,
isEnabled: true);
// Static overrides and mirrored native alarm state are reset on
// redeployment per design decision — the new config may bind different
// sources, and the source snapshot re-seeds the mirror on (re)subscribe.
await _storage.ClearStaticOverridesAsync(instanceName);
await _storage.ClearNativeAlarmsForInstanceAsync(instanceName);
// Join the reset dispatched ahead of actor creation, so a failed clear still
// fails the deployment exactly as it did when the clear ran inline here.
await staticStateResetGate;
// No explicit replication step: deployed_configurations is a replicated table,
// so the write above is captured and shipped to the peer like any other row.
@@ -1566,13 +1582,26 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
// Re-read the durable row off the actor thread. The stored row — NOT the requesting
// message — is the authority on the config body AND on enabled/disabled state, so a
// disable that landed after the reconcile fetch still wins and the instance stays
// stopped.
// disable that landed before this read still wins and the instance stays stopped.
//
// Row 101: a disable/delete that lands WHILE the read is in flight is a different
// problem, and the row itself cannot catch it — the read already saw the enabled row,
// and neither Disable nor Delete leaves any of the three markers
// ShouldAttemptEnsureStart re-checks. So capture the generation on the actor thread
// BEFORE the read starts and let the landing handler discard a superseded snapshot,
// exactly as the boot load and the sweep do. Without it the heal could start an
// instance the operator had just disabled or deleted, PERMANENTLY: the sweep only ever
// walks store rows, so it has no runtime→store stop leg to undo it.
var generationAtRead = _storeWriteGeneration;
Task.Run(async () =>
{
var configs = await _storage.GetAllDeployedConfigsAsync();
// Read through the same seam the boot load and the sweep read through
// (_configLoader defaults to _storage.GetAllDeployedConfigsAsync, so this is the
// identical production call), which lets a test hold the read open and land the
// result behind a committed disable/delete.
var configs = await _configLoader();
var config = configs.FirstOrDefault(c => c.InstanceUniqueName == instanceName);
return new EnsureInstanceStartedResult(command, config, null);
return new EnsureInstanceStartedResult(command, config, null, generationAtRead);
}).ContinueWith(t => t.IsCompletedSuccessfully
? t.Result
: new EnsureInstanceStartedResult(command, null, t.Exception?.GetBaseException().Message))
@@ -1620,6 +1649,14 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
return;
}
// Row 101, ensure-started half: this node disabled, deleted or redeployed an instance
// while the read was in flight, so the snapshot predates state the actor has already
// advanced past. Discard it — starting from it would resurrect an instance the operator
// just stopped, and nothing would ever stop it again. Nothing is lost by dropping it:
// the heal is advisory, and the instance reconcile sweep re-reads the store every tick.
if (IsStoreSnapshotStale(result.StoreGenerationAtRead, "ensure-started"))
return;
// No stored row: nothing to start. Normal when the config was superseded/removed
// between the reconcile fetch and this read, and on an UNREPLICATED site whose peer
// node healed its own file — this singleton's storage legitimately has no such row.
@@ -3216,7 +3253,15 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// the single creation funnel, so recording it here covers every path that can ever start an
/// Instance Actor.
/// </param>
internal void CreateInstanceActor(string instanceName, string configJson, string revisionHash)
/// <param name="staticStateResetGate">
/// The deploy path's static-override / native-alarm reset, dispatched BEFORE this call so the
/// new actor's PreStart override load can be ordered after it (arch-review 2026-08-19 P1-2).
/// <see langword="null"/> on every non-deploy start (boot batch, enable, self-heal), where
/// nothing is being reset and the load runs immediately as before.
/// </param>
internal void CreateInstanceActor(
string instanceName, string configJson, string revisionHash,
Task? staticStateResetGate = null)
{
// Deferred register row 85 — an Instance Actor is useless without the DCL connection
// actors its data-sourced attributes subscribe through, so ensuring them belongs in the
@@ -3262,7 +3307,9 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
_loggerFactory.CreateLogger<InstanceActor>(),
_dclManager,
_healthCollector,
_serviceProvider));
_serviceProvider,
null,
staticStateResetGate));
var actorRef = Context.ActorOf(props, instanceName);
_instanceActors[instanceName] = actorRef;
@@ -3461,8 +3508,15 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// the actor thread. No original sender: the ensure-started path is fire-and-forget, so
/// there is nobody to answer.
/// </summary>
/// <param name="StoreGenerationAtRead">
/// Row 101: <see cref="_storeWriteGeneration"/> captured on the actor thread when the read
/// was dispatched. A mismatch on landing means a disable/delete/deploy committed while the
/// read was in flight, and the heal is dropped whole (see
/// <see cref="HandleEnsureInstanceStartedResult"/>).
/// </param>
internal record EnsureInstanceStartedResult(
EnsureInstanceStartedCommand Command, DeployedInstance? Config, string? Error);
EnsureInstanceStartedCommand Command, DeployedInstance? Config, string? Error,
long StoreGenerationAtRead = 0);
/// <summary>
/// Periodic instance reconcile-sweep tick (deferred register rows 78 + 72). Interval comes
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs
+86 -12
View File
@@ -111,6 +111,25 @@ public class InstanceActor : ReceiveActor
private ICancelable? _staticOverrideFlushTimer;
private readonly TimeSpan _staticOverrideFlushInterval;
/// <summary>
/// The deploy path's static-override / native-alarm reset, supplied by
/// <see cref="DeploymentManagerActor.CreateInstanceActor"/> on a (re)deploy and
/// <see langword="null"/> on every other start (arch-review 2026-08-19 P1-2). When set,
/// <see cref="PreStart"/> starts the override load only once it has completed, so a
/// redeploy can never load the PREDECESSOR's overrides into the successor's memory.
/// </summary>
private readonly Task? _staticStateResetGate;
/// <summary>
/// How long <see cref="PostStop"/> waits for the final coalesced static-override flush to
/// commit. The wait exists so a redeploy's clear is ordered AFTER this actor's last write
/// (arch-review 2026-08-19 P1-2 leg 2) — the Deployment Manager gates the clear on the
/// Terminated signal, which a fire-and-forget flush could outlive and durably resurrect
/// overrides the redeploy had just cleared. Bounded so a wedged SQLite write cannot stall
/// the redeploy indefinitely; exceeding it is logged and the stop proceeds.
/// </summary>
private static readonly TimeSpan FinalStaticOverrideFlushTimeout = TimeSpan.FromSeconds(5);
/// <summary>
/// Cached string forms of <see cref="QualityCode"/> (perf remediation, arch-review
/// WP1.5) — the TagValueUpdate ingest path calls this once or twice per update, so a
@@ -182,6 +201,11 @@ public class InstanceActor : ReceiveActor
/// <param name="dclManager">Optional Data Connection Layer manager actor reference.</param>
/// <param name="healthCollector">Optional health collector for reporting metrics.</param>
/// <param name="serviceProvider">Optional DI service provider for script execution services.</param>
/// <param name="staticOverrideFlushInterval">Coalescing window for buffered static-override writes.</param>
/// <param name="staticStateResetGate">
/// Optional (re)deploy static-override / native-alarm reset to order the PreStart override
/// load behind — see <see cref="_staticStateResetGate"/>.
/// </param>
public InstanceActor(
string instanceUniqueName,
string configJson,
@@ -194,7 +218,8 @@ public class InstanceActor : ReceiveActor
IActorRef? dclManager = null,
ISiteHealthCollector? healthCollector = null,
IServiceProvider? serviceProvider = null,
TimeSpan? staticOverrideFlushInterval = null)
TimeSpan? staticOverrideFlushInterval = null,
Task? staticStateResetGate = null)
{
_instanceUniqueName = instanceUniqueName;
_storage = storage;
@@ -208,6 +233,7 @@ public class InstanceActor : ReceiveActor
_serviceProvider = serviceProvider;
// Defaults to 100 ms, mirroring NativeAlarmActor's persistFlushInterval (P4).
_staticOverrideFlushInterval = staticOverrideFlushInterval ?? TimeSpan.FromMilliseconds(100);
_staticStateResetGate = staticStateResetGate;
// Deserialize the flattened configuration
_configuration = JsonSerializer.Deserialize<FlattenedConfiguration>(configJson);
@@ -326,9 +352,20 @@ public class InstanceActor : ReceiveActor
// the actor), and on failover/restart; this single point covers them all.
LogLifecycleEvent($"Instance {_instanceUniqueName} started");
// Asynchronously load static overrides from SQLite and pipe to self
// Asynchronously load static overrides from SQLite and pipe to self. On a (re)deploy
// the read is chained behind the reset the Deployment Manager dispatched before this
// actor existed, so it can only ever observe the CLEARED table (arch-review
// 2026-08-19 P1-2). A faulted reset does not block the load — the deploy itself fails
// on that fault through the persistence join, which stops this actor anyway.
var self = Self;
_storage.GetStaticOverridesAsync(_instanceUniqueName).ContinueWith(t =>
var instanceName = _instanceUniqueName;
var storage = _storage;
var load = _staticStateResetGate == null
? storage.GetStaticOverridesAsync(instanceName)
: _staticStateResetGate
.ContinueWith(_ => storage.GetStaticOverridesAsync(instanceName))
.Unwrap();
load.ContinueWith(t =>
{
if (t.IsCompletedSuccessfully)
return new LoadOverridesResult(t.Result, null);
@@ -356,12 +393,15 @@ public class InstanceActor : ReceiveActor
timer.Cancel();
_tagSubscribeRetryTimers.Clear();
// Best-effort final flush of any coalesced static-override writes so a graceful
// stop does not lose the last <= flush-interval of writes (P4 pattern, mirrors
// NativeAlarmActor.PostStop). Fire-and-forget — the in-memory attribute value
// (already applied) remains authoritative even if this final persist is lost.
// Final flush of any coalesced static-override writes so a graceful stop does not lose
// the last <= flush-interval of writes (P4 pattern, mirrors NativeAlarmActor.PostStop).
// AWAITED, not fire-and-forget (arch-review 2026-08-19 P1-2 leg 2): a redeploy stops
// this actor, waits for Terminated, and only then clears the static overrides — so a
// detached final flush could commit up to one flush interval of the PREDECESSOR's dirty
// writes AFTER that clear and durably resurrect them. Blocking here is what puts the
// write provably before the Terminated signal the clear is gated on.
_staticOverrideFlushTimer?.Cancel();
FlushDirtyStaticOverrides();
WaitForFinalStaticOverrideFlush();
// Operational `instance_lifecycle` event — instance stopped. An
// instance stops on disable, delete, redeployment, and graceful shutdown;
@@ -571,11 +611,16 @@ public class InstanceActor : ReceiveActor
/// failed write never blocks the actor; the in-memory attribute value (already
/// applied) remains authoritative.
/// </summary>
private void FlushDirtyStaticOverrides()
/// <returns>
/// The persist task, so the stop path can order the FINAL flush before termination
/// (<see cref="WaitForFinalStaticOverrideFlush"/>). Every other caller ignores it, keeping
/// the fire-and-forget semantics unchanged. Completed when there was nothing dirty.
/// </returns>
private Task FlushDirtyStaticOverrides()
{
_staticOverrideFlushTimer = null;
if (_dirtyStaticOverrides.Count == 0)
return;
return Task.CompletedTask;
var batch = _dirtyStaticOverrides
.Select(kv => (AttributeName: kv.Key, Value: kv.Value))
@@ -584,12 +629,41 @@ public class InstanceActor : ReceiveActor
var instanceName = _instanceUniqueName;
var logger = _logger;
_storage.SetStaticOverridesAsync(instanceName, batch)
.ContinueWith(t => logger.LogWarning(
var persist = _storage.SetStaticOverridesAsync(instanceName, batch);
persist.ContinueWith(t => logger.LogWarning(
t.Exception?.GetBaseException(),
"Failed to persist {Count} static overrides for {Instance}; in-memory state is authoritative",
batch.Count, instanceName),
TaskContinuationOptions.OnlyOnFaulted);
return persist;
}
/// <summary>
/// Runs the stop-path static-override flush and waits for it to commit, bounded by
/// <see cref="FinalStaticOverrideFlushTimeout"/>. A fault is swallowed — the
/// OnlyOnFaulted continuation inside <see cref="FlushDirtyStaticOverrides"/> has already
/// logged it, and a persist failure must never turn a graceful stop into a crash.
/// </summary>
private void WaitForFinalStaticOverrideFlush()
{
var flush = FlushDirtyStaticOverrides();
if (flush.IsCompleted)
return;
try
{
if (!flush.Wait(FinalStaticOverrideFlushTimeout))
{
_logger.LogWarning(
"Final static-override flush for {Instance} did not complete within {Timeout}; "
+ "stopping anyway — a redeploy's override reset may race it",
_instanceUniqueName, FinalStaticOverrideFlushTimeout);
}
}
catch (Exception)
{
// Already logged by the flush's OnlyOnFaulted continuation.
}
}
/// <summary>
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptRunLauncher.cs
+57 -1
View File
@@ -138,6 +138,18 @@ internal static class ScriptRunLauncher
/// <param name="options">Site runtime configuration options, including the execution timeout.</param>
/// <param name="completionTarget">The owning <see cref="AlarmActor"/>, which receives the completion message.</param>
/// <param name="logger">Logger for execution diagnostics.</param>
/// <param name="healthCollector">
/// Optional health collector for recording execution metrics (arch-review 2026-08-19
/// P1-3): the alarm on-trigger run shares the script-error counter with
/// <see cref="RunScriptAsync"/> so a wedged/failed alarm body is visible the same way a
/// wedged/failed instance script is.
/// </param>
/// <param name="serviceProvider">
/// Optional DI service provider (arch-review 2026-08-19 P1-3): resolves the singleton
/// <see cref="ISiteEventLogger"/> (stuck-watchdog reporting) and
/// <see cref="ScriptRunSummaryRecorder"/> (aggregate run counters), exactly like the
/// instance-script path.
/// </param>
/// <param name="runId">Per-alarm run counter, used only in log messages.</param>
/// <param name="executionTimeoutSeconds">The on-trigger script's per-script timeout in seconds. Null or non-positive falls back to the global value.</param>
/// <param name="parentExecutionId">
@@ -158,6 +170,8 @@ internal static class ScriptRunLauncher
SiteRuntimeOptions options,
IActorRef completionTarget,
ILogger logger,
ISiteHealthCollector? healthCollector,
IServiceProvider? serviceProvider,
long runId,
int? executionTimeoutSeconds,
Guid? parentExecutionId,
@@ -174,7 +188,8 @@ internal static class ScriptRunLauncher
() => RunAlarmScriptAsync(
alarmName, instanceName, level, priority, message, compiledScript,
instanceActor, sharedScriptLibrary, options, completionTarget, logger,
runId, parentExecutionId, timeout, cts),
healthCollector, serviceProvider, runId, parentExecutionId, timeout,
executionScheduler, cts),
CancellationToken.None, TaskCreationOptions.DenyChildAttach, executionScheduler)
.Unwrap();
}
@@ -463,11 +478,42 @@ internal static class ScriptRunLauncher
SiteRuntimeOptions options,
IActorRef completionTarget,
ILogger logger,
ISiteHealthCollector? healthCollector,
IServiceProvider? serviceProvider,
long runId,
Guid? parentExecutionId,
TimeSpan timeout,
ScriptExecutionScheduler executionScheduler,
CancellationTokenSource cts)
{
// arch-review 2026-08-19 P1-3: singletons resolved from the root provider up front,
// exactly like RunScriptAsync, so they are available to the catch blocks regardless
// of scope state. Null in tests that pass no serviceProvider — degrades the same way
// the instance-script path does (no site event log / no aggregate counters).
var siteEventLogger = serviceProvider?.GetService<ISiteEventLogger>();
var summaryRecorder = serviceProvider?.GetService<ScriptRunSummaryRecorder>();
// The on-trigger run has no per-script name of its own in the site-event/summary
// sense — key it under the alarm, mirroring the "AlarmActor:{name}" source
// convention already used for this alarm's other site events.
var scriptName = $"AlarmActor:{alarmName}";
var runStopwatch = Stopwatch.StartNew();
// arch-review 2026-08-19 P1-3: identify the worker this body is holding, so the
// stuck-script watchdog can detach and replace it if the body wedges — same
// mechanism, same gauge, as the instance-script path. Read in the FIRST synchronous
// segment, before any await.
var workerSlot = ScriptExecutionScheduler.CurrentWorkerSlot;
var workerRunStamp = workerSlot is { } slot ? executionScheduler.CurrentRunStamp(slot) : 0L;
var completed = 0;
ArmStuckScriptWatchdog(
cts, options, logger, siteEventLogger, executionScheduler,
workerSlot, workerRunStamp, timeout,
() => Volatile.Read(ref completed) != 0,
$"Alarm on-trigger script for '{alarmName}' on instance '{instanceName}'",
instanceName, scriptName);
try
{
// Enqueue-anchored deadline: an already-cancelled token means this on-trigger run
@@ -511,12 +557,17 @@ internal static class ScriptRunLauncher
}
};
summaryRecorder?.RecordStarted(instanceName, scriptName);
await compiledScript.RunAsync(globals, cts.Token);
summaryRecorder?.RecordCompleted(instanceName, scriptName, runStopwatch.ElapsedMilliseconds);
completionTarget.Tell(new AlarmActor.AlarmExecutionCompleted(alarmName, true));
}
catch (OperationCanceledException)
{
healthCollector?.IncrementScriptError();
summaryRecorder?.RecordTimedOut(instanceName, scriptName, runStopwatch.ElapsedMilliseconds);
logger.LogWarning(
"Alarm on-trigger script for {Alarm} on {Instance} timed out (run {RunId})",
alarmName, instanceName, runId);
@@ -524,6 +575,8 @@ internal static class ScriptRunLauncher
}
catch (Exception ex)
{
healthCollector?.IncrementScriptError();
summaryRecorder?.RecordFailed(instanceName, scriptName, runStopwatch.ElapsedMilliseconds);
// Failures logged, alarm continues
logger.LogError(ex,
"Alarm on-trigger script for {Alarm} on {Instance} failed (run {RunId})",
@@ -532,6 +585,9 @@ internal static class ScriptRunLauncher
}
finally
{
// Mark the body finished so the stuck-script watchdog (registered on
// cts.Token above) treats a timely cancellation as NOT stuck.
Interlocked.Exchange(ref completed, 1);
cts.Dispose();
}
}
src/ZB.MOM.WW.ScadaBridge.Transport/Import/ArtifactDiff.cs
+7
View File
@@ -170,6 +170,13 @@ public sealed class ArtifactDiff
// previewed as "Identical". Mirrors CompareDatabaseConnection below.
AddIfDifferent(changes, "MaxRetries", existing.MaxRetries, incoming.MaxRetries);
AddIfDifferent(changes, "RetryDelay", existing.RetryDelay.ToString(), incoming.RetryDelay.ToString());
// arch-review 2026-08-19 P7-1 (row-27 shape): TimeoutSeconds is operative at
// runtime and the importer has always APPLIED it, but the diff omitted it — same
// "silently changed and invisible to preview" defect as the MaxRetries fix above.
// A null incoming value (bundle written before this field existed) compares as
// "no change" — it mirrors BundleImporter's Overwrite fallback, which leaves the
// target's existing timeout untouched in that case.
AddIfDifferent(changes, "TimeoutSeconds", existing.TimeoutSeconds, incoming.TimeoutSeconds ?? existing.TimeoutSeconds);
// Secrets: presence-only comparison (we never echo the value in the diff).
var existingHasSecret = !string.IsNullOrEmpty(existing.AuthConfiguration);
src/ZB.MOM.WW.ScadaBridge.Transport/Import/BundleImporter.cs
+6
View File
@@ -3090,6 +3090,9 @@ public sealed class BundleImporter : IBundleImporter
existing.AuthConfiguration = dto.Secrets?.Values.TryGetValue("AuthConfiguration", out var auth) == true ? auth : null;
existing.MaxRetries = dto.MaxRetries;
existing.RetryDelay = dto.RetryDelay;
// A bundle written before this field existed carries null — leave the
// target's existing timeout untouched rather than resetting it to 0/unset.
existing.TimeoutSeconds = dto.TimeoutSeconds ?? existing.TimeoutSeconds;
await _externalRepo.UpdateExternalSystemAsync(existing, ct).ConfigureAwait(false);
await _auditService.LogAsync(user, "Update", "ExternalSystem", existing.Id.ToString(), existing.Name,
new { existing.Name, existing.EndpointUrl }, ct).ConfigureAwait(false);
@@ -3123,6 +3126,9 @@ public sealed class BundleImporter : IBundleImporter
AuthConfiguration = dto.Secrets?.Values.TryGetValue("AuthConfiguration", out var auth) == true ? auth : null,
MaxRetries = dto.MaxRetries,
RetryDelay = dto.RetryDelay,
// Null (a bundle written before this field existed) falls back to the
// entity's own "0 = unset" default.
TimeoutSeconds = dto.TimeoutSeconds ?? 0,
};
return sys;
}
src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntityDtos.cs
+8 -1
View File
@@ -252,7 +252,14 @@ public sealed record ExternalSystemDto(
int MaxRetries,
TimeSpan RetryDelay,
IReadOnlyList<ExternalSystemMethodDto> Methods,
SecretsBlock? Secrets);
SecretsBlock? Secrets,
// Per-system HTTP round-trip timeout (arch-review 2026-08-19 P7-1, row-27 shape).
// Additive trailing field; null distinguishes a bundle written before this field
// existed from a bundle that explicitly carries the entity's own "0 = unset"
// sentinel. BundleImporter treats a null incoming value as "not specified by this
// bundle": Add falls back to the entity default (0), Overwrite leaves the target's
// existing value untouched.
int? TimeoutSeconds = null);
public sealed record ExternalSystemMethodDto(
string Name,
src/ZB.MOM.WW.ScadaBridge.Transport/Serialization/EntitySerializer.cs
+5 -1
View File
@@ -139,7 +139,8 @@ public sealed class EntitySerializer
MaxRetries: sys.MaxRetries,
RetryDelay: sys.RetryDelay,
Methods: methods,
Secrets: secrets);
Secrets: secrets,
TimeoutSeconds: sys.TimeoutSeconds);
}).ToList(),
DatabaseConnections: aggregate.DatabaseConnections.Select(dc => new DatabaseConnectionDto(
Name: dc.Name,
@@ -433,6 +434,9 @@ public sealed class EntitySerializer
AuthConfiguration = dto.Secrets?.Values.TryGetValue("AuthConfiguration", out var auth) == true ? auth : null,
MaxRetries = dto.MaxRetries,
RetryDelay = dto.RetryDelay,
// Null (a bundle written before this field existed) falls back to the
// entity's own "0 = unset" default.
TimeoutSeconds = dto.TimeoutSeconds ?? 0,
};
externalSystems.Add(sys);
foreach (var m in dto.Methods)
tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/AuditLogIngestActorCombinedTelemetryTests.cs
+167 -3
View File
@@ -89,15 +89,24 @@ public class AuditLogIngestActorCombinedTelemetryTests : TestKit, IClassFixture<
/// DataProtection wiring (we never write secret columns in these tests).
/// </summary>
private IServiceProvider BuildServiceProvider(
Func<ScadaBridgeDbContext, ISiteCallAuditRepository>? siteCallRepoFactory = null)
Func<ScadaBridgeDbContext, ISiteCallAuditRepository>? siteCallRepoFactory = null,
Func<ScadaBridgeDbContext, IAuditLogRepository>? auditRepoFactory = null)
{
var services = new ServiceCollection();
services.AddDbContext<ScadaBridgeDbContext>(opts =>
opts.UseSqlServer(_fixture.ConnectionString)
.ConfigureWarnings(w => w.Ignore(
Microsoft.EntityFrameworkCore.Diagnostics.RelationalEventId.PendingModelChangesWarning)));
services.AddScoped<IAuditLogRepository>(sp =>
new AuditLogRepository(sp.GetRequiredService<ScadaBridgeDbContext>()));
if (auditRepoFactory is null)
{
services.AddScoped<IAuditLogRepository>(sp =>
new AuditLogRepository(sp.GetRequiredService<ScadaBridgeDbContext>()));
}
else
{
services.AddScoped(sp =>
auditRepoFactory(sp.GetRequiredService<ScadaBridgeDbContext>()));
}
if (siteCallRepoFactory is null)
{
services.AddScoped<ISiteCallAuditRepository>(sp =>
@@ -338,6 +347,161 @@ public class AuditLogIngestActorCombinedTelemetryTests : TestKit, IClassFixture<
Assert.DoesNotContain(siteCallRows, r => r.TrackedOperationId == poisonTrackedId);
}
/// <summary>
/// arch-review 2026-08-19 P5-1: the per-entry fallback loop (used when the
/// batch fast path fails) must run under its own short
/// <see cref="AuditLogIngestActor.IngestFallbackBudget"/>, not an unbounded
/// wait — otherwise a single slow/wedged downstream call can freeze this
/// ReceiveAsync handler, and with it the central ingest singleton's mailbox,
/// for as long as the SQL retry ladder allows (minutes). Forces the batch
/// fast path to fail immediately (via <see cref="BatchFailingAuditRepo"/>) so
/// the per-entry fallback runs, then hangs the first entry's SiteCalls
/// upsert (via <see cref="HangingSiteCallRepo"/>) to prove the fallback's own
/// token — not <see cref="CancellationToken.None"/> — reaches the repository
/// call and bounds the wait.
/// </summary>
/// <remarks>
/// Pre-fix, the per-entry loop passed no token to
/// <c>BeginTransactionAsync</c>/<c>InsertIfNotExistsAsync</c>/<c>UpsertAsync</c>,
/// so <see cref="HangingSiteCallRepo"/>'s <c>Task.Delay(Timeout.InfiniteTimeSpan, ct)</c>
/// would never observe a cancellation and hang forever — this test's bounded
/// <c>ExpectMsg</c> below would time out and fail, which is the fail-unfixed
/// proof for this fix.
/// </remarks>
[SkippableFact]
public async Task Receive_CachedTelemetry_PerEntryFallbackHangs_BudgetBoundsWait_RemainderSkipped()
{
Skip.IfNot(_fixture.Available, _fixture.SkipReason);
var siteId = NewSiteId();
var entries = Enumerable.Range(0, 3).Select(_ =>
{
var (audit, siteCall) = NewEntry(siteId);
return new CachedTelemetryEntry(audit, siteCall);
}).ToList();
var hangingRepo = new HangingSiteCallRepo();
var sp = BuildServiceProvider(
siteCallRepoFactory: _ => hangingRepo,
auditRepoFactory: ctx => new BatchFailingAuditRepo(new AuditLogRepository(ctx)));
var actor = CreateActor(sp);
var stopwatch = System.Diagnostics.Stopwatch.StartNew();
actor.Tell(new IngestCachedTelemetryCommand(entries), TestActor);
// Budget is IngestFallbackBudget (5s). Bound the wait comfortably above
// that but far below "hangs indefinitely" — on unfixed code (no token
// passed to the fallback loop) this ExpectMsg times out and the test
// fails, which is the point: it proves the mailbox would otherwise be
// frozen for the life of the hang, not just for one short budget.
var reply = ExpectMsg<IngestCachedTelemetryReply>(TimeSpan.FromSeconds(20));
stopwatch.Stop();
Assert.True(
stopwatch.Elapsed < TimeSpan.FromSeconds(15),
$"handler took {stopwatch.Elapsed} — the per-entry fallback budget should bound the wait to ~{AuditLogIngestActor.IngestFallbackBudget.TotalSeconds}s, not the full hang.");
// Only the first entry pays the hang; once the budget is exhausted the
// remaining two are skipped without even being attempted. None of the
// three entries commit (the first was cancelled mid-transaction; the
// rest were never started).
Assert.Empty(reply.AcceptedEventIds);
Assert.Equal(1, hangingRepo.CallCount);
// The token reaching the repository call is the fallback's OWN fresh
// token — non-default and cancellable — never CancellationToken.None.
Assert.NotNull(hangingRepo.ReceivedToken);
Assert.NotEqual(CancellationToken.None, hangingRepo.ReceivedToken!.Value);
Assert.True(hangingRepo.ReceivedToken.Value.CanBeCanceled);
await using var read = CreateReadContext();
var siteCallCount = await read.Set<SiteCall>().CountAsync(s => s.SourceSite == siteId);
Assert.Equal(0, siteCallCount);
}
/// <summary>
/// Test double — forces <see cref="AuditLogIngestActor"/>'s cached-telemetry
/// batch fast path (<c>TryIngestCachedBatchAsync</c>) to fail immediately by
/// throwing from <see cref="InsertManyIfNotExistsAsync"/>, so tests can reach
/// the per-entry fallback loop without waiting out the real 20s
/// <see cref="AuditLogIngestActor.IngestBudget"/>. Every other member
/// delegates to a real <see cref="AuditLogRepository"/> so the fallback path
/// itself still exercises genuine SQL.
/// </summary>
private sealed class BatchFailingAuditRepo : IAuditLogRepository
{
private readonly IAuditLogRepository _inner;
public BatchFailingAuditRepo(IAuditLogRepository inner) { _inner = inner; }
public Task<int> InsertManyIfNotExistsAsync(
IReadOnlyList<AuditEvent> events, TimeSpan? commandTimeout = null, CancellationToken ct = default) =>
throw new InvalidOperationException("simulated batch ingest failure — forces the per-entry fallback");
public Task InsertIfNotExistsAsync(AuditEvent evt, CancellationToken ct = default) =>
_inner.InsertIfNotExistsAsync(evt, ct);
public Task<IReadOnlyList<AuditEvent>> QueryAsync(
AuditLogQueryFilter filter, AuditLogPaging paging, CancellationToken ct = default) =>
_inner.QueryAsync(filter, paging, ct);
public Task<long> SwitchOutPartitionAsync(
DateTime monthBoundary, TimeSpan? commandTimeout = null, CancellationToken ct = default) =>
_inner.SwitchOutPartitionAsync(monthBoundary, commandTimeout, ct);
public Task<long> PurgeChannelOlderThanAsync(
string channel, DateTime threshold, int batchSize, TimeSpan? commandTimeout = null, CancellationToken ct = default) =>
_inner.PurgeChannelOlderThanAsync(channel, threshold, batchSize, commandTimeout, ct);
public Task<IReadOnlyList<DateTime>> GetPartitionBoundariesOlderThanAsync(
DateTime threshold, CancellationToken ct = default) =>
_inner.GetPartitionBoundariesOlderThanAsync(threshold, ct);
public Task<ZB.MOM.WW.ScadaBridge.Commons.Types.AuditLogKpiSnapshot> GetKpiSnapshotAsync(
TimeSpan window, DateTime? nowUtc = null, CancellationToken ct = default) =>
_inner.GetKpiSnapshotAsync(window, nowUtc, ct);
public Task<IReadOnlyList<ExecutionTreeNode>> GetExecutionTreeAsync(
Guid executionId, CancellationToken ct = default) =>
_inner.GetExecutionTreeAsync(executionId, ct);
public Task<IReadOnlyList<string>> GetDistinctSourceNodesAsync(CancellationToken ct = default) =>
_inner.GetDistinctSourceNodesAsync(ct);
public Task<long> BackfillSourceNodeAsync(
string sentinel, DateTime before, int batchSize, CancellationToken ct = default) =>
_inner.BackfillSourceNodeAsync(sentinel, before, batchSize, ct);
}
/// <summary>
/// Test double — <see cref="UpsertAsync"/> hangs on
/// <c>Task.Delay(Timeout.InfiniteTimeSpan, ct)</c> until the caller's token
/// cancels, recording the call count and the token it was handed. Used to
/// prove the per-entry fallback loop's own budget bounds the wait (arch-review
/// 2026-08-19 P5-1) rather than blocking forever on a wedged downstream call.
/// </summary>
private sealed class HangingSiteCallRepo : ISiteCallAuditRepository
{
private int _callCount;
public int CallCount => _callCount;
public CancellationToken? ReceivedToken { get; private set; }
public async Task UpsertAsync(SiteCall siteCall, CancellationToken ct = default)
{
Interlocked.Increment(ref _callCount);
ReceivedToken = ct;
await Task.Delay(Timeout.InfiniteTimeSpan, ct).ConfigureAwait(false);
}
public Task<SiteCall?> GetAsync(TrackedOperationId id, CancellationToken ct = default) =>
throw new NotSupportedException();
public Task<IReadOnlyList<SiteCall>> QueryAsync(
SiteCallQueryFilter filter, SiteCallPaging paging, CancellationToken ct = default) =>
throw new NotSupportedException();
public Task<int> PurgeTerminalAsync(DateTime olderThanUtc, CancellationToken ct = default) =>
throw new NotSupportedException();
public Task<SiteCallKpiSnapshot> ComputeKpisAsync(
DateTime stuckCutoff, DateTime intervalSince, CancellationToken ct = default) =>
throw new NotSupportedException();
public Task<IReadOnlyList<SiteCallSiteKpiSnapshot>> ComputePerSiteKpisAsync(
DateTime stuckCutoff, DateTime intervalSince, CancellationToken ct = default) =>
throw new NotSupportedException();
public Task<IReadOnlyList<SiteCallNodeKpiSnapshot>> ComputePerNodeKpisAsync(
DateTime stuckCutoff, DateTime intervalSince, CancellationToken ct = default) =>
throw new NotSupportedException();
}
/// <summary>
/// Test double — throws unconditionally from <see cref="UpsertAsync"/> so
/// the dual-write transaction is forced to roll back. Lets the AuditLog
tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/SqliteAuditWriterCommitFailureTests.cs
+190
View File
@@ -0,0 +1,190 @@
using System.Reflection;
using Microsoft.Data.Sqlite;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using NSubstitute;
using ZB.MOM.WW.Audit;
using ZB.MOM.WW.ScadaBridge.AuditLog.Site;
using ZB.MOM.WW.ScadaBridge.AuditLog.Tests.TestSupport;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using IAuditWriter = ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services.IAuditWriter;
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Tests.Site;
/// <summary>
/// arch-review 2026-08-19 P4-2 regression coverage. <see cref="SqliteAuditWriter.FlushBatch"/>
/// used to call <c>pending.Completion.TrySetResult()</c> per row BEFORE
/// <c>transaction.Commit()</c> ran. A later failure in the SAME batch — another row's insert
/// throwing, or the commit itself failing (SQLITE_FULL, IOERR) — rolled the whole transaction
/// back, but the earlier row's <see cref="System.Threading.Tasks.Task"/> had already completed
/// successfully: the caller, and <see cref="FallbackAuditWriter"/> (whose ring-buffer fallback
/// exists precisely for this outage), both saw false success while the row was actually gone
/// from the database — <c>TrySetException</c> in the outer catch is a no-op against an
/// already-completed <see cref="System.Threading.Tasks.TaskCompletionSource"/>.
/// </summary>
/// <remarks>
/// A committed transaction can't practically be made to fail via public SQLite surface without
/// exhausting disk or injecting a custom VFS, so these tests force a genuine, non-swallowed
/// mid-batch INSERT failure instead (the "bonus defect" the finding also calls out: a mid-batch
/// insert failure on a later row loses the rows that inserted fine before it, the same way).
/// That failure is reached through a real SQLite trigger + a custom scalar function registered
/// directly on the writer's own connection (via reflection — <see cref="SqliteAuditWriter"/>
/// deliberately doesn't expose it), so the whole path under test — transaction, per-row
/// try/catch, rollback, TrySetException — is the writer's real production code, not a mock.
/// </remarks>
public class SqliteAuditWriterCommitFailureTests
{
private const string FailMarker = "P4-2-FORCED-FAILURE";
private static AuditEvent NewEvent(string target) => ScadaBridgeAuditEventFactory.Create(
channel: AuditChannel.ApiOutbound,
kind: AuditKind.ApiCall,
status: AuditStatus.Delivered,
eventId: Guid.NewGuid(),
occurredAtUtc: DateTime.UtcNow,
target: target);
/// <summary>
/// Installs a BEFORE INSERT trigger on <c>audit_event</c> that calls a custom scalar
/// function throwing whenever the inserted row's <c>Target</c> is <see cref="FailMarker"/>.
/// An exception thrown from a Microsoft.Data.Sqlite custom function surfaces as a generic
/// SQLITE_ERROR (code 1), NOT SQLITE_CONSTRAINT (code 19) — so it is NOT caught by
/// FlushBatch's duplicate-EventId swallow (which matches only code 19) and instead reaches
/// the outer catch exactly like a real disk-outage commit failure would.
/// </summary>
private static void InstallFailureTrigger(SqliteAuditWriter writer)
{
var connectionField = typeof(SqliteAuditWriter).GetField(
"_connection", BindingFlags.NonPublic | BindingFlags.Instance)
?? throw new InvalidOperationException("SqliteAuditWriter._connection not found — test seam broke.");
var connection = (SqliteConnection)connectionField.GetValue(writer)!;
connection.CreateFunction<string, long>("throw_if_marker", target =>
{
if (target == FailMarker)
{
throw new InvalidOperationException("P4-2 test-forced insert failure");
}
return 0;
});
using var cmd = connection.CreateCommand();
cmd.CommandText = $"""
CREATE TRIGGER test_fail_marker
BEFORE INSERT ON audit_event
FOR EACH ROW WHEN NEW.Target = '{FailMarker}'
BEGIN
SELECT throw_if_marker(NEW.Target);
END;
""";
cmd.ExecuteNonQuery();
}
private static (SqliteAuditWriter writer, string dataSource) CreateWriter(string testName)
{
var dataSource = $"file:{testName}-{Guid.NewGuid():N}?mode=memory&cache=shared";
var opts = new SqliteAuditWriterOptions
{
DatabasePath = dataSource,
// Generous flush window: both writes below are enqueued back-to-back with no
// await between them, so they are already both in the channel long before the
// background loop's first drain attempt — this just gives extra margin so the
// test can't flake into two separate flushes on a slow CI box.
FlushIntervalMs = 1000,
};
var writer = new SqliteAuditWriter(
Options.Create(opts),
NullLogger<SqliteAuditWriter>.Instance,
new FakeNodeIdentityProvider(),
connectionStringOverride: $"Data Source={dataSource};Cache=Shared");
return (writer, dataSource);
}
[Fact]
public async Task FlushBatch_MidBatchInsertFailure_FaultsEveryRowInBatch_NotJustTheFailingRow()
{
var (writer, dataSource) = CreateWriter(
nameof(FlushBatch_MidBatchInsertFailure_FaultsEveryRowInBatch_NotJustTheFailingRow));
await using var _ = writer;
InstallFailureTrigger(writer);
// evt1 is an ordinary row — its own two INSERTs succeed. evt2 carries the marker and
// throws mid-batch, from WITHIN the same transaction as evt1's already-successful
// inserts. Fired without individually awaiting so both land in the channel before the
// background loop's first drain — i.e. the same FlushBatch transaction.
var evt1 = NewEvent("normal-row");
var evt2 = NewEvent(FailMarker);
var t1 = writer.WriteAsync(evt1);
var t2 = writer.WriteAsync(evt2);
try
{
await Task.WhenAll(t1, t2).WaitAsync(TimeSpan.FromSeconds(10));
}
catch
{
// Expected: both tasks fault. Individually asserted below so a failure here
// reports which one didn't behave as expected, not just "something threw".
}
// THE regression assertion: evt1's own inserts succeeded, but the batch it shared
// with evt2 rolled back — its Task must fault, not silently report success. Pre-fix,
// t1 would already be RanToCompletion at this point (TrySetResult ran inside the loop
// before evt2's insert threw), and the later TrySetException in the outer catch would
// be a silent no-op against the already-completed TaskCompletionSource.
Assert.True(t1.IsFaulted,
"evt1's row was rolled back along with the rest of the batch, so its Task must fault instead of reporting false success.");
Assert.True(t2.IsFaulted, "evt2's own insert threw directly and must fault its Task.");
// No commit ever happened for this batch.
Assert.Equal(0, writer.FlushCountForTests);
// And the database genuinely holds neither row — proves the rollback actually
// happened, not just that the Task-completion contract changed.
using var verify = new SqliteConnection($"Data Source={dataSource};Cache=Shared");
verify.Open();
using var countCmd = verify.CreateCommand();
countCmd.CommandText = "SELECT COUNT(*) FROM audit_event;";
Assert.Equal(0L, (long)countCmd.ExecuteScalar()!);
}
[Fact]
public async Task FallbackAuditWriter_MidBatchInsertFailure_RingsEveryFaultedRow_AndIncrementsFailureCounterForEach()
{
// End-to-end proof the fix actually closes the loop the finding describes:
// FallbackAuditWriter composes the REAL SqliteAuditWriter as its primary. Before the
// fix, evt1's primary write would appear to SUCCEED (its Task completed before the
// rollback), so FallbackAuditWriter would never route it to the ring or count it as a
// failure — the whole batch would be silently lost with SiteAuditWriteFailures staying
// at 0, exactly as the finding describes.
var (writer, _) = CreateWriter(
nameof(FallbackAuditWriter_MidBatchInsertFailure_RingsEveryFaultedRow_AndIncrementsFailureCounterForEach));
await using var _w = writer;
InstallFailureTrigger(writer);
var ring = new RingBufferFallback(capacity: 16);
var counter = Substitute.For<IAuditWriteFailureCounter>();
var fallback = new FallbackAuditWriter(
writer, ring, counter, NullLogger<FallbackAuditWriter>.Instance);
var evt1 = NewEvent("normal-row");
var evt2 = NewEvent(FailMarker);
// FallbackAuditWriter.WriteAsync never throws (audit failures are always swallowed at
// that layer) — fired concurrently, same reasoning as above, so both land in one
// FlushBatch transaction.
var t1 = fallback.WriteAsync(evt1);
var t2 = fallback.WriteAsync(evt2);
await Task.WhenAll(t1, t2).WaitAsync(TimeSpan.FromSeconds(10));
// Both rows — including the one whose own INSERT succeeded before the batch rolled
// back — must have been routed to the ring and counted as a primary failure.
Assert.Equal(2, ring.Count);
counter.Received(2).Increment();
}
}
tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/Commands/AuditExportCommandTests.cs
+253
View File
@@ -1,3 +1,4 @@
using System.Diagnostics;
using System.Net;
using System.Text;
using System.Web;
@@ -376,4 +377,256 @@ public class AuditExportCommandTests
public override void SetLength(long value) => throw new NotSupportedException();
public override void Write(byte[] buffer, int offset, int count) => throw new NotSupportedException();
}
// ─────────────────────────────────────────────────────────────────────
// Deadline + Ctrl-C cancellation (arch-review 2026-08-19 P6-8: audit export
// was the one CLI HTTP path with NO deadline — a hung server hung the CLI
// forever). Every test below bounds its own wait well under what the old,
// deadline-less implementation would have done (indefinite hang) — a
// regression back to that state fails these on a timeout, proving the fix.
// ─────────────────────────────────────────────────────────────────────
/// <summary>A handler that never completes the request until its <see cref="CancellationToken"/> fires — simulates a server that accepted the connection but never answers.</summary>
private sealed class HangingHandler : HttpMessageHandler
{
public int CallCount { get; private set; }
protected override async Task<HttpResponseMessage> SendAsync(
HttpRequestMessage request, CancellationToken cancellationToken)
{
CallCount++;
await Task.Delay(Timeout.InfiniteTimeSpan, cancellationToken);
throw new InvalidOperationException("unreachable — Task.Delay(Infinite) only returns via cancellation");
}
}
/// <summary>
/// A stream that serves one small chunk, then goes silent — any subsequent
/// <see cref="ReadAsync(Memory{byte}, CancellationToken)"/> blocks until the
/// caller's token cancels. Simulates a connection that stalls mid-transfer.
/// </summary>
private sealed class StallingStream : Stream
{
private readonly byte[] _firstChunk;
private bool _servedFirstChunk;
public StallingStream(byte[] firstChunk) => _firstChunk = firstChunk;
public override bool CanRead => true;
public override bool CanSeek => false;
public override bool CanWrite => false;
public override long Length => throw new NotSupportedException();
public override long Position { get => throw new NotSupportedException(); set => throw new NotSupportedException(); }
public override int Read(byte[] buffer, int offset, int count) =>
ReadAsync(buffer.AsMemory(offset, count)).AsTask().GetAwaiter().GetResult();
public override async ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken cancellationToken = default)
{
if (!_servedFirstChunk)
{
_servedFirstChunk = true;
var n = Math.Min(buffer.Length, _firstChunk.Length);
_firstChunk.AsSpan(0, n).CopyTo(buffer.Span);
return n;
}
await Task.Delay(Timeout.InfiniteTimeSpan, cancellationToken);
return 0; // unreachable — Task.Delay(Infinite) only returns via cancellation
}
public override void Flush() { }
public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException();
public override void SetLength(long value) => throw new NotSupportedException();
public override void Write(byte[] buffer, int offset, int count) => throw new NotSupportedException();
}
/// <summary>A stream that serves one small chunk, then throws <see cref="IOException"/> — simulates a connection reset mid-transfer.</summary>
private sealed class FaultingStream : Stream
{
private readonly byte[] _firstChunk;
private bool _servedFirstChunk;
public FaultingStream(byte[] firstChunk) => _firstChunk = firstChunk;
public override bool CanRead => true;
public override bool CanSeek => false;
public override bool CanWrite => false;
public override long Length => throw new NotSupportedException();
public override long Position { get => throw new NotSupportedException(); set => throw new NotSupportedException(); }
public override int Read(byte[] buffer, int offset, int count) =>
ReadAsync(buffer.AsMemory(offset, count)).AsTask().GetAwaiter().GetResult();
public override ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken cancellationToken = default)
{
if (!_servedFirstChunk)
{
_servedFirstChunk = true;
var n = Math.Min(buffer.Length, _firstChunk.Length);
_firstChunk.AsSpan(0, n).CopyTo(buffer.Span);
return new ValueTask<int>(n);
}
throw new IOException("connection reset by peer (simulated)");
}
public override void Flush() { }
public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException();
public override void SetLength(long value) => throw new NotSupportedException();
public override void Write(byte[] buffer, int offset, int count) => throw new NotSupportedException();
}
/// <summary>Redirects <see cref="Console.Error"/> for the duration of the callback and returns what was written.</summary>
private static async Task<string> CaptureStderrAsync(Func<Task> action)
{
var original = Console.Error;
var capture = new StringWriter();
Console.SetError(capture);
try
{
await action();
}
finally
{
Console.SetError(original);
}
return capture.ToString();
}
[Fact]
public async Task RunExport_ServerNeverSendsHeaders_TimesOutWithFriendlyError()
{
var path = Path.Combine(Path.GetTempPath(), $"audit-export-hang-{Guid.NewGuid():N}.csv");
var handler = new HangingHandler();
var client = new ManagementHttpClient(new HttpClient(handler), "http://localhost:9001", "u", "p");
var output = new StringWriter();
var sw = Stopwatch.StartNew();
var exit = 0;
var stderr = await CaptureStderrAsync(async () =>
{
exit = await AuditExportHelpers.RunExportAsync(
client,
new AuditExportArgs { Since = "1h", Until = "0h", Format = "csv", Output = path },
output, DateTimeOffset.UtcNow,
headersTimeout: TimeSpan.FromMilliseconds(200));
});
sw.Stop();
Assert.Equal(1, exit);
// Bounded well under the old infinite-hang behaviour.
Assert.True(sw.Elapsed < TimeSpan.FromSeconds(5), $"took {sw.Elapsed}");
Assert.Contains("TIMEOUT", stderr);
Assert.False(File.Exists(path));
}
[Fact]
public async Task RunExport_StreamStallsMidTransfer_InactivityTimeoutFires()
{
var path = Path.Combine(Path.GetTempPath(), $"audit-export-stall-{Guid.NewGuid():N}.csv");
try
{
var firstChunk = Encoding.UTF8.GetBytes("EventId,OccurredAtUtc\n");
var handler = new BodyHandler(HttpStatusCode.OK,
() => new StreamContent(new StallingStream(firstChunk)));
var client = new ManagementHttpClient(new HttpClient(handler), "http://localhost:9001", "u", "p");
var output = new StringWriter();
var sw = Stopwatch.StartNew();
var exit = 0;
var stderr = await CaptureStderrAsync(async () =>
{
exit = await AuditExportHelpers.RunExportAsync(
client,
new AuditExportArgs { Since = "1h", Until = "0h", Format = "csv", Output = path },
output, DateTimeOffset.UtcNow,
inactivityTimeout: TimeSpan.FromMilliseconds(200));
});
sw.Stop();
Assert.Equal(1, exit);
Assert.True(sw.Elapsed < TimeSpan.FromSeconds(5), $"took {sw.Elapsed}");
Assert.Contains("TIMEOUT", stderr);
// Clean message, not a raw stack trace, and it notes the partial file.
Assert.DoesNotContain(" at ", stderr);
Assert.Contains(path, stderr);
// The first chunk that DID arrive before the stall must have been
// flushed to disk — this is a partial, not empty, export.
Assert.True(File.Exists(path));
Assert.Equal(firstChunk, await File.ReadAllBytesAsync(path));
}
finally
{
if (File.Exists(path)) File.Delete(path);
}
}
[Fact]
public async Task RunExport_MidStreamIOException_ProducesCleanMessageNotStackTrace()
{
var path = Path.Combine(Path.GetTempPath(), $"audit-export-fault-{Guid.NewGuid():N}.csv");
try
{
var firstChunk = Encoding.UTF8.GetBytes("EventId,OccurredAtUtc\n");
var handler = new BodyHandler(HttpStatusCode.OK,
() => new StreamContent(new FaultingStream(firstChunk)));
var client = new ManagementHttpClient(new HttpClient(handler), "http://localhost:9001", "u", "p");
var output = new StringWriter();
var exit = 0;
var stderr = await CaptureStderrAsync(async () =>
{
exit = await AuditExportHelpers.RunExportAsync(
client,
new AuditExportArgs { Since = "1h", Until = "0h", Format = "csv", Output = path },
output, DateTimeOffset.UtcNow);
});
Assert.Equal(1, exit);
Assert.Contains("STREAM_FAILED", stderr);
Assert.Contains("connection reset by peer", stderr);
Assert.Contains(path, stderr);
Assert.DoesNotContain(" at ", stderr);
Assert.DoesNotContain(nameof(FaultingStream), stderr);
}
finally
{
if (File.Exists(path)) File.Delete(path);
}
}
[Fact]
public async Task RunExport_CtrlCToken_CancelsPromptlyWithCleanMessage()
{
var path = Path.Combine(Path.GetTempPath(), $"audit-export-ctrlc-{Guid.NewGuid():N}.csv");
var handler = new HangingHandler();
var client = new ManagementHttpClient(new HttpClient(handler), "http://localhost:9001", "u", "p");
var output = new StringWriter();
using var cts = new CancellationTokenSource();
var sw = Stopwatch.StartNew();
var exit = 0;
var stderr = await CaptureStderrAsync(async () =>
{
var task = AuditExportHelpers.RunExportAsync(
client,
new AuditExportArgs { Since = "1h", Until = "0h", Format = "csv", Output = path },
output, DateTimeOffset.UtcNow,
cancellationToken: cts.Token,
// Deliberately much longer than the Ctrl-C delay below, so a pass
// here proves cancellation — not the timeout — ended the call.
headersTimeout: TimeSpan.FromSeconds(30));
await Task.Delay(50);
cts.Cancel();
exit = await task;
});
sw.Stop();
Assert.Equal(1, exit);
Assert.True(sw.Elapsed < TimeSpan.FromSeconds(5), $"took {sw.Elapsed}");
Assert.Contains("CANCELLED", stderr);
Assert.False(File.Exists(path));
}
}
tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/DataConnectionsPageTests.cs
+65
View File
@@ -40,6 +40,9 @@ public class DataConnectionsPageTests : BunitContext
// Satisfy the page's [Inject] IDialogService — the host that actually
// renders the dialog lives in MainLayout, not in bUnit's render scope.
Services.AddScoped<IDialogService, DialogService>();
// arch-review P6-4: the page now filters its site/connection tree through
// SiteScopeService, so it must be resolvable in every render test.
Services.AddScoped<ZB.MOM.WW.ScadaBridge.CentralUI.Auth.SiteScopeService>();
AddTestAuth();
// Default: no health known. Tests that assert badge state override this.
@@ -271,4 +274,66 @@ public class DataConnectionsPageTests : BunitContext
Assert.Contains(optionLabels, l => l.Contains("Plant-C"));
Assert.DoesNotContain(optionLabels, l => l.Contains("Plant-A"));
}
// ── arch-review P6-4: site scoping. This page reads ISiteRepository directly,
// bypassing ManagementActor's scope gate, and Designer is NOT exempt from
// site scope — so a scoped Designer used to see every site's connections. ──
[Fact]
public void SiteScoping_ScopedDesigner_SeesOnlyPermittedSitesAndConnections()
{
SeedRepos(
sites: new[]
{
new Site("Plant-A", "plant-a") { Id = 1 },
new Site("Plant-B", "plant-b") { Id = 2 }
},
connections: new[]
{
new DataConnection("PLC-A", "OpcUa", 1) { Id = 100 },
new DataConnection("PLC-B", "OpcUa", 2) { Id = 200 }
});
var scopedUser = new ClaimsPrincipal(new ClaimsIdentity(new[]
{
new Claim(JwtTokenService.UsernameClaimType, "scoped"),
new Claim(JwtTokenService.RoleClaimType, "Designer"),
new Claim(JwtTokenService.SiteIdClaimType, "1"), // Plant-A only
}, "TestAuth"));
Services.AddSingleton<AuthenticationStateProvider>(new TestAuthStateProvider(scopedUser));
var cut = Render<DataConnectionsPage>();
FindToggleForLabel(cut, "Plant-A")!.Click();
Assert.Contains("Plant-A", cut.Markup);
Assert.Contains("PLC-A", cut.Markup);
// Neither the out-of-scope site nor its connection may appear anywhere.
Assert.DoesNotContain("Plant-B", cut.Markup);
Assert.DoesNotContain("PLC-B", cut.Markup);
}
[Fact]
public void SiteScoping_SystemWideUser_SeesEveryConnection()
{
// The unscoped case is unchanged: the default test principal is an
// Administrator (system-wide), so the filter is a no-op.
SeedRepos(
sites: new[]
{
new Site("Plant-A", "plant-a") { Id = 1 },
new Site("Plant-B", "plant-b") { Id = 2 }
},
connections: new[]
{
new DataConnection("PLC-A", "OpcUa", 1) { Id = 100 },
new DataConnection("PLC-B", "OpcUa", 2) { Id = 200 }
});
var cut = Render<DataConnectionsPage>();
FindToggleForLabel(cut, "Plant-A")!.Click();
FindToggleForLabel(cut, "Plant-B")!.Click();
Assert.Contains("PLC-A", cut.Markup);
Assert.Contains("PLC-B", cut.Markup);
}
}
tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Deployment/DebugViewConnectDisposeRaceTests.cs
+211
View File
@@ -0,0 +1,211 @@
using System.Reflection;
using System.Security.Claims;
using Bunit;
using Microsoft.AspNetCore.Components.Authorization;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.CentralUI.Auth;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
using DebugViewPage = ZB.MOM.WW.ScadaBridge.CentralUI.Components.Pages.Deployment.DebugView;
namespace ZB.MOM.WW.ScadaBridge.CentralUI.Tests.Deployment;
/// <summary>
/// Regression tests for arch-review 2026-08-19 P6-6. <c>Connect()</c> awaits
/// <c>StartStreamAsync</c> — which can take up to 30s — and then publishes the result into
/// <c>_session</c>. If the circuit was torn down inside that window, <c>Dispose()</c> had
/// already run, found <c>_session</c> still null and stopped nothing, so the just-created
/// session stayed registered in the singleton <see cref="DebugStreamService"/>. Its
/// keepalive renews registered sessions forever (the orphan reaper only collects
/// UNregistered ones), so the site subscription, the bridge actor and the dead circuit's
/// object graph were pinned until process restart. The fix publishes the session and
/// re-checks disposal under one lock, so exactly one side tears the session down.
/// </summary>
public class DebugViewConnectDisposeRaceTests : BunitContext
{
/// <summary>
/// A <see cref="DebugStreamService"/> whose start can be held open on demand, so the
/// dispose-during-start window is deterministic rather than timing-dependent.
/// </summary>
private sealed class GatedDebugStreamService : DebugStreamService
{
private readonly TaskCompletionSource _gate = new(TaskCreationOptions.RunContinuationsAsynchronously);
private readonly TaskCompletionSource _entered = new(TaskCreationOptions.RunContinuationsAsynchronously);
private readonly List<string> _stopped = new();
public GatedDebugStreamService(
CommunicationService comms, IServiceProvider provider, SiteStreamGrpcClientFactory factory)
: base(comms, provider, factory, NullLogger<DebugStreamService>.Instance)
{
}
public string SessionId => "session-under-test";
/// <summary>Completes once <see cref="StartStreamAsync"/> has been entered.</summary>
public Task Entered => _entered.Task;
/// <summary>Session ids passed to <see cref="StopStream"/>, in call order.</summary>
public IReadOnlyList<string> Stopped
{
get { lock (_stopped) { return _stopped.ToList(); } }
}
public void ReleaseStart() => _gate.TrySetResult();
public override async Task<DebugStreamSession> StartStreamAsync(
int instanceId, Action<object> onEvent, Action onTerminated, CancellationToken ct = default)
{
_entered.TrySetResult();
await _gate.Task;
return new DebugStreamSession(
SessionId,
new DebugViewSnapshot(
"Inst-1",
Array.Empty<AttributeValueChanged>(),
Array.Empty<AlarmStateChanged>(),
DateTimeOffset.UtcNow));
}
public override void StopStream(string sessionId)
{
lock (_stopped) { _stopped.Add(sessionId); }
}
}
private sealed class StubAuthStateProvider : AuthenticationStateProvider
{
private readonly AuthenticationState _state;
public StubAuthStateProvider(AuthenticationState state) => _state = state;
public override Task<AuthenticationState> GetAuthenticationStateAsync()
=> Task.FromResult(_state);
}
private (IRenderedComponent<DebugViewPage> Cut, GatedDebugStreamService Stream) RenderPage()
{
JSInterop.Mode = JSRuntimeMode.Loose;
var repo = Substitute.For<ITemplateEngineRepository>();
var siteRepo = Substitute.For<ISiteRepository>();
siteRepo.GetAllSitesAsync().Returns(new List<Site>());
Services.AddSingleton(repo);
Services.AddSingleton(siteRepo);
var comms = new CommunicationService(
Options.Create(new CommunicationOptions()),
NullLogger<CommunicationService>.Instance);
Services.AddSingleton(comms);
// An empty throwaway provider — the gated stub never calls into the base start
// path, so nothing is resolved from it.
var stream = new GatedDebugStreamService(
comms,
new ServiceCollection().BuildServiceProvider(),
new SiteStreamGrpcClientFactory(NullLoggerFactory.Instance));
Services.AddSingleton<DebugStreamService>(stream);
var identity = new ClaimsIdentity(
new[] { new Claim(ClaimTypes.Name, "deployer") }, "TestCookie");
var stubAuth = new StubAuthStateProvider(
new AuthenticationState(new ClaimsPrincipal(identity)));
Services.AddSingleton<AuthenticationStateProvider>(stubAuth);
Services.AddScoped(_ => new SiteScopeService(stubAuth));
return (Render<DebugViewPage>(), stream);
}
private static void SetField(DebugViewPage page, string name, object value) =>
typeof(DebugViewPage).GetField(name, BindingFlags.Instance | BindingFlags.NonPublic)!
.SetValue(page, value);
private static object? GetField(DebugViewPage page, string name) =>
typeof(DebugViewPage).GetField(name, BindingFlags.Instance | BindingFlags.NonPublic)!
.GetValue(page);
/// <summary>
/// Starts <c>Connect()</c> on the renderer's dispatcher (where it really runs) and hands
/// back its task without awaiting it, so the test can act inside the start window.
/// </summary>
private static async Task<Task> StartConnectAsync(IRenderedComponent<DebugViewPage> cut)
{
var connect = typeof(DebugViewPage).GetMethod(
"Connect", BindingFlags.Instance | BindingFlags.NonPublic)!;
Task? task = null;
await cut.InvokeAsync(() => { task = (Task)connect.Invoke(cut.Instance, null)!; });
return task!;
}
[Fact]
public async Task Connect_DisposedMidStart_StopsTheJustCreatedSession()
{
var (cut, stream) = RenderPage();
SetField(cut.Instance, "_selectedSiteId", 1);
SetField(cut.Instance, "_selectedInstanceId", 7);
var connect = await StartConnectAsync(cut);
await stream.Entered.WaitAsync(TimeSpan.FromSeconds(5));
// The user navigates away while the start is still in flight. Dispose runs off the
// renderer's thread, exactly as a circuit teardown can.
cut.Instance.Dispose();
stream.ReleaseStart();
await connect.WaitAsync(TimeSpan.FromSeconds(10));
Assert.Equal(new[] { stream.SessionId }, stream.Stopped);
Assert.Null(GetField(cut.Instance, "_session"));
}
[Fact]
public async Task Connect_ThenDispose_StopsTheSessionExactlyOnce()
{
var (cut, stream) = RenderPage();
SetField(cut.Instance, "_selectedSiteId", 1);
SetField(cut.Instance, "_selectedInstanceId", 7);
// Normal path: the start completes while the component is very much alive.
stream.ReleaseStart();
var connect = await StartConnectAsync(cut);
await connect.WaitAsync(TimeSpan.FromSeconds(10));
Assert.Empty(stream.Stopped);
Assert.NotNull(GetField(cut.Instance, "_session"));
cut.Instance.Dispose();
Assert.Equal(new[] { stream.SessionId }, stream.Stopped);
// A second Dispose (bUnit tears the component down again at test end) must not
// stop an already-released session a second time.
cut.Instance.Dispose();
Assert.Equal(new[] { stream.SessionId }, stream.Stopped);
}
[Fact]
public async Task Disconnect_AfterConnect_StopsTheSessionOnceAndClearsTheSlot()
{
var (cut, stream) = RenderPage();
SetField(cut.Instance, "_selectedSiteId", 1);
SetField(cut.Instance, "_selectedInstanceId", 7);
stream.ReleaseStart();
var connect = await StartConnectAsync(cut);
await connect.WaitAsync(TimeSpan.FromSeconds(10));
var disconnect = typeof(DebugViewPage).GetMethod(
"Disconnect", BindingFlags.Instance | BindingFlags.NonPublic)!;
await cut.InvokeAsync(() => (Task)disconnect.Invoke(cut.Instance, null)!);
Assert.Equal(new[] { stream.SessionId }, stream.Stopped);
Assert.Null(GetField(cut.Instance, "_session"));
// Dispose after an explicit disconnect has nothing left to stop.
cut.Instance.Dispose();
Assert.Equal(new[] { stream.SessionId }, stream.Stopped);
}
}
tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/AuditLogExportServiceTests.cs
+92 -14
View File
@@ -68,14 +68,14 @@ public class AuditLogExportServiceTests
// 1 header + 5 rows + trailing empty token from final \r\n = 7 entries.
Assert.Equal(7, lines.Length);
Assert.StartsWith("EventId,OccurredAtUtc,IngestedAtUtc,Channel,Kind,CorrelationId,SourceSiteId,", lines[0]);
Assert.StartsWith("EventId,OccurredAtUtc,IngestedAtUtc,Channel,Kind,CorrelationId,ExecutionId,ParentExecutionId,SourceSiteId,SourceNode,", lines[0]);
Assert.StartsWith("11111111-1111-1111-1111-111111111111,", lines[1]);
Assert.StartsWith("55555555-5555-5555-5555-555555555555,", lines[5]);
Assert.Equal(string.Empty, lines[6]);
}
[Fact]
public async Task ExportAsync_HeaderHasAll21Columns_InSpecOrder()
public async Task ExportAsync_HeaderHasAll24Columns_InSpecOrder()
{
var repo = Substitute.For<IAuditLogRepository>();
repo.QueryAsync(Arg.Any<AuditLogQueryFilter>(), Arg.Any<AuditLogPaging>(), Arg.Any<CancellationToken>())
@@ -89,11 +89,14 @@ public class AuditLogExportServiceTests
var header = csv.Split("\r\n")[0];
var columns = header.Split(',');
Assert.Equal(21, columns.Length);
// arch-review 2026-08-19 P6-7: ExecutionId/ParentExecutionId/SourceNode
// were missing from this CSV (JSON/JSONL exports already carried them).
Assert.Equal(24, columns.Length);
Assert.Equal(new[]
{
"EventId", "OccurredAtUtc", "IngestedAtUtc", "Channel", "Kind",
"CorrelationId", "SourceSiteId", "SourceInstanceId", "SourceScript",
"CorrelationId", "ExecutionId", "ParentExecutionId",
"SourceSiteId", "SourceNode", "SourceInstanceId", "SourceScript",
"Actor", "Target", "Status", "HttpStatus", "DurationMs",
"ErrorMessage", "ErrorDetail", "RequestSummary", "ResponseSummary",
"PayloadTruncated", "Extra", "ForwardState",
@@ -136,6 +139,77 @@ public class AuditLogExportServiceTests
Assert.Contains("\"boom\r\nthen again\"", csv);
}
[Fact]
public async Task ExportAsync_FreeTextLeadingFormulaCharacters_ArePrefixedWithQuote()
{
// arch-review 2026-08-19 P6-7 (CWE-1236): a free-text cell starting with
// =, +, -, @, or TAB must be prefixed with a literal single quote so a
// spreadsheet application renders it as inert text, not a formula/DDE
// launch. Target, ErrorMessage, ResponseSummary, and Extra are
// inbound-API-caller-controlled and are the primary attack surface.
var row = ScadaBridgeAuditEventFactory.Create(
channel: AuditChannel.ApiOutbound,
kind: AuditKind.ApiCall,
status: AuditStatus.Delivered,
eventId: Guid.Parse("cccccccc-cccc-cccc-cccc-cccccccccccc"),
occurredAtUtc: new DateTime(2026, 5, 20, 12, 0, 0, DateTimeKind.Utc),
target: "=HYPERLINK(\"http://evil\",\"click\")",
errorMessage: "+cmd|'/c calc'!A1",
responseSummary: "-2+3",
extra: "@SUM(A1:A9)");
var repo = Substitute.For<IAuditLogRepository>();
repo.QueryAsync(Arg.Any<AuditLogQueryFilter>(), Arg.Any<AuditLogPaging>(), Arg.Any<CancellationToken>())
.Returns(
Task.FromResult<IReadOnlyList<AuditEvent>>(new[] { row }),
Task.FromResult<IReadOnlyList<AuditEvent>>(Array.Empty<AuditEvent>()));
var sut = new AuditLogExportService(repo);
using var ms = new MemoryStream();
await sut.ExportAsync(new AuditLogQueryFilter(), maxRows: 10, ms, CancellationToken.None);
var csv = Encoding.UTF8.GetString(ms.ToArray());
var dataLine = csv.Split("\r\n")[1];
Assert.Contains("'=HYPERLINK(\"\"http://evil\"\",\"\"click\"\")", dataLine);
Assert.Contains("'+cmd|'/c calc'!A1", dataLine);
Assert.Contains(",'-2+3,", dataLine);
Assert.Contains(",'@SUM(A1:A9)", dataLine);
}
[Fact]
public async Task ExportAsync_NumericAndDateColumns_NeverPrefixed()
{
// HttpStatus/DurationMs/OccurredAtUtc are formatted by the exporter
// itself (never free text) and must render unmangled even when a
// negative DurationMs would otherwise look like a formula-neutralized
// "-" leader if the neutralization were mistakenly applied there.
var row = ScadaBridgeAuditEventFactory.Create(
channel: AuditChannel.ApiOutbound,
kind: AuditKind.ApiCall,
status: AuditStatus.Delivered,
eventId: Guid.Parse("dddddddd-dddd-dddd-dddd-dddddddddddd"),
occurredAtUtc: new DateTime(2026, 5, 20, 12, 0, 0, DateTimeKind.Utc),
httpStatus: 200,
durationMs: -5);
var repo = Substitute.For<IAuditLogRepository>();
repo.QueryAsync(Arg.Any<AuditLogQueryFilter>(), Arg.Any<AuditLogPaging>(), Arg.Any<CancellationToken>())
.Returns(
Task.FromResult<IReadOnlyList<AuditEvent>>(new[] { row }),
Task.FromResult<IReadOnlyList<AuditEvent>>(Array.Empty<AuditEvent>()));
var sut = new AuditLogExportService(repo);
using var ms = new MemoryStream();
await sut.ExportAsync(new AuditLogQueryFilter(), maxRows: 10, ms, CancellationToken.None);
var csv = Encoding.UTF8.GetString(ms.ToArray());
var dataLine = csv.Split("\r\n")[1];
var fields = dataLine.Split(',');
Assert.Equal("200", fields[15]); // HttpStatus, unprefixed
Assert.Equal("-5", fields[16]); // DurationMs, unprefixed despite leading '-'
Assert.StartsWith("2026-05-20", fields[1]); // OccurredAtUtc, unprefixed
}
[Fact]
public async Task ExportAsync_NullField_WrittenAsEmpty()
{
@@ -161,19 +235,23 @@ public class AuditLogExportServiceTests
var fields = dataLine.Split(',');
// EventId(0), OccurredAtUtc(1), IngestedAtUtc(2), Channel(3), Kind(4),
// CorrelationId(5), SourceSiteId(6), SourceInstanceId(7), SourceScript(8),
// Actor(9), Target(10), Status(11), HttpStatus(12), DurationMs(13),
// ErrorMessage(14), ErrorDetail(15), RequestSummary(16), ResponseSummary(17),
// PayloadTruncated(18), Extra(19), ForwardState(20)
Assert.Equal(21, fields.Length);
// CorrelationId(5), ExecutionId(6), ParentExecutionId(7), SourceSiteId(8),
// SourceNode(9), SourceInstanceId(10), SourceScript(11), Actor(12),
// Target(13), Status(14), HttpStatus(15), DurationMs(16), ErrorMessage(17),
// ErrorDetail(18), RequestSummary(19), ResponseSummary(20),
// PayloadTruncated(21), Extra(22), ForwardState(23)
Assert.Equal(24, fields.Length);
Assert.Equal("bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb", fields[0]);
Assert.Equal(string.Empty, fields[2]); // IngestedAtUtc null
Assert.Equal(string.Empty, fields[5]); // CorrelationId null
Assert.Equal(string.Empty, fields[6]); // SourceSiteId null
Assert.Equal(string.Empty, fields[12]); // HttpStatus null
Assert.Equal(string.Empty, fields[14]); // ErrorMessage null
Assert.Equal("False", fields[18]); // PayloadTruncated
Assert.Equal(string.Empty, fields[20]); // ForwardState null
Assert.Equal(string.Empty, fields[6]); // ExecutionId null
Assert.Equal(string.Empty, fields[7]); // ParentExecutionId null
Assert.Equal(string.Empty, fields[8]); // SourceSiteId null
Assert.Equal(string.Empty, fields[9]); // SourceNode null
Assert.Equal(string.Empty, fields[15]); // HttpStatus null
Assert.Equal(string.Empty, fields[17]); // ErrorMessage null
Assert.Equal("False", fields[21]); // PayloadTruncated
Assert.Equal(string.Empty, fields[23]); // ForwardState null
}
[Fact]
tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/AuditQueryParamParsersTests.cs
+60
View File
@@ -145,6 +145,66 @@ public class AuditQueryParamParsersTests
new[] { "", " ", null }, "status"));
}
// ========================================================================
// Comma-joined single token (arch-review 2026-08-19 P6-3, register row 99
// closure defect): a bare Enum.TryParse splits and bitwise-ORs a comma list
// inside ONE token for ANY enum, not just [Flags] ones. AuditStatus is dense
// and 0-based, so the OR'd composite almost always collides with a REAL, but
// WRONG, member. Values arriving as separate tokens (repeated query param /
// already-split list) must be unaffected — only a comma INSIDE one token is
// rejected. Proof this closes the defect: reverting the fix (`git stash`)
// makes every one of these throw-expecting tests fail, because the call
// instead RETURNS the wrong composite value below.
// ========================================================================
[Fact]
public void ParseEnumListStrict_CommaJoinedTwoNames_IsRejected_NotSilentlyOrdedIntoWrongMember()
{
// Forwarded(1) | Attempted(2) = 3 = Delivered — the exact silent-wrong-data
// collision the finding calls out. Pre-fix this returned [Delivered].
var ex = Assert.Throws<AuditFilterParseException>(
() => AuditQueryParamParsers.ParseEnumListStrict<AuditStatus>(
new[] { "Forwarded,Attempted" }, "status"));
Assert.Contains("Forwarded,Attempted", ex.Message);
Assert.Contains("status", ex.Message);
}
[Fact]
public void ParseEnumListStrict_CommaJoinedNameAndOrdinal_IsRejected_NotSmuggledPastNumericGuard()
{
// Failed(4) | 7 = 7 = Skipped — a comma smuggles an ordinal past the
// LooksNumeric guard because the guard only inspects the WHOLE token, and
// a comma-joined token starts with a letter. Pre-fix this returned [Skipped].
var ex = Assert.Throws<AuditFilterParseException>(
() => AuditQueryParamParsers.ParseEnumListStrict<AuditStatus>(
new[] { "Failed,7" }, "status"));
Assert.Contains("Failed,7", ex.Message);
}
[Fact]
public void ParseEnumListStrict_CommaJoinedNamesDropAMember_IsRejected()
{
// Submitted(0) | Parked(5) = 5 = Parked — Submitted silently vanishes from
// the composite. Pre-fix this returned [Parked], dropping Submitted entirely.
var ex = Assert.Throws<AuditFilterParseException>(
() => AuditQueryParamParsers.ParseEnumListStrict<AuditStatus>(
new[] { "Submitted,Parked" }, "status"));
Assert.Contains("Submitted,Parked", ex.Message);
}
[Fact]
public void ParseEnumListStrict_RepeatedParamTokensOfValidNames_StillParseIndependently()
{
// The fix targets a comma INSIDE one token — values that legitimately
// arrive as separate tokens (a repeated query param, or an already-split
// list) must keep working exactly as before.
var result = AuditQueryParamParsers.ParseEnumListStrict<AuditStatus>(
new[] { "Forwarded", "Attempted", "Submitted" }, "status");
Assert.Equal(
new[] { AuditStatus.Forwarded, AuditStatus.Attempted, AuditStatus.Submitted },
result);
}
[Fact]
public void ParseStringList_NullInput_ReturnsNull()
{
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/DebugStreamServiceTests.cs
+89
View File
@@ -164,4 +164,93 @@ public class DebugStreamServiceTests : TestKit
DebugStreamBridgeActor.ConsumerLivenessCheckInterval = previousCheck;
}
}
// --- arch-review 2026-08-19 P3-3: single-endpoint sites ---
[Theory]
[InlineData("http://localhost:5100", null)]
[InlineData(null, "http://localhost:5200")]
public async Task StartStreamAsync_SingleEndpointSite_UsesTheSoleAddressForBothNodeSlots(
string? nodeA, string? nodeB)
{
// Debug View used to hard-require BOTH gRPC node addresses and threw before ever
// creating the bridge actor, though single-endpoint sites are supported everywhere else
// (SiteAlarmLiveCacheService's N9 degeneration, SitePairChannelProvider's null-tolerant
// pair). The sole configured address now fills both slots, so the bridge actor's
// NodeA↔NodeB flip degenerates to a reconnect against the same node.
var instance = new Instance("Site1.Pump01") { Id = 7, SiteId = 3 };
var site = new Site("Site One", "site-1")
{
Id = 3,
GrpcNodeAAddress = nodeA,
GrpcNodeBAddress = nodeB
};
var instanceRepo = Substitute.For<ITemplateEngineRepository>();
instanceRepo.GetInstanceByIdAsync(7, Arg.Any<CancellationToken>()).Returns(instance);
var siteRepo = Substitute.For<ISiteRepository>();
siteRepo.GetSiteByIdAsync(3, Arg.Any<CancellationToken>()).Returns(site);
var services = new ServiceCollection();
services.AddScoped(_ => instanceRepo);
services.AddScoped(_ => siteRepo);
using var provider = services.BuildServiceProvider();
var commProbe = CreateTestProbe();
var commService = new CommunicationService(
Options.Create(new CommunicationOptions()),
NullLogger<CommunicationService>.Instance);
commService.SetCommunicationActor(commProbe.Ref);
using var grpcFactory = new Grpc.MockSiteStreamGrpcClientFactory(
new Grpc.MockSiteStreamGrpcClient());
using var service = new DebugStreamService(
commService, provider, grpcFactory, NullLogger<DebugStreamService>.Instance);
service.SetActorSystem(Sys);
// Act — resolution must get past the address check and start a bridge actor.
var startTask = service.StartStreamAsync(instanceId: 7, onEvent: _ => { }, onTerminated: () => { });
commProbe.ExpectMsg<SiteEnvelope>(TimeSpan.FromSeconds(5));
var bridgeActor = commProbe.LastSender;
// The stream it opened dials the ONE configured address.
Assert.Equal(new[] { nodeA ?? nodeB }, grpcFactory.RequestedEndpoints);
// Unwind the session.
bridgeActor.Tell(new DebugStreamTerminated("site-1", "corr"));
await Assert.ThrowsAsync<InvalidOperationException>(() => startTask);
}
[Fact]
public async Task StartStreamAsync_SiteWithNoGrpcAddresses_StillThrows()
{
// Degenerating a single endpoint must not degenerate into accepting NONE.
var instance = new Instance("Site1.Pump01") { Id = 7, SiteId = 3 };
var site = new Site("Site One", "site-1") { Id = 3 };
var instanceRepo = Substitute.For<ITemplateEngineRepository>();
instanceRepo.GetInstanceByIdAsync(7, Arg.Any<CancellationToken>()).Returns(instance);
var siteRepo = Substitute.For<ISiteRepository>();
siteRepo.GetSiteByIdAsync(3, Arg.Any<CancellationToken>()).Returns(site);
var services = new ServiceCollection();
services.AddScoped(_ => instanceRepo);
services.AddScoped(_ => siteRepo);
using var provider = services.BuildServiceProvider();
var commService = new CommunicationService(
Options.Create(new CommunicationOptions()),
NullLogger<CommunicationService>.Instance);
commService.SetCommunicationActor(CreateTestProbe().Ref);
using var grpcFactory = new SiteStreamGrpcClientFactory(NullLoggerFactory.Instance);
using var service = new DebugStreamService(
commService, provider, grpcFactory, NullLogger<DebugStreamService>.Instance);
service.SetActorSystem(Sys);
var ex = await Assert.ThrowsAsync<InvalidOperationException>(() =>
service.StartStreamAsync(instanceId: 7, onEvent: _ => { }, onTerminated: () => { }));
Assert.Contains("no gRPC node addresses", ex.Message);
}
}
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/CentralControlDtoMapperTests.cs
+8
View File
@@ -554,6 +554,11 @@ public class CentralControlDtoMapperTests
LocalDbReplicationConnected = true,
LocalDbOplogBacklog = 250L,
StartupReconcilerActive = true,
// arch-review 2026-08-19 P8-1: these three never crossed the wire at all
// (mapper omitted them entirely, always reading back as 0) until this fix.
DetachedScriptThreads = 2,
ScriptRunShedCount = 13,
SiteStreamAlarmDropCount = 41L,
};
/// <summary>
@@ -603,6 +608,9 @@ public class CentralControlDtoMapperTests
Assert.Equal(expected.LocalDbReplicationConnected, actual.LocalDbReplicationConnected);
Assert.Equal(expected.LocalDbOplogBacklog, actual.LocalDbOplogBacklog);
Assert.Equal(expected.StartupReconcilerActive, actual.StartupReconcilerActive);
Assert.Equal(expected.DetachedScriptThreads, actual.DetachedScriptThreads);
Assert.Equal(expected.ScriptRunShedCount, actual.ScriptRunShedCount);
Assert.Equal(expected.SiteStreamAlarmDropCount, actual.SiteStreamAlarmDropCount);
}
/// <summary>
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteAlarmAggregatorActorTests.cs
+59 -23
View File
@@ -15,6 +15,15 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Tests.Grpc;
/// dedup by AlarmKey, placeholder seeded-but-not-on-stream coherence, NodeA↔NodeB reconnect
/// re-seed, and periodic reconcile authoritative-replace (drift correction).
/// </summary>
/// <remarks>
/// Serialized with <c>SiteAlarmLiveCacheTelemetryTests</c> (register row 102): every actor
/// this class spins up moves the process-global
/// <c>scadabridge.site.alarm_cache.aggregators.active</c> gauge via
/// <see cref="SiteAlarmAggregatorActor"/>'s PreStart/PostStop, which the telemetry test's
/// fixed-baseline assertions require to hold still (or only increase) between its capture
/// and its checks. Same collection ⇒ never concurrent.
/// </remarks>
[Collection("SiteAlarmAggregatorGauge")]
public class SiteAlarmAggregatorActorTests : TestKit
{
private const string SiteId = "site-alpha";
@@ -188,6 +197,28 @@ public class SiteAlarmAggregatorActorTests : TestKit
private static AlarmStateChanged? Find(IReadOnlyList<AlarmStateChanged>? rows, string alarmName, string sourceRef) =>
rows?.FirstOrDefault(r => r.AlarmName == alarmName && r.SourceReference == sourceRef);
/// <summary>
/// Row-40 house pattern for an ABSENCE assertion: proves <paramref name="checks"/> keep
/// holding throughout <paramref name="window"/> by polling, rather than a single blind
/// Thread.Sleep(window) followed by one end-of-window sample. A bare sleep-then-assert
/// cannot distinguish "the mechanism correctly stayed quiet" from "the mechanism just
/// hadn't run yet when the one sample was taken", and it also can't catch a value that
/// flips wrong and then flips back before that single sample. This fails immediately on
/// the first tick any check goes false, and only returns once the full window has
/// elapsed with every check still true.
/// </summary>
private static void AssertHoldsFor(TimeSpan window, params (Func<bool> Holds, string Because)[] checks)
{
var deadline = DateTime.UtcNow + window;
while (true)
{
foreach (var (holds, because) in checks)
Assert.True(holds(), because);
if (DateTime.UtcNow >= deadline) return;
Thread.Sleep(20);
}
}
// ── Tests ───────────────────────────────────────────────────────────────────
[Fact]
@@ -226,8 +257,11 @@ public class SiteAlarmAggregatorActorTests : TestKit
seed.CompleteNext(Alarm("PumpFault", "", 500, tSeed));
AwaitCondition(() => sink.Count >= 1, TimeSpan.FromSeconds(3));
// Give a beat to ensure no late (dropped) delta sneaks in.
Thread.Sleep(150);
// Prove no late (dropped) delta sneaks in over the window, not just at one sample.
AssertHoldsFor(TimeSpan.FromMilliseconds(150),
(() => Find(sink.Latest, "PumpFault", "") is not null, "seed row must still be present"),
(() => Find(sink.Latest, "PumpFault", "")?.Priority == 500,
"seed value kept; older buffered delta must stay dropped"));
var latest = sink.Latest!;
var pumpFault = Find(latest, "PumpFault", "");
Assert.NotNull(pumpFault);
@@ -382,8 +416,8 @@ public class SiteAlarmAggregatorActorTests : TestKit
seed.FaultNext();
// A failed INITIAL seed must not publish (page keeps its poll fallback — not live yet).
Thread.Sleep(200);
Assert.Equal(0, sink.Count);
AssertHoldsFor(TimeSpan.FromMilliseconds(200),
(() => sink.Count == 0, "a failed initial seed must not publish"));
// The periodic reconcile is the backstop: it re-runs the fan-out (flag was cleared
// on failure), completes, and the actor publishes → now live.
@@ -464,8 +498,8 @@ public class SiteAlarmAggregatorActorTests : TestKit
firstSub.OnError(new Exception("late zombie fault")); // stale gen-1 error races in
// Pre-fix this burns retry budget and opens a THIRD stream; post-fix it is ignored.
Thread.Sleep(400);
Assert.Equal(2, TotalSubs());
AssertHoldsFor(TimeSpan.FromMilliseconds(400),
(() => TotalSubs() == 2, "a stale gen-1 error must not open a third stream"));
}
// ── WP1.1: graceful (status OK) stream completion is a reconnect trigger ──
@@ -520,8 +554,8 @@ public class SiteAlarmAggregatorActorTests : TestKit
// … and without refunding it either: the next error is the 4th, so it exceeds
// MaxRetries and the stream is given up rather than reconnected.
factory.ClientFor(GrpcNodeB).Subs[2].OnError(new Exception("4"));
Thread.Sleep(400);
Assert.Equal(before, TotalSubs());
AssertHoldsFor(TimeSpan.FromMilliseconds(400),
(() => TotalSubs() == before, "budget exhausted — the 4th error must not reconnect"));
}
[Fact]
@@ -539,9 +573,11 @@ public class SiteAlarmAggregatorActorTests : TestKit
// A completion racing out of the dead gen-1 stream must not tear down the live
// gen-2 stream (which would go deltaless until the next reconcile tick).
firstSub.OnCompleted();
Thread.Sleep(300);
Assert.DoesNotContain("corr-1", factory.ClientFor(GrpcNodeB).Unsubscribed);
Assert.Single(factory.ClientFor(GrpcNodeB).Subs);
AssertHoldsFor(TimeSpan.FromMilliseconds(300),
(() => !factory.ClientFor(GrpcNodeB).Unsubscribed.Contains("corr-1"),
"a late gen-1 completion must not unsubscribe the live gen-2 stream"),
(() => factory.ClientFor(GrpcNodeB).Subs.Count == 1,
"the live gen-2 stream must not be torn down"));
}
// ── R2 T10: live-delta publish coalescing (N6) ──
@@ -621,15 +657,15 @@ public class SiteAlarmAggregatorActorTests : TestKit
factory.ClientFor(GrpcNodeA).Subs[1].OnError(new Exception("3"));
AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 2, TimeSpan.FromSeconds(5));
Thread.Sleep(300);
Assert.Equal(1, seed.CallCount); // pre-fix: 4
AssertHoldsFor(TimeSpan.FromMilliseconds(300),
(() => seed.CallCount == 1, "failed reconnect attempts must not re-seed (pre-fix: 4)"));
// The stream finally comes up - exactly ONE re-seed, on the connect.
factory.ClientFor(GrpcNodeB).Subs[1].OnConnected!();
AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(5));
seed.CompleteNext();
Thread.Sleep(300);
Assert.Equal(2, seed.CallCount); // and no second one
AssertHoldsFor(TimeSpan.FromMilliseconds(300),
(() => seed.CallCount == 2, "the successful connect must re-seed exactly once"));
}
[Fact]
@@ -644,8 +680,8 @@ public class SiteAlarmAggregatorActorTests : TestKit
Thread.Sleep(200);
actor.Tell(new RunReconcile()); // skipped - the initial seed covered this window
Thread.Sleep(300);
Assert.Equal(1, seed.CallCount);
AssertHoldsFor(TimeSpan.FromMilliseconds(300),
(() => seed.CallCount == 1, "a tick inside the just-seeded window must be skipped"));
actor.Tell(new RunReconcile()); // and the next one runs
AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(3));
@@ -668,8 +704,8 @@ public class SiteAlarmAggregatorActorTests : TestKit
AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(3));
seed.CompleteNext(Alarm("PumpFault", "", 500, t)); // identical snapshot
Thread.Sleep(300);
Assert.Equal(1, sink.Count); // no second publish
AssertHoldsFor(TimeSpan.FromMilliseconds(300),
(() => sink.Count == 1, "an unchanged reconcile snapshot must not republish"));
// A genuinely changed snapshot still publishes.
actor.Tell(new RunReconcile());
@@ -733,8 +769,8 @@ public class SiteAlarmAggregatorActorTests : TestKit
Thread.Sleep(150);
actor.Tell(new RunReconcile()); // consumed by the initial seed's flag
Thread.Sleep(200);
Assert.Equal(1, seed.CallCount);
AssertHoldsFor(TimeSpan.FromMilliseconds(200),
(() => seed.CallCount == 1, "the tick right after the initial seed must be skipped"));
// From here on, with no reconnect in between, EVERY tick must fan out.
for (var expected = 2; expected <= 5; expected++)
@@ -778,8 +814,8 @@ public class SiteAlarmAggregatorActorTests : TestKit
// That re-seed covers this window: the next tick is skipped…
actor.Tell(new RunReconcile());
Thread.Sleep(250);
Assert.Equal(3, seed.CallCount);
AssertHoldsFor(TimeSpan.FromMilliseconds(250),
(() => seed.CallCount == 3, "the connect-driven re-seed must stand the next tick down"));
// …and the one after it fans out again.
actor.Tell(new RunReconcile());
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SitePairChannelProviderDisposalTests.cs
+105
View File
@@ -0,0 +1,105 @@
using Grpc.Net.Client;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests.Grpc;
/// <summary>
/// Arch-review 2026-08-19 P3-2 — a channel replaced by the 60 s DB address refresh (or an admin
/// edit) must not be disposed out from under an in-flight <see cref="SitePairChannelProvider.ExecuteAsync{T}"/>
/// call. <c>ExecuteAsync</c> reads the channel under the pair gate but runs the RPC outside it, so an
/// inline dispose aborted whatever was running — a multi-minute <c>DeployArtifactsCommand</c>
/// included — with a status the cross-node retry deliberately never engages on.
/// </summary>
public class SitePairChannelProviderDisposalTests
{
private static SitePairChannelProvider Build() => new(
new NoKeyProvider(),
Options.Create(new CommunicationOptions()),
NullLogger<SitePairChannelProvider>.Instance);
[Fact]
public async Task UpdateSite_WhileCallInFlight_ParksTheOldChannelAndLetsTheCallComplete()
{
using var provider = Build();
provider.UpdateSite("site-1", "http://localhost:15101", nodeBEndpoint: null);
var entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
GrpcChannel? inFlightChannel = null;
// A long-running command (deploy) against the site's current channel.
var call = provider.ExecuteAsync("site-1", async (channel, ct) =>
{
inFlightChannel = channel;
entered.SetResult();
await release.Task;
// A disposed GrpcChannel throws here — this is the assertion that the address
// refresh did not pull the channel out from under the call.
_ = channel.CreateCallInvoker();
return 42;
}, CancellationToken.None);
await entered.Task.WaitAsync(TimeSpan.FromSeconds(5));
// The DB refresh tick re-points the site at a new address, mid-call.
provider.UpdateSite("site-1", "http://localhost:15102", nodeBEndpoint: null);
// The superseded channel is PARKED, not disposed synchronously.
Assert.Equal(1, provider.ParkedChannelCount);
release.SetResult();
Assert.Equal(42, await call.WaitAsync(TimeSpan.FromSeconds(5)));
// Once the grace window elapses the park drains and the channel really is disposed —
// deferred disposal, not a leak.
provider.SweepGraveyard(force: true);
Assert.Equal(0, provider.ParkedChannelCount);
Assert.Throws<ObjectDisposedException>(() => inFlightChannel!.CreateCallInvoker());
}
[Fact]
public void RemoveSite_ParksBothChannelsRatherThanDisposingThemInline()
{
using var provider = Build();
provider.UpdateSite("site-1", "http://localhost:15111", "http://localhost:15112");
provider.RemoveSite("site-1");
// Same hazard as UpdateSite: an admin deleting a site while a command is in flight.
Assert.Equal(2, provider.ParkedChannelCount);
provider.SweepGraveyard(force: true);
Assert.Equal(0, provider.ParkedChannelCount);
}
[Fact]
public void UpdateSite_UnchangedEndpoints_ParksNothing()
{
using var provider = Build();
provider.UpdateSite("site-1", "http://localhost:15121", "http://localhost:15122");
provider.UpdateSite("site-1", "http://localhost:15121", "http://localhost:15122");
Assert.Equal(0, provider.ParkedChannelCount);
}
[Fact]
public void Dispose_DrainsParkedChannelsRegardlessOfRemainingGrace()
{
var provider = Build();
provider.UpdateSite("site-1", "http://localhost:15131", nodeBEndpoint: null);
provider.UpdateSite("site-1", "http://localhost:15132", nodeBEndpoint: null);
Assert.Equal(1, provider.ParkedChannelCount);
provider.Dispose();
Assert.Equal(0, provider.ParkedChannelCount);
}
private sealed class NoKeyProvider : ISitePskProvider
{
public ValueTask<string> GetAsync(string siteId, CancellationToken ct) => new("k");
public void Invalidate(string siteId) { }
}
}
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcClientFactoryDisposeTests.cs
+68
View File
@@ -66,6 +66,74 @@ public class SiteStreamGrpcClientFactoryDisposeTests
});
}
/// <summary>
/// Test client whose disposal always throws — stands in for a client whose
/// <c>ReleaseResources</c> hit an <see cref="ObjectDisposedException"/> on a subscription
/// token source (arch-review 2026-08-19 P3-1).
/// </summary>
private sealed class ThrowingClient : SiteStreamGrpcClient
{
public override void Dispose() => throw new ObjectDisposedException(nameof(CancellationTokenSource));
public override ValueTask DisposeAsync() =>
throw new ObjectDisposedException(nameof(CancellationTokenSource));
}
/// <summary>Hands out one throwing client followed by tracking clients.</summary>
private sealed class OneThrowingClientFactory : SiteStreamGrpcClientFactory
{
private readonly ConcurrentBag<TrackingClient> _created = new();
private int _next;
public OneThrowingClientFactory() : base(NullLoggerFactory.Instance) { }
public IReadOnlyCollection<TrackingClient> Created => _created.ToList();
protected override SiteStreamGrpcClient CreateClient(string siteIdentifier, string grpcEndpoint)
{
if (Interlocked.Increment(ref _next) == 1)
{
return new ThrowingClient();
}
var client = new TrackingClient();
_created.Add(client);
return client;
}
}
[Fact]
public async Task DisposeAsync_OneClientThrowing_StillDisposesEveryOtherClient()
{
// The disposal loop used to abort on the first throwing client, leaving every LATER
// client — and its gRPC channel — undisposed for the life of the process.
var factory = new OneThrowingClientFactory();
factory.GetOrCreate("site-a", "http://localhost:5100"); // throwing
factory.GetOrCreate("site-b", "http://localhost:5200");
factory.GetOrCreate("site-c", "http://localhost:5300");
await factory.DisposeAsync();
Assert.Equal(2, factory.Created.Count);
Assert.All(factory.Created, c =>
Assert.True(c.AsyncDisposeCalled, "a throwing client must not strand its peers undisposed"));
}
[Fact]
public void Dispose_OneClientThrowing_StillDisposesEveryOtherClient()
{
var factory = new OneThrowingClientFactory();
factory.GetOrCreate("site-a", "http://localhost:5100"); // throwing
factory.GetOrCreate("site-b", "http://localhost:5200");
factory.GetOrCreate("site-c", "http://localhost:5300");
factory.Dispose();
Assert.Equal(2, factory.Created.Count);
Assert.All(factory.Created, c =>
Assert.True(c.SyncDisposeCalled, "a throwing client must not strand its peers undisposed"));
}
[Fact]
public void Dispose_DoesNotDeadlock_UnderSingleThreadedSynchronizationContext()
{
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcClientTests.cs
+71 -6
View File
@@ -677,26 +677,91 @@ public class SiteStreamGrpcClientTests
// --- Communication-003 regression tests ---
[Fact]
public void RegisterSubscription_ReusedCorrelationId_CancelsAndDisposesPriorCts()
public void RegisterSubscription_ReusedCorrelationId_CancelsPriorCts_WithoutDisposingIt()
{
// Two SubscribeAsync calls briefly sharing a correlation ID (reconnect race).
// Inserting the second must cancel + dispose the first so it does not leak.
// Inserting the second must CANCEL the first — and must NOT dispose it (arch-review
// 2026-08-19 P3-1, re-pinning the earlier cancel-AND-dispose contract): the prior
// handler's ConsumeStreamAsync is still running and still reads prior.Token, so
// disposing from the replacing path throws ObjectDisposedException inside that loop.
// The owning SubscribeAsync/SubscribeSiteAsync disposes it as its loop unwinds.
var client = SiteStreamGrpcClient.CreateForTesting();
var first = new CancellationTokenSource();
var second = new CancellationTokenSource();
using var first = new CancellationTokenSource();
using var second = new CancellationTokenSource();
client.RegisterSubscription("corr-shared", first);
client.RegisterSubscription("corr-shared", second);
Assert.True(first.IsCancellationRequested);
// Disposed CTS throws ObjectDisposedException when its token is touched.
Assert.Throws<ObjectDisposedException>(() => _ = first.Token);
// Still usable by the prior handler that owns it — no ObjectDisposedException.
Assert.True(first.Token.IsCancellationRequested);
// The second (live) CTS must remain intact.
Assert.False(second.IsCancellationRequested);
}
[Fact]
public void RegisterSubscription_ReplacingAnAlreadyDisposedPriorCts_DoesNotThrowIntoTheNewSubscription()
{
// The prior handler's loop already unwound and disposed its own CTS while a replacement
// subscription is registering. The unguarded prior.Cancel() threw ObjectDisposedException
// straight out of the NEW SubscribeAsync, which reads it as a fault of the CURRENT
// generation and burns a reconnect on a phantom failure.
var client = SiteStreamGrpcClient.CreateForTesting();
var prior = new CancellationTokenSource();
client.RegisterSubscription("corr-shared", prior);
prior.Dispose();
using var replacement = new CancellationTokenSource();
client.RegisterSubscription("corr-shared", replacement);
// The replacement owns the slot and is untouched.
Assert.False(replacement.IsCancellationRequested);
client.Unsubscribe("corr-shared");
Assert.True(replacement.IsCancellationRequested);
}
[Fact]
public void Unsubscribe_AfterClientDisposal_IsANoOp()
{
// Disposal cancels every live subscription and clears the map; a late Unsubscribe from
// the consuming actor's teardown must not throw (an ObjectDisposedException here used to
// escape into the caller's cleanup path).
var client = SiteStreamGrpcClient.CreateForTesting();
using var cts = new CancellationTokenSource();
client.AddSubscriptionForTesting("corr-1", cts);
client.Dispose();
Assert.True(cts.IsCancellationRequested);
client.Unsubscribe("corr-1");
client.Unsubscribe("never-registered");
}
[Fact]
public void Dispose_WithAnAlreadyDisposedSubscriptionCts_StillCancelsTheRest()
{
// ReleaseResources iterated the live CTSes unguarded, so one already-disposed entry
// aborted the whole loop — leaving later subscriptions uncancelled AND (via the
// exception escaping into SiteStreamGrpcClientFactory's disposal loop) later clients'
// channels undisposed.
var client = SiteStreamGrpcClient.CreateForTesting();
var disposedEarly = new CancellationTokenSource();
using var live = new CancellationTokenSource();
client.AddSubscriptionForTesting("corr-disposed", disposedEarly);
client.AddSubscriptionForTesting("corr-live", live);
disposedEarly.Dispose();
// Deterministic regardless of dictionary iteration order: the disposed entry is always
// visited, so the unguarded loop always threw out of Dispose().
Assert.Null(Record.Exception(client.Dispose));
Assert.True(live.IsCancellationRequested);
}
[Fact]
public void RemoveSubscription_OnlyRemovesOwnCts_NotAReplacement()
{
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/PendingDeploymentPurgeActorTests.cs
+19 -6
View File
@@ -87,12 +87,25 @@ public class PendingDeploymentPurgeActorTests : TestKit
// Fast interval so the self-scheduling timer fires repeatedly; the actor must
// swallow each per-tick exception and keep ticking (best-effort hygiene — a
// crash here would drop the singleton until failover).
CreateActor(repo, TimeSpan.FromMilliseconds(100));
var actor = CreateActor(repo, TimeSpan.FromMilliseconds(100));
AwaitAssert(
() => Assert.True(Volatile.Read(ref calls) >= 2,
$"expected >= 2 ticks despite failures, got {Volatile.Read(ref calls)}"),
duration: TimeSpan.FromSeconds(5),
interval: TimeSpan.FromMilliseconds(50));
try
{
AwaitAssert(
() => Assert.True(Volatile.Read(ref calls) >= 2,
$"expected >= 2 ticks despite failures, got {Volatile.Read(ref calls)}"),
duration: TimeSpan.FromSeconds(5),
interval: TimeSpan.FromMilliseconds(50));
}
finally
{
// Row-95/row-82 pattern: a 100ms self-scheduling ticker left running strands
// later tests in the same assembly (it keeps firing — and keeps calling the
// now-disposed repo/service-provider — for the rest of the process). Stop it
// deterministically before the test ends.
Watch(actor);
Sys.Stop(actor);
ExpectTerminated(actor, TimeSpan.FromSeconds(3));
}
}
}
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteAlarmLiveCacheTelemetryTests.cs
+12
View File
@@ -11,6 +11,18 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
/// relative to a baseline so the process-wide static instruments are robust to any
/// parallel test interleaving.
/// </summary>
/// <remarks>
/// Register row 102's shape: a fixed-baseline delta only survives contributors that hold
/// constant or increase — a parallel class's actors moving the SAME process-global gauge
/// and then STOPPING after this test captures its baseline makes <c>baseline + N</c>
/// unreachable forever. <see cref="SiteAlarmAggregatorActor"/> calls
/// <see cref="ScadaBridgeTelemetry.LiveAlarmAggregatorStarted"/>/
/// <see cref="ScadaBridgeTelemetry.LiveAlarmAggregatorStopped"/> from its own
/// PreStart/PostStop, so every actor <c>Grpc.SiteAlarmAggregatorActorTests</c> spins up
/// moves this exact gauge — serialized into the same collection (row-96 pattern) so the
/// two classes never interleave.
/// </remarks>
[Collection("SiteAlarmAggregatorGauge")]
public class SiteAlarmLiveCacheTelemetryTests
{
private static long ReadGauge(string instrumentName)
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/ZB.MOM.WW.ScadaBridge.Communication.Tests.csproj
+4
View File
@@ -22,6 +22,10 @@
<Using Include="Xunit" />
</ItemGroup>
<ItemGroup>
<Content Include="xunit.runner.json" CopyToOutputDirectory="PreserveNewest" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.Communication/ZB.MOM.WW.ScadaBridge.Communication.csproj" />
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.Commons/ZB.MOM.WW.ScadaBridge.Commons.csproj" />
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/xunit.runner.json
+4
View File
@@ -0,0 +1,4 @@
{
"$schema": "https://xunit.net/schema/current/xunit.runner.schema.json",
"maxParallelThreads": 4
}
tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Actors/DataConnectionOpcUaFaultClassificationTests.cs
+109
View File
@@ -0,0 +1,109 @@
using Akka.Actor;
using Akka.TestKit.Xunit2;
using NSubstitute;
using Opc.Ua;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Actors;
using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests.Actors;
/// <summary>
/// Arch-review 2026-08-19 <b>P2-4</b> (classification half) — how a thrown
/// <see cref="ServiceResultException"/> out of a batch subscribe is classified.
///
/// <para>
/// <b>The gap these pin.</b> <c>IsConnectionLevelFailure</c> knew only about
/// <c>InvalidOperationException</c> / socket / timeout / IO faults, so the OPC UA SDK's own way of
/// reporting a dead session — a <c>ServiceResultException</c> carrying a bad-session status —
/// was treated as a per-tag resolution failure for the WHOLE chunk. Every tag went unresolved and
/// the backoff probe re-issued the same batch against the same dead session forever, growing the
/// adapter's monitored-item bookkeeping each round, instead of reconnecting once.
/// </para>
///
/// <para>
/// The classification must stay narrow: a per-node status (<c>BadNodeIdUnknown</c>) is exactly the
/// "device still booting / wrong tag path" case the resolution timer exists for, and must NOT put
/// the connection into a reconnect loop.
/// </para>
/// </summary>
public class DataConnectionOpcUaFaultClassificationTests : TestKit
{
private readonly ISiteHealthCollector _health = Substitute.For<ISiteHealthCollector>();
private readonly IDataConnectionFactory _factory = Substitute.For<IDataConnectionFactory>();
private static DataConnectionOptions Options() => new()
{
ReconnectInterval = TimeSpan.FromMilliseconds(100),
TagResolutionRetryInterval = TimeSpan.FromMilliseconds(150),
TagResolutionRetryMaxInterval = TimeSpan.FromMilliseconds(600),
WriteTimeout = TimeSpan.FromSeconds(5),
SeedReadMaxAttempts = 1,
SeedReadRetryDelay = TimeSpan.FromMilliseconds(10),
SubscribeBatchSize = 5,
SubscribeBatchDelay = TimeSpan.FromMilliseconds(50),
SeedReadBatchSize = 4,
SeedReadMaxParallelism = 2,
SeedOverallTimeout = TimeSpan.FromSeconds(30),
QualityFlushInterval = TimeSpan.FromMilliseconds(200)
};
private IActorRef CreateActor(FakeBatchDataConnection adapter, string name) =>
Sys.ActorOf(Props.Create(() => new DataConnectionActor(
name, adapter, Options(), _health, _factory, "OpcUa")), name);
private int HealthTransitions(string connectionName, ConnectionHealth health) =>
_health.ReceivedCalls().Count(c =>
c.GetMethodInfo().Name == nameof(ISiteHealthCollector.UpdateConnectionHealth) &&
c.GetArguments() is [string name, ConnectionHealth reported, ..] &&
name == connectionName && reported == health);
/// <summary>
/// A bad-session fault is a CONNECTION fault: the request fails and the actor enters
/// Reconnecting (which pushes bad quality for the connection).
///
/// <para><b>Fails before the fix:</b> the chunk was treated per-tag, so the request was acked
/// Success and the connection stayed nominally up on a dead session.</para>
/// </summary>
[Fact]
public void BadSessionServiceResultException_IsConnectionLevel_AndDrivesReconnect()
{
const string name = "opcua-bad-session";
var adapter = new FakeBatchDataConnection
{
BatchSubscribeThrows = () =>
new ServiceResultException(StatusCodes.BadSessionIdInvalid, "session is no longer valid")
};
var actor = CreateActor(adapter, name);
actor.Tell(new SubscribeTagsRequest("c1", "inst1", name, ["tag1", "tag2"], DateTimeOffset.UtcNow));
ExpectMsg<SubscribeTagsResponse>(m => !m.Success && m.ErrorMessage != null, TimeSpan.FromSeconds(5));
ExpectMsg<ConnectionQualityChanged>(q => q.Quality == QualityCode.Bad, TimeSpan.FromSeconds(5));
}
/// <summary>
/// The narrowness control. <c>BadNodeIdUnknown</c> is a per-node fault: the tags go unresolved
/// (bad quality, retried on the backoff timer) and the connection is NOT torn down.
/// </summary>
[Fact]
public void BadNodeIdServiceResultException_StaysPerTag_AndDoesNotTearDownTheConnection()
{
const string name = "opcua-bad-nodeid";
var adapter = new FakeBatchDataConnection
{
BatchSubscribeThrows = () =>
new ServiceResultException(StatusCodes.BadNodeIdUnknown, "no such node")
};
var actor = CreateActor(adapter, name);
actor.Tell(new SubscribeTagsRequest("c1", "inst1", name, ["tag1", "tag2"], DateTimeOffset.UtcNow));
// Per-tag treatment: bad quality per tag, and the request itself is still acked Success
// (the tags are registered and retried, exactly as an unresolved tag path is).
FishForMessage<SubscribeTagsResponse>(m => m.Success, TimeSpan.FromSeconds(5));
Assert.Equal(0, HealthTransitions(name, ConnectionHealth.Disconnected));
}
}
tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Actors/DataConnectionStaleConnectResultTests.cs
+211
View File
@@ -0,0 +1,211 @@
using Akka.Actor;
using Akka.TestKit.Xunit2;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Actors;
using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests.Actors;
/// <summary>
/// Arch-review 2026-08-19 <b>P2-3</b> — the connect leg of the stale-completion family that
/// <c>DataConnectionStaleCompletionTests</c> pins for the subscribe leg.
///
/// <para>
/// <b>The gap these pin.</b> <c>ConnectResult</c> was piped untagged, so a connect launched
/// against adapter generation N and answered after the adapter had been swapped (failover, or the
/// row-97 in-place reconfigure) was applied to its REPLACEMENT. A stale success drove
/// <c>ReSubscribeAll</c> and <c>BecomeConnected</c> against an adapter that had never connected;
/// the replacement's own result then landed in the Connected state, where nothing handles it. With
/// a bad new endpoint the batch subscribes failed at connection level with a log line only, no
/// <c>AdapterDisconnected</c> ever fired, and the idempotent-reconfigure guard refused a re-send
/// of the same configuration — only a restart recovered the connection. A stale FAILURE was just
/// as wrong: it counted toward failover and re-armed the reconnect timer underneath the attempt
/// that was genuinely in flight.
/// </para>
///
/// <para>
/// The state machine must also stay able to finish the CURRENT attempt after a stale result was
/// dropped — Connecting/Reconnecting is held until the matching-generation result arrives.
/// </para>
/// </summary>
[Collection("DataConnectionManagerActor")]
public class DataConnectionStaleConnectResultTests : TestKit
{
private const string FirstEndpoint = "opc.tcp://first-plc:4840";
private const string MidEndpoint = "opc.tcp://mid-plc:4840";
private const string LastEndpoint = "opc.tcp://last-plc:4840";
private const string Tag = "sensor/temp";
private readonly ISiteHealthCollector _health = Substitute.For<ISiteHealthCollector>();
private readonly IDataConnectionFactory _factory = Substitute.For<IDataConnectionFactory>();
private readonly DataConnectionOptions _options = new()
{
ReconnectInterval = TimeSpan.FromMilliseconds(100),
TagResolutionRetryInterval = TimeSpan.FromMilliseconds(200),
WriteTimeout = TimeSpan.FromSeconds(5),
SeedReadMaxAttempts = 1,
SeedReadRetryDelay = TimeSpan.FromMilliseconds(10)
};
private static Dictionary<string, string> Config(string endpoint) =>
new() { ["Endpoint"] = endpoint };
private IActorRef CreateConnectionActor(string name, IDataConnection adapter) =>
Sys.ActorOf(Props.Create(() =>
new DataConnectionActor(
name, adapter, _options, _health, _factory, "OpcUa",
Config(FirstEndpoint), null, 3)), name);
private void ExpectAdapterFor(string endpoint, IDataConnection adapter) =>
_factory.Create("OpcUa", Arg.Is<IDictionary<string, string>>(d => d["Endpoint"] == endpoint))
.Returns(adapter);
/// <summary>An adapter whose connect completes when <paramref name="connect"/> does.</summary>
private static IDataConnection Adapter(Task connect, string subscriptionId)
{
var adapter = Substitute.For<IDataConnection>();
adapter.ConnectAsync(Arg.Any<IDictionary<string, string>>(), Arg.Any<CancellationToken>())
.Returns(connect);
adapter.Status.Returns(ConnectionHealth.Connected);
adapter.SubscribeAsync(Arg.Any<string>(), Arg.Any<SubscriptionCallback>(), Arg.Any<CancellationToken>())
.Returns(subscriptionId);
adapter.ReadAsync(Arg.Any<string>(), Arg.Any<CancellationToken>())
.Returns(new ReadResult(false, null, null));
return adapter;
}
private int HealthTransitions(string connectionName, ConnectionHealth health) =>
_health.ReceivedCalls().Count(c =>
c.GetMethodInfo().Name == nameof(ISiteHealthCollector.UpdateConnectionHealth) &&
c.GetArguments() is [string name, ConnectionHealth reported, ..] &&
name == connectionName && reported == health);
private static int CallCount(IDataConnection adapter, string method) =>
adapter.ReceivedCalls().Count(c => c.GetMethodInfo().Name == method);
/// <summary>
/// Drives the actor to: connected on the first adapter with one live tag subscription, then
/// reconfigured onto <paramref name="mid"/> (whose connect is still in flight), then
/// reconfigured again onto <paramref name="last"/> (whose connect is also still in flight).
/// The mid adapter's pending connect is now two generations stale.
/// </summary>
private IActorRef ArrangeTwoGenerationsBehind(
string name, IDataConnection first, IDataConnection mid, IDataConnection last)
{
ExpectAdapterFor(MidEndpoint, mid);
ExpectAdapterFor(LastEndpoint, last);
var actor = CreateConnectionActor(name, first);
AwaitCondition(() => HealthTransitions(name, ConnectionHealth.Connected) >= 1,
TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(10));
var probe = CreateTestProbe();
probe.Send(actor, new SubscribeTagsRequest("c1", "inst1", name, [Tag], DateTimeOffset.UtcNow));
probe.FishForMessage<SubscribeTagsResponse>(m => m.Success, TimeSpan.FromSeconds(5));
actor.Tell(new ReconfigureConnectionCommand(name, "OpcUa", Config(MidEndpoint), null, 3));
AwaitCondition(() => CallCount(mid, nameof(IDataConnection.ConnectAsync)) >= 1,
TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(10));
actor.Tell(new ReconfigureConnectionCommand(name, "OpcUa", Config(LastEndpoint), null, 3));
AwaitCondition(() => CallCount(last, nameof(IDataConnection.ConnectAsync)) >= 1,
TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(10));
return actor;
}
/// <summary>
/// The headline case: a success from the adapter the operator has just corrected AWAY from
/// must not declare the connection up, and must not re-issue the subscription registry
/// against the replacement adapter that has not connected yet.
///
/// <para><b>Fails before the fix:</b> the actor entered Connected and re-subscribed
/// <c>Tag</c> on the never-connected last adapter.</para>
/// </summary>
[Fact]
public void StaleConnectSuccess_DoesNotConnect_AndDoesNotResubscribeAgainstTheNewAdapter()
{
const string name = "stale-connect-success";
var midGate = new TaskCompletionSource();
var lastGate = new TaskCompletionSource();
var first = Adapter(Task.CompletedTask, "sub-first");
var mid = Adapter(midGate.Task, "sub-mid");
var last = Adapter(lastGate.Task, "sub-last");
ArrangeTwoGenerationsBehind(name, first, mid, last);
// The two-generations-old connect finally answers — successfully.
midGate.SetResult();
// Settle: nothing here subscribes the default test actor, so this is purely a wait long
// enough for the piped result to have been received and dropped.
ExpectNoMsg(TimeSpan.FromMilliseconds(500));
Assert.Equal(1, HealthTransitions(name, ConnectionHealth.Connected));
Assert.Equal(0, CallCount(last, nameof(IDataConnection.SubscribeAsync)));
}
/// <summary>
/// A stale FAILURE must not touch the attempt that is genuinely in flight. Acting on it
/// counted toward the failover budget and re-armed the reconnect timer, so a second
/// <c>ConnectAsync</c> was launched against an adapter already connecting.
///
/// <para><b>Fails before the fix:</b> the last adapter was asked to connect a second time
/// one <c>ReconnectInterval</c> after the stale failure landed.</para>
/// </summary>
[Fact]
public void StaleConnectFailure_DoesNotDisturbTheCurrentAttempt()
{
const string name = "stale-connect-failure";
var midGate = new TaskCompletionSource();
var lastGate = new TaskCompletionSource();
var first = Adapter(Task.CompletedTask, "sub-first");
var mid = Adapter(midGate.Task, "sub-mid");
var last = Adapter(lastGate.Task, "sub-last");
ArrangeTwoGenerationsBehind(name, first, mid, last);
midGate.SetException(new Exception("connection refused"));
// Comfortably longer than ReconnectInterval, so a re-armed timer would have fired.
ExpectNoMsg(TimeSpan.FromMilliseconds(600));
Assert.Equal(1, CallCount(last, nameof(IDataConnection.ConnectAsync)));
Assert.Equal(1, HealthTransitions(name, ConnectionHealth.Connected));
}
/// <summary>
/// Dropping a stale result must not wedge the state machine: the actor stays in Reconnecting
/// and the CURRENT generation's own result is still processed normally — connection up, and
/// the whole subscription registry transparently re-subscribed against the live adapter.
/// </summary>
[Fact]
public void CurrentGenerationConnectResult_IsStillProcessed_AfterAStaleOneWasDropped()
{
const string name = "stale-connect-then-live";
var midGate = new TaskCompletionSource();
var lastGate = new TaskCompletionSource();
var first = Adapter(Task.CompletedTask, "sub-first");
var mid = Adapter(midGate.Task, "sub-mid");
var last = Adapter(lastGate.Task, "sub-last");
ArrangeTwoGenerationsBehind(name, first, mid, last);
midGate.SetResult();
ExpectNoMsg(TimeSpan.FromMilliseconds(300));
Assert.Equal(1, HealthTransitions(name, ConnectionHealth.Connected));
// Now the live adapter answers.
lastGate.SetResult();
AwaitCondition(() => HealthTransitions(name, ConnectionHealth.Connected) >= 2,
TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(10));
AwaitCondition(() => CallCount(last, nameof(IDataConnection.SubscribeAsync)) >= 1,
TimeSpan.FromSeconds(5), TimeSpan.FromMilliseconds(10));
}
}
tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/RealOpcUaClientBatchRollbackTests.cs
+116
View File
@@ -0,0 +1,116 @@
using System.Collections;
using System.Collections.Concurrent;
using System.Reflection;
using NSubstitute;
using Opc.Ua;
using Opc.Ua.Client;
using ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters;
namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests.Adapters;
/// <summary>
/// Arch-review 2026-08-19 <b>P2-4</b> (rollback half) — a batch subscribe whose
/// <c>ApplyChanges</c> throws must leave NO residue behind.
///
/// <para>
/// <b>The gap this pins.</b> <c>CreateSubscriptionsCoreAsync</c> registers each monitored item as
/// it places it (<c>_callbacks</c>, <c>_itemShard</c>, the shard's item count and the SDK
/// subscription's own item list) and only issues <c>ApplyChanges</c> once the whole chunk is
/// placed. A throw there — a dead session being the common cause — propagated with every one of
/// those registrations still in place and nothing left holding a handle to release them. The actor
/// sees the throw as a whole-chunk fault and re-issues the SAME batch on its backoff timer, so
/// each probe grew the bookkeeping again: unbounded shard growth and phantom shard occupancy.
/// </para>
///
/// <para>
/// The seam is reached with a data shard whose SDK <see cref="Subscription"/> was never created,
/// which is precisely how the SDK reports "you have no usable subscription": placement and
/// <c>AddItem</c> succeed, <c>ApplyChangesAsync</c> throws. Private state is read by reflection
/// rather than widened for the test.
/// </para>
/// </summary>
[Trait("Category", "Unit")]
public class RealOpcUaClientBatchRollbackTests
{
private static object Field(object target, string name) =>
target.GetType().GetField(name, BindingFlags.Instance | BindingFlags.NonPublic)!.GetValue(target)!;
/// <summary>
/// Gives the client a mocked session plus ONE pre-placed (never created) data shard, so a
/// batch subscribe places its items and then throws out of ApplyChanges.
/// </summary>
private static Subscription ArrangeUncreatedShard(RealOpcUaClient client)
{
var session = Substitute.For<ISession>();
session.NamespaceUris.Returns(new NamespaceTable());
typeof(RealOpcUaClient)
.GetField("_session", BindingFlags.Instance | BindingFlags.NonPublic)!
.SetValue(client, session);
// The parameterless ctor is obsolete in the current SDK in favour of a telemetry-context
// overload; nothing here publishes telemetry, and the alternative would be to hand-build a
// context purely to satisfy the signature.
#pragma warning disable CS0618
var subscription = new Subscription { DisplayName = "uncreated-shard" };
#pragma warning restore CS0618
var shardType = typeof(RealOpcUaClient).GetNestedType("DataShard", BindingFlags.NonPublic)!;
var shard = Activator.CreateInstance(
shardType,
BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic,
null, [subscription], null)!;
((IList)Field(client, "_dataShards")).Add(shard);
return subscription;
}
private static int ShardItemCount(RealOpcUaClient client)
{
var shard = ((IList)Field(client, "_dataShards"))[0]!;
return (int)shard.GetType().GetProperty("ItemCount")!.GetValue(shard)!;
}
[Fact]
public async Task ApplyChangesThrow_RollsBackEveryPartiallyCreatedItem()
{
var client = new RealOpcUaClient();
var subscription = ArrangeUncreatedShard(client);
var fault = await Record.ExceptionAsync(() => client.CreateSubscriptionsAsync(
["i=2258", "i=2259", "i=2260"],
(_, _, _, _) => { }));
// Positive control: the batch really did fail at ApplyChanges, not before placement.
Assert.NotNull(fault);
// Nothing registered survives the throw — pre-call state, exactly.
Assert.Empty((ConcurrentDictionary<string, Action<string, object?, DateTime, uint>>)
Field(client, "_callbacks"));
Assert.Empty((ConcurrentDictionary<string, Subscription>)Field(client, "_itemShard"));
Assert.Empty((ConcurrentDictionary<string, MonitoredItem>)Field(client, "_monitoredItems"));
Assert.Equal(0, ShardItemCount(client));
Assert.Empty(subscription.MonitoredItems);
}
[Fact]
public async Task RepeatedFailedBatches_DoNotGrowTheAdapterBookkeeping()
{
// The actual operational damage: the actor re-issues the same chunk on its backoff timer,
// so an un-rolled-back batch compounded on every probe.
var client = new RealOpcUaClient();
ArrangeUncreatedShard(client);
for (var attempt = 0; attempt < 3; attempt++)
{
var fault = await Record.ExceptionAsync(() => client.CreateSubscriptionsAsync(
["i=2258", "i=2259"],
(_, _, _, _) => { }));
Assert.NotNull(fault);
}
Assert.Empty((ConcurrentDictionary<string, Action<string, object?, DateTime, uint>>)
Field(client, "_callbacks"));
Assert.Empty((ConcurrentDictionary<string, Subscription>)Field(client, "_itemShard"));
Assert.Equal(0, ShardItemCount(client));
}
}
tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/DeploySiteAsyncTests.cs
+152 -4
View File
@@ -45,7 +45,9 @@ public class DeploySiteAsyncTests : TestKit
}
/// <summary>Seeds <paramref name="count"/> deployable instances at the test site.</summary>
private List<Instance> ArrangeInstances(int count)
/// <param name="count">How many instances to seed.</param>
/// <param name="disabledIds">Ids seeded <c>Disabled</c> instead of <c>NotDeployed</c>.</param>
private List<Instance> ArrangeInstances(int count, params int[] disabledIds)
{
var instances = new List<Instance>();
for (var i = 1; i <= count; i++)
@@ -55,7 +57,7 @@ public class DeploySiteAsyncTests : TestKit
Id = i,
SiteId = SiteId,
TemplateId = 10,
State = InstanceState.NotDeployed
State = disabledIds.Contains(i) ? InstanceState.Disabled : InstanceState.NotDeployed
};
instances.Add(instance);
@@ -73,7 +75,11 @@ public class DeploySiteAsyncTests : TestKit
return instances;
}
private DeploymentService CreateService(IActorRef commActor, int maxParallelism, TimeSpan? perInstanceTimeout = null)
private DeploymentService CreateService(
IActorRef commActor,
int maxParallelism,
TimeSpan? perInstanceTimeout = null,
TimeSpan? reconcileTimeout = null)
{
var comms = new CommunicationService(
Options.Create(new CommunicationOptions { DeploymentTimeout = TimeSpan.FromSeconds(30) }),
@@ -84,7 +90,8 @@ public class DeploySiteAsyncTests : TestKit
{
OperationLockTimeout = TimeSpan.FromSeconds(5),
SiteDeploymentMaxParallelism = maxParallelism,
SiteDeploymentTimeoutPerInstance = perInstanceTimeout ?? TimeSpan.FromSeconds(30)
SiteDeploymentTimeoutPerInstance = perInstanceTimeout ?? TimeSpan.FromSeconds(30),
SiteDeploymentReconcileTimeout = reconcileTimeout ?? TimeSpan.FromSeconds(5)
});
return new DeploymentService(
@@ -305,6 +312,147 @@ public class DeploySiteAsyncTests : TestKit
Arg.Any<CancellationToken>());
}
/// <summary>
/// P7-2: a bulk site deploy must not restart deliberately-stopped equipment.
///
/// <para>
/// Deploy is a legal transition out of <c>Disabled</c> and the deploy path
/// forces the instance to <c>Enabled</c> on success, so a site-wide push over an
/// unfiltered instance list silently re-enabled every instance an operator had
/// disabled. Bulk deploy therefore skips them outright — no flatten, no send, no
/// state change — and reports them so the operator can see what was left alone.
/// The single-instance path is unchanged: that is the explicit "yes, re-enable
/// this one" action.
/// </para>
/// </summary>
[Fact]
public async Task DeploySiteAsync_SkipsDisabledInstances_WithoutDeployingOrReEnablingThem()
{
var instances = ArrangeInstances(5, disabledIds: [2, 4]);
var commActor = Sys.ActorOf(Props.Create(() =>
new ThrottledSiteActor(new ConcurrencyTracker(), slowInstanceName: null, slowDelay: TimeSpan.Zero)));
var result = await CreateService(commActor, maxParallelism: 3).DeploySiteAsync(SiteId, "admin");
Assert.True(result.IsSuccess);
var summary = result.Value;
// Only the three Enabled/NotDeployed instances were attempted.
Assert.Equal(3, summary.InstanceResults.Count);
Assert.Equal(3, summary.SuccessCount);
Assert.Equal(0, summary.FailureCount);
Assert.DoesNotContain(summary.InstanceResults, r => r.UniqueName is "Inst-02" or "Inst-04");
// The Disabled pair is reported as skipped, with the reason the operator needs.
Assert.NotNull(summary.SkippedInstances);
var skipped = summary.SkippedInstances;
Assert.Equal(2, skipped.Count);
Assert.Equal(
new[] { "Inst-02", "Inst-04" },
skipped.Select(s => s.UniqueName).Order().ToArray());
Assert.All(skipped, s => Assert.Contains("deploy individually to re-enable", s.Reason));
// ...and left completely untouched: still Disabled, and never handed to the
// repository at all (no deployment record, no instance update, no snapshot).
foreach (var id in new[] { 2, 4 })
{
Assert.Equal(InstanceState.Disabled, instances.Single(i => i.Id == id).State);
await _pipeline.DidNotReceive().FlattenAndValidateAsync(
id, Arg.Any<CancellationToken>(), Arg.Any<bool>(), Arg.Any<FlattenSession?>());
}
await _repo.DidNotReceive().UpdateInstanceAsync(
Arg.Is<Instance>(i => i.Id == 2 || i.Id == 4), Arg.Any<CancellationToken>());
await _repo.DidNotReceive().AddDeploymentRecordAsync(
Arg.Is<DeploymentRecord>(r => r.InstanceId == 2 || r.InstanceId == 4), Arg.Any<CancellationToken>());
}
/// <summary>
/// P7-3: phase 1 is serial, so an unreachable site used to cost the batch one
/// full 30-second <c>QueryTimeout</c> PER INSTANCE before a single deploy was
/// sent — hours for a large site, so the CLI's 5-minute bulk deadline cancelled
/// and the next attempt re-armed the same cost. Two bounds close it: a short
/// per-reconcile deadline, and a once-per-batch circuit-breaker that skips the
/// remaining reconciles after the first timeout.
///
/// <para>
/// The site here answers the deploy but NEVER answers the reconcile query, so
/// the assertion that exactly ONE query was issued for six instances is the
/// whole finding: without the breaker there would be six.
/// </para>
/// </summary>
[Fact]
public async Task DeploySiteAsync_UnreachableSite_ReconcilesOnceThenSkipsTheRestOfTheBatch()
{
const int instanceCount = 6;
ArrangeInstances(instanceCount);
// Reconciliation only runs when a prior record is stuck InProgress.
for (var i = 1; i <= instanceCount; i++)
{
_repo.GetCurrentDeploymentStatusAsync(i, Arg.Any<CancellationToken>())
.Returns(new DeploymentRecord($"prior-{i}", "admin")
{
InstanceId = i,
Status = DeploymentStatus.InProgress,
RevisionHash = "sha256:old"
});
}
var queryCount = 0;
var commActor = Sys.ActorOf(Props.Create(() =>
new SilentQuerySiteActor(() => Interlocked.Increment(ref queryCount))));
var reconcileTimeout = TimeSpan.FromMilliseconds(300);
var service = CreateService(commActor, maxParallelism: 3, reconcileTimeout: reconcileTimeout);
var sw = Stopwatch.StartNew();
var result = await service.DeploySiteAsync(SiteId, "admin");
sw.Stop();
Assert.True(result.IsSuccess);
// The circuit-breaker opened after the first unanswered reconcile.
Assert.Equal(1, Volatile.Read(ref queryCount));
// Prepare therefore cost ONE reconcile deadline, not six.
Assert.True(sw.Elapsed < TimeSpan.FromSeconds(2),
$"bulk prepare took {sw.ElapsedMilliseconds} ms — the per-batch reconcile bound is not holding");
// A skipped reconcile is not a failed deploy: every instance still went out.
Assert.Equal(instanceCount, result.Value.SuccessCount);
Assert.Equal(0, result.Value.FailureCount);
}
/// <summary>
/// Answers <c>RefreshDeploymentCommand</c> normally but NEVER answers a
/// <c>DeploymentStateQueryRequest</c>, counting how many arrived — a site that
/// is up for deploys yet unreachable for the reconcile query, which is the shape
/// a wedged or overloaded site presents.
/// </summary>
private sealed class SilentQuerySiteActor : ReceiveActor
{
public SilentQuerySiteActor(Action onQuery)
{
Receive<SiteEnvelope>(env =>
{
switch (env.Message)
{
case DeploymentStateQueryRequest:
onQuery();
return;
case RefreshDeploymentCommand cmd:
Sender.Tell(new DeploymentStatusResponse(
cmd.DeploymentId, cmd.InstanceUniqueName,
DeploymentStatus.Success, null, DateTimeOffset.UtcNow));
return;
}
});
}
}
/// <summary>
/// Answers every <c>RefreshDeploymentCommand</c> after a fixed delay, recording
/// how old each command's staging <c>Timestamp</c> already was on arrival.
tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/DeploymentManagerOptionsValidatorTests.cs
+19
View File
@@ -53,4 +53,23 @@ public class DeploymentManagerOptionsValidatorTests
Assert.True(result.Failed);
Assert.Contains("OperationLockTimeout", result.FailureMessage);
}
[Fact]
public void ZeroSiteDeploymentReconcileTimeout_IsRejected()
{
var result = Validate(new DeploymentManagerOptions { SiteDeploymentReconcileTimeout = TimeSpan.Zero });
Assert.True(result.Failed);
Assert.Contains("SiteDeploymentReconcileTimeout", result.FailureMessage);
}
[Fact]
public void NegativeSiteDeploymentReconcileTimeout_IsRejected()
{
var result = Validate(new DeploymentManagerOptions
{
SiteDeploymentReconcileTimeout = TimeSpan.FromSeconds(-1)
});
Assert.True(result.Failed);
Assert.Contains("SiteDeploymentReconcileTimeout", result.FailureMessage);
}
}
tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/DeploymentServiceTests.cs
+30
View File
@@ -1597,6 +1597,36 @@ public class DeploymentServiceTests : TestKit
Assert.Equal(InstanceState.Disabled, instance.State);
}
/// <summary>
/// The counterpart to the bulk path's Disabled skip (arch-review P7-2): a
/// SINGLE-instance deploy of a Disabled instance still deploys AND enables it.
/// That is the documented "Deploy on a Disabled instance also enables it"
/// transition and the explicit operator action bulk deploy now refuses to
/// perform implicitly, so it must keep working exactly as before.
/// </summary>
[Fact]
public async Task DeployInstanceAsync_DisabledInstance_StillDeploysAndEnables()
{
var instance = new Instance("SingleDisabled")
{
Id = 73, SiteId = 1, State = InstanceState.Disabled
};
_repo.GetInstanceByIdAsync(73, Arg.Any<CancellationToken>()).Returns(instance);
SetupValidPipeline(73, "SingleDisabled", "sha256:target");
_repo.GetCurrentDeploymentStatusAsync(73, Arg.Any<CancellationToken>())
.Returns((DeploymentRecord?)null);
var counters = new ReconcileProbeCounters();
var commActor = Sys.ActorOf(Props.Create(() =>
new ReconcileProbeActor(counters, siteHash: "sha256:target", failQuery: false)));
var result = await CreateServiceWithCommActor(commActor).DeployInstanceAsync(73, "admin");
Assert.True(result.IsSuccess);
Assert.Equal(1, counters.DeployCount);
Assert.Equal(InstanceState.Enabled, instance.State);
}
// ── DeploymentManager-016: reconciled record must carry the target revision hash ──
[Fact]
tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/StaleInstanceProbeTests.cs
+69
View File
@@ -1,4 +1,5 @@
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates;
using ZB.MOM.WW.ScadaBridge.Commons.Types;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Templates;
@@ -96,4 +97,72 @@ public class StaleInstanceProbeTests
Assert.Equal("sha256:second", await probe.GetCurrentRevisionHashAsync(InstanceId));
}
/// <summary>
/// P7-4: a non-structural template edit landing DURING the flatten must not be
/// captured as if it were already included.
///
/// <para>
/// The memo pairs a revision hash with the watermark readings that produced it.
/// Reading the per-template versions AFTER the flatten pairs the PRE-edit hash
/// with the POST-edit version, and every later probe then finds the watermark
/// unchanged and answers "not stale" forever — until an unrelated edit, a
/// restart, or the 4096-entry overflow clears it. The staleness sweep and
/// <c>ImportResult.StaleInstanceIds</c> both read through this memo.
/// </para>
///
/// <para>
/// The fix is that the versions come from the session, which records each
/// template's version at the instant it READ that template — before the edit —
/// so the mismatch on the next probe invalidates the memo. Only the recording is
/// changed: the mid-flatten edit still produces the pre-edit hash on this pass,
/// which is correct, since that is the graph the flatten actually saw.
/// </para>
/// </summary>
[Fact]
public async Task TemplateEditedDuringFlatten_InvalidatesTheMemo_RatherThanPinningTheStaleHash()
{
const int TemplateId = 42;
// The flatten reads the template through the session (recording the version
// it read it at), and the edit lands before the flatten returns.
_pipeline
.FlattenAndValidateAsync(InstanceId, Arg.Any<CancellationToken>(), Arg.Any<bool>(), Arg.Any<FlattenSession?>())
.Returns(async call =>
{
_flattenCount++;
var session = call.ArgAt<FlattenSession?>(3)!;
await session.GetTemplateAsync(TemplateId, _ => Task.FromResult<Template?>(new Template($"T{TemplateId}")));
// A non-structural template edit commits while the flatten is in flight.
_watermark.BumpTemplate(TemplateId);
_currentHash = "sha256:after-edit";
var config = new FlattenedConfiguration { InstanceUniqueName = "Inst-01" };
return Result<FlatteningPipelineResult>.Success(
new FlatteningPipelineResult(config, "sha256:before-edit", ValidationResult.Success()));
});
var probe = CreateProbe();
// Pass 1 legitimately returns the hash of the graph the flatten walked.
Assert.Equal("sha256:before-edit", await probe.GetCurrentRevisionHashAsync(InstanceId));
Assert.Equal(1, _flattenCount);
// Pass 2 must see the edit. Re-recording the versions after the flatten
// would have memoised the post-edit version against the pre-edit hash, and
// this call would still answer "sha256:before-edit" without re-flattening.
_pipeline
.FlattenAndValidateAsync(InstanceId, Arg.Any<CancellationToken>(), Arg.Any<bool>(), Arg.Any<FlattenSession?>())
.Returns(_ =>
{
_flattenCount++;
var config = new FlattenedConfiguration { InstanceUniqueName = "Inst-01" };
return Result<FlatteningPipelineResult>.Success(
new FlatteningPipelineResult(config, _currentHash, ValidationResult.Success()));
});
Assert.Equal("sha256:after-edit", await probe.GetCurrentRevisionHashAsync(InstanceId));
Assert.Equal(2, _flattenCount);
}
}
tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/HealthReportSenderTests.cs
+16 -1
View File
@@ -333,6 +333,16 @@ public class HealthReportSenderTests
/// un-sent report only, and the next successful report shipped with the
/// counters at zero.
/// </summary>
/// <remarks>
/// Extended for arch-review 2026-08-19 P8-1: <c>ScriptRunShedCount</c> is drained
/// by <c>CollectReport</c> the exact same way as the five counters above, but
/// <c>AddIntervalCounters</c> used to take only those five — a failed send
/// destructively lost the shed count forever. <c>DetachedScriptThreads</c> and
/// <c>SiteStreamAlarmDropCount</c> are deliberately NOT exercised here: they are
/// point-in-time gauges (set via <c>Interlocked.CompareExchange</c>/<c>Read</c>,
/// never zeroed by <c>CollectReport</c>), so a failed send loses nothing for them
/// to restore.
/// </remarks>
[Fact]
public async Task SendFailure_PreservesIntervalCountersForNextReport()
{
@@ -341,7 +351,7 @@ public class HealthReportSenderTests
collector.SetActiveNode(true);
// Pre-populate every per-interval counter so the restore path on each
// field is exercised — script error, alarm error, dead letter, site
// audit write failure, audit redaction failure.
// audit write failure, audit redaction failure, script run shed.
collector.IncrementScriptError();
collector.IncrementScriptError();
collector.IncrementAlarmError();
@@ -351,6 +361,10 @@ public class HealthReportSenderTests
collector.IncrementSiteAuditWriteFailures();
collector.IncrementAuditRedactionFailure();
collector.IncrementAuditRedactionFailure();
collector.IncrementScriptRunShed();
collector.IncrementScriptRunShed();
collector.IncrementScriptRunShed();
collector.IncrementScriptRunShed();
var options = Options.Create(new HealthMonitoringOptions
{
@@ -383,6 +397,7 @@ public class HealthReportSenderTests
Assert.Equal(3, firstSuccess.DeadLetterCount);
Assert.Equal(1, firstSuccess.SiteAuditWriteFailures);
Assert.Equal(2, firstSuccess.AuditRedactionFailure);
Assert.Equal(4, firstSuccess.ScriptRunShedCount);
}
/// <summary>
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs
+4 -2
View File
@@ -475,8 +475,10 @@ public class SiteCompositionRootTests : IDisposable
new object[] { typeof(IEventLogQueryService) },
new object[] { typeof(ISiteIdentityProvider) },
new object[] { typeof(IHealthReportTransport) },
// M2.15 (#29): the active-node purge gate must be registered on site nodes
// so EventLogPurge only runs on the active node.
// M2.15 (#29): the active-node check must be registered on site nodes so
// EventLogPurge's log line can report whether this node is active (arch-review
// 2026-08-19 P4-4: it no longer gates which node purges — every node purges its
// own site_events file).
new object[] { typeof(SiteEventLogActiveNodeCheck) },
};
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteKestrelSetupTests.cs
+56
View File
@@ -0,0 +1,56 @@
using Microsoft.AspNetCore.Server.Kestrel.Core;
using ZB.MOM.WW.ScadaBridge.Host;
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// <summary>
/// The site node's Kestrel configuration, asserted against the code that actually ships (the other
/// Host startup tests re-declare their own <c>ConfigureKestrel</c> lambda, which can never catch a
/// regression in <c>Program.cs</c>'s).
/// </summary>
public class SiteKestrelSetupTests
{
[Fact]
public void SiteListener_EnablesHttp2Keepalive()
{
// Arch-review 2026-08-19 P4-3. Kestrel's default KeepAlivePingDelay is TimeSpan.MaxValue —
// no server-initiated pings — so a half-open HTTP/2 connection is never detected on a
// stream that is legitimately silent. The LocalDb sync stream is exactly that: with the
// dialling node crashed or partitioned, this node's handler stays parked on a dead stream
// holding the single-stream guard, refusing the reconnect that would heal the pair.
var options = new KestrelServerOptions();
SiteKestrelSetup.Configure(options, grpcPort: 8083, metricsPort: 8084);
Assert.NotEqual(TimeSpan.MaxValue, options.Limits.Http2.KeepAlivePingDelay);
Assert.Equal(SiteKestrelSetup.Http2KeepAlivePingDelay, options.Limits.Http2.KeepAlivePingDelay);
Assert.Equal(SiteKestrelSetup.Http2KeepAlivePingTimeout, options.Limits.Http2.KeepAlivePingTimeout);
// Detection has to be quick enough to matter, and a ping must not be its own timeout.
Assert.True(SiteKestrelSetup.Http2KeepAlivePingTimeout < SiteKestrelSetup.Http2KeepAlivePingDelay);
Assert.True(
SiteKestrelSetup.Http2KeepAlivePingDelay + SiteKestrelSetup.Http2KeepAlivePingTimeout
<= TimeSpan.FromMinutes(2));
}
[Fact]
public void SiteListener_KeepsGrpcHttp2Only_AndMetricsHttp1Capable()
{
// The keepalive change must not disturb the protocol split: h2c-only on the gRPC port so
// Grpc.AspNetCore can serve it without TLS ALPN, HTTP/1.1-capable on the metrics port so a
// standard Prometheus scraper can reach /metrics.
var options = new KestrelServerOptions();
SiteKestrelSetup.Configure(options, grpcPort: 8083, metricsPort: 8084);
var endpoints = options.GetType()
.GetProperty("CodeBackedListenOptions", System.Reflection.BindingFlags.Instance
| System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Public)!
.GetValue(options) as System.Collections.IEnumerable;
Assert.NotNull(endpoints);
var protocols = endpoints!.Cast<ListenOptions>().Select(o => o.Protocols).ToList();
Assert.Equal(2, protocols.Count);
Assert.Equal(HttpProtocols.Http2, protocols[0]);
Assert.Equal(HttpProtocols.Http1AndHttp2, protocols[1]);
}
}
tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/AuditEndpointsTests.cs
+166
View File
@@ -315,6 +315,172 @@ public class AuditEndpointsTests
}
}
// ─────────────────────────────────────────────────────────────────────
// CSV export — formula-injection neutralization + column alignment
// (arch-review 2026-08-19 P6-7)
// ─────────────────────────────────────────────────────────────────────
[Fact]
public void CsvHeader_Has24Columns_SameOrderAsAuditExportRowProperties()
{
// The JSONL export serializes AuditExportRow directly, so its field
// order IS the AuditExportRow property-declaration order; the CSV
// header must match it exactly (finding: the CSV was missing
// ExecutionId/ParentExecutionId/SourceNode that JSONL already carried).
var columns = AuditEndpoints.CsvHeader.Split(',');
Assert.Equal(24, columns.Length);
Assert.Equal(new[]
{
"EventId", "OccurredAtUtc", "IngestedAtUtc", "Channel", "Kind",
"CorrelationId", "ExecutionId", "ParentExecutionId",
"SourceSiteId", "SourceNode", "SourceInstanceId", "SourceScript",
"Actor", "Target", "Status", "HttpStatus", "DurationMs",
"ErrorMessage", "ErrorDetail", "RequestSummary", "ResponseSummary",
"PayloadTruncated", "Extra", "ForwardState",
}, columns);
}
[Fact]
public void FormatCsvRow_IncludesExecutionParentExecutionAndSourceNode()
{
var evt = ScadaBridgeAuditEventFactory.Create(
channel: AuditChannel.ApiOutbound,
kind: AuditKind.ApiCall,
status: AuditStatus.Delivered,
eventId: Guid.Parse("11111111-1111-1111-1111-111111111111"),
occurredAtUtc: new DateTime(2026, 5, 20, 12, 0, 0, DateTimeKind.Utc),
sourceNode: "site-a-node-a",
executionId: Guid.Parse("22222222-2222-2222-2222-222222222222"),
parentExecutionId: Guid.Parse("33333333-3333-3333-3333-333333333333"));
var row = AuditEndpoints.FormatCsvRow(AuditExportRow.From(evt));
var fields = row.Split(',');
Assert.Equal(24, fields.Length);
Assert.Equal("22222222-2222-2222-2222-222222222222", fields[6]); // ExecutionId
Assert.Equal("33333333-3333-3333-3333-333333333333", fields[7]); // ParentExecutionId
Assert.Equal("site-a-node-a", fields[9]); // SourceNode
}
[Theory]
[InlineData("=HYPERLINK(\"http://evil\",\"click\")", "'=HYPERLINK(")]
[InlineData("+cmd|'/c calc'!A1", "'+cmd|")]
[InlineData("-2+3", "'-2+3")]
[InlineData("@SUM(1,1)", "'@SUM(1,1)")]
[InlineData("\tsneaky", "'\tsneaky")]
public async Task Export_Csv_FreeTextLeadingFormulaCharacter_IsPrefixedWithQuote(string malicious, string expectedPrefix)
{
// CWE-1236: Target is inbound-API-caller-controlled free text. A raw
// leading =, +, -, @, or TAB must never reach the CSV cell unescaped.
var evt = ScadaBridgeAuditEventFactory.Create(
channel: AuditChannel.ApiInbound,
kind: AuditKind.InboundRequest,
status: AuditStatus.Delivered,
eventId: Guid.Parse("44444444-4444-4444-4444-444444444444"),
occurredAtUtc: new DateTime(2026, 5, 20, 12, 0, 0, DateTimeKind.Utc),
target: malicious);
var (client, _, host) = await BuildHostAsync(
roles: new[] { "Administrator" },
queryPages: new[]
{
(IReadOnlyList<AuditEvent>)new[] { evt },
(IReadOnlyList<AuditEvent>)Array.Empty<AuditEvent>(),
});
using (host)
{
var response = await client.SendAsync(Get("/api/audit/export?format=csv"));
var body = await response.Content.ReadAsStringAsync();
Assert.Contains(expectedPrefix, body);
// The raw, un-neutralized value must never appear on its own —
// every occurrence must be the neutralized ('-prefixed) form.
Assert.DoesNotContain("," + malicious + ",", body);
}
}
[Fact]
public void FormatCsvRow_ErrorMessageResponseSummaryAndExtra_AreNeutralized()
{
// Covers the finding's named secondary carriers beyond Target.
var evt = ScadaBridgeAuditEventFactory.Create(
channel: AuditChannel.DbOutbound,
kind: AuditKind.DbWrite,
status: AuditStatus.Failed,
eventId: Guid.Parse("55555555-5555-5555-5555-555555555555"),
occurredAtUtc: new DateTime(2026, 5, 20, 12, 0, 0, DateTimeKind.Utc),
errorMessage: "=cmd()",
responseSummary: "+1+1",
extra: "@evil");
var row = AuditEndpoints.FormatCsvRow(AuditExportRow.From(evt));
var fields = row.Split(',');
Assert.Equal("'=cmd()", fields[17]); // ErrorMessage
Assert.Equal("'+1+1", fields[20]); // ResponseSummary
Assert.Equal("'@evil", fields[22]); // Extra
}
[Fact]
public void FormatCsvRow_NegativeDurationAndDateColumns_NeverNeutralized()
{
// DurationMs/OccurredAtUtc are formatted by the exporter itself, never
// free text — a negative duration's leading '-' must render as-is, not
// be mistaken for a formula trigger and quote-prefixed.
var evt = ScadaBridgeAuditEventFactory.Create(
channel: AuditChannel.ApiOutbound,
kind: AuditKind.ApiCall,
status: AuditStatus.Delivered,
eventId: Guid.Parse("66666666-6666-6666-6666-666666666666"),
occurredAtUtc: new DateTime(2026, 5, 20, 12, 0, 0, DateTimeKind.Utc),
httpStatus: 200,
durationMs: -7);
var row = AuditEndpoints.FormatCsvRow(AuditExportRow.From(evt));
var fields = row.Split(',');
Assert.Equal("-7", fields[16]); // DurationMs, unprefixed
Assert.StartsWith("2026-05-20", fields[1]); // OccurredAtUtc, unprefixed
Assert.Equal("200", fields[15]); // HttpStatus, unprefixed
}
[Fact]
public void FormatCsvRow_BenignFreeTextField_IsNotPrefixed()
{
var evt = ScadaBridgeAuditEventFactory.Create(
channel: AuditChannel.ApiOutbound,
kind: AuditKind.ApiCall,
status: AuditStatus.Delivered,
eventId: Guid.Parse("77777777-7777-7777-7777-777777777777"),
occurredAtUtc: new DateTime(2026, 5, 20, 12, 0, 0, DateTimeKind.Utc),
target: "plant-a-external-system");
var row = AuditEndpoints.FormatCsvRow(AuditExportRow.From(evt));
var fields = row.Split(',');
Assert.Equal("plant-a-external-system", fields[13]); // Target
}
[Fact]
public void CsvHeader_ColumnsMatchAuditExportRowPropertyOrder()
{
// The JSONL export serializes AuditExportRow verbatim (property
// declaration order = JSON field order once camelCased); assert the
// CSV header names + order are identical to that property surface, so
// a column can never again silently drift between the two formats.
// (Reflection, not a live JSONL round-trip: JsonOptions drops null
// properties on write — e.g. ForwardState is always null on the
// central read path — which would otherwise hide a real column from
// a JSON-based comparison.)
var propertyNames = typeof(AuditExportRow)
.GetProperties()
.Select(p => p.Name)
.ToArray();
Assert.Equal(propertyNames, AuditEndpoints.CsvHeader.Split(','));
}
[Fact]
public async Task Export_Parquet_Returns501()
{
tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/DebugStreamHubSubscribeRaceTests.cs
+179
View File
@@ -0,0 +1,179 @@
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Http.Connections.Features;
using Microsoft.AspNetCore.Http.Features;
using Microsoft.AspNetCore.SignalR;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
namespace ZB.MOM.WW.ScadaBridge.ManagementService.Tests;
/// <summary>
/// Regression tests for arch-review 2026-08-19 P6-6 on the hub side. <c>SubscribeInstance</c>
/// awaits <c>StartStreamAsync</c> — up to 30s — and only then stores the session id on the
/// connection. A client that disconnects inside that window ran <c>OnDisconnectedAsync</c>
/// while there was still no session id to release, so the session stayed registered in the
/// singleton <see cref="DebugStreamService"/> and its keepalive renewed it forever (the
/// orphan reaper only collects UNregistered sessions). The fix publishes the id and
/// re-checks connection liveness under the per-connection <c>Items</c> monitor, so exactly
/// one side tears the session down.
/// </summary>
public class DebugStreamHubSubscribeRaceTests
{
private sealed class GatedDebugStreamService : DebugStreamService
{
private readonly TaskCompletionSource _gate = new(TaskCreationOptions.RunContinuationsAsynchronously);
private readonly TaskCompletionSource _entered = new(TaskCreationOptions.RunContinuationsAsynchronously);
private readonly List<string> _stopped = new();
public GatedDebugStreamService()
: base(
new CommunicationService(Options.Create(new CommunicationOptions()), NullLogger<CommunicationService>.Instance),
new ServiceCollection().BuildServiceProvider(),
new SiteStreamGrpcClientFactory(NullLoggerFactory.Instance),
NullLogger<DebugStreamService>.Instance)
{
}
public string SessionId => "session-under-test";
/// <summary>Completes once <see cref="StartStreamAsync"/> has been entered.</summary>
public Task Entered => _entered.Task;
/// <summary>Session ids passed to <see cref="StopStream"/>, in call order.</summary>
public IReadOnlyList<string> Stopped
{
get { lock (_stopped) { return _stopped.ToList(); } }
}
public void ReleaseStart() => _gate.TrySetResult();
public override async Task<DebugStreamSession> StartStreamAsync(
int instanceId, Action<object> onEvent, Action onTerminated, CancellationToken ct = default)
{
_entered.TrySetResult();
await _gate.Task;
return new DebugStreamSession(
SessionId,
new DebugViewSnapshot(
"Inst-1",
Array.Empty<AttributeValueChanged>(),
Array.Empty<AlarmStateChanged>(),
DateTimeOffset.UtcNow));
}
public override void StopStream(string sessionId)
{
lock (_stopped) { _stopped.Add(sessionId); }
}
}
private sealed class TestHubCallerContext : HubCallerContext
{
private readonly FeatureCollection _features = new();
private readonly CancellationTokenSource _aborted = new();
public TestHubCallerContext(HttpContext http) =>
_features.Set<IHttpContextFeature>(new TestHttpContextFeature { HttpContext = http });
public override string ConnectionId => "test-conn";
public override string? UserIdentifier => null;
public override System.Security.Claims.ClaimsPrincipal? User => null;
public override IDictionary<object, object?> Items { get; } = new Dictionary<object, object?>();
public override IFeatureCollection Features => _features;
public override CancellationToken ConnectionAborted => _aborted.Token;
public override void Abort() => _aborted.Cancel();
private sealed class TestHttpContextFeature : IHttpContextFeature { public HttpContext? HttpContext { get; set; } }
}
private const string SessionIdKey = "DebugStreamSessionId";
private static (DebugStreamHub Hub, TestHubCallerContext Context, GatedDebugStreamService Stream) CreateHub()
{
var repo = Substitute.For<ITemplateEngineRepository>();
repo.GetInstanceByIdAsync(7).Returns(new Instance("Inst-1") { Id = 7, SiteId = 1 });
var services = new ServiceCollection();
services.AddSingleton(repo);
var http = new DefaultHttpContext { RequestServices = services.BuildServiceProvider() };
var stream = new GatedDebugStreamService();
var hub = new DebugStreamHub(
stream, Substitute.For<IHubContext<DebugStreamHub>>(), NullLogger<DebugStreamHub>.Instance)
{
Clients = Substitute.For<IHubCallerClients>()
};
var context = new TestHubCallerContext(http);
hub.Context = context;
// No roles / no permitted sites on the connection == system-wide scope, so the
// authorization branch is not what these tests are exercising.
return (hub, context, stream);
}
[Fact]
public async Task SubscribeInstance_ClientDisconnectsMidStart_StopsTheJustCreatedSession()
{
var (hub, context, stream) = CreateHub();
var subscribe = hub.SubscribeInstance(7);
await stream.Entered.WaitAsync(TimeSpan.FromSeconds(5));
// The client goes away while the start is still in flight — the disconnect path
// finds nothing registered yet.
await hub.OnDisconnectedAsync(null);
Assert.Empty(stream.Stopped);
stream.ReleaseStart();
await subscribe.WaitAsync(TimeSpan.FromSeconds(10));
Assert.Equal(new[] { stream.SessionId }, stream.Stopped);
Assert.False(context.Items.ContainsKey(SessionIdKey));
}
[Fact]
public async Task SubscribeInstance_ConnectionAbortedMidStart_StopsTheJustCreatedSession()
{
// Same window, but the abort token trips without OnDisconnectedAsync having run yet
// (SignalR aborts the connection before dispatching the disconnect handler).
var (hub, context, stream) = CreateHub();
var subscribe = hub.SubscribeInstance(7);
await stream.Entered.WaitAsync(TimeSpan.FromSeconds(5));
context.Abort();
stream.ReleaseStart();
await subscribe.WaitAsync(TimeSpan.FromSeconds(10));
Assert.Equal(new[] { stream.SessionId }, stream.Stopped);
Assert.False(context.Items.ContainsKey(SessionIdKey));
}
[Fact]
public async Task SubscribeInstance_ThenDisconnect_StopsTheSessionExactlyOnce()
{
var (hub, context, stream) = CreateHub();
stream.ReleaseStart();
await hub.SubscribeInstance(7);
Assert.Empty(stream.Stopped);
Assert.Equal(stream.SessionId, context.Items[SessionIdKey]);
await hub.OnDisconnectedAsync(null);
Assert.Equal(new[] { stream.SessionId }, stream.Stopped);
Assert.False(context.Items.ContainsKey(SessionIdKey));
// A second release (an explicit unsubscribe racing the disconnect) is a no-op.
await hub.UnsubscribeInstance();
Assert.Equal(new[] { stream.SessionId }, stream.Stopped);
}
}
tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/ManagementActorTests.cs
+123
View File
@@ -1309,6 +1309,129 @@ public class ManagementActorTests : TestKit, IDisposable
Assert.Equal(envelope.CorrelationId, response.CorrelationId);
}
// ========================================================================
// Site scoping: list filtering + resolve-before-scope oracles
// (arch-review 2026-08-19 P6-4 + P6-5)
// ========================================================================
/// <summary>
/// P6-4: the unfiltered list used to return EVERY site's connections to any
/// authenticated caller, handing a scoped caller exactly the rows
/// <c>GetDataConnection</c> denies them.
/// </summary>
[Fact]
public void ListDataConnections_SiteScopedUser_ReturnsOnlyPermittedSitesConnections()
{
var siteRepo = Substitute.For<ISiteRepository>();
siteRepo.GetAllDataConnectionsAsync(Arg.Any<CancellationToken>())
.Returns(new List<DataConnection>
{
new("ConnA", "OpcUa", 1) { Id = 1 },
new("ConnB", "OpcUa", 2) { Id = 2 },
});
_services.AddScoped(_ => siteRepo);
var actor = CreateActor();
actor.Tell(ScopedEnvelope(new ListDataConnectionsCommand(null), new[] { "1" }, "Deployer"));
var response = ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
Assert.Contains("ConnA", response.JsonData);
Assert.DoesNotContain("ConnB", response.JsonData);
}
[Fact]
public void ListDataConnections_SystemWideUser_ReturnsEveryConnection()
{
var siteRepo = Substitute.For<ISiteRepository>();
siteRepo.GetAllDataConnectionsAsync(Arg.Any<CancellationToken>())
.Returns(new List<DataConnection>
{
new("ConnA", "OpcUa", 1) { Id = 1 },
new("ConnB", "OpcUa", 2) { Id = 2 },
});
_services.AddScoped(_ => siteRepo);
var actor = CreateActor();
actor.Tell(Envelope(new ListDataConnectionsCommand(null), "Deployer"));
var response = ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
Assert.Contains("ConnA", response.JsonData);
Assert.Contains("ConnB", response.JsonData);
}
[Fact]
public void ListDataConnections_ScopedAdministrator_ReturnsEveryConnection()
{
// Administrator bypasses scope even with PermittedSiteIds set — same rule
// EnforceSiteScope applies everywhere else.
var siteRepo = Substitute.For<ISiteRepository>();
siteRepo.GetAllDataConnectionsAsync(Arg.Any<CancellationToken>())
.Returns(new List<DataConnection>
{
new("ConnA", "OpcUa", 1) { Id = 1 },
new("ConnB", "OpcUa", 2) { Id = 2 },
});
_services.AddScoped(_ => siteRepo);
var actor = CreateActor();
actor.Tell(ScopedEnvelope(new ListDataConnectionsCommand(null), new[] { "1" }, "Administrator"));
var response = ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
Assert.Contains("ConnA", response.JsonData);
Assert.Contains("ConnB", response.JsonData);
}
[Fact]
public void ListDataConnections_ExplicitOutOfScopeSiteFilter_ReturnsUnauthorized()
{
var siteRepo = Substitute.For<ISiteRepository>();
siteRepo.GetDataConnectionsBySiteIdAsync(2, Arg.Any<CancellationToken>())
.Returns(new List<DataConnection> { new("ConnB", "OpcUa", 2) { Id = 2 } });
_services.AddScoped(_ => siteRepo);
var actor = CreateActor();
var envelope = ScopedEnvelope(new ListDataConnectionsCommand(2), new[] { "1" }, "Deployer");
actor.Tell(envelope);
var response = ExpectMsg<ManagementUnauthorized>(TimeSpan.FromSeconds(5));
Assert.Equal(envelope.CorrelationId, response.CorrelationId);
}
/// <summary>
/// P6-5: the not-found check used to run before the scope check, so an
/// out-of-scope caller could tell a real instance id from a fake one by the
/// error they got back. Both cases must now answer with the same denial.
/// </summary>
[Fact]
public void DebugSnapshot_NonexistentInstanceForSiteScopedUser_ReturnsUnauthorized()
{
_templateRepo.GetInstanceByIdAsync(999, Arg.Any<CancellationToken>())
.Returns((Instance?)null);
var actor = CreateActor();
var envelope = ScopedEnvelope(new DebugSnapshotCommand(999), new[] { "1" }, "Deployer");
actor.Tell(envelope);
var response = ExpectMsg<ManagementUnauthorized>(TimeSpan.FromSeconds(5));
Assert.Equal(envelope.CorrelationId, response.CorrelationId);
}
[Fact]
public void DebugSnapshot_NonexistentInstanceForSystemWideUser_StillReportsNotFound()
{
// The fail-closed arm must not change the unscoped caller's answer.
_templateRepo.GetInstanceByIdAsync(999, Arg.Any<CancellationToken>())
.Returns((Instance?)null);
var actor = CreateActor();
actor.Tell(Envelope(new DebugSnapshotCommand(999), "Deployer"));
var response = ExpectMsg<ManagementError>(TimeSpan.FromSeconds(5));
Assert.Contains("not found", response.Error);
}
// ========================================================================
// Serialization (finding ManagementService-007)
//
tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/SecuredWriteHandlerTests.cs
+130
View File
@@ -1138,4 +1138,134 @@ public class SecuredWriteHandlerTests : TestKit, IDisposable
var response = ExpectMsg<object>(TimeSpan.FromSeconds(5));
Assert.IsNotType<ManagementUnauthorized>(response);
}
// ------------------------------------------------------------------------
// Resolve-before-scope oracles (arch-review 2026-08-19 P6-5)
//
// Submit/approve/reject used to answer "not found" (and, on approve/reject,
// "is 'Executed', not Pending") BEFORE the scope gate ran, so an out-of-scope
// caller could enumerate row ids and read their lifecycle status by the error
// shape alone. Every out-of-scope caller must now get the SAME scope denial
// whether or not the target exists, and whatever its status.
// ------------------------------------------------------------------------
/// <summary>Seeds a non-Pending (already executed) row for the status-oracle tests.</summary>
private void SeedExecutedWrite(long id) =>
_securedWriteRepo.GetAsync(id, Arg.Any<CancellationToken>()).Returns(new PendingSecuredWrite
{
Id = id, SiteId = "SITE1", ConnectionName = "Gw1", TagPath = "Tag.A",
ValueJson = "true", ValueType = "Boolean", Status = "Executed",
OperatorUser = "alice", SubmittedAtUtc = DateTime.UtcNow,
});
[Fact]
public void Approve_SiteScopedVerifier_NonexistentId_Unauthorized_NotNotFound()
{
_securedWriteRepo.GetAsync(999, Arg.Any<CancellationToken>())
.Returns((PendingSecuredWrite?)null);
var actor = CreateActor();
actor.Tell(ScopedEnvelope(new ApproveSecuredWriteCommand(999, "ok"), "bob", ["3"], "Verifier"));
ExpectMsg<ManagementUnauthorized>(TimeSpan.FromSeconds(5));
}
[Fact]
public void Approve_SiteScopedVerifier_OutOfScopeExecutedRow_Unauthorized_NotStatusError()
{
SeedSiteWithConnection(1, "SITE1", "Gw1", "MxGateway");
SeedExecutedWrite(42);
var actor = CreateActor();
actor.Tell(ScopedEnvelope(new ApproveSecuredWriteCommand(42, "ok"), "bob", ["3"], "Verifier"));
// A ManagementError naming 'Executed' would tell the caller the row exists
// and what happened to it — the oracle this hoist closes.
ExpectMsg<ManagementUnauthorized>(TimeSpan.FromSeconds(5));
Assert.Equal(0, _comms.CallCount);
}
[Fact]
public void Approve_SystemWideVerifier_NonexistentId_StillReportsNotFound()
{
// The fail-closed arm must leave the unscoped caller's answers untouched.
_securedWriteRepo.GetAsync(999, Arg.Any<CancellationToken>())
.Returns((PendingSecuredWrite?)null);
var actor = CreateActor();
actor.Tell(Envelope(new ApproveSecuredWriteCommand(999, "ok"), "bob", "Verifier"));
var response = ExpectMsg<ManagementError>(TimeSpan.FromSeconds(5));
Assert.Contains("not found", response.Error);
}
[Fact]
public void Reject_SiteScopedVerifier_NonexistentId_Unauthorized_NotNotFound()
{
_securedWriteRepo.GetAsync(999, Arg.Any<CancellationToken>())
.Returns((PendingSecuredWrite?)null);
var actor = CreateActor();
actor.Tell(ScopedEnvelope(new RejectSecuredWriteCommand(999, "no"), "bob", ["3"], "Verifier"));
ExpectMsg<ManagementUnauthorized>(TimeSpan.FromSeconds(5));
}
[Fact]
public void Reject_SiteScopedVerifier_OutOfScopeExecutedRow_Unauthorized_NotStatusError()
{
SeedSiteWithConnection(1, "SITE1", "Gw1", "MxGateway");
SeedExecutedWrite(42);
var actor = CreateActor();
actor.Tell(ScopedEnvelope(new RejectSecuredWriteCommand(42, "no"), "bob", ["3"], "Verifier"));
ExpectMsg<ManagementUnauthorized>(TimeSpan.FromSeconds(5));
_securedWriteRepo.DidNotReceiveWithAnyArgs().TryMarkRejectedAsync(
default, default!, default, default, default);
}
[Fact]
public void Reject_SystemWideVerifier_ExecutedRow_StillReportsStatusConflict()
{
// In-scope/unscoped behavior is unchanged: the status error still names
// the status that blocked the transition (the P6-1 CAS shape is intact).
SeedExecutedWrite(42);
var actor = CreateActor();
actor.Tell(Envelope(new RejectSecuredWriteCommand(42, "no"), "bob", "Verifier"));
var response = ExpectMsg<ManagementError>(TimeSpan.FromSeconds(5));
Assert.Contains("Executed", response.Error);
}
[Fact]
public void Submit_SiteScopedOperator_NonexistentSite_Unauthorized_NotNotFound()
{
_siteRepo.GetSiteByIdentifierAsync("GHOST", Arg.Any<CancellationToken>())
.Returns((Site?)null);
var actor = CreateActor();
actor.Tell(ScopedEnvelope(
new SubmitSecuredWriteCommand("GHOST", "Gw1", "Tag.A", "true", "Boolean", "go"),
"alice", ["3"], "Operator"));
ExpectMsg<ManagementUnauthorized>(TimeSpan.FromSeconds(5));
_securedWriteRepo.DidNotReceiveWithAnyArgs().AddAsync(default!, default);
}
[Fact]
public void Submit_SystemWideOperator_NonexistentSite_StillReportsNotFound()
{
_siteRepo.GetSiteByIdentifierAsync("GHOST", Arg.Any<CancellationToken>())
.Returns((Site?)null);
var actor = CreateActor();
actor.Tell(Envelope(
new SubmitSecuredWriteCommand("GHOST", "Gw1", "Tag.A", "true", "Boolean", "go"),
"alice", "Operator"));
var response = ExpectMsg<ManagementError>(TimeSpan.FromSeconds(5));
Assert.Contains("not found", response.Error);
}
}
tests/ZB.MOM.WW.ScadaBridge.Security.Tests/CookieSessionValidatorTests.cs
+45
View File
@@ -681,6 +681,51 @@ public class SecurityOptionsValidatorConfigGuardTests
Assert.True(result.Failed);
Assert.Contains(nameof(SecurityOptions.RoleRefreshThresholdMinutes), result.FailureMessage);
}
// ── arch-review 2026-08-19 P6-9: PerIpMaxFailures must exceed the per-pair ceiling ──
[Fact]
public void Validate_DefaultPerIpMaxFailures_Succeeds()
{
var result = new SecurityOptionsValidator().Validate(name: null, ValidOptions());
Assert.True(result.Succeeded);
}
[Fact]
public void Validate_PerIpMaxFailuresEqualsPerPairCeiling_Fails()
{
var options = ValidOptions();
options.PerIpMaxFailures = options.MaxLoginFailuresPerWindow; // 5 == 5
var result = new SecurityOptionsValidator().Validate(name: null, options);
Assert.True(result.Failed);
Assert.Contains(nameof(SecurityOptions.PerIpMaxFailures), result.FailureMessage);
Assert.Contains(nameof(SecurityOptions.MaxLoginFailuresPerWindow), result.FailureMessage);
}
[Fact]
public void Validate_PerIpMaxFailuresBelowPerPairCeiling_Fails()
{
var options = ValidOptions();
options.PerIpMaxFailures = 3; // < default MaxLoginFailuresPerWindow (5)
var result = new SecurityOptionsValidator().Validate(name: null, options);
Assert.True(result.Failed);
Assert.Contains(nameof(SecurityOptions.PerIpMaxFailures), result.FailureMessage);
}
[Fact]
public void Validate_PerIpMaxFailuresOneAbovePerPairCeiling_Succeeds()
{
var options = ValidOptions();
options.PerIpMaxFailures = options.MaxLoginFailuresPerWindow + 1; // tightest valid config
var result = new SecurityOptionsValidator().Validate(name: null, options);
Assert.True(result.Succeeded);
}
}
#endregion
tests/ZB.MOM.WW.ScadaBridge.Security.Tests/LoginThrottleTests.cs
+115 -1
View File
@@ -136,9 +136,123 @@ public class LoginThrottleTests
var (throttle, clock) = Create();
throttle.RecordFailure("stale", "10.0.0.1");
clock.Advance(TimeSpan.FromMinutes(10)); // window fully expired
// Distinct IPs (not just distinct usernames) so this stays a pure per-pair prune-cadence
// test — hammering PruneEveryNFailures (64) failures from a SINGLE IP would itself trip
// the P6-9 per-IP spray ceiling (default 25) and lock "10.0.0.1" out, which would make
// the IsLockedOut("stale", "10.0.0.1") assertion below fail for an unrelated reason.
for (var i = 0; i < LoginThrottle.PruneEveryNFailures; i++)
throttle.RecordFailure($"fresh{i}", "10.0.0.1"); // cadence tick fires within these
throttle.RecordFailure($"fresh{i}", $"10.0.1.{i}"); // cadence tick fires within these
Assert.False(throttle.IsLockedOut("stale", "10.0.0.1"));
Assert.True(throttle.TrackedKeyCount <= LoginThrottle.PruneEveryNFailures); // "stale" swept
}
// ── arch-review 2026-08-19 P6-9: per-IP spray ceiling ──────────────────────
[Fact]
public void Spray_OnePasswordAcrossManyUsernamesFromOneIp_TripsThePerIpCeiling()
{
// THE FAIL-UNFIXED PROOF: against the old per-{username}|{ip}-only throttle, one
// failure each across PerIpMaxFailures distinct usernames from a single IP never
// reaches MaxLoginFailuresPerWindow (5) on any one pair, so the next attempt from
// that IP — even against a brand-new username — sails through un-throttled. With the
// fix, the shared per-IP counter accumulates across usernames and locks the IP out.
var clock = new TestTimeProvider(Start);
var options = new SecurityOptions { PerIpMaxFailures = 10 };
var throttle = Create(clock, options);
// One failure each across the first (ceiling - 1) usernames: no individual pair ever
// gets near the per-pair ceiling (5), and the IP hasn't tipped over yet either.
for (var i = 0; i < options.PerIpMaxFailures - 1; i++)
{
throttle.RecordFailure($"user{i}", "10.0.0.1");
Assert.False(throttle.IsLockedOut($"user{i}", "10.0.0.1"));
}
// The PerIpMaxFailures-th failure — against yet another brand-new username — tips the
// shared IP counter over the ceiling.
throttle.RecordFailure("user-last", "10.0.0.1");
// Now EVERY username from that IP is refused, including ones never tried before.
Assert.True(throttle.IsLockedOut("brand-new-user", "10.0.0.1"));
}
[Fact]
public void Spray_DistinctIpsAreUnaffectedByOneIpsLockout()
{
var clock = new TestTimeProvider(Start);
var options = new SecurityOptions { PerIpMaxFailures = 10 };
var throttle = Create(clock, options);
for (var i = 0; i < options.PerIpMaxFailures; i++)
throttle.RecordFailure($"user{i}", "10.0.0.1");
Assert.True(throttle.IsLockedOut("anyone", "10.0.0.1"));
Assert.False(throttle.IsLockedOut("anyone", "10.0.0.2")); // untouched IP
Assert.False(throttle.IsLockedOut("user0", "10.0.0.2"));
}
[Fact]
public void PerPairLockout_StillFiresBeforeThePerIpCeiling_NormalFlowUnchanged()
{
// Byte-compatible per-pair behavior: five failures against ONE pair still locks that
// pair out well below the (much higher) default per-IP ceiling.
var clock = new TestTimeProvider(Start);
var throttle = Create(clock);
for (var i = 0; i < 5; i++) throttle.RecordFailure("alice", "10.0.0.1");
Assert.True(throttle.IsLockedOut("alice", "10.0.0.1"));
Assert.False(throttle.IsLockedOut("bob", "10.0.0.1")); // IP itself is nowhere near its ceiling
}
[Fact]
public void NatFriendly_AFewFailuresAcrossAFewUsers_StaysUnderThePerIpCeiling()
{
// A shared NAT gateway where a handful of users mistype their passwords must not be
// punished: a couple of failures each across a few distinct usernames stays well under
// the default PerIpMaxFailures (25).
var clock = new TestTimeProvider(Start);
var throttle = Create(clock); // default PerIpMaxFailures = 25
foreach (var user in new[] { "alice", "bob", "carol", "dave" })
for (var i = 0; i < 3; i++) // 4 users x 3 failures = 12 < 25, and < 5 per pair
throttle.RecordFailure(user, "10.0.0.1");
foreach (var user in new[] { "alice", "bob", "carol", "dave" })
Assert.False(throttle.IsLockedOut(user, "10.0.0.1"));
Assert.False(throttle.IsLockedOut("eve", "10.0.0.1"));
}
[Fact]
public void PerIpLockout_ReleasesOnWindowExpiry()
{
var clock = new TestTimeProvider(Start);
var options = new SecurityOptions { PerIpMaxFailures = 10 };
var throttle = Create(clock, options);
for (var i = 0; i < options.PerIpMaxFailures; i++)
throttle.RecordFailure($"user{i}", "10.0.0.1");
Assert.True(throttle.IsLockedOut("brand-new-user", "10.0.0.1"));
clock.Advance(TimeSpan.FromMinutes(6)); // > default LoginLockoutMinutes (5)
Assert.False(throttle.IsLockedOut("brand-new-user", "10.0.0.1"));
}
[Fact]
public void RecordSuccess_DoesNotResetThePerIpCounter()
{
// A spray that stumbles onto (or already knows) one valid account on the sprayed IP
// must not be able to reset the IP's failure budget via that success and keep spraying.
var clock = new TestTimeProvider(Start);
var options = new SecurityOptions { PerIpMaxFailures = 10 };
var throttle = Create(clock, options);
for (var i = 0; i < options.PerIpMaxFailures - 1; i++)
throttle.RecordFailure($"user{i}", "10.0.0.1"); // one short of the IP ceiling
throttle.RecordSuccess("valid-user", "10.0.0.1"); // a hit mid-spray
throttle.RecordFailure("user-last", "10.0.0.1"); // the IP ceiling is still reached
Assert.True(throttle.IsLockedOut("another-new-user", "10.0.0.1"));
}
}
tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests/SiteCallAuditPurgeTests.cs
+88 -50
View File
@@ -114,6 +114,20 @@ public class SiteCallAuditPurgeTests : TestKit
NullLogger<SiteCallAuditActor>.Instance)));
}
/// <summary>
/// Row-82/row-95 house pattern: every test in this file arms a
/// <c>PurgeIntervalOverride = 100ms</c> ticker. Left running past the test's own
/// assertions it keeps firing against the recording repo — and, worse, against a
/// scope-per-tick DI container whose lifetime the test no longer controls — for the
/// rest of the assembly's run, stranding later tests. Stop it deterministically.
/// </summary>
private void StopTicker(IActorRef actor)
{
Watch(actor);
Sys.Stop(actor);
ExpectTerminated(actor, TimeSpan.FromSeconds(3));
}
// ---------------------------------------------------------------------
// 1. PurgeTick_CallsPurgeTerminal_WithRetentionThreshold
// ---------------------------------------------------------------------
@@ -124,21 +138,27 @@ public class SiteCallAuditPurgeTests : TestKit
var repo = new RecordingRepo { RowsDeletedPerCall = 7 };
// Non-default retention (30 days) so the assertion isn't accidentally
// satisfied by the 365-day default.
CreateActor(repo, FastPurgeOptions(retentionDays: 30));
var actor = CreateActor(repo, FastPurgeOptions(retentionDays: 30));
try
{
AwaitAssert(
() => Assert.True(repo.PurgeThresholds.Count >= 1,
$"expected >= 1 PurgeTerminalAsync call, got {repo.PurgeThresholds.Count}"),
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
AwaitAssert(
() => Assert.True(repo.PurgeThresholds.Count >= 1,
$"expected >= 1 PurgeTerminalAsync call, got {repo.PurgeThresholds.Count}"),
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
// The threshold the actor passed must be ~UtcNow - 30 days. 1-minute
// slack covers scheduling jitter between the tick firing and the assert.
var threshold = repo.PurgeThresholds[0];
var expected = DateTime.UtcNow - TimeSpan.FromDays(30);
Assert.True(
Math.Abs((threshold - expected).TotalMinutes) < 1.0,
$"purge threshold {threshold:o} should be within 1 minute of {expected:o}");
// The threshold the actor passed must be ~UtcNow - 30 days. 1-minute
// slack covers scheduling jitter between the tick firing and the assert.
var threshold = repo.PurgeThresholds[0];
var expected = DateTime.UtcNow - TimeSpan.FromDays(30);
Assert.True(
Math.Abs((threshold - expected).TotalMinutes) < 1.0,
$"purge threshold {threshold:o} should be within 1 minute of {expected:o}");
}
finally
{
StopTicker(actor);
}
}
// ---------------------------------------------------------------------
@@ -149,18 +169,24 @@ public class SiteCallAuditPurgeTests : TestKit
public void PurgeTick_DefaultRetention_Uses365DayThreshold()
{
var repo = new RecordingRepo();
CreateActor(repo, FastPurgeOptions()); // default 365 days
var actor = CreateActor(repo, FastPurgeOptions()); // default 365 days
try
{
AwaitAssert(
() => Assert.True(repo.PurgeThresholds.Count >= 1),
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
AwaitAssert(
() => Assert.True(repo.PurgeThresholds.Count >= 1),
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
var threshold = repo.PurgeThresholds[0];
var expected = DateTime.UtcNow - TimeSpan.FromDays(365);
Assert.True(
Math.Abs((threshold - expected).TotalMinutes) < 1.0,
$"purge threshold {threshold:o} should be within 1 minute of {expected:o}");
var threshold = repo.PurgeThresholds[0];
var expected = DateTime.UtcNow - TimeSpan.FromDays(365);
Assert.True(
Math.Abs((threshold - expected).TotalMinutes) < 1.0,
$"purge threshold {threshold:o} should be within 1 minute of {expected:o}");
}
finally
{
StopTicker(actor);
}
}
// ---------------------------------------------------------------------
@@ -171,16 +197,22 @@ public class SiteCallAuditPurgeTests : TestKit
public void PurgeTick_PurgeThrows_ActorStaysAlive_RetriesNextTick()
{
var repo = new PurgeThrowingRepo();
CreateActor(repo, FastPurgeOptions());
// The singleton must NOT die on a purge fault — a second tick must still
// arrive (continue-on-error). Two purge calls prove the actor survived
// the first throw and the timer kept ticking.
AwaitAssert(
() => Assert.True(repo.PurgeCallCount >= 2,
$"expected >= 2 purge attempts (actor survived the throw), got {repo.PurgeCallCount}"),
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
var actor = CreateActor(repo, FastPurgeOptions());
try
{
// The singleton must NOT die on a purge fault — a second tick must still
// arrive (continue-on-error). Two purge calls prove the actor survived
// the first throw and the timer kept ticking.
AwaitAssert(
() => Assert.True(repo.PurgeCallCount >= 2,
$"expected >= 2 purge attempts (actor survived the throw), got {repo.PurgeCallCount}"),
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
}
finally
{
StopTicker(actor);
}
}
// ---------------------------------------------------------------------
@@ -206,24 +238,30 @@ public class SiteCallAuditPurgeTests : TestKit
.BuildServiceProvider();
var options = FastPurgeOptions(retentionDays: 30);
Sys.ActorOf(Props.Create(() => new SiteCallAuditActor(
var actor = Sys.ActorOf(Props.Create(() => new SiteCallAuditActor(
provider,
options,
NullLogger<SiteCallAuditActor>.Instance)));
try
{
// No reconciliation collaborators were registered, yet the purge tick
// must still fire on the production path.
AwaitAssert(
() => Assert.True(repo.PurgeThresholds.Count >= 1,
"purge timer must arm even when the reconciliation collaborators are absent "
+ $"(SiteCallAudit-007), got {repo.PurgeThresholds.Count} purge calls"),
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
// No reconciliation collaborators were registered, yet the purge tick
// must still fire on the production path.
AwaitAssert(
() => Assert.True(repo.PurgeThresholds.Count >= 1,
"purge timer must arm even when the reconciliation collaborators are absent "
+ $"(SiteCallAudit-007), got {repo.PurgeThresholds.Count} purge calls"),
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
var threshold = repo.PurgeThresholds[0];
var expected = DateTime.UtcNow - TimeSpan.FromDays(30);
Assert.True(
Math.Abs((threshold - expected).TotalMinutes) < 1.0,
$"purge threshold {threshold:o} should be within 1 minute of {expected:o}");
var threshold = repo.PurgeThresholds[0];
var expected = DateTime.UtcNow - TimeSpan.FromDays(30);
Assert.True(
Math.Abs((threshold - expected).TotalMinutes) < 1.0,
$"purge threshold {threshold:o} should be within 1 minute of {expected:o}");
}
finally
{
StopTicker(actor);
}
}
}
tests/ZB.MOM.WW.ScadaBridge.SiteCallAudit.Tests/SiteCallAuditReconciliationTests.cs
+236 -148
View File
@@ -251,6 +251,20 @@ public class SiteCallAuditReconciliationTests : TestKit
NullLogger<SiteCallAuditActor>.Instance)));
}
/// <summary>
/// Row-82/row-95 house pattern: every actor built above arms a fast-repeating
/// reconciliation (and, via <see cref="SiteCallAuditActor"/>, purge) ticker — as fast
/// as 100ms in most of the tests below. Left running past the owning test it keeps
/// firing against test doubles the test has stopped controlling, stranding later
/// tests in the same assembly. Stop it deterministically before the test ends.
/// </summary>
private void StopTicker(IActorRef actor)
{
Watch(actor);
Sys.Stop(actor);
ExpectTerminated(actor, TimeSpan.FromSeconds(3));
}
// ---------------------------------------------------------------------
// 1. AbsentRow_PulledFromSite_IsUpserted
// ---------------------------------------------------------------------
@@ -267,18 +281,24 @@ public class SiteCallAuditReconciliationTests : TestKit
new PullSiteCallsResponse(new[] { row }, MoreAvailable: false));
var repo = new RecordingRepo();
CreateActor(sites, client, repo, FastTickOptions());
AwaitAssert(
() =>
{
Assert.True(repo.Upserted.ContainsKey(id),
"reconciliation tick should upsert the row present at the site but absent centrally");
Assert.Equal("Parked", repo.Upserted[id].Status);
Assert.Equal(siteId, repo.Upserted[id].SourceSite);
},
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
var actor = CreateActor(sites, client, repo, FastTickOptions());
try
{
AwaitAssert(
() =>
{
Assert.True(repo.Upserted.ContainsKey(id),
"reconciliation tick should upsert the row present at the site but absent centrally");
Assert.Equal("Parked", repo.Upserted[id].Status);
Assert.Equal(siteId, repo.Upserted[id].SourceSite);
},
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
}
finally
{
StopTicker(actor);
}
}
// ---------------------------------------------------------------------
@@ -304,18 +324,24 @@ public class SiteCallAuditReconciliationTests : TestKit
new PullSiteCallsResponse(new[] { r1, r2, r3 }, MoreAvailable: false));
var repo = new RecordingRepo();
CreateActor(sites, client, repo, FastTickOptions());
var actor = CreateActor(sites, client, repo, FastTickOptions());
try
{
AwaitAssert(
() => Assert.True(client.Calls.Count >= 2,
$"need at least 2 pulls to assert cursor advancement, got {client.Calls.Count}"),
duration: TimeSpan.FromSeconds(5),
interval: TimeSpan.FromMilliseconds(50));
AwaitAssert(
() => Assert.True(client.Calls.Count >= 2,
$"need at least 2 pulls to assert cursor advancement, got {client.Calls.Count}"),
duration: TimeSpan.FromSeconds(5),
interval: TimeSpan.FromMilliseconds(50));
Assert.Equal(DateTime.MinValue, client.Calls[0].SinceUtc);
Assert.Equal(t3, client.Calls[1].SinceUtc);
// The batch size flows through from options.
Assert.Equal(500, client.Calls[0].BatchSize);
Assert.Equal(DateTime.MinValue, client.Calls[0].SinceUtc);
Assert.Equal(t3, client.Calls[1].SinceUtc);
// The batch size flows through from options.
Assert.Equal(500, client.Calls[0].BatchSize);
}
finally
{
StopTicker(actor);
}
}
// ---------------------------------------------------------------------
@@ -337,19 +363,25 @@ public class SiteCallAuditReconciliationTests : TestKit
.Script(siteB, new PullSiteCallsResponse(new[] { bRow }, MoreAvailable: false));
var repo = new RecordingRepo();
CreateActor(sites, client, repo, FastTickOptions());
AwaitAssert(
() =>
{
// siteA was attempted (and threw) yet siteB's row still landed —
// one offline site must not sink the rest of the tick.
Assert.Contains(client.Calls, c => c.SiteId == "siteA");
Assert.True(repo.Upserted.ContainsKey(bId),
"siteB must be reconciled even though siteA threw");
},
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
var actor = CreateActor(sites, client, repo, FastTickOptions());
try
{
AwaitAssert(
() =>
{
// siteA was attempted (and threw) yet siteB's row still landed —
// one offline site must not sink the rest of the tick.
Assert.Contains(client.Calls, c => c.SiteId == "siteA");
Assert.True(repo.Upserted.ContainsKey(bId),
"siteB must be reconciled even though siteA threw");
},
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
}
finally
{
StopTicker(actor);
}
}
// ---------------------------------------------------------------------
@@ -369,7 +401,7 @@ public class SiteCallAuditReconciliationTests : TestKit
// this actor upserted nothing on its own.
var repo = new RecordingRepo();
var bareProvider = TestSupport.StubRepositoryProvider.Build(repo);
Sys.ActorOf(Props.Create(() => new SiteCallAuditActor(
var bareActor = Sys.ActorOf(Props.Create(() => new SiteCallAuditActor(
bareProvider,
FastTickOptions(),
NullLogger<SiteCallAuditActor>.Instance)));
@@ -379,20 +411,28 @@ public class SiteCallAuditReconciliationTests : TestKit
// collaborator-less actor would have ticked too, had it been wired.
var liveClient = new ScriptedPullClient();
var liveRepo = new RecordingRepo();
CreateActor(
var liveActor = CreateActor(
new StaticEnumerator(new SiteEntry("siteX", "http://siteX:8083")),
liveClient,
liveRepo,
FastTickOptions());
AwaitAssert(
() => Assert.True(liveClient.Calls.Count >= 1),
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
try
{
AwaitAssert(
() => Assert.True(liveClient.Calls.Count >= 1),
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
// The collaborator-less actor never reconciles: it has no client to pull
// with, so it upserts nothing on its own.
Assert.Equal(0, repo.UpsertCallCount);
// The collaborator-less actor never reconciles: it has no client to pull
// with, so it upserts nothing on its own.
Assert.Equal(0, repo.UpsertCallCount);
}
finally
{
StopTicker(liveActor);
StopTicker(bareActor);
}
}
// ---------------------------------------------------------------------
@@ -431,26 +471,32 @@ public class SiteCallAuditReconciliationTests : TestKit
ReconciliationBatchSize = 2,
};
CreateActor(sites, client, repo, options);
var actor = CreateActor(sites, client, repo, options);
try
{
AwaitAssert(
() =>
{
// All three rows reconciled — including the page-2 row that only a
// within-tick continuation pull could have fetched.
Assert.True(repo.Upserted.ContainsKey(p1a.TrackedOperationId));
Assert.True(repo.Upserted.ContainsKey(p1b.TrackedOperationId));
Assert.True(repo.Upserted.ContainsKey(p2.TrackedOperationId),
"the page-2 row must be reconciled within the same tick via the MoreAvailable continuation drain");
},
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
AwaitAssert(
() =>
{
// All three rows reconciled — including the page-2 row that only a
// within-tick continuation pull could have fetched.
Assert.True(repo.Upserted.ContainsKey(p1a.TrackedOperationId));
Assert.True(repo.Upserted.ContainsKey(p1b.TrackedOperationId));
Assert.True(repo.Upserted.ContainsKey(p2.TrackedOperationId),
"the page-2 row must be reconciled within the same tick via the MoreAvailable continuation drain");
},
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
// Exactly two pulls happened (page 1 + the continuation page 2) and the
// second pull's `since` cursor advanced to the page-1 max (t2).
Assert.True(client.Calls.Count >= 2, $"expected >= 2 pulls within the tick, got {client.Calls.Count}");
Assert.Equal(DateTime.MinValue, client.Calls[0].SinceUtc);
Assert.Equal(t2, client.Calls[1].SinceUtc);
// Exactly two pulls happened (page 1 + the continuation page 2) and the
// second pull's `since` cursor advanced to the page-1 max (t2).
Assert.True(client.Calls.Count >= 2, $"expected >= 2 pulls within the tick, got {client.Calls.Count}");
Assert.Equal(DateTime.MinValue, client.Calls[0].SinceUtc);
Assert.Equal(t2, client.Calls[1].SinceUtc);
}
finally
{
StopTicker(actor);
}
}
// ---------------------------------------------------------------------
@@ -484,35 +530,41 @@ public class SiteCallAuditReconciliationTests : TestKit
ReconciliationBatchSize = 2,
};
CreateActor(sites, client, repo, options);
var actor = CreateActor(sites, client, repo, options);
try
{
AwaitAssert(
() => Assert.True(client.CallCount >= 1, "the first reconciliation tick should have pulled"),
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
AwaitAssert(
() => Assert.True(client.CallCount >= 1, "the first reconciliation tick should have pulled"),
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
// The rows it saw were still upserted (idempotent mirror refresh).
// Awaited, not asserted flat: the upsert follows the pull asynchronously,
// so gating on CallCount alone races the repo write under parallel load
// (observed once in a full-suite run, 2026-08-17).
AwaitAssert(
() =>
{
Assert.True(repo.Upserted.ContainsKey(r1.TrackedOperationId));
Assert.True(repo.Upserted.ContainsKey(r2.TrackedOperationId));
},
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
// The rows it saw were still upserted (idempotent mirror refresh).
// Awaited, not asserted flat: the upsert follows the pull asynchronously,
// so gating on CallCount alone races the repo write under parallel load
// (observed once in a full-suite run, 2026-08-17).
AwaitAssert(
() =>
{
Assert.True(repo.Upserted.ContainsKey(r1.TrackedOperationId));
Assert.True(repo.Upserted.ContainsKey(r2.TrackedOperationId));
},
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
// Critical SiteCallAudit-009 invariant: the within-tick drain BROKE on the
// no-progress pin rather than looping to the 50-page ceiling. With a 2s
// tick interval, only the first tick has fired in the window, so the pull
// count reflects ONE tick's within-loop behaviour. A correct break yields
// 1 pull for that tick; we allow a small margin for a possible second tick
// edge, but it must be far below the 50-page within-tick ceiling.
Assert.True(client.CallCount < 10,
$"a single-timestamp saturation pin must break the within-tick drain, not spin to the "
+ $"page ceiling; got {client.CallCount} pulls (an unbounded within-tick loop would be 50+)");
// Critical SiteCallAudit-009 invariant: the within-tick drain BROKE on the
// no-progress pin rather than looping to the 50-page ceiling. With a 2s
// tick interval, only the first tick has fired in the window, so the pull
// count reflects ONE tick's within-loop behaviour. A correct break yields
// 1 pull for that tick; we allow a small margin for a possible second tick
// edge, but it must be far below the 50-page within-tick ceiling.
Assert.True(client.CallCount < 10,
$"a single-timestamp saturation pin must break the within-tick drain, not spin to the "
+ $"page ceiling; got {client.CallCount} pulls (an unbounded within-tick loop would be 50+)");
}
finally
{
StopTicker(actor);
}
}
// ---------------------------------------------------------------------
@@ -546,24 +598,30 @@ public class SiteCallAuditReconciliationTests : TestKit
ReconciliationBatchSize = 2,
};
CreateActor(sites, client, repo, options);
AwaitAssert(
() =>
{
foreach (var r in rows)
var actor = CreateActor(sites, client, repo, options);
try
{
AwaitAssert(
() =>
{
Assert.True(repo.Upserted.ContainsKey(r.TrackedOperationId),
"every row in the single-timestamp burst must reconcile via the "
+ "composite keyset cursor within one tick");
}
},
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
foreach (var r in rows)
{
Assert.True(repo.Upserted.ContainsKey(r.TrackedOperationId),
"every row in the single-timestamp burst must reconcile via the "
+ "composite keyset cursor within one tick");
}
},
duration: TimeSpan.FromSeconds(3),
interval: TimeSpan.FromMilliseconds(50));
// Two pages (2 + 2) drained the burst; well under the 50-page ceiling.
Assert.True(client.CallCount is >= 2 and < 10,
$"expected the burst to drain in a couple of keyset pages, got {client.CallCount}");
// Two pages (2 + 2) drained the burst; well under the 50-page ceiling.
Assert.True(client.CallCount is >= 2 and < 10,
$"expected the burst to drain in a couple of keyset pages, got {client.CallCount}");
}
finally
{
StopTicker(actor);
}
}
// ---------------------------------------------------------------------
@@ -593,11 +651,17 @@ public class SiteCallAuditReconciliationTests : TestKit
ReconciliationBatchSize = 2,
};
CreateActor(sites, client, repo, options);
var evt = ExpectMsg<SiteCallReconciliationPinnedChanged>(TimeSpan.FromSeconds(5));
Assert.Equal(siteId, evt.SiteId);
Assert.True(evt.Pinned, "a legacy site that ignores after_id must publish Pinned=true");
var actor = CreateActor(sites, client, repo, options);
try
{
var evt = ExpectMsg<SiteCallReconciliationPinnedChanged>(TimeSpan.FromSeconds(5));
Assert.Equal(siteId, evt.SiteId);
Assert.True(evt.Pinned, "a legacy site that ignores after_id must publish Pinned=true");
}
finally
{
StopTicker(actor);
}
}
// ---------------------------------------------------------------------
@@ -648,22 +712,30 @@ public class SiteCallAuditReconciliationTests : TestKit
var actor = CreateActor(sites, client, repo, FastTickOptions());
// Wait until the drain is genuinely in flight and blocked inside PullAsync.
await client.Entered.WaitAsync(TimeSpan.FromSeconds(5));
try
{
// Wait until the drain is genuinely in flight and blocked inside PullAsync.
await client.Entered.WaitAsync(TimeSpan.FromSeconds(5));
// The mailbox must still be serving. A generous-but-finite budget: this
// fails at the pre-fix behaviour (the reply only arrives once the pull
// unblocks, which never happens until Release below).
var id = TrackedOperationId.New();
var reply = await actor.Ask<UpsertSiteCallReply>(
new UpsertSiteCallCommand(NewRow(id, sourceSite: siteId)),
TimeSpan.FromSeconds(3));
// The mailbox must still be serving. A generous-but-finite budget: this
// fails at the pre-fix behaviour (the reply only arrives once the pull
// unblocks, which never happens until Release below).
var id = TrackedOperationId.New();
var reply = await actor.Ask<UpsertSiteCallReply>(
new UpsertSiteCallCommand(NewRow(id, sourceSite: siteId)),
TimeSpan.FromSeconds(3));
Assert.True(reply.Accepted);
Assert.Equal(id, reply.TrackedOperationId);
// Let the drain finish so the actor shuts down cleanly.
client.Release();
Assert.True(reply.Accepted);
Assert.Equal(id, reply.TrackedOperationId);
}
finally
{
// Unblock the drain FIRST so the actor is free to process the Stop rather
// than parked inside the blocked pull, then stop it deterministically
// rather than leaving its 100ms ticker running (row-95).
client.Release();
StopTicker(actor);
}
}
[Fact]
@@ -678,16 +750,25 @@ public class SiteCallAuditReconciliationTests : TestKit
var client = new CountingBlockingPullClient();
var repo = new RecordingRepo();
CreateActor(sites, client, repo, FastTickOptions());
var actor = CreateActor(sites, client, repo, FastTickOptions());
await client.Entered.WaitAsync(TimeSpan.FromSeconds(5));
try
{
await client.Entered.WaitAsync(TimeSpan.FromSeconds(5));
// Several tick intervals elapse while the first pass is still blocked.
await Task.Delay(TimeSpan.FromMilliseconds(600));
// Several tick intervals elapse while the first pass is still blocked.
await Task.Delay(TimeSpan.FromMilliseconds(600));
Assert.Equal(1, client.CallCount);
client.Release();
Assert.Equal(1, client.CallCount);
}
finally
{
// Unblock the drain FIRST so the actor is free to process the Stop, then
// stop it deterministically rather than leaving its 100ms ticker running
// (row-95).
client.Release();
StopTicker(actor);
}
}
/// <summary>
@@ -715,25 +796,32 @@ public class SiteCallAuditReconciliationTests : TestKit
var actor = CreateActor(sites, client, repo, FastTickOptions());
// Once the pull has been served the drain is upserting; flood the mailbox
// with ingest commands so the two writers overlap in wall-clock time.
await client.Entered.WaitAsync(TimeSpan.FromSeconds(5));
try
{
// Once the pull has been served the drain is upserting; flood the mailbox
// with ingest commands so the two writers overlap in wall-clock time.
await client.Entered.WaitAsync(TimeSpan.FromSeconds(5));
var asks = Enumerable.Range(0, 6)
.Select(_ => actor.Ask<UpsertSiteCallReply>(
new UpsertSiteCallCommand(NewRow(TrackedOperationId.New(), siteId)),
TimeSpan.FromSeconds(10)))
.ToArray();
var asks = Enumerable.Range(0, 6)
.Select(_ => actor.Ask<UpsertSiteCallReply>(
new UpsertSiteCallCommand(NewRow(TrackedOperationId.New(), siteId)),
TimeSpan.FromSeconds(10)))
.ToArray();
await Task.WhenAll(asks);
Assert.All(asks, ask => Assert.True(ask.Result.Accepted));
await Task.WhenAll(asks);
Assert.All(asks, ask => Assert.True(ask.Result.Accepted));
// The drain's own upserts must have run in the same window.
AwaitAssert(
() => Assert.True(repo.UpsertCount >= 12, $"expected both writers to have run; saw {repo.UpsertCount}"),
TimeSpan.FromSeconds(5));
// The drain's own upserts must have run in the same window.
AwaitAssert(
() => Assert.True(repo.UpsertCount >= 12, $"expected both writers to have run; saw {repo.UpsertCount}"),
TimeSpan.FromSeconds(5));
Assert.Equal(1, repo.MaxObservedConcurrency);
Assert.Equal(1, repo.MaxObservedConcurrency);
}
finally
{
StopTicker(actor);
}
}
/// <summary>
tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogPurgeServiceTests.cs
+16 -17
View File
@@ -410,32 +410,32 @@ public class EventLogPurgeServiceTests : IDisposable
Assert.Empty(exceptions);
}
// ── SiteEventLogging-019: purge runs only on the active node ──
// ── arch-review 2026-08-19 P4-4: purge runs on every node, not just the active one ──
// (supersedes the old SiteEventLogging-019 "active-node-only gate" — DataConnectionActor
// writes site_events on both site nodes, so a standby that skipped its own purge would
// grow its file unbounded on an unreplicated site.)
[Fact]
public void RunPurge_OnStandbyNode_SkipsAllWork()
public void RunPurge_OnStandbyNode_StillRunsThePurge()
{
// SiteEventLogging-019: per design, the daily purge runs on the active
// node only. The standby's local SQLite receives no writes, so purging
// there is unnecessary; we gate the purge tick on the injected
// active-node check and early-exit when it returns false. The row
// inserted here is well past retention, so a real purge would delete
// it — the standby gate must leave it intact.
// P4-4 regression: before the fix, a standby (isActiveNode returning false) early-exited
// the purge tick entirely, leaving this well-past-retention row in place forever on an
// unreplicated standby. The gate is gone — the standby purges its own file exactly like
// the active node.
InsertEventWithTimestamp(DateTimeOffset.UtcNow.AddDays(-31));
Assert.Equal(1, GetEventCount());
var purge = CreatePurgeService(isActiveNode: () => false);
purge.RunPurge();
Assert.Equal(1, GetEventCount());
Assert.Equal(0, GetEventCount());
}
[Fact]
public void RunPurge_OnActiveNode_RunsTheRetentionPurge()
{
// SiteEventLogging-019: when the active-node check returns true the
// service runs the purge as before. Pinned alongside the standby case
// so a future regression that inverts the gate is caught.
// Pinned alongside the standby case above so a future regression that reintroduces
// asymmetric active/standby purge behaviour is caught from either side.
InsertEventWithTimestamp(DateTimeOffset.UtcNow.AddDays(-31));
InsertEventWithTimestamp(DateTimeOffset.UtcNow);
@@ -446,12 +446,11 @@ public class EventLogPurgeServiceTests : IDisposable
}
[Fact]
public void RunPurge_WithNullCheck_FallsBackToRunning()
public void RunPurge_WithNullCheck_StillRuns()
{
// SiteEventLogging-019: when no active-node check is supplied (the
// default for non-clustered hosts and pre-existing tests), the service
// preserves the pre-fix "run on every tick" behaviour rather than
// silently skipping every tick. Backward compatibility guard.
// When no active-node check is supplied (the default for non-clustered hosts and
// pre-existing tests), the service runs the purge — unaffected by P4-4 since the
// check was never gating anything for this caller either.
InsertEventWithTimestamp(DateTimeOffset.UtcNow.AddDays(-31));
var purge = CreatePurgeService(isActiveNode: null);
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/AlarmScriptWatchdogTests.cs
+237
View File
@@ -0,0 +1,237 @@
using Akka.Actor;
using Akka.TestKit;
using Akka.TestKit.Xunit2;
using Microsoft.CodeAnalysis.CSharp.Scripting;
using Microsoft.CodeAnalysis.Scripting;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening;
using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors;
/// <summary>
/// arch-review 2026-08-19 P1-3 — alarm on-trigger runs used to bypass the stuck-script
/// watchdog, the script-error health counter, and <see cref="ScriptRunSummaryRecorder"/>
/// entirely: <c>ScriptRunLauncher.RunAlarmScriptAsync</c> carried none of the machinery its
/// sibling <c>RunScriptAsync</c> (the instance-script path) carries, on the SAME bounded
/// <see cref="ScriptExecutionScheduler"/> pool. A wedged alarm body cost a pool thread
/// forever with no site event, no detach/replace, and <c>DetachedScriptThreads</c> stayed 0;
/// a timed-out or failed on-trigger run never touched the health collector or the run-summary
/// counters either.
///
/// <para>These tests mirror <see cref="StuckScriptWatchdogTests"/> and
/// <see cref="ScriptDeadlineAtEnqueueTests"/> exactly, but drive an <see cref="AlarmActor"/>
/// on-trigger run (via a <c>ValueMatch</c> raise) instead of a <see cref="ScriptActor"/>
/// <c>Call</c>.</para>
/// </summary>
public class AlarmScriptWatchdogTests : TestKit, IDisposable
{
private readonly SharedScriptLibrary _sharedLibrary;
public AlarmScriptWatchdogTests()
{
var compilationService = new ScriptCompilationService(
NullLogger<ScriptCompilationService>.Instance);
_sharedLibrary = new SharedScriptLibrary(
compilationService, NullLogger<SharedScriptLibrary>.Instance);
AlarmWatchdogHooks.Gate = new SemaphoreSlim(0);
}
void IDisposable.Dispose()
{
// Free any still-wedged worker before tearing down, so no test leaves a blocked thread.
AlarmWatchdogHooks.Gate.Release(8);
Shutdown();
}
private static Script<object?> CompileRaw(string code)
{
var scriptOptions = ScriptOptions.Default
.WithReferences(typeof(object).Assembly, typeof(Enumerable).Assembly,
typeof(AlarmWatchdogHooks).Assembly)
.WithImports("System", "System.Collections.Generic", "System.Linq", "System.Threading.Tasks");
var script = CSharpScript.Create<object?>(code, scriptOptions, typeof(ScriptGlobals));
script.Compile();
return script;
}
/// <summary>A body that blocks its worker thread outright — cooperative cancellation is never observed.</summary>
private static Script<object?> WedgeScript() => CompileRaw(
"ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors.AlarmWatchdogHooks.Gate.Wait(); return null;");
private TestActorRef<AlarmActor> BuildAlarmActor(
string alarmName,
Script<object?> compiled,
SiteRuntimeOptions options,
ScriptExecutionScheduler scheduler,
IServiceProvider? serviceProvider,
ISiteHealthCollector? healthCollector = null)
{
var instance = CreateTestProbe().Ref;
var config = new ResolvedAlarm
{
CanonicalName = alarmName,
TriggerType = "ValueMatch",
TriggerConfiguration = "{\"attributeName\":\"Status\",\"matchValue\":\"Critical\"}",
PriorityLevel = 100
};
return ActorOfAsTestActorRef<AlarmActor>(
Props.Create(() => new AlarmActor(
alarmName, "Inst1", instance, config, compiled, _sharedLibrary, options,
NullLogger<AlarmActor>.Instance, null, null, healthCollector, serviceProvider,
null, scheduler, null)),
"alarm-watchdog-" + Guid.NewGuid().ToString("N"));
}
/// <summary>Raises the ValueMatch condition (Normal → Active), spawning the on-trigger run.</summary>
private static void Raise(IActorRef alarm, string instanceName = "Inst1") =>
alarm.Tell(new AttributeValueChanged(
instanceName, "Status", "Status", "Critical", "Good", DateTimeOffset.UtcNow));
private static SiteRuntimeOptions WatchdogOptions() => new()
{
ScriptExecutionTimeoutSeconds = 1,
StuckScriptGraceMs = 200,
MaxScriptCallDepth = 10
};
[Fact]
public void WedgedAlarmScript_DetachesItsWorker_StartsAReplacement_AndDrainsWhenFreed()
{
using var scheduler = new ScriptExecutionScheduler(1);
var siteLog = new FakeSiteEventLogger();
var options = WatchdogOptions();
var wedged = BuildAlarmActor("Wedged", WedgeScript(), options, scheduler,
new SingleServiceProvider(siteLog));
Raise(wedged);
// Timeout (1 s) + grace (200 ms) later the watchdog fires: the worker is detached and
// a replacement thread starts — exactly like the instance-script path.
AwaitAssert(
() => Assert.Equal(1, scheduler.DetachedThreadCount),
TimeSpan.FromSeconds(15));
AwaitAssert(
() => Assert.Contains(siteLog.OfType("script"), r =>
r.Severity == "Error" &&
r.Message.Contains("still executing", StringComparison.OrdinalIgnoreCase) &&
r.Message.Contains("Wedged") &&
r.Message.Contains("DETACHED")),
TimeSpan.FromSeconds(5));
// The replacement worker is live: a fresh alarm on-trigger run completes even though
// the pool's only original thread is still blocked. Before this fix there was no
// detach at all, so a second wedge would have starved the whole pool.
var healthy = BuildAlarmActor("Healthy", CompileRaw("return 7;"), options, scheduler, null);
Raise(healthy);
AwaitAssert(
() => Assert.Equal(0, healthy.UnderlyingActor.RunsInFlight),
TimeSpan.FromSeconds(15));
// Freeing the wedged body lets the detached worker exit rather than pull more work,
// so the pool does not silently end up double-sized.
AlarmWatchdogHooks.Gate.Release();
AwaitAssert(
() => Assert.Equal(0, scheduler.DetachedThreadCount),
TimeSpan.FromSeconds(15));
Assert.Equal(1, scheduler.MaximumConcurrencyLevel);
}
[Fact]
public void AlarmRunTimedOutAtDequeue_IncrementsHealthCollectorScriptErrorCount()
{
using var scheduler = new ScriptExecutionScheduler(1);
var siteLog = new FakeSiteEventLogger();
var health = new SiteHealthCollector();
// Occupy the sole worker with a first alarm's on-trigger run, so the second alarm's
// run is queued behind it (mirrors ScriptDeadlineAtEnqueueTests' proven pattern).
var blockerOptions = new SiteRuntimeOptions
{ ScriptExecutionTimeoutSeconds = 120, StuckScriptGraceMs = 120_000 };
var blocker = BuildAlarmActor("Blocker", WedgeScript(), blockerOptions, scheduler, null);
Raise(blocker);
AwaitAssert(() => Assert.Equal(1, scheduler.BusyThreadCount), TimeSpan.FromSeconds(15));
// The target alarm has a 1 s budget — by the time the blocker frees the worker, this
// run's deadline has already passed, so it takes the timeout (OperationCanceledException)
// path at dequeue without ever executing its body.
var lateOptions = new SiteRuntimeOptions
{ ScriptExecutionTimeoutSeconds = 1, StuckScriptGraceMs = 1000 };
var late = BuildAlarmActor("Late", CompileRaw("return 1;"), lateOptions, scheduler,
new SingleServiceProvider(siteLog), health);
Raise(late);
AwaitCondition(() => scheduler.QueueDepth >= 1, TimeSpan.FromSeconds(15),
TimeSpan.FromMilliseconds(25));
var queuedFor = System.Diagnostics.Stopwatch.StartNew();
AwaitCondition(
() => queuedFor.Elapsed >= TimeSpan.FromSeconds(1.5) || scheduler.QueueDepth == 0,
TimeSpan.FromSeconds(15), TimeSpan.FromMilliseconds(25));
Assert.True(scheduler.QueueDepth >= 1,
"the late alarm run left the queue before its budget elapsed, so this run never " +
"exercised the timeout-at-dequeue path");
AlarmWatchdogHooks.Gate.Release();
// Before this fix, RunAlarmScriptAsync's OperationCanceledException catch never
// touched the health collector at all — this stayed 0 forever regardless of how many
// alarm on-trigger runs timed out.
AwaitAssert(
() => Assert.Equal(1, health.CollectReport("site-1").ScriptErrorCount),
TimeSpan.FromSeconds(15));
// The shed-at-dequeue run never held a worker (it was cancelled before its body ran),
// so — unlike the wedged-worker test above — it must NOT be reported as stuck.
Assert.DoesNotContain(siteLog.OfType("script"),
r => r.Message.Contains("'Late'", StringComparison.Ordinal) &&
r.Message.Contains("still executing", StringComparison.OrdinalIgnoreCase));
AwaitCondition(() => scheduler.BusyThreadCount == 0 && scheduler.DetachedThreadCount == 0,
TimeSpan.FromSeconds(20), TimeSpan.FromMilliseconds(25));
}
[Fact]
public async Task AlarmRunCompletion_IsRecordedByScriptRunSummaryRecorder()
{
using var scheduler = new ScriptExecutionScheduler(2);
var siteLog = new FakeSiteEventLogger();
var recorder = new ScriptRunSummaryRecorder();
var options = new SiteRuntimeOptions
{ ScriptExecutionTimeoutSeconds = 30, StuckScriptGraceMs = 30_000 };
// Before this fix, RunAlarmScriptAsync never resolved or touched
// ScriptRunSummaryRecorder at all — a flush after any number of alarm runs would
// always report zero activity for them.
var alarm = BuildAlarmActor("Recorded", CompileRaw("return 42;"), options, scheduler,
new SingleServiceProvider(siteLog, summaryRecorder: recorder));
Raise(alarm);
AwaitAssert(() => Assert.Equal(0, alarm.UnderlyingActor.RunsInFlight), TimeSpan.FromSeconds(15));
var emitted = await recorder.FlushAsync(siteLog);
Assert.True(emitted);
var rows = siteLog.OfType("script");
var summaryRow = Assert.Single(rows, r => r.Source == "ScriptRunSummary");
Assert.Equal("1 runs: 1 completed, 0 failed, 0 timed out across 1 scripts", summaryRow.Message);
Assert.NotNull(summaryRow.Details);
Assert.Contains("AlarmActor:Recorded", summaryRow.Details);
}
}
/// <summary>
/// Test hook the alarm stuck-script watchdog tests use to block a script-execution thread
/// deterministically: the compiled body waits on <see cref="Gate"/>, which the test releases
/// once it has observed the detach (or at teardown).
/// </summary>
public static class AlarmWatchdogHooks
{
/// <summary>Gate a blocking test script waits on; reset per test.</summary>
public static SemaphoreSlim Gate = new(0);
}
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs
+5 -1
View File
@@ -352,7 +352,11 @@ public class InstanceActorTests : TestKit, IDisposable
var overrides = await _storage.GetStaticOverridesAsync("PumpRedeploy");
Assert.Single(overrides);
// Clear overrides (simulates what DeploymentManager does on redeployment)
// Clear overrides BEFORE the successor exists — the ordering DeploymentManager now
// guarantees (arch-review 2026-08-19 P1-2). It used to clear inside the deploy's
// persistence task, i.e. after creating the actor, so this "simulation" pinned an
// ordering the production path did not have; the end-to-end contract lives in
// RedeployStaticOverrideResetTests.
await _storage.ClearStaticOverridesAsync("PumpRedeploy");
overrides = await _storage.GetStaticOverridesAsync("PumpRedeploy");

Some files were not shown because too many files have changed in this diff Show More